The Stats Map

      • active statistical inference
      • anisotropic distribution
      • anytime-valid
      • anytime-valid p-values
      • asymptotic confidence sequences
      • average calibration error
      • Banach space
      • basic inequalities
      • basic matrix inequalities
      • Bayes factors
      • Bayesian interpretation of probability
      • Bayesian nonparametrics
      • Bayesian parametrics
      • Bayesian statistics
      • Bernstein von-Mises theorem
      • Berry-Esseen bounds
      • betting strategies
      • BH procedure
      • bounded difference inequalities
      • calibration
      • Catoni-Giulini M-estimator
      • causal inference
      • cdf concentration
      • cdf estimation
      • central limit theorems
      • chaining
      • characteristic function
      • Chernoff bounds specify e-values
      • Chernoff method
      • chi-squared divergence
      • CLTs in Banach spaces
      • coarsened filtrations can increase power
      • comparing forecasters by betting
      • concentration in Banach spaces
      • concentration inequalities
      • concentration of functions
      • concentration of measure
      • concentration of self-bounding functions
      • concentration via covering
      • conditional independence testing
      • confidence intervals
      • confidence sequences
      • confidence sequences for quantiles
      • confidence sequences via conjugate mixtures
      • confidence sequences via predictable plug-ins
      • conformal prediction
      • conjugate transpose
      • contextual bandit
      • covering and packing
      • credible intervals
      • current statistical practice combines the Fisherian and Neyman-Pearson perspectives
      • deep density estimation
      • density estimation
      • differential privacy
      • Dirichlet process
      • distributional distance
      • Doob decomposition
      • doubly robust estimator
      • duality between hypothesis tests and CIs
      • Dudley chaining
      • Dudley's entropy bound
      • e-BH procedure
      • e-process
      • e-value
      • e-values enable post-hoc hypothesis testing
      • Efron-Stein inequality
      • empirical Bernstein bounds
      • empirical process theory
      • ensemble learning
      • entropy number
      • estimating means by betting
      • evidence against the null
      • evidence is quantifiable in small-worlds
      • exchangeable distribution
      • exponential families
      • exponential inequalities
      • external randomization
      • f-divergence
      • FDR control
      • Fisher information
      • Fisher information distance
      • Fisher's paradigm
      • fixed-time
      • fork-convex
      • foundations of statistics
      • frequentist interpretation of probability
      • frequentist statistics
      • from boundedness to variance adaptivity
      • game theory
      • game-theoretic concentration inequalities
      • game-theoretic convergence of opinions
      • game-theoretic expectations
      • game-theoretic hypothesis testing
      • game-theoretic LLN
      • game-theoretic probabilities
      • game-theoretic probability
      • game-theoretic statistics
      • Gaussian complexity
      • Gaussian process regression
      • Gaussian sequence model
      • generic chaining
      • Glivenko-Cantelli classes
      • GRO e-variable
      • GROW e-variable
      • growth rate conditions in sequential testing
      • heavy-tailed scalar concentration
      • Hellinger distance
      • Hermitian matrix
      • hidden Markov model
      • hilbert space
      • histograms
      • Hölder space
      • hypothesis testing
      • ideal metrics
      • infinitely divisible distribution
      • information processing inequality
      • information theory
      • instrumentalist theory of probability
      • inverse problems
      • irregular problems in hypothesis testing
      • isotropic distributions
      • issues with p-values
      • Jeffreys prior
      • Jeffreys' paradigm of hypothesis testing
      • Karlin-Rubin theorem
      • Kelly betting
      • kernel density estimation
      • kernel regression
      • kernel trick
      • KL divergence
      • knn
      • KS distance
      • lady tasting tea
      • law of likelihood
      • light-tailed maximal inequalities
      • light-tailed scalar concentration
      • likelihood principle
      • likelihood-ratio test
      • Lindeberg-Feller CLT
      • Lindeberg-Levy CLT
      • Linear Regression
      • linear regression diagnostics
      • linear smoothers
      • local differential privacy
      • local polynomial regression
      • Loewner order
      • log-concave distribution
      • Lp norm
      • Lyapunov CLT
      • M-estimation
      • marginal consistency
      • Markovian alternatives
      • martingale CLT
      • martingale concentration
      • matrix inequalities
      • maximal inequalities
      • maximizing log-wealth
      • Mayo's error statistics
      • median-of-means
      • Mercer kernel
      • method of moments for concentration
      • method of moments for estimation
      • metric entropy
      • metric space
      • MGF
      • minimal sufficiency
      • MLE
      • model selection
      • model-X assumption
      • monotone likelihood ratio
      • multi-group calibration
      • multi-group consistency
      • multiarmed bandit
      • multiple testing
      • multivariate concentration
      • multivariate heavy-tailed concentration
      • multivariate light-tailed concentration
      • Nash equilibrium
      • Neyman-Pearson lemma
      • Neyman-Pearson lemma for discrete distributions
      • Neyman-Pearson paradigm
      • Neyman-Pearson paradigm with losses
      • nonparametric classification
      • nonparametric density estimation
      • nonparametric regression
      • numeraire e-variable
      • online calibration
      • online gradient descent
      • online marginal estimation
      • Online Newton Step
      • operator norm
      • operator norm inequalities
      • optimal transport
      • optimization perspective on Markov's inequality
      • optional continuation
      • optional stopping
      • Orlicz norm
      • p-hacking
      • p-value
      • PAC-Bayes
      • parametric density estimation
      • parametric versus nonparametric statistics
      • partitions and trees
      • permutation test
      • permutation testing by betting
      • Petrov's CLT template
      • pinball loss
      • Pinelis approach to concentration
      • portfolio optimization
      • post-hoc confidence sequences via e-processes
      • post-hoc hypothesis testing
      • post-hoc hypothesis testing with losses
      • post-hoc valid confidence sequences
      • PRDS
      • prediction-powered inference
      • quantile estimation
      • quantitative CLT template with ideal metrics
      • Rademacher complexity
      • randomized inequalities
      • Rao-Blackwell theorem
      • REGROW e-variable
      • reinforcement learning
      • representer theorem
      • reverse information projection (RIPr)
      • RKHS
      • rkhs regression
      • Royall's three questions
      • safe, anytime-valid inference (SAVI)
      • score function
      • sequential hypothesis testing
      • sequential statistics
      • Simpson's paradox
      • small worlds vs large worlds
      • splines
      • squared error
      • statistical decision theory
      • statistical inference
      • stopping-time
      • strong approximations
      • sub-exponential distributions
      • sub-Gaussian distributions
      • sub-Gaussian process
      • sub-psi process
      • sufficiency and the likelihood
      • sufficient statistic
      • supermartingale
      • supervised learning
      • survey sampling
      • t-test
      • techniques for multivariate concentration
      • test-martingale
      • testing by betting—composite vs composite
      • testing by betting—simple vs composite
      • testing by betting—simple vs simple
      • testing by betting—two-sample testing
      • testing exchangeability
      • testing forecasters by betting
      • testing group invariance
      • time-uniform
      • total variation distance
      • two-sample testing
      • u-statistics
      • uncertainty quantification
      • uniformly most powerful test
      • universal inference
      • v-statistics
      • variational approach to concentration
      • variational autoencoders
      • variational inference
      • Ville's inequality
      • Wald interval
      • Wald test
      • Warner's randomized response
      • Wasserstein Distance
      • wavelets
      • weighted least squares
      • zero sum game

    evidence against the null

    Last modified Sep 02, 20241 min read

    In hypothesis testing, we are interested in quantifying evidence against the null so that we can decide when to reject it. Many statistical tools have been invented for this task, including p-values, e-values, and Bayes factors.

    As a philosophical aside, note that the word “evidence” is well-defined in hypothesis testing since it is a small-world (see small worlds vs large worlds and evidence is quantifiable in small-worlds.


    Graph View

    Backlinks

    • Bayes factors
    • Fisher's paradigm
    • Neyman-Pearson paradigm
    • Ville's inequality
    • current statistical practice combines the Fisherian and Neyman-Pearson perspectives
    • e-process
    • e-value
    • p-value
        • active statistical inference
        • anisotropic distribution
        • anytime-valid
        • anytime-valid p-values
        • asymptotic confidence sequences
        • average calibration error
        • Banach space
        • basic inequalities
        • basic matrix inequalities
        • Bayes factors
        • Bayesian interpretation of probability
        • Bayesian nonparametrics
        • Bayesian parametrics
        • Bayesian statistics
        • Bernstein von-Mises theorem
        • Berry-Esseen bounds
        • betting strategies
        • BH procedure
        • bounded difference inequalities
        • calibration
        • Catoni-Giulini M-estimator
        • causal inference
        • cdf concentration
        • cdf estimation
        • central limit theorems
        • chaining
        • characteristic function
        • Chernoff bounds specify e-values
        • Chernoff method
        • chi-squared divergence
        • CLTs in Banach spaces
        • coarsened filtrations can increase power
        • comparing forecasters by betting
        • concentration in Banach spaces
        • concentration inequalities
        • concentration of functions
        • concentration of measure
        • concentration of self-bounding functions
        • concentration via covering
        • conditional independence testing
        • confidence intervals
        • confidence sequences
        • confidence sequences for quantiles
        • confidence sequences via conjugate mixtures
        • confidence sequences via predictable plug-ins
        • conformal prediction
        • conjugate transpose
        • contextual bandit
        • covering and packing
        • credible intervals
        • current statistical practice combines the Fisherian and Neyman-Pearson perspectives
        • deep density estimation
        • density estimation
        • differential privacy
        • Dirichlet process
        • distributional distance
        • Doob decomposition
        • doubly robust estimator
        • duality between hypothesis tests and CIs
        • Dudley chaining
        • Dudley's entropy bound
        • e-BH procedure
        • e-process
        • e-value
        • e-values enable post-hoc hypothesis testing
        • Efron-Stein inequality
        • empirical Bernstein bounds
        • empirical process theory
        • ensemble learning
        • entropy number
        • estimating means by betting
        • evidence against the null
        • evidence is quantifiable in small-worlds
        • exchangeable distribution
        • exponential families
        • exponential inequalities
        • external randomization
        • f-divergence
        • FDR control
        • Fisher information
        • Fisher information distance
        • Fisher's paradigm
        • fixed-time
        • fork-convex
        • foundations of statistics
        • frequentist interpretation of probability
        • frequentist statistics
        • from boundedness to variance adaptivity
        • game theory
        • game-theoretic concentration inequalities
        • game-theoretic convergence of opinions
        • game-theoretic expectations
        • game-theoretic hypothesis testing
        • game-theoretic LLN
        • game-theoretic probabilities
        • game-theoretic probability
        • game-theoretic statistics
        • Gaussian complexity
        • Gaussian process regression
        • Gaussian sequence model
        • generic chaining
        • Glivenko-Cantelli classes
        • GRO e-variable
        • GROW e-variable
        • growth rate conditions in sequential testing
        • heavy-tailed scalar concentration
        • Hellinger distance
        • Hermitian matrix
        • hidden Markov model
        • hilbert space
        • histograms
        • Hölder space
        • hypothesis testing
        • ideal metrics
        • infinitely divisible distribution
        • information processing inequality
        • information theory
        • instrumentalist theory of probability
        • inverse problems
        • irregular problems in hypothesis testing
        • isotropic distributions
        • issues with p-values
        • Jeffreys prior
        • Jeffreys' paradigm of hypothesis testing
        • Karlin-Rubin theorem
        • Kelly betting
        • kernel density estimation
        • kernel regression
        • kernel trick
        • KL divergence
        • knn
        • KS distance
        • lady tasting tea
        • law of likelihood
        • light-tailed maximal inequalities
        • light-tailed scalar concentration
        • likelihood principle
        • likelihood-ratio test
        • Lindeberg-Feller CLT
        • Lindeberg-Levy CLT
        • Linear Regression
        • linear regression diagnostics
        • linear smoothers
        • local differential privacy
        • local polynomial regression
        • Loewner order
        • log-concave distribution
        • Lp norm
        • Lyapunov CLT
        • M-estimation
        • marginal consistency
        • Markovian alternatives
        • martingale CLT
        • martingale concentration
        • matrix inequalities
        • maximal inequalities
        • maximizing log-wealth
        • Mayo's error statistics
        • median-of-means
        • Mercer kernel
        • method of moments for concentration
        • method of moments for estimation
        • metric entropy
        • metric space
        • MGF
        • minimal sufficiency
        • MLE
        • model selection
        • model-X assumption
        • monotone likelihood ratio
        • multi-group calibration
        • multi-group consistency
        • multiarmed bandit
        • multiple testing
        • multivariate concentration
        • multivariate heavy-tailed concentration
        • multivariate light-tailed concentration
        • Nash equilibrium
        • Neyman-Pearson lemma
        • Neyman-Pearson lemma for discrete distributions
        • Neyman-Pearson paradigm
        • Neyman-Pearson paradigm with losses
        • nonparametric classification
        • nonparametric density estimation
        • nonparametric regression
        • numeraire e-variable
        • online calibration
        • online gradient descent
        • online marginal estimation
        • Online Newton Step
        • operator norm
        • operator norm inequalities
        • optimal transport
        • optimization perspective on Markov's inequality
        • optional continuation
        • optional stopping
        • Orlicz norm
        • p-hacking
        • p-value
        • PAC-Bayes
        • parametric density estimation
        • parametric versus nonparametric statistics
        • partitions and trees
        • permutation test
        • permutation testing by betting
        • Petrov's CLT template
        • pinball loss
        • Pinelis approach to concentration
        • portfolio optimization
        • post-hoc confidence sequences via e-processes
        • post-hoc hypothesis testing
        • post-hoc hypothesis testing with losses
        • post-hoc valid confidence sequences
        • PRDS
        • prediction-powered inference
        • quantile estimation
        • quantitative CLT template with ideal metrics
        • Rademacher complexity
        • randomized inequalities
        • Rao-Blackwell theorem
        • REGROW e-variable
        • reinforcement learning
        • representer theorem
        • reverse information projection (RIPr)
        • RKHS
        • rkhs regression
        • Royall's three questions
        • safe, anytime-valid inference (SAVI)
        • score function
        • sequential hypothesis testing
        • sequential statistics
        • Simpson's paradox
        • small worlds vs large worlds
        • splines
        • squared error
        • statistical decision theory
        • statistical inference
        • stopping-time
        • strong approximations
        • sub-exponential distributions
        • sub-Gaussian distributions
        • sub-Gaussian process
        • sub-psi process
        • sufficiency and the likelihood
        • sufficient statistic
        • supermartingale
        • supervised learning
        • survey sampling
        • t-test
        • techniques for multivariate concentration
        • test-martingale
        • testing by betting—composite vs composite
        • testing by betting—simple vs composite
        • testing by betting—simple vs simple
        • testing by betting—two-sample testing
        • testing exchangeability
        • testing forecasters by betting
        • testing group invariance
        • time-uniform
        • total variation distance
        • two-sample testing
        • u-statistics
        • uncertainty quantification
        • uniformly most powerful test
        • universal inference
        • v-statistics
        • variational approach to concentration
        • variational autoencoders
        • variational inference
        • Ville's inequality
        • Wald interval
        • Wald test
        • Warner's randomized response
        • Wasserstein Distance
        • wavelets
        • weighted least squares
        • zero sum game

      Ben Chugg © 2024

      • GitHub
      • Twitter