Fundamentals — A Guided Tour of Modern Statistics

Notes / Hypothesis Testing

3.1 Fundamentals

Hypothesis testing is just a particular type of decision problem. As usual, we assume that the data is sampled according to $X \sim \mathbb {P}_\theta $ and that $\mathbb {P}_\theta $ belongs to the model $\mathcal {P} = \{\mathbb {P}_\theta : \theta \in \Omega \}$. In addition to the standard setup, we divide the models in $\mathcal {P}$ into two disjoint subclasses known as “hypotheses”: \[ \begin {aligned} H_0 &: \theta \in \Omega _0 \subset \Omega && \text {(null hypothesis)} \\ H_1 &: \theta \in \Omega _1 = \Omega \setminus \Omega _0 && \text {(alternative hypothesis)} \end {aligned} \]

Our goal is to infer which hypothesis is correct. This can be cast as classification, so our decision space is \[ \mathcal {D} = \{\text {accept } H_0,\ \text {reject } H_0\}. \]


	$\theta \in \Omega _0$	$\theta \in \Omega _1$

Reject $H_0$	1 (Type I Error)	0 (Good)

Accept $H_0$	0 (Good)	1 (Type II Error)

Table 1. Canonical loss function $L(\theta , d)$.

We have two types of error which induce a loss. A Type I error or false positive occurs when we reject $H_0$ when it is in fact true. Similarly a Type II error or false negative occurs when we accept $H_0$ when it is false. Define a test function $\phi (X)$ as \[ \phi (X) = \mathbb {P}\!\left (\delta _\phi (X, U) = \text {Reject } H_0 \,\big |\, X\right ). \] where $U$ is as usual a uniform random variable independent of $X$.

Now we talk about Neyman-Pearson paradigm, which simply bounds it and focuses on minimizing the Type II error. Especially, we require a level $\alpha $ test \[ \sup _{\theta _0 \in \Omega _0} \mathbb {E}_{\theta _0} \phi (X) = \sup _{\theta _0 \in \Omega _0} \beta (\theta _0) \le \alpha . \] that maximizes the power $\beta (\theta _1) = \mathbb {E}_{\theta _1}[\phi (X)]$ for each $\theta _1 \in \Omega _1$. Such a test is called uniformly most powerful (UMP).

Consider the “simple” test \[ \begin {aligned} H_0 &: X \sim p_0 \\ H_1 &: X \sim p_1 \end {aligned} \] where $p_0, p_1$ denote the densities of $\mathbb {P}_{\theta _0}, \mathbb {P}_{\theta _1}$ with respect to some common measure $\mu $, and we call $\mathbb {E}_{p_1}[\phi (X)]$ the power of the test $\phi $. Our goal in the simple case can be compactly described as: \[ \begin {aligned} \max _{\phi } \quad & \mathbb {E}_{p_1}[\phi (X)] \\ \text {s.t.} \quad & \mathbb {E}_{p_0}[\phi (X)] \le \alpha . \end {aligned} \]

3.1.1 Neyman-Pearson Lemma

Lemma 3.3 (Neyman-Pearson).

(i)

Existence. For testing $H_0 : p_0$ vs. $H_1 : p_1$, there is a test $\phi (X)$ and a constant $k$ such that:

(a): $\mathbb {E}_{p_0}\phi (X) = \alpha $ (size $=$ level).
(b): $\phi (x) = \begin {cases} 1 & \text {if } \dfrac {p_1(x)}{p_0(x)} > k \text { \emph {(always reject if likelihood ratio is $> k$)}.} \\[2pt] 0 & \text {if } \dfrac {p_1(x)}{p_0(x)} < k \text { \emph {(always accept if likelihood ratio is $< k$)}.} \end {cases}$

Such a test is called a likelihood ratio test (LRT).

(ii)

Sufficient. If a test satisfies (a),(b) for some constant $k$, it is most powerful for testing $H_0 : p_0$ vs. $H_1 : p_1$ at level $\alpha $. (Hence, the LRT from part (i) is most powerful.)

(iii)

Necessary. If a test $\phi $ is MP at level $\alpha $ then it satisfies (b) for some $k$, and it also satisfies (a) unless there exists a test of size $< \alpha $ with power $1$. (In the latter case, we did not need to expend all of budgeted Type I error.)

Example 3.5 (One parameter exponential family). Consider the case where $X_1, \ldots , X_n \stackrel {\mathrm {i.i.d.}}{\sim } p_\theta (x) \propto h(x)\exp (\theta T(x))$, and we are interested in testing \[ H_0 : \theta = \theta _0 \quad \text {vs.} \quad H_1 : \theta = \theta _1. \] We want an MP test at level $\alpha $. The likelihood ratio is \[ \frac {\prod _i p_{\theta _1}(x_i)}{\prod _i p_{\theta _0}(x_i)} \propto \exp \!\left ((\theta _1 - \theta _0)\sum _i T(x_i)\right ). \] Since the exponential is just a monotone transformation, an MP test will reject for large $(\theta _1 - \theta _0)\sum _i T(x_i)$. Assuming $\theta _1 > \theta _0$, we will reject for large $\sum _i T(x_i)$. That is, an MP test has the form \[ \phi (x) = \begin {cases} 1 & \text {if } \sum _i T(x_i) > k \\ \gamma & \text {if } \sum _i T(x_i) = k \\ 0 & \text {if } \sum _i T(x_i) < k, \end {cases} \] where $k, \gamma $ are chosen to satisfy the size constraint \[ \alpha = \mathbb {E}_{\theta _0}\phi (X) = \mathbb {P}_{\theta _0}\!\left [\sum _i T(X_i) > k\right ] + \gamma \mathbb {P}_{\theta _0}\!\left [\sum _i T(X_i) = k\right ]. \] Note that $\sum _i T(x_i)$ has no explicit $\theta $ dependence and that $k, \gamma $ do not depend on $\theta _1$ (assuming $\theta _1 > \theta _0$). This means $\phi $ is in fact UMP for testing \[ H_0 : \theta = \theta _0 \quad \text {vs.} \quad H_1 : \theta > \theta _0. \] Here, $H_1$ is an example of a one-sided alternative, which arises when the parameter values of interest lie on only one side of the real-valued parameter $\theta _0$.

3.1.2 Monotone Likelihood Ratio

In the above example, we were able to extend our MP test for a simple hypothesis to a UMP test for a one-sided hypothesis. This phenomenon is not unique to exponential families. We can get the same behavior whenever the models have a so-called monotone likelihood ratio.

3.1.3 Composite Null

Now we introduce a new strategy to deal with cases with a composite null. Consider the case with a simple alternative: \[ \begin {aligned} H_0 &: X \sim f_\theta , \quad \theta \in \Omega _0 \\ H_1 &: X \sim g, \end {aligned} \] where $g$ is known. We now impose a prior distribution $\Lambda $ on $\Omega _0$. So we consider the new hypothesis \[ H_\Lambda : X \sim h_\Lambda (x) = \int _{\Omega _0} f_\theta (x)\, d\Lambda (\theta ), \] where $h_\Lambda (x)$ is the marginal distribution of $X$ induced by $\Lambda $. In order to reduce the problem to a simple versus simple case, let us test $H_\Lambda $ against $H_1$. Let $\beta _\Lambda $ be the power of the MP level-$\alpha $ test $\phi _\Lambda $ for testing $H_\Lambda $ vs. $g$.

Hence, $\Lambda $ will be the least favorable distribution if the MP test under $\Lambda $ has smaller power than the MP test under any other prior distribution. The following theorem can help us to deal with the case of composite null by using the notion of least favorable distribution, which tells that if we choose $\Lambda $ in the right way, we can get the MP.

3.1.4 Method of Undetermined Multipliers

Unbiasedness enforces the appealing property that the probability of rejection is greater under any alternative distribution than it is under any null distribution. A uniformly most powerful test is always unbiased if it exists.

The following lemma tells us we can find a UMPU test by looking only at $\alpha $-similar tests.

Proof. Firstly, because $\phi _0$ is UMP $\alpha $-similar tests, it is at least as powerful as $\phi _\alpha (X) \equiv \alpha $, and the power of $\phi _0$ on $\Omega _1$ is therefore $\ge \alpha $. Hence, $\phi _0$ is unbiased. □

Let us test $H_0: \theta = \theta _0$ vs. $H_1: \theta \neq \theta _0$ when $X$ is distributed according some member of the one-dimensional exponential family \[ p_\theta (x) = h(x) \exp \left (\theta T(x) - A(\theta )\right ) \] We have the $\alpha -$level condition \[ \beta _\phi (\theta _0) = \alpha , \] and the derivative condition \[ \frac {{\mathop{}\!\mathrm{d}} }{{\mathop{}\!\mathrm{d}} \theta } \beta _\phi (\theta _0) = 0. \] As now we have multiple constraints, we introduce the method of undetermined multipliers.

In this setting, $H_0 : \theta = \theta _0$. We will fix a simple alternative $\theta = \theta ' \neq \theta _0$ and hope that our best test has no $\theta '$ dependence. We would like to maximize power $\int \phi (x) p_{\theta '}(x) \, d\mu (x)$ subject to \begin{align*} \int \phi (x) p_{\theta _0}(x) \, d\mu (x) &= \alpha \\ \int \phi (x) \frac {d}{d\theta } p_{\theta _0}(x) \, d\mu (x) &= 0. \end{align*}

For a 1-parameter exponential family, we have \begin{align} p_\theta (x) &= h(x) e^{\theta T(x) - A(\theta )} \quad \text {and} \\ \frac {d}{d\theta } p_\theta (x) &= h(x) e^{\theta T(x) - A(\theta )} \left ( T(x) - A'(\theta ) \right ) = p_\theta (x) \left ( T(x) - \mathbb {E}_\theta [T(X)] \right ). \end{align}

Applying the reasoning from the previous section, we find that a most powerful test has rejection region defined by \[ p_{\theta '}(x) > k_1 p_{\theta _0}(x) + k_2 \frac {d}{d\theta } p_{\theta _0}(x) \] for some values of $k_1$ and $k_2$, which is equivalent to \[ \frac {e^{(\theta ' - \theta _0) T(x)}}{k_1' + k_2' T(x)} > \text {const} \] with some rearranging.

The first possibility will not give rise to an unbiased test, because the result would be a one-sided test with monotone power functions. Therefore any optimal $\phi $ is of the form \[ \phi (x) = \begin {cases} 1 & \text {if } T(x) > C_1 \text { or } T(x) < C_2 \\ \gamma _i & \text {if } T(x) = C_i \\ 0 & \text {otherwise} \end {cases}. \] A simplification is possible if $T(x)$ is symmetrically distributed under $\theta _0$. Then the optimal test rejects whenever $|T(x)| > \text {const}$. Such tests are called equitailed tests.

3.1.5 UMP Invariant Tests

Example 3.14. Suppose that we observe $X = (X_1, \ldots , X_d)$, with independent coordinates $X_i \sim \mathcal {N}(\theta _i, 1)$ and that we are interested in the hypothesis testing problem \[ H_0 : \theta _1 = \ldots = \theta _d = 0 \quad \text {vs.} \quad H_1 : \exists i \in \{1, \ldots , d\} : \theta _i \neq 0. \] If we transform our data so that $X' = OX$ for $O \in \mathbb {O}(d)$, the set of $d \times d$ orthogonal matrices, i.e., the set of square matrices such that $O^\top O = OO^\top = I$, then we have $X_i' \stackrel {\text {ind}}{\sim } \mathcal {N}(\theta _i', 1)$ for $\theta ' = O\theta $, and our hypothesis testing problem can be seen to be equivalent to testing \[ H_0 : \theta _1' = \ldots = \theta _d' = 0 \quad \text {vs.} \quad H_1 : \exists i \in \{1, \ldots , d\} : \theta _i' \neq 0. \] Hence, when searching for a test, the principle of invariance would suggest constraining $\phi (X) = \phi (OX) \; \forall O \in \mathbb {O}(d)$. In this case, it can be checked that $\phi $ is invariant in this way iff it is a function of the magnitude of the vector of samples, i.e. of $T = \sum _i X_i^2$. This tells us that if we only care about invariant tests, our optimality goal is to search for UMP tests among functions of $T$.

In this example, $T$ is non-central chi-squared distributed with $d$ degrees of freedom, i.e., $T \sim \chi _d^2(\psi ^2)$ with a non-centrality parameter $\psi ^2 = \sum _{i=1}^{d} \theta _i^2$. Thus, we can simplify the null and alternative hypotheses of our testing problem to \[ H_0 : \psi = 0 \quad \text {vs.} \quad H_1 : \psi > 0. \quad \text {(one-sided test)} \] Note that we have reduced both the relevant data and the relevant parameter space for our testing problem; these are common advantages of imposing invariance constraints. To derive our UMP invariant test, we will check that the $\chi _d^2(\psi ^2)$ has monotone likelihood ratios in $T$. Note that the density of non-central chi-squared distribution has the form: \[ f(t; \psi ) = e^{-\frac {\psi ^2}{2}} \sum _{k=0}^{\infty } \frac {\left (\frac {\psi ^2}{2}\right )^k}{k!} \cdot \frac {t^{\frac {d}{2} - 1 + k} e^{-\frac {t}{2}}}{2^{k + \frac {d}{2}} \Gamma \left (k + \frac {d}{2}\right )}. \] The likelihood ratio can therefore be computed as \[ \frac {p_{\psi ^2}(t)}{p_{\psi ^2 = 0}(t)} = \frac {f(t; \psi )}{\frac {t^{\frac {d}{2} - 1} e^{-t/2}}{2^{\frac {d}{2}} \Gamma \left (\frac {d}{2}\right )}} = e^{-\frac {\psi ^2}{2}} \sum _k c_k \left (\frac {\psi ^2}{2}\right )^k t^k. \] where $c_k$ are non-negative constants. We can see that each term in the sum above increases in $t$, and thus the ratio as a whole is increasing in $t$. (We only need to compare each parameter value with $0$, because our null hypothesis is simple.) Thus this family has MLR, and the UMPI test rejects when $T$ is large.

3.1.6 Confidence Regions

Given a model $\mathcal {P} = \{\mathbb {P}_\theta : \theta \in \Omega \}$, we begin by defining an appropriate collection of tests. For each $\theta _0 \in \Omega $, let $\Omega _0(\theta _0)$ be a set containing $\theta _0$, where $\Omega _0(\theta _0) \subset \Omega $. Next, define $\phi _{\theta _0}$ to be a level $\alpha $ test for $H_0: \theta \in \Omega _0(\theta _0)$ vs. $H_1: \theta \notin \Omega _0(\theta _0)$, and define $A(\theta _0)$ as the acceptance region of $\phi _{\theta _0}$. Because $\phi _{\theta _0}$ is a level $\alpha $ test, and $\theta _0 \in \Omega _0(\theta _0)$, we have $P_\theta (X \in A(\theta )) \ge 1 - \alpha $ for all $\theta \in \Omega $.

Now consider the region $S(X) = \{\theta \in \Omega : X \in A(\theta )\}$. Since $P_\theta (\theta \in S(X)) = P_\theta (X \in A(\theta )) \ge 1 - \alpha $, $S(X)$ is a $1 - \alpha $ confidence region! Different choices of null sets $\Omega _0(\theta _0)$ will lead to different forms of confidence regions. For example,

1.: One-sided tests $\Omega _0(\theta _0) = \{\theta : \theta \le \theta _0\}$ often yield confidence bounds \[ S(X) = \{\theta : u(X) \le \theta \} \] where $u(X)$ denotes a data-dependent lower bound.
2.: Two-sided tests $\Omega _0(\theta _0) = \{\theta _0\}$ often yield confidence intervals \[ S(X) = \{\theta : u(X) \le \theta \le v(X)\}, \] where $v(X)$ denotes a data-dependent upper bound.

Intuitively, we desire a $1 - \alpha $ confidence region that is as narrow as possible, which we can achieve by minimizing the number of extraneous it contains. We make this notion precise in the following optimality property for confidence regions.


	\(\theta \in \Omega _0\)	\(\theta \in \Omega _1\)

Reject \(H_0\)	1 (Type I Error)	0 (Good)

Accept \(H_0\)	0 (Good)	1 (Type II Error)