Large Sample Theory — A Guided Tour of Modern Statistics

Notes / Parameter Estimation

2.3 Large Sample Theory

2.3.1 Delta Method

Proof. By the definition of the derivative, we have that \[ \phi (t) = \phi (\theta ) + \phi '(\theta )(t - \theta ) + o(\|t - \theta \|), \] i.e. \begin{equation} \phi (t) = \phi (\theta ) + \phi '(\theta )(t - \theta ) + R(\|t - \theta \|) \end{equation} where \(\lim _{h \to 0} \frac {R(h)}{h} = 0\). Since \(r_n(T_n - \theta )\) converges in distribution, we know that \(r_n(T_n - \theta ) = O_p(1)\), which implies that \(r_n \|T_n - \theta \| = O_p(1)\). We also have that \(\|T_n - \theta \| = o_p(1)\), which implies \(R(\|T_n - \theta \|) = o_p(\|T_n - \theta \|)\). Thus \[ r_n R(\|T_n - \theta \|) = r_n o_p(\|T_n - \theta \|) = o_p(r_n \|T_n - \theta \|) = o_p(O_p(1)) = o_p(1). \] Using this along with (6), we have the second part of the theorem. Noting that \(r_n \phi '(\theta )(T_n - \theta ) \xrightarrow {d} \phi '(\theta ) T\), and applying Slutsky’s theorem, we get the first part as well. □

Proof. By definition, \[ \phi (t) = \phi (\theta ) + \boldsymbol {\nabla } \phi (\theta )^\top (t - \theta ) + \frac {1}{2}(t - \theta )^\top \boldsymbol {\nabla }^2 \phi (\theta )(t - \theta ) + R(\|t - \theta \|), \] where \(R(h) = o(\|h\|^2)\). Since \(\boldsymbol {\nabla } \phi (\theta ) = 0\), we actually have \begin{equation} \phi (t) = \phi (\theta ) + \frac {1}{2}(t - \theta )^\top \boldsymbol {\nabla }^2 \phi (\theta )(t - \theta ) + R(\|t - \theta \|). \end{equation} Note \(r_n^2 R(\|T_n - \theta \|) = r_n^2 o_p(\|T_n - \theta \|^2) = o_p(\|r_n(T_n - \theta )\|^2)\). Since \(r_n(T_n - \theta )\) converges in distribution, so does \(\|r_n(T_n - \theta )\|^2\), and so \(\|r_n(T_n - \theta )\|^2 = O_p(1)\). Thus \begin{equation} r_n^2 R(\|T_n - \theta \|) = o_p(O_p(1)) = o_p(1). \end{equation} Now by the continuous mapping theorem, we have that \begin{equation} \frac {1}{2}(r_n(T_n - \theta ))^\top \boldsymbol {\nabla }^2 \phi (\theta )(r_n(T_n - \theta )) \xrightarrow {d} \frac {1}{2} T^\top \boldsymbol {\nabla }^2 \phi (\theta ) T. \end{equation} So combining (7), (8), (9) and using Slutsky’s lemma, we get the desired convergence in distribution. □

Example 2.35. Suppose \(\theta \in (0,1)\), \(X_i \sim \text {Bernoulli}(\theta )\). To estimate \(\theta \), we may use the sample mean \(\hat {\theta }_n = n^{-1}\sum _{i=1}^n X_i\). Clearly, \(\mathbb {E}\hat {\theta }_n = \theta \), \(\text {Var}(\hat {\theta }_n) = \frac {\theta (1-\theta )}{n}\), \(\big [\sqrt {n}(\hat {\theta }_n - \theta )\big ]^2 \xrightarrow {d} \theta (1-\theta )\cdot \chi _{(1)}^2\). Instead of using mean squared error to measure the performance of \(\hat {\theta }_n\), let us use the Kullback-Leibler (KL) divergence (or the log loss). This is \[ D_{KL}(P \,\|\, Q) = \int dP \log \!\left (\frac {dP}{dQ}\right ). \] Let \(P_t = \text {Bernoulli}(t)\), \(t \in [0,1]\). So \[ D_{KL}(P_t \,\|\, P_\theta ) = t\log \frac {t}{\theta } + (1-t)\log \frac {1-t}{1-\theta }. \] Let \(\phi (t) = D_{KL}(P_t \,\|\, P_\theta )\). Then \[ \phi '(t) = \log \frac {t}{1-t} - \log \frac {\theta }{1-\theta }. \] Note \(\phi '(\theta ) = 0\). So we need the second derivative: \[ \phi ''(t) = \frac {1}{t} + \frac {1}{1-t} = \frac {1}{t(1-t)}, \] and so \(\phi ''(\theta ) = \frac {1}{\theta (1-\theta )}\). So by the second order Delta Method, \[ n D_{KL}(P_{\hat {\theta }_n} \,\|\, P_\theta ) \xrightarrow {d} \frac {1}{2}\chi ^2_{(1)}. \]

2.3.2 Fisher Information

Before we do anything, we have to make several assumptions.

1.: We have a “nice, smooth” model, i.e. the Hessian is Lipschitz-continuous. To be rigorous, the following must hold: \[ \big \|\nabla ^2 \ell _{\theta _1}(x) - \nabla ^2 \ell _{\theta _2}(x)\big \|_{\mathrm {op}} \le M(x)\,\|\theta _1 - \theta _2\|_2 \qquad \mathbb {E}_\theta [M^2(x)] < \infty \]
2.: The MLE, \(\hat {\theta }_n \in \arg \max _{\theta \in \Theta } P_n \ell _\theta (x)\), is consistent, i.e. \(\hat {\theta }_n \xrightarrow {p} \theta _0\) under \(P_{\theta _0}\).
3.: \(\Theta \) is a convex set.

Remark 2.38. Let us rewrite the asymptotic variance. Given that \(\nabla ^2 \ell _\theta = \nabla \!\left (\dfrac {\nabla p_\theta }{p_\theta }\right ) = \dfrac {\nabla ^2 p_\theta }{p_\theta } - \dfrac {\nabla p_\theta \nabla p_\theta ^\top }{p_\theta ^2}\): \[ \mathbb {E}_\theta \!\left [\frac {\nabla ^2 p_\theta }{p_\theta }\right ] = \int \frac {\nabla ^2 p_\theta }{p_\theta }\, p_\theta \, d\mu = \int \nabla ^2 p_\theta \, d\mu = \nabla ^2 \int p_\theta \, d\mu = 0. \]

As a result: \[ \mathbb {E}_\theta [\nabla ^2 \ell _\theta ] = -\mathbb {E}_\theta \!\left [\left (\frac {\nabla p_\theta }{p_\theta }\right ) \!\left (\frac {\nabla p_\theta }{p_\theta }\right )^{\!T}\right ] = -\operatorname *{Cov}_\theta (\nabla \ell _\theta (x)). \]

We define the Fisher Information as \(I_\theta := \mathbb {E}_\theta [\nabla \ell _\theta (x)\nabla \ell _\theta (x)^\top ] = \operatorname *{Cov}_\theta \nabla \ell _\theta \), where the final equality holds because \(\mathbb {E}_\theta [\nabla \ell _\theta (x)] = 0\) (\(\theta \) maximizes \(\mathbb {E}_\theta [\ell _\theta (x)]\)). To show this, assume that we can swap \(\nabla , \mathbb {E}\). Then, \(\nabla \ell _\theta (x) = \nabla \log p_\theta (x) = \dfrac {\nabla p_\theta (x)}{p_\theta (x)}\). Using that result, we see that: \[ \mathbb {E}_\theta [\nabla \ell _\theta ] = \mathbb {E}\!\left [\frac {\nabla p_\theta }{p_\theta }\right ] = \int \frac {\nabla p_\theta }{p_\theta }\, p_\theta \, d\mu = \int \nabla p_\theta \, d\mu = \nabla \int p_\theta \, d\mu = \nabla (1) = 0. \]

We now have a more compact representation of the asymptotic distribution described in the Theorem above. \[ \sqrt {n}(\hat {\theta }_n - \theta _0) \xrightarrow {d} \mathsf {N}(0,\, I_{\theta _0}^{-1} I_{\theta _0} I_{\theta _0}^{-1}) = \mathsf {N}(0,\, I_{\theta _0}^{-1}). \]

Consider \(I_\theta = -\nabla ^2 \mathbb {E}[\ell _\theta (x)]\). If the magnitude of the second derivative is “large,” that implies that the log-likelihood is steep around the global maximum (making it “easy” to find). Alternatively, if the magnitude of \(-\nabla ^2 \mathbb {E}[\ell _\theta (x)]\) is “small,” we do not have sufficient curvature to find the optimal \(\theta \).

Proof. Set \(\Psi (x) = \nabla _\theta \ell _\theta (x)\), we have that \(\mathbb {E}_\theta [\Psi ] = 0\), and that \begin{align*} \mathbb {E}[(\delta - g(\theta ))\Psi ] &= \mathbb {E}[\delta \Psi ] \\ &= \mathbb {E}[\delta \nabla \ell _\theta ] \\ &= \mathbb {E}\!\left [\delta \frac {\nabla p_\theta }{p_\theta }\right ] \\ &= \int \delta \nabla p_\theta \, d\mu (x). \end{align*}

Under good regularity conditions, we have that \[ \mathbb {E}[(\delta - g(\theta ))\Psi ] = \nabla \int \delta (x) p_\theta (x)\, d\mu (x) = \nabla g(\theta ). \]

We take \[ \gamma = \nabla g(\theta ), \quad C = I_\theta \] to get the desired result. □

Proof. Take \begin{align*} g(\theta ) &= v^\top \theta \\ \delta &= v^\top \hat {\theta }(X). \end{align*}

Applying the Cramer-Rao theorem, \[ \mathbb {E}\!\left [(v^\top (\hat {\theta } - \theta ))^2\right ] \ge v^\top I_\theta ^{-1} v \] and \[ \mathbb {E}\!\left [(v^\top (\hat {\theta } - \theta ))^2\right ] = \mathbb {E}\!\left [\mathrm {tr}\!\left ((\hat {\theta } - \theta )(\hat {\theta } - \theta )^\top v v^\top \right )\right ] = v^\top \mathrm {Cov}(\hat {\theta })\, v. \] □