Notes / Parameter Estimation
2.1 Not all data is relevant to a particular decision problem.
Definition 2.1 (Statistic) . A statistic \(T : \mathcal {X} \to \mathcal {T}\) is a function of the data.
Definition 2.2 (Sucient Statistic) . A statistic is sufficient for a model \(\mathcal {P} = \{\mathbb {P}_\theta : \theta \in \Omega \}\) if for all \(t\) , the conditional
distribution \(X \mid T(x) = t\) does not depend on \(\theta \) .
Theorem 2.3 (Neyman-Fisher Factorization Criterion (NFFC)) . Suppose each \(\mathbb {P}_\theta \in \mathcal {P}\) has density \(p(x; \theta )\) w.r.t. a
common \(\sigma \) -finite measure \(\mu \) , i.e., \(\frac {d\mathbb {P}_\theta }{d\mu } = p(x; \theta )\) . Then \(T(X)\) is sufficient if and only if \(p(x; \theta ) = g_\theta (T(x)) h(x)\) for some \(g_\theta , h\) .
Example 2.4. The model \(\{\mathbb {P}_\theta : \theta \in \Omega \}\) forms an \(s\) -dimensional exponential family if each \(\mathbb {P}_\theta \) has density of the form: \[ p(x; \theta ) = \exp \left ( \sum _{i=1}^{s} \eta _i(\theta ) T_i(x) - B(\theta ) \right ) h(x) \]
\(\eta _i(\theta ) \in \mathbb {R}\) are called the natural parameters .
\(T_i(x) \in \mathbb {R}\) are its sufficient statistics , which follows from NFFC .
\(B(\theta )\) is the log-partition function because it is the logarithm of a normalization factor: \[ B(\theta ) = \log \left ( \int \exp \left ( \sum _{i=1}^{s} \eta _i(\theta ) T_i(x) \right ) h(x)\, d\mu (x) \right ) \in \mathbb {R} \]
\(h(x) \in \mathbb {R}\) : base measure.
Exponential families are of particular interest to us, because many common distributions are exponential
families (e.g., Normal, Binomial, and Poisson)
Definition 2.5 (Minimal Suciency) . A sufficient statistic \(T\) is minimal if for every sufficient statistic \(T'\)
and for every \(x, y \in \mathcal {X}\) , \(T(x) = T(y)\) whenever \(T'(x) = T'(y)\) . In other words, \(T\) is a function of \(T'\) (there exists \(f\) such that \(T(x) = f(T'(x))\) for any \(x \in \mathcal {X}\) ).
Theorem 2.6. Let \(\{p(x; \theta ), \theta \in \Omega \}\) be a family of densities with respect to some measure \(\mu \) . Suppose that there exists
a statistic \(T\) such that for every \(x, y \in \mathcal {X}\) : \[ p(x; \theta ) = C_{x,y} p(y; \theta ) \quad \Longleftrightarrow \quad T(x) = T(y) \] for every \(\theta \) and some \(C_{x,y} \in \mathbb {R}\) . Then \(T\) is a minimal sufficient statistic.
Definition 2.7 (Ancillary) . A statistic \(A\) is ancillary for \(X \sim \mathbb {P}_\theta \in \mathcal {P}\) if the distribution of \(A(X)\) does not depend on \(\theta \) .
Definition 2.8 (First-Order Ancillary) . A statistic \(A\) is first-order ancillary for \(X \sim \mathbb {P}_\theta \in \mathcal {P}\) if \(\mathbb {E}_\theta [A(X)]\) does not depend on \(\theta \) .
From this we define the concept of complete statistics.
Definition 2.9 (Complete Statistic) . A statistic \(T\) is complete for \(X \sim \mathbb {P}_\theta \in \mathcal {P}\) if no non-constant function of \(T\) is
first-order ancillary. In other words, if \(\mathbb {E}_\theta [f(T(X))] = 0\) for all \(\theta \) , then \(f(T(X)) = 0\) with probability 1 for all \(\theta \) .
Theorem 2.11 (Complete Statistics for Exponential Family) . \((T_1, \ldots , T_s)\) is complete for any \(s\) -dimensional full
rank exponential family.
Theorem 2.12 (Basu's Theorem) . If \(T\) is complete and sufficient for \(\mathcal {P} = \{\mathbb {P}_\theta : \theta \in \Omega \}\) , and \(A\) is ancillary then \(T(X) \perp \!\!\!\perp A(X)\) .
Example 2.13. \(X_1, \ldots , X_n \stackrel {i.i.d}{\sim } \mathcal {N}(\mu , \sigma ^2)\) (\(\mu , \sigma ^2\) both unknown). \(\bar {X} \perp \!\!\!\perp \frac {1}{n} \sum _{i=1}^{n} (X_i - \bar {X})^2\) .
Proof. Fix any \(\sigma > 0\) , and consider the submodel \(\mathcal {P}_\sigma = \{\mathcal {N}(\mu , \sigma ^2) : \mu \in \mathbb {R}\}\) . In each submodel, \(\bar {X}\) is complete and sufficient, and \(\frac {1}{n} \sum _{i=1}^{n} (X_i - \bar {X})^2\) is
ancillary. By Basu’s Theorem, \(\bar {X} \perp \!\!\!\perp \sum _{i=1}^{n} (X_i - \bar {X})^2\) under \(\mathcal {N}(\mu , \sigma ^2)\) for any \(\mu \) . Since \(\sigma \) is arbitrary, we have \(\bar {X} \perp \!\!\!\perp \frac {1}{n} \sum _{i=1}^{n} (X_i - \bar {X})^2\) for the full model. □