Notes / Probability Theory and Stochastic Processes
1.2 Convergence
1.2.1 Convergence: Almost Sure, in Probability, and in \(L^p\)
It is straightforward to prove that almost sure convergence (\(a.s.\)) implies convergence in probability, and
convergence in \(L^p\) implies convergence in probability. However, almost sure convergence (\(a.s.\)) and \(L^p\) may not
imply each other. Now we talk about an example: using probability methods to prove Weierstrass
Theorem.
Proof.Suppose \( \{ X_j, j \geq 1 \} \) are i.i.d. Bernoulli variables with parameter \( p \in [0, 1] \): \( \mathbb {P}[X_j = 1] = p, \mathbb {P}[X_j = 0] = 1 - p \), and set \( S_n = \sum _{j=1}^{n} X_j \). Then we find \[ \mathbb {E}[X_j] = p, \quad \text {var}(X_j) = p(1 - p), \quad \mathbb {E}[S_n] = np, \quad \text {var}(S_n) = np(1 - p). \] Moreover, \[ \mathbb {P}[S_n = j] = \binom {n}{j} p^j (1 - p)^{n - j}, \quad \text {thus} \quad f_n(p) = \mathbb {E}[f(S_n/n)]. \] Note
that \( f \) is bounded: \( |f| \leq M \), and it is uniformly continuous: for any \( \epsilon > 0 \), there exists \( \delta > 0 \) such that \( |f(x) - f(y)| \leq \epsilon \) as long as \( |x - y| \leq \delta \). Thus, \[ |f_n(p) - f(p)| \leq \mathbb {E}[|f(S_n/n) - f(p)|] \leq \epsilon + 2M \mathbb {P}[|S_n/n - p| > \delta ]. \]
Note that \[ \mathbb {P}[|S_n/n - p| > \delta ] \leq \frac {\text {var}(S_n)}{n^2 \delta ^2} = \frac {p(1 - p)}{n \delta ^2} \leq \frac {1}{4n \delta ^2}. \] Thus, \[ |f_n(p) - f(p)| \leq \epsilon + \frac {M}{2n \delta ^2}. \] □
We give a useful lemma to prove almost surely convergence.
1.2.2 Weak Convergence
1.2.3 Convergence in Distribution
Convergence in probability implies convergence in distribution. Furthermore, convergence in distribution is
equivalent to the convergence of characteristic functions.
The distribution is uniquely identified by the characteristic function.
At the end of this subsection, we introduce some useful convergence results. Let \( (X_n) \) be a sequence of random
variables and let \( X \) be a random variable such that \( X_n \to X \) (a.s.): \[ \mathbb {P}(X_n \to X) = 1. \]
We state the convergence properties as follows:
(MON) If \( 0 \leq X_n \uparrow X \), then \( \mathbb {E}(X_n) \leq \mathbb {E}(X) < \infty \);
(FATOU) If \( X_n \geq 0 \), then \( \mathbb {E}(X_n) \leq \liminf \mathbb {E}(X_n) \);