Notes / Probability Theory and Stochastic Processes
1.1 1.1.1 A probability space \( (\Omega , \mathcal {F}, P) \) contains three elements:
The space \(\Omega \) : this is a non-empty set. It can be viewed as the set of all possible outcomes.
The \(\sigma \) -field \(\mathcal {F}\) : this can be viewed as a collection of all the events.
The probability measure \(P\) : this is a function from \(\mathcal {F}\) to \([0, 1]\) . It gives a probability to each event.
Definition 1.1 ( \(\sigma -\) Field) . Suppose \(\mathcal {F}\) is a non-empty collection of subsets of \(\Omega \) .
It is a field if it is closed under complementation and closed under union : \[ A \in \mathcal {F} \implies A^c \in \mathcal {F}, \quad A_1, A_2 \in \mathcal {F} \implies A_1 \cup A_2 \in \mathcal {F}. \]
It is a monotone class if \[ A_j \in \mathcal {F}, \, A_j \subset A_{j+1}, \, 1 \leq j < \infty \implies \bigcup _j A_j \in \mathcal {F}, \] and \[ A_j \in \mathcal {F}, \, A_j \supset A_{j+1}, \, 1 \leq j < \infty \implies \bigcap _j A_j \in \mathcal {F}. \]
It is a \(\sigma \) -field if it is closed under complementation and closed under countable union : \[ A \in \mathcal {F} \implies A^c \in \mathcal {F}, \quad A_j \in \mathcal {F}, \, 1 \leq j < \infty \implies \bigcup _j A_j \in \mathcal {F}. \]
Definition 1.2 (Generated \(\sigma \) -Fields) . Given any collection \( C \) of sets, the \(\sigma \) -field generated by \( C \) is the
intersection of all \(\sigma \) -fields containing \( C \) .
Definition 1.3 (Probability Measure) . Suppose \(\mathcal {F}\) is a \(\sigma \) -field on \(\Omega \) . A probability measure \(P\) is a function from \(\mathcal {F}\) to \([0, 1]\)
satisfying the following axioms:
\(P[E] \geq 0\) for all \(E \in \mathcal {F}\) ;
\(P[\Omega ] = 1\) ;
If \(\{E_j\}_{j}\) is a countable collection of pairwise disjoint sets in \(\mathcal {F}\) , then: \[ P\left [\bigcup _j E_j\right ] = \sum _j P[E_j]. \]
These axioms imply the following consequences:
\(P[E^c] = 1 - P[E]\) ;
\(P[E \cup F] + P[E \cap F] = P[E] + P[F]\) ;
Continuity: if \(E_n \uparrow E\) or \(E_n \downarrow E\) , then \(P[E_n] \to P[E]\) ;
\(P\left [\bigcup _j E_j\right ] \leq \sum _j P[E_j]\) .
Theorem 1.5 (Carathéodory's Extension Theorem) . Suppose \(\mathcal {F}_0\) is a field and \(\mathcal {F}\) is the \(\sigma \) -field generated by
\(\mathcal {F}_0\) . Suppose \(\mu \) is a probability measure on \(\mathcal {F}_0\) . Then there exists a unique probability measure on \(\mathcal {F}\) that
coincides with \(\mu \) on \(\mathcal {F}_0\) .
1.1.2
Definition 1.7 (Random Variables) . A real-valued random variable is a function \( X: \Omega \to \mathbb {R} \) such that \[ X^{-1}(B) \in \mathcal {F}, \quad \forall B \in \mathcal {B}. \] In other
words, a random variable is just a measurable function from \( (\Omega , \mathcal {F}) \) to \( (\mathbb {R}, \mathcal {B}) \) .
Definition 1.8 (Distribution) . Each random variable \( X \) induces a probability measure \( \mu \) on \( (\mathbb {R}, \mathcal {B}) \) by the
following correspondence: \[ \mu [B] = P[X^{-1}(B)] = P[X \in B], \quad \forall B \in \mathcal {B}. \] The measure \( \mu \) is called the law (or the distribution) of \( X \) , denoted by \( \mathcal {L}(X) \) ; its
associated distribution function is called the distribution function of \( X \) , denoted by \( F_X \) .
The concept of “expectation” is the same as integration in the probability space \( (\Omega , \mathcal {F}, \mathbb {P}) \) .
Corollary 1.9. If \( X \) takes only positive integer values, we have \[ \mathbb {E}[X] = \sum _{n=1}^{\infty } \mathbb {P}[X \geq n]. \]
Definition 1.10 ( \(p\) -th Moment) . For any \( p \in (0, \infty ) \) , define \[ L^p(\Omega , \mathcal {F}, \mathbb {P}) = \left \{ X \text { random variable on } (\Omega , \mathcal {F}, \mathbb {P}) : \mathbb {E}[|X|^p] < \infty \right \}. \] For \( X \in L^p \) , we call \( \mathbb {E}[|X|^p] \) the \( p \) -th moment of \( X \) .
Definition 1.11 (Independent) . The random variables \( \{ X_j, 1 \leq j \leq n \} \) are independent if, for any Borel sets \( \{ B_j, 1 \leq j \leq n \} \) , we
have \[ \mathbb {P}\left [\bigcap _{j=1}^n \{ X_j \in B_j \}\right ] = \prod _{j=1}^n \mathbb {P}[X_j \in B_j]. \] The random variables \( \{ X_j, j \geq 1 \} \) are independent if \( \{ X_j, 1 \leq j \leq n \} \) are independent for all \( n \) .
Theorem 1.12. Suppose \( \{ A_j, 1 \leq j \leq n \} \) are independent and each \( A_j \) is a \(\pi \) -system. Denote by \( \sigma (A_j) \) the \(\sigma \) -field generated
by \( A_j \) for each \( j \) . Then \( \{ \sigma (A_j), 1 \leq j \leq n \} \) are independent.
Example 1.13 (Kolmogorov's 0-1 Law) . Let \( \{ X_n \} \) be a sequence of independent random variables. Let \[ G_n = \sigma (X_k, k \geq n) \quad \text {and} \quad G_\infty = \bigcap _{n \geq 1} G_n. \]
Then \( G_\infty \) is trivial, i.e., for any \( A \in G_\infty \) , we have \[ \mathbb {P}[A] = 0 \text { or } 1. \] Define \( S_n = \sum _{j=1}^{n} X_j \) . It is clear that the following events are in \( G_\infty \) : \[ \lim _{n \to \infty } S_n \text { exists}, \quad \lim _{n \to \infty } \frac {S_n}{n} \text { exists}, \quad \limsup _{n \to \infty } \frac {S_n}{n} > 0. \] Whereas,
the following event is not in \( G_\infty \) : \[ \limsup _{n \to \infty } S_n > 0. \]
Proof. On the one hand, we have \( A \in G_{n+1} \) for any \( n \) . Thus, \( A \) is independent of \( \sigma (X_1, \dots , X_n) \) for any \( n \) . Therefore, \( A \) is independent
of \( \sigma (X_n, n \geq 1) \) . On the other hand, \( A \) is measurable with respect to \( \sigma (X_n, n \geq 1) \) . Therefore, \( A \) is independent of itself. This implies
\( \mathbb {P}[A] = \mathbb {P}[A \cap A] = \mathbb {P}[A]^2 \) , thus \( \mathbb {P}[A] \in \{ 0, 1 \} \) . □
Now let’s talk about Gaussians. The key thing to know is that Gaussians stay Gaussian under linear combinations and
projections.
Proposition 1.14. The following are equivalent:
1. The random vector \( \xi = (\xi _1, \dots , \xi _d)^\top \) follows a \( d \) -dimensional Gaussian distribution.
2. For any \( a_1, \dots , a_d \) , the linear combination \( \sum _{k=1}^d a_k \xi _k \) follows a one-dimensional Gaussian distribution.