Tutorial

• conditional expectation & variance
  • conditional expectation $$E(X|Y=y)=\sum _{x}x{p}_{X|Y}(x|y)$$ $$E\left(\sum _k{X}_k|Y=y\right)=\sum _{k}E(X_k|Y=y)$$
  • law of total expectation $$E[E(X|Y)]=E[X]$$
    • 👨‍🏫 idea: separating randomness
      • $g(y):=E(X|Y=y)$
      • then $E_Y[g(y)]=E[X]$
  • conditional variance $$\text{Var}(X|Y)=E((X-E(X|Y){)}^2|Y)$$ $$\text{Var}(X)=E(\text{Var}(X|Y))+\text{Var}(E(X|Y))$$
• moment generating function
  • $M_X(t=E(e^{tX}))$
  • $E(X^n)=M_X^{(n)}(0)=\frac{{\mathrm{d}}^n}{\mathrm{d}{t}^n}{M}_X(t){|}_{t=0}$
  • if $X,Y$ independent: $$M_{X+Y}(t)=M_X(t)M_Y(t)$$
  • uniqueness: MGF determines distribution uniquely
    • i.e. if two r.v. have same MGF: then same distribution
• law of large number (LLN)
  • if $X_1,X_2,\dots ,X_n$: i.i.d. r.v. w/ $\mu <\infty$ $${\bar{X}}_n\stackrel{\mathbb{P}/a.s.}{⟶}\mu$$
    • i.e. convergence in probability
    • ${\lim }_{n\to \infty }\mathbb{P}(|{\bar{X}}_n-\mu |>\epsilon )\to 0$
    • a.s.: almost sure
    • convergence in probability
      • 👨‍🏫 probability of not converging = 0
    • 👨‍🏫 for stronger result, a stronger assumption is needed
  • can be computed without knowing the distribution of individual r.v.
    • $\mu$ must exist, though
• central limit theorem (CLT)
  • if $X_1,X_2,\dots ,X_n$: i.i.d. r.v. w/ $\mu ,{\sigma }^2<\infty$ $$\frac{(\sum X_i)-n\mu }{\sigma \sqrt{n}}=\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}\stackrel{d}{⟶}N(0,1)$$
  • i.e. convergence in distribution
  • strength of convergence: almost sure > in probability > in distribution
    • one on the left: implies one on the right

Examples

1. skipped
2. solution
   1. let $N=\min \{n\ge 1;X_n>X_0\}$
      • show: $X_n$ w/ distribution $F+(1-F)\log (1-F)$
      • $X_i<X_0$ for all $i\in [1,n-1]$
      • idea: law of total probability, solving one by one
      • $P(X_N\le x)=\sum P(X_N\le x|N=n)P(N=n)$ $$\begin{array}{rl}P(N=n)& =P(X_0<X_n\cap X_1,\dots ,X_{n-1}<X_0)\\ & =\frac{1}{n+1}\cdot \frac{1}{n}\\ & =\frac{1}{n}-\frac{1}{n+1}\end{array}$$
      • explanation: set Xn​=max, X0​=max{{Xi​}</span>{Xn​}}
      • $P(X_N\le x|N=n)=[F(n){]}^{n+1}$ $$\begin{array}{rl}P(X_N\le x)& =\sum [F(n){]}^{n+1}\left(\frac{1}{n}-\frac{1}{n+1}\right)\\ & =F+(1-F)\log (1-F)\\ [F(X)-1]\sum \frac{[F(X){]}^n}{n}+F(n)& \end{array}$$
      • integrate both sides to show: one converges to $-\log$
   2. consider: a very simple case for intuition
      • $n=2$: $$\begin{array}{rl}P(X_2>X_0\le X_1)& =P(X_2>X_1)-P(X_1\ge X_0\ge X_2)\end{array}$$ $$\begin{array}{rlr}P(M>m)& =P(X_m>X_0\le \dots )& =P(X_2>\dots )-P(\dots )\\ & =\frac{1}{(n+1)!}& \end{array}$$
3. $X\sim$Poisson w/ $\Lambda \sim Exp(\mu )$
   • show: $X\sim Geom$
   • $Lambda\sim Exp(\mu )$
   • Poisson's MGF: $$\begin{array}{rlrl}{M}_X(t)& =E(e^{\Lambda (e^t-1)})& & \\ & =E(e^{t\Lambda })& & t=e^t-1\\ & =\frac{\mu }{\mu -(e^t-1)}& & \Lambda \approx \text{Exp}(\mu )\end{array}$$
   • $M_{\Lambda }(t)=\frac{\mu }{\mu -t}$