Moment generating function

Examples

• when $X\sim Be(p),M(t)=1-p+p{e}^t$ $$\begin{array}{rl}M(t)& =E(e^{tX})\\ & =e^{t\cdot 0}P(X=0)+e^{t\cdot 1}P(X=1)\\ & =1-p+p{e}^t\end{array}$$
• when $X\sim Bin(n,p),M(t)=(1-p+p{e}^t{)}^n$
  • $X={\sum }_{i=1}^n{X}_i$,$X_i$ being independent and $X_i\sim Be(p)$ $$\begin{array}{rl}M(t)& =E(e^{tX})\\ & =E(e^{t(X_1+\cdots +X_n)})\\ & =E(e^{t{X}_1}{e}^{t{X}_2}\dots e^{t{X}_n})\\ & =E(e^{t{X}_1})\dots E(e^{t{X}_n})\\ & =[M_{X_i}(t){]}^n=(1-p+p{e}^t{)}^n\end{array}$$
• $X\sim Geom(p),M(t)=\frac{p{e}^t}{1-(1-p)e^t}$
  • 👨‍🏫 exercise!
• $X\sim Poisson(\lambda ),M(t)=\exp (\lambda (e^t-1))$ $$\begin{array}{rl}M(t)& =E(e^{tx})\\ & =\sum _{k=0}^{\infty }{e}^{tk}P(X=k)\\ & =\sum _{k=0}^{\infty }{e}^{tk}\frac{e^{-\lambda }{\lambda }^k}{j!}\\ & =e^{-\lambda }\sum _{k=0}^{\infty }\frac{(e^t\lambda {)}^k}{j!}\\ & =e^{-\lambda }{e}^{e^t\lambda }\\ & =e^{(e^t-1)\lambda }\end{array}$$
• when $X\sim Bin(n,p),M(t)=(1-p+p{e}^t{)}^n$
  • suppose $n\ge 4$
  • derive the 4th moment by computing 4th derivative
  • 👨‍🏫 tedious!
• when $X\sim U(\alpha ,\beta ),M(t)=\frac{e^{\beta t}-e^{\alpha }t}{(\beta -\alpha )t}$ $$\begin{array}{rl}M(t)& =E(e^{tX})\\ & ={\int }_{-\infty }^{\infty }{e}^{tx}f(x)\mathrm{d}x\\ & ={\int }_{\beta }^{\alpha }{e}^{tx}\frac{1}{\beta -\alpha }\mathrm{d}x\\ & =\frac{1}{\beta -\alpha }{\int }_{\alpha }^{\beta }{e}^{tx}\mathrm{d}x\\ & =\frac{1}{\beta -\alpha }\frac{1}{t}{e}^{tx}{|}_{\alpha }^{\beta }\\ & =\frac{e^{\beta t}-e^{\alpha t}}{(\beta -\alpha )t}\end{array}$$
  • when $X\sim Exp(\lambda ),M(t)=\frac{\lambda }{\lambda -t}$ for $t<\lambda$ $$\begin{array}{rl}E\left[e^{tX}\right]& ={\int }_0^{\infty }{e}^{tx}\lambda e^{-\lambda x}\mathrm{d}x\\ & =\lambda {\int }_0^{\infty }{e}^{-(\lambda -t)x}\mathrm{d}x\\ & =\frac{\lambda }{\lambda -t}\end{array}$$
    • $t<\lambda$ as otherwise, $E^{e^{tX}}<0$ - somewhat impossible:
      • as $\lambda$ = frequency can't be negative!
  • when $X\sim N(\mu ,{\sigma }^2),M(t)=exp(\mu t+{\sigma }^2{t}^2/2)$
    • $X=\sigma Z+\mu$, where $Z\sim N(0,1)$
    • compute $M_Z(t)$ first $$\begin{array}{rlrl}{M}_Z(t)& =E(e^{tZ})& & \\ & ={\int }_{-\infty }^{\infty }{e}^{tZ}\frac{1}{\sqrt{2\pi }}{e}^{-(Z^2/2)}\mathrm{d}Z& & \\ & ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-(Z^2/2-tZ)}\mathrm{d}Z& & \\ Z^2/2-tZ& =\frac{1}{2}(Z^2-2tZ+t^2-t^2)& & \\ & =\frac{1}{2}(Z-t{)}^2-\frac{1}{2}(t^2)& & \\ {\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-(Z^2/2-tZ)}\mathrm{d}Z& ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(Z-t{)}^2}{e}^{\frac{t^2}{2}}\mathrm{d}Z& & \\ & =e^{\frac{t^2}{2}}{\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(Z-t{)}^2}\mathrm{d}Z& & \\ & =e^{\frac{t^2}{2}}& & \text{simplify pdf of }N(t,1)\\ M_X(t)& =E(e^{t(\sigma Z+\mu )})& & \\ & =E(e^{t\sigma Z+t\mu })& & \\ & =E_Z(e^{t\sigma Z}{e}^{t\mu })& & \\ & =e^{t\mu }{E}_Z(e^{t\sigma Z})& & \\ & =e^{t\mu }{M}_Z(t\sigma )& & \\ & =e^{t\mu }{e}^{\frac{t^2{\sigma }^2}{2}}& & \\ & =e^{t\mu +\frac{t^2{\sigma }^2}{2}}& & \end{array}$$
  • suppose $M_X(t)=e^{3(e^{(t-1)})}$, find $P(X=0)$
    • given: Poisson distribution
    • $P(X=0)=e^{-3}$
  • sum of independent Binomial r.v. w/ the same success probability: binomial
    • $X\sum Bin(n,p)$, $Y\sim Bin(m,p)$, $X,Y$ being independent $$\begin{array}{rl}E\left[e^{t(X+Y)}\right]& =E\left[e^{tX}\right]E\left[e^{tY}\right]\\ & =[1-p+p{e}^t{]}^n[1-p+p{e}^t{]}^m\\ & =[1-p+p{e}^t{]}^{r+m}\end{array}$$
    • thus $X+Y\sim Bin(n+m,p)$
  • similar for Poisson r.v.
  • sum of independent normal r.v. is normal
    • $X\sim N({\mu }_1,{\sigma }_1^2)$
    • $Y\sim N({\mu }_2,{\sigma }_2^2)$ $$\begin{array}{rl}E\left[e^{t(X+Y)}\right]& =E\left[e^{tX}\right]E\left[e^{tY}\right]\\ & =\exp ({\mu }_{1}t+{\sigma }_1^2{t}^2/2)\exp ({\mu }_{2}t+{\sigma }_2^2{t}^2/2)\\ & =\exp (({\mu }_1+{\mu }_2)t+({\sigma }_1^2+{\sigma }_2^2)t^2/2)\end{array}$$
  • find all moments of exponential dist. of parameter $\lambda >0$
    • $X\sim Exp(\lambda )$ $$\begin{array}{rl}E\left[e^{tX}\right]& =\frac{\lambda }{\lambda -t}\\ & =\frac{1}{1-t/\lambda }\\ & =\sum _{k=0}^{\infty }\frac{t^k}{{\lambda }^k}\\ & =\sum _{k=0}^{\infty }\frac{k!/{\lambda }^k}{k!}{t}^k\end{array}$$
    • thus: $E[X^k]=\frac{k!}{{\lambda }^k}$
  • chi-squared r.v. w/ $n$ degrees of freedom: ${\chi }_n^2$
    • given as: ${\sum }_{i=1}^n{Z}_1^2$
      • $Z_i$: independent std. normal r.v.
    • then $M(t)$: $$\begin{array}{rl}M(t)& =E(e^{t(Z_1^2+Z_2^2+\dots Z_n^2)})\\ & =E(e^{t{Z}_1^2+t{Z}_2^2+\dots t{Z}_n^2})\\ & =E(e^{t{Z}_1^2})E(e^{t{Z}_2^2})\dots E(e^{t{Z}_n^2})\\ & =E(e^{t{Z}_1^2}{)}^n\end{array}$$ $$\begin{array}{rl}E(e^{t{Z}^2})& ={\int }_{-\infty }^{\infty }{e}^{t{Z}^2}\frac{1}{\sqrt{2\pi }}{e}^{-Z^2/2}\mathrm{d}Z\\ & ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-Z^2/2(1-2t)}\mathrm{d}Z\\ & ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{Z^2}{2{\sigma }^2}}\mathrm{d}Z\\ & =\sigma {\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{Z^2}{2{\sigma }^2}}\mathrm{d}Z\\ & =\sigma \\ \frac{1}{{\sigma }^2}& =1-2t\\ {\sigma }^2& =\frac{1}{1-2t}\\ \sigma & =\sqrt{\frac{1}{1-2t}}\\ E(e^{t{Z}^2})& =\sqrt{\frac{1}{1-2t}}\\ M(t)& =(\frac{1}{1-2t}{)}^{n/2}\end{array}$$

Joint MGFs

• for multiple variables:
  • $M(t_1,\dots ,t_n)=E[e^{t_1{X}_1+\cdots +t_n{X}_n}]$
• thus: for joint distribution of $X,Y$
  • exists joint cdf, pmf/pdf, and joint mgf
    • joint mgf: $M(t,s)=E(e^{tx+sY})$
• individual mgf: can be obtained from joint mgf (like marginal)
  • $M_{X_i}(t)=E[e^{t{X}_i}]=M(0,\dots ,0,t,0,\dots ,0)$
• 👨‍🏫 not required for the final exam

Multiplication rule

• $M(t_1,\dots ,t_n)=M_{X_1}(t_1)\dots M_{X_n}(t_n)$
  • iff each $X_i$ independent
• proof: too advanced for this course
  • but some generalization of the original multiplication rule $$\begin{array}{rl}M(t_1,\dots ,t_n)& =E[e^{t_1{X}_1+\cdots +t_n{X}_n}]\\ & =E[e^{t_1{X}_1}\dots e^{t_n{X}_n}]\\ & =E[e^{t_1{X}_1}]\dots E[e^{t_n{X}_n}]\\ & =M_{X_1}(t_1)\dots M_{X_n}(t_n)\end{array}$$

Example

• let $X,Y$ be independent normal r.v. w/ $\mu ,{\sigma }^2$
  • as we know that $X+Y,X_Y$ are independent, we get: $$\begin{array}{rl}E\left[e^{t(X+Y)+s(X-Y)}\right]& =E\left[e^{(t+s)X+(t-s)Y}\right]\\ & =E\left[e^{(t+s)X}\right]E\left[e^{(t-s)Y}\right]\\ & =e^{\mu (t+s)+{\sigma }^2(t+s{)}^2/2}{e}^{\mu (t-s)+{\sigma }^2(t-s{)}^2/2}\\ & =e^{2\mu t+{\sigma }^2{t}^2}{e}^{{\sigma }^2{s}^2}\end{array}$$
    • if we considered the original preceding as a sum of normal r.v. with $2\mu ,2{\sigma }^2$
      • and indep. normal r.v. w/ $0,2{\sigma }^2$
      • joint MGF: uniquely determines the joint distribution
      • thus: $X+Y$ and $X-Y$ are independent normal r.v.

Introduction

• 2 important concepts: laws of large numbers & central limit theorem
• LLN: concerned w/ stating conditions under the average of a seq. of r.v. converges to expected average
• CLT: concerned w/ determining conditions: where sum of large number of r.v. has probability distribution approximately normal

Chebyshev's inequality and weak law of large numbers

• inequalities below: for estimating tail probabilities w/ minimal information on r.v.
  • e.g. its mean / variance
• Markov's inequality: for non-negative r.v. $X$ and for $a>0$: $$P(X\ge a)\le \frac{E(X)}{a}$$
• proof: (~= first moment) $$\begin{array}{rl}E(X)& ={\int }_{-\infty }^{\infty }x{f}_X(x)\mathrm{d}x={\int }_0^{\infty }x{f}_X(x)\mathrm{d}x\\ & \ge {\int }_a^{\infty }x{f}_X(x)\mathrm{d}x\ge {\int }_a^{\infty }a{f}_X(x)\mathrm{d}x=aP(X\ge a)\end{array}$$
• Chebyshev's inequality: for r.v. $X$ w/ finite mean $\mu$, variance ${\sigma }^2$, then for $a>0$: $$P(|X-\mu |\ge a)\le \frac{{\sigma }^2}{a^2}$$
• proof: (~= first moment) $$\begin{array}{rl}P(|X-\mu |\ge a)& =P(|X-\mu {|}^2\ge a^2)\\ & \le \frac{E(|X-\mu {|}^2)}{a^2}\\ & =\frac{\text{var}(X)}{a^2}\end{array}$$
• Markov's & Chebyshev's: deriving bounds on probabilities with very few information
  • mean / mean & variance
  • if the actual distribution is not known: it's a decent measure!

Example

• no. of items produced in factory: a r.v. w/ mean 50
  • what is: probability that production exceed 75?
    • $50/75=\frac{2}{3}$
  • if variance of production is 25, what is the probability of production being within $[40,60]$?
    • $25/10^2=\frac{2}{4}$
    • $1-\frac{1}{4}=\frac{3}{4}$