Example

• $n$ people: throw their hats to center of a room
  • hats: mixed up, find expected no. of people getting back their own hats
  • $E(\sum )=\sum (E)=\sum (P)=n\times \frac{1}{n}$
• suppose: $N$ different types of coupons exists
  • and each time: obtaining a coupon is equally likely to be any of $N$ types
  • find: expected no. of different types with $n$ run
    • denote answer by $X$
      • $X=\sum X_i$ for $i\in [1,N]$
    • $X_i$: 1 if $i$-th type is contained in the $n$ company
    • $E(X)=\sum E(X_i)$
      • $E(X)=P(X_i=i)=1-P(\text{none of tickets being type }i)$
      • $=1-\frac{(N-1{)}^n}{N^n}$
      • $E(X)=N\times (1-\frac{(N-1{)}^n}{N^n})$
  • find expected no. of coupons needed before completing at lease one set of each type
    • let $Y$: no. of coupons needed to get a complete set
    • $Y=Y_0+Y_1+\dots Y_{N-1}$
    • $Y_0$: no. of coupons you need in order to get a first new type
      • $=1$
    • $Y_1$: no. of coupons you need in order to get a second a new type (after $Y_0$)
      • $E(Y_1)=1\times (\frac{N-1}{N})+\frac{1}{N}(1+E(Y_1))$
      • $E(Y_1)=\frac{N}{N-1}$
    • $Y_i$: no. of coupons to get a new type after collecting $i-1$ types
      • $E(Y_i)=1\times (\frac{N-i}{N})+\frac{i}{N}(1+E(Y_i))$
      • $E(Y_i)=\frac{N}{N-i}$
      • $Y_i\sim Geom(p_i)$
      • where success probability $p_i=\frac{N-i}{N}$
    • thus: $Y={\sum }_{i=0}^{N-1}\frac{N}{N-i}=N({\sum }_{i=1}^N\frac{1}{i})\approx N\log N$

Covariance

• covariance of jointly distributed r.v. $X,Y$:
  • $\text{cov}(X,Y)=E[(X-{\mu }_X)(Y-{\mu }_Y)]$
  • measures: how $X,Y$ vary together
• common formula
  • $\text{cov}(X,Y)=\text{cov}(Y,X)$
  • $\text{cov}(X,X)=\text{var}(X)$
  • $\text{cov}(X,a)=E[(X-{\mu }_X)(a-a)]=0$
  • if, for larger value of $X$: corresponds to larger values of $Y$
    • then $\text{cov}(X,Y)>0$
    • else: $\text{cov}(X,Y)<0$
• remark: if $\text{cov}(X,Y)\ne 0$, $X,Y$: correlated
  • else: $X,Y$: uncorrelated
• correlation: does not imply causation
• theorem: $\text{cov}(X,Y)=E(XY)-E(X)E(Y)$
  • similar to $\text{var}(X)=E(X^2)-(E(X){)}^2$
• proof $$\begin{array}{rl}\text{cov}(X,Y)& =E[(X-{\mu }_X)(Y-{\mu }_Y)]\\ & =E[(X-{\mu }_X)(Y-{\mu }_Y)]\end{array}$$
• theorem: if $X,Y$ independent, then for any $g,h:\mathbb{R}\to \mathbb{R}$, we have
  • $E[g(X)h(Y)]=E[g(X)]E[h(Y)]$
  • $E[XY]=EXE[Y]$
• theorem: if $X,Y$ independent: then $\text{cov}(X,Y)=0$
  • 👨‍🏫 however: $\text{cov}(X,Y)$ doesn't mean that $X,Y$ are independent
• proof: set some $X$ s.t. $E(X)=0$, and let $Y$ dependent on $X$
  • and let $XY=0$ always
  • then $\text{cov}(X,Y)=E[XY]-E[X][E[Y]=0]$
• theorem: $$\text{var}\left(\sum _{k=1}^n{X}_k\right)=\sum _{k=1}^n\text{var}(X_k)+2\sum _{1\le i<j\le n}\text{cov}(X_i,X_j)$$
• proof:

• w/ independent $X_i$ $$\text{var}\left(\sum _{k=1}^n{X}_k\right)=\sum _{k=1}^n\text{var}(X_k)$$
  • under independence: $\text{var}(\sum )=\sum \text{var}$
• proof:

Properties of covariance

1. $\text{var}(X)=\text{cov}(X,X)$
2. $\text{cov}(X,Y)=\text{cov}(Y,X)$
3. $\text{cov}({\sum }_{i=1}^n{a}_i{X}_i,{\sum }_{i=1}^m{n}_j{Y}_i)={\sum }_{i=1}^n{\sum }_{j=1}^m{a}_i{b}_j\text{cov}(X_i,Y_j)$

• 👨‍🏫 no independence assumed
• d

Covariance

Example

• sample variance
  • $X_i-\bar{X}$ is called deviation
  • sample variance: $S^2={\sum }_{i=1}^n\frac{(X_i-\bar{X}{)}^2}{n-1}$
  • find $\text{var}(\bar{X})$
    • $\text{var}(\bar{X})=\text{var}(\frac{1}{n}{\sum }_{i=1}^n{X}_i)=\frac{1}{n^2}\text{var}({\sum }_{i=1}^n{X}_i)$
    • w/ independence: $=\frac{1}{n^2}{\sum }_{i=1}^n\text{var}(X_i)=\frac{1}{n^2}\cdot n{\sigma }^2=\frac{{\sigma }^2}{n}$
  • find $E[S^2]$ $$\begin{array}{rlrl}E(S^2)& =E(\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2)& & \\ & =\frac{1}{n-1}\sum _{i=1}^{n}E[(X_i-\bar{X}{)}^2]& & \\ E(X_i-\bar{X}{)}^2& =E[(X_i-\mu )-(\bar{X}-\mu ){]}^2& & \\ & =E[(X_i-\mu {)}^2+(\bar{X}-\mu {)}^2-2(X_i-\mu )(\bar{X}-\mu )]& & \\ & =E[(X_i-\mu {)}^2]+E[(\bar{X}-\mu {)}^2]-2E[(X_i-\mu )(\bar{X}-\mu )]& & \\ & =\text{var}(X_i)+\text{var}(\bar{X})-2\text{cov}(X_i,\bar{X})& & \\ & ={\sigma }^2+\frac{1}{n}{\sigma }^2-2\text{cov}(X_i,\bar{X})& & \\ \text{cov}(X_i,\frac{1}{n}\sum _{j=1}^n{X}_j)& =\frac{1}{n}\sum _{j=1}^n\text{cov}(X_i,X_j)& & \\ \text{cov}(X_i,X_j)& =0& & \text{independence if }i\ne j\\ \text{cov}(X_i,\bar{X})& =\frac{1}{n}\text{cov}(X_i,X_i)=\frac{1}{n}\text{var}(X_i)=\frac{{\sigma }^2}{n}& & \\ E(X_i-\bar{X}{)}^2& ={\sigma }^2-\frac{1}{n}{\sigma }^2=\frac{n-1}{n}{\sigma }^2& & \\ E(S^2)& ={\sigma }^2& & \end{array}$$
  • thus, sample variance w/ $n-1$ is a good estimator