Covariance

• correlation (coefficient) of r.v. $X,Y$: defined by $\rho (X,Y)$ defined by: $$\rho (X,Y)=\frac{\text{cov}(X,Y)}{\sqrt{\text{var}(X)\text{var}(Y)}}=\frac{\text{cov}(X,Y)}{{\sigma }_X{\sigma }_Y}$$
• thus: $-1\le \rho (X,Y)\le 1$
• proof $$0\le \text{var}(\frac{X}{{\sigma }_X}+\frac{Y}{{\sigma }_Y^2})=\frac{\text{var}(X)}{{\sigma }_X^2}+\frac{\text{var}(Y)}{{\sigma }_Y^2}+\frac{n\text{cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=$$
• remarks
  1. d
  2. $\rho X,Y=1$ iff $Y=aX+b$ w/ ${\sigma }_Y{\sigma }_X=0$
     • tjere $a={\sigma }_X/{\sigma }_X$
  3. s
  • if $Y=aX+b$ w/ $a={\sigma }_Y/{\sigma }_X$
  4. d
  5. independence of $X,Y$ $\rho (X,Y)$ implies
     • but the opposite: doesn't get support
• d

Example

• variance of binomial
  • $X\sim Bin(n,p)$: given by $X=I_1+I_2+\cdots +I_n$
  • $I_k$: whether $k$-th trial is success / not
  • $\text{var}(X)=\text{var}({\sum }_{k=1}^n)={\sum }_{k=1}^n\text{var}(I_k)={\sum }_{k=1}^{n}p(1-p)=np(1-p)$
• example
  • $X=\pm 1$, w/ fair probability
  • $Y=1$ if $X=1$, $Y=-1$ if $Y=-1$
  • $E(X)=E(Y)=0$, and as $E(XY)=1$, $\text{cov}(X,Y)=1$
  • $\rho (X,Y_1)=\rho (X,Y_2)=1$

Conditional expectation

1. w/ $X,Y$: jointly distributed discrete r.v. $$E[X[Y=u]]:=\sum _{x}x{p}_{X|Y}(x,y)$$
   • w/ $p_Y(y)>0$
   • $p_{X|Y}(c,y)=\frac{p_{X,Y}(x,y)}{p_Y(y)}=\frac{\text{joint}}{\text{marginal}}$
     • a valid PMF!
2. w/ $X,Y$: jointly distributed continuous r.v. $$E[X[Y=u]]:={\int }_{-1\infty }^{\infty }x{f}_{X|Y}(x,y)$$
   • w/ $p{f}_Y(y)>0$
   • $p_{X|Y}(c,y)=\frac{p_{X,Y}(x,y)}{p_Y(y)}=\frac{\text{joint}}{\text{marginal}}$
   • w/ $p_Y$ a valid PDF!

• important formula
  1. s $$E[g(X)|Y=y]=\{\begin{array}{llc}{\sum }_{x}g(x)p_{X|Y}(x|y)& & \text{discrete}\\ {\int }_{-\infty }^{\infty }g(x)f_{X|Y}(x|y)\mathrm{d}x& & \text{continuous}\end{array}$$
     • thus: conditional expectation of sum = sum of conditional expectation $$E\left[\sum _{k=1}^n{X}_k|Y=y\right]=\sum _{k=1}^{n}E[X_k|Y=y]$$
• furthermore, consider $E[X|Y]$ as function of $Y$, $g(Y)$
  • e.g. $E[X|Y=y]=y/2-5$, then $E[X|Y]=Y/2-5$
• theorem: $$E[X]=E[\underset{g(Y)}{\underbrace{E(X|Y)}}]=\{\begin{array}{ll}{\sum }_{y}E(X|Y=y)P(Y=y)& \text{discrete }Y\\ {\int }_{-\infty }^{\infty }E(X|Y=y)f_Y(y)\mathrm{d}y& \text{continuous }Y\end{array}$$
  • i.e. nesting property of expectation
  • $E_Y[E_X[X|Y]]$
  • and $g_Y=E_X[X|Y]$
• d

Example

• $f(x,y)=\{\begin{array}{llc}\frac{e^{-x/y}{e}^{-y}}{y}& & 0<x<\infty ,0<y<\infty \\ 0& & \text{otherwise}\end{array}$
• $f_{X|Y}(x|y)=\frac{f(x,y)}{f_Y(y)}$
• for $y>0$: $$\begin{array}{rl}{f}_Y(y)& ={\int }_{-\infty }^{\infty }f(x,y)\mathrm{d}x={\int }_0^{\infty }{e}^{-x/y}\cdot \frac{1}{y}{e}^{-y}\mathrm{d}x\\ & =e^{-y}{\int }_0^{\infty }{e}^{-x/y}\cdot \frac{1}{y}\mathrm{d}x\\ & =e^{-y}\end{array}$$
• $f_{X|Y}(x|y)=\{\begin{array}{l}d\end{array}$
• d