Joint distributions

Several random variables at once: their joint law, how it transforms, and how one variable informs another.

1. Joint densities and marginals (Def. 1.1); independence via factorisation (Prop. 1.2).
2. Transformations of random vectors by the Jacobian (Thm 1.3), e.g. sums by convolution (Cor. 1.4) and polar coordinates (Example 1.5).
3. Conditional densities (Def. 1.6) and conditional expectation $\mathbb{E}[X\mid Y]$ (Def. 1.7) with the tower property (Thm 1.8) and the law of total variance (Prop. 1.9).

Random variables $X, Y$ are jointly continuous with joint density $f_{X,Y}\ge0$ if $\mathbb{P}((X, Y)\in A) = \iint_Af_{X,Y}(x, y)\,\mathrm{d}x\,\mathrm{d}y$ for all (reasonable) $A\subseteq\mathbb{R}^2$. The marginal densities are $f_X(x) = \int f_{X,Y}(x, y)\,\mathrm{d}y$ and $f_Y(y) = \int f_{X,Y}(x, y)\,\mathrm{d}x$, and $\mathbb{E}[g(X, Y)] = \iint g\,f_{X,Y}$.

Jointly continuous $X, Y$ are independent iff $f_{X,Y}(x, y) = f_X(x)f_Y(y)$ (almost everywhere), iff $f_{X,Y}(x, y) = g(x)h(y)$ for some functions $g, h$ on a product region. A density supported on a non-rectangular region cannot factorise.

Let $(X, Y)$ have density $f_{X,Y}$ on an open set $D$, and $T: D\to E$ a $C^1$ bijection with $C^1$ inverse $(x, y) = (x(u, v), y(u, v))$. Then $(U, V) = T(X, Y)$ has density

If $X, Y$ are independent with densities $f_X, f_Y$, then $Z = X + Y$ has density $f_Z(z) = \int_{-\infty}^\infty f_X(x)f_Y(z - x)\,\mathrm{d}x$.

Let $X, Y$ be i.i.d. $N(0, 1)$ and $X = R\cos\Theta$, $Y = R\sin\Theta$. Then $f_{R,\Theta}(r, \theta) = \frac1{2\pi}e^{-r^2/2}\cdot r$ on $r > 0$, $0\le\theta < 2\pi$: $R$ and $\Theta$ are independent, $\Theta\sim U(0, 2\pi)$ and $R^2\sim\operatorname{Exp}(\frac12)$. Reversing this gives the Box–Muller method: from independent uniforms $U_1, U_2$, $\sqrt{-2\ln U_1}(\cos2\pi U_2, \sin2\pi U_2)$ are i.i.d. standard normals.

For jointly continuous $X, Y$ and $y$ with $f_Y(y) > 0$, the conditional density of $X$ given $Y = y$ is $f_{X\mid Y}(x\mid y) = \frac{f_{X,Y}(x, y)}{f_Y(y)}$, a density in $x$, and $\mathbb{E}[g(X)\mid Y = y] = \int g(x)f_{X\mid Y}(x\mid y)\,\mathrm{d}x$.

Let $\psi(y) = \mathbb{E}[X\mid Y = y]$. The conditional expectation $\mathbb{E}[X\mid Y]$ is the random variable $\psi(Y)$. It satisfies $\mathbb{E}[h(Y)X\mid Y] = h(Y)\mathbb{E}[X\mid Y]$ ("taking out what is known"), and $\mathbb{E}[X\mid Y] = \mathbb{E}[X]$ if $X, Y$ are independent.

$\mathbb{E}\big[\mathbb{E}[X\mid Y]\big] = \mathbb{E}[X]$ (when $\mathbb{E}|X| < \infty$). More generally $\mathbb{E}\big[h(Y)\mathbb{E}[X\mid Y]\big] = \mathbb{E}[h(Y)X]$ for bounded $h$.

$\operatorname{Var}(X) = \mathbb{E}\big[\operatorname{Var}(X\mid Y)\big] + \operatorname{Var}\big(\mathbb{E}[X\mid Y]\big)$, where $\operatorname{Var}(X\mid Y) = \mathbb{E}[X^2\mid Y] - \mathbb{E}[X\mid Y]^2$.

• Draw the support; it decides limits of integration and independence.
• Transformations: invert, compute the Jacobian of the inverse, find the new support, check the density integrates to $1$.
• Conditioning: $\mathbb{E}[X\mid Y]$ is a function of $Y$; average it with the tower property.