Proof. To compute the variance of $a X + b$ , we first need to compute the mean value of $a X + b$ . To avoid confusion, we will denote the mean of $X$ as $μ_{X}$ and the mean of $a X + b$ as $μ_{a X + b}$ . The mean of $a X + b$ is as follows:

\begin{aligned} μ_{a X + b} & = E (a X + b) \\ = a \cdot E (X) + b \\ = a \cdot μ_{X} + b \end{aligned}

Now, use the definition of variance:

\begin{aligned} V (a X + b) & = E [(a X + b - μ_{a X + b})^{2}] \\ = E [(a X + b - (a μ_{X} + b))^{2}] \\ = E [(a X - a μ_{X})^{2}] \\ = E [(a (X - μ_{X}))^{2}] \\ = E [a^{2} (X - μ_{X})^{2}] \\ = a^{2} \cdot E [(X - μ_{X})^{2}] \\ = a^{2} \cdot V (X) \end{aligned}

This completes the proof.

■

Theorem.

Variance of sum of two random variables

The variance of the sum of two random variables $X$ and $Y$ is:

\begin{array}{r} V (X + Y) = V (X) + V (Y) + 2 \cdot cov (X, Y) \end{array}

Where $cov (X, Y)$ is the covariance of $X$ and $Y$ .

Proof. Let the mean of random variables $X$ and $Y$ be $μ_{X}$ and $μ_{Y}$ , respectively. We also denote the mean of $X + Y$ as $μ_{X + Y}$ . From the definition of variance:

\begin{aligned} V (X + Y) & = E [(X + Y - μ_{X + Y})^{2}] \\ = E [(X + Y - E (X + Y))^{2}] \\ = E [(X + Y - E (X) - E (Y))^{2}] \\ = E [(X + Y - μ_{X} - μ_{Y})^{2}] \\ = E [((X - μ_{X}) + (Y - μ_{Y}))^{2}] \\ = E [(X - μ_{X})^{2} + (Y - μ_{Y})^{2} + 2 (X - μ_{X}) (Y - μ_{Y})] \\ = E [(X - μ_{X})^{2}] + E [(Y - μ_{Y})^{2}] + 2 E [(X - μ_{X}) (Y - μ_{Y})] \\ = V (X) + V (Y) + 2 \cdot cov (X, Y) \end{aligned}

This completes the proof.

■

Theorem.

Variance of the difference of two random variables

The variance of the difference of two random variables $X$ and $Y$ is:

V (X - Y) = V (X) + V (Y) - 2 \cdot cov (X, Y)

Where $cov (X, Y)$ is the covariance of $X$ and $Y$ .

Proof. We know from theoremlink that:

\begin{array}{r} V (X + Y) = V (X) + V (Y) + 2 \cdot cov (X, Y) \end{array}

Setting $Y = - Y^{'}$ gives:

\begin{aligned} V (X - Y^{'}) & = V (X) + V (- Y^{'}) + 2 \cdot cov (X, - Y^{'}) \\ = V (X) + V (Y^{'}) - 2 \cdot cov (X, Y^{'}) \end{aligned}

For the second equality, we used theoremlink and theorem TODO.

■

Theorem.

Variance of sum of two independent variables

The variance of a sum of two independent random variables $X$ and $Y$ is:

V (X + Y) = V (X) + V (Y)

Proof. We know from theoremlink that:

\begin{array}{r} V (X + Y) = V (X) + V (Y) + 2 \cdot cov (X, Y) \end{array}

If $X$ and $Y$ are independent, then their covariance is zero. This completes the proof.

■

Theorem.

Interchanging the summation sign and variance

If $X_{1}, X_{2}, \dots, X_{n}$ are independent random variables, then:

\sum_{i = 1}^{n} V (X_{i}) = V (\sum_{i = 1}^{n} X_{i})

Note that this is only true when the random variables are independent.

Proof. Let's start from the left-hand side:

\begin{matrix} (1) & \sum_{i = 1}^{n} V (X_{i}) = V (X_{1}) + V (X_{2}) + \dots + V (X_{n}) \end{matrix}

From theoremlink, we know that $V (X) + V (Y) = V (X + Y)$ if $X$ and $Y$ are independent random variables. Since $X_{1}, X_{2}, \dots, X_{n}$ are all independent, then $(1)$ becomes:

\begin{aligned} \sum_{i = 1}^{n} V (X_{i}) & = V (X_{1} + X_{2} + \dots + X_{n}) \\ = V (\sum_{i = 1}^{n} X_{i}) \end{aligned}

This completes the proof.

■

Theorem.

Law of total variance

The law of total variance states that:

V (X) = E [V (X | Y)] + V [E (X | Y)]

Here, $E [V (X | Y)]$ is referred to as the within-group variation, while $V [E (X | Y)]$ is referred to as the between-group variation.

Proof. Let us color-code the law of total variance:

\begin{matrix} (2) & V (X) = E [V (X | Y)] + V [E (X | Y)] \end{matrix}

Let's start with the green term. From theoremlink, we have that:

V (X | Y) = E (X^{2} | Y) - [E (X | Y)]^{2}

Taking the expected value on both sides:

E [V (X | Y)] = E [E (X^{2} | Y)] - E [E (X | Y)^{2}]

From the law of total expectation TODO, we know that $E [E (X^{2} | Y)] = E (X^{2})$ :

\begin{matrix} (3) & E [V (X | Y)] = E (X^{2}) - E [E (X | Y)^{2}] \end{matrix}

We then move on to the red term in $(2)$ . From theoremlink, we have that:

V [E (X | Y)] = E [(E (X | Y))^{2}] - [E (E (X | Y))]^{2}

Again, from the law of total expectation TODO, we have that $[E (E (X | Y))]^{2} = [E (X)]^{2}$ :

\begin{matrix} (4) & V [E (X | Y)] = E [(E (X | Y))^{2}] - [E (X)]^{2} \end{matrix}

Taking the sum of $(3)$ and $(4)$ yields:

\begin{aligned} E [V (X | Y)] + V [E (X | Y)] & = E (X^{2}) - E [E (X | Y)^{2}] + E [(E (X | Y))^{2}] - [E (X)]^{2} \\ = E (X^{2}) - [E (X)]^{2} \\ = V (X) \end{aligned}

This completes the proof.

■

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!