Moments and Variance

Moments and Variance#

Notation#

Notation

Let \(\P\) be a probability function defined over the probability space \(\pspace\).

Let \(X\) be a discrete random variable with \(\S = \lset \xi_1, \xi_2, \ldots \rset\) unless otherwise stated.

Moments#

Definition 80 (Moments)

The \(k\)-th moment of \(X\) is defined as:

\[ \expectation \lsq X^k \rsq = \sum_{x \in X(\S)} x^k \cdot \P \lsq X = x \rsq \]

This definition follows from Property 1 in Expectation.

Essentially, this means that the \(k\)-th moment is the expectation of \(X^k\).

Variance#

Definition 81 (Variance)

The variance of \(X\) is defined as:

(132)#\[ \var \lsq X \rsq = \expectation \lsq \lpar X - \mu \rpar^2 \rsq \]

where \(\mu = \expectation \lsq X \rsq\) is the expectation of \(X\).

We denote \(\sigma^2\) as \(\var\) for short-hand notation.

We also have an equivalent definition of variance, which is more used in practice.

Definition 82 (Variance (Alternative))

The variance of \(X\) is defined as:

(133)#\[ \var \lsq X \rsq = \expectation \lsq X^2 \rsq - \expectation \lsq X \rsq^2 \]

Standard Deviation#

Definition 83 (Standard Deviation)

In the definition of Definition 81, we have \(\var \lsq X \rsq\) to have a different unit than \(X\). If \(X\) is measured in meters, then \(\var \lsq X \rsq\) is measured in meters squared. To solve this issue, we define a new measure called the standard deviation, which is the square root of the variance [Pishro-Nik, 2014].

(134)#\[ \std \lsq X \rsq = \sqrt{\var \lsq X \rsq} \]

Properties of Moments and Variance#

The properties of moments and variance are as follows:

Property 6 (Scaling)

For any constant \(c\), we have:

(135)#\[ \expectation \lsq c \cdot X \rsq = c^k \cdot \expectation \lsq X \rsq \]

where \(k\) is the order of the moment.

Property 7 (DC Shift)

For any constant \(c\), we have:

(136)#\[ \expectation \lsq (X + c) \rsq = \expectation \lsq X \rsq \]

The intuition is that shifting the random variable by a constant does not change the spread of the random variable.

Property 8 (Linearity)

Combining Property 6 and Property 7, we have:

(137)#\[ \expectation \lsq a \cdot X + b \rsq = a^k \cdot \expectation \lsq X \rsq \]

where \(k\) is the order of the moment.

Concept#

Concept

  • Variance is a measure of how spread out a distribution is. More concretely, it is the expectation of the squared deviation from the expectation of the distribution. One can think of for every data point in the distribution, how far is each data point from the expectation (population mean). The variance is the average of these data points (squared to make it positive).

  • Variance is deterministic and is synonymous with Population Variance.

  • Sample Variance is the variance of a random sample from the true population, which is a random variable.

References and Further Readings#

  • Pishro-Nik, Hossein. “Chapter 3.2.4. Variance.” In Introduction to Probability, Statistics, and Random Processes, 202-206. Kappa Research, 2014.

  • Chan, Stanley H. “Chapter 3.4.4. Momenets and variance.” In Introduction to Probability for Data Science, 133-136. Ann Arbor, Michigan: Michigan Publishing Services, 2021.

Contents