Gaussian Distribution

Gaussian Distribution#

Perhaps the most important distribution is the Gaussian Distribution, also known as the Normal Distribution.

General Gaussian Distribution#

Definition#

Definition 114 (Gaussian Distribution (PDF))

\(X\) is a continuous random variable with a Gaussian distribution if the probability density function is given by:

(147)#\[ \pdf(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \e \lcurl -\frac{(x - \mu)^2}{2 \sigma^2} \rcurl \]

where \(\mu\) is the mean and \(\sigma^2\) is the variance and are parameters of the distribution.

Some conventions:

  1. We write \(X \sim \gaussian(\mu, \sigma^2)\) to indicate that \(X\) has a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\).

  2. We sometimes denote \(\gaussian\) as \(\normal\) or \(\gaussiansymbol\).

Definition 115 (Gaussian Distribution (CDF))

There is no closed form for the CDF of the Gaussian distribution. But we will see that it is easy to compute the CDF of the Gaussian distribution using the CDF of the standard normal distribution later.

Therefore, the CDF of the Gaussian distribution is given by:

(148)#\[ \cdf(x) = \int_{-\infty}^x \pdf(x) \, \mathrm{d}x \]

Yes, you have to integrate the PDF to get the CDF but there is no “general” solution.

The PDF of the Gaussian distribution is shown below. The code is referenced and modified from [Shea, 2021].

Hide code cell source 
 1import numpy as np
 2import matplotlib.pyplot as plt
 3import scipy.stats as stats
 4
 5mean_1, mean_2, mean_3, mean_4, mean_5 = -3, -0.3, 0, 1.2, 3
 6sigma = 1
 7
 8x = np.linspace(-10, 10, 1000)
 9
10all_params = [
11    (mean_1, sigma),
12    (mean_2, sigma),
13    (mean_3, sigma),
14    (mean_4, sigma),
15    (mean_5, sigma),
16]
17
18for params in all_params:
19    N = stats.norm(params[0], params[1])
20    label = f"Normal({N.mean()}, $\sigma=${N.std() :.1f})"
21    plt.plot(x, N.pdf(x), label=label)
22
23plt.xlim(-12, 12)
24plt.ylim(0, 0.5)
25plt.legend(prop={'size': 6})
26plt.title("Fixing $\sigma$ = 1, and varying $\mu$")
27plt.show()
28
29mean = 0
30sigma_1, sigma_2, sigma_3, sigma_4, sigma_5 = 0.3, 0.5, 1, 4, 6
31
32x = np.linspace(-10, 10, 1000)
33
34all_params = [
35    (mean, sigma_1),
36    (mean, sigma_2),
37    (mean, sigma_3),
38    (mean, sigma_4),
39    (mean, sigma_5),
40]
41
42for params in all_params:
43    N = stats.norm(params[0], params[1])
44    label = f"Normal({N.mean()}, $\sigma=${N.std() :.1f})"
45    plt.plot(x, N.pdf(x), label=label)
46
47plt.xlim(-12, 12)
48plt.ylim(0, 1.5)
49plt.legend(prop={"size": 6})
50plt.title("Fixing $\mu$ = 0, and varying $\sigma$")
51plt.show()
../../_images/f7a5d8fcb8e999e9ead552df5a5515987b9e89dbeed94a2a39fb115faa0dbc5c.png ../../_images/7e0aefdefbddd3e3a05d77e5ef0bff11edbf0f04fb0d1693ff98753fc1102673.png

Remark 39 (Gaussian Distribution (PDF))

Note in the plots above, the PDF of the Gaussian distribution is symmetric about the mean \(\mu\), and given a small enough \(\sigma\), the PDF can be greater than 1, as long as the area under the curve is 1.

Note that the PDF curve moves left and right as \(\mu\) increases and decreases, respectively. Similarly, the PDF curve gets narrower and wider as \(\sigma\) increases and decreases, respectively.

Expectation and Variance#

Theorem 30 (Expectation and Variance of the Gaussian Distribution)

The observant reader may notice that the mean and variance of the Gaussian distribution are parameters of the distribution. However, this is not proven previously.

If \(X\) is a continuous random variable with an gaussian distribution with mean \(\mu\) and variance \(\sigma^2\), then the expectation and variance of \(X\) are given by:

(149)#\[ \expectation \lsq X \rsq = \mu \qquad \text{and} \qquad \var \lsq X \rsq = \sigma^2 \]

Linear Transformation of Gaussian Distribution#

Theorem 31 (Linear Transformation of Gaussian Distribution)

Let \(X \sim \gaussiansymbol(\mu, \sigma^2)\) be a random variable with a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\).

Let \(Y = aX + b\) be a linear transformation of \(X\).

Then, \(Y \sim \gaussiansymbol(a\mu + b, a^2\sigma^2)\) is a Gaussian distribution with mean \(a\mu + b\) and variance \(a^2\sigma^2\).

In other words, \(Y\) experiences a shift and scale change in coordinates, and while the distribution may not longer be the same as \(X\), it still belongs to the same family.

Standard Gaussian Distribution#

Motivation#

In continuous distributions, we are often asked to evaluate the probability of an interval \([a, b]\). For example, if \(X \sim \gaussian(\mu, \sigma^2)\), what is the probability that \(X\) is between \(a\) and \(b\)? If we use PDF to compute the probability, we will have to integrate the PDF over the interval \([a, b]\).

\[ \P \lsq a \leq X \leq b \rsq = \int_a^b \pdf(x) \, \mathrm{d}x \]

We have seen that there is no closed form solution for this integral.

Furthermore, in practice, we use the CDF to compute the probability of an interval.

\[ \P \lsq a \leq X \leq b \rsq = \cdf(b) - \cdf(a) \]

However, we have also seen that there is no closed form solution for the CDF of the Gaussian distribution since the CDF is an integral of the PDF.

This prompted the existence of the standard normal distribution.

Definition#

Definition 116 (Standard Gaussian Distribution)

Let \(X\) be a continuous random variable with a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\). Then \(X\) is said to be a Standard Gaussian if it has a Gaussian distribution with mean 0 and variance 1.

(150)#\[ \pdf(x) = \frac{1}{\sqrt{2 \pi}} \exp \lcurl -\frac{x^2}{2} \rcurl \]

In notations, it is \(X \sim \gaussiansymbol(0, 1)\), with \(\mu = 0\) and \(\sigma^2 = 1\).

Definition 117 (Standard Normal Distribution (CDF))

The CDF of the Standard Gaussian is given by:

\[ \Phi(x) \defeq \cdf(x) = \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^x \exp \lcurl -\frac{t^2}{2} \rcurl \, \mathrm{d}t \]

Take note of the \(\Phi\) notation, which is the standard notation for the CDF of the Standard Gaussian.

Hide code cell source 
 1import sys
 2from pathlib import Path
 3parent_dir = str(Path().resolve().parents[2])
 4sys.path.append(parent_dir)
 5
 6import numpy as np
 7import scipy.stats as stats
 8
 9from omnivault.utils.reproducibility.seed import seed_all
10from omnivault.utils.probability_theory.plot import plot_continuous_pdf_and_cdf
11seed_all()
12
13# x = np.linspace(0, 10, 5000)
14mean, sigma = 0, 1
15X = stats.norm(mean, sigma)
16
17plot_continuous_pdf_and_cdf(X, -5, 5, title="Normal Distribution $\mu=0, \sigma=1$", figsize=(15, 5))

/home/runner/work/omniverse/omniverse/omnivault/utils/reproducibility/seed.py:120: UserWarning: Deterministic mode is activated. This will negatively impact performance and may cause increase in CUDA memory footprint.
  configure_deterministic_mode()
../../_images/9ad495be690bee24183d66821199364099f3b48ed9f20183a49fabd8748dc439.png

Error Function#

Definition 118 (Error Function)

The error function is defined as:

\[ \text{erf}(x) \defeq \frac{2}{\sqrt{\pi}} \int_0^x \exp \lcurl -t^2 \rcurl \, \mathrm{d}t \]

The connection of the error function to the CDF of the Standard Gaussian is given by:

(151)#\[ \Phi(x) = \frac{1}{2} \left( 1 + \text{erf} \lcurl \frac{x}{\sqrt{2}} \rcurl \right) \qquad \text{and} \qquad \text{erf}(x) = 2 \Phi(x\sqrt{2}) - 1 \]

CDF of Arbitrary Gaussian Distribution#

Theorem 32 (CDF of Arbitrary Gaussian Distribution)

Let \(X\) be a continuous random variable with a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\). Then we can convert the CDF of \(X\) to the CDF of the Standard Gaussian by:

(152)#\[ \cdf(x) = \Phi \lpar \frac{x - \mu}{\sigma} \rpar \]

Proof. CDF of Arbitrary Gaussian Distribution The definition of a Gaussian distribution is given by:

\[\begin{split} \begin{aligned} \cdf(x) &= \P \lsq X \leq x \rsq \\ &= \int_{-\infty}^x \pdf(t) \, \mathrm{d}t \\ &= \int_{-\infty}^x \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \lcurl -\frac{(t - \mu)^2}{2 \sigma^2} \rcurl \, \mathrm{d}t \\ \end{aligned} \end{split}\]

We can use the (integration by) substitution \(u = \frac{t - \mu}{\sigma}\) to get [Chan, 2021]:

\[\begin{split} \begin{aligned} \int_{-\infty}^x \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \lcurl -\frac{(t - \mu)^2}{2 \sigma^2} \rcurl \, \mathrm{d}t &= \int_{-\infty}^{\frac{x - \mu}{\sigma}} \frac{1}{\sqrt{2 \pi}} \exp \lcurl -\frac{u^2}{2} \rcurl \, \mathrm{d}u \\ &= \Phi \lpar \frac{x - \mu}{\sigma} \rpar \end{aligned} \end{split}\]

Corollary 8 (Probability of an Interval)

As a consequence of Theorem 32, we can compute the probability of an interval for an arbitrary Gaussian distribution. This solves our problem of computing the probability of an interval for a Gaussian Distribution earlier.

(153)#\[\begin{split} \begin{aligned} \P \lsq a \leq X \leq b \rsq &= \cdf(b) - \cdf(a) \\ &= \P \lsq X \leq b \rsq - \P \lsq X \leq a \rsq \\ &= \Phi \lpar \frac{b - \mu}{\sigma} \rpar - \Phi \lpar \frac{a - \mu}{\sigma} \rpar \end{aligned} \end{split}\]

Corollary 9 (Corollary of Standard Gaussian)

As a consequence of Theorem 32, we have the following corollaries [Chan, 2021]:

  • \(\Phi(y) = 1 - \Phi(-y)\)

  • \(\P \lsq X \geq b \rsq = 1 - \Phi \lpar \frac{b - \mu}{\sigma} \rpar\)

  • \(\P \lvert X \rvert \geq b = 1 - \Phi \lpar \frac{b - \mu}{\sigma} \rpar + \Phi \lpar \frac{-b - \mu}{\sigma} \rpar\)

References and Further Readings#

Contents