Moment Generating Function

Moment Generating Function#

Definition #

In probability theory and statistics, a moment generating function (MGF) is a function that uniquely determines the probability distribution of a random variable. Specifically, the MGF of a random variable is the expected value of the exponential function raised to a certain power of the random variable.

Definition 164 (Moment Generating Function)

For any random variable $X$ , the moment-generating function (MGF) $M_{X} (s)$ is

M_{X} (s) = E [e^{s X}] .

and it is well defined if there exists a positive constant $c$ such that $M_{X} (s)$ is finite for all $s \in R$ with $| s | \leq c$ (i.e. $s \in [- c, c]$ ]).

In other words, the MGF of $X$ is the expected value of the exponential function raised to a certain power of $X$ .

By the Law of the Unconscious Statistician, and Property 1 for the discrete case and Property 18 for the continuous case, we can easily see the following:

For the discrete case, the MGF is

M_{X} (s) = \sum_{x \in Ω} e^{s x} p_{X} (x),

whereas in the continuous case, the MGF is

M_{X} (s) = \int_{- \infty}^{\infty} e^{s x} f_{X} (x) d x .

The continuous case should remind us of the definition of a Laplace transform. For any function $f (t)$ , the Laplace transform is

L [f] (s) = \int_{- \infty}^{\infty} f (t) e^{s t} d t .

From this perspective, we can interpret the MGF as the Laplace transform of the PDF. The argument $s$ of the output can be regarded as the coordinate in the Laplace space. If $s = - j ω$ , then $M_{X} (j ω)$ becomes the Fourier transform of the PDF [Chan, 2021].

Example 38 (Moment Generating Function Example 1.)

Consider a random variable $X$ with three states $0, 1, 2$ and with probability masses $\frac{2}{6}, \frac{3}{6}, \frac{1}{6}$ respectively. Find the MGF.

The moment-generating function is

\begin{array}{r} \begin{aligned} M_{X} (s) = E [e^{s X}] & = e^{s 0} \cdot \frac{2}{6} + e^{s 1} \cdot \frac{3}{6} + e^{s 2} \cdot \frac{1}{6} \\ = \frac{1}{3} + \frac{e^{s}}{2} + \frac{e^{2 s}}{6} . \end{aligned} \end{array}

Question and solution from page 324 of [Chan, 2021].

Example 39 (Moment Generating Function Example 2.)

Find the MGF for an exponential random variable.

The MGF of an exponential random variable can be found as

M_{X} (s) = E [e^{s X}] = \int_{0}^{\infty} e^{s x} λ e^{- λ x} d x = \int_{0}^{\infty} λ e^{(s - λ) x} d x = \frac{λ}{λ - s}, if λ > s

Question and solution from page 325 of [Chan, 2021].

Properties #

Theorem 57 (Moment Generating Function Properties)

The $M G F$ has the properties that

$M_{X} (0) = 1$
${\frac{d^{k}}{d s^{k}} M_{X} (s) |}_{s = 0} = E [X^{k}]$ , for any positive integer $k$ . In other words, if $s$ is set to $0$ , then the $k$ -th derivative of the MGF is the $k$ -th moment of $X$ .

Proof can be found in [Chan, 2021], page 325.

Example 40 (Moments of Bernoulli Random Variable)

Let $X$ be a Bernoulli random variable with parameter $p$ . Find the first two moments using MGF.

The MGF of a Bernoulli random variable is

\begin{array}{r} \begin{aligned} M_{X} (s) & = E [e^{s X}] \\ = e^{s 0} p_{X} (0) + e^{s 1} p_{X} (1) \\ = (1) (1 - p) + (e^{s}) (p) \\ = 1 - p + p e^{s} . \end{aligned} \end{array}

The first and the second moment, using the derivative approach, are

\begin{array}{r} \begin{array}{c} E [X] = {\frac{d}{d s} M_{X} (s) |}_{s = 0} = {\frac{d}{d s} (1 - p + p e^{s}) |}_{s = 0} = {p e^{s} |}_{s = 0} = p, \\ E [X^{2}] = {\frac{d^{2}}{d s^{2}} M_{X} (s) |}_{s = 0} = {\frac{d^{2}}{d s^{2}} (1 - p + p e^{s}) |}_{s = 0} = {p e^{s} |}_{s = 0} = p . \end{array} \end{array}

Question and solution from page 326 of [Chan, 2021].

The table below lists the MGF of some common discrete and continuous random variables.

\begin{array}{r} \begin{array}{lllll} Distribution & PMF / PDF & E [X] & Var [X] & M_{X} (s) \\ Bernoulli & p_{X} (1) = p and p_{X} (0) = 1 - p & p & p (1 - p) & 1 - p + p e^{s} \\ Binomial & p_{X} (k) = (\begin{matrix} n \\ k \end{matrix}) p^{k} (1 - p)^{n - k} & n p & n p (1 - p) & {(1 - p + p e^{s})}^{n} \\ Geometric & p_{X} (k) = p (1 - p)^{k - 1} & \frac{1}{p} & \frac{1 - p}{p^{2}} & \frac{p e^{s}}{1 - (1 - p) e^{s}} \\ Poisson & p_{X} (k) = \frac{λ^{k} e^{- λ}}{k!} & λ & λ & e^{λ (e^{s} - 1)} \\ Gaussian & f_{X} (x) = \frac{1}{\sqrt{2 π σ^{2}}} \exp {- \frac{(x - μ)^{2}}{2 σ^{2}}} & μ & σ^{2} & \exp {μ s + \frac{σ^{2} s^{2}}{2}} \\ Exponential & f_{X} (x) = λ \exp {- λ x} & \frac{1}{λ} & \frac{1}{λ^{2}} & \frac{λ}{λ - s} \\ Uniform & f_{X} (x) = \frac{1}{b - a} & \frac{a + b}{2} & \frac{(b - a)^{2}}{12} & \frac{e^{s b} - e^{s a}}{s (b - a)} \end{array} \end{array}

Why is it useful?#

Moments #

The moment generating function of $X$ provides us with all the moments of $X$ , which is why it is called the moment generating function.

Uniquely determines the distribution #

Moreover, the MGF uniquely determines the distribution, provided it exists. This means that if two random variables have the same MGF, they must have the same distribution. Therefore, finding the MGF of a random variable enables us to determine its distribution. This approach is particularly useful when working with sums of multiple independent random variables. The proof can be found here.

Moment Generating Function

Contents

Moment Generating Function#

Definition #

Properties #

Why is it useful?#

Moments #

Uniquely determines the distribution #

Further Readings #

Moment Generating Function

Contents

Moment Generating Function#

Definition#

Properties#

Why is it useful?#

Moments#

Uniquely determines the distribution#

Further Readings#

Connect with me!

Share this page!

Definition #

Properties #

Moments #

Uniquely determines the distribution #

Further Readings #