In statistics, a moment is a quantitative measure that characterizes the shape of a probability distribution. The concept is borrowed from physics (where moment refers to rotational force) and applied to statistics. Moments are used to quantitatively describe the center, spread, asymmetry, and peakedness of a random variable $X$’s distribution.
Definition of Moments
Given a random variable $X$ with probability density function (or probability mass function) $f(x)$, the $n$-th moment about the origin $\mu’_n$ is defined as:
$$ \mu’n = \mathbb{E}[X^n] = \int{-\infty}^{\infty} x^n f(x) dx $$
The $n$-th moment about the mean $\mu = \mathbb{E}[X]$ (central moment) $\mu_n$ is defined as:
$$ \mu_n = \mathbb{E}[(X - \mu)^n] = \int_{-\infty}^{\infty} (x - \mu)^n f(x) dx $$
Meaning of Lower-Order Moments
0th Moment
- 0th moment about the origin: $\mu’0 = \int{-\infty}^{\infty} x^0 f(x) dx = \int_{-\infty}^{\infty} f(x) dx = 1$ This represents the fundamental property of probability that the integral of the probability density function over the entire domain equals 1.
1st Moment
- 1st moment about the origin: $\mu’1 = \int{-\infty}^{\infty} x^1 f(x) dx = \mathbb{E}[X] = \mu$ This represents the mean (expected value) of the random variable. It indicates the center of the distribution.
2nd Moment
- 2nd central moment: $\mu_2 = \mathbb{E}[(X - \mu)^2] = \sigma^2$ This represents the variance of the random variable. It indicates the spread of the distribution.
Meaning of Higher-Order Moments
3rd Moment (Skewness)
Using the 3rd central moment $\mu_3$, the skewness $\gamma_1$ of a distribution is defined as:
$$ \gamma_1 = \frac{\mu_3}{\sigma^3} $$
Skewness indicates the asymmetry of the distribution:
- $\gamma_1 > 0$: The distribution is right-skewed (has a longer right tail)
- $\gamma_1 < 0$: The distribution is left-skewed (has a longer left tail)
- $\gamma_1 = 0$: The distribution is symmetric (e.g., normal distribution)
4th Moment (Kurtosis)
Using the 4th central moment $\mu_4$, the kurtosis $\gamma_2$ of a distribution is defined as:
$$ \gamma_2 = \frac{\mu_4}{\sigma^4} - 3 $$
Kurtosis indicates the peakedness and tail heaviness of the distribution. Since the kurtosis of a normal distribution is 3, the value minus 3 (excess kurtosis) is typically used:
- $\gamma_2 > 0$: The distribution is more peaked than a normal distribution with heavier tails (data concentrated at the center with more outliers)
- $\gamma_2 < 0$: The distribution is flatter than a normal distribution with lighter tails
- $\gamma_2 = 0$: The distribution has the same peakedness as a normal distribution
Moments are fundamental tools for capturing various characteristics of probability distributions and play an important role in understanding data properties in data analysis and machine learning.