Closed-Form Derivation of Cross-Entropy Between Gaussian Distributions

Step-by-step mathematical derivation of the closed-form cross-entropy between two Gaussian distributions using properties of expectation and variance.

2 min read

Here, we derive the closed-form expression for the cross-entropy $H(p_1, p_2)$ between two Gaussian distributions $p_1(x)$ and $p_2(x)$.

Preparation

Gaussian Distribution (Normal Distribution)

The probability density function of a Gaussian distribution with mean $\mu$ and variance $\sigma^2$ is given by:

$$ p(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\lbrace-\frac{(x-\mu)^2}{2\sigma^2}\rbrace $$

Properties of Expectation and Variance

For a random variable $X$ following a Gaussian distribution:

  • Expectation: $\mathbb{E}[X] = \mu$
  • Variance: $\mathbb{V}[X] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2 = \sigma^2$
  • Therefore, $\mathbb{E}[X^2] = \mu^2 + \sigma^2$

Derivation of Cross-Entropy

The cross-entropy $H(p_1, p_2)$ between two Gaussian distributions $p_1(x) = \mathcal{N}(x | \mu_1, \sigma_1^2)$ and $p_2(x) = \mathcal{N}(x | \mu_2, \sigma_2^2)$ is defined as:

$$ H(p_1, p_2) = -\int_{-\infty}^{\infty} p_1(x) \log p_2(x) dx = -\mathbb{E}_{p_1}[\log p_2(x)] $$

First, let’s expand $\log p_2(x)$:

$$ \log p_2(x) = \log \left(\frac{1}{\sqrt{2\pi\sigma_2^2}} \exp\lbrace-\frac{(x-\mu_2)^2}{2\sigma_2^2}\rbrace\right) $$ $$ = -\frac{1}{2} \log(2\pi\sigma_2^2) - \frac{(x-\mu_2)^2}{2\sigma_2^2} $$

Now, substitute this into the expectation formula for cross-entropy:

$$ H(p_1, p_2) = -\mathbb{E}{p_1}\left[-\frac{1}{2} \log(2\pi\sigma_2^2) - \frac{(x-\mu_2)^2}{2\sigma_2^2}\right] $$ $$ = \frac{1}{2} \log(2\pi\sigma_2^2) + \frac{1}{2\sigma_2^2} \mathbb{E}{p_1}[(x-\mu_2)^2] $$

Next, expand $\mathbb{E}_{p_1}[(x-\mu_2)^2]$:

$$ \mathbb{E}{p_1}[(x-\mu_2)^2] = \mathbb{E}{p_1}[x^2 - 2x\mu_2 + \mu_2^2] $$ $$ = \mathbb{E}{p_1}[x^2] - 2\mu_2 \mathbb{E}{p_1}[x] + \mu_2^2 $$

Since $p_1(x)$ is a Gaussian distribution with mean $\mu_1$ and variance $\sigma_1^2$, we substitute $\mathbb{E}{p_1}[x] = \mu_1$ and $\mathbb{E}{p_1}[x^2] = \mu_1^2 + \sigma_1^2$:

$$ \mathbb{E}_{p_1}[(x-\mu_2)^2] = (\mu_1^2 + \sigma_1^2) - 2\mu_2 \mu_1 + \mu_2^2 $$ $$ = (\mu_1 - \mu_2)^2 + \sigma_1^2 $$

Finally, substitute this back into the cross-entropy equation:

$$ H(p_1, p_2) = \frac{1}{2} \log(2\pi\sigma_2^2) + \frac{(\mu_1 - \mu_2)^2 + \sigma_1^2}{2\sigma_2^2} $$

This is the closed-form expression for the cross-entropy between two Gaussian distributions.

関連記事