Here, we derive the closed-form expression for the cross-entropy \(H(p_1, p_2)\) between two Gaussian distributions \(p_1(x)\) and \(p_2(x)\).
Preparation
Gaussian Distribution (Normal Distribution)
The probability density function of a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\) is given by:
\[ p(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\lbrace-\frac{(x-\mu)^2}{2\sigma^2}\rbrace \]Properties of Expectation and Variance
For a random variable \(X\) following a Gaussian distribution:
- Expectation: \(\mathbb{E}[X] = \mu\)
- Variance: \(\mathbb{V}[X] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2 = \sigma^2\)
- Therefore, \(\mathbb{E}[X^2] = \mu^2 + \sigma^2\)
Derivation of Cross-Entropy
The cross-entropy \(H(p_1, p_2)\) between two Gaussian distributions \(p_1(x) = \mathcal{N}(x | \mu_1, \sigma_1^2)\) and \(p_2(x) = \mathcal{N}(x | \mu_2, \sigma_2^2)\) is defined as:
\[ H(p*1, p_2) = -\int*{-\infty}^{\infty} p*1(x) \log p_2(x) dx = -\mathbb{E}*{p_1}[\log p_2(x)] \]First, let’s expand \(\log p_2(x)\):
\[ \log p_2(x) = \log \left(\frac{1}{\sqrt{2\pi\sigma_2^2}} \exp\lbrace-\frac{(x-\mu_2)^2}{2\sigma_2^2}\rbrace\right) \]\[ = -\frac{1}{2} \log(2\pi\sigma_2^2) - \frac{(x-\mu_2)^2}{2\sigma_2^2} \]Now, substitute this into the expectation formula for cross-entropy:
\[ H(p*1, p_2) = -\mathbb{E}*{p*1}\left[-\frac{1}{2} \log(2\pi\sigma_2^2) - \frac{(x-\mu_2)^2}{2\sigma_2^2}\right] \]\[ = \frac{1}{2} \log(2\pi\sigma_2^2) + \frac{1}{2\sigma_2^2} \mathbb{E}*{p_1}[(x-\mu_2)^2] \]Next, expand \(\mathbb{E}_{p_1}[(x-\mu_2)^2]\):
\[ \mathbb{E}_{p_1}[(x-\mu_2)^2] = \mathbb{E}_{p*1}[x^2 - 2x\mu_2 + \mu_2^2] \]\[ = \mathbb{E}*{p*1}[x^2] - 2\mu_2 \mathbb{E}*{p_1}[x] + \mu_2^2 \]Since \(p_1(x)\) is a Gaussian distribution with mean \(\mu_1\) and variance \(\sigma_1^2\), we substitute \(\mathbb{E}_{p_1}[x] = \mu_1\) and \(\mathbb{E}_{p_1}[x^2] = \mu_1^2 + \sigma_1^2\):
\[ \mathbb{E}\_{p_1}[(x-\mu_2)^2] = (\mu_1^2 + \sigma_1^2) - 2\mu_2 \mu_1 + \mu_2^2 \]\[ = (\mu_1 - \mu_2)^2 + \sigma_1^2 \]Finally, substitute this back into the cross-entropy equation:
\[ H(p_1, p_2) = \frac{1}{2} \log(2\pi\sigma_2^2) + \frac{(\mu_1 - \mu_2)^2 + \sigma_1^2}{2\sigma_2^2} \]This is the closed-form expression for the cross-entropy between two Gaussian distributions.