Beyond residual errors: measuring difference in probability space

Mean squared error is often used for evaluating how model predictions fit observed data, but in probability distributions, observations don't have to be treated as points in Euclidean space. The interactive plots below highlight several divergence measures that express disagreement between distributions. These measures offer a statistically grounded notion of difference that avoids the arbitrary geometry imposed by common residual-based metrics.

For example, the mean square error (MSE) is the maximum likelihood estimator if the errors are normally distributed, but in practice it is often the case that the errors are not normally distributed. In this case, the MSE is no longer the maximum likelihood estimator.

In this explorer, the reference distribution is fixed. Change the mean and standard deviation of the comparison distribution. The plots show how Kullback-Leibler divergence, total variation distance, Earth mover's distance, and Jensen-Shannon divergence respond.

Notes

1. $P$ is the normal distribution with location 0 and scale 1 in all cases.
2. $M$ in the JSD is the mean of $P$ and $Q$.
3. The Jensen-Shannon Divergence is a lower bound on the TVD.

Kullback-Leibler divergence (KLD)

$$D_{KL}(P||Q) = \sum_i P(i) \log_2 \frac{P(i)}{Q(i)}$$

The Kullback-Leibler divergence is the expectation of the log-likelihood ratio between two distributions $P$ and $Q$. It is a measure of how one probability distribution diverges from another, and in the information-theoretic sense, it quantifies the expected number of extra bits required to code samples from distribution $P$ when using a code optimized for distribution $Q$. The KL divergence is asymmetric, meaning that $D_{KL}(P||Q) \neq D_{KL}(Q||P)$. It is 0 if and only if the two distributions are identical, it is undefined if $Q(i) = 0$ for any $i$ where $P(i) > 0$, and it is unbounded.

The example above compares two Gaussians, and for this specific case there is a closed-form solution for the KL divergence:

$$D_{KL}(P||Q) = \log \frac{\sigma_Q}{\sigma_P} + \frac{\sigma_P^2 + (\mu_P - \mu_Q)^2}{2\sigma_Q^2} - \frac{1}{2}$$

Total Variation Distance (TVD)

$$D_{TV}(P,Q) = \frac{1}{2} \sum_i |P(i) - Q(i)|$$

The total variation distance expresses the aggregated absolute difference between the probabilities assigned by the two distributions, and is expressed in the range $[0,1]$. A value of 0 means that the two distributions are identical, while a value of 1 means that the two distributions are completely disjoint.

Earth Mover's Distance (EMD)

$$D_{EMD}(P,Q) = \sum_i |CDF_P(i) - CDF_Q(i)|$$

The earth mover's distance expresses the difference between two probability distributions as the minimum amount of "work" required to transform one distribution into the other, where "work" is defined as the amount of probability mass moved multiplied by the distance it is moved.

Jensen-Shannon Distance (JSD)

$$D_{JS}(P,Q) = \frac{1}{2} (D_{KL}(P||M) + D_{KL}(Q||M))$$

The Jensen-Shannon distance measures how well a single averaged distribution can stand in for both P and Q; it is the average information loss when P and Q are each replaced by their average density mixture.

Interpretation

No single divergence is best for every problem. The useful measure depends on which differences matter to a specific application, or which mathematical properties the analysis requires.