Kullback-Leibler (KL) Divergence

What is Kullback-Leibler (KL) Divergence? KL Divergence  Explained

Kullback-Leibler Divergence (KL Divergence), also known as relative entropy, is a measure of the difference between two probability distributions. It is commonly used in information theory, statistics, and machine learning to quantify the dissimilarity or information loss when approximating one distribution with another.

Given two probability distributions P and Q, the KL Divergence between them is defined as:

KL(P || Q) = ∑ P(x) * log(P(x) / Q(x))

The KL Divergence is asymmetric, meaning KL(P || Q) is not the same as KL(Q || P). It measures how much additional information is needed to encode data samples from P using a code optimized for Q, compared to using a code optimized for P.

Key points about KL Divergence:

Non-negativity: KL Divergence is always non-negative, meaning it is zero if and only if the two distributions P and Q are identical.

Not a true distance metric: KL Divergence is not a true mathematical distance measure because it does not satisfy the triangle inequality property. In other words, the KL Divergence between two distributions is not the same as the KL Divergence between those distributions and a third distribution.

Application in machine learning: KL Divergence has various applications in machine learning. It is commonly used in generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to measure the difference between the model’s generated distribution and the true distribution. It is also used in information retrieval, natural language processing, and reinforcement learning.

Relationship to cross-entropy: KL Divergence is closely related to cross-entropy, another measure used to compare probability distributions. In fact, the KL Divergence between two distributions P and Q can be expressed as the sum of the cross-entropy between P and Q and the entropy of P:

KL(P || Q) = H(P, Q) – H(P)

Here, H(P, Q) represents the cross-entropy of P with respect to Q, and H(P) represents the entropy of P.

Sample-based estimation: In practice, when the true distributions are not known, the KL Divergence can be estimated using samples. This involves computing the empirical probabilities based on observed data and approximating the logarithm term using the samples.

KL Divergence provides a useful tool for quantifying the dissimilarity between probability distributions. It is widely used in various fields, particularly in information theory, statistics, and machine learning, to measure and compare the information content or divergence between two distributions.