Chat with Resume
10 min read · ML Fundamentals

KL Divergence
made visual

Kullback–Leibler divergence — no scary math. Just two distributions,
and how far apart they are.

01 — The Big Idea

What even is a distribution?

Before KL Loss, let's get comfortable with distributions. A probability distribution tells you how likely each outcome is. Think of it as a "belief" about what will happen.

Below are two distributions — P (the true one) and Q (our guess). Drag the sliders and watch them shift.

P peak position 35%
Q peak position 65%

When P and Q overlap perfectly, they're identical. When they drift apart — that gap is exactly what KL divergence measures.

02 — The Formula

What KL divergence actually measures

Kullback–Leibler divergence (KL) answers: "How much extra information do I waste if I use Q to describe a world that's actually P?"

KL(P ∥ Q) = Σ P(x) · log( P(x) / Q(x) )
Sum over all outcomes x. P is truth, Q is our approximation.

Three things to notice:

1. It's not symmetric — KL(P∥Q) ≠ KL(Q∥P). Direction matters.
2. It's always ≥ 0. Zero means the distributions are identical.
3. The log ratio captures surprise — when Q says something is rare but P says it's common, that's expensive.
03 — Live Calculation

Let's compute it ourselves

Hit Random to shuffle two full distributions over Rain, Sunny, and Cloudy. P is truth; Q is the forecast.

Total KL(P ∥ Q) P · log(P/Q) summed over all three outcomes
— nats
04 — Which Direction?

What if the loss is KL(P ∥ Q) or KL(Q ∥ P)?

Same two distributions — but the direction flips who weights the sum and which ratio you take. In training, P is frozen (truth / parent). Only Q, the student, gets updated.

Frozen P — truth / parent · Fixed reference · never moves during training
same P · two different losses on Q ↓
Trained Q — student model · Starts identical · diverges by loss function
KL(P ∥ Q) loss KL(Q ∥ P) loss
Step 0 / 48

Purple = P (frozen) · Green = Q (student updating each step)

Forward · KL(P ∥ Q)

Σ P(x) · log( P(x) / Q(x) )
  • P weights the sum · ratio P/Q
  • Q spreads to cover all modes — blurry but inclusive
  • Watch left panel: Q widens across both peaks

Reverse · KL(Q ∥ P)

Σ Q(x) · log( Q(x) / P(x) )
  • Q weights the sum · ratio Q/P
  • Q locks onto one mode — sharp, mode-seeking
  • Watch right panel: Q snaps to a single peak
05 — The Question Each KL Asks

Two different questions — RLHF picks one

Forward and reverse KL are not just swapped formulas. Each one asks a different question about frozen P and trained Q. In RLHF, we care far more about the reverse one.

Forward KL(P ∥ Q)

"If P is truth, where is Q missing mass that P cares about?"

Expectation under P · punishes Q for being too low where P is high · pushes Q to cover every mode

Reverse KL(Q ∥ P)

"Where Q actually puts mass, is it saying things P would not?"

Expectation under Q · punishes Q for being high where P is low · keeps the student on a leash

RLHF · PPO

Why reverse KL is what we use

  • P stays frozen — the SFT parent we trust. Q is trained with reward signals on its own outputs.
  • We only sample from Q during RL — reverse KL weights exactly those tokens, asking whether the student is drifting off-policy.
  • Forward KL would force Q to spread across all of P's modes — useful in VI, but too blurry for coherent language.
  • Reverse KL is the right guardrail: improve from reward, but stay close to the parent. Hence reward − β · KL(Q ∥ P).
06 — Token by Token

Reverse KL on every token the student writes

Ask "What is the capital of France?" — the student replies below. Click any word (or use ← →) to see how far Q drifted from frozen parent P at that step.

Student reply · click a token
—
Token —
KL(Q∥P) — nats

Where we are
What each model would say next
P · parent (frozen)
Q · student (trained)
Penalty on the word Q picked

In PPO, we sum KL(Q∥P) across every token the student actually writes.

Recap

The 30-second summary

KL divergence measures how different distribution Q is from distribution P.
Forward KL asks where Q misses P's mass. Reverse KL asks where Q oversteps P — the question RLHF actually needs.
RLHF / PPO uses reverse KL(Q∥P): reward improves Q, but −β·KL(Q∥P) keeps it near frozen parent P.
Reverse KL is used in self-distillation for sharper, mode-seeking behavior.