Introduction to Perplexity

How Confused Is Your Model?

Imagine you're playing a game where you have to guess the next word in a sentence. If the sentence is "The cat sat on the…", you'd probably guess "mat" or "floor." You're not very confused. But if the sentence is "After the strange symphony, the artist revealed a…", you'd be much less certain. You are more perplexed.

In Natural Language Processing (NLP), we use a similar idea to measure how well a language model predicts text. This measurement is called perplexity.

Perplexity

noun

A measurement of how well a probability model predicts a sample. In NLP, it quantifies a language model's uncertainty when predicting the next word in a sequence.

Think of perplexity as a measure of surprise. A model with low perplexity is less surprised by the text it sees, meaning its predictions are more accurate. A high perplexity means the model is very surprised, indicating its predictions are often wrong.

Lower perplexity is better. It means the model is more confident and accurate in its predictions.

The Math Behind the Metric

Perplexity is calculated based on the probability a model assigns to a sample of text. Let's say we have a test sentence $W$ made up of $N$ words: $w_1, w_2, \dots, w_N$ . The perplexity of this sentence is the N-th root of the inverse of the probability of the sentence.

PPL(W) = \sqrt[N]{\frac{1}{P(w_1, w_2, \dots, w_N)}}

This can also be written as:

PPL(W) = P(w_1, w_2, \dots, w_N)^{-\frac{1}{N}}

Here, $P(w_1, w_2, \dots, w_N)$ is the probability the language model assigns to that entire sequence of words. A model that assigns a higher probability to the test sentence will have a lower perplexity.

Interpreting the Score

So, what does a perplexity score actually mean in practice? You can think of the perplexity value as the number of choices the model is effectively considering for the next word.

For example, if a model has a perplexity of 100, it means that whenever it tries to guess the next word, it's as confused as if it had to choose between 100 different words with equal probability. A model with a perplexity of 20 is more certain, as if it's only choosing between 20 words.

A perplexity of 100 on a language task means the model is as uncertain as if it were choosing the next word uniformly from 100 possibilities.

This is why perplexity is so important for evaluating language models. It provides a single, intuitive number that tells us how well the model understands the language in a given dataset. When developers train a new model, they often use perplexity as a key benchmark to track its improvement. As they adjust the model's architecture and training data, they aim to drive the perplexity score down.

Quiz Questions 1/5

In the context of Natural Language Processing, what does perplexity primarily measure?

Quiz Questions 2/5

A language model with a low perplexity score is generally considered...

Understanding perplexity gives you a solid foundation for judging how language models perform.

Demystifying Perplexity in NLP