This version is intentionally simplified for clarity. The book provides a more rigorous treatment, including probabilistic analysis, empirical findings, and concrete architectural constraints for preventing error propagation in real-world AI systems.

How AI Hallucinations Snowball (and Why It’s Not Random)

When a language model writes text, it doesn’t “think” about meaning. It predicts the next word based on probability. At every step, the model creates a ranked list of possible next words. Some are very likely. Others are possible but strange.

Think of this list as a curve:

• The head contains safe, obvious choices

• The tail contains rare, odd choices

Most of the time, the model picks from the head. But sometimes it doesn’t.

A Simple Example

Imagine the model is finishing this sentence:

The cat sat on the ___.

The model might assign probabilities like this:

• Mat (75%) — the obvious choice

• Floor (15%) — still reasonable

• Map (0.01%) — technically possible, but odd

All three words fit the grammar, but only one fits common sense. Most of the time, the model picks the obvious choice—mat. But once in a while, we get unlucky and the model selects map.

The moment the model outputs map, it rewrites its own context. Instead of predicting words that usually follow mat—floor, rug, doorway—it begins predicting words associated with map, such as route, region, or location.

Instead of completing the sentence “The cat sat on the mat,” the result can drift into something like “The cat sat on the map, tracing a route across the galaxy.”

The model isn’t confused. It is doing exactly what it was designed to do: stay consistent with its previous output.

This Is Error Propagation

This effect is known as error propagation. A more descriptive name is hallucination snowballing.

Here’s what happens:

1. The model makes a small mistake

2. That mistake becomes part of the input

3. Future predictions bend around it

4. The output grows more detailed—and more wrong

Research shows that models often prefer to stay consistent with their own earlier output, even when it conflicts with what they “know.” Once an error appears early, it can shape everything that follows.

The result is text that sounds confident, clear, and polished, but is detached from reality.

Why This Matters in Real Systems

In a short sentence, the damage is small.

In a long workflow (like a report, a marketing campaign, or a multi-step analysis) the damage compounds. An early mistake can quietly guide hundreds of later decisions.

By the end, the system hasn’t just hallucinated.
It has built a whole story on top of the error.

Conclusion

When generation happens in one long, unbroken stream, mistakes have nowhere to go but forward. If nothing forces the system to pause, reset, or validate its own outputs, even a minor error can influence everything that follows.

In the next post, we’ll move from diagnosis to design. We’ll look at practical mitigation strategies and how architectural boundaries, validation steps, and controlled workflows can stop errors from spreading and turn probabilistic models into systems you can actually trust.