For a long time, models designed to understand language tried to mimic how we read: one word at a time. Recurrent Neural Networks (RNNs) were built on this principle. An RNN processes the first word of a sentence, keeps a summary of it in its hidden state, and then moves to the next word, carrying that memory forward. Each new word updates the hidden state, creating a running summary of everything seen so far.

This sequential process, however, creates a fundamental bottleneck. The network can't process the tenth word until it has finished with the first nine. This one-at-a-time approach means the computation time scales linearly with the length of the text. In computational terms, this is described as having a path length of O(N) for sequential operations, where N is the length of the sequence. For very long documents, this becomes incredibly slow.

The sequential bottleneck isn't the only problem. As an RNN processes a long sentence, it struggles to maintain context. Information from the beginning of the sentence can get diluted or lost by the time the network reaches the end. This is a classic issue known as the vanishing gradient problems.

Think of the gradient as a correction signal telling the network how to adjust its understanding. If that signal fades to a whisper, the network can't learn relationships between words that are far apart. The meaning of a word at the start of a paragraph might be lost by the final sentence.

Long Short-Term Memory (LSTM) networks were a brilliant solution. LSTMs are a special type of RNN with a more complex memory cell. They use a system of 'gates'—an input gate, an output gate, and a forget gate—to regulate the flow of information. These gates selectively decide what information to store, what to discard, and what to use for the output. This helps preserve important context over longer sequences, mitigating the vanishing gradient issue.

Even with this clever gating mechanism, LSTMs still had the same fundamental limitation as simpler RNNs: they processed data one step at a time. They were better at remembering, but just as slow.

The breakthrough came from rethinking the entire process. Instead of processing a sentence sequentially, what if a model could look at all the words at once and decide which ones were most relevant to understanding any other word in the sentence? This is the core idea of the attention mechanism.

Early versions of attention, developed by researchers like Dzmitry Bahdanau and Minh-Thang Luong, were still used with RNNs. They allowed the decoder in a machine translation model to 'look back' at all the words in the source sentence, focusing its attention on the most relevant ones at each step of generating the translation. This was a huge improvement, but the underlying sequential processing of the RNN remained.

The real revolution was to discard the recurrence entirely. The 2017 paper "Attention Is All You Need" introduced the Transformer, a model built solely on attention mechanisms. It could process all words in a sentence in parallel. There was no more waiting.

This shift changed the computational complexity. Because every word can be processed at the same time, the number of sequential operations is no longer tied to the sequence length N. Instead, it's a constant, O(1). This massive parallelization is what makes it feasible to train the enormous models we see today.

By moving from recurrence to attention, we traded a slow, step-by-step memory for a fast, parallelised system of contextual relationships. This was the key that unlocked the scaling and power of modern Large Language Models.