Early models for sequence-to-sequence (Seq2Seq) tasks, like machine translation, relied on Recurrent Neural Networks (RNNs) and their more capable cousins, LSTMs. The setup was logical: an encoder RNN would read the input sentence word by word, and a decoder RNN would generate the output sentence word by word.

The encoder's job was to understand the entire input sentence and condense its meaning into a single, fixed-size vector. This summary, often called the context vector or 'thought vector', was then handed off to the decoder. The decoder's only source of information about the input was this one vector. It had to generate a complete, accurate translation based entirely on that condensed summary.

This created a huge problem. For short sentences, it worked reasonably well. But for longer ones, the model struggled. Trying to cram the nuances of a 30-word sentence into a single, fixed-length vector is like trying to summarize a feature film in one short sentence. You're guaranteed to lose critical information.

This is the information bottleneck. It also worsened the infamous vanishing gradient problem. As the network tried to send error signals back through time to adjust its parameters, the signal would weaken with each step, becoming almost useless by the time it reached the early words in a long sequence. The model couldn't learn long-range dependencies, like how the first word of a sentence affects the tenth.

The solution was to get rid of the single context vector and instead allow the decoder to look back at the entire input sentence at every step of the generation process. This mechanism is called attention.

Instead of just receiving one summary vector, the decoder now gets access to the hidden state of the encoder for every single input word. At each step of generating an output word, the decoder does two things:

1. It calculates an attention score for each encoder hidden state. This score determines how relevant each input word is to the current output word being generated.
2. It creates a new, weighted context vector by taking a weighted sum of all the encoder hidden states. The weights are the attention scores it just calculated.

This new context vector is custom-built for each step. When translating the verb, the decoder can 'pay attention' to the source verb. When translating the noun, it can focus on the source noun. It's no longer forced to rely on a single, generic summary.

This direct connection between decoder and encoder states acts like a shortcut, mitigating the vanishing gradient problem. Error signals have a much shorter, more direct path to travel, making it easier for the model to learn long-range dependencies.

Two popular early approaches to calculating attention scores were Bahdanau and Luong attention. They differ subtly in how and when they compute the scores. Bahdanau attention is often called 'additive' attention, while Luong's is 'multiplicative'. The key idea is the same, but the mathematical implementation differs. Luong attention is generally simpler and more common in modern architectures.

Even with attention, these Seq2Seq models had a fundamental limitation: they were still sequential. The RNNs at their core had to process words one after another. The calculation for the fourth word depended on the result from the third, which depended on the second, and so on.

This sequential nature made them slow to train and impossible to scale to the massive datasets needed for true language mastery. You couldn't just throw more computers at the problem to speed it up, because each step had to wait for the previous one to finish.

This is the problem the Transformer architecture was designed to solve. It asked a radical question: what if we could get the benefits of attention without the sequential processing of RNNs? By getting rid of recurrence entirely and relying only on a more powerful form of attention, the Transformer opened the door to massive parallelization. This shift from sequential to parallel processing was the key that unlocked the incredible scaling we see in today's Large Language Models.

Understanding this evolution from the RNN bottleneck to attention, and finally to a purely attention-based parallel architecture, is key to grasping why models like GPT are possible.