The engine powering nearly all modern LLMs is the Transformer architecture. Introduced in 2017, its key innovation is the attention mechanism. This allows the model to weigh the importance of different words in the input text when processing and generating a response. It doesn't just read words in sequence; it understands the relationships between them, no matter how far apart they are.

Think of a chef preparing a complex dish. They don't just add ingredients in a fixed order. They constantly pay attention to how different elements interact—how the acidity of the lemon balances the richness of the butter, or how the spice level of the chilli affects the overall flavour. The attention mechanism works similarly, constantly assessing which words are most relevant to understanding the context of every other word.

At its core, the attention mechanism works with three vectors for each input token (a word or part of a word): a Query, a Key, and a Value.

1. Query: Represents the current word's request for information.
2. Key: Represents the label or identifier of other words in the sequence.
3. Value: Represents the actual content of those other words.

The model calculates a score by matching the Query of the current word with the Keys of all other words. These scores are then converted into weights (using a softmax function), which determine how much attention to pay to each word's Value. The final output for the current word is a weighted sum of all the Values in the sequence.

Not all Transformers are built the same. The two dominant architectures you'll encounter are Dense and Mixture of Experts (MoE).

A dense model, like GPT-3.5 or Llama 3 70B, uses its entire set of parameters for every single calculation. When it processes a token, every neuron in the network is activated. This is thorough and powerful but computationally expensive. It’s like having every specialist in a hospital consult on every single patient, regardless of their ailment.

An MoE model, like Mixtral 8x7B or GPT-4, takes a different approach. It consists of multiple smaller "expert" networks and a router or gating network. For each token, the router decides which one or two experts are best suited to handle it and only activates those. The other experts remain dormant.

This is like a hospital's triage system. A patient with a broken arm is sent to an orthopaedic specialist, not a cardiologist. This specialisation makes MoE models much faster and more efficient to run for a given parameter count, as only a fraction of the model is used for any single token. While an MoE model might have a huge total parameter count (e.g., 1.76 trillion for GPT-4), the number of active parameters used for inference is much smaller.

When choosing an LLM for an automation task, you need to look beyond marketing and evaluate a few key metrics.

Context Window: This is the amount of text (input + output) the model can remember in a single conversation, measured in tokens. A small context window (e.g., 4,000 tokens) means the model will forget the beginning of a long document or conversation. A large context window (e.g., 200,000 tokens) allows it to process entire books or extensive codebases at once. Managing this is critical for tasks that require understanding long histories or documents.

Parameter Count: This is often used as a rough proxy for a model's capability. A model with 70 billion parameters (like Llama 3 70B) generally has more capacity for knowledge and nuance than a 7 billion parameter model. However, architecture (Dense vs. MoE) and training data quality are just as important. A well-trained 13B model can outperform a poorly trained 30B model.

Scaling Laws: Research has shown a predictable relationship between model size, dataset size, and performance. In essence, as you increase the compute power, model parameters, and training data, the model's performance improves logarithmically. This is why companies are in a race to build bigger and bigger models—the returns are predictable, though increasingly expensive.

Finally, the choice between a proprietary and an open-weight model has significant implications for enterprise use.

Proprietary models (e.g., OpenAI's GPT series, Anthropic's Claude) are accessed via an API. You send them data, and they send back a response. You don't have access to the model's inner workings or weights. The main advantages are ease of use and access to state-of-the-art performance without needing to manage infrastructure. The downsides are cost, data privacy concerns (as your data is sent to a third party), and latency.

Open-weight models (e.g., Meta's Llama series, Mistral's models) release their model weights to the public. This allows anyone to download, modify, and run the model on their own hardware—from a powerful laptop to a private cloud server. This provides maximum control over data privacy and can significantly reduce latency. However, it requires technical expertise to set up and maintain the necessary infrastructure.

For many professional automation tasks, especially those involving sensitive customer data, open-weight models are becoming the standard. They offer a better balance of performance, cost, and security.

Let's test your understanding of these core architectural concepts.

Understanding these architectural trade-offs is the first step in selecting the right tool for an automation project. The choice of engine—from its core design to its deployment model—directly impacts the system's cost, speed, and security.