Neural Processing Units, or NPUs, are custom-built circuits designed for one primary job: running artificial intelligence models efficiently. Unlike a general-purpose Central Processing Unit (CPU) that has to be ready for anything, an NPU is a specialist. Its internal architecture is stripped down and optimized for the specific type of math that powers deep learning.

The heart of an NPU is its collection of tensor accelerators, often called MAC (Multiply-Accumulate) arrays. Neural networks rely heavily on multiplying large matrices of numbers together over and over again. A CPU handles these one by one, which is slow. A GPU does many at once, which is faster. But an NPU has dedicated hardware that does nothing but this one operation, making it incredibly fast and power-efficient.

Think of it like folding letters. A CPU is one person carefully folding each letter. A GPU is a large group of people folding letters at the same time. An NPU is a machine purpose-built to fold thousands of letters per minute with minimal energy. This specialized hardware is the NPU's main advantage.

To keep these speedy accelerator cores busy, NPUs use a sophisticated memory hierarchy. They have small, extremely fast on-chip memory (often SRAM) right next to the processing cores. This local memory stores the model's weights and the data being processed, so the chip doesn't have to constantly fetch information from slower system RAM. Minimizing data movement is key to the NPU’s efficiency, as moving data costs both time and energy.

Finally, NPUs employ a dataflow architecture. This isn't just about having the right parts; it's about how data moves between them. The architecture is engineered to ensure a constant, predictable flow of data streams through the tensor cores, maximizing their usage and preventing stalls. This is a fundamental departure from the more flexible, but less efficient, data handling in CPUs and GPUs.

CPUs, GPUs, and NPUs are all processors, but they are designed with very different goals in mind. A CPU is the brain of a computer, built for flexibility. It has a few very powerful cores that excel at handling complex, sequential tasks—like running your operating system or a web browser.

A GPU, originally designed for graphics, is a master of parallelism. It has thousands of simpler cores that can perform the same operation on many different pieces of data simultaneously. This makes it great for rendering video game worlds and, as it turns out, for training large AI models.

An NPU is the ultimate specialist. It sacrifices the general-purpose flexibility of a CPU and the broad parallelism of a GPU to do one thing exceptionally well: execute trained AI models (a process called inference). It's designed for maximum performance per watt on neural network calculations.

Several major chipmakers have developed their own NPU architectures, each with a slightly different focus.

Qualcomm's Hexagon Processor, found in their Snapdragon chips, is a leader in the mobile space. It's a key part of what makes modern smartphone features like computational photography, real-time translation, and voice assistants work quickly without draining the battery.

Intel has integrated NPUs into its latest laptop processors, like the Core Ultra series. Their "AI Boost" NPU is designed to offload sustained AI workloads, such as background blurring in video calls or running generative AI tools locally, from the CPU and GPU to save power and improve system responsiveness.

AMD leverages technology from its acquisition of Xilinx to power its "Ryzen AI" engine. This NPU is also aimed at laptops and PCs, focusing on bringing AI-enhanced features to everyday applications and improving overall system efficiency.

How do you measure an NPU's performance? The most common metric is TOPS, or Tera Operations Per Second. This tells you how many trillion calculations the chip can theoretically perform each second. A higher TOPS number generally means a more powerful NPU.

However, raw TOPS isn't the whole story. Efficiency is just as important, especially in battery-powered devices. That's why TOPS per Watt is a critical metric. It measures the computational performance relative to the power consumed. An NPU with a high TOPS/Watt rating can deliver strong AI performance without quickly draining your device's battery.

Another key factor is latency, which is the time it takes to complete a single inference task. For real-time applications like voice recognition or augmented reality, low latency is crucial for a smooth user experience. The NPU's architecture, especially its on-chip memory and dataflow, plays a huge role in minimizing latency.

Time to check what you've learned about the architecture of these specialized processors.

NPUs represent a major shift in processor design, moving from general-purpose computing toward specialized hardware that can handle the unique demands of artificial intelligence.