Traditional neural networks are powerful, but they're also resource-hungry. They rely on 32-bit floating-point numbers, which are precise but require a lot of memory and computational power. Think of each number as a highly detailed instruction, with lots of nuance.

Binary Neural Networks (BNNs) take a radically different approach. Instead of using a wide range of numbers for their weights and activations, BNNs constrain them to just two values: +1 or -1. It's like swapping a detailed instruction manual for a simple on/off switch. This simplification is the key to their efficiency.

This binary constraint applies to two key components:

• Weights: These are the parameters the network learns during training. In a BNN, every weight is either +1 or -1.
• Activations: These are the signals passed between neurons. They are also binarized to +1 or -1.

By forcing everything into this binary state, the entire calculation process changes dramatically.

Why go through the trouble of simplifying a network so much? The advantages are significant, especially for deploying AI on devices with limited resources, like smartphones or smart sensors.

Second, and perhaps more importantly, computations become incredibly fast and energy-efficient. The most common operation in a neural network is multiplication. In a BNN, multiplying by +1 or -1 is equivalent to a simple, lightning-fast bitwise operation called XNOR. This replaces computationally expensive arithmetic with basic logic.

If BNNs are so efficient, why aren't they used everywhere? The main hurdle is training. Neural networks learn using a process called backpropagation, which relies on calculating gradients. A gradient is essentially the slope of the loss function, telling the network which direction to adjust its weights to get better.

The problem is, the function that turns a real number into a binary value (a step function) has a gradient that is zero almost everywhere. Where it's not zero, it's infinite. A zero gradient provides no information for learning; it's like trying to find your way down a mountain by only looking at perfectly flat ground. You have no idea which way is down.

To solve this, researchers developed clever workarounds. The most common is the Straight-Through Estimator (STE). During the forward pass of the network, weights are binarized as usual. But during the backward pass (where learning happens), the network pretends the binarization function was a different, smoother function—one that has a useful gradient.

This technique essentially "passes through" a helpful gradient, allowing the network to learn how to adjust its underlying, real-valued weights. These real-valued weights are then binarized again for the next forward pass. It's a simple but effective trick that makes training BNNs possible.