You're likely familiar with linear regression, which models a relationship using the simple equation $y = mx + b$. This concept is the direct ancestor of a single 'neuron' in a neural network. Instead of a single input $x$ and slope $m$, a neuron takes multiple inputs ($x_1, x_2, ..., x_n$), each with its own importance, or 'weight' ($w_1, w_2, ..., w_n$).

These weighted inputs are summed together, and a 'bias' term ($b$) is added. The bias is a learnable constant that shifts the output, much like the y-intercept in our simple line equation. This entire operation is just a weighted sum: $z = (w_1x_1 + w_2x_2 + ... + w_nx_n) + b$. This is the first step a neuron takes to process information.

This weighted sum, $z$, is a linear transformation of the inputs. If this were all a neuron did, a network of these neurons would just be a complex linear model. It could only learn linear relationships, like fitting a straight line to data points. To model the complex, non-linear patterns found in the real world—like identifying a cat in a photo or understanding language—we need to introduce a non-linear step.

After calculating the weighted sum $z$, the neuron passes this value through an activation function. This function determines the final output of the neuron, effectively deciding how much of the signal to pass on to the next layer. It's the key component that introduces non-linearity, allowing networks to learn intricate decision boundaries beyond simple straight lines.

There are many activation functions, but three are foundational: Sigmoid, Tanh, and ReLU.

• Sigmoid: This function squashes any input value into a range between 0 and 1. This makes it useful for the output layer of a binary classification problem, where the output can be interpreted as a probability.

• Tanh (Hyperbolic Tangent): Tanh is similar to sigmoid but squashes values into a range between -1 and 1. Its output is zero-centered, which often helps models learn more efficiently than sigmoid in hidden layers.

• ReLU (Rectified Linear Unit): ReLU is the most popular activation function in deep learning. Its formula is simple: $f(x) = max(0, x)$. It's computationally efficient and helps mitigate the vanishing gradient problem that can plague Sigmoid and Tanh. However, it can also lead to 'dead neurons' if a neuron's input is consistently negative.

The entire process of taking input data, passing it through layers of neurons, and producing a final output is called the forward pass. It's a one-way trip from the input layer to the output layer.

Let's walk through it for a single neuron.

1. Input: The neuron receives its inputs. Let's say we have two inputs, $x_1 = 3$ and $x_2 = -2$.
2. Weighted Sum: The neuron has internal weights and a bias it has learned. Suppose $w_1 = 0.5$, $w_2 = -1.0$, and the bias $b = 1$.
3. Calculation: It computes the weighted sum: $z = (x_1w_1) + (x_2w_2) + b = (3 \times 0.5) + (-2 \times -1.0) + 1 = 1.5 + 2 + 1 = 4.5$.
4. Activation: The result $z=4.5$ is passed through an activation function, say ReLU. The output is $f(4.5) = max(0, 4.5) = 4.5$. This value is then passed to the next layer.

In a deep neural network, this process is chained together. The output of one layer of neurons becomes the input for the next layer. Each layer transforms the data, allowing the network to learn increasingly complex features and representations, building from simple lines and edges in early layers to complete objects or concepts in later ones. This cascade of simple calculations, powered by non-linear activations, is how a neural network makes its prediction.

These building blocks are the foundation upon which all modern neural networks are built. Understanding them is key to demystifying how AI learns.