So far, you've seen methods that learn a value function, like Q-learning, and then use that function to pick the best actions. These are called value-based methods. Now, let's switch gears to another approach: directly learning the policy itself.

Policy-based methods, like those using policy gradients, learn a function, $\pi_\theta(a|s)$, that maps a state directly to an action (or a probability distribution over actions). Instead of estimating how good a state or state-action pair is, we directly optimize the parameters, $\theta$, of our policy to maximize the expected total reward.

The core idea is to adjust the policy's parameters in the direction that leads to better outcomes. We calculate how the expected reward changes as we change our policy parameters, which is the gradient. Then, we update our parameters in the direction of this gradient, a process similar to hill climbing, where the goal is to reach the peak of highest reward.

A common algorithm that uses this principle is called REINFORCE. It collects a trajectory by following the current policy, calculates the total reward, and then updates the policy parameters to increase the probability of actions taken in that trajectory, weighted by how good the total reward was. While simple, it can have high variance because the reward from an entire episode is used to update every action, which can be noisy.

Policy gradient methods can be slow to learn due to their high variance. On the other hand, value-based methods can be more stable but struggle with continuous action spaces. Actor-Critic methods combine the best of both worlds.

Imagine two components working together: an actor and a critic.

The Actor is the policy. It decides which action to take in a given state. Its goal is to learn the best policy, just like in policy gradient methods.

The Critic is a value function. It doesn't choose actions. Instead, it evaluates the actions taken by the actor, providing feedback. The critic learns a value function (like $V(s)$ or $Q(s, a)$) to estimate how good a state or state-action pair is.

Here’s how they interact: The actor takes an action. The critic observes the outcome and calculates the Temporal Difference (TD) error, which measures how much better or worse the outcome was than expected. This TD error is then used as the signal to update both the actor and the critic. The actor uses it to update its policy parameters, making good actions more likely, and the critic uses it to improve its own value estimates.

Standard Q-learning is powerful, but it relies on a Q-table to store the value for every state-action pair. This is fine for small, discrete environments. But what about environments with a huge number of states, like processing pixels from a screen?

Enter Deep Q-Networks (DQNs). The key innovation of DQN is to use a deep neural network to approximate the Q-value function, $Q(s, a)$. Instead of a table, we have a network that takes the state (like an image) as input and outputs the Q-value for each possible action.

Training a neural network with RL data can be unstable. The data is sequential and not independent, and the target value for the Q-function is constantly changing as the policy improves. DQNs introduced two key techniques to solve this instability:

1. Experience Replay: Instead of training on consecutive experiences as they happen, the agent stores its experiences—$(s, a, r, s')$ tuples—in a replay buffer. During training, it samples random mini-batches from this buffer. This breaks the correlation between consecutive samples, making the training process more stable and data-efficient.

2. Fixed Q-Targets: The network is updated by minimizing the difference between its current Q-value prediction and a target Q-value. If the same network is used to generate both, the target moves with every update, leading to instability. DQN uses a second, separate target network to generate the target Q-values. The weights of this target network are frozen for a period of time and only periodically updated with the weights from the main network. This provides a stable target for the main network to chase.

These advanced methods—policy gradients, actor-critic architectures, and DQNs—form the bedrock of many modern reinforcement learning applications, allowing agents to solve complex problems in high-dimensional spaces.