In reinforcement learning, we've seen two main ways to update our value estimates. Monte Carlo (MC) methods wait until the end of an episode to update every state visited, based on the final outcome. Temporal Difference (TD) learning, specifically TD(0), updates its estimate for a state after just one step, using the reward from that step and the estimated value of the next state. This is called bootstrapping.

TD(0) is efficient but myopic. It only looks one step ahead. MC considers the entire episode's return, giving it a more complete picture, but it can be slow and have high variance. What if there was a way to combine the best of both worlds?

This is where TD(λ) comes in. It's a sophisticated algorithm that elegantly bridges the gap between the one-step updates of TD(0) and the full-episode updates of MC. It allows an agent to look multiple steps into the future, creating a more flexible and often more efficient learning process.

The core mechanism behind TD(λ) is the eligibility trace. Think of it as a short-term memory for the agent. It keeps track of which states have been visited recently, marking them as 'eligible' for learning updates.

Imagine you're navigating a maze and find a piece of cheese. The move you just made was clearly important, but what about the move before that? And the one before that? An eligibility trace assigns credit (or blame) to the whole sequence of recent states that led to an outcome. The most recent states get the most credit, while the credit for states visited longer ago fades away.

This 'memory' is stored for each state $s$ in a value called $E(s)$. At every time step $t$, the eligibility traces for all states decay slightly and the trace for the currently visited state is increased.

When a state is visited, its eligibility trace gets a boost of 1. At every step, all traces are multiplied by a decay factor, $\gamma\lambda$. This ensures that the influence of past states gradually diminishes over time.

With eligibility traces in place, the TD(λ) update rule becomes quite powerful. First, just like in TD(0), we calculate the TD error at each step. This error measures the difference between our prediction and what actually happened one step later.

Instead of using this error to update only the value of the current state, $V(S_t)$, TD(λ) uses it to update the value of every single state in proportion to its eligibility trace. This is how credit is assigned back through time.

States that were visited recently (high eligibility) get a large share of the update from the current TD error. States visited long ago, or not at all, have traces near zero and are barely affected.

The parameter λ (lambda) controls the rate at which the eligibility traces decay, ranging from 0 to 1. It acts as a slider between TD(0) and Monte Carlo methods.

When λ = 0, the eligibility trace for any state other than the current one becomes zero immediately. The update rule simplifies to updating only the current state, making it identical to TD(0).

When λ = 1, the traces decay very slowly (only by the discount factor $\gamma$). This means credit from a reward is passed back almost fully to all prior states in the episode. The learning updates become very similar to the updates made by a Monte Carlo method.

By choosing a value for λ between 0 and 1, we can create a hybrid algorithm. An intermediate value, like λ = 0.9, often performs best in practice. It allows the agent to learn from multiple future steps without having to wait until the end of the episode, striking a balance between the bias of TD(0) and the variance of MC.

Like many other reinforcement learning algorithms, TD(λ) is guaranteed to converge to the optimal value function, provided the learning rate α decreases appropriately over time and all states are explored sufficiently.

TD(λ) provides a powerful framework for reinforcement learning, offering a way to adjust the depth of backups and balance the trade-off between bias and variance. Its use of eligibility traces makes learning more efficient by propagating information from rewards back to all the states that led to them.