No history yet

Introduction to Reinforcement Learning

Learning from Experience

Imagine teaching a dog to fetch. You don't give it a textbook on aerodynamics. Instead, you throw a ball. If the dog brings it back, you give it a treat. If it gets distracted by a squirrel, it gets nothing. Over time, the dog learns that bringing the ball back leads to a reward.

This is the core idea behind reinforcement learning (RL). It's a way for a machine to learn by doing. An 'agent' learns to make decisions by performing actions in an 'environment' to achieve a goal.

In Reinforcement Learning (RL), an agent learns by interacting with its environment.

Unlike other types of machine learning where a model learns from labeled data, an RL agent learns from the consequences of its actions, much like humans and animals do. The agent isn't told which action is best; it must discover it through trial and error.

The Agent-Environment Loop

The learning process in RL happens in a continuous cycle called the agent-environment loop. It’s a constant back-and-forth conversation. Let's break down the key parts of this interaction.

Here's how the loop works:

  1. State (sts_t): The agent observes the current situation of the environment. A state is a snapshot of everything the agent needs to know. For a robot cleaning a room, the state might be its location and the location of any trash.
  2. Action (ata_t): Based on the state, the agent chooses an action. The robot might decide to move forward, turn left, or pick up trash.
  3. Reward (rt+1r_{t+1}): The environment gives the agent feedback in the form of a reward. It's a numerical value. The robot might get a +10 reward for picking up trash and a -1 for bumping into a wall. The environment also transitions to a new state (st+1s_{t+1}) as a result of the agent's action.

Observe state -> Take action -> Get reward -> Observe new state. This cycle repeats, and with each turn, the agent learns a little more about which actions lead to good outcomes.

Learning the Best Strategy

An agent's goal isn't just to get the biggest immediate reward. It wants to maximize its cumulative reward over the long run. Sometimes, a small immediate reward can lead to a much larger reward later. Think of a chess game: sacrificing a pawn (a small negative reward) might be necessary to checkmate the king later (a huge positive reward).

The agent's strategy for choosing actions is called its policy, often represented by the Greek letter pi (\\[pi\\]). A policy is essentially the agent's brain; it's a map that tells the agent what action to take in any given state.

policy

noun

In reinforcement learning, a strategy used by an agent to determine the next action based on the current state.

The entire goal of reinforcement learning is to find the best possible strategy, known as the optimal policy (\\[pi\\]^*). This is the policy that, if followed, will rack up the most reward over time. Finding this optimal policy is the central challenge. The agent refines its policy through thousands or millions of cycles in the agent-environment loop, slowly improving its decision-making until its actions consistently lead to the best long-term outcomes.

Quiz Questions 1/5

What is the primary way a reinforcement learning agent learns to make better decisions?

Quiz Questions 2/5

In the reinforcement learning framework, an agent's main objective is to maximize its __________ reward.

With these core concepts, you now understand the fundamental loop that drives all of reinforcement learning.