Supervised learning had an answer key; unsupervised learning had unlabeled structure. Reinforcement learning has neither. An agent is dropped into an environment, takes actions, and receives rewards, and from that feedback alone it must figure out how to behave. No one demonstrates the right move; the agent learns from consequences.
An agent acts in an environment to maximize reward: Q-learning discovers a policy and a value function from pure trial and error, and every RL system must balance exploring new actions against exploiting the best one it knows.
A Third Way to Learn
RL reframes learning as a loop. At each step the agent observes the current state, chooses an action, and the environment replies with a reward and the next state. Repeat. The agent's job is to pick actions that maximize reward over the long run, not just the next step.
This tiny loop is astonishingly general. A game-playing agent sees the board (state), makes a move (action), and wins or loses (reward). A robot senses its joints, sends motor torques, and is rewarded for walking. A recommendation engine shows an item and is rewarded by a click. Same skeleton every time, which is why one set of ideas covers them all.
States, Actions, Rewards & the Goal
The agent does not maximize the next reward, it maximizes the return: the total reward collected from now until the episode ends. Because a reward now is usually worth more than the same reward later, future rewards are shrunk by a discount factor gamma (say 0.95) per step. A high gamma makes the agent far-sighted, willing to accept small costs now (each step in our maze costs -0.1) to reach a big payoff later (the goal, +10).
Two objects capture what the agent knows. The policy is its behavior, which action to take in each state. The value function grades states: how much return can I expect starting here, if I act well? Learning good values lets the agent compare actions ("this move leads somewhere valuable"), and the best policy simply picks the highest-value action in every state. Getting these from experience is the job of Q-learning.
Q-learning & the Policy It Finds
Q-learning keeps a table of action-values Q(state, action): an estimate of the return from taking that action, then behaving well afterward. Every experience nudges the table toward reality with a single rule, move Q(s,a) a little toward reward + gamma × (best Q of the next state). Run enough episodes and the estimates converge, no model of the world required.
In our 5×5 GridWorld, a random agent averages a return of about -10 (it blunders into the trap). After a few hundred episodes of Q-learning the same agent scores about +9, and the table it built encodes a clean policy: in every cell, take the highest-value action.
The arrows form a flow that runs along the top row and down the far column, threading past the trap and walls to the goal, a route nobody programmed. The matching value function (highest near the goal, lowest by the trap) and this policy are two views of the same learned knowledge.
Real-World Example: Exploration vs Exploitation
Every RL agent faces one dilemma: should it exploit the best option it has found, or explore others that might be better? The cleanest place to see it is the multi-armed bandit, and a perfect real-world case is choosing which ad to show.
Six ad variants (ad_id, headline), each with a
hidden true click-through rate. The agent never sees the rate, it only observes clicks (reward 1
or 0) and must learn which ad wins. AD2 is best at 11.2%.
Turn each ad into a bandit "arm" and let different strategies choose who sees what:
| Strategy | How it explores | Result |
|---|---|---|
| Pure greedy | never, always shows the current best | can lock onto a loser forever |
| Epsilon-greedy | random 10% of the time | finds AD2, sent it ~82% of traffic |
| UCB | favors options it is uncertain about | finds AD2 fast, explores smartly |
Pure greedy is a cautionary tale: commit to whatever looked best after a few noisy clicks and you may never gather the evidence to know better. A little exploration, random (epsilon-greedy) or uncertainty-driven (UCB), pays a small short-term cost to find the truly best ad, then exploits it. This same trade-off drives A/B testing, recommendation, and clinical-trial design.
Reinforcement Learning in Machine Learning & AI
RL powers some of the most striking results in modern AI, precisely where decisions unfold over time.
| Idea | Where it is used |
|---|---|
| Q-learning / value methods | game AI, navigation, inventory and pricing control |
| Bandits (explore/exploit) | ad and content selection, A/B testing, adaptive trials |
| Policy methods | robotics, continuous control, self-driving simulation |
| Deep RL | Atari, Go and chess (AlphaGo/AlphaZero), data-center energy |
| RLHF | aligning large language models to human preferences |
Reinforcement learning is how AI systems learn to act, not just predict. It powered the superhuman game players (AlphaGo, AlphaZero) that first showed machines could out-plan people, and it drives robotics and control, where the goal is a sequence of good decisions rather than a single label. Most visibly today, RLHF, reinforcement learning from human feedback, is a key step in aligning large language models: the model's helpful, harmless responses are shaped by a reward signal learned from human preferences. The core ideas from this chapter, reward, policy, value, and the exploration-exploitation balance, are exactly what the next chapter scales up with neural networks.
Train an agent in Python
The companion notebook builds a GridWorld from scratch, trains an agent with the Q-learning update, visualizes the learned policy and value function, and runs a multi-armed bandit on the ad data to show why exploration pays, all in pure numpy, each cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, matplotlib, and openpyxl.
🎓 Key Takeaways
- ✓RL is the third paradigm: an agent learns from rewards by acting, no labels and no fixed dataset.
- ✓Agent, environment, state, action, reward form a loop; the goal is to maximize cumulative discounted return.
- ✓The policy says what to do in each state; the value function says how good each state is.
- ✓Q-learning learns action-values from experience with one update rule; the greedy policy falls out of it.
- ✓Exploration vs exploitation is the central dilemma, you must try new actions to learn what to exploit.
Practice Challenges
Five short challenges. Try them in numpy before checking the solutions.
Random baseline
Run a random agent in the GridWorld for 200 episodes and report its average return.
rng.integers(4) each step; sum the rewards.Train with Q-learning
Train a Q-table and report the trained agent's average return.
Q[s,a] += alpha*(r + gamma*max(Q[s2]) - Q[s,a]).Read the policy
Extract the greedy policy and name the best action at the start cell.
Q.argmax(axis=2) gives the action per state.Discount factor
Compare a myopic agent (gamma=0) with a far-sighted one (gamma=0.95).
Exploration pays
On the ad bandit, compare pure-greedy vs epsilon-greedy traffic sent to the best ad.
N[best] / T for each strategy.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on reinforcement learning. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Q-learning used a table, one entry per state, which cannot scale to a video game screen or a robot's cameras. Deep RL & Applications replaces the table with neural networks (DQN, policy gradients, actor-critic), and shows how RLHF uses these ideas to align large language models.