The last chapter's Q-learning kept a table with one row per state. That works for a 25-cell maze and collapses everywhere else: a chess position, a camera frame, and a robot's joint angles all have astronomically many states. Deep reinforcement learning replaces the table with a neural network that generalizes across states, and that single change unlocks the headline results of modern AI.
Swap the Q-table for a neural network: DQN (value-based), policy gradients (policy-based), and actor-critic (both) scale RL to huge problems, and RLHF, a reward model learned from human preferences plus PPO, is how those ideas align today's chatbots.
From Table to Network
A Q-table stores a separate number for every state-action pair. Double the resolution of a camera and the table explodes past the number of atoms in the universe, you can never fill it, let alone visit every entry. The escape is function approximation: instead of memorizing a value per state, train a neural network that reads the state and outputs the values, so nearby states share what they learn.
A Deep Q-Network (DQN) is exactly this: a net that maps a state to one Q-value per action. Training it naively is unstable, so DQN adds two fixes, experience replay (train on random past transitions to break correlations) and a target network (a slow-moving copy that provides a steady learning target). On our GridWorld the DQN reaches a return of 9.3, matching the tabular agent, but now it learns a function that would scale to a pixel screen. This is the algorithm that first played Atari from raw images.
Three Families: Value, Policy & Actor-Critic
Deep RL splits into three approaches, and our GridWorld shows two of them solving the same maze.
| Family | What it learns | Strength / cost |
|---|---|---|
| Value-based (DQN) | Q(s, a), then act greedily | sample-efficient; needs discrete actions |
| Policy-based (REINFORCE) | the policy (action probabilities) directly | handles continuous actions; higher variance |
| Actor-critic (A2C, PPO, SAC) | a policy (actor) + a value (critic) | low-variance, stable; the modern default |
A policy-gradient method skips values and pushes up the probability of actions that led to high reward. It also solves the maze (return 9.2) but learns more noisily, policy gradients are high-variance. The cure is to subtract a baseline (a value estimate) from the return. A learned value used only as that baseline gives REINFORCE with a baseline. Let the value network bootstrap, scoring each step by its reward plus its estimate of the next state, and you get actor-critic, in which the actor chooses actions while the critic scores states, trading a little bias for a steadier signal at every step. PPO, a robust actor-critic, is the algorithm behind aligning language models, next.
RLHF: Aligning Language Models
Why does a chatbot feel helpful and safe? A large part of the answer is RLHF, reinforcement learning from human feedback, which turns fuzzy human preferences into a reward the model can be trained to maximize. It runs in three stages.
The middle stage is the clever one, and it is just a classifier. Humans compare pairs of responses and pick the better one; a reward model learns to predict those choices, giving a single scalar "how good is this response?" for anything the model produces. Stage three then uses PPO to push the language model toward high-reward responses, with a leash that keeps it from drifting too far and gaming the score.
Real-World Example: Building the Reward Model
Let us build stage two ourselves, from a teaching set of preference comparisons written for the book. The insight: preferring A over B means reward(A) > reward(B), which is exactly a Bradley-Terry model, fittable as a logistic regression on the difference between the two responses' features.
600 preference comparisons, a realistic teaching dataset written for the book, standing in for human raters' choices. Each row holds the rubric scores
(helpfulness, factuality, conciseness, safety) of a
chosen response and a rejected one. The true preference weights are hidden, the
reward model must recover them from the choices alone.
Fitting the reward model on the feature differences recovers what the raters value, from nothing but pairwise choices:
| Rubric feature | Learned reward weight | Reads as |
|---|---|---|
| safety | 1.75 | matters most |
| factuality | 1.42 | close second |
| helpfulness | 1.28 | important |
| conciseness | 0.80 | a modest plus |
Trained on 450 comparisons and tested on the 150 it never saw, the model ranks the preferred response higher in 81% of them (122 of 150), learning, with no access to the hidden weights, that safety and factuality dominate. That scalar reward is precisely what PPO would then optimize. The whole chapter meets here: a neural network, trained by an actor-critic method, against a reward learned from preference comparisons (in a real system, human raters' choices), that pipeline is a large part of why modern assistants are helpful and safe.
Deep RL Applications in Machine Learning & AI
Deep RL is behind many of the field's most visible milestones, and a growing share of production AI.
| System | What deep RL did |
|---|---|
| AlphaGo / AlphaZero | AlphaGo beat Go champions Lee Sedol (2016) and Ke Jie (2017) AlphaZero learned Go, chess and shogi by self-play alone and beat the strongest programs |
| Atari (DQN) | learned to play dozens of games from raw pixels, superhuman on many |
| Robotics & control | locomotion and manipulation via policy methods |
| RLHF (ChatGPT, Claude) | aligned large language models to be helpful, harmless, and honest |
| Recommendation & ops | long-horizon decisions in ads, logistics, and energy |
Deep RL is where learning to predict becomes learning to act. It produced the first systems to beat top professionals at Go (AlphaGo, then AlphaZero) and, through RLHF, is now a standard step in building the large language models at the center of AI. The frontier is active: researchers are refining preference optimization (methods like DPO that skip the RL step), scaling RLAIF where the feedback itself comes from AI, and using RL to teach models to reason through multi-step problems. The four ideas from these two chapters, reward, policy, value, and exploration, remain the vocabulary of it all, now expressed with neural networks at enormous scale.
Build deep RL in Python
The companion notebook trains a DQN (with experience replay and a target network) and a
policy-gradient agent on the GridWorld with torch, explains
actor-critic, then fits the RLHF reward model from teaching preference data,
each cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, torch, scikit-learn,
matplotlib, and openpyxl.
🎓 Key Takeaways
- ✓Deep RL replaces the Q-table with a neural network, so RL scales to huge and continuous state spaces.
- ✓DQN stabilizes neural Q-learning with experience replay and a target network; it reached human-level play on many Atari games from pixels.
- ✓Policy-gradient methods learn the policy directly (higher variance); actor-critic adds a value critic to steady them.
- ✓RLHF aligns LLMs in three stages: supervised fine-tuning, a reward model from human preferences, and PPO.
- ✓The reward model is just a classifier, we fit one that recovered safety and factuality as the top human priorities.
Practice Challenges
Five short challenges. Try them with torch and scikit-learn before checking the solutions.
Train a DQN
Train a neural Q-network with experience replay and a target network; report its evaluation return.
Policy gradient
Train a REINFORCE policy with a baseline; report its final return.
Categorical(logits=...); loss = -(logprob × advantage).Value vs policy
State the key difference between DQN and policy-gradient methods.
Reward model
Fit the RLHF Bradley-Terry reward model and name the feature humans weight most.
chosen - rejected features.Does it rank correctly?
Report the fraction of held-out comparisons the reward model ranks the way humans did.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on deep RL and RLHF. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You have met every ML paradigm, supervised, unsupervised, and reinforcement. The ML Case Study turns theory into practice. Case Study: An End-to-End ML Project walks a realistic dataset from raw data through splits, a baseline, cross-validation, and tuning, the full workflow, start to finish.