The last chapter's Q-learning kept a table with one row per state. That works for a 25-cell maze and collapses everywhere else: a chess position, a camera frame, and a robot's joint angles all have astronomically many states. Deep reinforcement learning replaces the table with a neural network that generalizes across states, and that single change unlocks the headline results of modern AI.
Swap the Q-table for a neural network: DQN (value-based), policy gradients (policy-based), and actor-critic (both) scale RL to huge problems, and RLHF, a reward model learned from human preferences plus PPO, is how those ideas align today's chatbots.
From Table to Network
A Q-table stores a separate number for every state-action pair. Double the resolution of a camera and the table explodes past the number of atoms in the universe, you can never fill it, let alone visit every entry. The escape is function approximation: instead of memorizing a value per state, train a neural network that reads the state and outputs the values, so nearby states share what they learn.
A Deep Q-Network (DQN) is exactly this: a net that maps a state to one Q-value per action. Training it naively is unstable, so DQN adds two fixes, experience replay (train on random past transitions to break correlations) and a target network (a slow-moving copy that provides a steady learning target). On our GridWorld the DQN reaches a return of 9.3, matching the tabular agent, but now it learns a function that would scale to a pixel screen. This is the algorithm that first played Atari from raw images.
Three Families: Value, Policy & Actor-Critic
Deep RL splits into three approaches, and our GridWorld shows two of them solving the same maze.
| Family | What it learns | Strength / cost |
|---|---|---|
| Value-based (DQN) | Q(s, a), then act greedily | sample-efficient; needs discrete actions |
| Policy-based (REINFORCE) | the policy (action probabilities) directly | handles continuous actions; higher variance |
| Actor-critic (A2C, PPO, SAC) | a policy (actor) + a value (critic) | low-variance, stable; the modern default |
A policy-gradient method skips values and pushes up the probability of actions that led to high reward. It also solves the maze (return 9.2) but learns more noisily, policy gradients are high-variance. The cure is to subtract a baseline (a value estimate) from the return. Promote that baseline into its own learned network and you get actor-critic: the actor chooses actions while the critic scores states to steady the learning. PPO, a robust actor-critic, is the algorithm behind aligning language models, next.
RLHF: Aligning Language Models
Why does a chatbot feel helpful and safe? A large part of the answer is RLHF, reinforcement learning from human feedback, which turns fuzzy human preferences into a reward the model can be trained to maximize. It runs in three stages.
The middle stage is the clever one, and it is just a classifier. Humans compare pairs of responses and pick the better one; a reward model learns to predict those choices, giving a single scalar "how good is this response?" for anything the model produces. Stage three then uses PPO to push the language model toward high-reward responses, with a leash that keeps it from drifting too far and gaming the score.
Real-World Example: Building the Reward Model
Let us build stage two ourselves, from real preference data. The insight: preferring A over B means reward(A) > reward(B), which is exactly a Bradley-Terry model, fittable as a logistic regression on the difference between the two responses' features.
600 human preference comparisons. Each row holds the rubric scores
(helpfulness, factuality, conciseness, safety) of a
chosen response and a rejected one. The true preference weights are hidden, the
reward model must recover them from the choices alone.
Fitting the reward model on the feature differences recovers what humans value, from nothing but pairwise choices:
| Rubric feature | Learned reward weight | Reads as |
|---|---|---|
| safety | 1.53 | matters most |
| factuality | 1.36 | close second |
| helpfulness | 1.11 | important |
| conciseness | 0.74 | a modest plus |
The model reaches 85% accuracy predicting held-out human judgments and correctly ranks the preferred response about 84% of the time, learning, with no access to the hidden weights, that safety and factuality dominate. That scalar reward is precisely what PPO would then optimize. The whole chapter meets here: a neural network, trained by an actor-critic method, against a reward learned from human preference, that pipeline is a large part of why modern assistants are helpful and safe.
Deep RL Applications in Machine Learning & AI
Deep RL is behind many of the field's most visible milestones, and a growing share of production AI.
| System | What deep RL did |
|---|---|
| AlphaGo / AlphaZero | beat world champions at Go, chess, and shogi via self-play RL |
| Atari (DQN) | learned to play dozens of games from raw pixels, superhuman on many |
| Robotics & control | locomotion, manipulation, and data-center cooling via policy methods |
| RLHF (ChatGPT, Claude) | aligned large language models to be helpful, harmless, and honest |
| Recommendation & ops | long-horizon decisions in ads, logistics, and energy |
Deep RL is where learning to predict becomes learning to act. It produced the first systems to out-plan human experts (AlphaGo/AlphaZero) and, through RLHF, is now a standard step in building the large language models at the center of AI. The frontier is active: researchers are refining preference optimization (methods like DPO that skip the RL step), scaling RLAIF where the feedback itself comes from AI, and using RL to teach models to reason through multi-step problems. The four ideas from these two chapters, reward, policy, value, and exploration, remain the vocabulary of it all, now expressed with neural networks at enormous scale.
Build deep RL in Python
The companion notebook trains a DQN (with experience replay and a target network) and a
policy-gradient agent on the GridWorld with torch, explains
actor-critic, then fits the RLHF reward model from real preference data,
each cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, torch, scikit-learn,
matplotlib, and openpyxl.
🎓 Key Takeaways
- ✓Deep RL replaces the Q-table with a neural network, so RL scales to huge and continuous state spaces.
- ✓DQN stabilizes neural Q-learning with experience replay and a target network; it mastered Atari from pixels.
- ✓Policy-gradient methods learn the policy directly (higher variance); actor-critic adds a value critic to steady them.
- ✓RLHF aligns LLMs in three stages: supervised fine-tuning, a reward model from human preferences, and PPO.
- ✓The reward model is just a classifier, we fit one that recovered safety and factuality as the top human priorities.
Practice Challenges
Five short challenges. Try them with torch and scikit-learn before checking the solutions.
Train a DQN
Train a neural Q-network with experience replay and a target network; report its evaluation return.
Policy gradient
Train a REINFORCE policy with a baseline; report its final return.
Categorical(logits=...); loss = -(logprob × advantage).Value vs policy
State the key difference between DQN and policy-gradient methods.
Reward model
Fit the RLHF Bradley-Terry reward model and name the feature humans weight most.
chosen - rejected features.Does it rank correctly?
Report the fraction of held-out comparisons the reward model ranks the way humans did.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on deep RL and RLHF. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You have met every ML paradigm, supervised, unsupervised, and reinforcement. The ML Case Study turns theory into practice. Case Study: An End-to-End ML Project walks a real dataset from raw data through splits, a baseline, cross-validation, and tuning, the full workflow, start to finish.