Wait, What?
An algorithm can learn what to do without being shown the correct action for every situation.
Reinforcement learning studies decision-making through interaction. An agent observes a state, chooses an action, receives a reward and moves to a new state. The central difficulty is delayed consequence: an action that looks good now may create a poor future, while a small sacrifice can open a much better route later.
This article owns the reinforcement-learning algorithm learning job. The existing Dynamic Programming article owns general state-and-recurrence design, the MCMC article owns sampling from probability distributions, and the Algorithmic Game Theory article owns strategic interaction. Reinforcement learning is different: it asks how an agent should improve sequential decisions from experience.
Quick Answer
Learn reinforcement learning through the route state → action → reward → transition → return → policy → value → Bellman equation → dynamic programming → Monte Carlo estimation → temporal-difference learning → Q-learning → exploration → function approximation → DQN → policy gradients → actor–critic → offline evaluation → safety and deployment. A beginner should be able to trace a tiny Markov decision process by hand. A professional should be able to identify whether a method is on-policy or off-policy, model-based or model-free, value-based or policy-based, explain its bias–variance and exploration trade-offs, evaluate it without reward leakage, and know when reinforcement learning is the wrong tool.
1. Begin With a Tiny World You Can Draw
Use a five-cell grid. One cell is the start, one gives +10, one gives −10 and every move costs −1. The learner can move left or right. Before writing code, ask: which action is best from each cell, and why?
The grid makes delayed reward visible. It also exposes an important truth: the best action depends on both immediate reward and what becomes possible afterwards.
2. A Markov Decision Process Gives the Mathematical Skeleton
A finite Markov decision process is usually described by states, actions, transition probabilities, rewards and a discount factor. The Markov assumption says that, given the current state and action, the future does not need the complete earlier history.
Do not teach the notation before the learner can point to each part in the grid. Mathematics should compress an understood process, not replace understanding.
3. Return Is the Quantity the Agent Ultimately Tries to Improve
The return is the accumulated future reward. With discount factor γ, later rewards are usually weighted by powers of γ. Discounting can represent time preference, uncertainty about distant outcomes or simply make an infinite-horizon sum mathematically manageable.
A useful exercise is to compare two action sequences with the same immediate reward but different future outcomes. That breaks the common novice habit of judging actions one step at a time.
4. A Policy Is a Decision Rule
A policy tells the agent how to choose actions from states. A deterministic policy selects one action. A stochastic policy assigns probabilities to actions.
Beginners often confuse a policy with a value function. Keep them separate: the policy answers “what should I do?” while a value function answers “how good is this situation, assuming future behaviour follows some policy?”
5. State Value and Action Value Answer Different Questions
Vπ(s) estimates expected return from state s under policy π. Qπ(s,a) estimates expected return if the agent first takes action a in state s and then follows π.
Q-values are especially useful when choosing among actions because each candidate action can be compared directly.
6. The Bellman Equation Is a Recursive Consistency Condition
A value equals expected immediate reward plus discounted value of what comes next. This simple decomposition is the structural heart of many reinforcement-learning algorithms.
Connect this to the existing Dynamic Programming lesson: both disciplines exploit repeated substructure. The difference is that reinforcement learning may need to estimate the dynamics or values from data rather than receiving a fully specified model.
7. Policy Evaluation Asks: How Good Is This Fixed Policy?
If the transition model is known, iterative policy evaluation repeatedly applies the Bellman expectation backup until values stabilize. Each sweep propagates information about future reward backwards through the state space.
Have learners calculate two sweeps by hand. The visible movement of value is more educational than starting with a library call.
8. Policy Improvement Turns Values Into Better Decisions
Once a policy has been evaluated, choose actions that are greedy with respect to the estimated values. Alternating evaluation and improvement gives policy iteration.
This teaches a general pattern: estimate consequences, improve the decision rule, then repeat.
9. Value Iteration Compresses Evaluation and Improvement
Value iteration applies the Bellman optimality backup directly, replacing the fixed-policy expectation with a maximum over actions. It repeatedly pushes each state’s value toward the best achievable future return.
For a known finite MDP, value iteration is an excellent bridge from textbook dynamic programming into reinforcement-learning thinking.
10. What If the Transition Model Is Unknown?
Now the agent must learn from sampled experience. This is where the field becomes recognizably “reinforcement learning”: states, actions, rewards and next states arrive through interaction or logged trajectories rather than a complete transition table.
11. Monte Carlo Methods Learn From Complete Returns
Monte Carlo value estimation waits until an episode finishes, computes the observed return and updates estimates toward that return. The target is conceptually clean because it uses an actual sampled outcome rather than another current estimate.
The cost is variance and delay: a learner may need to wait many steps before an update is available.
12. Temporal-Difference Learning Bootstraps
TD(0) updates after each transition using the target r + γV(s’). Because V(s’) is itself an estimate, the algorithm bootstraps.
This gives faster, incremental learning but creates a moving-target problem. The distinction between observed return and bootstrapped target is one of the most important conceptual boundaries in the subject.
13. SARSA Learns the Value of the Behaviour It Actually Follows
SARSA updates Q(s,a) using the next action actually selected by the current behaviour policy. This makes it an on-policy control method.
In a risky grid-world, an exploratory policy may learn a safer route because its value estimates include the consequences of its own exploratory behaviour.
14. Q-Learning Uses a Greedy Target
Q-learning updates toward r + γ max_a’ Q(s’,a’). The behaviour may explore, but the update target assumes the best estimated next action. This is why standard Q-learning is called off-policy.
Q[s,a] += alpha * (reward + gamma * max(Q[next_state]) - Q[s,a])The one-line update is easy to memorize and easy to misunderstand. Require learners to identify the prediction, target and temporal-difference error before they are allowed to code it.
15. Exploration Is a Decision Problem Inside the Decision Problem
If the agent always chooses the current best-looking action, it may never discover a better one. If it explores forever without concentrating on good actions, performance remains poor.
ε-greedy is the simplest starting point: choose a random action with probability ε and a greedy action otherwise. More advanced approaches include optimistic initialization, upper-confidence methods, entropy bonuses and intrinsic-motivation signals.
16. Tables Stop Scaling When the State Space Explodes
A table works when every relevant state-action pair can be stored. Images, continuous controls and large combinatorial states make direct tables impossible. Function approximation replaces individual entries with a parameterized model that generalizes across related states.
17. Deep Q-Networks Approximate Q-Values With Neural Networks
DQN combines Q-learning with a neural network, experience replay and a separate target network. Replay reduces strong temporal correlation in update batches, while a target network slows the movement of bootstrapped targets.
The current TensorFlow Agents DQN tutorial walks through environment setup, data collection, replay and training. Use such implementations after the learner can already trace tabular Q-learning.
18. Policy-Gradient Methods Optimize the Policy Directly
Instead of learning action values and taking an argmax, policy-gradient methods parameterize a stochastic policy and adjust parameters in a direction that increases expected return.
REINFORCE is the canonical starting point. It is simple but can have high variance because complete sampled returns are noisy gradient signals.
19. Baselines Reduce Variance Without Changing the Desired Gradient
Subtracting a baseline, often a learned state-value estimate, turns raw return into an advantage-like signal. The question becomes not merely “was the outcome good?” but “was it better than expected from this state?”
20. Actor–Critic Splits Action Selection From Evaluation
The actor represents the policy. The critic estimates value information used to train the actor. This architecture supports incremental updates and forms the basis of many modern methods.
Berkeley’s current CS 285 course treats policy gradients, value/Q-function learning and actor–critic as central reinforcement-learning families. That grouping is useful pedagogically because it lets learners compare what each family represents, what target it estimates and how data are reused.
21. On-Policy and Off-Policy Data Are Not Interchangeable by Default
On-policy algorithms learn from trajectories generated by the policy being improved. Off-policy algorithms can learn about one target policy while data come from another behaviour policy.
This matters enormously in practice because fresh interaction can be expensive or unsafe. Reusing old data is attractive, but distribution shift and extrapolation error can make naive reuse unreliable.
22. Offline Reinforcement Learning Raises the Evidence Bar
Offline RL learns from a fixed logged dataset without freely gathering new experience. The learner must reason about which actions are well supported by the data and which are speculative. A policy that appears strong under an inaccurate learned value model can fail badly when deployed.
23. Reward Design Can Quietly Redefine the Task
An agent optimizes the reward it is given, not the intention in a designer’s head. Proxy rewards can create reward hacking, perverse shortcuts or undesirable trade-offs.
For educational examples, ask learners to invent two reward functions for the same grid and predict how the learned behaviour changes. This makes objective misspecification concrete.
24. Evaluation Must Separate Training Experience From Decision Quality
Report return over multiple random seeds, confidence intervals, environment variations and failure cases. Inspect learning curves rather than only the best checkpoint. A single lucky run is not evidence of a reliable algorithm.
Professionals also distinguish online evaluation, simulation evaluation and off-policy evaluation from logs. Each answers a different question and carries different assumptions.
25. Common Learning Failure States
- Optimizing immediate reward instead of return.
- Confusing state value with action value.
- Using Bellman equations as symbols without understanding the one-step decomposition.
- Calling Q-learning on-policy because the behaviour is ε-greedy.
- Ignoring exploration and declaring the first good policy optimal.
- Using a neural network before understanding the tabular update.
- Training and evaluating on the same random seed.
- Treating one reward definition as a neutral description of the task.
- Using offline data without checking coverage of candidate actions.
- Reporting only average return and hiding catastrophic runs.
26. A Beginner-to-Professional Learning Ladder
- Level 1: trace states, actions and rewards in a small grid.
- Level 2: calculate discounted returns.
- Level 3: evaluate a fixed policy by hand.
- Level 4: run value iteration on a tiny MDP.
- Level 5: implement tabular Monte Carlo, TD(0), SARSA and Q-learning.
- Level 6: compare on-policy and off-policy learning under exploration.
- Level 7: replace a Q-table with simple function approximation.
- Level 8: implement or study DQN with replay and target networks.
- Level 9: derive the logic of REINFORCE and actor–critic updates.
- Level 10: evaluate offline, distribution-shift and safety constraints before deployment.
27. Teach With Predict → Run → Investigate → Modify → Make
Programming education research supports beginning with code reading and prediction before independent construction. The current Raspberry Pi Foundation PRIMM course structures programming as Predict, Run, Investigate, Modify and Make, while the National Centre for Computing Education pedagogy guidance emphasizes reading, tracing and explaining code before writing.
For reinforcement learning, give learners a complete 12-line tabular Q-learning loop. Ask them to predict which table entry changes, run one controlled transition, identify the target and error, then modify one element such as ε or γ. Only after that should they build a new environment.
28. Use Counterexamples, Not Only Success Cases
Create a cliff-walking environment where a greedy shortcut is risky, an environment with sparse reward, and a dataset that omits one critical action. Ask which algorithmic assumption breaks in each case.
29. Professional Direction
Advanced study includes eligibility traces and TD(λ), distributional RL, double and dueling value methods, PPO, SAC, deterministic policy gradients, model-based RL, planning with learned dynamics, offline RL, imitation learning, inverse reinforcement learning, multi-agent RL, constrained MDPs, safe exploration and formal evaluation under distribution shift.
Algorithm-learning rule: never ask only whether the agent earned a high reward. Ask what state representation it saw, what data generated the update, whether the target bootstrapped, which policy produced the experience, how exploration changed evidence, and whether the evaluation actually supports deployment.
eduKate Sengkang 