Wait, What?
A computer can differentiate a program without turning the whole program into a symbolic formula.
Automatic differentiation, or AD, works by applying the chain rule to the actual sequence of elementary operations performed by a program. That makes it different from symbolic differentiation and different again from finite differences. The subject begins with a tiny idea—carry derivative information alongside values—and grows into the machinery behind modern optimisation, scientific computing and machine learning.
Quick Answer
Learn automatic differentiation through derivatives as local rules → computational graphs → dual numbers → forward mode → Jacobian-vector products → reverse mode → adjoints → vector-Jacobian products → backpropagation → higher-order derivatives → checkpointing → custom derivatives → gradient checking → performance and memory trade-offs. A beginner should be able to trace derivatives through a short expression. A professional should know which AD mode matches the input/output shape, how intermediate values affect memory, and how to validate custom gradient code against independent numerical checks.
1. Begin With the Chain Rule, Not a Framework
If a program computes y = f(g(x)), the derivative is built from local derivatives. AD scales this principle to long programs by recording or transforming the sequence of primitive operations.
2. Separate AD From Symbolic Differentiation
Symbolic differentiation manipulates expressions. AD differentiates executed operations. The distinction matters when programs include loops, branches, arrays and library operations that are awkward to expand into one symbolic expression.
3. Separate AD From Finite Differences
Finite differences estimate derivatives by perturbing inputs. They are useful for checking, but step size creates truncation and floating-point trade-offs. AD instead propagates derivatives through operations and can compute derivatives to working floating-point precision without choosing a differencing step.
4. Dual Numbers Make Forward Mode Concrete
Write a value as a + bε with ε² = 0. When arithmetic is extended to these objects, the ordinary part carries the function value while the ε coefficient carries a directional derivative. This is a powerful beginner model because the derivative moves forward beside the value.
5. Forward Mode Computes JVPs Naturally
For a function with Jacobian J and direction v, forward mode efficiently computes Jv, a Jacobian-vector product. It is especially attractive when the number of input directions is small relative to the number of outputs.
JAX exposes forward- and reverse-mode transformations directly; see JAX Advanced Automatic Differentiation.
6. Reverse Mode Runs Sensitivity Backward
Reverse mode first evaluates the program, then propagates adjoint values from outputs back toward inputs. Instead of asking how one input direction changes every output, it asks how a weighted combination of outputs depends on every input.
7. Reverse Mode Computes VJPs Naturally
Given a cotangent vector u, reverse mode computes uᵀJ, a vector-Jacobian product. When a scalar loss depends on millions of parameters, one reverse pass can produce the full gradient, which is why reverse-mode AD is central to machine learning.
8. Backpropagation Is a Specialised Reverse-Mode Pattern
Backpropagation through a neural network is reverse-mode differentiation applied to a layered computation. Learning AD therefore clarifies backpropagation rather than treating it as a separate magical algorithm.
PyTorch describes its autograd engine as a reverse automatic differentiation system that records operations into a directed acyclic graph and traverses that graph during backward computation. See PyTorch Autograd Mechanics.
9. The Computational Graph Is a Dependency Structure
Nodes represent intermediate values or operations; edges represent dependencies. A forward trace computes values in dependency order. A reverse trace accumulates sensitivities in the opposite direction. The graph is a learning tool, but production systems may implement the same logic through tapes, tracing, source transformation or compiler-level techniques.
10. Source Transformation and Operator Overloading Are Different Implementation Routes
Some AD systems overload arithmetic so operations build derivative-aware objects or traces. Others transform program representations before or during compilation. Enzyme, for example, differentiates LLVM-level code; see Enzyme AD documentation.
11. Shape Determines the Best Mode
If a function maps a few inputs to many outputs, forward mode can be economical. If many inputs feed a few outputs, reverse mode is usually preferable. Large Jacobians are often never formed explicitly; professional code asks for JVPs or VJPs instead.
12. Reverse Mode Pays a Memory Bill
Backward computation may require intermediate values from the forward pass. PyTorch documents saved tensors because reverse mode often trades memory for the ability to compute gradients later. That trade-off becomes significant in deep networks and long simulations.
13. Checkpointing Trades Recalculation for Memory
Instead of saving every intermediate, checkpointing stores selected states and recomputes others during the reverse pass. This reduces peak memory at the cost of extra computation. Professional AD work treats this as an explicit time-memory design decision.
14. Higher-Order Derivatives Compose AD Transforms
Gradients can themselves be differentiated. Hessian-vector products, Jacobians and higher-order derivatives can often be built by composing forward and reverse transformations. The order matters for efficiency.
15. Not Every Operation Is Smooth
Absolute value at zero, discrete indexing decisions, comparisons and branch boundaries raise questions about derivative definitions. Frameworks choose conventions. Learners should distinguish “the software returned a gradient” from “the mathematical derivative exists and means what this optimisation problem needs.”
16. In-Place Mutation Can Break Derivative Reasoning
Reverse mode depends on the history of values. Mutating a tensor that a backward rule still needs can invalidate that history, which is why AD frameworks track versions or restrict certain in-place operations.
17. Custom Gradients Need Independent Tests
A custom backward formula can be fast and wrong. Compare analytical JVPs or VJPs against finite differences on small, well-conditioned test cases. JAX includes gradient-checking utilities that compare derivative products against finite-difference directions; see jax.test_util.check_vjp.
18. Link AD to Optimisation Without Giving Away Ownership
The existing Numerical Optimisation Algorithms article owns optimisation methods such as gradient descent, Newton methods and line search. This article owns how derivatives are computed efficiently for programs that those optimisation methods may consume.
19. Common Learning Failure States
- Calling finite differences automatic differentiation.
- Thinking reverse mode means reversing the numerical function itself.
- Forming a full Jacobian when only a JVP or VJP is needed.
- Ignoring the input/output dimensionality when choosing a mode.
- Forgetting that reverse mode may need saved intermediates.
- Trusting a custom backward rule without gradient checking.
- Assuming every program branch has a mathematically smooth derivative.
- Confusing backpropagation with a completely different theory from reverse-mode AD.
20. A Beginner-to-Professional Learning Ladder
- Level 1: differentiate a short scalar expression by hand.
- Level 2: draw its computational graph.
- Level 3: propagate dual-number derivatives forward.
- Level 4: compute one JVP.
- Level 5: propagate adjoints backward.
- Level 6: compute one VJP and a scalar-loss gradient.
- Level 7: compare forward and reverse complexity by input/output shape.
- Level 8: implement and test a custom derivative rule.
- Level 9: measure graph memory and checkpointing costs.
- Level 10: select mixed-mode derivatives for Jacobians, Hessian-vector products and production workloads.
21. Teach by Predicting Before Running
Give learners a three-operation program and ask them to predict both the value and derivative at each node before running an AD framework. PRIMM’s predict-run-investigate sequence is useful because it delays code production until the learner can read the computation. See Sentance, Waite and Kallia on PRIMM.
22. Fade Worked Examples
Start with a fully annotated forward- and reverse-mode trace. Then remove one derivative, then an entire local rule, then the traversal order. Research on programming instruction has found benefits from faded worked examples paired with metacognitive scaffolding for novice problem solving.
See Shin et al. (2023).
23. Use Delayed Retrieval
After the first lesson, ask learners later to reconstruct the difference between JVP and VJP without notes. Then give a new function shape and ask which mode they would choose. Retrieval plus transfer is more informative than re-reading the same graph.
24. Professional Validation Checklist
- Define the exact derivative object required: gradient, JVP, VJP, Jacobian or Hessian-vector product.
- Check numerical types and differentiability assumptions.
- Verify custom rules on small cases.
- Measure peak memory, not only runtime.
- Test branch and boundary cases.
- Compare forward, reverse and mixed-mode alternatives.
- Profile saved intermediates and recomputation.
- Keep an independent finite-difference check for selected regression tests.
Professional Direction
Advanced study includes source-transformation AD, compiler-level differentiation, sparse Jacobians, implicit differentiation, differentiating through optimisation procedures, checkpoint scheduling, higher-order reverse mode, custom primitives, nondifferentiable programming constructs and AD for scientific simulation.
Algorithm-learning rule: never ask only “did the framework return a gradient?” Ask which derivative product was computed, why that mode fits the problem shape, what intermediate information had to be retained, and what independent evidence says the derivative is correct.
eduKate Sengkang 