How to Learn Numerical Optimisation Algorithms: Gradient Descent, Newton Methods, Line Search and Convergence

Wait, What?

In real numerical optimisation, “take the derivative and set it to zero” is often only the beginning of the problem.

When an objective depends on many variables, has no convenient closed-form minimiser, or is expensive to evaluate, optimisation becomes an iterative algorithmic process. A professional method must choose a direction, choose a step size, decide whether progress is trustworthy, stop for a defensible reason, and account for conditioning, finite precision and model structure. The learning goal is therefore not to memorise gradient descent. It is to understand how local information is converted into a sequence of increasingly useful candidate solutions.

Quick Answer

Learn numerical optimisation through the route objective function → geometry → gradient → descent direction → step size → line search → conditioning → Hessian curvature → Newton step → damping/trust → quasi-Newton approximation → stochastic gradients → convex versus non-convex behaviour → stopping criteria → residuals and tolerances → measured performance. Begin on one- and two-dimensional quadratics where every update can be drawn.

1. Separate the Model From the Solver

An optimisation problem defines variables, an objective and possibly constraints. The solver is the algorithm used to search for a solution. If the objective does not represent the intended real task, an optimiser can converge beautifully to the wrong answer.

This article complements the existing linear programming algorithms guide. Linear programming exploits a special linear structure. Here the focus is iterative numerical methods for broader smooth optimisation, especially unconstrained or simply constrained problems.

2. The Gradient Points Uphill

For a differentiable objective, the gradient collects the partial derivatives. Locally, it identifies the direction of steepest increase under the standard Euclidean geometry. Negating it therefore gives a natural descent direction.

Gradient descent updates x_next = x - step_size × gradient. The expression is short; the difficult questions are how large the step should be, what geometry the objective has, and how to know when the method has done enough.

3. Trace a Quadratic Before General Functions

Use a one-dimensional bowl such as f(x) = (x - 3)^2. Choose a starting point and several step sizes. Compute the derivative, take one update, and mark the new point on the graph. Then move to an elongated two-dimensional quadratic and draw contour lines.

This reveals two foundational behaviours: a reasonable step approaches the minimum, while an oversized step can overshoot or diverge. On badly conditioned contours, steepest descent can zig-zag across a narrow valley even while each step decreases the objective.

4. Step Size Is Part of the Algorithm

A tiny fixed step may be safe but painfully slow. A large one may fail. A learning sequence should therefore treat step-size choice as a first-class design decision, not a number chosen after the “real” algorithm.

MIT’s nonlinear-optimisation notes develop gradient descent together with descent lemmas and step-size conditions, and the notes were updated in July 2026. See MIT 6.7220 / 15.084: Gradient Descent and Descent Lemmas.

5. Line Search Tests a Step Before Committing to It

Backtracking line search starts with a candidate step and shrinks it until a sufficient-decrease condition is satisfied. This converts “How far should I move?” into a controlled local test.

The important lesson is not one constant in an Armijo-style rule. It is the algorithmic pattern: propose a move, test whether observed improvement is adequate relative to the local model, and reduce the move when the evidence is poor.

6. Conditioning Explains Slow Zig-Zagging

If curvature differs greatly by direction, a single global step size must respect the steep direction while making only modest progress along the shallow one. This can produce long zig-zag paths.

At intermediate level, connect this to eigenvalues of the Hessian on quadratic objectives. At professional level, recognise conditioning as both a mathematical and numerical issue that can motivate scaling, preconditioning or methods that use curvature information.

7. Newton’s Method Uses Curvature to Rescale the Step

Newton’s method builds a local quadratic model using the gradient and Hessian. In multiple dimensions, the Newton direction solves a linear system involving the Hessian rather than simply moving opposite the gradient.

Near a well-behaved solution, Newton steps can converge very rapidly. Far away, an undamped Newton step may point somewhere unhelpful, and computing or solving with the Hessian can be expensive.

8. “Second Order” Does Not Mean “Always Better”

• Gradient descent: cheap first-order information, often simple and scalable.
• Newton: richer curvature information, potentially fast local convergence, but higher per-iteration cost.
• Damped Newton: combines Newton directions with step control.
• Quasi-Newton: approximates curvature from successive gradients rather than forming the exact Hessian.

The correct comparison includes total work, memory, derivative cost and reliability—not iteration count alone.

9. Quasi-Newton Methods Trade Exact Curvature for Learned Curvature

Methods such as BFGS update an approximation to the Hessian or its inverse using changes in positions and gradients. Limited-memory variants such as L-BFGS retain only a compact history and are useful when the variable dimension is large.

This is an instructive professional pattern: the algorithm can accumulate structural information from previous iterations and use it to improve future directions without paying the full cost of an exact second-order model.

10. Stochastic Gradient Descent Changes the Cost Model

If the objective is a large average over data, computing the full gradient may be expensive. Stochastic or mini-batch gradient methods estimate the gradient from a subset of data. Individual steps become cheaper but noisier.

MIT’s nonlinear-optimisation course includes stochastic gradient descent and empirical risk minimisation, with notes updated in August 2026. See MIT: Stochastic Gradient Descent and Empirical Risk Minimisation.

11. Convexity Changes What “Local” Means

For a convex differentiable objective, every local minimum is global, giving powerful guarantees that are unavailable in general non-convex problems. Strong convexity and smoothness allow still sharper convergence statements.

Stanford’s EE364a, active in Summer 2026, teaches recognition of convex optimisation problems together with theory and computational methods. See Stanford EE364a: Convex Optimization I.

12. Non-Convex Optimisation Requires More Cautious Claims

In non-convex objectives, a stationary point need not be a global minimum. It may be a local minimum, maximum or saddle. Therefore “the gradient is near zero” can be a valid stopping signal for a stationarity goal without proving global optimality.

The professional report should state what the algorithm guarantees under the actual assumptions, not upgrade a local convergence result into a global claim.

13. Stopping Is a Decision, Not a Feeling

Possible stopping tests include small gradient norm, small step norm, small relative objective improvement, a satisfied optimality residual, a maximum iteration/time budget, or a problem-specific tolerance. Each answers a different question.

• A small objective change can mean convergence—or simply a step size that became too small.
• A small gradient may indicate stationarity but not necessarily global optimality.
• A time limit is an operational stop, not mathematical convergence.
• Absolute tolerances can be misleading when variables or objectives have very different scales.

14. Numerical Precision Is Part of Correctness in Practice

Finite-precision arithmetic affects gradients, Hessian solves, line-search comparisons and stopping rules. Poor scaling can magnify error. Nearly singular curvature can make Newton systems unstable. Numerical optimisation therefore needs checks on residuals and conditioning, not just symbolic derivations.

Recent solver work continues to emphasise the implementation layer: for example, Stanford researchers reported GPU acceleration and mixed-precision linear algebra in a 2026 conic optimisation solver study. See CuClarabel: GPU Acceleration for a Conic Optimization Solver. The lesson for learners is not to imitate one solver, but to recognise that professional optimisation performance depends on numerical linear algebra and hardware as well as mathematical iteration formulas.

15. Benchmark the Whole Method, Not One Update Formula

When comparing gradient descent, Newton or quasi-Newton methods, report total objective and gradient evaluations, linear solves, wall-clock time, memory, final residuals and tolerance settings. Comparing only iteration counts can favour an expensive method unfairly.

The existing professional algorithm-evaluation guide provides the wider benchmarking discipline.

16. Common Learning Failure States

• Using the gradient direction instead of the negative gradient for minimisation.
• Choosing a large step because “larger movement means faster convergence.”
• Stopping when the objective barely changes without checking whether the step size collapsed.
• Assuming Newton always descends.
• Inverting a Hessian explicitly when solving a linear system is the safer computational formulation.
• Claiming global optimality from a stationary point in a non-convex objective.
• Comparing stochastic and full-gradient methods by iteration count rather than comparable work.
• Ignoring scaling, precision and residuals.

17. A Scaffold-Fade Learning Ladder

• Level 1: draw a one-dimensional objective and identify uphill/downhill directions.
• Level 2: perform several gradient-descent steps by hand with different step sizes.
• Level 3: trace backtracking line search and explain why each rejected step failed.
• Level 4: diagnose zig-zagging on an ill-conditioned quadratic.
• Level 5: compute a one-dimensional Newton step and compare it with gradient descent.
• Level 6: solve a multidimensional Newton direction using a linear system.
• Level 7: compare gradient, Newton and quasi-Newton methods under equal evaluation budgets.
• Level 8: run a real numerical problem and report assumptions, residuals, tolerances, conditioning and measured resource cost.

Worked examples and metacognitive scaffolding are useful when the learner must coordinate several state changes. Shin and colleagues found that faded worked examples combined with metacognitive scaffolding supported programming problem solving and self-regulation. See Shin et al. (2023). In optimisation lessons, fade the gradient, direction, step test and stopping interpretation separately so the learner must reconstruct why each move is allowed.

18. Immediate, Delayed and Transfer Checks

• Immediate: compute one gradient update and predict whether the objective should fall.
• Step-size: explain why one candidate step diverges while another converges.
• Curvature: compare gradient and Newton directions on the same quadratic.
• Stopping: decide which residual or tolerance actually supports the desired claim.
• Delayed: reconstruct gradient descent and line search from geometric ideas rather than memorised code.
• Transfer: decide whether a new problem is linear programming, smooth unconstrained optimisation, stochastic optimisation or a constrained convex problem.

19. AI Assistance Boundary

AI can generate toy objectives, calculate candidate gradients, plot trajectories and compare solver traces. The learner should independently verify derivatives, state the assumptions behind convergence claims, inspect stopping evidence and distinguish a solver status from proof of global optimality.

Professional Direction

Advanced study includes conjugate-gradient methods, trust-region algorithms, BFGS and L-BFGS, accelerated first-order methods, proximal methods, coordinate descent, stochastic variance reduction, automatic differentiation, constrained optimisation, interior-point methods, operator splitting, ADMM, preconditioning and differentiable optimisation. Stanford’s Spring 2026 EE364b course, for example, covers proximal methods, ADMM, robust and stochastic optimisation, global optimisation and modern non-convex optimisers. See Stanford EE364b: Convex Optimization II.

Algorithm-learning rule: optimisation expertise begins when “take a step downhill” becomes a disciplined sequence of modelling, direction choice, step control, convergence evidence and numerical verification.