How to Learn BFGS: Quasi-Newton Updates, Secant Conditions, Wolfe Line Searches, Curvature and L-BFGS Engineering

Can an optimiser learn the shape of a curved surface without ever being given the full matrix of second derivatives? BFGS does exactly that. It uses successive positions and gradients to build an increasingly useful approximation of curvature, then uses that approximation to choose better search directions.

This Learning Hall article develops BFGS from gradient geometry to quasi-Newton reasoning, line search, numerical safeguards and limited-memory engineering. It complements the broader Numerical Optimisation Algorithms article, as well as specialist pages such as Nelder–Mead. It does not replace their wider jobs.

Quick Read

Gradient descent uses slope but ignores most local curvature information.
Newton’s method uses the Hessian but computing and factorising it can be expensive.
BFGS is a quasi-Newton method: it updates a Hessian or inverse-Hessian approximation from observed changes in gradients.
The secant relation connects a step s with the gradient change y.
A positive curvature quantity yᵀs is central to preserving a useful positive-definite approximation.
A line search, commonly designed around Wolfe conditions, is part of the method’s practical behaviour—not an optional decoration.
Full BFGS stores a dense n×n approximation; L-BFGS keeps only a limited history and is more suitable for very high-dimensional problems.

1. Beginner Level: Why Gradient Direction Is Not Enough

Suppose you are standing on a bowl-shaped surface and want to reach its lowest point. The gradient tells you the direction of steepest uphill change, so moving against the gradient takes you downhill. But a single gradient does not tell you whether the surface is broad in one direction and sharply curved in another.

On an elongated valley, plain gradient descent can zig-zag because the same scalar step size must cope with very different curvature along different directions. Newton’s method corrects for that geometry using the Hessian matrix of second derivatives. BFGS tries to obtain much of that benefit without requiring the exact Hessian.

2. Newton’s Step and the Quasi-Newton Question

For a smooth objective f(x), Newton’s method uses a local quadratic model. If g is the gradient and ∇²f is the Hessian, a Newton direction solves:

Hessian * p = -gradient

The quasi-Newton question is: can we build a matrix that behaves enough like the Hessian, or its inverse, using only function and gradient information observed during the optimisation?

BFGS answers yes by updating the approximation after every accepted step.

3. The Two Observations BFGS Learns From

After moving from xₖ to xₖ₊₁, define:

s_k = x_(k+1) - x_k
y_k = grad f(x_(k+1)) - grad f(x_k)

The vector s says how the parameters moved. The vector y says how the gradient changed because of that movement. For a true quadratic objective with Hessian B, these quantities satisfy y = B s exactly. For a general smooth objective, they approximate local curvature over the accepted step.

That relation is the conceptual heart of quasi-Newton methods: curvature can be inferred from how slopes change as we move.

4. The Secant Condition

If Bₖ₊₁ is the next Hessian approximation, a natural requirement is:

B_(k+1) s_k = y_k

If Hₖ₊₁ approximates the inverse Hessian instead, the corresponding relation is Hₖ₊₁ yₖ = sₖ. This is called a secant condition because it is a multidimensional analogue of using finite slope changes to approximate derivative information.

Many matrices could satisfy one vector equation. BFGS chooses a structured rank-two update that preserves symmetry and, under suitable curvature conditions, positive definiteness.

5. The Inverse-Hessian BFGS Update

A common implementation stores H, an approximation to the inverse Hessian. Let ρ = 1/(yᵀs). Then:

H_new = (I - rho*s*y.T) H (I - rho*y*s.T) + rho*s*s.T

The notation is compact, but learners should not begin by memorising it. First remember the job: update H so that the latest step/gradient-change pair is respected while retaining useful information learned from previous iterations.

The search direction is then:

p_k = -H_k g_k

When H is positive definite and the gradient is nonzero, this is a descent direction because gᵀp is negative.

6. Curvature: Why yᵀs Matters

The scalar yᵀs measures whether the observed gradient change is consistent with positive curvature along the step. When yᵀs > 0, the BFGS update can preserve positive definiteness if the previous approximation was positive definite.

If yᵀs is zero, extremely small or negative, blindly applying the update can be unstable or destroy useful geometry. This can happen because the objective is nonconvex, gradients are noisy or inaccurate, the step is poorly chosen, or finite-precision effects dominate.

Professional implementations therefore monitor curvature and may skip, damp or otherwise safeguard an update rather than assuming every observed pair is trustworthy.

7. BFGS Needs a Line Search

BFGS proposes a direction p, but it still needs a step length α:

x_new = x + alpha * p

A line search tries to find a step that gives sufficient improvement without moving so far that the local model becomes uninformative. Wolfe-style conditions are commonly used because they combine a sufficient-decrease requirement with a curvature requirement involving the directional derivative.

This is an important learning correction: “BFGS” in a practical solver is not just the matrix update. Search direction, line search, stopping criteria, gradient quality and update safeguards work together as one optimisation procedure.

8. A Small Quadratic Thought Experiment

Consider f(x, y) = 100x² + y². The surface is much steeper in x than y. Gradient descent with a single step scale can bounce across the narrow x direction while making slow progress in y. An inverse-Hessian model learns that movement in x should be scaled very differently from movement in y.

On an exact quadratic and under ideal arithmetic/line-search conditions, quasi-Newton information becomes progressively aligned with the true curvature. On general nonlinear objectives, the approximation keeps adapting as the local geometry changes.

9. Transparent Pseudocode

x = initial point
H = identity matrix
g = gradient(x)

repeat:
    p = -H @ g
    alpha = line_search(f, gradient, x, p)
    x_new = x + alpha * p
    g_new = gradient(x_new)

    s = x_new - x
    y = g_new - g

    if y.T @ s is safely positive:
        rho = 1 / (y.T @ s)
        H = (I - rho*s*y.T) @ H @ (I - rho*y*s.T) \
            + rho*s*s.T
    else:
        safeguard_update()

    x, g = x_new, g_new

until stopping_condition

The exact line-search and safeguard policies are major implementation choices. A teaching implementation should make them visible rather than hiding them inside a magical call.

10. Using SciPy Without Giving Up Understanding

import numpy as np
from scipy.optimize import minimize

def f(x):
    return 100.0 * x[0]**2 + x[1]**2

def grad(x):
    return np.array([200.0 * x[0], 2.0 * x[1]])

result = minimize(
    f,
    x0=np.array([2.0, 2.0]),
    jac=grad,
    method="BFGS",
    options={"gtol": 1e-8},
)

print(result.x)
print(result.success)
print(result.message)

Current SciPy documentation exposes BFGS through scipy.optimize.minimize and returns an inverse-Hessian approximation in the optimisation result. The professional habit is to inspect the termination message and diagnostics rather than treating a returned vector as proof of successful optimisation.

11. Full BFGS Versus L-BFGS

A dense inverse-Hessian approximation uses O(n²) storage and matrix-vector work. That is practical for moderate dimensions but becomes expensive when n is very large.

L-BFGS avoids storing the full dense matrix. It keeps a limited history of recent s and y pairs and applies an implicit inverse-Hessian action through a compact recursion. Its memory is roughly proportional to n times the retained history length, making it a standard choice for large unconstrained problems.

L-BFGS-B adds bound handling. It should not be described as “BFGS but smaller” without also recognising that the solver’s feasible-set logic and stopping measures differ when bounds are present.

12. Gradient Quality Can Decide Everything

BFGS learns curvature from gradient differences. If gradients are wrong, inconsistent or dominated by noise, the curvature model learns the wrong world. Analytic gradients, automatic differentiation and carefully controlled numerical differentiation can therefore change solver reliability dramatically.

A valuable debugging check compares supplied gradients against finite-difference or complex-step estimates on representative points. Do not wait until the optimiser fails to test whether the derivative routine is actually differentiating the same objective.

13. Stopping Is a Measurement Problem

Common stopping signals include a small gradient norm, a small parameter step, a small objective change, iteration/evaluation budgets and application-specific tolerances. None alone means “the global optimum has been proven.”

A flat region can produce a small gradient far from the desired solution. A tiny step can reflect line-search trouble. A stable objective can coexist with materially changing parameters. Professional reporting records why the solver stopped and whether the final point satisfies the requirements of the actual application.

14. Failure Modes Strong Learners Should Test

Wrong gradient: often more damaging than a mediocre initial point.
Poor scaling: variables with radically different natural scales can make the geometry difficult.
Nonpositive or tiny yᵀs: signals an unsafe curvature update.
Noisy objective/gradient: secant information may become unreliable.
Line-search failure: can block progress even if the direction formula is correct.
Nonconvex objectives: convergence to a stationary point does not imply a global minimum.
Excessive dimension: dense BFGS memory becomes the bottleneck.
NaN/Inf values: require explicit detection and recovery policy.
Badly chosen finite-difference steps: can amplify cancellation or truncation error.

15. Compare Methods by Their Information Budget

Gradient descent uses first-order information with little memory. Newton methods use explicit second-order information and can take powerful geometry-aware steps when Hessians are affordable. BFGS infers curvature from first-order observations. Nelder–Mead uses function values and a simplex rather than gradients.

The correct choice depends on dimension, smoothness, derivative availability, noise, constraints, cost per function evaluation and the accuracy required. Algorithm choice is part of modelling the optimisation environment.

16. Learn BFGS by Fading the Scaffolds

For a learner, the full matrix formula arrives too early if s, y and the secant condition are not yet meaningful. A more productive sequence is to predict how the gradient changes on a quadratic, run a worked iteration, investigate why yᵀs must be positive, modify the initial scaling or line search, and only then reconstruct the update from its required properties.

Faded worked examples can remove one stage at a time: first compute s and y; later compute ρ; later form the update; finally choose diagnostics for a failing optimisation. This preserves the path from symbols to decisions.

17. Beginner-to-Professional Progression

Beginner: understand gradients, curvature and why a badly scaled valley causes zig-zagging.
Intermediate: calculate s, y, yᵀs and one inverse-Hessian update by hand on a two-variable problem.
Advanced: implement BFGS with a line search, verify gradients, derive the descent property and compare convergence against gradient descent.
Professional: choose BFGS versus L-BFGS/L-BFGS-B, safeguard curvature updates, diagnose termination, scale variables, benchmark realistic workloads and validate the solution against domain requirements.

18. Practice Problems

For f(x,y)=100x²+y², compute the gradient at (1,1) and explain the anisotropic curvature.
Given s and y vectors, calculate yᵀs and decide whether an undamped BFGS update is safe.
Implement a small backtracking line search and compare it with a Wolfe-condition implementation.
Introduce a deliberate gradient bug and design checks that detect it before optimisation.
Compare BFGS and Nelder–Mead on the Rosenbrock function using function evaluations as well as iterations.
Measure memory use as dimension grows and identify when full BFGS becomes impractical.
Explain how L-BFGS can apply an inverse-Hessian approximation without materialising an n×n matrix.

19. Sources and Further Reading

Final idea: BFGS is best understood as disciplined learning from movement. Each accepted step leaves a receipt: how far the parameters moved and how the gradient changed. The algorithm turns those receipts into a better geometric model. Professional optimisation depends on judging when that evidence is trustworthy, when it is not, and what the solver’s final state actually proves.

Small Group Tutorials