Wait, What?
Sometimes the expensive part is not the optimization algorithm. It is asking the real world for one more answer.
Bayesian optimization is designed for situations where evaluating the objective is costly: running an experiment, training a large model, testing a physical design or calling an expensive simulation. Instead of sampling blindly, the algorithm builds a model of what it has learned so far, represents uncertainty about what remains unknown and chooses the next evaluation strategically.
This article owns the Bayesian-optimization learning job. The existing Numerical Optimisation article owns gradient-based optimisation and line search, the MCMC article owns posterior sampling methods, and the Krylov-Subspace article owns iterative linear-system solvers. Bayesian optimization has a different job: choosing expensive evaluations under uncertainty.
Quick Answer
Learn Bayesian optimization through the route expensive objective → initial design → surrogate model → prediction → uncertainty → exploration vs exploitation → acquisition function → optimize acquisition → evaluate real objective → update surrogate → repeat → noise → constraints → batch selection → multi-fidelity → high-dimensional limits → stopping → validation. A beginner should be able to explain why the algorithm sometimes samples an uncertain point that is not currently predicted to be best. A professional should be able to choose a surrogate and acquisition strategy, understand the assumptions behind Gaussian processes, handle noisy or constrained objectives, measure sample efficiency and recognize when dimensionality or objective cost makes another method preferable.
1. Begin With a Function You Are Not Allowed to See
Draw an unknown one-dimensional curve behind a sheet of paper. Let the learner request only six x-values. After each request, reveal the corresponding y-value. The goal is to find the maximum with as few queries as possible.
This physical exercise makes the central scarcity visible: the budget is not CPU time for arithmetic; it is the number of expensive observations.
2. Random Search Is the Baseline Bayesian Optimization Must Beat
Before introducing a probabilistic model, sample points uniformly at random. Record the best observed value after 1, 2, 5 and 10 evaluations. This produces a simple sample-efficiency baseline.
Grid search is another useful comparison, but it scales poorly with dimensionality because the number of grid points grows exponentially with the number of coordinates.
3. Bayesian Optimization Uses a Surrogate for the Expensive Objective
A surrogate is a cheap model fitted to observed input–output pairs. It predicts the objective at untested locations and, crucially, represents uncertainty about those predictions.
The current Ax introduction to Bayesian optimization describes the repeated cycle clearly: fit a surrogate, use predictive uncertainty to define an acquisition function, optimize that acquisition to choose a new evaluation and refit after observing the result.
4. A Gaussian Process Models a Distribution Over Functions
A Gaussian process does not merely predict one curve. Conditioned on observed data, it defines a posterior distribution over possible functions. At any candidate x, this gives a predictive mean and uncertainty.
For beginners, visualize the posterior mean as a central line and predictive uncertainty as a band. After a new sample, ask where the band should shrink and why.
5. The Kernel Encodes Assumptions About Similarity
A kernel says how strongly observations at two inputs should be related. Squared-exponential kernels encode very smooth behaviour; Matérn kernels allow rougher functions; periodic kernels encode repeating structure.
Changing the kernel changes what the surrogate considers plausible between observed points. Therefore kernel choice is not a cosmetic parameter—it is a modelling assumption.
6. Length Scale Controls How Quickly Correlation Decays
A large length scale makes distant inputs look similar and produces a smoother posterior. A small length scale lets the function vary rapidly. Automatic relevance determination can assign different length scales to different input dimensions.
Have learners fit the same observations with two length scales and predict how the next recommended point changes.
7. Predictive Mean Alone Would Be Greedy
If the algorithm always sampled the location with the highest predicted mean, it would exploit what it already believes and could miss an unexplored region containing a much better optimum.
Bayesian optimization therefore uses uncertainty deliberately. The next evaluation should usually depend on both predicted quality and how much remains unknown.
8. Acquisition Functions Turn Belief Into a Next-Point Decision
An acquisition function is cheap to evaluate and scores how useful it would be to sample each candidate point. The algorithm optimizes the acquisition function—not the expensive real objective—to decide where to query next.
The current BoTorch acquisition-function documentation includes classical Expected Improvement, Probability of Improvement and Upper Confidence Bound methods alongside Monte Carlo acquisition functions for more complex settings.
9. Probability of Improvement Is Simple but Can Become Myopic
Probability of Improvement asks how likely a candidate is to beat the current best by some threshold. It can prefer locations with a high chance of a tiny improvement while ignoring a lower-probability location with much larger upside.
10. Expected Improvement Values Both Probability and Magnitude
Expected Improvement asks how much improvement is expected above the current best, averaging over predictive uncertainty. A point can score well because it has a strong predicted mean, large uncertainty or both.
This makes EI a particularly good teaching tool for exploration–exploitation trade-offs.
11. Upper Confidence Bound Uses an Explicit Exploration Weight
A UCB-style acquisition combines predictive mean with a multiple of predictive standard deviation. Increasing the uncertainty weight makes the search more exploratory; reducing it makes the search greedier.
Let learners vary that weight on the same posterior and predict which region will be selected next.
12. The Acquisition Function Has Its Own Optimization Problem
Once an acquisition function is defined, the algorithm must maximize it. This inner problem is usually much cheaper than evaluating the real objective, so multistart gradient optimization, random restarts or other numerical methods can be used aggressively.
This is a useful distinction: Bayesian optimization contains an optimization problem inside another optimization problem, but only the outer objective carries the expensive evaluation cost.
13. Initial Design Matters Before the Surrogate Knows Anything
A surrogate needs initial observations. Random designs, Latin hypercube sampling and low-discrepancy sequences can provide broad early coverage. Too few initial points may create unstable hyperparameter estimates; too many waste the adaptive advantage of Bayesian optimization.
14. Observation Noise Changes the Meaning of “Best So Far”
With noisy evaluations, the highest observed value may simply be a lucky draw. The surrogate should model observation noise, and acquisition functions may need noisy variants that reason about latent objective values rather than trusting the raw maximum.
This is why repeated measurements and uncertainty calibration matter when optimizing physical or stochastic experiments.
15. Constraints Turn “Best” Into “Best Feasible”
Real optimization often includes constraints: temperature must stay below a safety threshold, memory must fit a device, or accuracy must exceed a minimum. Constrained Bayesian optimization models objective and feasibility together so the acquisition favors promising points likely to satisfy constraints.
16. Batch Bayesian Optimization Selects Several Points at Once
If four experiments can run in parallel, choosing them independently may waste budget by selecting nearly identical points. Batch acquisition functions reason jointly about the value of a set of candidates.
BoTorch’s Monte Carlo framework supports q-batch acquisition functions specifically for this kind of parallel decision-making.
17. Multi-Fidelity Optimization Uses Cheap Imperfect Signals
Sometimes a low-resolution simulation, smaller dataset or shorter training run provides a cheap approximation to the expensive objective. Multi-fidelity methods decide both where to evaluate and at what fidelity.
The key question becomes value of information per unit cost, not merely predicted improvement.
18. Categorical and Conditional Spaces Need Special Treatment
Hyperparameter spaces often contain choices such as optimizer type, architecture family or activation function. Some parameters only exist when another choice is active. Standard continuous kernels may not represent these spaces naturally.
Professionals match the model and search representation to the topology of the design space rather than forcing every variable into a Euclidean vector.
19. High Dimensionality Is a Genuine Weakness
Traditional Gaussian-process Bayesian optimization is strongest in low-to-moderate dimensions. As dimensions grow, data become sparse, kernel hyperparameters become harder to estimate and acquisition optimization becomes difficult.
Structured kernels, trust-region methods, additive models, embeddings or random-search hybrids can help, but Bayesian optimization should not be treated as a universal replacement for other optimizers.
20. Sample Efficiency and Wall-Clock Efficiency Are Different
An algorithm may reach a good objective value in fewer real evaluations but spend much more compute fitting the surrogate. If the real evaluation costs hours, that overhead is trivial. If the objective costs milliseconds, Bayesian optimization may be unnecessarily elaborate.
Always compare total decision cost, not only number of evaluations.
21. Regret Provides a Sequential View of Search Quality
Simple regret measures how far the best point found so far is from the true optimum. Cumulative regret also penalizes poor decisions made along the way. Which measure matters depends on whether only the final configuration is valuable or each intermediate decision has consequences.
22. Stopping Is a Decision, Not an Afterthought
Stop when the evaluation budget is exhausted, expected improvement falls below a meaningful threshold, uncertainty around the incumbent is sufficiently small, or the value of another experiment is lower than its cost.
A fixed “100 iterations” rule is convenient but may have no relationship to the economics of the real experiment.
23. Validate the Recommended Configuration Outside the Search Loop
After optimization, rerun the selected configuration under fresh randomness, repeated trials or independent measurements. The optimizer searched specifically for unusually good observations, so selection can amplify noise.
Separate optimization evidence from confirmation evidence.
24. Common Learning Failure States
- Using Bayesian optimization when evaluations are already cheap.
- Learning only the Gaussian-process mean and ignoring uncertainty.
- Treating the kernel as an implementation detail.
- Using predicted mean as the next-point rule and calling it Bayesian optimization.
- Confusing Probability of Improvement with Expected Improvement.
- Ignoring observation noise.
- Optimizing the acquisition function poorly and blaming the surrogate.
- Using high-dimensional spaces without questioning sample efficiency.
- Reporting best observed value without a random-search baseline.
- Failing to validate the final selected point independently.
25. A Beginner-to-Professional Learning Ladder
- Level 1: choose six points to locate the maximum of an unknown 1D curve.
- Level 2: compare grid search and random search.
- Level 3: interpret a Gaussian-process mean and uncertainty band.
- Level 4: compute or visualize PI, EI and UCB on a simple posterior.
- Level 5: implement the sequential fit–acquire–evaluate loop.
- Level 6: optimize a noisy black-box function with a realistic baseline.
- Level 7: handle constraints and repeated measurements.
- Level 8: compare batch and sequential acquisition.
- Level 9: test multi-fidelity or conditional search spaces.
- Level 10: evaluate sample efficiency, compute overhead, calibration and final confirmation under deployment constraints.
26. Teach Uncertainty Before Library Syntax
Give learners three observed points and three candidate next points. Provide a predicted mean and standard deviation for each. Ask which point a greedy strategy chooses, which an uncertainty-seeking strategy chooses and which EI might prefer. Only then introduce code.
27. PRIMM Works Well for Sequential Optimizers
The current Raspberry Pi Foundation PRIMM course structures learning as Predict, Run, Investigate, Modify and Make. For Bayesian optimization, learners can predict the next acquisition point, run one iteration, investigate why uncertainty changed, modify the kernel or acquisition rule, then build a new black-box experiment.
The CS2023 Computer Science Foundations curriculum identifies algorithms, data structures, complexity and problem solving as essential application-level knowledge. Bayesian optimization adds a useful professional extension: algorithmic decisions can also be constrained by the cost of obtaining evidence.
28. Compare Against Strong Non-Bayesian Baselines
Random search, quasi-random search, evolutionary methods and gradient-based optimization should remain visible alternatives. A world-class experiment asks not whether Bayesian optimization improved the objective, but whether it improved it more efficiently than methods appropriate to the same search space and cost model.
29. Professional Direction
Advanced study includes noisy expected improvement, knowledge gradient, entropy search, q-batch acquisitions, Thompson sampling, trust-region Bayesian optimization, multi-objective BO, multi-fidelity BO, constrained BO, preference-based BO, contextual BO, neural and deep-kernel surrogates, sparse Gaussian processes and decision-theoretic value-of-information methods.
Algorithm-learning rule: never ask only which point the model predicts is best. Ask how uncertain that prediction is, what another evaluation would teach, how expensive the evidence is, what assumptions the surrogate makes, and whether the final recommendation survives an independent confirmation run.
eduKate Sengkang 