When a trial is expensive
Bayesian optimisation is a budget of trials, not a search of space
Bound → model → pick → stop
Each trial is a full run, so you do not walk the space. After a few points you hold a cheap belief about the rest, and the next run goes where that belief is still worth paying for.
BOUNDKnobs — legal range, grain, metric named onceMODELSurrogate — a posterior over the surface, not the truthACQNext trial — pick where the budget is spentSTOPKill — budget, delta, or a human ApproveBelief → next spend
After a few expensive trials you fit a cheap surrogate, usually a Gaussian process. That surrogate is Bayesian inference: a mean for what you expect, a band for where you are still unsure. An acquisition function turns that posterior into one next point, which you run and then fold back into the belief until STOP. Inference that never buys a next trial — a PD update, an A/B posterior, a lifetime ECL — is not this card.
Problem → use → example
| If the real problem is… | Use | Example case |
|---|---|---|
| The objective runs in seconds on a laptop | Grid / random | Notebook fit on a small sample: exhaust the grid. A Gaussian process here is theatre. |
| You have traffic and one treatment to judge | A/B (fixed n) | One offer vs control, sample size named up front. Don’t re-pick the treatment every hour and call it optimisation. |
| You will peek before the planned n | Sequential test | Early stop on a lift that already crossed a boundary. Peeking without a sequential rule is just cheating the p-value. |
| Each trial is a full PD / scorecard backtest | Bayesian opt | Cut-off search when a portfolio run is the cost. Don’t grid forty cut-offs if last week’s surface already tells you where to look. Analytics card |
| ECL lifetime run is the expensive step | Sequential model | Parameter / scenario knobs on an auto-loan lifetime forecast: freeze the grain, spend the next run where uncertainty still moves the decision. Credit card |
| Each trial is a full agent eval pack | Bayesian opt | Prompt / tool / loop knobs when 20 golden tickets is the cost of one point. Grid the cheap smoke test; model the expensive pack. AI card |
| Many knobs, some categorical, GP is a pain | TPE / Optuna | Mixed search (flags, depths, thresholds) where a Gaussian process wants a smooth box. TPE is a lane, not a different job. |
| The next try is a freeze window | Adaptive experiment | One middleware cutover slot this month: pick the next flag mix from what already held, not a 12-cell grid. Delivery card |
| Policy is already written; no search left | Named rule | Credit-engine threshold that compliance already locked. Log it. Don’t “optimise” a bound you are not allowed to move. |
| You still can’t name the metric | Stop. Ask first | No ASK, no search. Optimising an unofficial number just burns the budget on the wrong hill. |
Default order: name the decision and metric → bound the knobs (and what’s illegal) → price one trial → grid if cheap → surrogate + acquisition if expensive → stop when the budget or the delta dies.
Tool picker
Grid / randomCheap objective; exhaust it
Sequential A/BTraffic + a peek rule
scikit-optimizeGP search on a NumPy objective
BoTorchSurrogate + acquisition in PyTorch
AxExperiment loop if you are not writing a paper
OptunaTPE / mixed knobs; GP is optional
scipy.optimizeSmooth, cheap, differentiable-enough
Bound vs search
Bound · legal
Knobs, grain, and metric owned before the first trial. Illegal values are not “exploration.”
Search · spent
The next point costs a run. Acquisition proposes; it does not publish.
Inference · MODEL
The surrogate is a posterior over the objective. Updating a PD or an A/B posterior with no next-knob spend is inference, not optimisation.
Ladder + gates
- Named decision + metric
- Legal bounds (and what’s forbidden)
- Price of one trial
- Grid / random while cheap
- Surrogate + acquisition when it isn’t
- Stop: budget, delta, or Approve
Kill switch
If you can’t name the bound, the cost of one trial, and how you stop, you don’t start a search.
Anti-patterns
- Bayesian opt on a two-second objective
- No stop rule, just more trials
- Optimising a metric that isn’t the decision
- Proposing illegal cut-offs as “exploration”
- AutoML as the owner of the call
- Warm-start from a different grain
- Peeking an A/B with no sequential boundary
- Library shopping (BoTorch vs Ax vs Optuna) before the budget is named
Always on
SecurityWho can run a trial against live books
GovernanceNamed owner of metric, bounds, and stop
ObservabilityEvery trial logged with grain and model id
EvalsHold-out check before the “best” point ships
Human ApproveAcquisition proposes; a person publishes