How do attackers use black-box optimisation?

How black-box optimisation works against a model

Black-box optimisation is the attacker’s way of learning from outputs when internals are hidden. Instead of inspecting weights, prompts, or training data, they treat the model like a system to probe, score, and steer. The practical goal is to discover inputs that reliably change behaviour, whether that means bypassing filters, extracting information, or coaxing unsafe content.

That makes the attack iterative rather than one-shot. The attacker submits many candidate queries, watches how the system responds, and keeps the variations that move the output in the desired direction. In AI threat work, this is the same basic search problem described in MITRE ATLAS adversarial AI threat matrix and in NHIMG’s MITRE ATLAS Guide, where black-box optimisation appears as a search and evasion technique rather than a model-inspection technique.

Because the attacker only sees outputs, they rely on feedback patterns: changes in confidence, refusal rate, verbosity, token choice, or side effects in downstream tools. Even small differences can be enough to guide the next query. That is why the method scales from simple prompt probing to more advanced extraction, adversarial example generation, and tool-abuse workflows where the model’s own responses reveal what to try next.

Why repeated probing is the attacker’s advantage

Black-box optimisation works best when the defender allows cheap, repeated testing. Each query can be harmless in isolation, but together they create a signal about decision boundaries, safety thresholds, or hidden policy behaviour. Attackers can use that signal to approximate the model’s response surface and then search for inputs that sit just across the boundary they want to cross.

This is why rate limits, anomaly detection, and response consistency matter. If the system answers every probe in a slightly different way, the attacker gets more information to optimise against. If the system is stable, slow to reveal useful gradients, and hard to automate at scale, the search becomes costlier and less reliable. The defender’s objective is not to make every output identical, but to reduce the value of each probe and the speed of the optimisation loop.

One useful way to think about the risk is that the attacker is trying to turn uncertainty into leverage. The more they can learn from the model’s behaviour, the better they can steer future inputs. That is why query campaigns, prompt sweeps, and systematic parameter variation are more concerning than isolated unusual prompts.

What defenders should look for in practice

Defenders should focus on the pattern, not just the single prompt. Black-box optimisation usually leaves traces such as repeated near-duplicate requests, wide prompt variation from the same source, alternating benign and malicious wording, or bursts of traffic that appear designed to measure the model’s response boundaries. Those patterns are often more important than any one suspicious input.

They should also watch for signs that the model is being used as an oracle for discovery. If requests are gradually converging on a specific behaviour, output format, or refusal edge case, the traffic may be part of an optimisation loop. The defensive response is to correlate query volume, source reputation, session behaviour, and response anomalies, then decide whether to throttle, challenge, block, or sandbox the caller.

For a broader view of adversarial model behaviour, The State of NHI & AI Agent Breach Report 2026 is useful because it connects attack paths to the operational assets attackers actually target, including credentials, tokens, and service access. That helps defenders distinguish ordinary experimentation from repeated probing aimed at exploiting a model or the systems around it.

Practitioner Guidance

What to prioritise: Treat repeated low-and-slow querying, prompt mutation, and response harvesting as the key indicators of black-box optimisation. The most important question is whether the caller is learning from the model faster than your controls are making that learning expensive.

What to verify: Check whether your API gateway, model proxy, or application logs can tie probes to a stable identity, session, or source. If you cannot reconstruct the sequence of related requests, you will struggle to distinguish normal use from optimisation-driven abuse.

Decision rule: If a caller is changing inputs in a systematic way to elicit boundary behaviour, reduce rate limits and add challenge friction before you wait for confirmed harm. Once the attacker has a reliable feedback loop, their next queries become more effective.

Practitioner takeaway: Black-box optimisation is dangerous because it converts ordinary model interaction into a measurement campaign, so the best defence is to make probing noisy, costly, and easy to spot.

Related resources from NHI Mgmt Group

• How do attackers turn a supply-chain incident into wider NHI compromise?
• How do attackers operationalise stolen OAuth tokens at scale?
• Why do attackers often check model availability before trying to generate content?
• How do attackers turn stolen npm secrets into broader compromise?