How can security teams detect model extraction?

What model extraction looks like in practice

model extraction is usually visible as a pattern, not a single request. Attackers probe the system with many inputs, vary prompts in small steps, and try to learn how outputs change at decision boundaries. A defender should think in terms of accumulated behaviour across time, accounts, and endpoints, not just individual queries. That is why anomaly detection is more useful than static signature checks alone.

One useful reference point for defenders is the broader monitoring and response model in NIST Cybersecurity Framework 2.0, which helps connect unusual request patterns to detection and response workflows. For teams operating API-based services, the request pattern itself is the signal: repeated boundary testing, many near-duplicate prompts, and requests that appear designed to map model behaviour rather than use the service normally.

Which signals are most worth hunting for

The strongest clues are behavioural. High-volume API calls from a small set of accounts, scripted timing, and systematic changes to input strings often indicate automated collection rather than legitimate user activity. Near-duplicate requests are especially important when they are arranged to compare output differences across a narrow range of prompts, because that is a common way to infer decision rules or recreate a model surrogate.

Defenders should also treat request diversity as a clue. Normal customers usually vary in topic and pacing, while extraction attempts often concentrate on one model, one endpoint, or one output format. If a workflow shows lots of similar queries without corresponding product usage, that gap deserves review. For teams building detections, the MITRE ATT&CK Enterprise Matrix is useful for thinking about adversary behaviour as a chain of actions, even when the end goal is model replication rather than classic intrusion.

How to distinguish extraction from normal heavy usage

Volume alone is not enough. A genuine enterprise integration may also generate many API calls, so the key is whether the traffic looks exploratory, repetitive, and boundary-focused. Extraction attempts usually show low semantic variety, little sign of end-user business context, and a steady effort to observe how output changes when inputs are nudged. That is different from normal bursts caused by real usage, testing, or batch processing.

Good detection therefore combines request-rate analytics with content-aware review. Look for the same client asking for a broad sweep of similar questions, repeated attempts to elicit probability-like behaviour, and request sequences that seem designed to sample the model rather than consume a service outcome. If the service exposes a public API, controls such as authN, authZ, and resource throttling should be paired with review of request distribution, not used as the only defense. Guidance in the NIST AI Risk Management Framework is helpful here because it reinforces the need to monitor how AI systems are used, not only how they are built.

Risk and Threat Considerations

Model extraction matters because it can expose expensive intellectual property, erode competitive advantage, and give attackers a cheaper path to imitate your system’s behaviour. It can also become a precursor to follow-on abuse, including adversarial testing at scale, prompt crafting for evasion, or cloning a safer-looking surrogate that preserves the original model’s weaknesses.

Failure mechanism: The attacker sends enough structured queries to approximate internal decision logic, especially around boundaries where the model is most informative. Repeated probing can succeed even when each individual request looks harmless.

Impact: Once the behaviour is replicated well enough, the attacker can reduce dependence on the original service, study its responses offline, and potentially use the copied behaviour to support fraud, bypass controls, or target weaknesses in the original system.

Practitioner Guidance

What to prioritise: Start with telemetry that ties requests to identity, rate, and content similarity. If you cannot reliably answer who is querying, how fast, and with what repetition pattern, extraction attempts will be difficult to separate from legitimate usage.

What to verify: Confirm that quotas are actually enforced per meaningful unit, not just per shared API key or tenant. Also verify that your anomaly rules can detect many small requests over time, because extraction campaigns often stay below obvious burst thresholds.

Practitioner takeaway: The best extraction detections combine behavioural similarity, rate patterns, and access context, because an attacker trying to recreate a model usually leaves a recognizable sampling pattern before they leave a technical exploit.

Related resources from NHI Mgmt Group

• Why are NHIs a critical concern for security teams?
• What steps should security teams take to prevent Shadow AI risks?
• Why is the abuse of NHIs a priority for security teams?
• What is the difference between SAST and DAST for security teams?