How can security teams detect creation of a proxy AI model?

What proxy AI model creation looks like in telemetry

Proxy model creation is usually less visible as a single event than as a pattern. The strongest signal is repeated querying of a protected model at scale, followed by harvesting outputs to train a substitute model that imitates behaviour, style, or decision boundaries. Security teams should look for clustered request patterns, synthetic data generation workflows, and training activity that appears to depend on large volumes of model outputs rather than organic business data.

That pattern often shows up in adjacent systems first. Build pipelines, notebook jobs, data labeling queues, and batch training runs can reveal whether a team is converting external model responses into a local model asset. The key question is not whether a model exists, but whether its creation path is inconsistent with normal product development and whether the request and training cadence suggest imitation rather than legitimate experimentation.

For a broader view of the surrounding risk surface, the AI Security Posture Management (AI-SPM) Guide is useful because proxy-model detection depends on discovering model assets, AI pipelines, and unusual attack paths before they blend into routine AI operations.

Teams that already manage agent or model governance can also benefit from the AI Security Maturity Model, which helps frame whether detection coverage exists across inventory, monitoring, and response rather than only at the model endpoint.

Signals that indicate imitation rather than normal usage

High-volume, structured requests are one of the clearest behavioural cues. A proxy builder typically needs many examples of responses from the target model, so look for bursty access from the same actor, repeated prompts with small variations, and patterns that resemble systematic probing rather than ordinary user interaction. If the target is an API, the request shape may be unusually regular, with frequent retries, pagination-like sequences, or prompt families that span many related topics.

Downstream data movement matters just as much. Unusual exports from logs, stored completions, prompt archives, or response caches can indicate output harvesting. Likewise, new datasets that appear to have been assembled from model answers, especially where labels or fine-tuning corpora are derived from generated text, deserve scrutiny. Proxy creation often leaves a trail in data engineering systems before it becomes obvious in the model layer.

Threat hunters should pair behavioural review with adversary technique mapping. MITRE ATLAS is a good reference for understanding how model probing, output harvesting, and imitation fit into AI threat patterns, while the MITRE ATLAS Guide helps teams connect observed activity to AI-specific techniques and mitigations.

For teams that already investigate adversary behaviour in enterprise environments, the MITRE ATT&CK Enterprise Matrix remains useful for surrounding behaviours such as credential abuse, automation abuse, and lateral movement that may support the collection effort even when the proxy model itself is the main objective.

How to confirm and respond without blocking legitimate development

Confirmation should combine request telemetry, downstream training evidence, and ownership review. A legitimate model team can usually explain why a job exists, what data it uses, and why its training cadence matches an approved use case. A suspicious effort is harder to justify, especially if the data source is primarily model output, the prompt patterns are designed for broad behavioural imitation, or the project lacks a clear business sponsor.

When the pattern looks suspicious, tighten controls around the protected model first. Rate limits, output controls, and misuse monitoring should reduce the ability to harvest enough material for a convincing substitute. If the model is exposed through an API, review authentication, client identity, and quota enforcement. If the model is embedded in internal tooling, inspect whether the same actor is using approved interfaces in a way that effectively bypasses intended usage constraints.

The NIST Cybersecurity Framework 2.0 is a useful external anchor here because proxy-model detection spans identify, protect, detect, respond, and recover activities rather than a single control family.

For AI-specific governance and threat modelling, the MITRE ATLAS adversarial AI threat matrix is especially relevant when you need to separate benign experimentation from adversarial imitation and understand how model extraction or mimicry attempts tend to evolve.

Practitioner Guidance

What to prioritise: Start with the model interfaces that can be queried repeatedly, then inspect the downstream jobs that turn those outputs into training data. If the request pattern is normal but the training pipeline is strange, the compromise may be in data engineering rather than in the live model service.

What to verify: Confirm whether the actor has a legitimate development relationship to the model, whether the output volume matches an approved use case, and whether the training data lineage is documented. If you cannot trace the dataset back to an authorised business process, treat the activity as higher risk.

Common mistake: Teams often watch only for direct theft of weights or prompts and miss proxy creation because it can look like ordinary usage. The more important signal is a sequence of normal-looking outputs being converted into an unapproved imitation workflow.

Practitioner takeaway: Proxy model detection is a correlation problem, not a single-alert problem, so the best control point is the combination of repeated querying, output harvesting, and suspicious training lineage.

Related resources from NHI Mgmt Group

• How should security teams detect AI model abuse when attackers use legitimate AI services to blend into normal traffic?
• How should security teams detect and contain AI misuse when attackers abuse legitimate model APIs or cloud credentials?
• How can security teams detect discovery of AI model ontology?
• How can security teams detect discovery of an AI model family?