AI/ML models are statistical systems trained on data to make predictions, generate content, or classify inputs. They apply learned parameters at runtime, so their behaviour depends on the training data, model design, and any fine-tuning or prompt inputs. Security teams treat the model itself, its weights, and its training data as part of the AI attack surface.
What AI/ML models are, and why they matter to security
AI/ML models are not just files or code, they are trained statistical systems whose behaviour reflects both design choices and the data used to train and tune them. That makes the model itself part of the attack surface, because a compromise can alter predictions, generated output, or classification decisions without visibly changing the surrounding application.
For security teams, the practical implication is that model integrity matters as much as software integrity. A model can be manipulated before deployment through poisoned training data, during deployment through tampered weights or unsafe updates, or at runtime through inputs crafted to steer outputs in unintended ways.
What changes at training time and inference time
The security posture of a model depends on where the control boundary sits. Training time introduces risks around data provenance, label integrity, and unintended memorisation, while inference time introduces risks around query abuse, leakage through outputs, and manipulation through adversarial inputs. The same model can therefore be safe in one stage and fragile in another.
Fine-tuning and prompt inputs can also change behaviour, which is why many organisations treat the base model, adapted weights, and surrounding instruction layer as related but distinct assets. That distinction matters when you decide what to sign, validate, restrict, or monitor.
NHIMG’s AI Model Security Guide is useful here because it separates model theft, inversion, adversarial examples, and inference API abuse into distinct control problems rather than treating them as one generic AI issue.
Common model failure modes and attack paths
Three broad failure modes recur across AI and ML systems. First, poisoning can distort the training set so the model learns the attacker’s intended behaviour. Second, extraction or inversion can reveal aspects of the trained model or the data it learned from. Third, adversarial inputs can cause the model to misclassify or emit unsafe output even when the underlying weights are intact.
These are security problems because they undermine trust in the model’s outputs, not just its availability. In regulated or high-impact use cases, a compromised model can create downstream legal, financial, operational, or safety consequences even when no traditional server breach has occurred.
MITRE ATLAS provides a strong threat-modeling lens for this class of behaviour, and NHIMG’s MITRE ATLAS Guide helps map AI-specific attack techniques, including model evasion, poisoning, and exfiltration paths.
How organisations govern model integrity and provenance
Good model governance starts with knowing what exists, where it came from, and whether it has been modified. That usually means maintaining an inventory of models, recording training-data lineage, tracking fine-tuning and deployment history, and applying integrity checks before a model is promoted into production.
Provenance controls become especially important when models move across teams, vendors, or deployment environments. If you cannot verify the source of the weights or the integrity of the pipeline that produced them, you cannot reliably trust the output of the model that follows.
For that reason, model signing and provenance controls are not just supply-chain hygiene, they are part of AI assurance. NHIMG’s Model Signing and OpenSSF Model Signing Guide explains how signing identities and verification gates help preserve model integrity across the release pipeline.
How security teams should think about model monitoring
Monitoring for AI/ML models needs to cover more than uptime. Teams should watch for drift in output quality, unusual query patterns, signs of extraction activity, and unexpected changes in model behaviour after retraining or tuning. The goal is to detect both silent degradation and active compromise.
Post-deployment review matters because model incidents can be subtle. A model may continue serving responses while gradually leaking information, becoming easier to evade, or losing the decision boundaries that made it reliable in the first place. That is why model review should include both security telemetry and periodic validation against expected behaviour.
NHIMG’s AI Model Risk Management Guide is a useful companion for organisations that need a lifecycle view of inventory, validation, and ongoing monitoring.
Risk and Threat Considerations
AI/ML models concentrate multiple kinds of exposure in one place: poisoned training data, tampered weights, sensitive memorisation, and runtime manipulation. Because the model influences decisions directly, a compromise can stay hidden longer than a conventional application fault and can affect many downstream users at once.
Failure mechanism: An attacker can corrupt training data, steal or alter model weights, or craft inputs that force the model into unreliable or unsafe behaviour. In some cases, they can also infer sensitive information from the model’s outputs or confidence patterns.
Impact: The result can be false decisions, data leakage, unsafe automation, loss of trust in model output, and costly post-incident retraining or rollback.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS addresses the attack and risk surface, while NIST SP 800-53 Rev 5, SLSA and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATLAS | Adversarial AI Threat Matrix | Covers AI/ML attack techniques, poisoning, evasion, and exfiltration patterns. |
| Recommendation — Map model threats to ATLAS techniques and hunt for poisoning, evasion, and extraction activity. | ||
| NIST SP 800-53 Rev 5 | SC-28 — Protection of Information at Rest | Supports protecting model weights and training data stored as sensitive AI assets. |
| AU-6 — Audit Review, Analysis, and Reporting | Supports review of model, inference, and access telemetry for compromise indicators. | |
| Recommendation — Apply SC-28 to protect stored model artefacts and training data from unauthorized access or tampering. Use AU-6 to review model activity logs for abuse, drift, and compromise indicators. | ||
| SLSA | SLSA — Supply-chain Levels for Software Artifacts | Model pipelines need provenance and integrity controls similar to software artifacts. |
| Recommendation — Adopt SLSA-style provenance controls for model build, training, and release pipelines. | ||
| OWASP ASVS | V11 — Cryptography | Model signing and verification rely on strong cryptographic assurance for integrity. |
| Recommendation — Use V11 controls to verify model signatures and protect integrity checks in the delivery pipeline. | ||
Practitioner Guidance
Why practitioners should care: AI/ML models should be governed as production assets with their own ownership, integrity checks, and validation criteria. Treating them as disposable artefacts creates blind spots around provenance, tuning history, and runtime abuse.
What to watch for: Pay close attention to unexpected output shifts after retraining, unexplained confidence changes, repeated probing behaviour, and any path that lets untrusted data or unreviewed updates reach the model.
Practitioner takeaway: The model is not just the thing that uses data, it is itself a security-sensitive dependency whose trustworthiness must be established, monitored, and revalidated over time.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on November 3, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org