Artificial Intelligence and Machine Learning (AI/ML)

AI/ML as a security and governance subject

Artificial intelligence and machine learning are not just technical implementation choices, they are decision systems that can change how organisations handle data, model outputs, and operational control. That means the security conversation extends beyond accuracy to provenance, access, drift, and whether the system is being used in ways the organisation did not intend.

In practice, AI/ML becomes a governance subject when teams need to decide what data may train a model, who may modify it, how outputs are approved, and what evidence proves a model is behaving as expected. Those questions are central to risk management, not side effects of the technology.

How AI/ML systems differ from fixed-rule systems

A fixed-rule system produces outputs from explicit logic, while AI/ML systems infer patterns from examples and may generalise beyond the exact data they were trained on. That flexibility is the main benefit, but it also means behaviour can change when the training set changes, the prompt changes, or the surrounding environment changes.

This difference matters because security controls that work well for deterministic software, such as code review alone, do not fully address model behaviour. Organisations must think about the model as an evolving asset with its own lifecycle, dependencies, and failure modes.

For that reason, AI/ML security often overlaps with model risk, data governance, and software security at the same time. The model can be technically sound and still produce unsafe or unreliable results if the underlying data, access paths, or deployment chain are weak.

Core security properties of AI/ML

The most important security properties are provenance, integrity, confidentiality, and controlled use. Provenance asks where the model, training data, and artefacts came from. Integrity asks whether they were altered. Confidentiality asks who can see training data, model parameters, prompts, and outputs. Controlled use asks whether the system is being used only for approved purposes.

Model outputs also deserve special attention because they can influence decisions, trigger downstream automation, or reveal information about the data they were trained on. If outputs are reused without validation, the model effectively becomes part of a wider trust chain.

AI/ML systems also create a lifecycle problem: they may be retrained, tuned, or updated after deployment. That means the security posture of the system can change over time, even when the application code has not changed.

Common failure modes and operational consequences

AI/ML systems can fail through poor data quality, poisoned training data, insecure model supply chains, excessive access to training resources, or misuse of inference outputs. They can also fail quietly through model drift, where the system remains available but becomes less reliable as real-world patterns change.

Another common issue is that teams treat the model as a single component, when in reality it is a chain of data, code, infrastructure, and human decisions. Weakness in any one part can undermine the whole system, especially when outputs are used for security decisions, eligibility decisions, or automated actions.

Because of that, AI/ML should be evaluated as both a technical system and a governance object. The practical question is not only “does it work?” but also “can we trust it, explain it, and constrain it well enough for the intended use?”

Risk and Threat Considerations

AI/ML creates meaningful exposure when attackers can poison training data, steal model artefacts, evade model detection, or abuse model outputs to influence downstream decisions. Even without a direct attacker, weak provenance or uncontrolled retraining can create the same business risk through bad or manipulated results.

Failure mechanism: Compromised data, artefacts, or inference paths can change model behaviour, expose sensitive training information, or let an attacker shape outputs without touching the underlying application logic.

Impact: The result can be incorrect decisions, disclosure of sensitive information, broken trust in automated workflows, and wider downstream compromise if model outputs drive security, operational, or financial actions.

Practitioner Guidance

Why practitioners should care: AI/ML systems need ownership across data, model, and deployment boundaries, because no single team usually controls the full risk surface. Security and governance teams should treat the model as a managed asset, not as an isolated feature.

What to watch for: Be alert to training data with unclear provenance, models that can be updated without review, and inference paths that expose sensitive information or enable unapproved downstream automation. Those are often the first signs that the control model is too loose.

Practitioner takeaway: The safest AI/ML programmes are the ones that can explain where the model came from, what it is allowed to do, and how its outputs are validated before they influence real decisions.

Related resources from NHI Mgmt Group

• Intelligence
• What do regulators expect from AI and machine learning risk models?
• How should teams govern AI workflows that span multiple machine learning platforms?
• How should security teams reduce adversarial machine learning risk in production AI systems?