Inference

Inference is the process of using a trained model to generate outputs from new inputs. In security terms, it matters because inference cost can vary dramatically based on prompt size, tool calls, and model selection, which makes it a target for abuse and cost escalation.

Expanded Definition

Inference is the runtime phase where a trained model applies learned patterns to fresh input and returns a prediction, completion, classification, or action. In AI security, the term is not limited to text generation: it also covers policy decisions, content moderation, tool selection, and any model-driven step that produces a result after training is complete. That distinction matters because inference is usually where operational risk becomes visible, including prompt injection, excessive tool invocation, and exposure of sensitive context. NHI Management Group treats inference as a security boundary because the model may handle secrets, identities, and delegated authority during execution.

Definitions vary across vendors when inference is bundled together with orchestration, retrieval, or agent execution, but the core idea remains the same: a deployed model is being used on live data, not being trained. For governance and control mapping, the NIST Cybersecurity Framework 2.0 is useful for framing the operational protections around this phase. The most common misapplication is treating inference as harmless read-only activity, which occurs when organisations ignore that live prompts can trigger tool use, data exposure, or costly model calls.

Examples and Use Cases

Implementing inference rigorously often introduces latency and cost controls, requiring organisations to weigh response quality against throughput, spend, and security inspection overhead.

• A customer support assistant generates a response from a live prompt and retrieves policy documents during inference, which creates a need to monitor both model output and retrieval permissions.
• An agentic workflow uses a model to decide whether to open a ticket, query an internal API, or escalate to a human, making inference part of an execution chain rather than a simple prediction step.
• A fraud detection model scores a transaction in real time, where inference must be fast enough for business use but still constrained by data minimisation and audit requirements.
• A code assistant suggests a patch from repository context, and the inference layer must prevent leakage of secrets, credentials, or restricted source fragments included in the prompt.
• A security analyst uses a model to classify alerts, where inference quality affects triage speed, but overreliance can hide weak signals or reinforce bad assumptions.

For organisations defining controls around live AI use, the operational question is whether the inference path can be observed, bounded, and revoked with the same discipline as any other privileged runtime. The NIST Cybersecurity Framework 2.0 helps teams translate that question into governance, detection, and response expectations. In practice, the strongest inference examples are those where an output directly changes state, privileges, or downstream trust.

Why It Matters for Security Teams

Inference is where model behaviour meets production reality, so weaknesses here can become security incidents even when training was clean. If prompts can be manipulated, inference may follow attacker-controlled instructions. If tool access is excessive, a model can become an unintended broker of secrets, API calls, or identity actions. If cost controls are absent, repeated or oversized inference requests can create an economic denial of service. For teams managing NHI and agentic AI, inference is especially important because the model may operate with delegated credentials, service tokens, or workflow authority, which turns output generation into a control problem as much as an AI problem.

Security teams should treat inference logs, prompt boundaries, tool permissions, and output validation as first-class controls, not afterthoughts. That is where identity and AI security intersect most sharply: the question is not only what the model said, but what it was allowed to do when it said it. Organisations typically encounter inference risk only after an abuse event, a spend spike, or an unsafe action, at which point inference becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• How should security teams secure internet-facing local AI inference servers?
• How should security teams govern in-house AI inference workloads?
• How should teams govern identity and access for AI inference platforms?
• Why do inference stacks create new NHI risk?