Use runtime testing, data-boundary checks, and threat modelling that explicitly covers prompt injection, context poisoning, and guardrail bypass. These features create behaviours that static code analysis alone cannot see, so the control set has to move from code correctness to application behaviour and data handling at execution time.
Securing AI and LLM Features Starts at Runtime, Not at the Code Review
AI and LLM features behave more like dynamic application components than static logic. The security question is therefore not only whether the code is correct, but whether the feature behaves safely when it sees hostile prompts, ambiguous context, or unexpected data. That shifts assurance toward execution-time testing, boundary enforcement, and adversarial validation.
Security teams should treat these features as behaviourally testable surfaces. A prompt can change the model’s effective instructions, retrieved content can contaminate context, and a downstream tool or connector can turn an innocuous response into an unsafe action. The practical goal is to prove that the feature remains bounded when the runtime environment is actively trying to bend it.
That is why runtime evaluation matters alongside traditional secure development practices. It is the same reason NIST AI Risk Management Framework emphasises governing AI risk across the full lifecycle, not just at build time. For application teams, the equivalent control question is whether the deployed feature still resists manipulation after deployment, especially when external inputs and internal context meet.
What to Test in the Application Path
The most important control areas are prompt injection resistance, context isolation, and guardrail robustness. Prompt injection tests should cover direct attacks in user input, indirect attacks carried in documents, emails, web pages, or retrieved records, and attempts to override system instructions or policy text. Context tests should verify that one user’s data cannot leak into another user’s session, memory, or retrieved results.
Guardrail testing should go beyond “does the filter block bad words” and ask whether the application can still be induced to reveal secrets, summarize restricted content, call dangerous tools, or ignore policy after a multi-step conversation. The feature is safe only if the control survives the exact sequence of messages, retrievals, and tool calls that real users and attackers can trigger.
A useful reference point is OWASP Agentic AI Top 10, because many application-layer failures show up as identity and privilege abuse, tool misuse, memory poisoning, or trust exploitation rather than as obvious code defects. Even when the feature is not a full autonomous agent, those patterns still help structure testing around where the runtime can be steered.
For organisations with richer AI stacks, NIST AI 600-1 GenAI Profile reinforces the need for pre-deployment testing, content provenance awareness, and incident handling for generative ai features. That makes it a good fit for teams that need to connect application testing to a broader governance and assurance process.
How Data Boundaries and Threat Modelling Keep the Feature Contained
Data-boundary checks are the control that stops an LLM feature from becoming a generic exfiltration layer. The application should know which inputs are untrusted, which outputs are safe to display, which retrieved sources are allowed, and which actions require explicit human approval. Without that separation, the model can blend sensitive and public data in ways that are hard to detect after the fact.
Threat modelling should therefore focus on how information crosses boundaries, not just on model quality. Ask where prompt text can be influenced, where retrieved data can be poisoned, where memory can retain unsafe content, and which connectors or APIs can convert model output into side effects. The key failure mode is not that the model “hallucinates”, but that the surrounding application trusts the model in places where it should have enforced a hard boundary.
That is why the security conversation belongs alongside AI-specific threat modelling resources such as CSA MAESTRO agentic AI threat modeling framework and the MITRE ATLAS adversarial AI threat matrix. Both help teams think in terms of attack paths, poisoning, manipulation, and control failure rather than treating AI behaviour as an opaque black box.
When the feature uses tools, retrieval, or shared memory, the boundary problem becomes a trust problem. The application must limit what the model can see, what it can retain, and what it can act on. If the model can reach privileged data or operational tools without explicit constraints, the risk is no longer theoretical, because the application itself has become the compromise path.
Risk and Threat Considerations
AI and LLM features expand the attack surface because they turn unstructured language into an input channel for policy bypass, data leakage, and unsafe tool use. The main risk is not only malicious prompting, but also indirect poisoning through content the application ingests and then trusts.
Failure mechanism: An attacker manipulates prompts, retrieved context, memory, or connectors so the feature reveals restricted data, ignores guardrails, or triggers a side effect that the surrounding application would normally block.
Impact: This can lead to data exposure, unauthorized actions, privilege misuse, or cross-user contamination, especially when the feature sits close to sensitive workflows or production systems.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GV.RR-01 — Roles and Responsibilities | AI app security needs clear ownership for runtime testing and boundary controls. |
| MAP-01 — Map Context | Threat modelling for LLM features depends on mapping inputs, context, and outputs. | |
| MEASURE-01 — Measure and Monitor | Runtime testing and guardrail checks require ongoing measurement in production-like conditions. | |
| Recommendation — Assign accountable owners for AI runtime testing and boundary enforcement. Map AI inputs, context sources, and outputs before release. Measure AI behavior continuously for boundary and guardrail failures. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | AI features often fail when the runtime can misuse privilege or act beyond intended authority. |
| ASI06 — Memory & Context Poisoning | Context poisoning and poisoned memory are central runtime threats to LLM features. | |
| ASI02 — Tool Misuse | Tool-using LLM features must be tested for unsafe or unintended actions. | |
| Recommendation — Constrain agent and application privileges to the minimum required. Isolate memory and validate context sources before reuse. Restrict tool access and require explicit approval for high-impact actions. | ||
| CSA MAESTRO | GRC — Governance, Risk and Compliance | AI feature security needs structured risk governance and testing discipline. |
| SEF — Security Evaluation and Testing | The question centers on runtime testing of AI and LLM behaviour. | |
| Recommendation — Define governance for AI risk reviews and runtime assurance. Execute adversarial tests against prompt, memory, and tool paths. | ||
| OWASP ASVS | V13 — Configuration | Application guardrails and boundary settings are configuration-dependent. |
| V15 — Secure Coding and Architecture | Secure AI features require architectural controls around data flow and trust boundaries. | |
| Recommendation — Harden application configuration for AI boundary enforcement. Design AI features with explicit trust boundaries and least-privilege flows. | ||
Practitioner Guidance
What to verify: Test the feature with hostile prompts, indirect injection, poisoned retrieval sources, and cross-session context leakage. The question is not whether the model answers, but whether it still respects data boundaries and action limits under pressure.
What good looks like: Unsafe inputs are contained, sensitive context is not reused outside its boundary, and any tool-using action is observable, approved, and limited to the smallest necessary scope. If a control only works in clean lab traffic, it is not yet a production control.
Practitioner takeaway: secure ai features by proving that the runtime cannot be steered past its intended boundaries, because the decisive failures happen at the point where input, context, and action meet.
Related resources from NHI Mgmt Group
- How should organisations govern embedded AI features inside SaaS apps?
- How should organisations automate EU AI Act governance for LLM applications across different risk categories?
- What breaks when organisations try to secure cloud native AI applications with siloed teams and point tools?
- How should healthcare organisations secure AI and LLM systems that process sensitive patient data?