TL;DR: LLM risk is not just prompt injection but a full input-output governance problem, where malicious prompts, secret exposure, and unsafe responses can all bypass basic filters if they are not inspected at runtime, according to AccuKnox. The practical implication is that enterprise AI security now depends on dual-layer policy enforcement, auditability, and control over what the model sees and emits.
At a glance
What this is: AccuKnox argues that securing LLMs requires filtering both prompts and responses, not just blocking bad input, because runtime inspection is needed to stop secret leaks, unsafe outputs and prompt injection.
Why it matters: For IAM and security teams, the key issue is that AI systems need governed input, output and traceability controls that behave more like policy enforcement than static content filtering.
Context
LLM security fails when teams treat the model as a single control point instead of a conversation with two attack surfaces. Input can be weaponised through prompt injection, secret leakage or abusive requests, while output can expose confidential data, generate insecure code or return harmful advice.
That matters for identity and governance work because LLMs increasingly sit inside enterprise workflows that already depend on access control, auditability and policy enforcement. The practical problem is not whether the model is clever enough, but whether the surrounding controls can inspect what is sent to it and what comes back out.
Key questions
Q: How should security teams secure LLM system prompts in production applications?
A: Security teams should treat system prompts as governed runtime assets, not informal configuration. That means version control, change approval, least-privilege editing, and security review before release. Prompts should be tested for injection, leakage, and unsafe instruction handling, because prompt text can change behavior without changing code. The control objective is to keep model behavior within approved boundaries.
Q: Why do prompt filters alone fail to protect LLM applications?
A: Prompt filters only inspect one visible part of the system, while the real attack path often enters through retrieved documents, tool responses, uploads, or hidden context. If an instruction is concealed or arrives through another channel, filtering the prompt does not stop the model from acting on it. Protection has to follow the whole context path.
Q: What are the signs that LLM response controls are not working?
A: Warning signs include confidential text appearing in completions, insecure code reaching developers, blocked queries with no traceable reason and repeated policy violations tied to the same workflow. If teams cannot explain which rule fired or why a response passed, the control is too weak to support governance or audit needs.
Q: What should teams do when an LLM policy blocks a prompt or response?
A: Teams should preserve the original prompt, the model output if available, the policy decision and the session context, then review whether the block reflects a true security issue or an overbroad rule. That evidence supports containment, tuning and compliance reporting without relying on guesswork.
Technical breakdown
Why prompt policies are only half the control plane
Prompt policies sit in front of the model and inspect user input before it is processed. They are designed to detect patterns such as embedded secrets, injection attempts and abusive prompts, then block or sanitise them before the model can act on them. That helps reduce direct abuse, but it does not address what the model later emits. In practice, prompt control is an ingress filter, not a complete governance model. Enterprises that stop there still leave themselves exposed to data leakage, harmful completions and insecure code generation downstream.
Practical implication: treat prompt filtering as an entry control and pair it with output inspection for any production LLM workflow.
How response policies reduce model output risk
Response policies inspect the model’s output before a user sees it. This is important because an LLM may reveal internal content from its training context, produce vulnerable code or generate unsafe guidance even when the original prompt looked harmless. Response control is therefore a egress and quality layer, not just a content moderation feature. In governance terms, it creates a second checkpoint that can stop disclosure after inference has already happened. That is especially relevant where models are embedded in developer tooling, support workflows or regulated environments.
Practical implication: apply response scanning wherever LLM output can influence users, systems or code repositories.
Why runtime traceability matters in AI governance
The article’s dashboard and session tracking point to a larger control issue: without traceability, teams cannot explain why a prompt was blocked, what policy fired or whether a response was suppressed for the right reason. In LLM operations, that evidence becomes part of auditability, incident investigation and policy tuning. A session-linked trace also helps connect input and output events into a single governance record. For security teams, that shifts LLM control from a black-box moderation problem to an operational control plane with reviewable decisions.
Practical implication: require session-level logging and policy-level traces so AI security decisions are auditable and adjustable.
Threat narrative
Attacker objective: The attacker aims to coerce the LLM into leaking information, producing harmful output or executing unintended behaviour that weakens enterprise controls.
- Entry occurs when a malicious or careless user submits a prompt containing injection text, hidden instructions or embedded credentials to the LLM application.
- Credential or data exposure follows when the model or surrounding workflow processes that input and may surface secrets, internal context or other sensitive material.
- Impact occurs when the model returns unsafe output, leaks confidential content or generates insecure code that can propagate into business workflows.
Breaches seen in the wild
- ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.
- Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Input-only filtering is an incomplete control model for enterprise LLMs. The article correctly shows that prompt injection and secret leakage are only one side of the problem. In operational terms, a model that is protected only at ingress can still become a disclosure channel at egress, especially where outputs are copied into code, tickets or customer-facing workflows. Practitioners should treat LLM security as conversation governance, not text moderation.
AI security now needs policy enforcement at the point of use, not after the fact. The strongest control point is runtime, where the system can inspect prompts, outputs and session context before business impact occurs. That aligns with the broader move in AI governance toward measurable controls rather than trust in model behaviour. For security teams, the implication is that LLM review processes must be built into the workflow, not bolted on as an audit exercise.
Traceability is becoming a core AI governance requirement, not a nice-to-have dashboard feature. Session IDs, blocked-event logging and policy-specific violation analysis create evidence that can support investigations, tuning and compliance reporting. Without that record, teams cannot explain why a model was blocked or whether a leak was prevented in time. Practitioners should regard traceable decisioning as part of the control, not a reporting layer.
Dual-layer LLM policy creates a named control pattern: conversation boundary enforcement. The article’s prompt and response split reflects a useful governance idea, namely that the boundary around an LLM must be enforced on both ingress and egress. That is the right mental model for enterprises embedding LLMs into customer support, development and regulated workflows. Teams should design controls around the conversation boundary, not the model alone.
From our research library:
- A May 2025 Gartner poll of 147 CIOs and IT leaders found that 24% had already deployed AI agents, 50% were experimenting and 17% planned to deploy by the end of 2026.
- Read next: Agentic AI Security Guide
What this signals
Conversation boundary enforcement is the right way to think about LLM security because the model can expose risk at both ingress and egress, so policy has to inspect what goes in and what comes out before business use.
AI deployment confidence is a weak proxy for control maturity: organisations that describe themselves as confident in their AI deployment actually experience a 72% security incident rate, compared to 33% for those who remain cautious, according to the 2026 Infrastructure Identity Survey.
For practitioners, the implication is that AI governance must move from static content rules to traceable runtime decisions that are visible to security, compliance and engineering teams.
For practitioners
- Implement dual-layer prompt and response controls Inspect every LLM interaction before it reaches the model and again before output is released to users or downstream systems.
- Block secrets at the prompt boundary Detect API keys, passwords and other credentials in user input, then stop the request before the model can process it.
- Add session-linked audit trails Correlate prompt scans, response scans and policy decisions to a session ID so investigators can reconstruct each AI interaction.
- Review unsafe-code and DLP rules together Treat insecure code generation, internal data leakage and harmful text as related output risks, because the same model can create all three.
Key takeaways
- LLM security failures are not limited to prompt injection, because unsafe outputs can still expose secrets, generate vulnerable code or spread harmful advice after the model has already processed a request.
- The practical control gap is the boundary around the conversation itself, where both inbound prompts and outbound responses need policy enforcement and auditability.
- Enterprises that cannot trace blocked interactions or policy decisions will struggle to prove that their AI governance controls are actually working.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | LLM prompt abuse and unsafe output can drive privilege misuse in agentic workflows. |
| Recommendation — Apply ASI03 controls to restrict what an LLM can access, reveal or execute in workflow context. | ||
| MITRE ATT&CK | TA0006;TA0010 — Credential Access; Exfiltration | The article’s core risks include secret exposure and data leakage through prompt and response paths. |
| Recommendation — Map prompt and output abuse to TA0006 and TA0010, then hunt for leaked credentials and exfiltration patterns. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | The article is fundamentally about policy controls, traceability and accountability around LLM use. |
| Recommendation — Use GOVERN to assign ownership for prompt, response and audit controls across the AI lifecycle. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | LLM interfaces need authorised access boundaries because prompts can carry sensitive business content. |
| Recommendation — Apply PR.AA-05 to constrain who and what may interact with production LLM workflows. | ||
| NIST AI 600-1 | GenAI Profile — Generative AI Profile | The article addresses generative AI deployment controls, monitoring and safe-use enforcement. |
| Recommendation — Align LLM policy and monitoring to the GenAI profile when deploying models into enterprise workflows. | ||
Key terms
- Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
- Response policy: Response policy is the pre-approved set of rules that determines what an automated security workflow is allowed to do. In an AI SOC, it defines when containment, account suspension, or enrichment can happen, who owns the rule, and how actions are logged and reversed if needed.
- Conversation Boundary Enforcement: Conversation boundary enforcement is the practice of controlling both sides of an LLM interaction, not just the prompt. It treats the input and output paths as one governed workflow, which is essential when a model can leak information or produce risky content after inference.
- Session-Linked Audit Trail: A session-linked audit trail connects prompts, outputs, policy decisions and enforcement reasons to a single interaction record. This gives security and compliance teams evidence for investigations, tuning and reporting, especially where LLM decisions affect regulated or high-risk workflows.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management for practitioners securing modern AI-enabled systems. It gives security teams a practical foundation for governing identities and credentials across the broader enterprise stack.
Published by the NHIMG editorial team on August 21, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org