AI security guardrails are failing against real prompt attacks

At a glance

What this is: Prompt||GTFO Season 1 is a practitioner-focused AI security event that surfaced working prompt attacks, model persistence tricks, and large-scale red teaming methods.

Why it matters: It matters because AI security teams now need controls for model behaviour, prompt handling, and data exposure, not just static filtering and policy statements.

By the numbers:

• Running a single prompt across 30+ models simultaneously, he could quickly benchmark weaknesses and defenses across the entire AI landscape.

👉 Read Knostic's analysis of Prompt||GTFO Season 1 and the AI security attacks it surfaced

Context

AI guardrails fail when defenders treat prompts as simple input and models as passive systems. The article shows that protocol confusion, persistence across versions, and automated social engineering can all bypass controls that were designed for narrower, more predictable interactions. For identity and access teams, the new issue is not only what data the model can see, but what the AI system itself is allowed to do with that data.

Prompt security has become an identity and governance problem as much as a model safety problem. When AI tools can carry context across sessions, generate their own prompts, and interact with enterprise data sources, they begin to resemble governed non-human actors with access boundaries that need lifecycle control. That makes AI security a shared concern for IAM, data governance, and AI risk owners.

Key questions

Q: How should security teams govern AI systems that can influence tools and workflows?

A: Security teams should treat AI systems as governed runtime actors, not passive applications. That means defining their allowed data sources, tool permissions, logging requirements, and escalation paths before deployment. When an AI system can shape actions or decisions, least privilege, approval boundaries, and auditability become core controls, not optional enhancements.

Q: Why do prompt attacks bypass traditional AI guardrails so easily?

A: Prompt attacks succeed when controls only inspect surface text and ignore how the model transforms, stores, and reuses context. An attacker can hide intent through encoding, delegation, or self-referential output, then exploit the system's own behaviour. Effective defence needs layered inspection, version-aware policy checks, and tool-use restrictions.

Q: How do teams know if AI guardrails are actually working?

A: Guardrails are working only if they produce consistent refusal, logging, and containment behaviour across models, versions, and input transformations. If the same prompt behaves differently after encoding, rerouting, or session replay, the control boundary is leaking. Measure consistency, not just block rates, because attackers look for exceptions.

Q: Who should be accountable when an AI system leaks unsafe instructions or data?

A: Accountability should sit with the team that owns the AI system's permissions, logging, and release controls, not only the model provider. If the system can persist context or reach enterprise data, the operating team must define acceptance criteria, monitoring thresholds, and incident response steps for model misuse.

Technical breakdown

Protocol confusion attacks and guardrail bypass

Protocol confusion attacks exploit the difference between what a model receives, what it re-encodes, and what downstream controls inspect. In the example described, malicious prompts were hidden with ROT-13 and then trusted when echoed back by the system, which caused the model to validate content that should have been blocked. The failure mode is not just weak filtering. It is a mismatch between transformation layers, inspection points, and the model's tendency to trust its own output.

Practical implication: validate prompts before transformation and after decoding, not only at the content-filter layer.

Model persistence and cross-version memory abuse

Model persistence attacks rely on the fact that conversational state, policy behaviour, and safety responses can differ across versions or sessions. The article describes a case where dangerous instructions were recovered by carrying a conversation across ChatGPT versions, exploiting gaps between model updates and safety resets. That means the attacker is not breaking one model instance only. They are using the system's memory boundary, update cadence, and session continuity as the attack surface.

Practical implication: treat session continuity and version transitions as security boundaries that need explicit reset and re-validation.

Industrialised red teaming across many LLMs

Modern red teaming is becoming a parallel testing problem, not a one-off review. The article describes methods that run a single prompt across dozens of models to identify where protections diverge, then use those differences to infer weaknesses in filtering logic, tool use, and policy enforcement. This matters because AI security failure is often inconsistent across model providers and deployment configurations, so the attacker only needs one weak path.

Practical implication: benchmark policy enforcement across every model and deployment path, not just the primary production model.

Threat narrative

Attacker objective: The attacker wants to extract hidden model behaviour or sensitive instructions and use the AI system itself as a scalable attack and influence channel.

1. Entry occurs through prompt injection or hidden prompt encoding that bypasses the first layer of inspection.
2. Escalation happens when the attacker leverages model memory, version differences, or self-referential outputs to recover restricted information or alter behaviour.
3. Impact is achieved when the model leaks hidden instructions, exposes unsafe guidance, or becomes a scalable tool for manipulation and data exposure.

NHI Mgmt Group analysis

Prompt security is now a governance problem, not a content-filter problem. The article shows that attackers can exploit protocol layers, context handling, and model self-reference in ways static filters do not anticipate. That shifts the control question from "what words are allowed" to "who can influence model state, tool use, and downstream action." Practitioners should treat AI prompt pathways as governed access surfaces.

AI systems are increasingly acting like non-human identities with their own access boundaries. Once a model can retain context, call tools, and generate instructions that affect later sessions, it needs lifecycle thinking similar to other machine actors. That does not mean every model is autonomous, but it does mean its privileges, context scope, and permitted actions must be explicitly bounded. Practitioners should align AI governance with identity and privilege management.

Industrialised red teaming will expose inconsistency faster than policy committees can close it. Running the same prompt across many models reveals that security assumptions often hold in one place and fail in another. That makes standardisation around testing, logging, and model approvals essential. Practitioners should assume adversaries will benchmark the AI estate the same way defenders should.

Model persistence shows why AI risk cannot be measured per session alone. The attack surface includes version transitions, memory carryover, and hidden state that persists beyond a single prompt exchange. Security teams that only inspect isolated interactions will miss the broader control failure. Practitioners should govern the full conversation lifecycle, not just the current request.

AI security is converging with identity, data, and workflow governance. The real exposure comes when models can reach enterprise data, generate actions, and influence human decisions without equivalent oversight. That is where NHI and agentic AI governance becomes relevant, because the AI system itself behaves like a controlled runtime actor. Practitioners should plan for AI security controls that span authentication, authorisation, logging, and policy enforcement.

What this signals

Prompt governance will increasingly sit alongside identity governance in AI programmes. The practical shift is toward inventories of prompts, tools, and model permissions, with logs that can stand up to audit and incident review. Teams that already manage non-human access can extend those controls to AI systems faster than teams starting from scratch.

The next governance gap is not model accuracy, but model authority. Once AI can reach documents, systems, or workflows, the question becomes whether the AI boundary is enforced with the same discipline as human access, machine credentials, and service accounts.

Need-to-know for AI systems is emerging as a real control concept. If the model can only see the data and tools required for a narrow task, attack impact stays bounded. If it inherits broad context or excessive tool reach, prompt abuse becomes an enterprise-wide exposure vector.

For practitioners

• Harden prompt ingress and decoding paths Inspect prompts before and after any transformation step, including encoding, templating, and retrieval augmentation. Apply separate controls to user input, system prompts, and tool-generated output so one layer cannot validate another layer's unsafe content.
• Reset security state across model version changes Define explicit reset and re-validation steps when a model, policy, or conversation context changes. Do not assume that a session safe under one version remains safe after upgrade, failover, or context replay.
• Test model behaviour across the full AI estate Run the same adversarial prompt set across every production model, gateway, and configuration. Compare logging, refusals, and tool invocation behaviour to find the weakest path rather than the nominal control path.
• Treat AI tool access as privileged access Restrict which models can reach enterprise data, external tools, or workflow actions, and record those permissions in a governed inventory. Use least privilege for model-to-tool access and review it on the same cadence as other high-risk machine accounts.
• Build persistence-aware monitoring for AI sessions Monitor for repeated attempts to recover hidden instructions, policy fragments, or unsafe outputs across sessions and versions. Tie alerts to conversation reuse, state carryover, and unusual prompt chaining rather than single-message content alone.

Key takeaways

• Prompt attacks succeed when models trust transformed content, cross-session state, or their own output more than the original control boundary.
• The article shows that AI red teaming has become industrialised, with one prompt able to expose weaknesses across dozens of models in minutes.
• Practitioners should govern AI tool access, session state, and version transitions as privileged control points, not just model features.

Key terms

• Prompt Injection: Prompt injection is an attack where malicious instructions are embedded in user content, retrieved content, or tool output to influence model behaviour. The goal is to override intended policy, steer responses, or trigger unsafe actions by exploiting how the model interprets context.
• Model Persistence: Model persistence is the tendency for an AI system to retain useful context, state, or behavioural cues across turns, versions, or sessions. In security terms, that persistence becomes a control risk when attackers use it to recover blocked content or carry unsafe influence forward.
• Tool Use Authorization: Tool use authorisation is the set of permissions that determines which systems an AI model may query, modify, or trigger during runtime. It is the machine-access equivalent of privileged access, and it must be scoped, logged, and reviewed with the same discipline as other high-risk access paths.

What's in the full article

Knostic's full research covers the operational detail this post intentionally leaves for the source:

• Step-by-step examples of the prompt confusion and self-incrimination techniques used against models.
• Concrete demonstrations of multi-model jailbreak testing across different providers and configurations.
• Practical logging and debugging patterns for non-deterministic AI applications.
• Additional session summaries from the Prompt||GTFO lineup that show how offensive AI tradecraft is evolving.

👉 Knostic's full post covers the prompt attacks, red team methods, and practical lessons in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and related control patterns that security practitioners need for governed access. It is suitable for teams extending identity discipline into AI systems and other non-human actors.