Join our Newsletter — 33% off our NHI Course

Why did prompt injection beat AI agent safeguards in the CTF?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 20230
Topic starter  

TL;DR: AI agent prompt attacks worked best when they looked like normal work, Verno Labs found in a CTF with Imperial College. AI assistants need task-scope, structured-output and memory controls before they can reconstruct or reveal hidden state.

Editorial analysis by NHI Mgmt Group, based on content published by Verno Labs: “Verno Labs and Imperial College CTF”.

By the numbers:

  • Verno Labs collected 1,024 conversations across all five levels and saw 212 out of 1,024 conversations result in some form of data leakage.

Key questions

Q: Why do structured prompts create more risk than direct jailbreaks?

A: Structured prompts often succeed because they look like normal machine-readable work rather than a hostile override.

Q: How can teams tell whether an AI agent is becoming an exfiltration path?

A: Look for repeated partial disclosures, unusual format coercion, rising use of encoding and attempts to reconstruct hidden state across turns.

Q: What breaks when AI agents have broader access than their tasks require?

A: Over-privileged agents break segregation of duties, weaken auditability, and expand blast radius across transactions, data lookups, and workflow triggers.

Practitioner guidance

  • Implement task-scoped disclosure policies Allow agents to answer only within a narrowly defined task and deny any response that would reveal hidden prompts, secrets or transaction data outside that scope.
  • Constrain multi-turn memory reuse Expire or isolate conversational context that could be recombined into a secret, and prevent earlier partial disclosures from authorising later ones.
  • Block structured output exfiltration paths Apply the same authorisation checks to JSON, YAML and other structured responses that you use for free-text replies, because schema-shaped output is often the delivery wrapper.

Bottom line: AI agent prompt attacks succeed most often when they are disguised as legitimate work, not when they read like obvious jailbreak attempts.

Glossary terms: Prompt Injection (Agentic), Structured-schema Extraction, Task-Scoped Access, Conversational Authority

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 2 weeks ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20712
 

Prompt injection is now an authorisation problem, not just a content-safety problem. The challenge shows that the model will comply with a convincing workflow even when the underlying request is malicious. That is an identity and access issue because the assistant is effectively making disclosure decisions on behalf of a user or system. Practitioners should treat agent output permissions as governed access, not as an untrusted text problem.

A few things that frame the scale:

  • Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.

A question worth separating out:

Q: Should organisations use the same controls for humans, NHIs, and AI agents?

A: No. The control family may overlap, but the operating assumptions differ. Human identity controls focus on authentication and user context, while NHIs need lifecycle and credential governance, and AI agents require both NHI controls and runtime oversight for autonomous action. The correct model is shared governance with actor-specific enforcement.

👉 Read our full editorial: AI agent prompt injection stays effective when attacks look legitimate


This post was modified 2 weeks ago by NHI Mgmt Group

   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.