Should organisations trust assistants just because the system prompt is visible?

Why a Visible System Prompt Is Not a Trust Boundary

Seeing the system prompt can help reviewers understand the assistant’s intended role, but it does not guarantee the model will keep behaving that way. Runtime behaviour can shift because of prompt injection, tool output, hidden instructions, memory contamination, or a compromised orchestration layer. Trust has to come from controls around execution, not from transparency alone.

Visible instructions are useful for auditability, especially when teams need to compare intended policy with observed outputs. The problem is that a prompt is only one input to a larger runtime, and it may not be the strongest one.

How Assistants Still Drift After Reviewable Prompts

An assistant can appear well governed at design time and still diverge at runtime. A malicious prompt can redirect it, a downstream tool can return tainted content, or a connected service can cause it to reveal information in a format that looks legitimate. The danger is not only direct jailbreaks, but also compliant-looking output that leaks data, bypasses intended boundaries, or repeats sensitive context in a new channel.

This is why output handling matters as much as input handling. If an assistant can echo secrets, repackage confidential text, or call tools beyond its intended scope, a visible prompt does not stop misuse. The control question is whether the system can constrain what the assistant is allowed to do even when it is influenced at runtime.

What Organisations Should Trust Instead of the Prompt Alone

Organisations should treat assistants as governed non-human identities with scoped authority, explicit change control, and enforceable data handling rules. That means constraining which tools they can use, what data they can access, what formats they can emit, and which changes require human approval. In practice, the reviewable prompt is evidence, not a control.

For agent-to-agent and multi-hop workflows, the question becomes even sharper because delegation can expand the blast radius quickly. NHIMG’s Multi-Agent and A2A Security Guide is useful here because it frames authentication, delegation chains, and containment as first-class concerns rather than assuming the visible prompt is enough.

Risk and Threat Considerations

Visible prompts can create a false sense of control. The real exposure is that runtime instructions, tool responses, and formatting channels can be abused to leak data or extend behaviour beyond what reviewers expected, especially when assistants sit inside business processes or hold access to sensitive systems.

Failure mechanism: The assistant follows a later instruction, a compromised tool response, or an allowed output path that carries sensitive data in a seemingly legitimate form, bypassing the intent of the visible prompt.

Impact: Organisations may overestimate safety, miss policy drift, and allow data exposure or unauthorised actions through an assistant that still appears compliant on paper.

Practitioner Guidance

What to verify: Confirm that the assistant’s effective permissions are smaller than its apparent conversational capability. Review tool scopes, output filters, logging, and approval points as separate controls, because a readable prompt does not prove any of them are enforced.

Decision rule: If the assistant can influence production data, external systems, or confidential text, require runtime controls such as allowlisted tools, content filtering, and human approval for material actions before treating the system as trustworthy.

Practitioner takeaway: A visible prompt improves transparency, but trust only becomes defensible when the assistant’s authority, outputs, and data flows are constrained independently of what the prompt says.

Related resources from NHI Mgmt Group

• When should organisations treat an AI agent as a privileged system?
• When should organisations use AI to help manage NHIs?
• How can organisations prevent AI agents from becoming overprivileged?
• How can organisations govern AI agents that use service accounts and tokens?