An LLM relay is an intermediary service that forwards prompts and responses between a client and upstream model providers. In malicious cases, the relay can inspect, modify, or exfiltrate content in transit, turning a convenience layer into a trust boundary that attackers control.
Expanded Definition
An LLM relay is not the model itself, but the traffic path that sits between a user, application, or agent and one or more upstream model providers. In legitimate deployments, it can centralise routing, logging, policy checks, key management, and model selection. In security-sensitive environments, that same position makes it a high-value intermediary because it may see prompts, system instructions, tool calls, retrieval content, and generated outputs before they reach their destination.
The security significance of an LLM relay is that it creates an additional trust boundary. A well-designed relay can reduce exposure by enforcing content filtering, quota controls, tenant isolation, and audit logging. A compromised or malicious relay can do the opposite by altering prompts, injecting instructions, capturing secrets, or selectively censoring responses. This makes the concept closely related to agentic AI governance in the OWASP Agentic AI Top 10 and to risk treatment under the NIST AI Risk Management Framework. The most common misapplication is treating the relay as a neutral transport layer when it is actually a privileged inspection and enforcement point that can shape model behaviour if compromised.
Examples and Use Cases
Implementing an LLM relay rigorously often introduces added latency and operational complexity, requiring organisations to weigh centralised control against the cost of another component that can fail, be abused, or become a bottleneck.
- A security team routes all employee prompts through a relay that redacts secrets before forwarding them to external model providers.
- An AI product uses a relay to apply policy checks, model allowlists, and data residency rules before requests reach the upstream service.
- A compromised relay modifies a system prompt to weaken guardrails or inject tool instructions into an autonomous agent workflow.
- A regulated organisation logs every relay transaction for auditability, while ensuring the logs do not capture unnecessary personal data or sensitive credentials.
- A multi-tenant platform uses a relay to isolate customers, but a configuration error allows one tenant’s context to leak into another tenant’s session.
These use cases map directly to the kinds of control and threat questions raised in the NIST AI 600-1 Generative AI Profile and the CSA MAESTRO agentic AI threat modeling framework, which both emphasise governance over intermediary layers that can alter model inputs and outputs.
Why It Matters for Security Teams
Security teams need to understand an LLM relay because it often becomes the real enforcement layer for confidentiality, integrity, and access control in AI applications. If the relay is weakly governed, attackers may not need to break the model at all; they can target the path in front of it to capture prompts, manipulate responses, or harvest tokens and API keys. That risk is especially acute when the relay also performs tool orchestration for agents, because prompt tampering can translate into unsafe execution authority.
From a governance perspective, the relay should be treated as part of the system’s attack surface, not as a passive network hop. Relevant threat patterns are echoed in the MITRE ATLAS adversarial AI threat matrix, particularly where adversarial manipulation or exfiltration occurs through the AI pipeline. Security teams should validate who controls the relay, what it can observe, what it can modify, and how its logs are protected. Organisations typically encounter the operational cost of an LLM relay only after a prompt leak, policy bypass, or model-routing compromise, at which point the relay becomes operationally unavoidable to investigate and contain.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI RMF governs trustworthy AI risk management, including intermediary services that shape model inputs and outputs. | |
| NIST AI 600-1 | The GenAI Profile addresses operational controls for generative AI systems and their surrounding service layers. | |
| OWASP Agentic AI Top 10 | OWASP highlights agentic application risks where intermediaries can alter prompts, tools, or execution paths. | |
| CSA MAESTRO | MAESTRO models agentic AI threats across orchestration layers, including relays and brokers in the path. | |
| MITRE ATLAS | ATLAS covers adversarial AI tactics involving manipulation and exfiltration through AI workflows. |
Treat the relay as a governed AI risk component and document controls for integrity, transparency, and accountability.
