An AI gateway ends up holding every model provider key and seeing every prompt and response that passes through it, so the security requirements have to be written into the request for proposal, where bidders must answer them, rather than discovered during rollout. This template is for buying an AI gateway (also called an LLM gateway or model proxy), whether self-hosted software or a managed service, that sits between applications, agents and users on one side and model providers on the other. Requirements are numbered, typed and written so each answer can be tested and scored, and the template ends with a proof of concept plan. It is vendor-neutral and draws on the OWASP Top 10 for LLM Applications 2025, MITRE ATLAS, the NCSC Guidelines for Secure AI System Development and the CSA AI Controls Matrix. It is written for security architects, AI platform owners, procurement and evaluation panels. What a gateway does and its architecture options are explained in the AI Gateway and LLM Gateway Guide; the capability areas to shortlist against are in the AI Gateway Buyer's Guide. This page turns both into contract-ready questions.
How to use this template
- Replace text in square brackets, such as [Organisation], [identity provider], [secrets manager], [SIEM] and [number], with your own details. Delete requirements that do not apply rather than leaving them unscored.
- Each requirement has an ID (for example AGW-IDN-01) and a type: M mandatory (pass or fail), S scored (0 to 5) or I informational. Keep M items few; caller identity, provider key custody, admin plane protection and logging in sections 4.1, 4.2, 4.6 and 5 are the strongest candidates.
- Before issue, complete Appendix A: the two or three workloads the gateway must carry first, for example an internal chat assistant used by staff, a customer-facing retrieval application, and an agent that calls models many times per task. Bidders answer against these and the proof of concept runs them.
- Weights in brackets are illustrative. Fix them, and the scoring rules, before responses arrive.
- Issue sections 1 to 8 to bidders. Implementation guidance is for the panel only. Ask for responses in a spreadsheet with one row per requirement ID and a column for evidence.
1. Purpose and scope
[Organisation] invites proposals for an AI gateway that will route requests from approximately [number] applications, [number] agents and [number] staff to [list of model providers and self-hosted models], within [period]. The gateway must authenticate every caller, hold provider credentials on their behalf, enforce model access, budgets and rate limits per caller, apply input and output policy, and produce logs suitable for security monitoring and cost allocation. In scope: the gateway runtime, its administration interface and API, policy and guardrail functions, key and budget management, logging and export, and deployment options. Out of scope: model hosting, agent build platforms and dedicated Model Context Protocol (MCP) tool gateways, unless offered as an option in section 7.
2. Current state and intended use
- Identity: [identity provider], [workload identity method in each cloud or cluster], [secrets manager].
- Model providers and endpoints: [providers, cloud model services and self-hosted inference servers in use today], with the number of provider keys found during discovery: [number].
- Callers: [applications, agents, coding assistants, notebooks and batch jobs], with owners and data classes sent in prompts.
- Security tooling: [SIEM], [DLP], [web proxy or security service edge], [API gateway].
- Constraints: data residency [region], hosting model [self-hosted in our cloud / managed service / either], regulatory obligations [list], latency budget [milliseconds added at the 95th percentile].
- Appendix A workloads: [describe two or three workloads, their traffic volume, data and failure tolerance].
3. Instructions to bidders
- Answer each requirement as Met, Met with configuration, Roadmap (with date) or Not met, and attach evidence: documentation link, configuration export, screenshot or independent report.
- State which features need a higher licence tier, add-on or enterprise edition, in one availability note per section.
- For open source or open-core offerings, state which requirements are met by the community edition and which by the commercial edition.
- Describe every default credential, bootstrap secret or default administrator account shipped with the product and how it is removed.
- Shortlisted bidders must take part in the proof of concept in section 8 using the Appendix A workloads.
4. Security requirements
4.1 Caller identity and access
| ID | Requirement | Type |
|---|---|---|
| AGW-IDN-01 | Each application, agent and team can be issued its own gateway credential, so that usage, spend and policy verdicts are attributed to one caller and that caller can be revoked alone. | M |
| AGW-IDN-02 | Callers can authenticate with short-lived tokens from [identity provider] or cloud workload identity federation instead of static keys; list each method supported and its token validation (issuer, audience, expiry). | S |
| AGW-IDN-03 | Where gateway keys are issued, they support expiry dates, a model allow list, a budget and a rate limit per key, and are shown once at creation and stored only as a hash. | M |
| AGW-IDN-04 | Staff access through chat or tool front ends uses [identity provider] sign-in, with model access granted by group membership. | S |
| AGW-IDN-05 | The gateway can record an end-user or agent identifier passed by the calling application, separate from the caller credential, and include it in logs. | S |
| AGW-IDN-06 | All caller credentials, their owners, scopes, budgets, creation dates and last use can be listed through an API for inventory and access review. | S |
See the Workload Identity Federation Guide and the Agent Identity and Access Guide.
4.2 Provider credential custody
| ID | Requirement | Type |
|---|---|---|
| AGW-KEY-01 | Provider API keys and cloud credentials can be read at run time from [secrets manager] by reference, so configuration files and repositories hold references, not values. | M |
| AGW-KEY-02 | Where cloud model services support it, the gateway authenticates with a cloud role or workload identity rather than a static key; describe each provider integration. | S |
| AGW-KEY-03 | Provider credentials are never returned to callers, never written to logs and never visible in full in the admin interface after entry. | M |
| AGW-KEY-04 | A provider credential can be rotated without downtime and without changing caller configuration; describe the procedure and time taken. | S |
| AGW-KEY-05 | Separate provider credentials can be configured per environment (development, test, production) and per business unit for cost isolation. | S |
Stolen model keys are resold and abused for compute, which MITRE ATLAS records as AML.T0034 Cost Harvesting; see the LLMjacking Guide.
4.3 Model access, budgets and rate limits
| ID | Requirement | Type |
|---|---|---|
| AGW-USE-01 | Model allow lists can be set per caller, team and environment, and a request for a model outside the list is refused and logged. | M |
| AGW-USE-02 | Budgets in currency or tokens can be set per caller, team and period, with alert thresholds and a hard stop; state how quickly a hard stop takes effect under concurrent load. | M |
| AGW-USE-03 | Rate limits on requests and tokens per minute can be set per caller, and maximum input size, output tokens and request timeouts per route (OWASP LLM10:2025 Unbounded Consumption). | S |
| AGW-USE-04 | Routing, fallback and load balancing rules cannot send a request to a provider or region outside the caller's permitted list, including during failover. | M |
| AGW-USE-05 | Exposure of token probability data (logit bias and log probabilities) to callers can be restricted per route. | S |
| AGW-USE-06 | Response caching can be disabled per route, and where enabled, cache entries are partitioned so one caller cannot receive another caller's cached response; describe the cache key. | S |
OWASP LLM10:2025 lists input validation, limiting exposure of logits and log probabilities, rate limiting and user quotas among its mitigations. See How to set spend limits on LLM API keys.
4.4 Input and output policy
| ID | Requirement | Type |
|---|---|---|
| AGW-POL-01 | Input and output filters for personal data, secrets and [Organisation] patterns can be configured per route, with actions of log, redact or block (OWASP LLM02:2025 Sensitive Information Disclosure). | S |
| AGW-POL-02 | Describe any prompt injection detection offered, the categories it covers (direct, and indirect through retrieved content), the test data used and published detection and false positive rates (OWASP LLM01:2025 Prompt Injection). | S |
| AGW-POL-03 | Policies run in report-only mode before enforcement, are versioned, and can be rolled back to a previous version per route. | M |
| AGW-POL-04 | External guardrail or classification services can be called from the gateway, with a defined behaviour (fail open or fail closed, chosen per route) when the service is unavailable. | S |
| AGW-POL-05 | Every policy verdict (allowed, redacted, blocked) is logged with the policy version and rule that produced it. | M |
| AGW-POL-06 | State whether the gateway modifies prompts (for example inserting system instructions) and how such changes are recorded. | I |
See the AI Guardrails and AI Firewalls Guide and the Generative AI Data Loss Prevention Guide.
4.5 Agent and tool traffic
| ID | Requirement | Type |
|---|---|---|
| AGW-AGT-01 | Per-agent limits on tokens per task, requests per task and spend per hour can be set, so a looping or hijacked agent stops without human action (OWASP LLM06:2025 Excessive Agency; OWASP Agentic Top 10 ASI08 Cascading Failures). | S |
| AGW-AGT-02 | A task, session or trace identifier passed by the caller is preserved in gateway logs so model calls can be joined with the agent's tool calls. | M |
| AGW-AGT-03 | If the product also brokers MCP or other tool traffic, describe how tool servers are authenticated, how tokens are validated for audience, and confirm that client tokens are not passed through to downstream APIs, as the MCP security best practices forbid token passthrough. | I |
| AGW-AGT-04 | Tool or function definitions sent in requests can be inspected and logged, and requests that declare tools outside a caller's permitted list can be refused. | S |
Tool traffic usually needs its own control point; see the MCP Gateway and MCP Server Governance Guide.
4.6 Administration plane
| ID | Requirement | Type |
|---|---|---|
| AGW-ADM-01 | Administrator sign-in uses [identity provider] SSO with phishing-resistant MFA; any built-in master or bootstrap key can be disabled after setup and is not needed for daily administration. | M |
| AGW-ADM-02 | The admin interface and API can be served on a separate listener, network or hostname from the inference endpoint, so only inference is reachable by callers. | M |
| AGW-ADM-03 | Roles separate configuration, key issuance, budget management and read access to prompt logs; describe the built-in roles and custom role support. | S |
| AGW-ADM-04 | Configuration can be managed as code through an API or declarative files, with changes reviewable before apply. | S |
| AGW-ADM-05 | Administrator provisioning and removal can be automated through SCIM or group sync from [identity provider]. | S |
Default master keys left in production are a recurring cause of gateway credential exposure; the hardening steps after purchase are in Securing the AI gateway itself.
5. Logging, audit and monitoring
| ID | Requirement | Type |
|---|---|---|
| AGW-AUD-01 | Each request is logged with caller credential, end-user or agent identifier, source address, model and provider requested and used, token counts, cost, latency, status and policy verdicts. | M |
| AGW-AUD-02 | Prompt and response content logging is configurable per route (off, metadata only, redacted, full), with separate retention for content and metadata. | M |
| AGW-AUD-03 | Administrative events (key creation and deletion, budget and policy changes, provider credential changes, role changes, sign-ins and failures) are logged separately and cannot be altered by administrators. | M |
| AGW-AUD-04 | Logs stream to [SIEM] in a documented schema in near real time; state the format, transport and delay. | M |
| AGW-AUD-05 | Built-in alerts or exportable signals for spend spikes per caller, use of a credential from a new network location, requests for disallowed models, high-volume systematic querying and repeated policy blocks. | S |
High-volume systematic querying can indicate model extraction, recorded in ATLAS as AML.T0024 Exfiltration via AI Inference API. See the AI Agent Audit Log Schema Template for fields to require.
6. Platform security, assurance and hosting
| ID | Requirement | Type |
|---|---|---|
| AGW-PLT-01 | Releases are versioned and published with checksums or signatures, and a software bill of materials is available on request (OWASP LLM03:2025 Supply Chain). | S |
| AGW-PLT-02 | Describe the vulnerability disclosure process, security advisory channel, and target time to fix critical issues; state how customers are notified. | M |
| AGW-PLT-03 | For a managed service: state where prompts, responses, logs and cache are processed and stored, all sub-processors, and whether any customer content is used to train or improve models or products, with a contractual opt-out. | M |
| AGW-PLT-04 | Provide current independent assurance (for example a SOC 2 Type II report or ISO/IEC 27001 certificate with scope) and state any ISO/IEC 42001 certification. | S |
| AGW-PLT-05 | Complete the CSA AI-CAIQ, or map controls to the CSA AI Controls Matrix, stating which controls sit with [Organisation]. | I |
| AGW-PLT-06 | Describe secure development practice against the NCSC Guidelines for Secure AI System Development stages: secure design, development, deployment, and operation and maintenance. | I |
| AGW-PLT-07 | State encryption in transit (caller to gateway and gateway to provider, with certificate validation), encryption at rest for logs and cache, customer-managed key options and private connectivity options. | S |
| AGW-PLT-08 | State availability targets, horizontal scaling method and the measured added latency at the 95th percentile for the Appendix A workloads. | S |
| AGW-PLT-09 | Notify [Organisation] of security incidents affecting its data or credentials within [hours]. | M |
The CSA AI Controls Matrix (243 control objectives across 18 domains in v1) ships with the AI-CAIQ questionnaire, intended for self-assessment or vendor evaluation. For supplier questions beyond this RFP use the AI Vendor Security Questionnaire Template.
7. Services, support, pricing and exit
- Onboarding: implementation plan, hardening guide, and the configuration steps [Organisation] must complete before production traffic.
- Support: severity definitions and response times, including for gateway outage, credential exposure and a policy blocking production traffic.
- Pricing: price at [current], [2x] and [5x] request volume, stating the unit (requests, tokens, callers, nodes or flat fee) and any charge on top of provider costs.
- Exit: export of configuration, routes, policies, caller inventory and logs in documented formats; confirmed deletion of [Organisation] data and credentials within [days].
8. Evaluation method and proof of concept
| Criterion | Weight | Evidence |
|---|---|---|
| Caller identity and key custody (4.1, 4.2) | [20%] | Per-caller credentials issued in the proof of concept; provider keys held in [secrets manager] |
| Model access, budgets and limits (4.3, 4.5) | [20%] | Live demonstration of allow lists, hard stops and agent limits |
| Policy and guardrails (4.4) | [15%] | Report-mode results, block and false positive rates on [Organisation] test set |
| Admin plane and logging (4.6, 5) | [20%] | Exported logs in [SIEM]; admin role test |
| Performance and operations (6) | [10%] | Measured latency and failover behaviour |
| Assurance and cost over the term (6, 7) | [15%] | Reports, AI-CAIQ, pricing at three volumes |
Scores use 0 to 5: 0 not met; 1 roadmap only; 3 met with configuration; 5 met fully and demonstrated. Shortlisted bidders run a [three]-week proof of concept:
- Deploy with no default credentials active, admin interface on a separate listener, and administrator sign-in through [identity provider].
- Load provider credentials from [secrets manager] by reference; confirm they do not appear in configuration exports, logs or the admin interface.
- Issue a credential per Appendix A workload; request a disallowed model and exceed a budget, and record whether each is refused and logged.
- Run [Organisation] test prompts containing personal data, secrets and embedded instructions in report mode, then in enforce mode; record verdicts and false positives.
- Force a provider outage and confirm fallback stays inside each caller's permitted providers and regions.
- Rotate a provider credential under load; revoke one caller credential and time how long until requests fail.
- Export a day of logs to [SIEM] and trace one agent task from caller to model calls using the task identifier.
Implementation guidance
This section is for the evaluation panel, not for bidders.
- Run discovery before issue: the number of provider keys, direct callers and agents found today sizes the RFP and fills section 2. See the Shadow AI Discovery Guide.
- Weight custody and identity above features: routing and model support change monthly; how the gateway holds keys and identifies callers is hard to change once hundreds of callers depend on it.
- Self-hosted vs managed: self-hosted keeps prompts in your environment but makes AGW-PLT-01 and AGW-PLT-02 your patching burden; managed shifts operations but makes AGW-PLT-03 and AGW-PLT-04 mandatory.
- Small vs large organisations: a small organisation with few callers can score provider-side controls (per-project keys, spend caps) as partial credit and keep M items to identity, custody and logging; a large one should make SSO, secrets manager integration and SIEM streaming mandatory.
- Use dedicated trial credentials: capped, time-limited provider keys and synthetic test data for the proof of concept, revoked at the end.
- Plan the rollout and the bad day: phase onboarding with the AI gateway rollout playbook, check settings after go-live with the AI gateway health check, and prepare the AI gateway compromise playbook before production.
Standards and references
- OWASP: Top 10 for LLM Applications 2025
- OWASP: Top 10 for Agentic Applications for 2026
- MITRE ATLAS: Cost Harvesting
- MITRE ATLAS: Exfiltration via AI Inference API
- Model Context Protocol: Security Best Practices
- NCSC: Guidelines for Secure AI System Development
- Cloud Security Alliance: AI Controls Matrix
This template is provided for guidance and must be adapted to your organisation. Related NHI Mgmt Group resources: AI Gateway and LLM Gateway Guide · AI Gateway Buyer's Guide · Agent platform RFP template · Security Vendor Evaluation and RFP Guide