Notifications
Clear all
Topic starter
24/06/2026 8:53 pm
TL;DR: GenAI applications can move from operational chaos to controlled routing when the API layer centralises governance, secret handling, rate limits, observability, and prompt-injection defences, according to Kong. The real lesson is that GenAI governance fails when teams treat the model layer as the only risk boundary; access control and telemetry around the application flow matter just as much.
NHIMG editorial — based on content published by Kong: From Chaos to Control: How Kong AI Gateway Streamlined My GenAI Application
Questions worth separating out
Q: How should security teams govern GenAI applications that rely on external model APIs?
A: They should put policy, logging, rate limiting, and secret handling in front of the model rather than inside individual applications.
Q: Why do exposed API keys create such a large risk in GenAI workloads?
A: Because the key is the workload’s identity, so compromise can grant direct access to model APIs and related data flows.
Q: What do security teams get wrong about prompt injection in production AI apps?
A: They often treat prompt injection as a content moderation issue when it is really a request-control issue.
Practitioner guidance
- Separate model access from application secrets Move API keys and other secrets into a dedicated vault and keep them out of application code, build artefacts, and client-visible configuration.
- Enforce policy at the AI ingress layer Apply rate limiting, quotas, request filtering, and prompt inspection before traffic reaches the model.
- Instrument prompt and token telemetry Capture prompt flow, token counts, latency, and policy decisions in the same operational view so security and platform teams can detect abnormal usage patterns.
What's in the full article
Kong's full blog post covers the operational detail this post intentionally leaves for the source:
- The Kong AI Gateway plugin flow used to centralise routing, prompt protection, and observability.
- The AWS-based reference architecture that shows how the application, gateway, and model provider interact.
- The feature set behind AI Manager, semantic routing, caching, quotas, and Vault integration.
- The implementation context for prompt decorating, AI proxy behaviour, and RAG orchestration in the described build.
👉 Read Kong's analysis of AI gateway governance for GenAI applications →
AI gateway governance for GenAI apps: are your controls keeping up?
Explore further
25/06/2026 5:45 am
AI gateway governance is becoming the control point for GenAI applications, not a convenience layer. Once teams move from demos to production, routing, quotas, telemetry, and secret handling become governance functions rather than engineering extras. The article reflects that shift clearly: the application remained usable only after the control plane was tightened around it. Practitioners should treat the AI gateway as part of identity and access governance for the workload.
A few things that frame the scale:
- 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so, according to AI Agents: The New Attack Surface report.
- Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
A question worth separating out:
Q: Who should be accountable for AI gateway governance in an enterprise?
A: Accountability should sit with the teams that own identity, platform policy, and operational risk together, not with model developers alone. When a gateway controls secrets, routing, and usage, it becomes part of the governance stack. That means IAM, security architecture, and platform engineering need shared oversight.
👉 Read our full editorial: AI gateway governance for GenAI applications needs stronger control
