What is the difference between embedding model keys in apps and centralising them in a gateway?

Embedding model keys in apps means every caller holds a credential and talks directly to the provider. Centralising them in a gateway keeps the provider key in one place, authenticates callers with identity, and forwards requests on their behalf. The gateway model improves control, auditability, and guardrails without forcing teams to distribute secrets widely.

Why This Matters for Security Teams

The decision to embed model keys in applications or centralise them in a gateway is really a decision about identity, control, and blast radius. If every app carries its own provider key, the organisation gets faster initial integration but also wider secret sprawl, weaker revocation, and far less observability. NHI Mgmt Group notes that 96% of organisations store secrets outside of secrets managers in vulnerable locations, including code and CI/CD tools, which is exactly the failure mode this pattern creates.

Security teams often underestimate how quickly embedded keys become operational debt. A key in code is hard to rotate, hard to scope, and hard to prove is still in use only where intended. A gateway changes the model: callers authenticate with identity, policy is enforced in one place, and the provider secret stays hidden behind a managed control point. That lines up with least privilege principles in NIST SP 800-53 Rev 5 Security and Privacy Controls and with NHI governance guidance in Ultimate Guide to NHIs — What are Non-Human Identities.

In practice, many security teams encounter key leakage only after a repository, build log, or third-party integration has already exposed the secret.

How It Works in Practice

Embedding a model key in an app means the application itself is trusted to hold and use the provider credential directly. That is simple, but it also makes every deployment, developer environment, and downstream replica a place where the secret can leak. Centralising the key in a gateway shifts trust away from the app binary and toward an intermediate control plane that authenticates the caller, evaluates policy, and forwards requests to the model provider using a managed secret.

In a well-designed gateway pattern, the application proves who it is, the gateway decides what it may do, and the provider key never leaves the boundary of the gateway. This improves auditability because all model traffic passes through a single choke point. It also enables controls such as rate limiting, request logging, prompt filtering, tenant separation, and per-workload policy. The pattern aligns with the NIST guidance on access control and system monitoring, and it reflects the NHI governance emphasis in Ultimate Guide to NHIs — What are Non-Human Identities.

• Use app identity or workload identity for caller authentication instead of sharing the provider key broadly.
• Store the model provider key only in the gateway, ideally in a secrets manager or equivalent protected store.
• Apply policy at the gateway for model choice, user context, rate limits, and logging.
• Rotate the provider credential centrally so all consuming apps inherit the change without code edits.

For implementation detail, teams can compare this model against broader control guidance in NIST SP 800-53 Rev 5 Security and Privacy Controls and treat the gateway as the enforcement point for secrets handling, access review, and telemetry. These controls tend to break down in highly distributed edge environments where apps must operate offline and cannot reliably reach a central gateway.

Common Variations and Edge Cases

Tighter gateway centralisation often increases operational dependency, requiring organisations to balance stronger control against latency, uptime, and platform complexity. That tradeoff is real, especially where applications are event-driven, latency-sensitive, or deployed across multiple clouds and regions.

There is no universal standard for this yet, but current guidance suggests avoiding a false binary. Some teams centralise only the provider credential while still allowing local service-to-service identity for internal requests. Others use multiple gateways by domain or tenant to reduce blast radius. A gateway is also not a complete control by itself if it simply proxies requests without caller authentication, policy enforcement, and logging. In that case, it is just a thinner secret distribution layer.

Embedding may still appear in prototypes, air-gapped systems, or single-purpose tools where a gateway would add more risk than benefit. Even there, best practice is to treat the embedded key as a temporary exception with explicit rotation, restricted scope, and a removal plan. NHI Mgmt Group’s broader research shows why this matters: 79% of organisations have experienced secrets leaks, and 71% of NHIs are not rotated on time, so the long-term risk of leaving keys in apps is rarely theoretical.

For teams deciding between the two patterns, the practical question is not just where the key lives, but whether the organisation can prove who used it, when, and for what purpose.

Related resources from NHI Mgmt Group

• What is the difference between gateway-first and SDK-first agent security?
• What is the difference between privilege reduction and secret rotation?
• What is the difference between a rules-based secret scanner and a hybrid scanner?
• What is the difference between code scanning and runtime identity monitoring?