They should treat direct integrations as a temporary pattern and move policy, routing, and observability into a central control plane. The key is to remove provider-specific logic from application code so identity, logging, and failover are enforced consistently across services instead of being rebuilt per team.
Why This Matters for Security Teams
Direct LLM integrations look simple at the application layer, but at scale they become a control problem. Each team that embeds provider logic, custom prompts, and ad hoc logging creates a separate trust boundary, which makes identity, audit, routing, and incident response inconsistent. NHI Management Group’s research on agentic risk shows how quickly AI systems exceed intended scope, and the same pattern appears in direct integrations when controls are left inside app code instead of a shared control plane.
That matters because security teams lose the ability to answer basic questions: which model handled the request, what data was sent, what policy approved it, and how the request failed over. Current guidance from NIST AI Risk Management Framework and OWASP Agentic AI Top 10 both point toward centralized governance, but many teams still treat model selection as a developer convenience rather than a security control.
The practical failure mode is familiar: one service ships with stronger logging, another bypasses review for latency, and a third hardcodes a vendor key with no consistent revocation path. In practice, many security teams encounter exposure only after an integration has already leaked data or bypassed policy, rather than through intentional design.
How It Works in Practice
The scalable pattern is to move provider-specific decisions out of application code and into a central control plane that owns policy, routing, identity, and observability. The application should request LLM access through a governed interface, while the control plane decides which model can be used, what data can be sent, whether redaction is required, and where telemetry is recorded. That aligns with the control-plane approach reflected in NHIMG research on OWASP NHI Top 10 and with implementation guidance in NIST Cybersecurity Framework 2.0.
A practical design usually includes:
- Central policy-as-code for request approval, content filtering, and data classification rules.
- Workload identity for each service so the control plane can authenticate the caller without shared static secrets.
- Request-time routing to approved providers based on jurisdiction, sensitivity, latency, or cost.
- Unified logging for prompts, responses, tool calls, and failover decisions so investigations do not depend on each team’s custom code.
- Short-lived credentials and automated revocation so provider access is not left standing after deployment.
Security teams should also standardise fallback behaviour. If one model is unavailable, the control plane should fail closed or route only to preapproved alternatives under the same policy. That prevents developers from building their own retry and failover logic, which is where governance usually fragments. This approach is reinforced by the CSA MAESTRO agentic AI threat modeling framework and by NHIMG analysis of The State of Non-Human Identity Security, which highlights the visibility gap that appears when access is spread across disconnected systems.
There is also a governance benefit: the control plane becomes the place where security, legal, and platform teams can agree on policy once, instead of re-litigating the same model-access decisions in every product squad. These controls tend to break down when legacy services must call multiple model providers directly because each service then becomes its own shadow control plane.
Common Variations and Edge Cases
Tighter central control often increases release friction, requiring organisations to balance governance against developer autonomy and latency. Best practice is evolving here, and there is no universal standard for how much model choice should be abstracted versus exposed to product teams.
One common variation is a multi-region or regulated-data environment where routing must account for residency constraints. In that case, the control plane needs policy boundaries by tenant, region, and data class, not just by model name. Another edge case is experimentation: teams may need a sandbox path for prompt testing, but that path should be isolated from production identities, production telemetry, and production data sources.
Security teams should be careful not to assume the control plane eliminates all risk. It reduces sprawl, but it does not remove prompt injection, unsafe tool use, or downstream data leakage. It also does not help if teams copy model keys into code repositories or bypass the gateway for “temporary” integrations. NHIMG’s reporting on AI Agents: The New Attack Surface shows how quickly organisations lose visibility when access patterns are not centrally governed.
For teams adopting this pattern, the operational test is simple: if the security team cannot disable a model, revoke access, or reconstruct a full request path from one place, the integration is not governed enough for scale. Guidance from NIST AI Risk Management Framework and OWASP suggests central ownership, but implementation still depends on disciplined platform design.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 | Direct LLM integrations expand attack surface and bypassable controls. |
| CSA MAESTRO | T1 | MAESTRO addresses threat modeling and control placement for agentic systems. |
| NIST AI RMF | AI RMF supports governance, mapping, and monitoring for LLM integrations. |
Put policy, routing, and observability in the control plane and enforce it at request time.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on July 4, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
