How do security teams evaluate GenAI frameworks in code without losing control of sensitive data?

Why This Matters for Security Teams

GenAI frameworks in code are not just another dependency concern. They can change how prompts, embeddings, retrieval data, API keys, and model outputs move through an application, which makes sensitive data exposure easier to miss during ordinary software review. That risk is amplified when frameworks connect to secrets, logs, plugins, or external tools. Current guidance from NIST AI 600-1 GenAI Profile and the NIST Cybersecurity Framework 2.0 points teams toward risk-based evaluation, but the practical challenge is mapping where the framework touches real data paths. NHIMG research also shows why this matters: in the State of Non-Human Identity Security, only 1.5 out of 10 organisations were highly confident in securing NHIs. In practice, many security teams discover data leakage paths only after developers have already wired the framework into production workflows.

How It Works in Practice

Security teams should evaluate GenAI frameworks the same way they evaluate other high-risk code dependencies, but with extra attention to data handling and identity boundaries. Start by inventorying where the framework runs, what it can read, and what it can send. Then trace whether it can access environment variables, secret stores, vector databases, prompt history, telemetry, or third-party connectors. If the framework has access to sensitive data, the question is not only whether it is vulnerable, but whether it can expose data through prompts, logs, caching, debug output, or tool execution.

A useful review sequence is:

• Confirm whether the framework stores or forwards prompts, embeddings, and completion traces.
• Check whether secrets are passed through code, config files, or runtime variables.
• Validate whether access is scoped to the minimum data set needed for the use case.
• Require logging controls that avoid capturing credentials, tokens, or regulated data.
• Test whether retrieval and plugin features can reach systems beyond the intended boundary.

Framework-level review should also include identity and workload controls. If a GenAI framework uses service accounts, agent connectors, or tool access, those identities should be treated as privileged non-human identities and governed accordingly. NHIMG’s Top 10 NHI Issues is a useful companion for spotting over-privilege, weak rotation, and missing monitoring. For implementation guidance, teams can pair that with NIST SP 800-53 Rev 5 Security and Privacy Controls to map data protection, access control, and audit requirements into the review process. These controls tend to break down when developers allow the framework to inherit broad environment access in containerized or serverless deployments because the runtime boundary becomes too permissive.

Common Variations and Edge Cases

Tighter framework review often increases delivery overhead, so organisations have to balance developer speed against exposure risk. That tradeoff is especially visible in retrieval-augmented generation, internal copilots, and agentic workflows where the framework needs broad read access to be useful. Current guidance suggests treating those cases as higher risk by default, but there is no universal standard for how much data access is acceptable across all use cases.

One common edge case is open-source code that is safe in isolation but becomes risky once teams add plugins, vector stores, or telemetry packages. Another is local development, where engineers assume non-production data is harmless and end up exposing secrets through notebooks, traces, or shared debug artifacts. The DeepSeek breach illustrates how quickly sensitive records can become visible when AI systems are misconfigured or exposed. For governance maturity, the Ultimate Guide to NHIs — Regulatory and Audit Perspectives helps teams align framework review with audit evidence, retention, and accountability. The right control is not to ban GenAI frameworks outright, but to classify where they can operate, what they can touch, and what must be blocked before code reaches production.

Related resources from NHI Mgmt Group

• How should security teams implement cloud data loss prevention in Google Cloud environments without losing control of sensitive data elsewhere?
• How should security teams automate access governance with Infrastructure as Code without losing control over sensitive approvals?
• How should security teams use AI-generated code fixes without losing control of AppSec risk?
• How should security teams scale GenAI applications in production without losing reliability or control?