Agents and model endpoints on Databricks run a few steps away from the most sensitive data a company holds, so a single over-scoped service principal or a missing egress rule turns a chatbot into a data exfiltration path. This guide covers the AI layer of Databricks: Agent Bricks agents, Model Serving endpoints, the Unity Gateway that governs models and MCP tools, AI Search indexes and the logs they produce. It assumes the workspace basics (SSO, SCIM, Unity Catalog, token policy) are already handled, as described in the Databricks Security Guide, and goes deeper on how agents authenticate, what they can reach and how to detect misuse. Databricks has renamed several features over 2025 and 2026: older material calls the gateway "Mosaic AI Gateway" and AI Search "Vector Search", and the current documentation uses Agent Bricks, Unity Gateway and AI Search. Feature names and preview status here were checked against Databricks documentation on 27 October 2026. It is written for platform engineers, data security leads and anyone approving an agent for production.
This is an independent guide from NHI Mgmt Group, based on Databricks's public documentation as of November 2026. It is not sponsored or reviewed by Databricks. Product features change, so confirm settings against Databricks's current documentation.
Key takeaways
- Every agent has two possible identities. App authorization runs the agent as its own service principal shared by all users; user authorization (Public Preview) runs it with each caller's downscoped credentials. Pick per resource, not per agent.
- Unity Catalog is the permission plane for AI too. Models, MCP services, functions used as tools, AI Search indexes and inference tables are Unity Catalog securables, so the same grants, tags and reviews apply.
- The gateway is where you enforce cost and guardrails. Unity Gateway applies consumption limits, budgets, service policies and fallbacks across model and MCP services; leaving it unconfigured means no ceiling on spend or tool use.
- Egress control is the exfiltration backstop. Serverless network policies in Restricted Access mode limit model serving and apps to listed FQDNs and storage, which blunts prompt-injection-driven data leaks.
- Logs exist but need joining. system.access.audit records endpoint, index, MCP and secret changes; inference tables hold the payloads. Correlate them by principal to see who asked an agent to do what.
How identity and access work in Databricks Mosaic AI and Agent Bricks
Agent Bricks is Databricks' agent developer platform. Agents run on Agent Runtime (managed compute with a durable agent server), are called through an invocation API, the Responses API or SQL, keep memory in Lakebase-backed stores, reach tools through managed, custom or external MCP servers, and reach models through Unity Gateway. Generated code runs in Databricks Sandbox with scoped access to governed data. Every one of those hops is an authorisation decision.
The two agent authentication modes
| Mode | Identity used | Status | Best for | Main risk |
|---|---|---|---|---|
| App authorization (previously service principal authentication) | A service principal Databricks creates automatically with the app; all users share its permissions | Default | Shared, non-sensitive resources: a common AI Search index, a public reference table, a model endpoint | Every user inherits the service principal's full access, so one prompt from any user can reach anything the principal can |
| User authorization (previously on-behalf-of, OBO) | The requesting user's credentials, downscoped to the API scopes added to the app, forwarded in the x-forwarded-access-token header | Public Preview; a workspace admin must enable it | Row-level or user-specific data: HR tables, customer records, Genie spaces | If the forwarded token is missing, get_user_workspace_client() silently falls back to the app's service principal |
Databricks documents combining both in one agent: a shared index under app authorization and user-specific tables under user authorization. Resources that support user authorization include SQL warehouses, Genie, files and directories, Model Serving endpoints, AI Search indexes, Unity Catalog connections and Unity Catalog tables. Call get_user_workspace_client() inside the invoke or stream handler, never at startup, because the user's credentials only exist at query time.
- Test for silent fallback: Databricks recommends a temporary whoami tool that returns the effective identity, so you can prove user authorization is working before production, then remove it.
- Never log the forwarded token: it is a live credential. Log the user identity and the action, not the header.
- MCP services: the caller needs EXECUTE on the service plus USE CATALOG and USE SCHEMA on its parents. Built-in system.ai.* services are granted to account users by default, so review whether that default suits you. On-behalf-of access to MCP services needs the ai-gateway user API scope, and the first call returns JSON-RPC error -32042 with a login URL for the user to authorise.
For the general model of agents acting with delegated user authority, see the Delegation Chains and Attenuated Authority for AI Agents and the AI Agent Authorisation Guide.
Admin roles and privileged access
AI features add new privileged actions on top of the account admin, workspace admin and metastore admin roles covered in the platform guide. Treat these as privileged and restrict them to named groups.
| Privileged action | Permission involved | Why it matters | Control |
|---|---|---|---|
| Enable user authorization for apps | Workspace admin | Changes how every agent can act for users | Change ticket, test with whoami, record in change log |
| Change a serving endpoint's config, AI Gateway settings or permissions | CAN MANAGE on the endpoint | Can swap the model, disable logging or widen query access | Few holders; alert on serverlessRealTimeInference permission and config events |
| Query a serving endpoint | CAN QUERY | Anyone with it can use the model, and through an agent, its tools | Grant to groups or the agent's service principal, never to all users by default |
| Grant model, MCP or tool access in system.ai | GRANT policies with governed tags | Controls which principals can use which models and MCP services | Tag-based policies owned by the platform team, reviewed quarterly |
| Enable inference tables | MANAGE on the model service, CREATE TABLE, USE CATALOG and USE SCHEMA on the target | The table holds raw prompts and responses | Put tables in a restricted schema; the table owner is the endpoint creator, so transfer ownership to a group |
| Read or change secret scopes | Secret scope ACLs | Holds provider API keys used by endpoints | Separate scopes per agent; alert on ACL changes |
Run permission changes through infrastructure as code where possible, and keep CAN MANAGE on production endpoints with a deployment service principal rather than individual developers. See the Privileged Access Management Guide.
Integrations, API tokens and service accounts
- One service principal per agent or app: Databricks' own guidance is a dedicated service principal per app with least privilege. Shared principals across agents make it impossible to attribute actions or revoke one agent cleanly.
- Grants for app authorization: give the principal exactly what it needs: Can Use on the SQL warehouse, Can Query on the serving endpoint, Can Run on a Genie space, and SELECT or EXECUTE on specific Unity Catalog objects. Avoid catalog-wide grants.
- Provider keys in secrets, never plain variables: serving endpoint environment variables reference secrets as {{secrets/scope/key}}. If the syntax is wrong, the value is treated as a plain text variable, so a typo can publish a credential in configuration. The endpoint creator needs READ on the secret.
- External models through the gateway: Unity Gateway supports external providers with bring your own key. Keep those keys in a dedicated secret scope and rotate them on a schedule; see the AI Agent Credential Lifecycle and Rotation Guide.
- Personal access tokens: do not run production agents on a person's token. Use OAuth for service principals and track token events under the accounts service in audit logs. See the Databricks Service Principals and Token Security Guide.
- MCP servers and HTTP connections: external MCP servers and APIs are registered as Unity Catalog connections. Each connection stores credentials for a third party, so treat it like a secret: owner, scope, rotation, review.
Give agent service principals narrow grants and egress restrictions, so a stolen credential reaches as little data as possible.
Logging and detection
Three sources matter: the audit log system table, inference tables and MLflow traces.
| Source | What it shows | Watch for |
|---|---|---|
| system.access.audit, service_name serverlessRealTimeInference | Serving endpoint create, update, delete, start and stop, permission changes, usage policy and AI Gateway configuration changes | Permission widening, AI Gateway or logging config changes outside change windows |
| system.access.audit, service_name vectorSearch | AI Search endpoint and index create or delete, ACL changes, query, scan, upsert and delete | Bulk scans by an agent principal, upserts from unexpected principals (index poisoning) |
| system.access.audit, service_name mcpService and aiGatewaySkills | MCP requests through Unity Gateway; skill fetch, list and load | New MCP services used by an agent for the first time, spikes in tool calls |
| system.access.audit, service_name apps and secrets | App deploy and access changes; secret scope and ACL changes, secret reads | Access updates on agent apps, secret reads by new principals |
| system.access.audit, service_name accounts | Token generation, revocation and garbage collection | Personal tokens created for agent use |
| Inference tables | Raw request and response JSON, requester ID, status code, latency, destination model | Prompts containing injection markers, responses containing secrets or bulk records |
| MLflow Tracing | Each agent step and tool call, stored and governed in Unity Catalog | Tool sequences that read data then call an external URL |
- Know the gaps: inference tables skip payloads over 10 MiB, rows can take up to an hour to arrive, and requests that return 401, 403, 429 or 500 may not be logged. Do not use them as the only record of denied or failed calls.
- Protect the logs: inference tables hold whatever users typed, including personal data. Apply Unity Catalog grants and masking as you would for the source data.
- Query on event_date: filter audit queries on event_date for performance, and keep user_identity and source_ip_address for correlation.
See the AI Agent Observability and Incident Response Guide.
Hardening baseline
Phased rollout
- Inventory: list every serving endpoint, agent app, AI Search index, MCP service and Unity Catalog connection, with owner and service principal. Use the audit log to find endpoints nobody claims.
- Identity: give each agent its own service principal; move user-specific data to user authorization; prove it with the whoami tool in a test workspace.
- Grants: replace catalog-wide grants with object grants; review system.ai defaults and restrict foundation model access to the groups that need it.
- Gateway: set consumption limits and budgets with per-user hard caps; attach guardrail service policies to model and MCP services; configure fallbacks only to approved models.
- Egress: create a serverless network policy in dry-run mode, review what would be blocked, add the model provider DNS names you use, then switch to Restricted Access. Roll back by returning the workspace to its previous policy.
- Logging: enable inference tables into a restricted schema, transfer table ownership to a group, and build detections on the audit services above.
- Production gate: no agent goes live without a named owner, threat model, egress policy and a tested kill switch (stop the endpoint or revoke CAN QUERY). See the AI Agent Kill Switch and Containment Guide.
Decision guidance
- App versus user authorization: use app authorization when every caller should see the same data; use user authorization when results must respect each user's table, row or column permissions. Because user authorization is in Public Preview, weigh support terms before relying on it for regulated data, and keep the sensitive part of an agent off until it is enabled.
- Small teams: start with one restricted egress policy for all serverless workloads, one secret scope per agent and gateway budgets. Most of the value is there.
- Large organisations: separate workspaces for development and production agents, tag-based GRANT policies in system.ai managed centrally, and audit logs exported to the SIEM for cross-platform correlation.
Framework mapping
| Control area | Databricks feature | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | Threat reference |
|---|---|---|---|---|
| Per-agent identity and grants | App authorization service principal, Unity Catalog grants | AC-2, AC-6 | PR.AA-05 | ATT&CK T1078.004 Cloud Accounts |
| Delegated user access | User authorization with downscoped tokens | AC-6 | PR.AA-05 | OWASP LLM06:2025 Excessive Agency |
| Credentials for providers and tools | Secret scopes, {{secrets/scope/key}} | IA-5 | PR.AA-05 | ATT&CK T1552.001 Credentials In Files |
| Egress restriction | Serverless network policies, Restricted Access | SC-7 | PR.DS-01 | ATLAS AML.T0085 Data from AI Services |
| Guardrails and limits | Unity Gateway service policies, consumption limits, budgets | AC-6 | DE.CM-09 | ATLAS AML.T0051 LLM Prompt Injection; OWASP LLM10:2025 Unbounded Consumption |
| Audit and payload logging | system.access.audit, inference tables, MLflow Tracing | AU-2, AU-12 | DE.CM-09 | OWASP LLM02:2025 Sensitive Information Disclosure |
Common misconfigurations
- Agent principal with catalog-wide SELECT. One prompt from any user can then reach every table in the catalog. Grant objects, not catalogs.
- User authorization assumed but not working. The forwarded token is missing, the client falls back to the service principal and nobody notices. Test with whoami.
- Open egress on serverless. An injected instruction can send data to any URL. The ForcedLeak 2025 case on another platform shows the pattern: hidden instructions in a record, data sent out through an image URL to a domain the platform still trusted.
- Provider key in a plain environment variable. A wrong secret reference syntax stores the key in clear configuration.
- Inference tables in an open schema. Prompts and responses with customer data become readable by anyone with schema access.
- No gateway limits. A looping agent or abused endpoint runs up cost with no hard cap; see the Denial-of-Wallet and AI Agent Cost Runaway Guide.
Measuring it
- Dedicated principals: percentage of production agents with their own service principal (target 100%).
- Grant breadth: number of agent principals with catalog-level or schema-level grants (target falling to zero).
- Egress coverage: percentage of workspaces running serverless in Restricted Access mode.
- Gateway coverage: percentage of model and MCP services with consumption limits and a guardrail policy.
- Detection latency: time from an endpoint permission change to alert.
Practitioner checklist
- Inventory endpoints, agents, AI Search indexes, MCP services and connections with owners.
- Give every agent a dedicated service principal with object-level grants.
- Use user authorization for user-specific data and test it with a whoami tool.
- Keep CAN MANAGE on production endpoints with a deployment principal.
- Store every provider and tool credential in secret scopes and check the reference syntax.
- Set Unity Gateway consumption limits, budgets and guardrail policies.
- Move serverless egress to Restricted Access after a dry run.
- Write inference tables to a restricted schema owned by a group.
- Alert on serverlessRealTimeInference, vectorSearch, mcpService and secrets events.
- Review system.ai default grants and agent permissions quarterly.
Standards and references
- Databricks: Authentication for AI agents
- Databricks: Unity Gateway
- Databricks: Agent Bricks
- Databricks: Serverless egress control with network policies
- Databricks: Inference tables
- Databricks: Audit log reference
- Databricks: Configure access to resources from model serving endpoints
- OWASP: Top 10 for LLM Applications 2025
Related NHI Mgmt Group resources: Databricks Security Guide · Databricks Service Principals and Token Security Guide · Agent Identity and Access Guide · Agent Egress Control and Network Isolation Guide