A Databricks account gives a small number of admins, and a much larger number of service principals and tokens, reach into an organisation's data, compute and AI models at once. Access is split across three layers: the account console, each workspace, and Unity Catalog for data. A setting secured in one layer can be undone in another, and long-lived personal access tokens (PATs) still work by default in every workspace. This guide explains how identity and access work in Databricks, which admin roles matter, how to manage service principals, OAuth and tokens as non-human identities, what to log, and a hardening baseline. It is for platform owners, cloud security engineers and identity teams. Facts come from Databricks documentation for the AWS deployment; Azure and Google Cloud differ mainly in networking and identity provider integration.
Key takeaways
- Three layers, three admin sets. Account admins, workspace admins and metastore admins each hold separate powers, and an account admin can grant themselves workspace admin at any time.
- Service principals should run automation. Jobs and pipelines that run as a person fail when that person leaves and carry that person's access; run them as service principals with the Service Principal User role tightly assigned.
- Prefer federation to secrets. Workload identity federation removes OAuth secrets and PATs entirely; OAuth M2M secrets (up to 730 days) are next best; PATs are the legacy option.
- PATs are on by default. Restrict who can create them, cap lifetime well below the 730-day default, and monitor the token report.
- Unity Catalog is the data control plane. Ownership, the MANAGE privilege, storage credentials and external locations decide who reaches the underlying cloud storage.
How identity and access work in Databricks
| Layer | What it governs | Who administers it | Key objects |
|---|---|---|---|
| Account | Workspaces, identities, metastores, account-wide settings, billing | Account admins (account console at accounts.cloud.databricks.com on AWS) | Users, groups, service principals, federation policies, IP access lists, token report |
| Workspace | Notebooks, jobs, clusters, SQL warehouses, workspace settings | Workspace admins (members of the reserved admins group) | Entitlements, object ACLs, cluster policies, secret scopes, PAT settings |
| Unity Catalog | Data and AI assets across workspaces | Metastore admins and object owners | Catalogs, schemas, tables, volumes, models, storage credentials, external locations, connections, shares |
- Human sign-in: Databricks recommends single sign-on with unified login (one SSO configuration for the account and all workspaces), with MFA enforced by the identity provider. Without SSO, users can sign in with an external account or a one-time passcode, which is not what an enterprise wants.
- Provisioning: automatic identity management lets admins add users, groups and service principals from the identity provider, which stays the source of record; SCIM syncs users and groups but not service principals. On Azure, Microsoft Entra ID service principals can be synced automatically. See the SCIM Provisioning Guide.
- Manage identities at account level: with Unity Catalog, Databricks says identities should be managed in the account console and then assigned to workspaces, so one deprovisioning action removes access everywhere.
- Data access: users start with no data access in a metastore. Unity Catalog is enabled automatically for workspaces created after 8 November 2023; older workspaces need upgrading, and until they are, legacy table ACLs and cluster-level cloud credentials may still control data.
- Emergency access: account admins can designate up to 20 users who can sign in with password plus MFA if SSO fails. Treat these as break-glass accounts: few, monitored, credentials vaulted.
Admin roles and privileged access
| Role | Can do | Main risk | Control |
|---|---|---|---|
| Account admin | Create workspaces, create the Unity Catalog metastore, manage identities, enable system tables, manage previews, assign account and workspace admin roles | Full tenant control, including granting itself workspace admin | Small named group, phishing-resistant MFA in the IdP, alert on setAdmin |
| Workspace admin | Manage workspace identities and entitlements, settings, clusters, SQL warehouses, cluster policies, instance profiles, PAT settings; default owner of the workspace catalog | Can read secrets in any scope, create tokens for service principals they use, change security settings | Assign to individuals (the role cannot be assigned to groups), review quarterly, separate from data ownership |
| Metastore admin | Manage privileges and ownership on all securables in a metastore; transfer ownership of views, functions and models to any principal | Data-wide access by grant | Assign to a group, not a person, and log grant changes |
| Object owner / MANAGE holder | All privileges on the object; grant privileges to others; transfer ownership | Owner sprawl; leavers owning production catalogs | Groups as owners, ownership reviews |
| Billing and Marketplace admins | Budgets and serverless usage policies; Marketplace listings | Low data risk, cost and publishing risk | Restrict to finance and data product owners |
- The account admin who creates a workspace becomes its workspace admin. Other account admins have no workspace access by default but can grant it to themselves, so account admin is the real top privilege.
- Workspace admin is close to data admin: it can read secret contents, manage cluster policies and instance profiles (which may carry cloud storage roles), and owns the workspace catalog by default. Treat it as privileged access. See the Privileged Access Management Guide.
- Who can grant data: the object owner, the owner of the containing catalog or schema, a holder of
MANAGE, or a metastore admin. AREVOKEsucceeds even if nothing was granted, so verify revocation by checking effective grants. - Storage credentials and external locations sit directly under the metastore and wrap cloud IAM roles or managed identities. Whoever can create tables on an external location can expose that storage path; restrict privileges on them as tightly as cloud IAM itself.
Integrations, API tokens and service principals
Databricks automation authenticates in one of four ways. They are listed below from strongest to weakest for unattended workloads.
| Method | Credential stored by the workload | Lifetime | Use for |
|---|---|---|---|
| Workload identity federation (service principal federation policy) | None: the CI/CD or cloud runtime token is exchanged for a Databricks OAuth token | Short-lived OAuth token | CI/CD pipelines and external workloads; Databricks calls it strongly recommended |
| OAuth M2M (client credentials) | Client ID and OAuth secret | Access token 1 hour; secret up to 730 days; up to 5 secrets per service principal | Workloads where federation is not available |
| OAuth U2M on behalf of a service principal (Beta) | None; user signs in interactively | Session-bound | Humans who manage a service principal and need to act as it without a secret |
| Personal access token | Bearer token | Default maximum 730 days; Databricks recommends under 90 | Legacy integrations, Partner Connect and partner tools that require PATs |
- Service principals: API-only identities for CI/CD, scheduled jobs, cross-system integrations, testing and infrastructure as code. Running a job as a service principal stops it failing when a user leaves and stops it carrying that user's broader access.
- Service Principal Manager and Service Principal User: managers administer roles on the principal; users can run jobs as it. Managers do not inherit the user role. Anyone holding Service Principal User effectively holds the principal's data access, so treat assignment of that role as a privilege grant. Workspace admins with the user role can create tokens on the principal's behalf.
- OAuth secrets: shown once at creation, can be scoped (for example
sqlorjobsinstead of the defaultall-apis) but scopes cannot be changed later, and they trigger no expiry notifications, so track them yourself. Use environment variablesDATABRICKS_HOST,DATABRICKS_CLIENT_IDandDATABRICKS_CLIENT_SECRETfrom a secrets manager, never a committed config file. - PAT controls: workspace admins can disable PATs entirely (existing tokens then stop working but are not deleted, and work again if PATs are re-enabled), set fine-grained token permissions on who may create or use tokens, and cap lifetime with
maxTokenLifetimeDays(applies only to new tokens). Databricks automatically revokes PATs unused for 90 days or more. Account admins can review, export and revoke tokens in Security > Token report; workspace admins can usedatabricks token-management listanddelete. - Secret scopes: values returned by
dbutils.secrets.get()are shown as[REDACTED]in notebook output, but redaction applies only to literal values; transformed secrets can be printed, and workspace admins, creators and permitted users can read them. Align scopes with applications, not people, and restrict who can run code with access to them. Scope names are readable by all workspace users. - MCP servers and AI agents: agents and MCP servers that query Databricks act through a token or service principal like any other integration. Give each its own service principal with narrow Unity Catalog grants rather than a user PAT. See the MCP Security Guide.
The 2024 Snowflake customer breaches are the closest warning for any data platform: UNC5537 used infostealer-harvested credentials against around 165 organisations whose accounts had no MFA and no network allow lists. Long-lived PATs on unmanaged developer laptops present the same exposure in Databricks. See the NHI Authentication Guide and the Workload Identity Federation Guide.
Logging and detection
Audit events are available in the system.access.audit system table (account admins enable system tables) and through audit log delivery. Key columns are service_name, action_name, user_identity, source_ip_address, response and request_params. Account-level events carry workspace_id 0, most audit data stays in the workspace's region, and when a workspace is deleted its audit events older than 14 days are removed from the table, so export to your own store. Filter on event_date for performance.
Event (service accounts) | What to alert on | ATT&CK |
|---|---|---|
login, tokenLogin, mfaLogin, oidcTokenAuthorization | Logins from new countries, VPN or hosting ranges; token logins from unexpected IPs | T1078.004 |
generateDbToken, revokeDbToken, garbageCollectDbToken | Tokens created for service principals outside change windows; long-lifetime tokens | T1098.001, T1528 |
setAdmin, removeAdmin | Any admin grant; correlate with change tickets | T1078.004 |
addPrincipalToGroup, removePrincipalFromGroup | Additions to admin or data-owner groups | T1078.004 |
createIpAccessList, updateIpAccessList, deleteIpAccessList, IpAccessDenied | Weakening of network restrictions; repeated denials suggesting credential testing | T1078.004 |
Add Unity Catalog events (service unityCatalog, for example getTable) to spot unusual bulk reads against sensitive tables (T1530 Data from Cloud Storage covers the storage side), and watch for data shared to new external recipients. Some SQL definition fields in request_params are hidden unless the viewer is an account admin or in databricks_pii_access. See the Detection Engineering Guide.
Metrics:
- Number of active PATs, and percentage with lifetime over 90 days (target: falling to zero).
- Percentage of production jobs running as service principals rather than users.
- Percentage of service principals authenticating by federation rather than stored secrets.
- Count of account and workspace admins per workspace, reviewed quarterly.
- Percentage of catalogs and schemas owned by groups rather than individuals.
- Open Security Analysis Tool findings by severity and age.
Hardening baseline
| Area | Setting | Why |
|---|---|---|
| Sign-in | SSO with unified login; MFA in the IdP; emergency access users limited and vaulted | Removes password-only access paths |
| Identities | Account-level identity management via IdP; no local users | Single deprovisioning point |
| Tokens | Token permissions restricted to named groups; maxTokenLifetimeDays 90 or less; PATs disabled where no integration needs them | Limits long-lived bearer tokens |
| Automation | Service principals with federation policies; OAuth secrets scoped and tracked where federation is not possible | No user credentials in pipelines |
| Network | Workspace and account IP access lists; front-end PrivateLink for user access; back-end PrivateLink and customer-managed VPC for classic compute; serverless network policies to restrict egress | Stolen credentials fail from outside; reduces exfiltration routes |
| Compute | Cluster policies; cluster-scoped init scripts managed through policies instead of global init scripts | Prevents untrusted bootstrap code and over-privileged instance profiles |
| Data | Unity Catalog on every workspace; groups as owners; workspace-catalog bindings so production catalogs are only reachable from production workspaces; row filters and column masks on sensitive tables | One enforcement point for data |
| Logging | System tables enabled; audit logs exported and retained outside Databricks | Evidence survives workspace deletion |
| Posture | Security Analysis Tool (open source, from Databricks industry solutions, not covered by support SLAs) run on a schedule | Continuous check against Databricks best practices |
Rollout order
- Baseline: enable system tables and export audit logs; run the Security Analysis Tool; pull the token report.
- Admins: reduce account and workspace admins to named individuals, enforce SSO and MFA, set up emergency access users.
- Tokens in audit mode: use audit logs to find every PAT in use and its caller before restricting. Set
maxTokenLifetimeDaysfirst (affects only new tokens, so nothing breaks), then restrict token permissions. - Migrate automation: create service principals per pipeline, move jobs to run as them, then switch to federation or OAuth M2M. Pilot on non-production workspaces first.
- Disable PATs where no remaining dependency exists. Rollback is simple: re-enabling restores non-expired tokens, which is also why disabled tokens should be revoked once migration is confirmed.
- Network and data: apply IP access lists (test in a single workspace, keep an admin path open), then PrivateLink and catalog bindings.
Small versus large deployments: one or two workspaces can be run with IP access lists, SSO and strict token settings. Many workspaces across business units need account-level identity management, catalog bindings, PrivateLink and a central log pipeline, and benefit most from per-domain catalogs with group ownership.
Common misconfigurations
- Production jobs run as a person: the job carries that user's access, breaks when they leave, and often uses their PAT.
- Default token settings left in place: anyone can create PATs valid for up to two years, including on unmanaged laptops; infostealer exposure then becomes platform access, the pattern behind the Snowflake customer breaches.
- Service Principal User granted broadly: many users can run jobs as a highly privileged principal, which is privilege escalation by design.
- Individuals owning production catalogs: leavers keep ownership, or ownership falls back to admins with no review.
- Cloud credentials on clusters: instance profiles or global init scripts give every user on a cluster the same cloud role, bypassing Unity Catalog. Move storage access to storage credentials and external locations.
- Secrets assumed safe because redacted: notebook redaction only hides literal values; anyone who can run code with scope access can extract the secret.
- Logs left in the platform: audit data not exported, so a deleted workspace or a compromised admin removes the evidence.
Framework mapping
| Control area | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | CIS Controls v8.1 | ISO/IEC 27001:2022 Annex A |
|---|---|---|---|---|
| Identity lifecycle and service principals | AC-2, IA-5 | PR.AA-01 | 5.5, 6.1, 6.2 | 5.16, 5.17 |
| Admin roles and least privilege | AC-6 | PR.AA-05 | 5.4, 6.8 | 8.2 |
| Network restrictions | SC-7 | PR.DS-02 | 12.2 | 8.20, 8.22 |
| Configuration baseline | CM-6 | PR.PS-01 | 4.1 | 8.9 |
| Audit logging and monitoring | AU-2, AU-12, SI-4 | PR.PS-04, DE.CM-09 | 8.2, 8.5 | 8.15, 8.16 |
For AI workloads on the platform, Databricks publishes the Databricks AI Security Framework (DASF) 2.0, which maps AI system risks to controls; use it alongside the platform baseline rather than instead of it.
Practitioner checklist
- Enforce SSO with unified login and MFA in the identity provider; limit emergency access users and vault their credentials.
- Manage users, groups and service principals at account level from the identity provider.
- Keep account admins and workspace admins to a small named list and alert on
setAdmin. - Run every production job as a service principal and restrict the Service Principal User role.
- Use workload identity federation for CI/CD; where not possible, OAuth M2M with scoped secrets tracked for expiry.
- Set
maxTokenLifetimeDaysto 90 or less, restrict token permissions, and disable PATs where nothing depends on them. - Assign catalog and schema ownership to groups and bind production catalogs to production workspaces.
- Apply IP access lists and PrivateLink, and restrict serverless egress with network policies.
- Enable system tables, export audit logs and alert on token, admin and IP access list events.
- Run the Security Analysis Tool on a schedule and track findings to closure.
Standards and references
- Databricks: Administration concepts
- Databricks: Service principals
- Databricks: Authorize service principal access with OAuth
- Databricks: Authenticate access using OAuth token federation
- Databricks: Monitor and revoke personal access tokens
- Databricks: Manage privileges in Unity Catalog
- Databricks: Audit log system table reference
- Databricks Industry Solutions: Security Analysis Tool
Related NHI Mgmt Group resources: Enterprise Data Lake Security Guide · Snowflake Security Guide · Service Account Security Guide · Data Security Guide · Cloud Security Guide