Databricks Service Principals and Token Security Guide

Databricks automation runs on service principals and tokens, and a single long-lived personal access token or OAuth secret can read every table a job touches. Personal access tokens are enabled by default with a default maximum lifetime of 730 days, OAuth secrets can also live up to 730 days, and production jobs are still often owned by and run as individual engineers. This guide goes deeper than the platform overview: it covers how service principals are created and governed, which credential type to use for which workload, how to move to workload identity federation, the roles that let people act as a service principal, what to log and query in system.access.audit, and a hardening baseline. It is for Databricks platform owners, data platform security teams and NHI programme leads. Facts were checked against Databricks documentation (AWS edition) in October 2026; Azure and Google Cloud editions differ in places, especially for Microsoft Entra ID managed service principals.

Key takeaways

Automation should never run as a person. Databricks recommends service principals for CI/CD, scheduled jobs, integrations and infrastructure as code, and run-as settings so jobs survive leavers.
Federation beats secrets. Workload identity federation lets a pipeline exchange its own identity token for a Databricks OAuth token, which Databricks says removes the need to manage and rotate Databricks secrets.
OAuth secrets are better than PATs, but still secrets. They issue one-hour access tokens and can be scoped to APIs, yet the secret itself can last up to 730 days and must be vaulted and rotated.
Change the PAT defaults. Set maxTokenLifetimeDays well below 730 (Databricks recommends under 90), restrict who can create tokens, and disable PATs in workspaces that do not need them.
The Service principal User role is a privilege. Anyone who holds it can run jobs as that principal, and workspace admins who hold it can create tokens on its behalf.

How identity and access work in Databricks

Service principals are API-only identities. They hold roles, entitlements and data permissions exactly like users, and they exist at two levels.

Account level: account admins add service principals to the account, assign account roles and assign them to workspaces. Manage service principals at account level so the same identity and its permissions are consistent across workspaces.
Workspace level: workspace admins can add service principals to a workspace, grant entitlements and object permissions, and assign the workspace admin role.
Source of truth: service principals can be Databricks-managed or, on Azure, Microsoft Entra ID service principals synchronised through automatic identity management. Databricks documents that SCIM provisioning does not sync service principals, so do not assume your user provisioning flow covers them.
System service principals: Databricks creates system and application service principals tied to resources such as metastores. You cannot create, modify or impersonate them; exclude them from rotation and review processes but keep them in the inventory so they are not mistaken for orphans.
Entitlements: Workspace access, Databricks SQL access, Allow unrestricted cluster creation, Allow pool creation and Consumer access. All workspace users and service principals join the users group, which has traditionally granted Workspace access and Databricks SQL access. Databricks states that from 15 June 2026 newly rolled-out workspaces give the users group no entitlements, so principals must be granted them explicitly.
Data permissions: Unity Catalog privileges on catalogs, schemas, tables, volumes, storage credentials and external locations decide what a service principal can read and write. Grant them to groups that the service principal joins, not to the principal directly, so review and transfer are simpler.

Admin roles and privileged access

Role	Scope	What it can do that matters for NHIs	Control
Account admin	Whole account	Create workspaces, manage all identities, delegate account and workspace admin, enable Unity Catalog	Small named group of people, phishing-resistant MFA through SSO, no service principals except a tightly controlled IaC principal if needed
Workspace admin	One workspace	Manage workspace identities, settings, token policy, compute and policies; always holds Workspace access, unrestricted cluster creation and pool creation	Assigned individually (it cannot be assigned to a group); review quarterly
Metastore admin	Unity Catalog metastore	Manage privileges and ownership of all securable objects	Assign to a group, not a person; never to a general automation principal
Service principal Manager	One service principal	Manage roles on the principal, including granting Manager and User to others; creator becomes Manager automatically	Platform team group; review on each new principal
Service principal User	One service principal	Run jobs as the principal; workspace admins with this role can create tokens on its behalf	Grant to the owning team only; treat as equivalent to holding the principal's credential
Billing and Marketplace admins	Account	Budgets, serverless usage policies, Marketplace listings	Keep separate from identity administration

Account admin who creates a workspace becomes its workspace admin: remove that assignment afterwards if the person should not administer the workspace day to day.
Infrastructure as code principal: a Terraform principal that manages workspaces, identities and grants is effectively an account or workspace admin. Give it federation rather than a secret, restrict its trust to the protected deployment branch or environment, and alert on every use outside a deployment. See the Terraform and IaC Pipeline Credential Security Guide.

Integrations, API tokens and service principals

Choose the credential by where the workload runs, not by what is quickest to set up.

Credential	Lifetime	Best for	Main risk
Workload identity federation (federation policy on a service principal)	No Databricks secret; short-lived OAuth token per exchange	CI/CD and workloads that already have an identity token from their platform	Federation policy that trusts too broad an issuer, subject or audience
Account-wide token federation	IdP-issued tokens exchanged for Databricks tokens	Organisations that want their IdP to issue tokens for all users and principals	IdP policy becomes the control plane; misconfiguration affects the whole account
OAuth M2M (client ID and secret)	Secret up to 730 days, set at creation; access tokens valid one hour	Workloads with no federated identity, partner integrations that support OAuth	Secret theft; up to five secrets per principal can accumulate
OAuth on-behalf-of	No secret created for the principal	Acting as a service principal from an authorised context without minting a secret	Who is allowed to act on behalf; review the Service principal User role
Personal access token	Default maximum 730 days unless reduced	Legacy tools and partner integrations that only support PATs	Long-lived bearer token; auto-revoked only after 90 days unused

Making each credential type safer

Federation policies: bind a service principal's federation policy to a specific issuer, subject and audience, such as one repository and one protected environment, and create separate principals for plan or test stages and for production deployment. Treat the policy like an IAM trust policy. See the Workload Identity Federation Guide.
OAuth secrets: set a short lifetime at creation (90 days or less where the integration can rotate), scope the secret to the APIs the workload uses (for example sql or jobs) rather than all APIs, store it in a secrets manager, and delete unused secrets so a principal never holds more than one or two. The secret is shown once; if it was pasted into a ticket or chat, regenerate it.
Environment variables: the Databricks SDKs and CLI read DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET. Inject them at run time from a vault, never commit them to .databrickscfg files in repositories or images.
PATs: set maxTokenLifetimeDays through the workspace configuration API or CLI (it applies only to new tokens, so existing long-lived tokens must be found and revoked), restrict token creation and use permissions to named groups and service principals, and disable PATs in workspaces that do not need them. Databricks notes that disabling PATs breaks Partner Connect and partner integrations that depend on them, so inventory first.
Secret scopes: jobs that read external credentials should use secret scopes with ACLs limited to the job's principal. Databricks warns that workspace admins, secret creators and permitted users can read secret values, and that [REDACTED] output masking only covers literal values. Redaction is not a security boundary.
Integrations and AI tools: BI tools, ingestion services, MCP servers and agent frameworks should each get their own service principal with group-based Unity Catalog grants. Never let them reuse an engineer's PAT. See the Databricks Mosaic AI and Agent Bricks Security guide for agent-specific controls.

Service principal lifecycle

Request: record owner team, purpose, workspaces, data needed and credential type. Create at account level with a naming convention that shows owner and environment.
Grant: add to groups that hold the Unity Catalog privileges and entitlements; give the Manager role to the platform team and the User role to the owning team only.
Credential: federation first, OAuth secret second with a short lifetime, PAT only with a documented exception.
Review: quarterly, confirm owner, last use, grants and credentials. Remove principals that have not authenticated in 90 days after confirming with the owner.
Retire: delete secrets and tokens, transfer ownership of jobs and objects to a group, then remove the principal from workspaces and the account.

Logging and detection

Enable system tables and query system.access.audit. Its key columns are user_identity, service_name, action_name, source_ip_address, request_params and response. The audit log reference groups token management and authentication events under the accounts service, OAuth secret generation, listing and deletion under servicePrincipalCredentials, secret scope activity under secrets and job changes under jobs. Query distinct action_name values in your own account to confirm exact event names before building rules.

Detection	Signal	Why it matters
New OAuth secret or PAT for a service principal	`servicePrincipalCredentials` and `accounts` token events, especially outside change windows	Persistence through extra credentials (ATT&CK T1098.001 Additional Cloud Credentials)
Service principal used from a new network	`source_ip_address` outside known runner and integration ranges	Stolen token or secret in use (T1528 Steal Application Access Token, T1550.001 Application Access Token)
Interactive behaviour by an automation principal	Notebook or SQL editor activity, unusual `service_name` mix for that principal	Someone using a principal's credential by hand
Run-as or permission changes	`jobs` setting changes, Service principal User grants, Unity Catalog grant events	Privilege escalation by running code as a more privileged principal
Bulk data reads	Unity Catalog read volume per principal against its baseline	Exfiltration (T1530 Data from Cloud Storage)
Secret reads from unexpected principals	`secrets` events	Credential harvesting from scopes

Use the Token Management API (databricks token-management list) to export all tokens, their creators and scopes into your NHI inventory, and the account console token report to filter and revoke at account level. Databricks emails users about seven days before PAT expiry and sends service principal token expiry notices to workspace admins; route those to the owning team.

Metrics: percentage of automation authenticating with federation; number of PATs, with maximum remaining lifetime; service principals with an OAuth secret older than 90 days; production jobs whose run-as identity is a person; service principals with no named owner team; principals holding direct (non-group) Unity Catalog grants.

Hardening baseline

Inventory (week 1): export service principals, tokens, OAuth secrets, run-as identities of all jobs and pipelines, and Service principal User assignments.
Set token policy (audit first): set maxTokenLifetimeDays to 90 or less and restrict token permissions to named groups. Measure who creates PATs for two weeks before revoking long-lived ones.
Move jobs off people: set run-as to service principals for every production job and pipeline, owned by a team group.
Pilot federation: move one CI/CD deployment to workload identity federation, run in parallel with the existing secret for a release cycle, then delete the secret.
Replace PATs: migrate integrations to OAuth M2M or federation; keep a dated exception list for those that only support PATs.
Network controls: apply context-based ingress, which Databricks recommends as the primary control and which combines identity, request type and network source, or IP access lists (Enterprise tier) so automation principals can only connect from known ranges. IP access lists do not cover PrivateLink traffic.
Disable PATs in workspaces where the inventory shows no remaining need.

Rollback: if a token policy change breaks an integration, re-enable PAT use for that principal or group only, with an expiry date, rather than relaxing the workspace policy.

Small versus large estates: a single-workspace team can reach most of the baseline with account-level service principals, OAuth secrets in a vault, a 90-day PAT limit and run-as on all jobs. Multi-workspace estates should add federation for all CI/CD, account-wide token federation if the IdP can carry policy, context-based ingress across workspaces, and automated inventory reconciliation into the NHI programme.

Common misconfigurations

Production jobs owned and run by an engineer: the job breaks when the person leaves, or keeps running with their personal access to everything they could read.
Default 730-day tokens: PATs and OAuth secrets created with maximum lifetime and never rotated, relying on the 90-day unused auto-revocation, which never fires for a token in daily use.
Service principal User granted widely: broad groups can run arbitrary jobs as a highly privileged principal.
One principal for everything: a shared "etl" principal with all-apis OAuth secrets and wide Unity Catalog grants used by several teams and tools.
Credentials in notebooks and config files: secrets pasted into notebooks or .databrickscfg committed to Git, relying on redaction for protection.
Data platform sign-in without strong controls: the Snowflake Breach showed how stolen credentials without MFA or network restrictions opened customer data warehouses at scale; the same lesson applies to Databricks tokens used without network or context-based ingress limits.

Framework mapping

Framework	Reference	Databricks control
NIST SP 800-53 Rev. 5	AC-2 Account Management	Service principal lifecycle, ownership and quarterly review
NIST SP 800-53 Rev. 5	AC-6 Least Privilege	Group-based Unity Catalog grants, scoped OAuth secrets, restricted Service principal User
NIST SP 800-53 Rev. 5	IA-5 Authenticator Management	Token lifetime limits, secret rotation, federation
NIST SP 800-53 Rev. 5	AC-17 Remote Access; AU-6 Audit Record Review, Analysis, and Reporting	Context-based ingress and IP access lists; audit table detections
NIST CSF 2.0	PR.AA-01, PR.AA-05, DE.CM-03	Credential management, least privilege, activity monitoring
CIS Controls v8.1	Control 5 Account Management; Control 6 Access Control Management; Control 8 Audit Log Management	Principal inventory, scoped access, audit logs
MITRE ATT&CK	T1528; T1550.001; T1098.001; T1078.004; T1530	Token theft and use, extra credentials, valid cloud accounts, data access

Practitioner checklist

Create all automation identities as account-level service principals with an owner team and naming convention.
Run every production job and pipeline as a service principal, owned by a group.
Use workload identity federation for CI/CD; bind each federation policy to a specific issuer, subject and audience.
Give OAuth secrets short lifetimes and API scopes, store them in a vault and remove spare secrets.
Set maxTokenLifetimeDays to 90 or less, restrict PAT permissions, revoke old tokens and disable PATs where unused.
Grant Service principal User only to the owning team, and review Manager and User roles quarterly.
Restrict automation principals by network with context-based ingress or IP access lists.
Build detections in system.access.audit for new credentials, new source IPs, run-as changes and bulk reads.
Feed service principals, tokens and secrets into the NHI inventory and reconcile monthly.

Databricks Service Principals and Token Security Guide: OAuth, PATs, Federation and Run-As Controls