Cloud-agnostic secrets management for Azure Databricks workloads

At a glance

What this is: This is a practical guide to centralising secrets management for Azure Databricks, with the key finding that cloud-agnostic retrieval can reduce secret scattering across DataOps and MLOps workflows.

Why it matters: It matters because IAM, PAM, and NHI teams need a repeatable way to govern workload secrets across clouds, notebooks, and languages without creating new exposure paths.

By the numbers:

• Only 44% of organisations are currently using a dedicated secrets management system.
• Secrets management is a top five cybersecurity priority for only 33% of organisations, behind cloud security 45%, API security 42%, and endpoint security 36%.

👉 Read Akeyless's analysis of secrets management for Azure Databricks workloads

Context

Secrets management in Databricks is about controlling how workloads retrieve and use credentials inside analytics and machine learning pipelines. In practice, the challenge is not just storage. It is preventing secrets from being scattered across notebooks, configuration files, cloud-specific vaults, and ad hoc handoffs between Python, Scala, R, and SQL.

The governance gap is familiar to IAM and NHI teams: access becomes easy to consume but hard to lifecycle-manage. When a platform-specific pattern ties secrets to a single cloud or workspace, teams often inherit portability problems, inconsistent controls, and weaker offboarding discipline for workload credentials.

A cloud-agnostic approach changes the operating model by separating the workload from the storage location of the secret. That is especially relevant for DataOps and MLOps programmes where the same application logic may move between environments, while the underlying secret still needs central oversight and auditability.

Key questions

Q: How should security teams govern secrets in Azure Databricks workloads?

A: Treat the notebook, workspace, and secret store as one access chain. Require identity-bound retrieval at runtime, avoid hardcoded values in code or configuration, and scope each workload to only the secrets it needs. Governance should cover secret ownership, rotation, and offboarding for the workload, not just storage location.

Q: Why do cloud-specific secret vaults create governance problems for data platforms?

A: Cloud-specific vaults can tie access to one provider or one workspace, which makes migration, cloning, and multi-cloud operation harder to control. That often leads to duplicate secrets, inconsistent permissions, and uneven lifecycle management. The risk is not only technical lock-in. It is fragmented accountability for workload credentials.

Q: What breaks when secrets are passed through notebooks and Spark configuration?

A: The main failure is visibility. Once secrets move through notebook cells and runtime configuration, they can be copied, reused, or inherited by downstream jobs without clear ownership. That makes offboarding, audit, and incident scoping much harder because the credential’s path is no longer obvious.

Q: Should managed identity replace all secrets in Databricks?

A: No. Managed identity reduces the need to embed static credentials, but it does not eliminate access governance. Teams still need least privilege, workload separation, and review of which identities can reach which secrets. A better test is whether the identity is bounded tightly enough to keep blast radius small.

Technical breakdown

How centralized secrets retrieval works in Databricks

A centralized secrets pattern places the credential in a governed repository and exposes it to the workload at runtime through authentication and lookup. In the Databricks example, the notebook authenticates first, then retrieves a secret value, then passes that value into Spark configuration or a downstream API call. This avoids embedding the secret in code, but it does not eliminate the need for strong identity binding between the workspace, the workload, and the vault. The important architectural point is separation of code from credential storage, not elimination of trust.

Practical implication: define which workloads may retrieve which secrets and require identity-backed access rather than notebook-local convenience.

Why cloud-specific vault ties create governance friction

Cloud-native secret stores can work well inside one ecosystem, but they often create operational friction when workloads span clouds or teams. Databricks secret scopes are workspace-bound, and cloud vault integrations can make portability depend on provider-specific access paths. That increases the likelihood of secret duplication, inconsistent policy enforcement, and manual reconfiguration when environments change. For NHI governance, the issue is not just where the secret lives. It is whether the access model survives cloud migration, environment cloning, and application refactoring without creating new standing exposure.

Practical implication: map where secrets become environment-bound and remove unnecessary duplication before workloads spread across cloud boundaries.

Managed identity reduces secret handling without removing identity risk

Azure managed identity shifts authentication away from embedded credentials and toward a cloud-issued workload identity. That reduces direct secret handling in code, but it also moves the governance problem to token issuance, scope, and trust in the identity binding. If the managed identity is over-permissioned, reused broadly, or not separated by workload, the platform still carries privilege risk. For identity teams, this is a classic workload identity pattern: removing the static secret is valuable, but access scope and lifecycle controls remain the deciding factors.

Practical implication: treat managed identity as an access path that still needs least privilege, review, and workload-specific scoping.

NHI Mgmt Group analysis

Cloud-agnostic secrets management is now a workload identity problem, not just a vault problem. Databricks users are not only choosing where to store a secret, they are defining how a workload proves who it is across notebook runtimes, languages, and cloud boundaries. That makes the control surface broader than a single vault or SDK. Practitioners should treat retrieval policy, runtime identity, and secret lifecycle as one governed system.

Secret scattering remains the real failure mode in DataOps and MLOps estates. Once secrets move between Python cells, Spark configuration, API calls, and cloud services, the estate becomes easy to use and difficult to attest. The result is duplicated credentials, opaque ownership, and offboarding gaps when a workspace, pipeline, or service account changes. The practical conclusion is that centralisation matters because it reduces invisible copies, not because it is administratively neat.

Managed identity is helpful only when the underlying workload scope is tightly bounded. Removing hardcoded secrets lowers exposure, but it does not fix over-broad access or identity reuse across applications. In a platform like Databricks, the governance question becomes whether one managed identity is serving one workload or many. IAM and NHI teams should judge the pattern by blast radius, not by whether a password disappeared from code.

Static vs dynamic secrets is the right conceptual split for this use case. The operational value here comes from short-lived retrieval and central policy enforcement, not from relocating a long-lived secret into a different platform. That distinction matters because teams often mistake centralisation for security maturity. The better test is whether the secret can be issued, used, and constrained without becoming a standing credential.

NHI governance should extend into analytics pipelines and not stop at infrastructure boundaries. Data engineering and MLOps teams increasingly depend on machine identities that behave like first-class production actors. If those identities are not reviewed, named, and lifecycle-managed, the result is hidden privilege accumulation inside the analytics stack. The practitioner takeaway is to bring notebook and pipeline identities into the same governance model used for other NHIs.

From our research:

• 54% of organisations are dissatisfied with their current secrets management solution because not all secrets are secured, and 43% cite lack of central management, according to The 2024 State of Secrets Management Survey.
• 88% of security professionals are concerned about secrets sprawl, with 49% of those in larger organisations described as very concerned.
• That concern aligns with the Guide to the Secret Sprawl Challenge, which frames duplication and uncontrolled distribution as the core operational problem.

What this signals

Secret sprawl debt: once credentials are copied into notebooks, configs, and pipeline steps, the governance cost compounds faster than the platform stack changes. IAM and NHI teams should assume that analytics and MLOps environments will create hidden copies unless a central retrieval pattern is enforced.

The next maturity step is not simply adopting a vault. It is proving that workload identities, secret scopes, and offboarding controls still work when data teams refactor code, move clouds, or clone environments for testing. That is where many programmes discover that cloud-agnostic access is easier to promise than to govern.

For teams building a broader machine identity programme, this use case aligns closely with Ultimate Guide to NHIs , Lifecycle Processes for Managing NHIs and OWASP Non-Human Identity Top 10 because the real issue is lifecycle control of workload credentials, not just retrieval mechanics.

For practitioners

• Centralise workload secret retrieval Move Databricks credentials into a governed repository and require runtime lookup rather than embedding values in notebooks, jobs, or configuration files.
• Bind access to workspace-specific identity Use managed identity or equivalent workload identity to tie secret retrieval to a specific workspace and avoid reusable shared credentials across environments.
• Eliminate duplicate secret copies Search for secrets replicated in code, job parameters, Spark configs, and documentation, then remove redundant copies before the pipeline expands further.
• Review over-permissioned workload identities Confirm that each Databricks identity only reaches the secrets it actually needs, with separate permissions for separate pipelines or application contexts.

Key takeaways

• Azure Databricks secrets management becomes a governance issue when workload credentials are scattered across notebooks, configs, and cloud-specific vaults.
• Managed identity reduces hardcoded secret exposure, but access scope and lifecycle control still determine the true blast radius.
• Centralised retrieval only improves security when it is paired with workload-specific privilege, offboarding discipline, and duplicate-secret removal.

Key terms

• Workload Identity: The identity assigned to a software workload — such as a containerised application, serverless function, or microservice — enabling it to authenticate to other services without storing static credentials.
• Secret Scattering: Secret scattering is the uncontrolled spread of credentials across notebooks, config files, pipelines, tickets, and multiple vaults. It creates hidden copies, weak ownership, and inconsistent revocation paths, making governance and incident response harder than the original storage problem.
• Managed Identity: A cloud-provider-managed identity assigned to a compute resource, allowing it to authenticate to cloud services without storing credentials in application code.
• Cloud-Agnostic Secrets Management: Cloud-agnostic secrets management stores and governs credentials in a way that is not tied to one cloud provider or one workspace. The value is portability and central policy enforcement, but only if the access model remains consistent across environments.

What's in the full article

Akeyless's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step Databricks notebook code for authenticating to Akeyless with Azure cloud ID and managed identity.
• Python, Scala, and R retrieval patterns for passing secrets through Databricks runtime without hardcoding values.
• Example API request and Spark table write flow that shows how the retrieved secret is used in an actual workload.
• Implementation sequence for setting up Azure AD authentication and workspace-specific access paths.

👉 Akeyless's full blog post covers the Databricks implementation steps, code samples, and runtime secret retrieval flow.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM or NHI governance programme, it is worth exploring.