MLflow and Kubeflow Security Guide: Identity, Hardening and Detection for MLOps

MLflow tracking servers and Kubeflow clusters hold an organisation's models, training data references, prompts and the cloud credentials that reach them, yet they are often deployed with authentication switched off and published to a network that is wider than intended. Both platforms grew up as data science tools, so their security features are opt-in and their defaults favour a quick start on a laptop. This guide explains how identity and access actually work in self-hosted MLflow and in the Kubeflow Platform, which service accounts and tokens they rely on, what to log, a hardening baseline and the misconfigurations that recur. It is written for platform engineers, ML engineers and security teams who run or review these systems. Model file integrity and provenance are covered in the Model Supply Chain Guide; this page focuses on the platforms that store and serve those models.

Key takeaways

• Authentication is off unless you turn it on. MLflow enables its built-in authentication only when the server starts with --app-name basic-auth; without that flag anyone who can reach the port can read and change experiments, models and artefacts.
• The artefact proxy is a shared cloud identity. When MLflow proxies artefact access, every user of the tracking server inherits the server's storage credentials, so the IAM role or key behind it must be scoped to the artefact bucket and nothing else.
• Kubeflow isolation is namespace isolation, with stated limits. Profiles map users to Kubernetes namespaces, but the Kubeflow Pipelines documentation states there are no hard security guarantees and that pipeline definitions and ML Metadata are shared across namespaces.
• Change the defaults on day one. Kubeflow's reference manifests ship a static Dex user, [email protected], with the password 12341234; earlier MLflow releases shipped an admin password of password1234. Both must be replaced before anything else.
• Patch these servers like internet-facing applications. CISA added an MLflow server-side request forgery flaw (CVE-2026-64849) to the Known Exploited Vulnerabilities catalogue in August 2026, and SSRF from an MLOps server leads directly to cloud metadata credentials.

How identity and access work in MLflow and Kubeflow

The two platforms take very different approaches, which matters when you decide where to enforce controls.

MLflow

• Enabling auth: install mlflow[auth], set MLFLOW_FLASK_SERVER_SECRET_KEY (required for CSRF protection and identical across all servers in a multi-node setup), and start with --app-name basic-auth. Users and permissions persist in the auth database, so omitting the flag after a restart silently removes protection without deleting the users.
• Auth database: database_uri defaults to a local SQLite file, basic_auth.db. Multi-node deployments should point it at a central PostgreSQL database, which then holds password hashes and must be protected like any credential store.
• Pluggable decisions: authorization_function accepts a module:function hook so a proxy-supplied identity or token can be mapped to an MLflow user. Treat the hook as security-critical code with review and tests.
• Browser protections: since MLflow 3.5.0 the default FastAPI server includes security middleware: --allowed-hosts (MLFLOW_SERVER_ALLOWED_HOSTS, default localhost and private IPs) against DNS rebinding, --cors-allowed-origins (default any localhost origin), and --x-frame-options (default SAMEORIGIN). The middleware is not available when the server runs with --gunicorn-opts or --waitress-opts.

Kubeflow

• Authentication path: requests enter through the Istio ingress gateway, where oauth2-proxy acts as the OIDC client. Authorisation is separate and is handled by Kubernetes RBAC and Istio policies.
• Profiles: each Profile has one owner bound to ClusterRole/kubeflow-admin and contributors bound to kubeflow-edit or kubeflow-view. Contributor status is simply the presence of the RoleBinding and AuthorizationPolicy, so anyone who can edit those objects can grant access.
• Header-based identity: Istio AuthorizationPolicies match the kubeflow-userid header and accept traffic only from the ingress gateway and the ml-pipeline-ui service accounts. The header must therefore be set only by the authenticating proxy; any path that lets a client reach workloads without passing the gateway undermines the model.
• Pipelines: the Kubeflow Pipelines API server rejects requests for namespaces the caller cannot access, and runs execute in the user's namespace. The same documentation states that Kubeflow makes no hard security guarantees about Profile isolation, that pipeline definitions are shared across all namespaces, and that artefacts and executions in ML Metadata are shared without access control. Treat a shared Kubeflow cluster as a soft multi-tenancy boundary.

Admin roles and privileged access

• Default MLflow admin password: current releases have no default admin password and refuse to start without one. Earlier releases shipped password1234 (advisory GHSA-gq3w-7jj3-x7gr); current servers reject it and log a startup warning if an admin still uses it. Upgraded deployments must set a new password and restart.
• No login throttling: the MLflow documentation notes the UI has no limit on login attempts and no logout other than closing the browser. Put the server behind SSO or a reverse proxy that adds rate limiting and session control.
• Kubeflow default user: the reference manifests use [email protected] with password 12341234 and say this must be changed for any production deployment. Better still, connect your identity provider and remove static passwords from Dex (enablePasswordDB).

Integrations, API tokens and service accounts

MLOps platforms are dense with non-human identities. Each one below should have an owner, a scope and a rotation plan. See the AI Infrastructure Workload Identity Guide for the wider pattern.

• MLflow artefact store credentials: with proxied artefact access (the default, using --artifacts-destination), the server holds the storage credentials and the MLflow documentation warns that all users with access to the tracking server can reach artefacts through that assumed role. Use a workload identity (IAM role for the pod or VM) rather than AWS_ACCESS_KEY_ID environment variables or AZURE_STORAGE_CONNECTION_STRING, and scope it to one bucket prefix.
• Non-proxied mode: with --no-serve-artifacts and --default-artifact-root, clients write directly to storage and every client needs its own storage credential. That spreads keys to notebooks and CI runners; prefer it only where each client already has federated workload identity.
• Backend store URI: --backend-store-uri postgresql://user:pass@host/db puts a database password on the command line, where process listings and logs can expose it. Supply it from a secret mount or environment injected by your secrets manager.
• Client credentials: ~/.mlflow/credentials stores the password unencrypted, protected only by file permissions. Training jobs should receive short-lived credentials at runtime, not a shared username and password baked into images. See the Secrets in Container Images and Build Artefacts Guide.
• Kubeflow service accounts: each Profile namespace has default-editor (bound to kubeflow-edit) and default-viewer. Pipeline and notebook pods run as default-editor unless told otherwise, so any code in a notebook can act with edit rights across the namespace. Use projected, audience-bound tokens and dedicated service accounts per pipeline where possible. See the Kubernetes Service Account Token Guide.
• Pipelines SDK tokens: kfp.Client with ServiceAccountTokenVolumeCredentials reads a token from KF_PIPELINES_SA_TOKEN_PATH (default /var/run/secrets/kubeflow/pipelines/token). Machine-to-machine access from outside the cluster requires an oauth2-proxy overlay that trusts your cluster's token issuer; review which issuers are trusted.
• Cloud credentials in namespaces: data scientists frequently add cloud keys, Hugging Face tokens and LLM API keys as Kubernetes secrets in their namespace. Inventory them, replace static keys with workload identity federation and scan for them. See the Kubernetes Secrets Security Guide.

Logging and detection

Neither platform gives a complete security audit trail by itself, so build one from the layers around it.

Relevant ATT&CK techniques for detection mapping are Exploit Public-Facing Application (T1190), Cloud Instance Metadata API (T1552.005), Credentials In Files (T1552.001), Deploy Container (T1610) and Compute Hijacking (T1496.001). In MITRE ATLAS, AI Supply Chain Compromise: Model (AML.T0010.003) and AI Artifact Collection (AML.T0035) describe what an attacker does with a compromised model registry.

Metrics that show the controls are working:

• Percentage of MLflow and Kubeflow endpoints that require authentication from every network path (target 100%).
• Number of static cloud or storage keys used by MLOps components (target zero, replaced by workload identity).
• Days from a KEV or vendor security release for MLflow or Kubeflow components to patched production (target within the KEV due date).
• Number of users with MLflow admin or Kubeflow cluster-admin rights, reviewed each quarter.
• Percentage of registered production models whose artefact hash is recorded and verified at deployment.

Hardening baseline

MLflow tracking server

1. Turn on authentication: run with --app-name basic-auth or an SSO plugin, set default_permission to NO_PERMISSIONS for sensitive deployments, and set a strong MLFLOW_FLASK_SERVER_SECRET_KEY from your secrets manager.
2. Bind and restrict hosts: bind to an internal interface, set --allowed-hosts to the exact DNS names users reach, set --cors-allowed-origins to named notebook and app origins, and never use "*" in production. Do not use --disable-security-middleware unless a reverse proxy provides equivalent protection.
3. Terminate TLS in front: place the server behind a reverse proxy or ingress with TLS and, ideally, SSO and rate limiting.
4. Scope the storage role: one bucket or prefix, no list or delete on other buckets, versioning on, and access logs enabled.
5. Move the auth and backend databases to managed PostgreSQL with encryption at rest, private networking and credentials injected at runtime.
6. Block metadata access: enforce IMDSv2 with a hop limit of 1 on AWS, or equivalent network policy, so an SSRF in the server cannot reach instance credentials.
7. Patch on a schedule: track MLflow security advisories and the CISA KEV catalogue; treat CVE-2026-64849 as the minimum patch level for any reachable server.

Kubeflow Platform

1. Replace the default user: connect Dex or oauth2-proxy to your identity provider and remove static passwords, or at minimum replace 12341234 with a bcrypt hash of a strong password before the cluster is reachable.
2. Expose only through HTTPS ingress: the manifests advise using ClusterIP for the Istio ingress gateway service rather than NodePort, and putting HTTPS in front. Keep APP_SECURE_COOKIES at its default.
3. Apply Pod Security Standards: run user workloads under the restricted profile; Kubeflow Pipelines 2.16.0 and later can set runAsNonRoot, runAsUser and runAsGroup. See the Kubernetes Admission Control Guide.
4. Lock down Profile management: disable automatic registration, make a platform account the owner of each Profile, and restrict who can edit RoleBindings and AuthorizationPolicies in Profile namespaces.
5. Add network policies: deny-by-default between Profile namespaces, egress allowlists for training jobs, and no direct route to workloads that skips the ingress gateway.
6. Separate sensitive tenants: because pipeline definitions and ML Metadata are shared, put teams with regulated data on separate clusters rather than separate Profiles.

Roll out in stages: enable authentication and allowed hosts in a staging environment first, run existing pipelines and notebooks for a week to find clients that relied on anonymous access, then switch production during a maintenance window. Keep the previous server configuration and a database backup so you can roll back if critical training jobs break, but do not roll back to anonymous access on a network-reachable server.

Common misconfigurations

• MLflow on 0.0.0.0 with no auth: started with --host 0.0.0.0 for convenience and reachable from a VPN, a peered network or the internet. Anyone can replace a registered model artefact that a serving system will later load.
• Over-scoped artefact role: the tracking server runs with a broad storage or administrator role, so an SSRF or remote code execution flaw in the server becomes account-wide access. Exposed AI compute services have been exploited this way: in ShadowRay attackers used unauthenticated Ray clusters to steal cloud credentials and AI API tokens.
• Static keys in notebooks and namespaces: cloud keys and LLM API keys stored as plain environment variables or Kubernetes secrets readable by default-editor, then copied into experiment parameters or logs.
• Default Kubeflow credentials left in place: the static Dex user survives from a proof of concept into production.
• Treating Profiles as hard isolation: sharing one Kubeflow cluster between teams with different data classifications, despite the documented sharing of pipeline definitions and metadata.
• Auth that disappears on restart: a new deployment manifest omits --app-name basic-auth and the server comes back open, with the users still in the database giving a false sense of protection.

Framework mapping

Practitioner checklist

• Inventory every MLflow server and Kubeflow cluster, with owner, network exposure and version.
• Confirm authentication is enforced on every network path, including internal and VPN routes.
• Remove the Kubeflow default user and any MLflow admin still on a legacy default password.
• Replace static cloud keys used by tracking servers and pipelines with scoped workload identity.
• Set --allowed-hosts and --cors-allowed-origins to explicit values and keep the security middleware on.
• Block instance metadata access from MLOps hosts and pods.
• Restrict who can edit Profiles, RoleBindings and AuthorizationPolicies, and alert on changes.
• Record and verify hashes of production model artefacts and enable object versioning.
• Patch MLflow and Kubeflow components against KEV entries within the due date.
• Separate tenants with regulated data onto their own clusters or servers.

Standards and references

• MLflow: Basic HTTP Authentication
• MLflow: Protect Your Tracking Server from Network Exposure
• MLflow: Artifact Stores
• Kubeflow: Profiles and Namespaces
• Kubeflow: Multi-user Isolation for Pipelines
• Kubeflow: Manifests (installation and security notes)
• CISA: Known Exploited Vulnerabilities Catalog
• MITRE ATLAS: AI Supply Chain Compromise

Related NHI Mgmt Group resources: Model Supply Chain Guide · Ray and AI Compute Cluster Security Guide · Jupyter Server Security Guide · Kubernetes NHI Security Guide · AI Platforms Guide