Apache Spark Security: Secure RPC, web UIs and cluster access

Apache Spark authentication and encryption are settings you turn on (off by default), and a secure deployment keeps standalone master and REST submission ports away from untrusted networks. The project's own security documentation asks operators to configure security for every deployment type, and to rely only on documented settings. This guide sets out how identity works in Spark (shared-secret RPC authentication, servlet filters and ACLs for the web UIs, Kerberos and delegation tokens, Kubernetes service accounts), which settings to turn on, how to encrypt traffic and local storage, and how to spot an exposed cluster. It is written for data platform engineers and security teams running open source Spark on YARN, Kubernetes or standalone clusters. For the wider Hadoop stack, see the Hadoop Security Guide.

This is an independent guide from NHI Mgmt Group, based on the Apache Spark project's public documentation as of November 2026. It is not sponsored or reviewed by the Apache Spark project. Product features change, so confirm settings against the Apache Spark project's current documentation.

Key takeaways

• Nothing is on by default. spark.authenticate, spark.acls.enable, spark.io.encryption.enabled and RPC encryption all default to false; a default cluster accepts unauthenticated applications.
• Never expose ports 7077, 6066, 8080 or 4040 to the internet. The documentation expects Spark services to be private and reachable only from the hosts that need them.
• The UI has no login by default. You must configure a servlet filter for authentication (your own, or the JWT-based JWSFilter that Spark ships); spark.acls.enable does nothing without one.
• Prefer TLS for RPC, and enable it explicitly. spark.ssl.rpc.enabled is not inherited from spark.ssl.enabled, and the AES-based alternative is a legacy bespoke protocol.
• The shared secret is only as private as its distribution. On Kubernetes it reaches executors as an environment variable, so anyone who can list pods in the namespace can read it.

How identity and access work in Apache Spark

Spark has no user directory of its own. It relies on four separate mechanisms, and each deployment type supports them differently.

On standalone clusters, plan around the single shared secret: the documentation notes it limits security, especially on multi-tenant clusters, because every application holds the key that every other application uses. Multi-tenant estates usually run Spark on YARN or Kubernetes for this reason. On YARN the secret's distribution is only safe if YARN RPC encryption is enabled. Set spark.yarn.shuffle.server.recovery.disabled to true for applications that should not have their secret saved in the External Shuffle Service's database, accepting that their shuffle data is not recovered after a restart.

Admin roles and privileged access

Spark's authorisation model is per application, and it distinguishes view rights (see the UI) from modify rights (for example, kill jobs). By default only the submitting user is in the ACLs. A wildcard (*) in any list grants that right to every user, which is a common shortcut on shared clusters and should be removed.

The other privileged identities sit outside Spark: the Kerberos superuser that may impersonate users through --proxy-user, the service account of the History Server (which must own the event log directory), and on Kubernetes the driver's service account, which can create pods. Treat each as a privileged non-human identity with its own owner and review. On proxy users, the documentation points out a subtle difference: in cluster mode the driver JVM runs as the proxy user, but in client mode it runs as the superuser, which matters for local file access on multi-tenant gateways.

Staying on supported releases

Apply Spark security updates promptly and follow the Apache Spark security advisories. For standalone mode, disable spark.master.rest.enabled if unused, enable spark.authenticate or restrict network access, put the UI behind an authenticating proxy or filter, and keep UI and submission ports off the internet even when ACLs are on.

Integrations, API tokens and service accounts

Kubernetes

• Driver service account: the driver needs a Role or ClusterRole that allows it to create pods and services. Without a setting it uses the default service account in spark.kubernetes.namespace. Create a dedicated service account per team namespace, bind a namespaced Role (not a cluster-wide edit role) and set spark.kubernetes.authenticate.driver.serviceAccountName.
• Authentication secret exposure: Spark generates a per-application secret and passes it to executor pods as an environment variable, so restrict list and get on pods in Spark namespaces. Alternatively mount the secret from a file with spark.authenticate.secret.file (and the .driver.file and .executor.file variants); Spark does not mount these files for you.
• Credentials for data sources: mount Kubernetes Secrets with spark.kubernetes.driver.secrets.[SecretName] and spark.kubernetes.executor.secrets.[SecretName], or as environment variables with secretKeyRef. Secrets must be in the same namespace as the pods. Prefer workload identity from your cloud provider over static keys where it is available.
• User identity in images: images built from the project Dockerfiles run as UID 185. Security-conscious deployments should build images with their own unprivileged UID, or set runAsUser through a pod template, and enforce it with the Pod Security Admission controller.
• hostPath volumes: Spark on Kubernetes can mount hostPath volumes, which the Kubernetes documentation flags as risky; block them with Pod Security Admission.

See the Kubernetes Security Guide for cluster-side controls.

Kerberos and Hadoop services

Spark obtains delegation tokens for HDFS (the filesystem hosting the user's home directory), Hive when hive.metastore.uris is set, and HBase when hbase.security.authentication=kerberos. Other secure filesystems must be listed in spark.kerberos.access.hadoopFileSystems. For long-running jobs, a principal and keytab let Spark renew tokens indefinitely, but in cluster mode the keytab is copied to the driver host; on YARN it is staged in HDFS, so the documentation strongly recommends securing YARN and HDFS with encryption. The alternative is spark.kerberos.renewal.credentials=ccache, which uses the local ticket cache and needs a fresh kinit after the ticket's renewable life ends. Keytabs are long-lived secrets: store them with tight file permissions, one principal per service, and rotate them with the KDC. See the Kerberos Security Guide.

REST submission and JWT filters

The standalone REST Submission Server listens on 6066 when spark.master.rest.enabled is on. Recent releases can require an HTTP Authorization header carrying a JSON Web Token signed with a shared key, by setting spark.master.rest.filters=org.apache.spark.ui.JWSFilter and spark.org.apache.spark.ui.JWSFilter.param.secretKey. The same filter can protect all UI ports through spark.ui.filters. If you do not need REST submission, disable it. Treat the JWS key as a secret: keep it out of spark-defaults.conf files that many users can read.

Encryption

Output data written by applications is not covered by Spark's local storage encryption; protect it at the storage layer (HDFS encryption zones or object store encryption). See the Database and Storage Encryption Guide.

Logging and detection

The default network exposure is the first thing to measure. Spark's documented ports are the place to start.

• External exposure: scan your public address space and cloud security groups for the ports above. An internet-facing master is an open job-submission service (ATT&CK T1133 External Remote Services, often followed by T1496.001 Compute Hijacking).
• Unexpected applications: alert on applications submitted by unknown users, from unknown hosts, or with names and resource requests that do not match scheduled workloads. The master UI and History Server list them; cluster manager logs (YARN ResourceManager, Kubernetes API audit logs) give the authoritative record.
• Kubernetes signals: pod creation by Spark driver service accounts outside their namespaces, reads of Spark pod specs by unexpected principals, and hostPath or privileged pod requests rejected by admission.
• Event logs: create spark.eventLog.dir manually with permissions drwxrwxrwxt (sticky bit) owned by the History Server's user, so users can write but cannot read, delete or rename others' logs. Apply the same to spark.driver.log.dfsDir if driver logs are persisted.
• Configuration leakage: the Environment tab of the UI shows configuration values; Spark redacts properties and environment variables matching spark.redaction.regex (by default secret, password, token, access key and credential), so extend it for your own secret naming conventions.

Metrics that show the controls are working:

• Spark ports reachable from the internet (target zero).
• Percentage of clusters with spark.authenticate=true and RPC encryption enabled.
• History Servers with ACLs enabled and an authentication filter (target 100%).
• Spark namespaces where the driver uses the default service account (target zero).
• Clusters on a release with outstanding security updates (target zero).

Hardening baseline

1. Close the network first: remove internet exposure of all Spark ports, restrict security groups to known hosts, and disable the REST submission server if unused. This is the fastest risk reduction and rarely breaks jobs.
2. Patch: move to a supported Spark release and check the Apache Spark security page for advisories that affect your version.
3. Turn on RPC authentication: set spark.authenticate=true. On YARN and Kubernetes the secret is generated per application; on standalone, generate a strong secret and distribute it by file, not on the command line. Test on a staging cluster: a mismatch fails executor registration, which is the expected symptom.
4. Encrypt: enable spark.ssl.rpc.enabled with keystores (or AES version 2 where TLS is impractical), spark.io.encryption.enabled with 256-bit keys, and TLS on UIs.
5. Authenticate and authorise UIs: deploy a servlet filter or an authenticating reverse proxy, then enable spark.acls.enable and spark.history.ui.acls.enable with group-based ACLs and no wildcards.
6. Scope identities: dedicated Kubernetes service accounts and namespaced Roles, per-service Kerberos principals and keytabs, and Pod Security Admission blocking root, privileged and hostPath pods.
7. Lock down logs: sticky-bit event log directories and extended redaction patterns.

Rollback: keep the previous spark-defaults.conf and pod templates under version control. Authentication and encryption changes are all configuration, so a failed rollout is reversed by restoring the file and restarting the affected daemons or resubmitting jobs.

Choosing a deployment model: a single-team cluster on a private network can run standalone mode with a file-distributed secret, TLS and a proxy in front of the UIs. Shared, multi-tenant estates should prefer YARN with Kerberos or Kubernetes with per-namespace service accounts, because both give each application its own secret and integrate with an external identity system.

Common misconfigurations

• Internet-facing standalone master or REST port: with authentication off, anyone who can reach 7077 or 6066 can run code on the cluster.
• ACLs enabled without a filter: spark.acls.enable requires an authenticated user; with no filter installed it does nothing, giving false assurance.
• Assuming spark.ssl.enabled covers RPC: it does not; spark.ssl.rpc.enabled must be set separately.
• History Server open to all: spark.history.ui.acls.enable defaults to false, exposing every application's configuration and SQL plans to anyone who can reach port 18080.
• Default Kubernetes service account with broad rights: the driver inherits whatever the namespace default can do, and anyone who can list pods can read the application's authentication secret.
• Keytabs and secrets in world-readable configuration: spark.authenticate.secret or JWS keys in shared spark-defaults.conf files, or keytabs staged on unencrypted HDFS.

Framework mapping

Practitioner checklist

• No Spark port (7077, 6066, 8080, 8081, 4040, 18080) is reachable from the internet.
• spark.authenticate=true on every cluster; standalone secrets distributed by file with tight permissions.
• spark.ssl.rpc.enabled set explicitly, or AES with authEngineVersion 2; UIs on TLS.
• spark.io.encryption.enabled=true with 256-bit keys where shuffle data is sensitive.
• An authentication filter or proxy in front of all UIs, with spark.acls.enable and spark.history.ui.acls.enable on and no wildcard ACLs.
• REST submission disabled or protected with JWSFilter.
• Dedicated Kubernetes service accounts with namespaced Roles; pod list access restricted in Spark namespaces.
• Pod Security Admission blocks root, privileged and hostPath pods in Spark namespaces.
• Keytabs stored securely, one principal per service, YARN and HDFS encrypted where keytabs are staged.
• Spark on a supported release with current security updates applied.

Standards and references

• Apache Spark: Spark Security
• Apache Spark: Running Spark on Kubernetes
• Apache Spark: Spark Security (vulnerability reporting and advisories)
• Kubernetes: Pod Security Admission
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls

Related NHI Mgmt Group resources: Hadoop Security Guide · Kubernetes Security Guide · Kerberos Security Guide · ETL and ELT Pipeline Security Guide