ETL and ELT Pipeline Security Guide: Securing Data Pipelines and Their Credentials

Data pipelines hold standing credentials to almost every important data store in the organisation, and they are rarely governed like the privileged access they are. An ETL or ELT job reads from production databases, SaaS applications and object storage, then writes into a warehouse or lake, which means one orchestrator, one connector service or one pipeline service account can usually read more customer data than any single employee. This guide covers how to secure the pipeline itself: the identities and secrets it runs on, the orchestration layer, data in transit and at rest between stages, integrity of what lands in the warehouse, and the logging needed to spot bulk extraction. It is written for data platform engineers, security architects and identity teams. For the broader data protection programme see the Data Security Guide; this page goes deeper on the pipeline.

Key takeaways

• Treat every pipeline identity as privileged. Connector service accounts, warehouse loader users and orchestrator roles often have read access to whole schemas and write access to the warehouse, so they need the same ownership, rotation and review as admin accounts.
• Prefer federated or platform-managed identity over stored secrets. Managed identities, workload identity and key-pair or token based service users remove the long-lived password that infostealers and leaked config files expose.
• The orchestrator is a code execution platform. In Apache Airflow, DAG authors can run arbitrary code on workers and reach the credentials their DAGs use, so write access to the DAG repository is effectively access to every connection.
• ELT moves raw sensitive data further. Loading before transforming lands unmasked personal data in the warehouse, so classification, masking and access policy must apply at the raw layer, not only at the curated layer.
• Detect volume and novelty, not just failure. Pipeline credentials used from a new network, at an unusual hour or for unusually large reads are the main signal of a stolen connector credential.

ETL vs ELT: where the security boundary moves

ETL (extract, transform, load) transforms data in a separate processing tier before loading it. ELT (extract, load, transform) loads raw data into the warehouse or lakehouse and transforms it there, typically with SQL models. The choice changes where sensitive data sits unprotected and which identities can see it.

Most organisations run both: managed ingestion connectors for SaaS sources, an orchestrator such as Airflow for scheduling, and SQL transformation inside the warehouse. Map each hop as a data flow with a named identity on each end. That data flow map is what NIST CSF 2.0 ID.AM-03 asks for and it is the starting point for every control below.

Pipeline identities and their credentials

A typical pipeline has more non-human identities than people realise. Inventory them all, assign each an owner, and record what data it can reach. The Service Account Inventory Template works for this.

Platform specifics worth knowing

• Google Cloud Dataflow: worker VMs run as the worker service account. By default this is the Compute Engine default service account ([email protected]), which depending on organisation policy may hold the broad Editor role. Google recommends a user-managed worker service account with roles/dataflow.worker plus only the resource roles the pipeline needs. Whoever launches the job needs iam.serviceAccounts.actAs on that account, so control who holds roles/iam.serviceAccountUser. See the Google Cloud IAM and Service Accounts Guide.
• Azure Data Factory: a system-assigned managed identity is tied to the factory's lifecycle; a user-assigned managed identity is a separate resource that can be shared. Linked services can authenticate with managed identity to storage, Azure SQL, Synapse and others, and credentials that must remain can be stored in Azure Key Vault. Microsoft notes that users with Data Factory Contributor can run pipelines that use those identities and inherit their downstream permissions, so the Contributor role on a factory is effectively access to every connected store.
• Snowflake: set pipeline users to TYPE = SERVICE. Service users cannot log in with a password; LEGACY_SERVICE still allows passwords and is deprecated. Key-pair authentication needs at least a 2048-bit RSA key. The named key pair method (ALTER USER ... ADD KEY PAIR) supports role restriction and key expiration, and ROTATE KEY PAIR keeps the previous key valid for 24 hours by default, adjustable with EXPIRE_ROTATED_KEY_PAIR_AFTER_HOURS. The older RSA_PUBLIC_KEY and RSA_PUBLIC_KEY_2 slots allow zero downtime rotation but no expiry. See the Snowflake Security Guide.
• AWS Glue: jobs run under an IAM role passed at creation, so iam:PassRole on that role is the real permission to govern. Connection passwords can be referenced from AWS Secrets Manager rather than typed into the connection.

Securing the orchestrator

The Apache Airflow security model is explicit: DAG authors can execute arbitrary code on workers, in the DAG File Processor and in the Triggerer, Airflow does not sandbox that code, and DAG authors can access the credentials their DAGs use. Deployment Managers have access to all credentials. The same logic applies to Dagster, Prefect and similar tools. Design around it:

1. Protect the DAG repository like production code: branch protection, mandatory review, signed commits where practical, and no direct writes to the DAG folder on the scheduler. Anyone who can merge a DAG can read every connection that DAG can reference.
2. Move connections out of the metadata database: set [secrets] backend to a community provider backend (HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault) with [secrets] backend_kwargs. Airflow searches the configured backend first, then environment variables, then the metastore, and the order is not configurable, so stale metastore copies can still be read if a lookup misses. Delete them after migration.
3. Scope secrets per team or DAG: use path prefixes in the secrets backend and give each worker queue or deployment an identity that can read only its own prefix. Airflow supports a worker-specific backend via [workers] secrets_backend.
4. Protect the Fernet key: it encrypts connections and variables at rest and is needed by the API server, scheduler, workers, DAG File Processor and Triggerer. Keep it in a secrets manager, not in airflow.cfg committed to a repository, and rotate it with the documented re-encryption process.
5. Restrict UI roles: since Airflow 3, users who configure connections can write but not view sensitive connection fields; on Airflow 2 they could view them, which is a reason to prioritise upgrading. Scope UI roles down to individual DAGs where the deployment allows.
6. Isolate by trust boundary: separate deployments for teams that should not see each other's data, rather than relying on DAG level permissions inside one deployment.

The CI side matters too: if DAGs or transformation models deploy from CI, the CI system holds deploy credentials and often warehouse credentials. Apply the CI/CD Pipeline Identity Security Guide patterns, especially OIDC federation in place of stored cloud keys.

Protecting data in transit, at rest and in staging

• Transit: require TLS on every hop, including source database connections (many JDBC and ODBC drivers default to unencrypted or unverified TLS), connector to warehouse, and orchestrator to API endpoints. Verify certificates; sslmode=require in PostgreSQL clients encrypts but does not verify the server, verify-full does. Use private connectivity (VPC endpoints, Private Link, Private Service Connect) for cloud to cloud hops where available.
• Staging buckets: ETL scratch locations and ELT stage areas collect full extracts. Block public access, apply lifecycle rules to delete staged files after load, encrypt with a customer-managed key where policy requires it, and restrict the bucket to the pipeline's runtime role.
• Glue security configurations: an AWS Glue security configuration encrypts S3 targets, CloudWatch Logs and job bookmarks for ETL jobs, and CloudWatch Logs for crawlers. It overrides an SSE-S3 setting passed as a job parameter. If the job runs in a VPC it needs a route to AWS KMS (NAT gateway or a KMS VPC endpoint) or it fails with KMS timeouts.
• Logs and intermediate outputs: job logs, failed record queues and dead letter topics often contain sample rows. Classify and protect them like the data itself, and avoid printing rows or connection strings in task logs.
• Masking at the raw layer: for ELT, apply column masking or tokenisation policies to raw tables that hold personal data, so analysts with access to downstream views never need raw access. See the Data Classification Guide for the labelling that drives those policies.

Integrity of pipeline code and data

Confidentiality gets the attention, but a pipeline that loads manipulated data into finance or risk reporting is an integrity incident (ATT&CK T1565.001 Stored Data Manipulation). Controls that help:

• Code provenance: pin connector, provider and package versions; build orchestrator images from a reviewed lockfile; scan DAG and model repositories for secrets before merge.
• Separation of duties: the role that writes raw data should not be able to alter curated tables, and the transformation role should not be able to rewrite raw history.
• Data tests: row count, null rate, uniqueness and freshness checks in the transformation layer catch both bugs and tampering. Fail the run rather than publish suspect data.
• Immutable raw history: keep raw loads append only, with time travel or object versioning, so a corrupted run can be rebuilt.
• Untrusted sources: data from third party feeds and webhooks is input. Validate schemas and reject unexpected types before load rather than casting blindly.

Implementation approach

1. Inventory (weeks 1 to 3): list every pipeline, its sources and sinks, the identity on each hop, where its secret is stored and who owns it. Pull this from the orchestrator's connection list, the warehouse's user list and cloud IAM.
2. Remove the worst exposures first: password-only warehouse users without network policies, connections stored in plain config files or repositories, and default cloud service accounts with Editor or Contributor rights.
3. Pilot one pipeline end to end: move it to a dedicated identity, a secrets backend or managed identity, least privilege roles, and network allow lists. Run it in parallel with the old credentials for one full schedule cycle and compare outputs before cutting over.
4. Roll out by domain: migrate pipelines in batches per data domain. Keep the old credential disabled, not deleted, for one cycle so you can roll back by re-enabling it, then delete it.
5. Turn on detection: baseline normal query volume, source IPs and run times for each pipeline identity before alerting, to keep false positives manageable.
6. Govern: add pipeline identities to quarterly access reviews and to joiner, mover, leaver processes for their human owners.

Decision guidance

• Managed identity vs stored secret: use managed identity or workload identity whenever source and sink are in the same cloud or support federation. Keep stored secrets only for sources that offer nothing else, and put them in a secrets manager with rotation.
• Managed connector service vs self-hosted: a SaaS ELT service is a third party holding credentials to your sources. Check its credential storage, tenancy isolation and whether it supports customer-hosted agents or private connectivity. Self-hosting removes that third party but makes you responsible for patching and secrets.
• Small organisations: one orchestrator, one secrets manager, one warehouse service user per pipeline with key-pair auth and a network policy covers most risk. Skip separate deployments per team.
• Large organisations: split orchestrator deployments by trust boundary, use per-domain cloud roles, automate credential rotation and feed pipeline identity telemetry into the SOC.

Detection and measurement

Log authentication and query history for every pipeline identity, orchestrator audit logs (connection changes, DAG deployments, manual triggers), secrets manager access logs and cloud IAM changes to pipeline roles. Alert on:

• A pipeline identity authenticating from an IP outside its expected ranges, or interactively from a client tool.
• Reads far above the pipeline's normal volume, or queries against tables it never touches.
• New connections, changed connection hosts or new secrets backend paths outside a change window.
• actAs, PassRole or role assignment changes on pipeline runtime identities.

Framework mapping

In CIS Controls v8.1 the work sits mainly in Control 3 (Data Protection), Control 5 (Account Management), Control 6 (Access Control Management) and Control 8 (Audit Log Management). In ISO/IEC 27001:2022 Annex A the closest controls are A.5.17 Authentication information, A.8.3 Information access restriction, A.8.11 Data masking, A.8.15 Logging and A.8.24 Use of cryptography.

Common failure patterns

• Password service users with no network restriction: the Snowflake Breach campaign used valid usernames and passwords harvested by infostealers, often from contractor devices, against accounts with no MFA and no network allow lists, then used normal tools to copy tables in bulk. Pipeline and service users are exactly this kind of credential.
• Default cloud service accounts: Dataflow workers or Glue jobs running under broad default identities turn a single job compromise into project-wide access.
• Connections in code: connection strings in DAG files, notebooks or .env files committed to repositories. Once committed, they spread to forks, CI caches and backups and cannot be reliably recalled, so rotation is the only fix.
• One loader role for everything: the same user writes raw data, runs transformations and owns the database, so any compromise or bad model can alter all history.
• Orphaned SaaS connector tokens: tokens created by an engineer who left, still syncing CRM data to a forgotten destination.
• Logs full of data: task logs and failed record files with personal data, retained indefinitely and readable by every engineer.

Lessons from incidents

The Snowflake campaign of 2024 is the clearest lesson for pipeline owners. No platform vulnerability was involved: credentials that had not been rotated, some exposed in infostealer logs years earlier, were enough because the accounts lacked MFA and network policies. Snowflake has since moved to phase out single-factor password sign-ins, and the SERVICE user type now blocks password login for non-human users. The practical lesson is to remove password authentication from every pipeline identity and pin each one to known networks.

Practitioner checklist

• Map every pipeline hop with source, sink, identity, secret location and owner.
• Replace password authentication for warehouse pipeline users with key-pair, federated or managed identity, and set Snowflake users to TYPE = SERVICE.
• Apply network policies or private connectivity to every pipeline identity.
• Move orchestrator connections to an external secrets backend, delete metastore copies and protect the Fernet key.
• Treat DAG and transformation repositories as privileged: mandatory review, branch protection, secret scanning.
• Replace default cloud service accounts with dedicated least privilege roles and govern actAs and PassRole.
• Restrict ELT raw schemas to loader and transformation roles, with masking on personal data columns.
• Encrypt and auto-expire staging locations and keep raw loads append only.
• Baseline and alert on pipeline identity location, volume and new table access.
• Include pipeline identities and SaaS connector tokens in quarterly access reviews.

Standards and references

• NIST: SP 800-53 Rev. 5 Security and Privacy Controls for Information Systems and Organizations
• NIST: Cybersecurity Framework 2.0
• Apache Airflow: Airflow Security Model
• Apache Airflow: Secrets Backend
• Google Cloud: Dataflow security and permissions
• Microsoft Learn: Managed identity for Azure Data Factory
• Snowflake: Key-pair authentication and key-pair rotation
• AWS: Encrypting data written by AWS Glue

Related NHI Mgmt Group resources: Data Security Guide, Service Account Security Guide, Secrets Management Guide, Data Lake Security Guide, Kafka Security Guide.