A compromised workflow orchestrator gives the attacker scheduled code execution on your workers and a ready-made bag of credentials for every database, warehouse, cloud account and API your pipelines touch. Airflow-class schedulers run Python workflow definitions (DAGs) on a timetable, store connection credentials and variables, and hand them to tasks at run time. Their own security model is explicit that workflow authors can run arbitrary code and potentially reach the credentials that code uses, so a foothold in the workflow repository, the UI or a worker is a foothold in everything downstream. The urgent questions are which code ran, which credentials it could read, what it changed in downstream data, and how to restart pipelines without re-running the attacker's tasks. This playbook is for data platform, DevOps, security operations and incident response teams. It is vendor-neutral; where a concrete example helps, it uses the open-source Apache Airflow documentation as the reference design for the class. Preventive design is in Securing the workflow orchestrator (Airflow-class schedulers) itself.
Key takeaways
- Pause schedules, do not just block users. Scheduled and sensor-driven tasks keep running attacker code with no one logged in.
- Every stored connection is in scope. Anyone who can author workflows can, by design, write code that reads the credentials tasks use, so rotate all of them, not only those you can see were touched.
- Four platform secrets matter most. The metadata database password, the encryption key for stored connections, the token signing key and the secrets backend credential each unlock the rest.
- Downstream data is evidence and victim. Malicious tasks can alter tables, files and models as easily as they exfiltrate them, so check outputs for the window.
- Rebuild workers and redeploy workflow code from reviewed source. Workers run arbitrary code and are not safe to clean in place.
Scope: what an orchestrator compromise covers
| Entry point | Typical cause | What the attacker can do | ATT&CK |
|---|---|---|---|
| Workflow code repository or bundle location | Compromised developer account, weak branch protection, writable shared folder | Add or modify DAG code that runs on workers, the parser and the trigger service | T1059.006 Python, T1195.002 Compromise Software Supply Chain |
| Web UI or REST API | Exposed UI, weak or shared admin login, stolen API token | Edit connections and variables, trigger and clear runs, read logs | T1078 Valid Accounts |
| Worker host or task pod | Malicious task, vulnerable dependency, container escape | Read task credentials, reach the execution API, pivot to data stores | T1552.001 Credentials In Files |
| Metadata database | Exposed database, credentials leaked in logs or environment | Read encrypted connections, alter run state, plant variables | T1565.001 Stored Data Manipulation |
| Platform secrets | Encryption or signing key in environment variables readable by task code | Decrypt every stored connection or forge tokens | T1552.001 |
| Cloud identity of workers | Broad role attached to worker nodes or pods | Call cloud APIs, create new keys for persistence | T1552.005 Cloud Instance Metadata API, T1098.001 Additional Cloud Credentials |
The ShadowRay campaign of 2024 against exposed AI compute clusters shows the pattern: an exposed job submission interface became a route to cloud credentials, AI API tokens and compute. If the incident began in CI, run the CI/CD Pipeline Compromise Playbook alongside this one.
Why the blast radius is wide: the trust model
The Apache Airflow security model sets out three roles that apply across this class of scheduler:
- Deployment managers: install and configure the platform and have access to all credentials.
- Workflow authors: can create, modify and delete workflow files. Their code is not verified, checked or sandboxed by the platform, so they can effectively execute arbitrary code on workers, the file parser and the trigger service, and can potentially access the credentials that code uses.
- UI users: range from viewers to administrators. Users who can edit connections can, for some providers, configure options that lead to code execution on workers.
The same document states that workflow authors have access to all workflows in an installation and can change the state of any of them, and that task tokens for the internal execution API do not carry per-workflow authorisation. An experimental multi-team mode adds isolation at the UI and API level but, as the documentation says, does not yet guarantee task-level isolation. For incident scoping, that means one malicious workflow author or one compromised worker should be treated as able to reach every connection and variable in the installation.
How the compromise shows up
- Workflow code that does not match the repository: files in the deployed bundle or DAG folder that differ from the reviewed commit, or new workflows with no pull request.
- Connection and variable changes: edits to connection hosts, extras or login fields, or new variables, especially just before a run.
- Odd runs: manual triggers or cleared runs by unexpected users, tasks with unusual durations, or runs outside the schedule.
- Worker network behaviour: outbound connections from workers to destinations no connection is configured for.
- Cloud and data store activity: worker identities creating keys, reading secrets they do not normally read, or bulk exports from warehouses.
- Secrets in task logs: the security model warns that configuration values can leak through logs, and that values exposed by task code through environment variables are not automatically masked.
Triage: the first hour
- Pause all schedules: pause every workflow, or stop the scheduler, so no further attacker code runs. Record which workflows were paused by you, so restart is controlled.
- Freeze code delivery: stop the job that syncs workflow code from the repository, and lock the branch and bundle location to responders.
- Snapshot the state: back up the metadata database, export the deployed workflow files with hashes, export connections and variables metadata, users and roles, the audit log and task logs for the window.
- Preserve workers: snapshot or image running workers and keep pod specifications and logs before they are recycled; ephemeral task pods disappear quickly.
- Restrict access: limit the UI and API to responders and disable API tokens not needed for response.
Containment by priority
- Cut worker egress: restrict worker outbound traffic to the data stores and APIs that connections legitimately target, so exfiltration and command channels stop.
- Remove malicious code: replace the deployed workflow set with the last reviewed commit, and quarantine any added or modified files as evidence.
- Revoke access: disable unknown users, rotate admin passwords, revoke API tokens and identity provider sessions for workflow authors and admins.
- Contain downstream identities: for every connection, disable or restrict the target credential at the target system (database user, warehouse role, cloud key, API token) until it has been rotated.
- Restrict worker cloud roles: remove permissions not needed for recovery and look for keys or identities they created (T1098.001).
- Protect the logs: ship scheduler, API, worker and audit logs to a store that platform administrators cannot alter.
Credential rotation inventory
| Credential | Where it lives | Rotation action | Better pattern afterwards |
|---|---|---|---|
| Connection credentials (databases, warehouses, cloud, APIs, SFTP) | Metadata database (encrypted), environment variables, secrets backend | Rotate at each target system, then update the stored value | Secrets backend with per-connection scope; federated identity where targets support it |
| Encryption key for stored connections | Configuration or environment | Generate a new key and re-encrypt (see below) | Available only to components that need it |
| Token signing key (JWT secret or private key) | Configuration of API server and scheduler | Replace; invalidate tokens issued with the old key | Asymmetric keys, never present on workers |
| Metadata database password | Configuration of platform components | Rotate; review database logs | Kept away from workers and parsers |
| Secrets backend credential | Platform configuration or workload identity | Rotate and review what was read in the window | Workload identity scoped to the paths the platform needs |
| Worker cloud roles and keys | Nodes, pods, service accounts | Rotate keys, delete unknown keys, narrow roles | Per-workflow workload identity with minimum scope |
| Variables holding secrets | Metadata database | Rotate the underlying secrets and move them to connections or the backend | No secrets in variables |
| UI admin accounts and API tokens | Platform, identity provider | Reset, enforce SSO and phishing-resistant MFA | SSO only; break-glass in a vault |
- Encryption key rotation: the Airflow documentation describes rotating the Fernet key without breaking stored values: set the key setting to the new key followed by the old one, run the rotate command to re-encrypt, then remove the old key. In an incident, re-encrypting is not enough on its own: anyone who held the old key could already decrypt every stored value, so the underlying credentials must be rotated at their targets as well.
- Signing key: the security model notes that with a symmetric signing key, any component holding it can forge tokens with arbitrary claims, and that remote workers which sign their own requests must hold it. Treat those hosts as part of the control plane and rotate accordingly.
- Lookup order: by default the platform checks environment variables first and then the metadata database; an alternative secrets backend is searched before both, and on duplicate keys reads return the first match. Check all three locations so a stale or planted value does not shadow the rotated one.
See the If your secrets manager is compromised playbook if the backend itself may be affected, and the Workload Identity Federation Guide for replacing static keys.
Eradication and rebuild
- Rebuild platform components: redeploy scheduler, API server, parser, trigger service and workers from trusted images on a supported release, with configuration from reviewed source.
- Separate secrets by component: follow the documented hardening: the signing key only on components that issue or validate tokens, database credentials and the signing key never on workers, and connection credentials served through the execution API rather than exposed to every component.
- Re-review workflow code: review every workflow changed in the window and all custom operators, hooks and plugins; pin dependencies.
- Check downstream data: compare tables, files and models written by affected runs with backups or source systems, and restore where integrity cannot be shown. See the If your data warehouse and lakehouse is compromised playbook.
- Re-gate code delivery: require reviewed, protected-branch merges for workflow code and deploy it by a pipeline identity, not by hand.
Recovery, testing and rollback
- Staged restart: unpause a small set of low-risk workflows first, confirm they succeed with rotated credentials and that no unexpected egress occurs, then unpause by business priority.
- Backfill with care: decide per workflow whether to backfill missed runs; never re-run runs from the compromise window without reviewing their code version.
- Negative tests: confirm a token signed with the old key is rejected, and that a worker cannot read the database password or signing key.
- Rollback: keep the reviewed commit hash and the post-rotation secret versions as the known-good set.
Small and large organisations
| Decision | Small team, one installation | Large estate, many teams and installations |
|---|---|---|
| Credential rotation | Rotate every connection at once | Rotate by target criticality; production data stores and cloud admin first |
| Isolation afterwards | Single installation, strict author list | Separate installations per trust boundary, since task-level isolation between teams is not guaranteed |
| Data integrity review | Check all outputs in the window | Check outputs of affected workflows first, then sample the rest |
Evidence and notification
- Preserve: database backup, deployed workflow files with hashes, repository history, task and scheduler logs, worker images and pod specifications, audit logs, and target-side logs for each connection, with chain of custody (CSF 2.0 RS.AN-07).
- Notification: orchestrators often move personal and regulated data, so assess what the attacker could read through connections. See the Cyber Incident Reporting Requirements Map and coordinate with affected parties as NIST SP 800-61 Rev. 3 describes.
Detection and measurement
- Deployed workflow files whose hash does not match a reviewed commit (target zero).
- Connections with a named owner and a rotation date inside policy (NHI Mgmt Group recommendation: 100%, rotated at least every 90 days or federated).
- Workers able to read the database password or signing key (target zero).
- Worker egress to destinations not linked to a connection (target zero; each one investigated).
- Manual triggers and clears by non-owner users (tracked and reviewed weekly).
- Secrets found in task logs (target zero).
Framework mapping
| Area | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | ATT&CK addressed |
|---|---|---|---|
| Workflow code integrity | SI-7 | RS.AN-03 | T1059.006, T1195.002 |
| Connection and platform secrets | IA-5, SC-12 | RS.MI-02 | T1552.001, T1552.005 |
| Accounts, roles and least privilege | AC-2, AC-6 | RS.MI-01 | T1078, T1098.001 |
| Egress restriction | SC-7 | RS.MI-01 | T1552.005 |
| Logging and monitoring | AU-9, SI-4 | RS.AN-07 | T1685 Disable or Modify Tools |
| Data integrity and recovery | IR-4, CP-10 | RC.RP-02 | T1565.001 |
Common failure patterns
- Disabling users but leaving schedules running: planted workflows keep executing every interval.
- Rotating only the connections that look touched: any author-level code could read all of them.
- Re-encrypting without rotating: a new encryption key does not help if the old one was read; the credentials themselves must change.
- Restoring the database from a backup that holds planted variables: check restored connections and variables before restarting.
- Forgetting environment variables and the secrets backend: stale values shadow rotated ones because of the lookup order.
- Skipping the data review: altered outputs in a warehouse or feature store outlive the incident and feed reports and models.
Practitioner checklist
- Pause all schedules and stop workflow code sync.
- Back up the database, export deployed code, logs, users and connection metadata, and image workers.
- Cut worker egress to legitimate targets only.
- Replace deployed code with the last reviewed commit and quarantine changed files.
- Rotate every connection at its target, plus the encryption key, signing key, database password and backend credential.
- Check environment variables, the database and the backend for shadowed values.
- Rebuild components with secrets separated by component and review downstream data.
- Restart in stages and track the metrics above.
Standards and references
- Apache Software Foundation: Airflow security model
- Apache Software Foundation: Airflow Fernet key and rotation
- Apache Software Foundation: Airflow secrets backend
- NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- MITRE ATT&CK: T1552.001 Credentials In Files
- MITRE ATT&CK: T1059.006 Python
Related NHI Mgmt Group resources: Apache Airflow Security Guide · Securing the workflow orchestrator (Airflow-class schedulers) itself · ETL and ELT Pipeline Security Guide · If your ETL and iPaaS platform is compromised · Workflow orchestrator (Airflow-class schedulers) health check