Most incident response playbooks stop at Windows, Linux and cloud, so when a z/OS system is involved the responders often do not know where the evidence is, who can collect it or which containment step will stop the batch window. A mainframe keeps an unusually rich record of what happened, through System Management Facilities (SMF) records, the operations log and the security manager's own audit trail, but that record is only useful if it was switched on, protected from the people being investigated and collected in a way that proves it was not altered. This guide covers the evidence sources on z/OS, preparing them before an incident, triage and containment that does not break production, forensic acquisition and integrity, analysis and recovery. It is written for incident responders who have to work with mainframe teams, and for z/OS system programmers and security administrators who will be on the bridge call. Access control itself is covered in our Mainframe Identity and RACF Security Guide; this page assumes those controls and focuses on what to do when they have failed.
Key takeaways
- SMF is the primary evidence source. Security events (type 80), job activity (type 30), data set access and changes (types 14, 15, 17, 18 and the catalog records), system changes (type 90) and network sessions (type 119) together answer who did what, from where and to which data.
- Integrity has to be designed in. SMF record signing on log streams lets you detect a changed, added or removed record, and Db2 tamper-proof audit policies stop a database administrator switching off the audit trace. Neither can be added after the fact.
- Containment is a production change. Revoking a user ID, cancelling a job or stopping the FTP server can halt batch or online services, so decide containment with the operations lead and record each command and its time.
- Collect before you clean. Dump SMF with signatures kept, unload the security database, copy PARMLIB, PROCLIB and the APF and link lists, and hash everything before anyone resets passwords or restores profiles.
- Recovery from logical corruption needs known-good copies. Immutable point-in-time copies analysed in an isolated environment let you find the last clean state and choose surgical or full recovery.
Why mainframe incidents are different
The usual DFIR toolkit of endpoint agents, memory images and disk images does not translate directly to z/OS. There is no EDR agent on most mainframes, a full memory image of a production logical partition (LPAR) is not a routine collection, and disks are shared across a sysplex and owned by storage teams. What z/OS offers instead is structured, centralised recording.
- Shared everything: several LPARs can share one security database, one catalog and the same DASD volumes, so a change on one system applies everywhere. Scope the incident by sysplex and shared resource, not by host name.
- Few, powerful people: a small team of system programmers often holds the privileges an attacker would want and the access a responder needs. Decide early which of them are witnesses, which are responders and which may be subjects, and apply segregation of duties to evidence handling. See the Segregation of Duties Guide.
- Business-critical timing: batch windows and online CICS or IMS regions have hard deadlines, so a routine containment step can stop payments or settlement.
- Non-human identities everywhere: batch jobs, started tasks, FTP partners and API connectors run under service user IDs, and these are a common path to data.
Evidence sources on z/OS
The table lists the sources responders use most, with the questions each answers. Record names follow IBM's SMF documentation; enable the ones you need in SMFPRMxx before an incident.
| Source | What it records | Questions it answers |
|---|---|---|
| SMF type 80 (security product processing) | RACF events: logons and job initiation, resource access successes and failures, and every security command | Who logged on, what was denied, who changed a profile, permit or SETROPTS option and when |
| SMF type 81 and 83 | Type 81 at RACF initialisation; type 83 subtype 1 lists data sets affected by security label changes on data set profiles, and other subtypes carry LDAP, EIM, R_auditx and WebSphere audit data | What the security configuration was at IPL; which data sets a label change touched |
| SMF type 30 (common address space work) | Job, step and address space start, end and resource use | Which jobs and programs ran, under which user, on which system, for how long |
| SMF types 14, 15, 17, 18 | Data sets opened for input (14) or output and update (15), scratched (17) and renamed (18) | What was read, written, deleted or renamed, and by which job |
| SMF types 61, 65, 66 | Catalog define, delete and alter activity | Were data sets created, deleted or redirected in the catalog |
| SMF type 90 (system status) | System changes, including SET DATE, LNKLST activation after IPL (subtype 29) and LPA module adds or deletes (subtype 31) | Did anyone change the clock, the link list or the link pack area during the incident |
| SMF type 92 | z/OS UNIX file system activity | What happened in z/OS UNIX files used by web, API and Java workloads |
| SMF type 119 (TCP/IP) | TCP connection start and end (subtypes 1, 2), FTP server logon failure (72), FTP server transfers (70, 100), FTP sessions (103, 104) and TN3270 SNA sessions (20, 21) | Where a session came from, what was transferred out by FTP, how many logons failed |
| Db2 audit trace (SMF type 102) | Authorisation failures, grants and revokes, and access to audited tables, driven by audit policies in SYSIBM.SYSAUDITPOLICIES | Who queried or changed sensitive tables, and whether auditing was stopped |
| CICS SMF type 110 | CICS statistics and monitoring data | Which transactions ran in which region during the window |
| SYSLOG and OPERLOG | Console messages and operator commands; SYSLOG is per system on JES spool, OPERLOG is a sysplex-wide log stream | Which commands were issued from consoles, which subsystems were started or stopped |
| Logrec and dumps | Hardware and software error records; SVC dumps analysed with IPCS | Did an exploit or tampering attempt cause abends in system or subsystem code |
| Security database unload | A point-in-time copy of every user, group, profile and permit (IRRDBU00 for RACF; the list functions in ACF2 and Top Secret) | What access existed at collection time, compared with a known-good unload |
RACF event codes in the unloaded type 80 data identify the activity, for example 1 JOBINIT (logon or job start), 2 ACCESS, 13 ALTUSER, 19 PERMIT and 24 SETROPTS. IRRADU00 converts type 80, 81 and 83 records into a readable format that a SIEM or query tool can load.
Preparing the evidence before an incident
Investigations fail when records were never written or a privileged attacker could change them. Fix these in peacetime.
- Record to log streams: SMF can write to data sets or to system logger log streams. Log streams allow per-record-type routing and are a prerequisite for record signing. IBM warns that if SMF runs out of buffers there may be a loss of data, so size buffers and monitor for it.
- Turn on SMF record signing: RECSIGN in SMFPRMxx (off by default as NORECSIGN) adds digital signatures using a key pair held in ICSF and identified by a PKCS#11 token name, with HASH and SIGNATURE options such as SHA512 and RSA. IBM's material notes the log stream must be defined with a MAXBUFSIZE of 65532. Protect the key through the CRYPTOZ, CSFSERV and CSFKEYS classes so the people being audited cannot use it.
- Set RACF auditing: SAUDIT, CMDVIOL, OPERAUDIT and failure logging, as set out in the RACF guide linked above. More logging rarely breaks anything; missing logging cannot be fixed later.
- Protect database audit: create Db2 audit policies for sensitive tables and privileged authorities, and mark the important ones secure (DB2START 'S', which only SECADM can stop) or tamper-proof (DB2START 'T', which needs external authority to modify or stop).
- Get SMF off the platform in near real time: forward type 80, 30, 90, 119 and 102 events to the SIEM as they are written, not in a nightly batch. See the SIEM Guide.
- Set retention to match investigations: keep SMF and OPERLOG long enough to cover a typical dwell time and any regulatory minimum, on storage the security team controls. See the Log Management and Retention Guide.
- Keep baselines: regular, hashed copies of the security database unload, PARMLIB, PROCLIB, the APF list and the link list give you something to compare against.
- Write the mainframe annex: add z/OS contacts, out-of-hours escalation, who may issue which commands and the batch calendar to the incident response plan, and exercise it once a year.
Triage: the first questions
Detection usually arrives as a SIEM alert on type 80 data, an unexpected large FTP transfer, an application owner reporting changed data, or a ransomware event elsewhere that reached mainframe-connected systems. Ask these questions in the first hour.
- Which identity: a person, a service ID, a started task or an API connector? Was the user ID used from its normal source (IP address, terminal, partner system) and at its normal time?
- Which privilege: did the ID hold or gain SPECIAL, OPERATIONS or AUDITOR, a new PERMIT, SURROGAT authority or a PassTicket path? Look for type 80 ALTUSER, CONNECT, PERMIT and SETROPTS events in the window.
- Which data: types 14 and 15 show data sets read and written; 17 and 18 show deletions and renames; Db2 audit shows table access. Is this read-only exfiltration or modification?
- Which path in and out: type 119 shows TN3270, FTP and TCP sessions. A large FTP server transfer completion record (subtype 70) to an unknown address is a strong exfiltration signal.
- Is the system itself changed: type 90 subtypes 29 and 31, new APF-authorised libraries, new PROCLIB members or started task definitions suggest persistence below the application layer.
- Is logging intact: check for gaps in SMF intervals, SETROPTS changes that reduce logging, stopped Db2 audit traces and failed signature validation. A gap is itself a finding.
Containment without stopping the business
Agree each step with the operations and application owners, record the exact command, the issuer and the time, and prefer narrow actions first. The table is ordered roughly from least to most disruptive.
| Action | How it is done | Effect and risk |
|---|---|---|
| Raise logging on the suspect ID | ALTUSER with UAUDIT | No service impact; captures every further action by that ID |
| Revoke a human user ID | ALTUSER with REVOKE (ESM equivalents in ACF2 and Top Secret) | Blocks new logons; existing sessions may continue until cancelled |
| Remove a new or suspicious permit | PERMIT with DELETE, then refresh in-storage profiles with SETROPTS RACLIST REFRESH where the class is RACLISTed | Removes the access path; save the unload first so the change is recorded as evidence |
| Cancel a running job or session | Operator CANCEL for the job or TSO user | Stops the activity immediately; may leave data sets partly updated |
| Disable a delegation path | Remove SURROGAT or PassTicket (PTKTDATA) access for the affected application | Stops impersonation through a portal or connector; the application may fail until reissued |
| Restrict network entry | Block the source address in the IP filter or upstream firewall; stop the FTP server if exfiltration is live | Stops the transfer path; stopping FTP also stops legitimate partner transfers |
| Revoke a service ID | ALTUSER with REVOKE on the batch, started task or connector ID | High impact: jobs and connectors using it fail. Prefer removing the specific compromised access path first |
| Isolate an LPAR | Vary network interfaces offline or remove the system from the sysplex under the site's recovery procedure | Last resort for system-level compromise; plan with the capacity and DR teams |
For a compromised service ID, the safer pattern is often to issue a replacement ID with narrowed access, move the jobs to it during a planned window, then revoke the original. See the Mainframe Service Accounts and Batch Job Identity Guide.
Forensic acquisition and integrity
z/OS evidence is mostly structured records and configuration, which makes acquisition fast if it is planned. Collect in this order, hash each output file and record it on a chain of custody form. See the Evidence Handling and Chain of Custody Guide.
- SMF: dump the relevant log streams for the incident window plus a margin with IFASMFDL, using NOSIGSTRIP so the signature records are kept (they are dropped by default).
- Validate SMF signatures: run IFASMFDP with SIGVALIDATE and the same HASH and TOKENNAME values as SMFPRMxx. A report ending VALIDATION SUCCEEDED supports integrity; message IFA742I reasons such as records not matching expected counts point to inserted or deleted records. Only one error is reported per run, so rerun after narrowing the range.
- OPERLOG and SYSLOG: copy the sysplex OPERLOG for the window, and the per-system SYSLOG from spool before it is purged.
- Security database: unload with IRRDBU00 (or the ACF2 or Top Secret equivalent) and keep a copy of the database itself if a full restore may be needed.
- System configuration: copy the active PARMLIB members (SMFPRMxx, PROGxx, IEASYSxx), PROCLIB members for started tasks, the APF list and LNKLST as displayed by operator commands, and TCP/IP and FTP server configuration.
- Job output and dumps: preserve JES job logs for suspect jobs and any SVC dumps taken during the incident.
- Data and volumes: where data tampering is suspected, take or protect a point-in-time copy of the affected volumes and analyse it outside production.
Run collection under a dedicated responder ID with UAUDIT, write outputs where suspect IDs cannot alter them, and copy them off-platform. The general method is in the Digital Forensics (DFIR) Guide.
Analysis and timeline
Normalise everything to UTC, then join type 80, 30, 14/15, 119 and Db2 records on user ID, job name and time to build a single timeline. The type 30 job identifier links a logon to the programs it ran; type 119 adds the network source. Check for clock changes (type 90 SET DATE subtypes) before trusting the order.
| Behaviour | ATT&CK technique | Mainframe evidence |
|---|---|---|
| Logon with stolen or shared credentials | T1078 Valid Accounts | Type 80 JOBINIT from an unusual source; type 119 TN3270 or FTP session from a new address |
| Password guessing against TSO, FTP or TN3270 | T1110 Brute Force | Bursts of type 80 failures and revocations; type 119 subtype 72 FTP server logon failures |
| Bulk transfer of data sets | T1048 Exfiltration Over Alternative Protocol | Type 14 reads of sensitive data sets followed by type 119 FTP transfer completions |
| Changing records in application data | T1565.001 Stored Data Manipulation | Type 15 updates or Db2 audit records from IDs or jobs that do not normally write |
| Deleting data sets or catalogs | T1485 Data Destruction | Type 17 scratch, type 65 catalog delete records in volume |
| Encrypting data for ransom | T1486 Data Encrypted for Impact | Mass type 15 updates by one job; application failures reading data that is no longer valid |
Persistence on z/OS tends to sit in configuration: a new APF-authorised library, a PROCLIB member that runs under a powerful started task ID, a catch-all started task profile, or an extra permit on a sensitive class.
Eradication and recovery
- Access: remove every permit, connect and privileged attribute added by the attacker, using the before and after unloads as the list. Reset or replace credentials for affected IDs, and regenerate PassTicket keys and any keys the attacker could have read.
- System: remove unauthorised APF entries, PROCLIB members and link list changes, then verify with the same display commands used during collection. Where system-level tampering is confirmed, rebuild from a known-good system copy rather than editing in place.
- Data: for logical corruption or ransomware, find the last clean point. Immutable point-in-time copies, such as IBM's Safeguarded Copy within the IBM Z Cyber Vault approach, are analysed in an isolated system and used for surgical recovery of specific data or catastrophic recovery of the whole environment. See the Ransomware Response and Recovery Guide.
- Logging: confirm SMF, RACF and Db2 auditing are back at their intended settings and that signature validation succeeds on fresh records before declaring recovery.
- Notification: mainframes hold regulated financial, health and personal data, so involve legal early. See the Breach Notification Laws by Country Guide.
Choosing an approach by estate size
- Single LPAR, small team: SMF forwarding, a written mainframe annex, pre-built collection JCL and an external retainer with z/OS skills.
- Multi-sysplex estates: standard SMFPRMxx and signing everywhere, central OPERLOG, a security-owned collection ID and an isolated recovery system.
- Outsourced operations: make evidence collection, response times and command authority contractual.
Detection and measurement
- Time from a privileged RACF change to its SIEM alert (target minutes, not hours).
- Percentage of SMF log streams with record signing turned on, and signature validation success on a weekly sample.
- Number of SMF buffer shortage or data loss events per quarter (target zero).
- Db2 subsystems with secure or tamper-proof audit policies on sensitive tables.
- Time to collect the standard evidence set in an exercise, measured from call-out.
- Configuration drift findings against PARMLIB, PROCLIB, APF and security database baselines.
Framework mapping
| Framework | Reference | How this guide applies it |
|---|---|---|
| NIST CSF 2.0 | PR.PS-04, DE.CM-09 | SMF generation and forwarding; monitoring of system and data activity |
| NIST CSF 2.0 | RS.AN-03, RS.AN-06, RS.AN-07 | Timeline analysis; recording responder actions; integrity and provenance of collected SMF and unloads |
| NIST CSF 2.0 | RS.MI-01, RS.MI-02 | Containment and eradication steps on z/OS |
| NIST SP 800-53 Rev 5 | AU-9, AU-10, AU-11, AU-12 | Protected, signed and retained audit records |
| NIST SP 800-53 Rev 5 | IR-4, SI-7, CP-9, CP-10 | Incident handling, integrity checks, immutable copies and recovery |
| ISO/IEC 27001:2022 Annex A | A.5.24 to A.5.28, A.8.15 | Incident management planning, response, learning, collection of evidence and logging |
| CIS Controls v8.1 | Control 8, Control 17 | Audit log management and incident response management |
Common failure patterns
- SMF only on tape: records are dumped nightly and the SIEM sees nothing in real time, so the attacker has a day of undetected activity.
- Subject controls the evidence: the system programmer under suspicion runs the collection jobs, or SPECIAL users can alter SMF data sets and the signing key.
- Containment by mass revoke: every service ID touched by the incident is revoked at once and the batch window fails, turning a security incident into an outage.
- Profiles fixed before unload: administrators clean up permits before anyone takes a copy of the security database, destroying the record of what the attacker added.
- Mainframe missing from the IR plan: the team has no z/OS contact, no collection JCL and no agreed command authority, so the first hours are spent finding people.
- Restore without validation: data is restored from a backup that already contained the corruption because no one analysed copies before recovery.
Practitioner checklist
- Add a mainframe annex to the IR plan with contacts, command authority and the batch calendar.
- Record SMF to log streams, turn on RECSIGN and protect the signing key in ICSF.
- Forward types 80, 30, 90, 119 and 102 to the SIEM in near real time.
- Create secure or tamper-proof Db2 audit policies for sensitive tables and privileged authorities.
- Keep hashed baselines of the security database unload, PARMLIB, PROCLIB, APF and link lists.
- Pre-build and test collection jobs (IFASMFDL with NOSIGSTRIP, IFASMFDP with SIGVALIDATE, IRRDBU00) under a dedicated responder ID.
- Agree a containment ladder with operations so narrow actions come before revoking service IDs.
- Maintain immutable point-in-time copies and an isolated environment to validate them before recovery.
- Exercise a mainframe scenario at least once a year and measure time to collect evidence.
Standards and references
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- IBM: z/OS SMF records
- IBM: Signing and validating SMF records
- IBM: SMF digital signatures (RACF presentation)
- IBM: RACF event codes
- IBM: SMF 119 record subtypes
- IBM: Db2 for z/OS audit policies
- IBM Redbooks: IBM Z Cyber Vault
Related NHI Mgmt Group resources: Mainframe Identity and RACF Security Guide · Mainframe and COBOL Application Security Guide · Incident Response Guide · Anti-Forensics Detection Guide