Structured Analytic Techniques for CTI Guide: Reducing Bias in Threat Assessments

Threat intelligence assessments fail more often through flawed reasoning than through missing data: an analyst anchors on the first plausible actor, reads every new artefact as confirmation, and presents a guess with the confidence of a fact. Structured analytic techniques (SATs) are disciplined methods, developed in government intelligence analysis, that make assumptions visible, force alternatives to be considered and express uncertainty consistently. This guide explains the techniques that matter most for cyber threat intelligence (CTI), especially Key Assumptions Check and Analysis of Competing Hypotheses (ACH), shows how to apply them to attribution, incident scoping and strategic assessments, and covers how to express likelihood and confidence. It is for CTI analysts, incident responders writing findings and the managers who consume their judgements. For the CTI programme itself see the Threat Intelligence Guide.

Key takeaways

• Structure exposes reasoning to challenge. The value of SATs is that a colleague can see the assumptions and evidence behind a judgement and disagree with a specific step.
• Start with a Key Assumptions Check. It is quick, and it catches the most common CTI failure: an attribution or scoping conclusion resting on an unstated premise such as "this toolkit is exclusive to one group".
• ACH works by trying to disprove, not prove. The strongest hypothesis is the one with the least inconsistent evidence, not the one with the most supporting evidence.
• Separate likelihood from confidence. US intelligence analytic standards (ICD 203) define fixed likelihood terms and say a confidence level and a likelihood term should not be combined in one sentence.
• Use the techniques in proportion. A quick check for a daily brief, full ACH for attribution, insider or major incident questions, and contrarian techniques before a high-stakes decision.

Why CTI needs structured techniques

Richards J. Heuer Jr.'s Psychology of Intelligence Analysis (CIA, 1999) brings cognitive psychology research to the question of how analysts judge incomplete and ambiguous information. Two observations from its introduction apply directly to cyber work: information that confirms an existing judgement is perceived more vividly than information that contradicts it, and analysts often dismiss deception simply because they see no evidence of it, before any systematic search has been made. CTI is particularly exposed because:

• Artefacts are shared and stolen: malware, infrastructure and tooling are reused, sold, leaked and deliberately copied, so technical overlap is weaker evidence of identity than it appears.
• Vendor naming fragments the picture: one cluster of activity can carry several vendor names, and different vendors draw cluster boundaries differently. See the Cyber Kill Chain and Diamond Model Guide for structuring the evidence itself.
• Visibility is partial: logs are missing, retention is short and telemetry is biased towards what the organisation already monitors.
• Time pressure is real: incident decisions are made in hours, which favours the first plausible story.

The biases that show up most often in cyber assessments, and the technique that counters each:

• Anchoring: the first actor name mentioned in a vendor alert or chat channel becomes the reference point. Counter with ACH, which forces every alternative onto the matrix.
• Confirmation bias: new artefacts are read as supporting the current story. Counter by listing inconsistent evidence first.
• Mirror imaging: assuming the adversary values what you would value, for example that a criminal group would never sit quietly for months. Counter with Red Team Analysis.
• Availability: the group in last week's headlines seems most likely. Counter with a Quality of Information Check on how the hypothesis entered the discussion.
• Premature closure: the case is "solved" before scoping is complete. Counter with Indicators of Change and a scheduled re-test.

The core techniques at a glance

The US Government's A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis (March 2009) groups techniques into three families. The table maps them to CTI questions.

Key Assumptions Check in practice

The Tradecraft Primer describes four steps: write down the current analytic line; identify all premises, stated and unstated, it depends on; challenge each by asking whether it must be true and whether it holds in all conditions; then reduce the list to assumptions that must be true and identify what information could invalidate them. Applied to a typical incident judgement:

Record the result alongside the assessment: which assumptions were well supported, which were caveated and which were removed. Revisit it when new evidence arrives.

Analysis of Competing Hypotheses step by step

ACH is described in Chapter 8 of Heuer's book as an eight-step methodology, and the Tradecraft Primer gives a closely related sequence. In CTI form:

1. Generate hypotheses: brainstorm with people of different perspectives (incident responders, threat intelligence, the business owner). Include unwelcome options: insider, red team or penetration test activity, misconfiguration, a second unrelated intruder.
2. List evidence and arguments: indicators, timelines, tools, infrastructure, victimology, absence of expected artefacts, and key assumptions from the check above.
3. Build the matrix: hypotheses across the top, evidence down the side; mark each cell consistent, inconsistent or not applicable.
4. Refine: merge or split hypotheses, remove evidence that does not discriminate between any of them (it adds no diagnostic value).
5. Try to disprove: focus on inconsistent evidence. The hypothesis with least inconsistency is the strongest, however much evidence "fits" the others.
6. Test sensitivity: identify the few items the conclusion depends on and ask what happens if one is wrong, misread or planted.
7. Look for missing evidence and deception: what should exist if each hypothesis were true, but does not? Could an actor have staged false flags?
8. Report all hypotheses: state the leading judgement and the alternatives still worth monitoring.
9. Set indicators: define observations that would strengthen or weaken each hypothesis, and who will watch for them.

In this illustrative example H2 has no inconsistent evidence, so it leads, but it rests heavily on the "no extortion contact" item: if a demand arrives later, the assessment must be revisited. The commodity tool row does not discriminate and could be dropped. That is the kind of reasoning a reader can audit and challenge.

Expressing likelihood, confidence and source quality

Consistent language matters as much as method. Intelligence Community Directive 203, Analytic Standards (signed 2 January 2015 and later technically amended), sets nine tradecraft standards, including properly describing the quality and credibility of sources, properly expressing uncertainty, distinguishing information from assumptions and judgements, incorporating analysis of alternatives and explaining changes to judgements. It also defines a set of likelihood terms:

ICD 203 encourages analysts not to mix terms from different rows and says products should not combine a confidence level and a degree of likelihood in the same sentence. Confidence describes the analyst's basis for a judgement: the logic, the quantity and quality of sources and the understanding of the topic. A practical CTI pattern is two separate sentences: "We assess it is likely that the intrusion was conducted by a state-aligned actor. We have moderate confidence in this judgement because it rests on victimology and the absence of extortion, not on unique tooling." Whichever scale you adopt (ICD 203, a national equivalent, or your own), publish it with your products so readers interpret words the same way.

• Source grading: many teams grade source reliability and information credibility separately, for example with the Admiralty (NATO) system of letters A to F for reliability and numbers 1 to 6 for credibility. A single vendor blog repeating an unverified claim should not be graded like a direct observation in your own logs.
• Handling: mark products with TLP so sharing matches source sensitivity. See the Traffic Light Protocol Guide.

Embedding techniques in the CTI workflow

Phased rollout

1. Agree a language standard: adopt a likelihood scale, a confidence definition and a source grading scheme, and add them to every product template.
2. Mandate a Key Assumptions Check: for every attribution, scoping or "are we affected?" judgement, recorded in the case notes.
3. Pilot ACH on two cases: one closed incident (to compare with what was later proven) and one live question. Use a spreadsheet; tools are optional.
4. Add peer challenge: a second analyst reviews the matrix or plays devil's advocate before high-impact products are released.
5. Track judgements: keep a log of key judgements with likelihood and confidence, and review them when outcomes become known.
6. Adjust: if techniques slow urgent work, keep the check and the language standard, and reserve full ACH for questions that drive major decisions. Reverting is simple because the techniques are process, not tooling.

Decision guidance

Small teams: a one-person CTI function gains most from the language standard and the assumptions check, plus borrowing a colleague from incident response as a challenger. Large teams: formalise peer review, maintain a judgement log and run periodic retrospective accuracy reviews.

Measuring the effect

• Coverage: share of attribution and scoping products with a recorded assumptions check.
• Language compliance: share of products using the agreed likelihood and confidence terms correctly.
• Revision rate: how often key judgements are later revised, and whether revisions were anticipated by recorded alternatives.
• Calibration: for resolved judgements, whether "likely" outcomes occurred roughly as often as the scale implies.
• Consumer feedback: whether decision makers report that uncertainty was clear enough to act on.

Common failure patterns

• Attribution by tool overlap: treating a shared or leaked tool as a fingerprint. Run the assumptions check on exclusivity first.
• ACH as decoration: filling in a matrix after the conclusion is fixed, with only supporting evidence listed. The test is whether inconsistent evidence was actively sought.
• Mixed likelihood language: "possibly", "may" and "could" used for anything from 10 to 70 per cent, so readers cannot compare judgements.
• Confidence and likelihood in one phrase: "highly likely with high confidence" hides which part is the probability and which is the basis.
• Ignoring absence of evidence: concluding "no data was taken" because no exfiltration was seen, when the logs that would show it were never collected.
• Never revisiting: an early incident hypothesis becomes the final report without re-testing as evidence arrives.

Framework mapping

Define intelligence requirements before analysis starts, so techniques are applied to questions that matter. See the Intelligence Requirements Template and the Threat Intelligence Maturity Model.

Practitioner checklist

• Adopt and publish a likelihood scale, a confidence definition and a source grading scheme.
• Record a Key Assumptions Check for every attribution, scoping or exposure judgement.
• Include unwelcome hypotheses (insider, authorised testing, misconfiguration, second intruder) in every ACH.
• Rank hypotheses by inconsistent evidence and run a sensitivity check on the few items the conclusion depends on.
• State likelihood and confidence in separate sentences, never combined.
• Have a second analyst challenge high-impact products before release.
• Define indicators that would change each judgement and assign someone to watch them.
• Keep a judgement log and review accuracy when outcomes are known.

Standards and references

• ODNI: Intelligence Community Directive 203, Analytic Standards
• US Government: A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis
• CIA Center for the Study of Intelligence: Psychology of Intelligence Analysis (Heuer)
• NIST: Cybersecurity Framework 2.0
• NIST: SP 800-53 Rev. 5
• FIRST: Traffic Light Protocol (TLP) 2.0

Related NHI Mgmt Group resources: Threat Intelligence Guide · Cyber Kill Chain and Diamond Model Guide · Intelligence Requirements Template · Threat Intelligence Analyst Career Guide