Most organisations buy or subscribe to threat intelligence feeds without ever measuring whether a feed adds anything the others do not, arrives in time to matter, or floods the SIEM with indicators that match legitimate traffic. The research is not encouraging: the largest academic comparison of feeds, published at USENIX Security 2019, found that most indicators appeared in only one feed and that many IP feeds contained unroutable or content delivery network addresses. This guide gives threat intelligence and detection engineering teams a repeatable way to evaluate feeds with measurable criteria: volume, overlap and exclusive contribution, latency, coverage, accuracy and false positive rate, ageing, and fit to your own threat model. It covers how to collect the data, how to use MISP's built-in overlap, warninglist, sightings and decaying features, and how to decide what to keep. Choosing a platform to hold the feeds is covered in the Threat Intelligence Platform Buyer's Guide; this page is about judging the feeds themselves.
Key takeaways
- Measure feeds against each other and against your own telemetry. Overlap, exclusive contribution and latency only make sense relative to other feeds; accuracy and relevance only make sense relative to what you actually see.
- Low overlap is the norm, not a sign of quality. Li et al. found that over 97 per cent of IP indicators in most categories were unique to one feed; a feed that shares nothing may be finding new threats or may be noise.
- False positives are measurable before deployment. Check every feed against lists of known benign infrastructure (cloud and CDN ranges, top domains, reserved address space); MISP warninglists package many of these.
- Indicators age, so scoring must too. An IP address or phishing domain has a short useful life; use expiry, sightings and decay models rather than blocking on indicators indefinitely.
- Atomic indicators sit at the bottom of the Pyramid of Pain. Hashes and IPs are cheap for an adversary to change; weight feeds that supply context, tooling and behaviour, not only volume.
The evaluation metrics
Li, Dunn, Pearce, McCoy, Voelker, Savage and Levchenko defined six metrics in "Reading the Tea Leaves: A Comparative Analysis of Threat Intelligence" (USENIX Security 2019). They remain the clearest vocabulary for feed evaluation. The third column adds how to compute each one in practice.
| Metric | Definition (Li et al.) | How to measure it in your environment |
|---|---|---|
| Volume | How many indicators a feed contains over the measurement period. | Count unique indicators by type (IPv4, domain, URL, SHA256) per day over 30 to 90 days. |
| Differential contribution | The share of feed A's indicators not in feed B: |A\B| / |A|. | Pairwise matrix across all feeds of the same indicator type; MISP's feed overlap analysis matrix does this on cached feeds. |
| Exclusive contribution | The share of a feed's indicators that appear in no other feed. | Set difference against the union of every other feed you hold, including free and government sources. |
| Latency | How much later a feed reports an indicator than the first feed that reported it. | Record first-seen timestamps per feed at ingestion; compute the median delay for shared indicators. |
| Coverage | The share of indicators a feed should contain that it actually includes (like recall). | Use a ground truth set: indicators from your confirmed incidents, honeypots or a trusted advisory, and check how many each feed had and when. |
| Accuracy | The share of a feed's indicators that are correctly included (like precision), measured as an upper bound. | Test against benign lists and against sightings marked false positive by your analysts. |
What the 2019 study found: scan and brute force IP feeds overlapped the most, with half of their pairwise intersection rates above 5 per cent, while botnet, exploit, malware and spam feeds mostly shared under 1 per cent. Six of eight malware hash feeds had over 95 per cent exclusive contribution. On accuracy, 33 of 47 IP feeds contained at least one unroutable address, 12 contained reserved addresses, 21 included IPs tied to top Alexa domains, and some reputation feeds overlapped significantly with CDN ranges. Treat these as illustrations of the problem from that study period, not as current scores for any feed.
Relevance: does the feed match your threat model?
A technically accurate feed can still be worthless to you. Before measuring anything, write down what you need the feed for.
- Use case: blocking at the perimeter, SIEM enrichment and alerting, retrospective hunting, or strategic reporting. Blocking needs very high accuracy; hunting tolerates more noise.
- Threat profile: sectors, geographies and actor types that matter to you. A ransomware-focused feed is of little value to an organisation whose top risk is state espionage against edge devices, and the reverse.
- Indicator types you can act on: if you have no TLS or JA3-style telemetry, network fingerprint indicators cannot be matched. Map each indicator type to a log source you actually hold.
- Context: does each indicator carry a first seen date, a confidence or score, a threat type and a link to an actor, malware family or ATT&CK technique? Context is what turns a match into a decision.
- Format and transport: STIX 2.1 over TAXII, MISP format, CSV or free text. MISP can ingest MISP, CSV and free-text feeds; structured formats preserve context. See the STIX and TAXII Guide.
Pyramid of Pain check: David Bianco's model ranks indicators by how much pain detecting them causes an adversary, from hash values, IP addresses and domain names at the bottom, through network and host artefacts and tools, to tactics, techniques and procedures at the top. A feed made only of hashes and IPs is cheap for attackers to outrun. Give weight to feeds and reports that describe artefacts, tools and behaviours you can turn into durable detections.
Running a feed evaluation: step by step
- Define scope and ground truth. Pick the indicator types and use cases in scope. Build a ground truth set from your last 12 months of confirmed incidents, phishing reports and honeypot or sandbox output, plus indicators from trusted government advisories.
- Ingest in parallel without acting. Load candidate feeds into a test MISP instance or TIP with blocking and alerting disabled. In MISP, enable caching so feeds can be correlated and compared without creating events, and use "Override IDS Flag" for free-text or CSV feeds under test so nothing reaches detection exports.
- Normalise. Lower-case domains, strip URL parameters consistently, expand or reject CIDR ranges by policy, and convert all hashes to one type where the feed supplies several. Without normalisation, overlap figures are meaningless.
- Collect for 30 to 90 days. Record per indicator: feed, first seen in feed, last seen, type, any confidence or tags. Shorter windows overstate exclusivity because slow feeds have not caught up.
- Compute the metrics. Volume, pairwise differential contribution, exclusive contribution, median latency against the earliest feed, coverage against ground truth, and the benign match rate described below.
- Retro-hunt against your telemetry. Match each feed's indicators against 30 to 90 days of DNS, proxy, firewall, EDR and email logs. Count matches, then have analysts classify a sample as true or false positive, recording the result as MISP sightings.
- Score and decide. Combine the metrics into a scorecard (next section), compare against cost and licence terms, and decide keep, replace, restrict to hunting only, or drop.
- Promote gradually. Move accepted feeds to alerting first, then to blocking only for indicator types and confidence levels that proved accurate. Keep a documented rollback: a single switch that removes a feed's indicators from blocking lists.
Measuring false positives and accuracy
| Check | Method | Signal of a poor feed |
|---|---|---|
| Known benign infrastructure | Match against MISP warninglists such as cloud provider ranges (for example the amazon-aws, cloudflare and akamai lists), CDN ranges and top-domain lists (Cisco Umbrella and Cloudflare Radar top lists). | Many matches to shared cloud or CDN addresses that would block legitimate services. |
| Reserved and unroutable address space | Compare IPs with private, loopback, documentation and other special-purpose ranges. | Any match suggests poor validation; Li et al. found this in most IP feeds studied. |
| Common false positive hashes | Match hashes against the common-ioc-false-positive warninglist and your software inventory. | Hashes of empty files, common installers or your own signed software. |
| Scanner and research infrastructure | Warninglists for research scanners (for example Censys and CIRCL scanning ranges). | Feed treats internet-wide research scanners as attackers, inflating volume. |
| Analyst verdicts | Sample retro-hunt matches and record true and false positive sightings. | False positive share above your use-case tolerance, for example above 1 per cent for blocking. |
| Self-references | Look for the publisher's own report URLs or domains as indicators; MISP's exclusion regex can strip them on ingest. | Indicators that are just links back to the feed's own reports. |
MISP's warninglists are described by the project as lists of well-known indicators that can be associated with potential false positives, errors or mistakes. They raise a warning at event and attribute level and can filter at the API level. Run them on every feed at evaluation and keep them enabled in production. Detail on running MISP itself is in the MISP Threat Sharing Guide.
Freshness, latency and indicator ageing
- Latency matters most for fast-changing indicators: phishing URLs and C2 IPs may be useful for days. A feed that reports them a week after others adds little for blocking, though it may still help hunting.
- Last seen matters as much as first seen: feeds that never expire indicators grow without limit and accumulate reassigned addresses. Check whether the feed supplies expiry, valid until or last seen fields, and whether it removes indicators.
- Decay scoring: MISP decaying models compute a current score from a base score (from taxonomy tags such as confidence) multiplied by a time-decay factor based on how long since the indicator was last seen. Parameters include lifetime (days until the score reaches 0), decay speed and a threshold below which an indicator is marked decayed. MISP ships Polynomial, Polynomial (Extended), which honours the retention taxonomy, and Sightings formulas, the last of which weights the score by the ratio of true to false positive sightings.
- Simulate before enabling: MISP's model editor plots the score over time and can score real attributes from a REST search, so you can tune lifetime and threshold against your data.
- Delta merge for snapshot feeds: for feeds that publish a full current list each pull, MISP's Delta Merge option keeps only attributes from the most recent fetch and soft-deletes old ones.
Building a feed scorecard
| Criterion | Weight (example for SIEM alerting) | Scoring guide |
|---|---|---|
| Relevance to threat model | 20 | Proportion of indicators tagged to sectors, regions or actors in your profile. |
| Accuracy | 20 | Benign match rate and analyst false positive rate; zero if above tolerance. |
| Exclusive contribution | 15 | Unique indicators that also produced true positive matches, not raw uniqueness. |
| Coverage against ground truth | 15 | Share of your confirmed incident indicators present, and how early. |
| Latency | 10 | Median hours behind the first feed for shared indicators. |
| Context richness | 10 | First and last seen, confidence, threat type, actor or ATT&CK mapping present. |
| Operational fit | 10 | Format, API stability, rate limits, licence terms and TLP handling. |
Decision guidance: for perimeter blocking, accuracy dominates and a feed with any meaningful benign match rate should be restricted to alerting. For hunting, exclusive contribution and context dominate and some noise is acceptable. Small teams should start with a few well-curated sources, such as government advisories and sector ISAC feeds, before paying for breadth they cannot triage; see the ISACs and Information Sharing Communities Guide. Large teams should rerun the evaluation annually and before every renewal, because feed quality changes.
Licensing and sharing terms
Read the terms before measuring: some commercial feeds forbid redistribution or derived products, and many community feeds are marked with the Traffic Light Protocol. Under FIRST TLP 2.0 (authoritative from August 2022), TLP:CLEAR has no disclosure limit, TLP:GREEN may be shared within the community but not publicly, TLP:AMBER within the recipient's organisation and its clients on a need-to-know basis, TLP:AMBER+STRICT within the recipient's organisation only, and TLP:RED with the individual recipients only. Your TIP must carry these markings through to any export, including feeds you push to managed security providers.
Where identity fits
Feed integrations run on API keys and tokens, and MISP's feed configuration accepts HTTP headers such as an Authorization header to consume authenticated feeds. These are machine credentials that unlock paid data and, for bidirectional sharing, the ability to publish into a community. Store them in a secrets manager, scope them to read-only where the provider allows, rotate them at renewal, and record an owner for each. Identity-specific indicators, such as leaked credentials, malicious OAuth applications and token theft infrastructure, need their own evaluation against identity telemetry; see the Threat Intelligence for Identity and NHI Guide.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | ID.RA-02 | Cyber threat intelligence is received from information sharing forums and sources. |
| NIST CSF 2.0 | DE.AE-07 | Threat intelligence and other context are integrated into analysis. |
| NIST SP 800-53 Rev. 5 | PM-16 Threat Awareness Program; SI-5 Security Alerts, Advisories, and Directives | Programme-level intake and handling of external threat information. |
| NIST SP 800-53 Rev. 5 | RA-3 Risk Assessment | Threat information informs risk assessment and therefore feed relevance. |
| ISO/IEC 27001:2022 | Annex A 5.7 Threat intelligence | Collect and analyse information about threats to produce threat intelligence. |
Detection and measurement in production
- True positive yield per feed: confirmed true positive alerts per 1,000 indicators per month.
- False positive rate per feed: share of feed-driven alerts closed as benign; review any feed above tolerance.
- Exclusive true positives: true positives that no other feed would have produced; the strongest argument for renewal.
- Median latency: hours behind the earliest source for indicators confirmed in your incidents.
- Share of decayed or expired indicators still in blocking lists: target zero.
- Cost per exclusive true positive: annual feed cost divided by exclusive true positives, for renewal decisions.
Feed-driven alert volume is one of the main causes of alert fatigue; tie these measures into the Alert Fatigue Reduction Guide work and the wider Security Operations Metrics Guide.
Common failure patterns
- Buying on volume: choosing the feed with the most indicators, which often means the most stale and benign entries.
- Blocking straight from a new feed: pushing an untested feed to the firewall and blocking a cloud or CDN range that hosts business services.
- Never expiring indicators: years-old IPs, now reassigned, still generating alerts or blocks.
- Counting overlap without normalising: case, trailing dots and URL variants make identical indicators look unique and inflate exclusive contribution.
- No feedback loop: analysts close false positives without recording sightings, so the same bad indicators keep firing and the feed is never challenged at renewal.
- Ignoring sharing terms: re-sharing TLP:AMBER or licensed data with partners or into public reports.
Practitioner checklist
- Write down the use case and threat profile each feed must serve before evaluating it.
- Build a ground truth set from your confirmed incidents and trusted advisories.
- Ingest candidate feeds into a test instance with alerting and blocking disabled.
- Normalise indicators, then compute volume, differential and exclusive contribution, latency, coverage and accuracy.
- Run every feed against MISP warninglists and your own allow-lists.
- Retro-hunt 30 to 90 days of telemetry and record analyst verdicts as sightings.
- Score with a weighted scorecard and restrict low-accuracy feeds to hunting.
- Configure expiry or decay models before any feed reaches blocking.
- Vault feed API keys, assign owners and rerun the evaluation before each renewal.
Standards and references
- USENIX: Reading the Tea Leaves: A Comparative Analysis of Threat Intelligence
- MISP Project: MISP user guide (Managing Feeds, Sightings, Decaying Models)
- MISP Project: misp-warninglists
- FIRST: Traffic Light Protocol (TLP) 2.0
- David J Bianco: The Pyramid of Pain
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: Threat Intelligence Guide · Threat Intelligence Platform Buyer's Guide · MISP Threat Sharing Guide · Detection Engineering Guide