Hierarchical F1 Score

How Hierarchical F1 Score Works

Hierarchical F1 score extends ordinary F1 by giving partial credit when a model predicts a label that is not exact but still sits higher in the taxonomy, or is otherwise semantically close to the right class. That makes it useful when categories are nested, and where a near miss can still preserve meaningful security signal.

It is not a replacement for exact-match evaluation. Instead, it answers a different question: did the model land somewhere useful in the label structure, or did it miss by an amount that really matters to the task?

Why Taxonomy Structure Changes the Score

In flat classification, every mistake is equally wrong. In hierarchical settings, some errors are less severe because the label system itself encodes relationships such as parent, child, or sibling categories. Hierarchical F1 reflects that structure, so the metric can better match operational usefulness when the output space contains domain groupings rather than isolated labels.

That matters in security classification because analysts often care about whether a prediction points toward the right family of events, assets, or control issues even if it does not hit the finest-grained label. A parent category may be sufficient for triage, routing, or summarisation, while an unrelated category is much less helpful.

Where It Is Most Useful

Hierarchical F1 is most valuable when the label taxonomy is deliberately designed with semantic nesting, such as attack types, alert families, policy classes, product groups, or incident subtypes. In those settings, the metric can distinguish between a model that is broadly directionally right and one that is simply noise.

It is especially useful when downstream consumers prefer “good enough” categorisation over brittle exactness, or when training data is sparse and fine-grained labels are hard to separate reliably. The metric can expose improvements that exact F1 hides, while still penalising broad misses.

For security teams, that makes it a practical way to evaluate classifiers that support investigation workflows, label enrichment, or risk triage, where coarse correctness can still create value.

Limits, Trade-offs, and Interpretation

Hierarchical F1 is only as good as the taxonomy behind it. If the label tree is inconsistent, poorly curated, or not aligned to how practitioners actually work, the score can reward predictions that look structurally close but are operationally unhelpful. The metric also depends on how distance or partial credit is defined, which means different implementations can produce different results.

Because of that, hierarchical F1 should be read alongside exact F1, per-class results, and error analysis. A high hierarchical score can conceal weak fine-grained discrimination, while a lower score may still be acceptable if the taxonomy is deep and the task truly rewards near misses. The point is to measure usefulness, not to relax evaluation discipline.

Practitioner Guidance

What to watch for: Use hierarchical F1 when the taxonomy itself is part of the task design, not when labels are effectively flat. If the model is meant to support analyst decisions, make sure the hierarchy reflects how those decisions are actually made, otherwise the metric can overstate practical performance.

Practitioner takeaway: Treat hierarchical F1 as a complement to exact-match evaluation, not a substitute. It is most defensible when a partial prediction still carries real operational value in the target workflow.

Related resources from NHI Mgmt Group

• How should teams choose F1 score instead of accuracy for imbalanced classification problems?
• Why does F1 score give a more honest signal than accuracy in skewed datasets?
• What is the difference between F1 score and accuracy in model evaluation?
• What is the difference between F1 score and false discovery rate in model evaluation?