Hierarchical F1 Score

Hierarchical F1 score is an evaluation method that gives partial credit when a prediction lands on a parent or closely related category rather than the exact label. In security classification tasks, it helps measure semantic usefulness when taxonomies have nested relationships.

Expanded Definition

Hierarchical f1 score is a classification metric used when labels sit inside a tree or nested taxonomy, so a prediction can be partially correct even when it misses the exact leaf label. That matters in security, where incident types, malware families, policy domains, and identity abuse patterns are often organised into parent-child structures rather than flat classes. Instead of treating every error as equally wrong, the metric rewards outputs that remain semantically close to the target, which makes it more informative than standard F1 in domains with overlapping or nested labels.

For example, a model that predicts credential compromise when the ground truth is a more specific token theft event may still receive partial credit if the taxonomy explicitly places both under the same parent. Definitions vary across vendors and research teams because there is no single standard formula governing how much credit a parent label should receive. NHI Management Group recommends treating the metric as a taxonomy-aware evaluation choice, not as a universal replacement for precision and recall. For broader operational context, teams often compare it with governance expectations in the NIST Cybersecurity Framework 2.0. The most common misapplication is using hierarchical F1 on a flat label set, which occurs when no validated parent-child taxonomy exists.

Examples and Use Cases

Implementing hierarchical F1 score rigorously often introduces taxonomy-maintenance overhead, requiring organisations to balance more meaningful evaluation against the cost of curating and versioning label hierarchies.

• An email security team evaluates phishing classification where credential harvesting, brand impersonation, and BEC sit under a shared abuse tree, so near-miss predictions still count as partially useful.
• A SOC fine-tunes a model to classify alerts into nested families such as malware, ransomware, and fileless malware, using the metric to see whether the model stays within the correct branch.
• A fraud or IAM analytics team scores detections across a hierarchy of account abuse patterns, where a prediction of impossible travel may be closer to the truth than a completely unrelated access anomaly.
• An NHI governance team evaluates agent activity classification, where a predicted secret misuse category is treated as closer to token exfiltration than to generic suspicious execution.
• A threat intelligence group uses a taxonomic label set mapped to MITRE ATT&CK-aligned internal categories, but applies hierarchical F1 only after confirming the hierarchy is stable and consistently annotated.

The metric is especially useful when the business needs semantic proximity, not only exact-match performance. It can show that a model is learning the right area of the taxonomy even when annotation boundaries remain imperfect.

Why It Matters for Security Teams

Security teams rely on metrics to decide whether a classifier is ready for automation, triage prioritisation, or analyst support. If hierarchical F1 is misunderstood, teams may overestimate model quality by rewarding near misses that are operationally too broad, or underestimate value by judging a partially correct prediction as a total failure. That risk is acute in security taxonomies, where the difference between a parent and a leaf label can change response actions, escalation paths, or containment decisions.

This matters for identity and NHI workflows as well. In environments that classify service accounts, API tokens, workload identities, or agent behaviours, a hierarchical label set can reflect how real incidents move from broad suspicion to specific cause. Used well, the metric helps teams see whether a model is learning useful structure before it is trusted in production. Used poorly, it can hide weak boundary precision and create false confidence in downstream automation. For governance alignment, teams should anchor taxonomy design to the expectations expressed in NIST Cybersecurity Framework 2.0 and evaluate whether the hierarchy supports consistent decision-making. Organisations typically encounter the operational cost of a bad metric only after a model routes the wrong case to the wrong playbook, at which point hierarchical F1 becomes necessary to explain why the system looked better on paper than in practice.

Related resources from NHI Mgmt Group

• F1 Score
• When should organisations escalate a high-risk identity score?
• What is the best way to score AI agent workflows in production-like environments?
• How do compliance teams turn score improvement into real risk reduction?