Macro F1

Expanded Definition

Macro F1 is a class-balanced evaluation metric, so it treats each label as equally important even when the dataset is imbalanced. That makes it especially useful in security and governance contexts where rare classes can represent the outcomes that matter most, such as abuse, compromise, or policy violation.

Unlike weighted or micro-averaged scores, Macro F1 does not let large classes dominate the final result. A model can look strong overall while still failing badly on minority classes, and Macro F1 is designed to expose that gap. In practice, this means the metric is often more honest for detection problems than accuracy alone.

The boundary to watch is that Macro F1 is a measurement choice, not a guarantee of better real-world performance. It can reward balanced treatment of all classes even when one class is operationally far more important, so the right use depends on whether each class truly deserves equal attention. For a broader control context, NIST SP 800-53 Rev 5 Security and Privacy Controls helps frame the surrounding governance and monitoring expectations.

Examples and Use Cases

Macro F1 is most often used when a model must perform reliably across all categories, not just the majority class. In security workflows, that is common when rare classes carry high operational value or risk.

• Evaluating a phishing classifier where legitimate mail is common, but rare malicious campaigns must still be detected.
• Comparing alert triage models that classify events into multiple incident types, including low-frequency but high-impact categories.
• Testing a secrets detection system that labels findings by severity or leak type, where uncommon classes should not be ignored.
• Assessing an access-review classifier that separates routine access from anomalous or unauthorized patterns across several labels.
• Measuring an agent or workflow classifier that routes requests into multiple action classes, especially when minority classes represent exceptions or escalations.

The main tradeoff is that Macro F1 can penalise a model heavily for poor performance on rare classes even when the overall volume is small. That is useful when minority failures matter, but it can be misleading if some classes are noisy, weakly defined, or not equally actionable.

Security Implications

In security programs, Macro F1 matters because skewed datasets can hide failure on the very class an organisation cares about most. A detector that performs well on normal traffic but misses rare malicious cases may still produce a respectable aggregate score if the evaluation is not class-balanced.

That creates a governance problem as well as a technical one: teams may believe a model is ready for production when it is actually uneven across critical labels. The practical symptom is often a system that looks stable in aggregate reports but fails during edge cases, escalation paths, or low-frequency abuse patterns.

NHIMG research shows that 80% of identity breaches involved compromised non-human identities such as service accounts and API keys, which is a useful reminder that rare classes can be disproportionately important in real security outcomes. Macro F1 helps surface whether a model is truly treating those minority outcomes as first-class cases rather than statistical noise. In other words, the metric can expose whether detection quality is balanced enough for operational trust.

Domain and Governance Relevance

Macro F1 is not a governance control by itself, but it is highly relevant to model oversight, especially when security teams use ML for detection, classification, prioritisation, or routing. It forces reviewers to ask whether the evaluation standard matches the operational stakes of the problem.

In NHI and agentic AI settings, this matters when models classify events involving service accounts, tokens, secrets, or autonomous actions. If rare compromise states, privilege-abuse patterns, or exception paths are under-scored, teams can miss the failure modes that matter most to identity assurance and response readiness.

For that reason, Macro F1 is most valuable when paired with clear class definitions, documented label ownership, and an explicit decision about whether equal class weighting is justified. The metric is strongest when it supports governance decisions about model acceptance rather than merely producing a single headline score.

Related resources from NHI Mgmt Group

• Why do grounded findings matter more than raw F1 scores for security tools?
• F1 Score
• Hierarchical F1 Score
• Macro Recall