Custom Metric

A custom metric is a measurement defined for a specific business or operational objective rather than a generic model score. It helps teams assess whether a model is producing outcomes that matter to the organisation, not just outputs that look technically correct.

Expanded Definition

A custom metric is a purpose-built measure that translates model behaviour into a business-relevant signal, such as task success, false escalation rate, policy violation rate, or time saved per workflow. Unlike a generic evaluation score, a custom metric reflects how an organisation defines value in context, so it can capture operational quality, risk, and compliance outcomes at the same time. In AI and automation programmes, this is especially important because a model can appear accurate while still producing the wrong practical result for the user, the process, or the control environment.

Definitions vary across vendors and teams because custom metrics are usually designed around the specific deployment, not a universal standard. A custom metric may be deterministic, threshold-based, weighted, or composite, and it may combine outputs from multiple systems. For governance purposes, the metric should be documented, reproducible, and tied to an explicit decision or control objective. NIST’s Cybersecurity Framework 2.0 is useful here because it emphasises measurable outcomes and governance alignment rather than isolated technical indicators.

The most common misapplication is treating a generic model score as a custom metric, which occurs when teams reuse an off-the-shelf accuracy measure without validating that it matches the actual business objective.

Examples and Use Cases

Implementing custom metrics rigorously often introduces measurement overhead, requiring organisations to balance better decision quality against added design, validation, and maintenance effort.

• A fraud team tracks “manual review rate avoided” rather than only model precision, because the operational goal is to reduce analyst workload without increasing financial risk.
• A customer support organisation measures “first-contact resolution assisted by AI” to determine whether the model actually improves service outcomes, not just response generation quality.
• An NHI governance team defines “unauthorised tool invocation rate” for an autonomous agent, because a technically successful output still represents a security failure if the agent used an unapproved action path.
• A security operations team uses a custom metric for “high-confidence alert reduction without missed incidents” to evaluate whether a triage model improves SIEM and SOAR workflows.
• A compliance team measures “policy-conformant output rate” when testing LLM-assisted drafting, so that evaluation reflects regulatory and control expectations, not only linguistic fluency.

Where a metric affects identity, access, or automation controls, the evaluation design should also consider whether the measured behaviour can be reliably attributed to the right user, workload, or system governance process. That is especially important when outputs influence privileged actions or delegated authority.

Why It Matters for Security Teams

Security teams need custom metrics because generic measures can hide risk. A model that optimises for speed may still increase false approvals, policy drift, or unsafe automation. A model that looks “accurate” in a lab may fail under adversarial inputs, messy production data, or unusual identity states. That makes custom metrics essential for judging whether an AI system, workflow, or control actually supports the intended security outcome.

This matters across AI security, cybersecurity operations, and identity governance. For example, if an agent is allowed to act on behalf of a human operator, the organisation needs metrics that capture action correctness, privilege containment, and exception handling, not just response quality. In governance terms, custom metrics help turn abstract requirements into auditable evidence that a control is working as intended. They also support continual improvement by showing where a deployment is drifting from its original risk posture.

Practitioners typically encounter the cost of weak metrics only after a failed review, an audit challenge, or an incident reveals that the system was optimising the wrong outcome, at which point custom metrics become operationally unavoidable to correct the failure.

Related resources from NHI Mgmt Group

• When should organisations prefer standards over custom implementations?
• When do custom IAM roles become a governance problem?
• How should security teams choose between basic, predefined, and custom GCP IAM roles?
• What is the difference between predefined roles and custom roles in GCP IAM?