Why do token-based leaderboards create risk when organisations try to measure AI productivity?

Why This Matters for Security Teams

Token-based leaderboards are risky because they can turn AI usage into a vanity metric. When teams are ranked on consumption, the metric starts shaping behavior: people generate more prompts, longer outputs, and unnecessary iterations to climb the board. That can increase spend, hide low-quality work, and create false confidence that AI adoption is delivering value. For security teams, the concern is not just cost control. It is also governance, because incentives that reward volume can undermine review discipline, data handling expectations, and change control.

This matters in environments where AI tools touch code, customer data, or operational decisions. A leaderboard can pressure staff to use AI in ways that look productive but are hard to validate, especially when the organisation has not defined what “good output” means. Current guidance from the NIST Cybersecurity Framework 2.0 is to measure outcomes and risk together, not activity alone. In practice, many security teams discover the problem only after spend has spiked and the organisation has already normalised noisy, low-value AI usage rather than disciplined delivery.

How It Works in Practice

The failure mode is simple: a leaderboard introduces a score, and the score becomes the objective. If the metric is tokens consumed, users will naturally optimise for more prompts, more generated text, and more AI-assisted activity, regardless of whether that activity improves the result. In mature environments, this can distort engineering, support, and knowledge work. It may also create hidden security exposure if people paste sensitive material into models just to increase throughput or appear active.

Better practice is to build a measurement model around outcomes, then keep AI spend as a separate control signal. That usually means pairing productivity metrics with quality, timeliness, and risk checks.

• Measure tasks completed, defects reduced, tickets resolved, or decisions supported.
• Track AI spend, usage volume, and approval boundaries as governance indicators.
• Review output quality through sampling, peer review, or workflow checkpoints.
• Define which data types are permitted in prompts and which require redaction or blocking.

This is aligned with broader AI governance thinking in the NIST AI Risk Management Framework, which emphasises managing impacts, not just observing activity. For organisations deploying agentic workflows, the question becomes even more important because autonomous systems can generate large amounts of action with limited human oversight. The practical control is to make the leaderboard secondary to validated business outcomes, then use anomalies in token usage as a signal for review rather than a badge of performance. These controls tend to break down when AI usage is embedded into informal team culture without clear approval rules, because volume is then mistaken for value before anyone notices the drift.

Common Variations and Edge Cases

Tighter measurement often increases reporting overhead, requiring organisations to balance visibility against administrative burden. That tradeoff is real, especially where teams want a simple scorecard and leadership wants quick evidence of AI adoption. But the simplicity of token-based ranking is exactly what makes it fragile.

There is no universal standard for AI productivity scoring yet, so current guidance suggests treating leaderboards as an internal engagement tool at most, not a performance truth. In some cases, usage can still be informative. For example, a support team may want token trends to understand demand patterns, or a platform team may use them to estimate cost allocation. Even then, the metric should be contextual, not competitive.

Edge cases matter most when the work is regulated, customer-facing, or safety-sensitive. In those settings, high usage can indicate rework, poor prompts, weak model selection, or a control gap rather than genuine efficiency. The same caution applies where AI agents can act on behalf of users, because volume may reflect delegated activity rather than human productivity. The better approach is to combine output measures with review controls, documented acceptable-use rules, and periodic checks against NIST Cybersecurity Framework 2.0 implementation goals. Organisations that rely on token leaderboards alone often end up rewarding the most AI-hungry behaviour, not the most effective work.

Related resources from NHI Mgmt Group

• Why do chat-based AI systems create new identity risk for organisations?
• Why does shadow AI create more risk when organisations try to prohibit it?
• Why do consumption based AI contracts create budgeting risk for organisations?
• When does AI adoption create more identity risk than productivity gain?