Progress Per Dollar

Expanded Definition

Progress per dollar describes the practical value an AI system delivers for each unit of cost, not just how strong it looks on a benchmark. In security work, that means measuring whether the model can sustain useful output across many attempts, long runtimes, and constrained budgets, especially where the system is being used operationally rather than in a one-off demonstration.

The term is most relevant when comparing systems that may have similar peak capabilities but very different operating costs. A model that is slightly weaker in a single test can still be more useful if it can be run more often, at larger scale, or with better recovery from failure. This is why the metric is often more meaningful than leaderboard placement for offensive use cases.

Guidance-vs-consensus note: there is no single industry standard for calculating progress per dollar. Some practitioners include inference cost only, while others factor in orchestration, retries, human review, and integration overhead. The key boundary is that the metric should reflect usable security output, not abstract model quality.

Examples and Use Cases

Progress per dollar shows up in security planning whenever teams compare AI systems by operational value rather than headline capability. It is especially useful in environments where repeated execution, prompt iteration, or batch analysis matters more than a single perfect answer.

• An adversarial testing team compares two models and selects the one that can generate enough high-quality attempts within a fixed budget.
• A security research group evaluates whether a cheaper model can produce adequate triage output across thousands of indicators instead of using a premium model for every query.
• A red team assesses whether a system remains usable when cost controls, rate limits, or token limits force shorter runs and more retries.
• An AI security lead compares vendor options by the amount of actionable output produced per dollar spent on inference and orchestration.

The trade-off is simple: the cheapest model is not always the best, but the most capable model on paper can still be the worst choice if it is too expensive to use repeatedly in the workflow.

Security Implications

When progress per dollar is ignored, attackers and defenders can both make poor decisions. A model that looks strong in isolation may be ineffective in a real campaign if it is too expensive to run at the volume needed for iterative probing, tool use, or large-scale filtering. Conversely, a low-cost model may be good enough to automate noisy but persistent activity at scale.

The main failure mode is budget misalignment. Teams may overestimate capability because they focus on benchmark rank, then discover that operational cost prevents sustained use. In offensive settings, that gap can make repeated attempts unaffordable or limit exploit development throughput. In defensive settings, the same gap can cause underinvestment in automation because the cost model hides the actual amount of security work the AI can perform.

A practitioner should watch for systems that perform well once but collapse in usefulness when repeated queries, long context, or tool calls are introduced. That is often where the real economic boundary appears.

Domain and Governance Relevance

Progress per dollar matters in AI security because cost efficiency changes who can use a model, how often it can be used, and at what scale. For offensive security, lower cost can translate into more attempts, more probing, and more persistence. For defensive security, better value per dollar can make detection, classification, and content review more sustainable.

In NHI and agentic environments, the metric becomes especially important when AI is connected to tools, service accounts, or automated workflows. Cost efficiency affects whether autonomous actions can be repeated reliably enough to matter, which in turn influences how much trust operators place in the system. That makes the term relevant to control design, budget ownership, and workload selection, not just model comparison.

NHIMG treats this as a governance question as well as a performance question: organisations should understand whether they are paying for occasional capability or for durable operational output. That distinction often determines whether the system can support real security work at scale.

Related resources from NHI Mgmt Group

• What do teams get wrong about per-seat licensing in agentic environments?
• Should security teams prefer tenant-scoped sync over per-realm provisioning models?
• What breaks when session policy is global instead of per application?
• How should security teams enforce per-client authorization in MCP environments?