A concurrency unit is a capacity measure that defines how many active generations a tenant can run at once. It is not the same as request volume or connection count. For governed AI platforms, concurrency units determine how rate limits, queueing, and rejection behaviour should be enforced for streaming workloads.
Expanded Definition
Concurrency unit is a governed capacity measure for how many active generations a tenant may run at the same time. In practice, it is an execution guardrail for streaming AI workloads, not a proxy for total request count, open connections, or overall throughput. That distinction matters because tenant-facing limits must account for how long a generation stays active, how much state it consumes, and how long it occupies shared model resources.
In NHI and agentic AI environments, concurrency units help security and platform teams separate identity and workload authorisation from raw traffic shaping. A tenant may be permitted many requests but only a small number of simultaneous generations, especially when model tools, retrieval systems, or downstream secrets are involved. Industry usage is still evolving, and no single standard governs this term yet, so vendors may measure it differently across burst windows, per-session tokens, or worker occupancy. For that reason, governance documents should define how the unit is counted and when it is decremented or released. The most common misapplication is treating concurrency units as simple request rate limits, which occurs when teams ignore active-session duration and overcommit shared inference capacity.
For broader identity and control context, the NHI Mgmt Group guidance on Ultimate Guide to NHIs is useful alongside the NIST NIST Cybersecurity Framework 2.0 for capacity governance and resilience planning.
Examples and Use Cases
Implementing concurrency units rigorously often introduces queueing and rejection tradeoffs, requiring organisations to weigh predictable tenant isolation against lower peak utilisation.
- A customer support agent tenant is allowed five concurrent generations, so a sixth prompt is queued until one active stream ends.
- An internal coding assistant gets a higher concurrency allowance during business hours, but tool-enabled generations are capped more tightly than plain text responses.
- A regulated workload uses per-tenant concurrency units to prevent one business unit from consuming shared inference capacity during incident response.
- Platform teams tie concurrency units to session duration so that long-running streaming outputs do not starve shorter, higher-priority jobs.
- Review of service-account execution patterns in the Ultimate Guide to NHIs helps teams map workload behaviour to identity governance rather than treating it as a pure infrastructure metric.
Capacity definitions also benefit from external control language. The NIST Cybersecurity Framework 2.0 is useful when organisations need to connect availability controls, recovery planning, and operational limits to business services.
Why It Matters in NHI Security
Concurrency units become a governance issue when AI agents, service accounts, and API-driven workflows share model access under different trust levels. If the limit is too loose, one tenant or one compromised NHI can monopolise active generations, creating denial-of-service conditions and masking abusive automation. If it is too strict, legitimate workflows stall, and teams may disable controls to restore productivity. That tension is especially important in agentic systems where a single generation can trigger tool calls, secrets retrieval, or multi-step action chains.
NHI Mgmt Group research shows that 97% of NHIs carry excessive privileges, which means capacity controls cannot be treated as an afterthought when execution authority is already broad. The practical security goal is not just to cap volume, but to ensure that simultaneous execution is bounded, observable, and attributable to the right tenant or identity. Pairing concurrency policy with service-account visibility from the Ultimate Guide to NHIs helps teams detect when automated actors begin exceeding expected runtime patterns. Organisations typically encounter concurrency issues only after a production slowdown or abuse event, at which point the term becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST Zero Trust (SP 800-207) and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | IA-2 | Agent execution limits and tool use depend on governing concurrent active sessions. |
| OWASP Non-Human Identity Top 10 | NHI-05 | Concurrency policy relies on the identity and workload boundaries of non-human actors. |
| NIST CSF 2.0 | PR.AC-4 | Least-privilege access should extend to how much simultaneous workload a tenant can consume. |
| NIST Zero Trust (SP 800-207) | SC.L3 | Zero trust limits blast radius by constraining what each actor may consume at once. |
| NIST AI RMF | Risk controls for AI systems include operational guardrails on active workload execution. |
Document concurrency as a risk treatment and monitor for overload, abuse, and unfair allocation.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 27, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
