The knowledge problem is why smarter systems still fail

At a glance

What this is: This is an analysis of why increasingly capable data and AI systems still break in practice, with the central finding that knowledge is dispersed, tacit, and constantly changing.

Why it matters: IAM, NHI, and AI governance teams should treat context loss as a control problem, because autonomy built on incomplete knowledge can produce brittle decisions, bad escalations, and misplaced trust.

Context

Modern AI and data systems often fail not because the models are weak, but because the environment they operate in is fragmented. A centralised platform can store information, yet still miss the local, tacit, and time-sensitive knowledge that people use to make safe decisions. In identity and security programmes, that gap matters because access, privilege, and trust decisions are rarely reducible to a single canonical record.

Tonic frames this as a knowledge problem rather than a compute problem. That framing is relevant to agentic AI and to governance more broadly, because the risk is not just automation error, but automation acting confidently when the needed context lives outside the system. The result is brittle orchestration, misplaced confidence in dashboards, and control models that assume more shared understanding than actually exists.

Key questions

Q: What breaks when agentic systems assume complete knowledge?

A: They become brittle because centralised state cannot capture the local, tacit, and time-sensitive knowledge that real decisions depend on. When an agent acts as if it has full context, it can overstep, misclassify risk, or execute the wrong workflow. The failure is not intelligence, but overconfidence under partial information.

Q: Why does dispersed knowledge increase automation risk?

A: Because the most relevant information often lives outside the model, outside the record, and inside human judgement. If that knowledge has to be centralised before it can be used, it may already be stale or incomplete. Automated systems then make decisions on what looks complete but is no longer accurate.

Q: How should teams design agentic systems for partial information?

A: Use bounded tasks, explicit stop conditions, and escalation paths for uncertainty. The goal is not to make agents omniscient, but to keep them useful when they only have fragmentary evidence. That requires narrow authority, clear signals, and a human decision point when confidence is low.

Q: When should human judgment override automated decisions?

A: Whenever the decision depends on context that cannot be reliably encoded, verified, or kept current. If a system is acting on stale state, conflicting ownership, or a situation that changes faster than the control cycle, human review should take precedence before the action is committed.

Technical breakdown

Why centralised state breaks under distributed knowledge

Complex organisations rarely hold all operational truth in one place. Data can be centralised, but the meaning of that data remains distributed across people, teams, and moments in time. That is why a single source of truth can still produce poor decisions: the source is complete only on paper, while reality changes faster than normalisation, approval, and reporting cycles. In AI and identity systems, this becomes a governance issue when downstream automation treats stale state as current fact.

Practical implication: Design controls so that high-risk decisions can be challenged by local context before automation acts.

Tacit knowledge and the limits of rule-based automation

Tacit knowledge is the judgement people use without being able to fully write it down. It shows up when an operator knows an alert is noise, or when an engineer senses that a fix will break something else. Rule engines, schemas, and agentic workflows struggle here because they require explicit inputs, but the most important signals are often incomplete, situational, and hard to codify. The technical failure is not lack of intelligence; it is overconfidence in formalisation.

Practical implication: Keep humans in the loop for cases where context cannot be reliably encoded or inferred.

Signals over state for agentic coordination

Agentic systems become brittle when they assume shared world models and global understanding. A better pattern is signal-based coordination, where agents act on bounded evidence, local feedback, and narrow objectives rather than on an illusion of full awareness. This reduces the chance that one wrong model, stale record, or missing context cascades into broader failure. For identity-adjacent systems, that means governing what an agent can infer and act upon, not just what data it can access.

Practical implication: Constrain agent behaviour to bounded signals and scoped decisions rather than open-ended autonomy.

NHI Mgmt Group analysis

Knowledge scarcity is now a governance problem, not just an AI design problem. The article correctly identifies that organisations often overestimate how much can be centralised, normalised, and safely automated. In identity and security programmes, that same mistake appears when teams trust dashboards more than operational context. The practitioner lesson is to govern for decision quality under partial knowledge, not for perfect data completeness.

Signal-based coordination is the right mental model for agentic systems. Agents do not need omniscience to be useful, but they do need bounded authority, clear stop conditions, and explicit escalation paths. This is especially relevant where AI agents interact with credentials, approvals, or workflow state, because overconfident action on incomplete context can create policy drift. The practitioner implication is to design for narrow decisions and visible handoffs, not broad autonomous reasoning.

Context decay is the hidden failure mode behind many brittle systems. By the time operational knowledge reaches a central system, its meaning may already have changed. That creates a context-to-control gap, where the control plane believes it has reliable state but the business has already moved on. The practitioner implication is to treat stale context as a control defect, especially in identity, access, and security workflows.

Human judgment remains a permanent control layer in complex environments. The article’s strongest point is that automation does not eliminate the need for interpretation; it changes where interpretation must happen. For IAM, NHI, and agentic AI programmes, that means escalation design matters as much as detection design. The practitioner implication is to preserve judgment at the point where uncertainty is highest, not after the system has already committed to an action.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Identity Guide

What this signals

Context-to-control gap: agentic systems fail when the control layer assumes it has a full and current picture, while operational knowledge remains local and transient. Practitioners should treat stale context as a governance defect, not a data-quality nuisance.

The next design question for security and identity teams is not whether an agent can reason, but whether it can be constrained to act safely on incomplete information. That shifts attention toward bounded authority, explicit escalation, and decision points where human judgement can still interrupt automation.

For practitioners

• Define escalation points for uncertain decisions Require AI and workflow systems to route ambiguous or high-impact cases to a human reviewer before they can change access, approve actions, or close incidents.
• Limit agent autonomy to bounded objectives Scope each agent to narrow tasks with explicit inputs, outputs, and stop conditions so it cannot infer broad intent from incomplete context.
• Preserve local operational knowledge in controls Capture frontline exceptions, owner notes, and temporary workarounds in the process that governs decisions, not only in a central record.
• Test whether dashboards reflect reality Compare system state with what operators actually know in live scenarios to expose where normalised data has drifted away from working knowledge.

Key takeaways

• Smart systems fail when they are built on the assumption that knowledge can be fully centralised, because the most important context is often local and transient.
• The core risk is not compute shortage but false certainty, where automation acts as if it understands more than it actually does.
• Security and identity teams should design for bounded autonomy, explicit escalation, and human override when decisions depend on unresolved context.

Key terms

• Tacit Knowledge: Operational judgment that people use every day but do not fully capture in policy, tickets, or systems of record. In enterprise security, tacit knowledge often includes maintenance timing, dependency risk, and exception handling that determine whether a technically correct action is actually safe.
• Context-to-Control Gap: The context-to-control gap is the distance between what an automated system can technically see and what it needs to know to act safely. It appears when tacit business knowledge, undocumented dependencies, or exception handling are missing from the data available to the system.
• Signal-Based Coordination: A design pattern where systems act on bounded, local signals rather than assuming a shared, complete world model. It is useful in agentic environments because it limits overreach when no single system has enough context to decide safely.
• Bounded Autonomy: Bounded autonomy means a system can act independently within defined limits, but cannot exceed those limits without human or policy control. In agentic governance, the boundary must be explicit, testable, and logged, because the real compliance question is where autonomous action stops.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners build governance that holds when automation must operate with incomplete context.