The knowledge problem is why smarter systems still fail

At a glance

What this is: This article argues that the real failure mode in modern data and agentic systems is a knowledge problem, not a compute or modelling problem.

Why it matters: That matters to IAM practitioners because autonomy, access decisions, and identity governance all break down when systems assume they know more than the people and processes that actually hold the context.

👉 Read Tonic's analysis of the knowledge problem in smart systems

Context

Modern data and AI systems often fail when architects assume centralisation produces understanding. The article frames this as a knowledge problem: the information needed to make good decisions is dispersed across people, teams, and moments, which means a single model or platform cannot safely absorb every decision context. In identity and access programmes, that same flaw shows up when policy engines or automation are asked to act without local nuance.

For IAM, PAM, and NHI governance teams, the lesson is that more automation does not automatically mean better control. Agentic systems, like human-operated workflows, still depend on context, escalation paths, and limits on when a machine should act versus when it should defer. That makes the topic relevant across human identity, workload identity, and agentic AI governance.

The starting position here is common in mature organisations, not exceptional. Most security teams already know that access decisions can fail when ownership, asset state, or operational impact is not captured correctly, but the article usefully extends that problem into AI systems and autonomous workflows.

Key questions

Q: How should security teams govern context access for autonomous workers?

A: Security teams should treat context as a separate control surface from action. An autonomous worker may be allowed to perform a task without being allowed to read, combine, or retain every supporting data source. That separation reduces the chance that a session accumulates more knowledge than the task requires.

Q: Why do AI agents and automation tools break down in complex organisations?

A: They break down because complex organisations distribute knowledge across people, systems, and moments. When an agent acts as if that knowledge is centralised and stable, it will make confident decisions based on incomplete evidence. That creates brittle behaviour, especially where privilege, ownership, or downstream impact are changing quickly.

Q: What do security teams get wrong about centralising data for smarter decisions?

A: Teams often assume that more centralised data automatically creates better decisions. In reality, centralisation can strip away local nuance, tacit judgment, and short-lived operational context. If the decision depends on those factors, the model or platform will be precise without being correct, which is a serious governance failure.

Q: Who is accountable when automated privacy workflows make the wrong decision?

A: Accountability remains with the organisation, not the workflow. Privacy, security, legal, and system owners must define decision boundaries, review thresholds, and escalation paths so automation supports policy enforcement instead of replacing human responsibility for sensitive cases.

Technical breakdown

Dispersed knowledge and control decisions

Hayek’s central idea is that the knowledge needed to run a complex system is distributed across many actors, not held in one place. In security operations, that means the right decision may depend on a local workaround, a short-lived exception, or a business context that never appears in the platform record. When systems centralise data too aggressively, they often remove the very signal that makes the decision safe. That is especially relevant in identity programmes where access, privilege, and ownership all change faster than records are updated.

Practical implication: design approval and escalation paths that preserve local context instead of forcing every decision through a single rigid model.

Tacit knowledge in agentic workflows

Tacit knowledge is what experienced people use without being able to fully write it down. In agentic workflows, that becomes a control problem because an AI system cannot reliably infer what operators know about risk, fragility, or downstream blast radius. The result is not always a visible failure. More often, the system acts confidently in situations where a human would pause, escalate, or narrow scope. That makes tacit knowledge a governance input, not just a human habit.

Practical implication: require human override points wherever business impact or privilege scope cannot be fully formalised.

Signals over state in autonomous systems

The article argues that complex systems work better through signals than through claims of global understanding. In practice, a signal is a narrow, decision-useful indicator that an agent can act on safely, such as policy boundaries, risk thresholds, or lifecycle state. This is a useful model for NHI and agentic AI security because it shifts design away from pretending the system knows everything and toward making bounded decisions with verifiable inputs. That is closer to how resilient identity controls should operate.

Practical implication: replace broad autonomy with tightly scoped signals, policy checks, and explicit stop conditions.

NHI Mgmt Group analysis

False certainty is the real control failure in autonomous systems. The article’s strongest point is that centralised platforms often create confidence without context, which is a governance weakness rather than a technical achievement. In identity and access terms, this is what happens when policy engines, automation, or AI workflows are asked to decide without enough operational knowledge. Practitioners should treat overconfident automation as a control gap, not an optimisation.

Knowledge dispersion should be treated as a design constraint, not an exception. Security programmes still fail when they assume that ownership, risk, and context can all be normalised into one authoritative record. That assumption is especially fragile in NHI and agentic AI environments, where access scope, execution timing, and business meaning can change faster than governance records. The practical conclusion is that control design must tolerate incomplete knowledge.

Signals are a better governance primitive than global state. The article correctly points toward coordination through bounded signals rather than universal visibility. That maps well to identity security, where the useful questions are often about whether a system has enough evidence to proceed, not whether it has perfect understanding. For practitioners, the lesson is to govern by policy-relevant signals, not by the illusion of complete system knowledge.

Agentic AI amplifies existing IAM assumptions rather than replacing them. The article’s argument extends beyond AI architecture because agentic workflows still depend on identity, privilege, and escalation rules. If a system cannot know enough to act safely, then its identity model must make that limitation visible and enforceable. Teams should therefore treat agent governance as an IAM problem as much as an AI problem.

What this signals

Knowledge dispersion will become a governance test for AI programmes. As autonomous systems move from experiment to operations, teams will need to prove that policy, ownership, and escalation can survive incomplete context. That is a design challenge for IAM, PAM, and AI governance teams, not just a data architecture issue.

Context-aware controls are more valuable than broad claims of intelligence. The most resilient programmes will treat bounded signals, explicit overrides, and scoped authorisation as first-class controls. Where a workflow cannot explain why it is acting, it should not be allowed to proceed on autonomy alone.

The practical signal for readers is that identity governance must now extend into agent behaviour, not just user lifecycle. That means aligning policy, review, and accountability with the moments where automation loses context, rather than assuming the platform will infer the right answer. See also the OWASP NHI Top 10 for adjacent agentic risk framing.

For practitioners

• Map where automation depends on tacit context Identify workflows where operators routinely override the system because the documented state is incomplete, wrong, or stale. Those paths should be treated as governance hotspots, especially when access decisions, privilege changes, or agent actions are involved.
• Add explicit human escalation points Require intervention whenever an agent or workflow cannot prove that it has enough context to proceed safely. This is most important for privileged actions, lifecycle exceptions, and any decision that could create persistent access or unexpected blast radius.
• Constrain autonomy with bounded signals Give systems narrow, verifiable signals such as policy state, risk thresholds, and task scope instead of asking them to reason over an entire world model. That makes the control plane easier to audit and reduces false confidence in automated decisions.
• Review identity records for context loss Check where CMDB data, ownership tags, and access inventories strip away the contextual detail operators actually use. If the record cannot support the decision, the workflow needs either better metadata or a mandatory review step before execution.

Key takeaways

• The article’s core warning is that smart systems fail when they are asked to replace dispersed human knowledge with centralised certainty.
• For identity and access programmes, the risk is not just automation error but governance collapse when context, ownership, and escalation paths are stripped away.
• Practitioners should govern autonomous systems with bounded signals, explicit overrides, and context-preserving controls rather than relying on broad claims of intelligence.

Key terms

• Tacit Knowledge: Operational judgment that people use every day but do not fully capture in policy, tickets, or systems of record. In enterprise security, tacit knowledge often includes maintenance timing, dependency risk, and exception handling that determine whether a technically correct action is actually safe.
• Dispersed Knowledge: Dispersed knowledge is information spread across many people, systems, and time periods rather than held in one central record. In governance terms, it explains why central platforms often miss local context, especially when decisions depend on ownership, business impact, or transient operational conditions.
• Signal-Driven Coordination: Signal-driven coordination is an operating model in which agents act on meaningful changes such as thresholds, ownership shifts, or stalled remediation. It avoids asking each agent to model the whole enterprise, while still allowing coordinated action through predefined triggers and escalation logic.

What's in the full article

Tonic's full article covers the conceptual argument and examples that this post intentionally leaves for the source:

• The Hayek framing behind the knowledge problem and why it matters for system design
• The sequence of examples showing how local context gets lost in centralised platforms
• The case for signals-based coordination in agentic workflows rather than global understanding
• The next-part teaser that connects this concept to omniscient-agent design

👉 Tonic's full article expands the argument with the Hayek framing, examples, and the next step in the series.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It is designed for practitioners who need to connect identity controls to broader security operations.