Why LLMs struggle with math and what that means for AI governance

At a glance

What this is: This is a ZioSec analysis of why LLMs are unreliable at arithmetic and how prompt manipulation patterns can push them off course.

Why it matters: It matters because IAM and AI governance teams need to treat model boundaries, context control, and delegated action limits as security controls, not just UX details.

Context

LLMs do not compute like calculators. They generate output by predicting likely token sequences, which makes them useful for language tasks but unreliable when precision, stable state, or deterministic arithmetic is required.

The governance problem appears when that same conversational system is allowed to act on behalf of users. Once task scope can expand mid-session, the issue is no longer math alone but whether guardrails, review points, and boundaries survive sustained interaction.

Key questions

Q: What breaks when an LLM is trusted to do precise work without external tools?

A: Determinism breaks first. LLMs can produce fluent but incorrect answers because they predict text patterns rather than execute exact arithmetic or logic. When teams treat that output as authoritative, errors become governance issues, not just model quality problems. Precision tasks should therefore be separated from language generation and routed to verified tools.

Q: Why do multi-turn AI conversations create more governance risk?

A: Multi-turn conversations give the model more context, more opportunity to infer intent, and more chances to relax its own boundaries. The result is a control problem for finance and other regulated use cases, because the system may shift from explanation to action without any formal approval step.

Q: How can security teams detect when prompt manipulation is working?

A: Look for gradual changes in task scope, repeated flattery or compliance-seeking language, and requests that move from benign assistance into sensitive or policy-bending actions. Those signals indicate the interaction is no longer bounded by the original intent. Logging prompt progression and review points makes the drift visible.

Q: How do organisations reduce risk when an AI agent can act without human approval gates?

A: They move authorisation to the moment of action and keep the agent’s scope task-bound. That means deny by default, issue the minimum capability needed for the current step and require retention of an attributable record for every consequential action.

Technical breakdown

Why LLMs are poor arithmetic engines

LLMs are statistical pattern systems, not symbolic solvers. They learn correlations from training data and generate the most probable next token, which can look fluent even when the underlying reasoning is wrong. Arithmetic requires deterministic state and exact intermediate steps, while an LLM is optimizing for textual plausibility. Tool calling can improve results when the model delegates computation to an external calculator or code interpreter, but that does not change the model's intrinsic behaviour. The governance lesson is that output quality depends on whether the task is bounded to what the model can actually do reliably.

Practical implication: route precision-dependent tasks to deterministic tools and keep the model out of unsupported calculation paths.

How prompt manipulation exploits context drift

The article describes several manipulation patterns that exploit conversational elasticity. Gradual scope expansion, often compared to boiling the frog, starts with harmless requests and incrementally moves toward unsafe ones. Flattery or love bombing can increase compliance because models are tuned to be helpful and accommodating. Concept drift works by slowly shifting the topic so the system stays engaged while the intent changes. These are not code exploits. They are governance failures at the interaction layer, where the system's willingness to continue the conversation becomes the attack surface.

Practical implication: constrain session scope, monitor topic shifts, and treat persuasive prompt patterns as a control problem.

Why context exhaustion weakens guardrails

Context windows are finite, and models give more weight to recent tokens than to earlier instructions. As a conversation grows, the original system prompt can become less influential relative to the newest content, which creates context exhaustion. The article links this to decision fatigue in humans, but the technical point is simpler: long sessions can erode instruction fidelity. In governance terms, the longer an AI interaction runs, the more likely earlier constraints are to be diluted by later prompts, making task boundaries and session resets operationally important.

Practical implication: enforce short session lifetimes, reassert policy at key steps, and log when the context boundary is crossed.

NHI Mgmt Group analysis

Arithmetic weakness is not the governance risk. Delegated action on top of arithmetic weakness is. An LLM that miscalculates is a quality problem; an LLM that miscalculates while still being trusted to act is an identity governance problem. The field should stop treating model fluency as a proxy for operational reliability. The practitioner takeaway is that decision rights must be separated from language competence.

Context drift is a control boundary problem, not a content quality problem. When prompt scope can be expanded gradually, the system is no longer responding to a fixed request set. That means the relevant control is not only model accuracy but whether the session preserves intent boundaries from start to finish. The implication for practitioners is that conversation length, role changes, and instruction drift need governance, not just moderation.

Task boundaries matter more than conversational helpfulness in AI governance. Helpful systems are easier to steer, but helpfulness is not the same as safe delegation. A model that stays compliant under pressure can be pushed into actions outside the original intent if no approval boundary exists. The practitioner conclusion is that AI governance must define where a session ends, where authority changes, and when a human must re-enter.

AI decision support should be designed around controllability, not assumed reasoning quality. This article reinforces a broader rule for human, NHI, and autonomous programmes: trust is earned through bounded execution, audit trails, and revocation points. When those are missing, even a benign assistant can become an unbounded decision surface. The practitioner conclusion is to govern the interaction model as tightly as the model itself.

Model usefulness does not reduce the need for explicit authority design. LLMs can assist with language, summarisation, and bounded tool use, but none of that creates inherent accountability or precision. The article's central lesson is that governance failures begin when organisations let conversational convenience stand in for control design. Practitioners should separate assistance from authority in every AI workflow.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

Context drift is now an operating risk, not a curiosity: LLMs can be nudged off-task through gradual scope expansion, flattery, and context exhaustion, so programme owners need session boundaries that are enforced rather than assumed.

The deeper issue is governance design. If a system can be steered by conversational pressure, then approval gates, prompt resets, and action logging belong in the control plane, not in ad hoc operator practice. AI governance should treat the interaction layer as part of the attack surface.

For practitioners

• Define task boundaries before allowing AI assistance Specify what the model may do, what it may not infer, and which actions require human confirmation before execution.
• Route arithmetic and other deterministic work to tools Use calculators, code, or verified external services for precision tasks instead of relying on the LLM's internal reasoning.
• Reset long conversations at policy checkpoints Break extended sessions into smaller units so earlier guardrails are reasserted and scope expansion is easier to detect.
• Log prompt drift and authority changes Track when user intent changes, when the conversation moves outside the original task, and when approval boundaries are crossed.

Key takeaways

• LLM math failures are a symptom of a broader design limit: these systems are fluent pattern matchers, not deterministic reasoning engines.
• Prompt drift techniques can erode guardrails inside a single conversation, especially when the session is long and instructions are not reasserted.
• The control lesson is to bound AI authority, use external tools for precision tasks, and make session resets part of governance.

Key terms

• Context Window: The context window is the text a model receives at one time, including prompts, retrieved documents, and conversation history. Security teams care about it because it becomes the practical boundary between trusted instructions and untrusted content, especially when the application assembles that text automatically.
• Answer Drift: Answer drift is the gradual change in a model’s responses over time, often showing up as reduced consistency or increasing error rates. It can signal degraded grounding, shifting data quality, or prompt and retrieval issues. Monitoring drift helps teams catch reliability problems before they become widespread user-facing failures.
• Tool calling: Tool calling is the pattern where a model selects and invokes an external function during runtime. In agent systems this turns text generation into action execution, so the access decision must be constrained, logged, and governed like any other privileged interaction.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.