Compare the same session and cost views before and after the change. A good fix should reduce the specific waste pattern it targeted, such as retries, idle time, or repeated reading, while leaving productive work intact. If the savings do not appear in later data, the proposed control change did not solve the underlying issue.
How to tell whether the optimisation actually changed behaviour
Cost optimisation only worked if the operational pattern changed, not just the bill line item. The cleanest test is to compare the same session and spend views before and after the change, then look for a drop in the waste you meant to remove, such as retries, idle time, or repeated reading, without a matching drop in useful work.
That means you are looking for a causal footprint in the telemetry, not a one-off dip. If the agent still performs the same task mix but consumes fewer tokens, fewer calls, or less wall-clock time per successful outcome, the optimisation likely did something real.
A useful readout should separate efficiency from suppression. If spend falls because the agent stops retrying a failing step, that is good only when the success rate and task completion quality stay stable. If spend falls because the agent no longer reaches the useful parts of the workflow, the change may have reduced activity rather than waste.
What evidence shows the fix worked for the right reason?
The best evidence is paired: cost metrics and task-quality metrics moving in the right direction together. For example, fewer repeated reads or shorter idle intervals should coincide with the same or better completion rate, lower error rate, or unchanged user outcome. That is the signal that you removed friction rather than degraded capability.
Look for the metric that the proposed fix was designed to move. A prompt, routing, caching, or tool-use change should alter the specific bottleneck it targeted. If the change was supposed to reduce redundant context reads, verify that the agent is actually reading less often on the same class of sessions, not merely that overall usage declined across unrelated work.
When the environment is variable, compare like with like. Session length, task type, tool availability, and request complexity all affect spend. A fix that looks effective in a simple batch may not hold in longer, more complex sessions, so the before-and-after comparison should be segmented rather than averaged into a single global number.
When the numbers say the change did not work
If later data shows the same waste pattern persisting, the control change did not solve the underlying issue. That usually means the optimisation targeted a symptom, not the cause, or it was introduced too narrowly to affect the dominant cost driver.
Watch for three common failure modes: the agent still retries the same way, the agent still spends time waiting or re-reading without progress, or the spend reduction disappears once the workload shifts. In each case, the apparent win was either temporary or misattributed.
If the fix reduced cost but also reduced useful output, treat that as a design failure rather than a successful optimisation. The goal is not cheaper activity at any price; it is lower waste with preserved task effectiveness.
Risk and Threat Considerations
Optimisation can create false confidence when teams judge success by spend alone. If the control change suppresses retries, tool calls, or context access without preserving success rate and quality, the agent may look cheaper while becoming less reliable, less observable, or easier to stall.
Failure mechanism: A change removes visible waste in the short term but does not alter the underlying behaviour, so the same inefficiency returns under different inputs or after normal workload variation.
Impact: Teams may deploy a weak control, understate operating cost, or miss a degradation in task completion that only shows up in later sessions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-01 — Monitoring for Anomalies and Events | Session cost changes need ongoing monitoring to confirm the behaviour actually changed. |
| ID.RA-01 — Asset Vulnerabilities Identified and Documented | The optimisation should target a documented waste pattern or bottleneck. | |
| GV.OV-01 — Oversight of Cyber Risk Management Strategy | You must validate that the control improved outcomes, not just spend. | |
| Recommendation — Monitor session telemetry for the waste pattern you intended to reduce. Document the specific inefficiency before changing controls. Review optimisation results against the intended operational outcome. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Comparing before-and-after session evidence requires review of telemetry and usage records. |
| Recommendation — Analyze session logs to confirm the cost reduction came from the intended behaviour. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Cost validation depends on retained logs that show retries, waits, and repeated reads. |
| Recommendation — Keep logs that prove whether the optimisation changed session behaviour. | ||
Practitioner Guidance
What to verify: Check the same cohort before and after the change, and validate both the cost metric and the outcome metric. If one improves while the other worsens, treat the optimisation as incomplete.
Decision rule: If the targeted waste pattern disappears only in the short-term view but returns in later data, roll back the assumption that the fix addressed root cause and remeasure at the session level.
What good looks like: The agent completes the same work with fewer avoidable retries, less idle time, or fewer redundant reads, while task success remains stable.
Practitioner takeaway: A real optimisation changes the behaviour that created the cost, not just the cost curve for a single reporting window.