Join our Newsletter — 33% off our NHI Course

What would it take for fine-tuning to stop stale CRM answers?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 20225
Topic starter  

TL;DR: Fine-tuning should stay reserved for stable behaviour patterns, while live information such as product catalogues, CRM records and prices belongs in tools, according to Archestra. This keeps systems of record current, auditable and easier to govern.

Editorial analysis by NHI Mgmt Group, based on content published by Archestra: “Fine-tune vs Tool Call: When to Use Each for LLM Agents”.

Key questions

Q: Why do live data lookups reduce AI governance risk?

A: Live lookups keep the authoritative record outside the model, where access can be logged, revoked and scoped.

Q: What are the signs that a model should not be fine-tuned?

A: If the answer depends on prices, customer states, inventory, tickets or any other fast-changing record, fine-tuning is the wrong mechanism.

Q: What breaks when teams fine-tune models without dataset controls?

A: Without dataset controls, fine-tuned models can encode bad, stale, or overexposed information into the model itself, which makes later correction harder than fixing a prompt or a document source.

Practitioner guidance

  • Separate stable behaviour from live information Classify each AI use case as either behaviour shaping or data retrieval before deciding on fine-tuning or tools.
  • Route changing records through governed tools Use tool calls for anything that can change more than once a year, including prices, customer data, tickets and inventories.
  • Bind tools to least-privilege machine identities Issue narrowly scoped service credentials for each tool, and make revocation and logging part of the design from day one.

Bottom line: Fine-tuning is the wrong control for data that changes frequently because it freezes a snapshot of reality inside the model.

Glossary terms: Fine-tuning, Tool Call Event, System of Record, Workload Identity

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 1 week ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20715
 

Fine-tuning is a behaviour-control decision, not a data-governance strategy. Teams repeatedly use the wrong control for the wrong problem because the model can produce an answer that looks correct. But if the underlying information changes, the control plane has to stay outside the model and remain revocable, logged and permissioned. For identity teams, that means the governing question is who or what can fetch the data, not how cleverly the model was trained.

A few things that frame the scale:

  • AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.

A question worth separating out:

Q: Should teams fine-tune for tone and format or use prompting first?

A: Use prompting first when the goal is a style, structure or refusal pattern that may be stable but is not yet proven to need training. Fine-tuning is justified when the behaviour must be consistent at scale and prompting cannot reliably hold the line. The decision should be about stable behaviour, not about teaching the model new facts.

👉 Read our full editorial: Fine-tune vs tool calls: choose live data over frozen weights


This post was modified 1 week ago by NHI Mgmt Group

   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.