Four enterprise LLM deployment patterns and their governance trade-offs

At a glance

What this is: This is Fiddler’s overview of four enterprise LLM deployment approaches, with the key finding that deployment choice is driven by maturity, data, cost, and control trade-offs.

Why it matters: It matters to IAM and security practitioners because every LLM deployment pattern changes how data, prompts, and model access need to be governed, especially where AI systems touch sensitive or regulated information.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing credentials.

👉 Read Fiddler’s full guide to four enterprise LLM deployment patterns

Context

LLM deployment is not a single architecture choice. It is a governance decision about where prompts, retrieval paths, training data, and model behaviour are allowed to interact with enterprise information, and that becomes an identity and access problem as soon as AI systems touch sensitive data or operational workflows. The article’s core value is in showing that the security boundary changes as teams move from simple prompt use to retrieval, tuning, and training.

For security and identity programmes, the relevant question is not which deployment method is technically possible, but which control model can sustain it. Prompt-based integrations, retrieval-augmented generation, fine-tuning, and training from scratch each create different exposure patterns for secrets, entitlements, and data provenance. That makes the article useful as a governance lens rather than just a model-selection guide.

As LLMs move from experimentation into production, the starting position described here is common: teams optimise first for speed and cost, then discover the control consequences later. That pattern is typical across emerging AI programmes, and it is where governance gaps usually appear first.

Key questions

Q: How should security teams govern LLM applications that call tools and data sources?

A: Security teams should govern LLM applications as identity-rich systems, not as isolated models. Access must be enforced before retrieval or tool use, tools must run under scoped credentials, and high-impact actions should require human approval. The model can assist with language and orchestration, but it should not be trusted to make authorization decisions on its own.

Q: Why does retrieval-augmented generation create new governance risk?

A: Retrieval-augmented generation creates new governance risk because the model can only be as trustworthy as the knowledge sources it retrieves from. If documents, embeddings, or vector stores are poisoned or too broadly exposed, the model may generate confident but wrong answers from unverified context. That makes source governance and provenance tracking essential.

Q: What breaks when teams fine-tune models without dataset controls?

A: Without dataset controls, fine-tuned models can encode bad, stale, or overexposed information into the model itself, which makes later correction harder than fixing a prompt or a document source. The control gap is usually provenance: teams can explain that a model was tuned, but not always who approved the data or whether the training set matched policy boundaries.

Q: How do organisations know if an enterprise LLM is operating outside its intended scope?

A: The clearest signals are unexpected data exposure, retrieval from sources not linked to the user’s role, and outputs that reveal information the requesting identity should not have reached. If audit logs cannot show which data influenced the response, the system is already outside a defensible governance boundary.

Technical breakdown

Prompt engineering and context: why simple API access changes the trust boundary

Prompt engineering means sending a model a request and, often, a set of examples or instructions that shape its output. The model may be external, shared, and opaque, so the real control point becomes the data included in the prompt and the permissions behind the application calling the model. That means the trust boundary is not the model alone, but the combination of application identity, prompt content, and downstream data handling. Where prompts contain sensitive material, the risk is not only leakage but also reuse in logs, tracing, or vendor-side processing.

Practical implication: bind prompt access to tightly scoped application identities and classify prompt content as data with explicit handling controls.

Retrieval-augmented generation: how external data becomes part of the model answer

RAG works by retrieving chunks of enterprise content, converting the query and documents into embeddings, ranking likely matches, and feeding the selected text back to the model as context. This improves relevance, but it also creates a new control surface: if retrieval is wrong, stale, over-broad, or poorly permissioned, the model can answer from the wrong evidence set. In security terms, RAG introduces a data access path that behaves like a dynamic authorization layer, even when teams think they are just adding search. It therefore needs governance over source quality, chunking, indexing, and access filtering.

Practical implication: treat retrieval as an access control layer and enforce source permissions, freshness checks, and evidence logging before production use.

Fine-tuning and training from scratch: when model ownership shifts control requirements

Fine-tuning updates a base model using enterprise examples so the model internalises patterns that once lived in prompts or documents. Training from scratch goes further by building the model on domain-specific corpora and infrastructure. Both approaches reduce some runtime dependency on external prompts, but they raise the stakes for dataset curation, lineage, and model governance. The more the model absorbs enterprise knowledge, the more important it becomes to control who can prepare data, approve training sets, and alter model versions. In practice, these are not just ML tasks; they are governance and change-control tasks.

Practical implication: apply dataset approval, provenance tracking, and model-change controls before treating tuned models as production assets.

NHI Mgmt Group analysis

LLM deployment strategy is now an identity governance decision, not just a model architecture choice. The article shows that each deployment pattern moves the security boundary in a different place, from prompt inputs to retrieved documents to model weights. That means enterprise AI governance has to track who can call the model, what data can be injected, and which identities are allowed to shape outputs. The practitioner conclusion is that AI controls must be designed alongside identity and data controls, not after deployment.

Retrieval-based AI creates a new permissioning problem: the model can only be as safe as the data path behind it. RAG is often described as a way to improve answer quality, but the deeper issue is that it turns retrieval into an implicit authorisation decision. If the index exposes content beyond intended scope, the model faithfully amplifies that exposure. This is where NHI governance matters, because the application calling the retriever often acts like a non-human identity with broad data reach. The practitioner conclusion is that retrieval permissions need lifecycle and scope control just like any other privileged access path.

Model governance debt accumulates when teams optimise for speed first and control later. The article’s deployment ladder encourages fast adoption, but each step increases technical and governance complexity. Prompting is easy to start, yet it can leave data handling undefined. Fine-tuning and training improve capability, but they also increase the blast radius of bad datasets, bad approvals, or bad versioning. The practitioner conclusion is that AI programmes need explicit ownership before scale creates hard-to-reverse control debt.

Data provenance becomes the hidden control plane for enterprise LLMs. Once a model consumes prompts, retrieval content, or tuning data, the security question is no longer only what the model says but where its inputs came from and who was allowed to supply them. This is especially important in regulated environments where evidence quality matters. The practitioner conclusion is that teams should treat provenance, traceability, and approval workflows as core AI governance capabilities, not optional metadata.

AI agent identity will converge with LLM deployment governance as organisations operationalise these patterns. Even when an LLM is not yet acting as a full agent, the calling application already behaves like a machine identity with access to data, tools, and workflows. That makes the boundary between LLMOps and NHI governance increasingly thin. The practitioner conclusion is that organisations should prepare to govern AI systems through the same identity, privilege, and audit disciplines they apply to other non-human actors.

What this signals

Prompt-to-production LLM programmes now need the same access discipline that identity teams already apply to other non-human actors. The practical signal is that model-facing applications should be catalogued, owned, and constrained like service accounts, not left as anonymous integration glue. Where RAG is in use, teams should pair access logging with source permissions and evidence retention so they can explain what the model saw and why. For governance baselines, the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 provide useful external anchors.

Data provenance is becoming the control plane for enterprise AI. Once retrieval, fine-tuning, or prompt context starts influencing decisions, the question shifts from model accuracy to traceable authority over inputs. Organisations should prepare for more scrutiny over which identities can supply training data, approve datasets, and publish retrieval sources. That is especially true where personal, regulated, or operationally sensitive information can enter the generation path.

Model governance debt: teams that scale LLM usage before defining ownership, approval, and logging will spend far more effort retrofitting controls later. The most durable programmes will collapse AI operations into the same lifecycle model used for other privileged enterprise assets, with clear ownership, review points, and audit evidence.

For practitioners

• Map each LLM deployment pattern to a distinct control set Document which teams use prompt-only, RAG, fine-tuned, or trained models, then assign controls for data exposure, change approval, and access review to each pattern.
• Treat retrieval as governed access Limit which documents can be indexed, separate sensitive sources by policy, and record which retrieved passages influenced each response for audit and incident response.
• Establish model and dataset change control Require approval for training data, fine-tuning sets, and model version promotion so that governance follows the same discipline used for other production changes.
• Bind AI workflows to explicit non-human identities Assign scoped service accounts or workload identities to model-facing applications, then monitor their access to retrieval stores, logs, and downstream tools.

Key takeaways

• Enterprise LLM deployment is fundamentally a governance choice because each pattern changes where data exposure and control failures can occur.
• RAG, fine-tuning, and training from scratch each increase operational power, but they also expand the scope of what must be permissioned, logged, and reviewed.
• Identity, provenance, and change control are the controls that determine whether production AI remains explainable and defensible.

Key terms

• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Fine-Tuned Model: A model updated with domain-specific examples so it learns enterprise patterns rather than relying only on generic pretraining. Fine-tuning improves relevance and can reduce prompt complexity, but it also makes dataset quality, provenance, and change control part of the security model.
• Model Governance Debt: The accumulation of control gaps that appear when AI experimentation moves faster than oversight. It usually shows up as unversioned prompts, shared credentials, unclear approval authority, and evaluation results that cannot be reproduced or audited later.
• Dataset provenance: Dataset provenance is the record of where training, validation, or testing data came from, how it was changed, and which model version used it. It gives auditors a way to trace results back to inputs and to understand whether a system’s outputs can be reproduced or explained.

What's in the full article

Fiddler's full blog covers the operational detail this post intentionally leaves for the source:

• Implementation distinctions between prompt engineering, RAG, fine-tuning, and training from scratch
• Examples of where each deployment pattern fits different data volumes, latency needs, and cost constraints
• Technical workflow details for embeddings, retrieval, and fine-tuning that help platform teams operationalise the approach
• Observability and pre-production testing concepts that matter once an LLM moves from prototype to production

👉 The full Fiddler post covers the deployment workflow details, cost trade-offs, and LLMOps considerations behind each approach.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and access control patterns that matter in AI-enabled environments. It gives security practitioners a common control language for managing non-human systems and the identities that support them.