MLOps vs LLMOps Security: What Changes for Foundation Model Apps

Security teams that copy their MLOps controls straight onto LLM applications protect the wrong assets: the risk moves from the training pipeline to prompts, retrieval data, API keys and model behaviour at runtime. A classic MLOps pipeline trains, registers and serves a model the organisation owns, so its threat model centres on training data, model artefacts and the registry. An LLMOps pipeline usually consumes a foundation model through an API or an open-weight download, then adds prompts, retrieval-augmented generation (RAG), tools and evaluation suites around it. This guide compares the two pipelines stage by stage: the threat model of each, how they are built, how they fail, which controls belong where, and how to decide which control set a given system needs. It is written for platform, ML and security engineers who own either pipeline.

Key takeaways

• Decide first whether you train or consume. Training or fine-tuning puts data poisoning, artefact integrity and registry tampering at the top of the list; consuming a hosted model puts prompt injection, key sprawl and output handling there.
• LLMOps adds new release artefacts. System prompts, prompt templates, RAG indexes, tool definitions and evaluation datasets all change system behaviour, so each needs versioning, review and rollback just like a model version.
• The identity layer changes shape. MLOps secrets are mostly hub tokens, registry credentials and cloud roles on training compute; LLMOps multiplies provider API keys across apps, notebooks, gateways and CI, which is where LLMjacking starts.
• Evaluation becomes a security gate. Accuracy metrics on a held-out set are enough for many predictive models; LLM applications need regression, injection and leakage test suites that run on every prompt, model or index change.
• Most real systems are hybrids. A team that fine-tunes an open-weight model and serves it behind a RAG application needs both control sets, so map controls per pipeline stage rather than per team label.

Short answer: what actually changes

MLOps security protects a build pipeline for a model: data in, artefact out, artefact deployed. LLMOps security protects a composition pipeline for an application: a model you mostly did not build, wrapped in instructions, retrieved context and tools that you change far more often than the model itself. The UK NCSC's Guidelines for secure AI system development anticipate this: they note that changes to data, models or prompts can lead to changes in system behaviour, and they ask developers to treat major changes like new versions and test them accordingly. In MLOps the "prompt" item on that list rarely exists. In LLMOps it is the most frequently changed artefact in production.

The deeper control set for each side already exists on nhimg.org. See the MLOps Security Guide for the training and serving stack, the Prompt Lifecycle Management Guide for prompt change control, and the LLM Observability Guide for runtime telemetry. This page is about the differences and the decision between them.

Side-by-side comparison

Decision criteria: which control set does your system need?

Classify each AI system on three questions before choosing controls. The answers decide which stages exist and therefore which threats apply.

1. Do you change model weights? If you pre-train, fine-tune, use LoRA or other parameter-efficient adapters, or distil, you run a training pipeline and need the MLOps control set for data intake, artefact integrity and registry access. OWASP LLM03:2025 Supply Chain notes that LoRA and PEFT on public hubs introduce new supply-chain risks. If you only call a hosted model, those controls shrink to supplier assurance.
2. Do you ship prompts, retrieval or tools? If yes, you have LLMOps release artefacts. Each needs an owner, version history, review and an evaluation gate, and RAG adds a data pipeline (ingestion, chunking, embedding, indexing) with its own poisoning and access-control risks.
3. What evaluation do you need to trust a release? A fraud classifier can be gated on precision and recall against labelled data. A customer assistant needs behavioural tests: injection resistance, refusal behaviour, leakage of system instructions or personal data, and correct tool use. The more open-ended the output, the more the gate looks like a security test suite. See Model-Level vs System-Level AI Evaluation.

Threat model: MLOps pipeline

The adversary's goal in an MLOps pipeline is usually to change what the model learns or to take the model and its data. Access is gained through the data supply, the build and experiment tooling, or the credentials that write to the registry.

• Training data poisoning: malicious or mislabelled samples enter training data and implant a backdoor or bias (ATLAS AML.T0020 Training Data Poisoning). ATLAS lists dataset provenance (AML.M0025), sanitising training data (AML.M0007), verifying datasets against protected manifests (AML.M0014) and maintaining an AI bill of materials (AML.M0023) among its mitigations. See the AI Data Poisoning and Backdoors Guide.
• Unsafe model artefacts: pickle-based checkpoints can execute code on load (ATLAS AML.T0011.000 Unsafe AI Artifacts), and a compromised public repository can deliver one (AML.T0010.003 Model Repository).
• Registry tampering: anyone with write access to the model registry can register a new version or move a deployment alias. MLflow's documentation describes an alias as a mutable, named reference to a model version, so changing which version @champion points to changes production without changing the URI the serving code loads. See Model and dataset registry security.
• Experiment tooling: notebooks, tracking servers and compute frameworks that assume a trusted network.
• Weight and data theft: exfiltration from object storage or the registry.

Threat model: LLMOps pipeline

In LLMOps the adversary rarely needs pipeline access. Untrusted text reaches the model at runtime through user input, retrieved documents, tool results and memory, and the application then acts on the output. The pipeline threats that remain are about who can change prompts, indexes and keys.

• Prompt injection: direct and indirect instructions that redirect the model (OWASP LLM01:2025; ATLAS AML.T0051). OWASP notes that RAG and fine-tuning do not fully mitigate it, so it cannot be trained away and must be contained by design.
• RAG and embedding risks: poisoned documents in the corpus (AML.T0070 RAG Poisoning) and weak access control across the vector store (OWASP LLM08:2025 Vector and Embedding Weaknesses). See the RAG Security Guide and Memory Poisoning vs RAG Poisoning.
• Prompt tampering and leakage: whoever can edit the system prompt can change behaviour for every user, and OWASP LLM07:2025 states the system prompt should not be treated as a secret or a security control.
• API key sprawl: provider keys copied into notebooks, environment files, CI variables and plugins. ATLAS AML.T0055 Unsecured Credentials covers the technique; the LLMjacking 2024 to 2026 page documents stolen cloud credentials and AI keys being used to run hosted models at the victim's expense.
• Tooling supply chain: gateways, orchestration and evaluation libraries sit next to every key. In the LiteLLM PyPI Package Breach 2026, a compromised scanner in the project's CI exposed its publishing token, and the malicious releases stole credentials, including LLM API keys, from machines that installed them.
• Unbounded consumption: excessive inference for denial of service, cost or behaviour cloning (OWASP LLM10:2025).

Architecture in words: where the stages differ

An MLOps pipeline runs left to right: data sources feed a feature or dataset store; training jobs on GPU compute read them; experiment tracking records runs; a validated model is registered, signed and promoted; a serving endpoint loads a pinned version; monitoring watches drift. Trust boundaries sit at data intake, at artefact intake from public hubs, and at the registry.

An LLMOps pipeline is a loop around a model endpoint. A prompt registry or repository holds templates; an ingestion job chunks and embeds documents into a vector index; an orchestration layer assembles system prompt, user input and retrieved context; a gateway holds provider credentials and enforces quotas; tools and MCP servers execute actions; an evaluation harness replays test sets on every change; tracing captures each request. Trust boundaries sit at every point where untrusted content enters the context window and at every point where model output triggers an action. The OWASP GenAI Security Project describes this lifecycle as nine stages: Scope and Plan, Augment and Fine-tune Data, Develop and Experiment, Test and Evaluate, Release, Deploy, Operate, Monitor and Govern.

Controls per pipeline stage

The NCSC guidelines support both columns: they ask developers to compute and share cryptographic hashes or signatures of model files, to be able to restore to a known good state after compromise, and to monitor and log inputs such as inference requests, queries or prompts in line with privacy requirements. See the Model Signing and OpenSSF Model Signing Guide for signing workflows.

Where identity fits

The non-human identities differ more than the models do, and they are the shortest route to compromise in both pipelines.

• MLOps identities: training job roles that can read all training data, registry service accounts that can write versions, and model hub tokens. Hugging Face documents read, write and fine-grained token types, recommends one token per app or usage so one can be revoked without affecting others, recommends fine-grained tokens in production, and offers Trusted Publishers so CI systems exchange an OIDC token for a short-lived Hub token instead of storing a secret.
• LLMOps identities: provider API keys per application, gateway master and virtual keys, embedding service keys, vector database credentials and OAuth tokens for tools. Each app should have its own key with a spend limit and an owner, issued from a secrets manager rather than a .env file. See the LLMjacking Guide.
• Shared rule: the identity that promotes to production (registry alias, prompt alias, index swap) should be a separate release identity, not the developer's or the training job's.

Failure modes compared

Worked scenario: one team, both pipelines

An insurer fine-tunes an open-weight model on claims notes to classify and summarise claims, then serves it behind an assistant that retrieves policy documents and can open tickets. Treated as "an MLOps project", the team signs the fine-tuned model and locks down the registry, but the system prompt sits in an application config file, the assistant's provider key for a fallback hosted model is shared across three apps, and the policy index is rebuilt nightly from a shared drive anyone can write to.

Mapped per stage, the controls become clear. Training data intake gets provenance records and personal data minimisation, and the claims notes are screened before fine-tuning. The base checkpoint is pulled by pinned digest in safetensors format, and the fine-tuned output is signed and verified at load. The release bundle records the model digest, prompt version, index snapshot ID and tool manifest together. The ingestion job reads only from an approved library with document ACLs carried into the index. The evaluation gate replays claims-summary regression tests plus an injection suite seeded with hostile text inside policy documents. Each app gets its own provider key from the secrets manager, with a spend cap. Traces record the prompt version and retrieved document IDs per request.

Implementation roadmap

1. Inventory and classify: list every AI system, mark train, consume or both, and record the release artefacts each one has. Use the AI Supply Chain and AI-BOM Guide for the record format.
2. Fix identities first: find and rotate provider keys outside a secrets manager, split shared keys per app, set spend limits, and restrict registry and prompt alias write access to release identities.
3. Bring prompts and indexes under change control: move prompts to version control or a prompt registry, snapshot indexes, and define the release bundle.
4. Add evaluation gates in audit mode: run regression and security suites on every change, report failures without blocking for two to four weeks, tune thresholds, then enforce.
5. Integrity and supply chain: enforce safe formats and signature verification for model artefacts; pin orchestration and gateway libraries; review MCP servers and tools before use.
6. Telemetry: trace retrieval and tool calls, log prompt and model versions per request, and alert on alias changes and token anomalies.
7. Rollback drills: practise reverting a model version, a prompt alias and an index snapshot separately, and rotating a provider key, and time each.

Detection and measurement

• Signals to log: registry and prompt alias changes with actor, index rebuilds and source changes, provider key creation and use per app, token usage per caller, guardrail and evaluation failures, artefact signature verification results.
• Metric 1: share of production AI releases deployed as a recorded bundle that passed the evaluation gate (target 100%).
• Metric 2: number of provider API keys not issued from a secrets manager or without an owner and spend limit (target zero).
• Metric 3: share of model artefacts verified by signature or pinned digest at load.
• Metric 4: mean time to roll back a prompt, index or model version in a drill.
• Metric 5: injection and leakage suite pass rate per release, tracked over time.

Framework mapping

NIST SP 800-218A, the SSDF community profile for generative AI and dual-use foundation models (July 2024), extends the SSDF with AI-specific practices for model producers, producers of AI systems that use those models, and acquirers, which maps to the train and consume split used here.

Common failure patterns

• One control set for all AI: applying registry and signing controls to an API-based assistant while leaving its prompts and keys unmanaged.
• Prompts treated as configuration, not code: edited live, never evaluated, and impossible to roll back to a known state.
• Keys shared across apps and environments: one leak exposes every workload and makes attribution of spend impossible, the pattern behind many LLMjacking cases.
• Trusting the tooling supply chain: unpinned gateway or orchestration libraries running next to every provider key.
• RAG indexes built from writeable sources: anyone who can edit the source can inject instructions into every answer.

Practitioner checklist

• Classify every AI system as train, consume or both, and record its release artefacts.
• Define a release bundle: model ID or digest, prompt version, index snapshot, tool manifest, guardrail config.
• Restrict registry versions, model aliases, prompt aliases and index swaps to a release identity with approval.
• Issue one provider key per app and environment from a secrets manager, with an owner and a spend limit.
• Use fine-grained, short-lived or OIDC-exchanged tokens for model hubs and CI.
• Enforce safe model formats and signature or digest verification at load.
• Gate every prompt, model and index change on regression plus injection and leakage suites.
• Carry source permissions into retrieval and screen ingested content for embedded instructions.
• Trace prompt version, retrieved documents and tool calls per request; alert on alias changes and token anomalies.
• Drill rollback of each artefact type and key rotation at least twice a year.

Standards and references

• NCSC: Guidelines for secure AI system development
• NIST: SP 800-218A Secure Software Development Practices for Generative AI and Dual-Use Foundation Models
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: Training Data Poisoning
• NIST: AI 600-1 Generative Artificial Intelligence Profile
• MLflow: Model Registry
• Hugging Face: User access tokens
• OpenSSF and Sigstore: Model transparency (model signing)

Related NHI Mgmt Group resources: MLOps Security Guide · Prompt Lifecycle Management Guide · LLM Observability Guide · Model and dataset registry security · LLMjacking Guide