On-Device AI Security Guide: Protecting Local Models and Data

When an AI model runs on a phone, laptop or edge box instead of in the cloud, the model file, its prompts and the data it reads all sit on hardware the user, or an attacker, physically controls. On-device AI removes the network round trip and keeps raw data local, which is a genuine privacy gain, but it also hands a copy of the model to every endpoint and puts local inference servers, model files and AI-generated indexes of user activity within reach of malware. This guide covers three situations: AI features built into the operating system (such as Windows Recall or Android's Gemini Nano), models your own apps ship to devices, and local model runtimes such as Ollama and llama.cpp that staff install on workstations. It is written for endpoint, mobile and application security teams, and for ML engineers shipping models to devices. For hosting open models on servers, see the Self-Hosted LLM Security Guide.

Key takeaways

• A model on a device is a model you have published. Encryption and obfuscation raise the cost of extraction but cannot prevent it on hardware the attacker controls, so never rely on model secrecy for security and keep valuable logic and system prompts server-side.
• Integrity matters more than confidentiality. A swapped model or malicious adapter silently changes app behaviour; sign and verify model files before loading, and prefer non-executable formats such as safetensors or GGUF over pickle.
• Local inference servers are network services. Ollama binds to 127.0.0.1:11434 by default; changing OLLAMA_HOST to 0.0.0.0 without an authenticating proxy publishes an unauthenticated model API.
• AI features create new sensitive data stores. Screen snapshots, embeddings and conversation caches need the same encryption, retention and access controls as the data they summarise.
• Prompt injection does not stop at the device. On-device assistants still read untrusted content; keep tool permissions narrow and require user confirmation for actions.

What changes when AI runs on the device

OWASP's Top 10 for LLM Applications 2025 notes explicitly, under LLM03 Supply Chain, that the emergence of on-device LLMs increases the attack surface and supply-chain risk. See the OWASP Top 10 for LLM Applications Guide.

Threats mapped to MITRE ATLAS

Identifiers below were checked against MITRE ATLAS 5.6.0. See the AI Threats and Model Security Guide for the wider model threat landscape.

OWASP LLM03 describes a real case in which attackers reverse-engineered mobile apps and replaced the embedded model with a tampered version, affecting 116 Google Play apps including cash recognition, parental control and face authentication apps. That is the core risk of shipping models: a repackaged app is indistinguishable to the user unless your backend checks integrity.

Protecting models you ship to devices

• Use platform model encryption where it exists: Core ML lets you generate a model encryption key in Xcode (Utilities, Create Encryption Key), which produces a .mlmodelkey file tied to your development team, and encrypt the bundled model at compile time by adding --encrypt with the key path to the model's compiler flags. The same key can encrypt model archives deployed through CloudKit. This protects the model at rest in the package; it does not stop a determined attacker on a jailbroken device from reading decrypted weights in memory.
• Verify integrity before every load: ship a signed manifest with model hashes and verify it at load time. OWASP LLM03's mitigations recommend encrypting models deployed at the AI edge with integrity checks and using vendor attestation APIs to detect tampered apps and models.
• Use app attestation to protect the backend: Android Play Integrity and Apple App Attest let your server check that requests come from your genuine app on a genuine device. Google's guidance for hybrid Gemini Nano and cloud inference recommends protecting the cloud path with Firebase App Check (backed by Play Integrity) and keeping sensitive system prompts server-side. See the Mobile App Security Guide.
• Apply resilience controls proportionately: OWASP MASVS-RESILIENCE lists theft of AI or machine learning models among the business risks that obfuscation, anti-tampering and runtime protection help mitigate, and also states that their absence is not itself a vulnerability. Use them to raise cost, not as the security architecture.
• Keep secrets out of the model and the app: no API keys, system prompts with business rules, or customer data in the bundle. Symantec's study of 1,859 mobile apps with hard-coded AWS keys shows how routinely embedded credentials are extracted; an app that falls back to a cloud LLM with an embedded key becomes a free LLM proxy for anyone. See the LLMjacking Guide.
• Prefer system-provided models where they fit: on Android, Gemini Nano runs inside the AICore system service, which Google documents as isolated from most other packages, without direct internet access (requests and model downloads go through the open-source Private Compute Services component) and without retaining request inputs or outputs. The OS vendor then owns model updates and isolation, though you remain responsible for your app's use of the output.

Protecting data that on-device AI creates

AI features turn scattered activity into a searchable store: snapshots, embeddings, transcripts and summaries. Windows Recall is the most documented example and a good template for what to ask of any such feature.

The governing policies are AllowRecallEnablement and DisableAIDataAnalysis under the WindowsAI CSP (./Device/Vendor/MSFT/Policy/Config/WindowsAI/) or Group Policy at Computer Configuration, Administrative Templates, Windows Components, Windows AI. Disabling AllowRecallEnablement removes the component and deletes saved snapshots after a restart. See the Endpoint DLP Guide and the AI and Privacy Guide.

For your own apps, apply the same pattern: store embeddings and chat history in platform-protected storage (Keychain or Data Protection classes on iOS, Android Keystore-wrapped keys), set retention, and remember that embeddings can leak the text they were made from. See the OWASP LLM Top 10 LLM08 Vector and Embedding Weaknesses Guide.

Local model runtimes on workstations

Developers and analysts increasingly run open-weight models locally. These runtimes are servers, model loaders and download managers at once, and they are often installed outside IT's view. See the Open-Weight Model Security Guide.

• Network exposure: Ollama listens on 127.0.0.1 port 11434 by default and changes bind address through OLLAMA_HOST. Its documentation shows setting 0.0.0.0:11434 to share it on a network; do not do this on corporate devices without an authenticating reverse proxy, because the documented way to share it is a reverse proxy or tunnel in front of the local API, and access control is left to that layer. Allowed browser origins are controlled by OLLAMA_ORIGINS; avoid wildcards such as chrome-extension://*, which lets every installed extension call the model.
• Unexpected cloud use: Ollama can route to cloud-hosted models; set "disable_ollama_cloud": true in ~/.ollama/server.json or OLLAMA_NO_CLOUD=1 where policy requires local-only processing, and check the log line confirming it.
• Untrusted models and inputs: the llama.cpp security policy advises running untrusted models in a sandbox (container or VM), verifying the hash of downloaded weights against a known-good value, and not using its RPC backend or llama-server on untrusted networks. It also warns that GPU side channels can leak data between models sharing hardware.
• Safe formats: pickle-based checkpoints can execute code when loaded. Prefer safetensors, which Hugging Face designed to store tensors safely as opposed to pickle, or GGUF for llama.cpp runtimes. The PyTorch torchtriton attack shows that the ML toolchain itself is a target: a dependency confusion package stole SSH keys and environment variables from machines installing nightly builds. See the Model Supply Chain Guide.
• Inventory and policy: discover runtimes through EDR process and listening-port inventory (for example a listener on TCP 11434), maintain an approved list of runtimes and model sources, and route requests to approved models through managed configuration. See the Shadow AI Discovery Guide.

Where identity fits

On-device assistants increasingly act: they read mail, call APIs and run tools with the signed-in user's tokens. That makes them non-human actors operating under a human identity. Scope OAuth grants to what the feature needs, prefer per-feature tokens over reusing the user's broad session, require confirmation for sending, paying or deleting, and log tool calls locally with a path to central collection. See the Agent Identity and Access Guide and the Prompt Injection Guide.

On-device, private cloud or public cloud

Small organisations should mostly govern built-in OS features (policy, retention, DLP) and block unapproved local runtimes. Larger organisations shipping their own models need a model signing pipeline, attestation-backed backends and a mobile red-team pass. See the Confidential AI and GPU Enclaves Guide for server-side alternatives.

Implementation roadmap

1. Discover: inventory OS AI features enabled per device, apps embedding models, and local runtimes (process names, listening ports, model directories such as ~/.ollama/models).
2. Set default policy: keep built-in features such as Recall disabled for managed devices until a risk review; pilot with a small ring and the retention, deny-list and DLP policies set.
3. Control local runtimes: publish an approved runtime list, enforce loopback binding through configuration management, and block unapproved model downloads at the proxy.
4. Secure the model pipeline for apps you ship: signed manifests, safe formats, encryption where the platform offers it, integrity check on load. Test by tampering with a model in a staging build and confirming the app refuses it.
5. Harden the backend: require app attestation on every cloud fallback call, move system prompts and policy server-side, and remove any embedded keys.
6. Test: include model extraction, model swap and prompt injection scenarios in mobile and endpoint red teaming. See the AI Red Teaming Guide.
7. Roll back safely: keep a kill switch (remote config flag) to disable an on-device AI feature or force cloud-only mode if a model is found to be vulnerable, and a forced model update path.

Detection and measurement

• Endpoint signals: new listeners on model runtime ports, binds to 0.0.0.0, processes loading files with pickle-based extensions from user directories, changes to Recall policies or OLLAMA_HOST, and large reads of AI data stores by unexpected processes.
• Backend signals: requests failing app attestation, model version or hash mismatches reported by clients, and cloud fallback volume spikes that suggest an extracted key or repackaged app.

Common failure patterns

• Treating the model as a secret: putting proprietary rules or credentials into the model or system prompt and assuming encryption keeps them safe.
• On-device guardrails as the only control: relying on a local safety filter or liveness model for fraud or access decisions without server-side checks, inviting offline adversarial search.
• Exposed local inference servers: developers set OLLAMA_HOST=0.0.0.0 for convenience and leave an unauthenticated model API on office or home networks.
• Loading pickle checkpoints from the internet: a downloaded model executes code on the analyst's workstation, which holds cloud and source-control credentials.
• Embedded cloud keys for fallback: the app calls a cloud LLM with a key in the binary, which is extracted and resold, as in many LLMjacking cases.
• Unreviewed AI data stores: snapshot or transcript databases enabled across a fleet with no retention, DLP or access review.

Framework mapping

NIST AI 100-2 E2025 (Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, March 2025) provides the shared vocabulary for evasion, poisoning, privacy and abuse attacks referenced above.

Practitioner checklist

• Inventory OS AI features, embedded models and local runtimes across managed devices.
• Govern built-in features such as Recall through AllowRecallEnablement, DisableAIDataAnalysis, retention and deny-list policies.
• Enforce loopback binding for local runtimes and block unapproved model sources.
• Load only safetensors, GGUF or signed formats; sandbox untrusted models.
• Sign and verify every model you ship, and use platform model encryption where available.
• Keep system prompts, business rules and API keys server-side; never in the app or model.
• Require app attestation for every cloud fallback call.
• Encrypt local embeddings, transcripts and caches with platform keystores and set retention.
• Scope assistant tool permissions and require confirmation for consequential actions.
• Maintain a kill switch and forced update path for on-device models.

Standards and references

• OWASP: Top 10 for LLM Applications 2025
• OWASP: Mobile Application Security Verification Standard
• MITRE ATLAS: Full AI Model Access
• NIST: AI 100-2 E2025 Adversarial Machine Learning
• Microsoft Learn: Manage Recall
• Android Developers: Gemini Nano
• Apple Developer: Encrypting a Model in Your App
• Apple Security Research: Private Cloud Compute

Related NHI Mgmt Group resources: AI Model Security Guide · Self-Hosted LLM Security Guide · Edge Computing Security Guide · Mobile App Security Guide · Model Supply Chain Guide