How should organisations govern embedded AI in third-party software under the EU AI Act?

How should embedded AI be governed when it lives inside someone else’s software?

Embedded AI should be governed like an internal capability that happens to be delivered through a vendor product, not as a black-box feature that sits outside your control boundary. That means the buyer still needs to know where the AI is used, who approves it, what data it can reach, and how evidence is retained. Under the eu ai act, the deployer cannot outsource accountability simply because the software is third-party supplied.

What governance controls matter most for embedded AI?

The practical starting point is inventory and ownership. If a third-party product contains AI-assisted decisioning, content generation, triage, or ranking, the organisation should record the use case, business owner, data inputs, outputs, and the operational purpose of the feature. That creates the minimum governance perimeter needed to decide whether the system is prohibited, limited, transparency-bound, or higher risk under the Act.

Ownership should be paired with a formal approval path, because embedded AI often enters through procurement, platform upgrades, or feature toggles rather than a standalone AI programme. The right control question is not only “is this vendor compliant?” but also “can our organisation explain why this use is acceptable, what evidence supports that view, and who can stop it if the risk changes?”

Evidence quality matters as much as policy wording. If the vendor cannot show traceability for model behaviour, testing, logging, data provenance, or human oversight, the deployer should treat that gap as a governance defect rather than a documentation nuisance. For vendor-delivered AI, the EU AI Act regulatory framework is the anchor reference, because it defines the obligations that sit on providers and deployers across the lifecycle.

Third-party AI governance is also an access problem. If the embedded model can see customer records, operational queues, or internal knowledge bases, then access scope, audit logging, and change control become part of the compliance posture. In practice, organisations should govern embedded AI with the same discipline used for privileged software integrations: minimal data exposure, explicit approval, and a reviewable record of what the feature can do.

How do third-party dependencies change the compliance burden?

Third-party software changes the burden, but not the responsibility. Vendor assurances can reduce due diligence work, yet they rarely eliminate the need for internal classification, oversight, and monitoring. If the AI feature is embedded inside a business system you operate, you still need to decide whether the use case is acceptable in your environment, whether the vendor’s safeguards are sufficient, and whether any residual obligations remain on your side.

That is especially important when the vendor’s evidence is incomplete. Missing model cards, opaque training data claims, weak logging, or vague human-oversight statements do not just create procurement friction, they make it harder to defend your own deployment decision. The safer governance pattern is to map each embedded AI feature to a named owner, a documented purpose, and a review cadence, then require evidence that is specific enough to support that exact use case rather than generic AI marketing language.

Vendor reliance also affects incident handling. If the embedded AI output can influence customer communications, operational decisions, or regulated processes, your organisation must be able to suspend the feature, contain the blast radius, and preserve records even when the supplier is slow to respond. Agentic AI Compliance Guide is useful here because it connects AI governance, audit evidence, and accountability in a way that aligns with deployer obligations. IAM and IGA Basics also helps frame the ownership and approval model, especially where access, entitlements, and reviewability shape the control boundary.

What should practitioners verify before relying on a vendor’s AI feature?

Risk and Threat Considerations

Third-party embedded AI can fail in ways that are easy to miss: hidden model changes, undocumented data exposure, weak logging, or outputs that influence decisions without adequate oversight. Those failures matter because they can shift a regulated use case into an ungoverned one even though the feature still looks like ordinary product functionality.

Failure mechanism: The vendor supplies capability without enough traceability for versioning, training lineage, access scope, or output review, so the deployer cannot prove what was used, when it changed, or who approved the change.

Impact: The organisation may lose control of compliance evidence, misclassify a high-risk use, or be unable to demonstrate accountability after an incident, audit, or regulatory inquiry.

Practitioner Guidance

What to verify: Confirm that every embedded AI feature has a named business owner, a documented use case, a data-access boundary, and a retained evidence trail for approvals and changes. If any of those four are missing, treat the feature as not yet governable rather than simply “vendor managed”.

Decision rule: If the vendor cannot provide traceable evidence for logging, model changes, or oversight, require compensating controls on your side or disable the feature for regulated workflows. If the feature can affect legal, financial, safety, or customer-impacting decisions, escalate the review before rollout.

Practitioner takeaway: Embedded AI should be governed by the organisation that deploys it, because accountability follows operational use, not procurement label.

Related resources from NHI Mgmt Group

• When should organisations prioritise third-party conformity assessment over internal assessment under the EU AI Act?
• Should organisations govern AI tools like third-party software dependencies?
• How should organisations govern agentic AI under EU and UK regulations?
• How can organisations govern third-party AI systems without losing accountability?