Most AI coding assistant evaluations compare suggestion quality and price, then discover the security questions after the tool is on every developer laptop. Modern assistants are no longer autocomplete: they run terminal commands, edit many files, call MCP servers and act with the developer's own credentials. That makes the choice a security decision about data handling, legal exposure, admin control and audit, as much as a productivity one. This vendor-neutral guide gives security, platform and procurement teams the requirements, evaluation criteria, vendor questions, red flags and a proof of concept plan for selecting an enterprise coding assistant. It does not cover application security scanners for AI-generated code, which are in the AI Code Security and Agentic Coding Security Buyer's Guide; configuration of tools you already run is in the AI IDE and Coding Agent Security Guide.
Key takeaways
- Evaluate the agent, not the autocomplete. The highest risk features are autonomous command execution, file system reach and tool connections, so weight admin control over those above suggestion quality.
- Get data handling in the contract. Retention periods, training use, sub-processors and data residency must be written commitments for your plan tier, not marketing pages.
- Central policy beats developer settings. Prefer tools where administrators can enforce permission modes, approved models and approved MCP servers, and developers cannot override them locally.
- Audit logs are a buying criterion. If you cannot see which user ran which agent action, with which tool, you cannot investigate an incident.
- Test with hostile repositories. A proof of concept should include prompt injection through repository files, because attackers already plant instructions there.
Define requirements
Write these down before talking to vendors, and get each signed off by its owner.
- Use cases: inline completion only, chat in the IDE, autonomous agent mode in the IDE, terminal agents, cloud-hosted agents that open pull requests, code review bots. Each step up adds execution and credential risk.
- Code sensitivity: which repositories may be sent to a hosted model at all; whether any code is export-controlled, regulated or customer-owned; whether secrets may appear in context.
- Deployment model: vendor-hosted SaaS, your cloud tenant, or self-hosted models; data residency requirements.
- Identity integration: SSO with your identity provider, SCIM provisioning and deprovisioning, group-based policy assignment.
- Tooling reach: which MCP servers, CLIs and cloud credentials developers expect the agent to use, and which must be off limits.
- Legal position: the IP indemnity you need, licence-matching controls for generated code, and acceptable use terms. Involve legal early.
- Evidence: what audit, SIEM export and retention you need for your own compliance programme. NIST SP 800-218A, the SSDF community profile for generative AI, names buyers who acquire AI systems among its audiences and is a useful frame for supplier questions.
Capability areas
| Area | What good looks like | Why it matters |
|---|---|---|
| Data handling | Contracted retention limits (including a zero or minimal retention option), no training on your code by default, named sub-processors, residency options | Source code, secrets in context and prompts leave your environment on every request |
| Admin policy | Org-level enforcement of agent permission modes, command allow and deny lists, approved models, feature toggles, file and repository exclusions | Developer-level settings drift and can be changed by a malicious repository file |
| Execution control | Approval before commands run, sandboxed or containerised execution, network egress restrictions for the agent | Agent mode runs code with the developer's privileges (OWASP LLM06 Excessive Agency) |
| MCP and tool governance | Central allow list of MCP servers, consent showing the exact command before a local server starts, scoped tokens per server | The MCP security best practices require clients to show the exact command and get explicit approval for one-click local server configuration, and forbid token passthrough |
| Instruction and context files | Visibility and control over repository rule files, warnings on hidden characters, admin control over whether they are honoured | Repository files are a proven prompt injection route (OWASP LLM01) |
| Identity and lifecycle | SSO, SCIM, group policy, fast seat removal, revocable personal and machine tokens | Leavers and contractors must lose access immediately |
| Audit and monitoring | Per-user logs of agent sessions, commands run, files changed, tool calls and policy blocks, exportable to your SIEM | Investigation and evidence |
| Legal and output controls | IP indemnity terms, filters or references for output matching public code, licence reporting | Copyright and licence exposure from generated code |
| Vendor security | Independent assurance reports, vulnerability disclosure programme, model and extension supply chain controls (OWASP LLM03 Supply Chain) | The vendor sits inside your development pipeline |
Evaluation criteria
| Criterion | Suggested weight | Evidence to request | Pass threshold |
|---|---|---|---|
| Contracted data retention and training terms | High | Data processing agreement, product terms for your tier | Written retention limit and no training on customer code without opt-in |
| Central enforcement of agent permissions | High | Admin console demo, policy documentation | Admins can block auto-approval of commands and developers cannot override |
| MCP server allow listing | High | Demo of blocking an unapproved server | Unapproved servers cannot be added by developers |
| Audit log depth and export | High | Sample log records, SIEM integration docs | User, session, command, tool call and timestamp per action, exportable |
| Resistance to repository prompt injection | High | Your own red-team test in the proof of concept | No unapproved command executes from planted instructions |
| SSO and SCIM | Medium | Integration docs, deprovisioning test | Access removed within your leaver SLA |
| Sandboxing and egress control | Medium | Architecture documentation | Agent execution can be confined to a workspace and limited network |
| IP indemnity | Medium | Contract clause and its conditions | Covers your plan tier with conditions you can meet |
| Residency and deployment options | Depends on requirements | Region list, private deployment docs | Meets your stated residency needs |
| Vendor assurance | Medium | Independent audit reports, penetration test summary | Current report covering the product in scope |
Small organisations can usually accept a vendor-hosted service with strong contract terms and central policy, and focus the proof of concept on agent controls. Large or regulated organisations should expect to negotiate data terms, require SIEM export and may need private or regional deployment, which narrows the field quickly.
Questions to ask vendors
- For our plan tier, how long are prompts, code context and outputs retained, where, and by which sub-processors? Is any of it used for training or evaluation?
- Which agent settings can administrators enforce centrally, and which can a developer or a repository configuration file override?
- Can we block automatic approval of terminal commands and file writes outside the workspace for all users?
- How do you control which MCP servers and extensions developers can connect, and what does the user see before a local server command runs?
- How does the product handle instruction files in a repository, and can administrators disable or restrict them? Do you detect hidden Unicode characters?
- What exactly is in the audit log for an agent session, how long is it kept and how do we export it?
- What credentials does the agent use when it calls Git hosting, package registries or cloud APIs, and can they be scoped or short-lived?
- What are the conditions of your IP indemnity, and which product settings must be enabled to qualify?
- How do you secure your model supply chain, extensions marketplace and update channel?
- How quickly is access revoked through SCIM, and are existing sessions and tokens terminated?
Red flags
- Data retention and training terms exist only on a web page that "may change", not in the contract.
- Agent permission settings are per-developer only, with no central enforcement.
- No way to restrict MCP servers or extensions, or a one-click install that starts local commands without showing them.
- Audit logs cover licence usage and seat counts but not agent actions.
- The agent reuses the developer's full Git or cloud credentials with no option to scope them.
- Indemnity that applies only under conditions you cannot verify or that exclude your tier.
- The vendor cannot explain how repository-supplied instructions are treated. The TrapDoor supply chain campaign in May 2026 planted assistant instruction files hidden with zero-width characters to make assistants run a fake security scan that exfiltrated secrets.
Proof of concept
- Set success criteria first from the evaluation table, with named owners for security, platform, legal and developer experience.
- Use a pilot group and non-production repositories that reflect your real languages and build systems, with no production credentials on pilot machines.
- Apply your intended central policy before developers start, then confirm developers cannot override it.
- Run hostile repository tests: planted instruction files, hidden characters, a malicious MCP server configuration and a dependency that prompts the agent to read secrets. Record whether the agent asks, blocks or executes.
- Verify audit: reconstruct one full agent session from exported logs in your SIEM.
- Test lifecycle: remove a pilot user through SCIM and confirm access and tokens end within your SLA.
- Review the contract against what you observed, and record exceptions before any wider rollout.
How NHI Mgmt Group can help
We provide independent market analysis, RFP strategy and evaluation support for agentic AI and non-human identity programmes, including the credentials and tool access that coding agents depend on. Browse vendors in our NHI and AI products directory or contact us about independent evaluation support.
Related NHI Mgmt Group resources: AI IDE and Coding Agent Security Guide · AI Code Security and Agentic Coding Security Buyer's Guide · Coding Agent Configuration File Attacks · MCP Security Guide