AI agent skills: what changes when procedure beats prompting?

TL;DR: Effective AI agent skills are compact, procedural packages that load only when relevant and prove value through matched evaluations, with research showing curated skills can lift pass rates by 16.6 points while oversized libraries and self-generated skills often regress, according to Arize. The governance lesson for identity and security teams is that agent behaviour improves only when the harness, routing, and acceptance criteria are treated as one controlled system.

NHIMG editorial — based on content published by Arize: How to write effective AI agent skills: 6 data-backed practices

By the numbers:

• Curated skills added 16.6 percentage points on average in SkillsBench, but gains varied by model-harness pair and some tasks regressed.
• Compact and standard-length skills improved pass rate by 19.0 and 21.5 points, while comprehensive documentation lifted it by just 0.7 points.
• Loading all 196 skills reached a 29.3% pass rate, while task-conditioned composition reached 45.3% and used about 23% fewer input tokens.

Questions worth separating out

Q: How should teams govern AI agent skills in production?

A: Treat skills as controlled runtime assets, not informal text.

Q: Why do AI agent skills need evaluation gates?

A: Because a skill that looks good can still reduce performance by displacing a better default, adding token bloat, or encoding the wrong procedure.

Q: What breaks when too many AI agent skills are loaded?

A: Context gets crowded, the agent spends more tokens on selection than execution, and the model may pick the wrong procedure or ignore the best one.

Practitioner guidance

• Inventory every skill as a governed runtime artefact Track ownership, purpose, required inputs, allowed actions, and review cadence for each skill so teams can see which workflows depend on it.
• Reduce each SKILL.md to the minimum viable procedure Keep only the load condition, ordered steps, critical branches, and definition of done in the main file, then move examples and scripts into separate files.
• Build routing tests for skill selection accuracy Measure whether the harness discovers, ranks, and loads the right skills for the task before you measure whether the task succeeded.

What's in the full article

Arize's full article covers the operational detail this post intentionally leaves for the source:

• The exact six-practice framework for writing AI agent skills that the article distils from the research.
• Study-by-study comparisons showing where curated skills, generated skills, and large skill bundles succeed or regress.
• Practical guidance on evaluation design, including how to compare with-skill and without-skill runs.
• Examples of how to structure SKILL.md, supporting files, and harness logic for better routing and reuse.

👉 Read Arize's analysis of effective AI agent skills and evaluation →

AI agent skills: what changes when procedure beats prompting?

Explore further

View Full Forum →  |  NHI Foundation Course →