PyRIT Guide: Set Up Scans, Attacks and Scorers Safely

Teams shipping generative AI features need a repeatable way to find out whether a model or application can be pushed into leaking data, ignoring its instructions or producing harmful output, and one-off manual testing does not scale. PyRIT (Python Risk Identification Tool for generative AI) is an open source toolkit from the Microsoft AI Red Team for finding security and safety weaknesses in generative AI systems through automated scans, adaptive multi-turn attacks and human-led testing. This guide covers what the toolkit contains, how its scenarios, executors, converters, targets and scorers fit together, how to deploy it without creating new credential and data risks, and how to turn its results into hardening work. It is for AI security engineers, application security teams and red teams running authorised tests against systems they own or are permitted to assess. Methodology (scoping, threat modelling, reporting) sits in the AI Red Teaming Guide; this page stays on the tool. Facts were checked against the PyRIT repository and documentation on 8 November 2026.

This is an independent guide from NHI Mgmt Group, based on PyRIT's public documentation as of November 2026. It is not sponsored or reviewed by Microsoft. Product features change, so confirm settings against PyRIT's current documentation.

Key takeaways

• Three ways in, one engine. PyRIT offers a command-line scanner (pyrit_scan and pyrit_shell), a browser interface called CoPyRIT for human-led testing, and a Python framework, all built on the same components.
• Orchestrators are now executors and attacks. Older tutorials talk about orchestrators such as CrescendoOrchestrator; current documentation describes executors, with attacks such as PromptSendingAttack, CrescendoAttack and TreeOfAttacksWithPruningAttack as the main family.
• Scorers decide what counts as a finding. Every scorer returns either a true_false verdict or a float_scale value between 0.0 and 1.0, and the choice and calibration of scorers matter as much as the attacks.
• The test rig holds sensitive material. PyRIT stores every prompt, response and score in its memory database (SQLite by default) and needs API keys for targets and helper models, so treat it as a sensitive system.
• Use Key Vault, not plaintext files, for shared runs. The configuration docs describe Azure Key Vault as the canonical environment source for shared, CI/CD and deployed setups, and PyRIT warns when it auto-loads a plaintext ~/.pyrit/.env.

What the tool does

PyRIT is MIT licensed and published by Microsoft. The repository is at github.com/microsoft/PyRIT; check that remotes and pinned URLs point there. The package requires Python 3.11 or later (below 3.15, per pyproject.toml). On 8 November 2026 the main branch carried version 1.2.0.dev0 and the documentation site defaulted to the 1.1.0 release; the docs ask you to use the version picker so that examples match your installed version.

The framework documentation describes the architecture as a chain: entry points (scanner, GUI, notebooks) run scenarios, which package datasets with attack techniques; each technique drives an executor that coordinates converters, targets and scorers, with a shared memory and registry layer underneath.

Executors in plain terms

The executor documentation splits attacks by a simple rule: a single-turn attack sends exactly one crafted message to the objective target and scores the reply; a multi-turn attack sends more than one and adapts, often using a separate adversarial model to write the next prompt. Compound executors chain other attacks, workflow executors cover multi-step cases such as cross-domain prompt injection (XPIA), where instructions are planted in content an AI system later ingests, and benchmark executors evaluate a target against a fixed dataset. Attack configuration (objective target versus adversarial target, prepended conversations, memory labels) is shared across all of them.

Scorers in plain terms

A scorer answers questions such as "was the prompt blocked?" or "how harmful was the response?". true_false scorers suit success criteria and refusal detection; float_scale scorers quantify severity and can be turned into a pass or fail with a threshold scorer. Scorers can run inside an attack, directly on a response, or in batch over stored results. LLM-as-judge scorers (the SelfAsk family) need their own chat target, which is another credential and another place your test data is sent.

Defender use cases

• Pre-release assessment: run AIRT scenarios such as Jailbreak, Leakage and Cyber against a staging deployment before a model, system prompt or guardrail change goes live, and keep the results as release evidence.
• Prompt injection testing for RAG and agents: use the XPIA workflow pattern to test whether an assistant that summarises web pages, emails or documents follows instructions hidden in that content. Pair this with the Prompt Injection Guide for the control side.
• Regression after model swaps: rerun the same scenario with the same datasets and memory labels after a provider or model version change to see whether refusal or leakage behaviour moved.
• Guardrail validation: point PyRIT at the application endpoint (HTTP, WebSocket or Playwright target) rather than the raw model, so content filters, input classifiers and output handling are tested together. The AI Guardrails and AI Firewalls Guide covers what those layers should do.
• Human-led exploration: use CoPyRIT for interactive sessions where a red teamer probes a system, tracks findings and shares them with colleagues, then converts promising lines into automated scenarios.
• Agent permission checks: where an AI system can call tools, design objectives around actions (create, delete, send, pay) and score tool calls, using the tool call scorer documented in the scoring section. The Red Teaming AI Agents for Identity Abuse guide lists identity-focused objectives worth adding.

Deployment and safe use

PyRIT sends adversarial content to real endpoints and stores what comes back. Safe use is mostly about authorisation, credentials and data.

Phased rollout

1. Get written authorisation. Agree scope, targets, objectives, test windows and stop conditions with the system owner and, for third-party model APIs, check the provider's usage terms. The Red Team Rules of Engagement Template works for AI tests too.
2. Install in isolation. The docs offer a user Docker image with JupyterLab and a local pip or uv install; contributor installs track the main branch. Pin a released version for assessments so results can be reproduced.
3. Use dedicated, low-privilege credentials. Create separate API keys or identities for the target, the adversarial model and any judge model, scoped to test deployments with spending limits. Never reuse production keys. The API Key Management Guide covers rotation and scoping.
4. Load secrets from a vault. Configure env_akv_ref in ~/.pyrit/.pyrit_conf so values come from Azure Key Vault; keep ~/.pyrit/.env.local for deliberate local overrides only. When a Key Vault reference is set, PyRIT ignores an auto-discovered ~/.pyrit/.env and emits a security warning.
5. Start small. Run one scenario with a single technique and a capped dataset (--datasets name:N, --max-total) against staging, check scorer verdicts by hand, then widen.
6. Respect rate limits. Set max_requests_per_minute on targets and use --max-concurrency and --max-retries on pyrit_scan so tests do not trip provider throttling or look like an outage.
7. Label and keep results. Use --memory-labels (for example experiment, version, operator) so every result can be traced to a run, then export with scenario-results in JSON or HTML format.
8. Roll back cleanly. Revoke the test credentials at the end of an engagement and archive or delete the memory database according to your retention rules.

Configuration settings that matter for security

Data handling

Datasets include harmful and offensive seed prompts by design, and responses may contain generated harmful content or real data leaked by the target. Store the memory database on encrypted storage, limit who can read it, and agree up front whether findings that contain personal data go into tickets verbatim or are summarised. Report vulnerabilities in PyRIT itself through the Microsoft Security Response Center as described in the repository security policy, and apply PyRIT updates promptly by reviewing the release notes.

Detection and hardening value

A PyRIT run is only useful if it changes the system. Map each finding to a control owner and a retest.

In control terms, PyRIT supports NIST SP 800-53 CA-8 (Penetration Testing), SA-11 (Developer Testing and Evaluation) and RA-5 (Vulnerability Monitoring and Scanning), and NIST CSF 2.0 ID.RA-01 (vulnerabilities identified, validated and recorded) and ID.IM-02 (improvements identified from security tests).

Metrics

• Attack success rate per scenario: share of objectives achieved, tracked per release and per model version.
• Scorer agreement: share of a weekly sample of automated verdicts that a human reviewer confirms; low agreement means the scorer, not the system, needs work.
• Time to retest: days from a finding being fixed to a passing rerun of the same scenario and dataset.
• Coverage: number of production AI applications with at least one labelled PyRIT run in the last quarter.
• Credential hygiene: test keys revoked within one day of engagement close.

Common failure patterns

• Testing the bare model, not the application. Results look worse or better than reality because guardrails, retrieval and tool permissions were not in the path.
• Trusting judge scores blindly. LLM-as-judge scorers misclassify; without human sampling, false positives swamp teams and false negatives give false comfort.
• Production keys in a notebook. A shared Jupyter environment with a plaintext .env becomes a credential store nobody manages.
• Unpinned versions. Following latest-branch docs with a different installed release breaks examples and makes results impossible to reproduce.
• No retention plan for memory. The database quietly accumulates harmful content and leaked data from every run.

Alternatives

PyRIT is one of several ways to test generative AI. Choose by the question you need answered, not by tool popularity.

• Manual expert red teaming: best for novel systems, business logic abuse and high-impact agents; slower and harder to repeat. See the OWASP GenAI Red Teaming Manual Guide.
• Probe-based vulnerability scanners: broad libraries of fixed and adaptive probes run against a model endpoint; quick coverage of known failure classes.
• Evaluation and test harnesses in CI: declarative test cases and assertions run on every prompt or model change; strong for regression gates.
• Continuous automated red teaming services: scheduled testing operated by a provider. See the Continuous Automated Red Teaming (CART) Guide.

Many teams combine a framework such as PyRIT for deep, scripted campaigns with lighter regression checks in the pipeline.

Practitioner checklist

• Written authorisation and scope exist for every target, including provider terms for hosted models.
• A released PyRIT version is pinned and the matching documentation version is used.
• Separate, scoped, spend-limited credentials exist for target, adversarial and judge models.
• Shared and CI runs load secrets through env_akv_ref; no plaintext .env in images or repositories.
• Tests run against the full application path, not only the raw model.
• Scorer verdicts are sampled and confirmed by a human before findings are reported.
• Every run carries memory labels and results are exported for evidence.
• The memory database is encrypted, access controlled and covered by a retention rule.
• Findings map to OWASP LLM Top 10 and ATLAS IDs with an owner and a retest date.

Standards and references

• Microsoft: PyRIT repository
• Microsoft: PyRIT documentation
• Microsoft: PyRIT security policy
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: LLM Prompt Injection
• MITRE ATLAS: LLM Jailbreak
• NIST: SP 800-53 Rev. 5
• NIST: Cybersecurity Framework 2.0

Related NHI Mgmt Group resources: AI Red Teaming Guide · AI Red Teaming and Testing Tools Buyer's Guide · Prompt Injection Guide · Red Teaming AI Agents for Identity Abuse