garak Guide: Run Probes, Read Reports and Gate CI

Before an LLM-backed application goes live, someone has to check whether it can be talked into leaking data, following injected instructions, writing malware or inventing software packages, and doing that by hand for every model change is not realistic. garak (Generative AI Red-teaming and Assessment Kit) is an open source command-line scanner that probes a model or dialogue system for those failure modes and reports which attempts succeeded. Its maintainers describe it as doing for LLMs something similar to what nmap or Metasploit do for networks. This guide explains how garak's generators, probes, detectors, buffs and harnesses work, how to scope and run scans safely, how to read the JSONL and HTML reports, and how to use garak as a recurring check in a pipeline. It is written for AI security engineers, application security teams and platform owners testing systems they own or are authorised to assess. Facts were checked against the garak repository and documentation on 8 November 2026.

This is an independent guide from NHI Mgmt Group, based on garak's public documentation as of November 2026. It is not sponsored or reviewed by NVIDIA or the garak maintainers. Product features change, so confirm settings against garak's current documentation.

Key takeaways

• Probes ask, detectors judge. A probe sends crafted prompts; each probe names a primary detector and extended detectors that decide whether a response is a hit (the failure happened) or a pass.
• Scope every run. By default garak runs all active probes, and its documentation notes a default run can involve 80,000 or more inference requests; use --spec, tags such as tag:owasp:llm01, tiers or --config fast.
• Read scores as trends, not grades. The FAQ explains that probe scores are not on a normalised scale, so compare a probe with itself over time, and the HTML report adds a Z-score against a calibrated bag of models.
• The report is your evidence. Each run writes garak.<uuid>.report.jsonl with model parameters, every prompt, every response and the scores, plus an HTML summary and a hit log.
• Rescan regularly. Probes are added and improved often, and the FAQ advises against relying on scores more than six months old.

What the tool does

garak is licensed under Apache 2.0, maintained under the github.com/NVIDIA/garak organisation, and installed from PyPI with python -m pip install -U garak. It requires Python 3.11 or later and is developed on Linux and macOS. On 8 November 2026 the main branch was at version 0.17.2.pre1. The FAQ states that NVIDIA officially contributes to the project and that garak will continue to be licensed under Apache 2.0.

Selecting what to run

run.spec (CLI --spec) is the single place to select probes and buffs. Selectors include probes.<module>, probes.<module>.<Class>, tag:<prefix> (for example tag:owasp:llm06), tier:<N> (inclusive, where tier 1 means "of concern"), and intent:<code> for intent-based probes. A leading minus removes a selection. --list_probes, --list_detectors and --plugin_info show what is installed and the tags each probe carries.

Key run settings

Defender use cases

• Model selection and acceptance: scan candidate models with the same spec before adoption and record the report as part of supplier assessment, alongside the OWASP LLM Top 10 LLM03 Supply Chain Guide.
• Prompt injection exposure: run promptinject, latentinjection and encoding probes against the application endpoint to see how often injected or encoded instructions are followed. The FAQ notes that garak already includes probes for indirect prompt injection relevant to RAG systems.
• Secret and key handling: the apikey probe checks whether a system will generate API keys for named services, and sysprompt_extraction checks whether a system prompt can be recovered. Both matter if prompts or context contain credentials, which they should not. See the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide.
• Code assistant safety: packagehallucination tests whether code generations name packages that do not exist, a supply chain risk if developers install them blindly; malwaregen tests refusal of malware requests.
• Output handling: xss and web_injection look for outputs that could enable cross-site scripting or data exfiltration when rendered, which feeds the OWASP LLM Top 10 LLM05 Improper Output Handling Guide.
• Agents with tools: agent_breaker is a multi-turn probe that uses a red team model to analyse an agent's tools and test whether they can be misused, mapped to OWASP LLM06 Excessive Agency.

Deployment and safe use

Phased rollout

1. Authorise and scope. Agree targets, probe families, volumes and test windows with the system owner, and check hosted model providers' terms. Use the Red Team Rules of Engagement Template.
2. Install in its own environment. Use a dedicated virtual environment or container, pin a released version for assessments, and record it in every report.
3. Use dedicated credentials. Generators read keys from environment variables such as OPENAI_API_KEY, BEDROCK_API_KEY, NIM_API_KEY, HF_INFERENCE_TOKEN (read role) and REST_API_KEY. Issue test-only keys with spending limits, inject them from your secrets manager at runtime and revoke them after the engagement. See the API Key Management Guide.
4. Configure REST targets carefully. For rest.RestGenerator, put the key in a $KEY placeholder filled from the environment rather than in the YAML file, keep verify_ssl on (it defaults to true and accepts a CA bundle path), and set request_timeout and ratelimit_codes (default 429) to match the target.
5. Dry run the plumbing. Use the test.Blank or test.Repeat generators to check configuration and reporting without sending anything to a real model.
6. Start narrow. Begin with --config fast or a single tagged family and --generations 1, review hits by hand, then widen to tier 1 probes and higher generation counts.
7. Write a config file. Move settings into a YAML or JSON file passed with --config so runs are repeatable and reviewable in source control (without secrets).
8. Close down. Revoke keys, archive reports to controlled storage and delete local copies under the XDG data directory.

Data handling

Reports hold every prompt and response, which can include harmful text and any sensitive data the target leaked. Set reporting.report_dir to an absolute path on encrypted, access-controlled storage. garak.log stays in the XDG data directory; the FAQ notes that the XDG_DATA_HOME environment variable can relocate it. Report security issues in garak through NVIDIA's PSIRT process described in the repository security policy, and apply garak updates promptly.

Detection and hardening value

Reading the results

The console shows, per probe and detector, how many generations behaved acceptably and marks any undesirable behaviour as FAIL with a failure rate. The JSONL report has an entry_type on each row: attempt entries carry a status (0 not sent, 1 response received, 2 response evaluated), and eval entries summarise each probe and detector pair. The hit log lists only the attempts scored as hits, which is the fastest place to start triage. Two newer metrics help with risk communication:

• hit@k: the chance that at least one of k attempts on a prompt succeeds. The reporting docs make the point that a jailbreak that works once in five replies is still a working jailbreak, which pooled success rates understate.
• Z-score: how the target compares with a regularly calibrated bag of models, shown in the HTML report or on the CLI with system.show_z=True.

To group findings by OWASP category, run python -m garak.analyze.report_digest <report.jsonl> owasp. The tags documentation lists support for OWASP LLM Top 10, AVID, Language Model Risk Cards, CWE and EU AI Act tags, and cautions that tag coverage does not mean every part of a framework is tested.

Mapping findings to controls

garak runs support NIST SP 800-53 RA-5 (Vulnerability Monitoring and Scanning) and SA-11 (Developer Testing and Evaluation), and NIST CSF 2.0 ID.RA-01.

Using garak in CI

Run a short, fixed config (for example tier 1 probes for the injection and leakage families with low generations) on model, prompt or guardrail changes, and a fuller scan on a schedule. Keep the same spec between runs so pass rates are comparable, store the JSONL and HTML reports as build artefacts, and fail the stage on a team-defined threshold that your pipeline script reads from the eval entries. Requests carry the user agent garak/{version} (LLM vulnerability scanner https://garak.ai) by default, so monitoring teams can recognise authorised scans in API gateway and WAF logs and tell them apart from real abuse.

Metrics

• Pass rate per probe family per release, with confidence intervals.
• hit@k for tier 1 probes, reported alongside pooled rates.
• Open tier 1 findings and median days to fix and rescan.
• Scan freshness: share of production AI applications with a scan less than six months old.
• Detector review rate: share of hits confirmed by a human reviewer.

Common failure patterns

• Running everything by default. A full run is slow and costly; scoped specs give faster, more useful results.
• Treating a pass rate as a certificate. A score only covers the probes run, and probes keep changing.
• Scanning the raw model only. Use the REST generator to test the deployed application with its guardrails and retrieval in place.
• Keys in config files. Hard-coded keys in committed YAML end up in repositories and CI logs; use the $KEY placeholder and environment injection.
• Unprotected report folders. Reports accumulate sensitive prompts and responses in a home directory nobody reviews.

Alternatives

• Adaptive red teaming frameworks: scripted multi-turn campaigns with attacker and judge models, suited to deep testing of one system.
• Evaluation harnesses: declarative test cases and assertions in CI for quality and security regressions.
• Manual expert testing: needed for business logic, novel agents and high-impact actions. See the AI Red Teaming Guide.
• Managed continuous testing: see the Continuous Automated Red Teaming (CART) Guide.

Practitioner checklist

• Written authorisation covers every target and probe family.
• A released garak version is pinned and recorded with each report.
• Test-only keys are injected at runtime and revoked afterwards; no keys in config files.
• REST targets keep verify_ssl on and have timeouts and rate limit codes set.
• Every run uses a committed config with an explicit run.spec.
• Hits are reviewed by a human before tickets are raised.
• Reports are stored on encrypted, access-controlled storage with a retention rule.
• Scans rerun after every model, prompt or guardrail change and at least every six months.
• Monitoring teams know the garak user agent and scan windows.

Standards and references

• NVIDIA: garak repository
• garak: User guide
• garak: Reference documentation
• NVIDIA: garak FAQ
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: LLM Prompt Injection
• NIST: SP 800-53 Rev. 5
• NIST: Cybersecurity Framework 2.0

Related NHI Mgmt Group resources: AI Red Teaming and Testing Tools Buyer's Guide · AI Red Teaming Guide · Prompt Injection Guide · OWASP LLM Top 10 LLM01 Prompt Injection Guide