Judgment Labs builds Judgment, an observability and evaluation platform for long-running AI agents. It is for engineering teams that have put agents into production and need to know whether they are behaving correctly, find the failures they did not anticipate, and turn real usage into tests that make the next version better.
The company’s view is that agents should improve from what actually happens in production, not only from tests written before release.
At A Glance
| Vendor | Judgment Labs |
|---|---|
| Category | AI agent observability and evaluation |
| Deployment | Judgment cloud, or inside the customer’s own AWS or GCP account |
| Agentic AI | Agent Judge, Issue Discovery agents, MCP server for coding agents |
| Customers | Fluidstack, Modal, Listen Labs, Monaco, Atomicwork |
| Market classification | AI Governance & Compliance; also Agentic AI Identity |
What Judgment Labs Does
An agent that runs for hundreds of steps can fail in ways that never show up as an error. Judgment works from the agent’s traces, whether they come from production or from tests in the team’s own environment, and applies the same judges to both. That lets a team monitor how agents act, spot failure patterns it did not know existed, and turn real usage into evaluations, alerts and improvement loops. Each project represents one agent, with its development, staging and production traffic held as environments inside it.
Products
Tracing
Tracing records agents of any depth, including subagents, tool calls and hundreds of spans per run. Traces are stored and indexed so that even very large ones stay searchable.
Monitoring And Judges
Monitoring can use filters, code checks, decision models or agentic judges. A judge reads a trace or a session and applies one of two or more labels that the team defines. There are three types:
| Agent judge | Reads the trace and decides against a rubric the team writes. |
|---|---|
| Condition judge | Checks duration, LLM cost, errors, or the names and attributes of spans. |
| Code judge | Runs Python the team uploads, created through the API or MCP. |
Issue Discovery
Issue Discovery runs background agents that search traces for recurring problems, so production failures surface without anyone writing a rule for them first.
Search And Evaluation Tools
Hybrid search combines structured and semantic filters across traces. The platform also includes Offline Evals, Scheduled Tasks, Automations and Judge Alignment.
How It Works
Judgment runs as a hosted cloud service or inside the customer’s own AWS or GCP account.
| Component | What it does |
|---|---|
| Trace ingestion | Agents send traces through built-in OpenTelemetry support in common frameworks, through OTel instrumentation libraries, or as OTLP sent directly. A coding agent can set this up itself through the Judgment MCP server. |
| Labelling | Once a judge is switched on for live traffic, every new trace gets a label, and the same judges grade each trace when a test runs over a dataset. Labels are stored as judge results next to the traces. |
| Actions | When a judge returns a given label, an action can send an email, post to Slack, add the trace to a dataset or run another judge. |
Agentic AI Capabilities
Agent Judge
Agent Judge is an agentic evaluation harness for long-horizon production agents. It runs as a multi-agent system that combines three parts: search, which makes long trajectories navigable; verification, which checks tool evidence and environment state to handle stateful actions; and rubrics that adapt as the agent changes. On internal production traffic it achieved a reported 0.86 accuracy and 0.79 F1 for trajectory-level hallucination detection.
MCP Server For Coding Agents
Through the MCP server, a coding agent can read Issues, traces, judge labels and test runs. Clients need remote HTTP MCP support. Browser-based OAuth is the preferred sign-in, with a personal API key sent as a bearer token as a fallback. Ingest API keys are not accepted, and a key carries the permissions of the user’s role.
Integrations
| Area | Integrations |
|---|---|
| Agent frameworks | LangGraph and LangChain, OpenAI Agents SDK, Claude Agent SDK, Vercel AI SDK |
| Coding agents (MCP) | Cursor, Claude Code, Codex |
| Collaboration | Slack |
Company Background
| Leadership | Alex Shan, co-founder and CEO. |
|---|---|
| Funding | $32 million across seed and Series A, in 2026. |
| Investors | Lightspeed Venture Partners led the seed, and Greenoaks and Lightspeed co-led the Series A, joined by Nova, Valor, Dynamic, Chris Manning and the founders of DoorDash and Mercor. |
Note: This overview is compiled from Judgment Labs’s publicly available information as of 4 November 2026. It is provided for information only. NHI Mgmt Group does not endorse any vendor, and nothing on this page represents the opinions of NHI Mgmt Group.