Flakiness
Repeat, then require passes
LLM answers vary between runs. Run each case N times and require at least M passes, set per suite in bench.yaml or per case, so one unlucky sample does not fail the build.
repeat: 3 · min_pass: 2
RAG evaluation benchmarks for test-driven quality
DocsGPT benchmarking runs YAML test suites against agents with the docsgpt-cli bench command: a suite is a directory of cases with assertions on answer content, JSON fields, sources, tool use, latency, tokens, golden snapshots and LLM-as-judge rubrics, with JSON or JUnit output and baseline diffs for CI/CD.
brew tap arc53/docsgpt-cli && brew install docsgpt-cli
Last updated:
zsh — docsgpt-cli bench
Benchmarking is done with docsgpt-cli
Suites are directories on disk and run from the terminal or a CI runner.
Suites run with docsgpt-cli bench, a command of the open-source docsgpt-cli (MIT, Go). The Homebrew formula installs the binary as docsgpt-cli. bench takes a suite directory; with no argument it runs ./bench. To scaffold one, use bench init.
A suite targets any agent on your instance: target: v1 for the REST chat-completions endpoint, stream for SSE, or webhook for an async webhook agent. The same cases therefore test a chat agent, a streaming widget and a webhook-triggered workflow; token assertions and attachments need the v1 target.
The CLI also does ask, chat and an agentic terminal mode with approvals. See DocsGPT CLI.
bench init · bench · bench record · v1 · stream · webhook
# install docsgpt-cli (Homebrew; release binaries also available)
brew tap arc53/docsgpt-cli && brew install docsgpt-cli
# scaffold a suite, then run it — bench takes a directory, not a file
docsgpt-cli bench init my-suite
docsgpt-cli bench ./my-suite
# with no argument it runs ./bench
docsgpt-cli benchA suite is a directory of cases
Shared defaults in bench.yaml, one sub-directory per case, and each case's files beside it.
The suite directory holds an optional bench.yaml with the defaults every case inherits: the agent, the target, concurrency, the per-case timeout, the judge agent, repeat and min_pass. Each case is its own sub-directory containing a case.yaml, which can override those defaults.
Because a case is a directory, its attachments sit next to it: list them under attachments and they are uploaded before the question is asked, so a case can test what the agent does with a specific document. The recorded golden.json lands there too.
Both files support ${VAR} interpolation, so the suite is safe to commit with the keys left in the environment. The loader rejects unknown fields, so a typo fails the run rather than silently skipping an assertion.
bench.yaml · NN-case-name/case.yaml · attachments · golden.json · .env
# a suite is a directory: shared defaults, then one sub-directory per case
my-suite/
├── bench.yaml # agent, target, concurrency, timeout, repeat/min_pass
├── .env # ${VAR} values — git-ignored, never committed
├── 01-refund-policy/
│ ├── case.yaml # description, tags, question, expect block
│ ├── policy.pdf # attachment, uploaded with the question
│ └── golden.json # written by `docsgpt-cli bench record`
└── 02-contract-notice/
└── case.yamlAssertion types
Each case states its question and the assertions a good answer has to satisfy. The deterministic assertions (content, JSON fields, sources, tools, latency, tokens, golden snapshot) are exact; the LLM-as-judge rubric scores what exact matching cannot, on a 0–1 scale.
| What it checks | What it catches | |
|---|---|---|
| Answer content | Substrings, patterns or an exact answer the reply has to match — and, as negative assertions, the phrases it must never contain | Wrong or missing facts, and the “I don’t know” non-answer |
| JSON fields | Parses the answer as JSON and asserts field by field: equality, one of a permitted set, numeric bounds, list length, presence | Broken schemas, wrong extracted fields |
| Sources | A floor and a ceiling on the number of cited sources | Answers with no grounding, and answers padded with sources |
| Tool usage | Which tools the agent called — and, as a negative assertion, that a tool was not called | Skipped, wrong or forbidden tool calls |
| Latency limit | Wall-clock seconds per case | Slow prompts or models |
| Token limit | Total tokens per case, on the REST target | Cost regressions |
| LLM-as-judge rubric | A judge agent scores the answer against a written rubric on a 0–1 scale; the threshold defaults to 0.7 | Quality that no exact match can express; the judge agent can be a local model |
| Golden snapshot | Compares the answer with the golden.json written by a bench record run | Silent drift in answers that still satisfy every other assertion |
Eight kinds of assertion; a case can combine any of them.
Sample suite
Suite defaults, then one case combining deterministic assertions with a judged one.
bench.yaml names the agent and the target and sets the defaults. Each case.yaml carries a description, optional tags for filtering, its attachments, the question, and an expect block: substrings the answer must and must not contain, fields in a JSON answer, a minimum number of sources, tools that must be called, latency and token ceilings, a judge rubric with a minimum score, and whether to compare against the recorded golden answer.
Negative assertions carry as much weight as positive ones. For a policy assistant, the phrase the answer must never use and the tool it must never call are the cases that keep it inside its remit; the guide has the full field reference.
bench.yaml · case.yaml · expect · judge · golden
# bench.yaml — suite defaults, overridable per case
agent: ${DOCSGPT_BENCH_KEY} # key name from `docsgpt-cli keys`, or a literal key
target: v1 # v1 | stream | webhook
judge:
agent: judge-agent # agent that grades the judge rubrics
concurrency: 2
timeout: 120s
repeat: 3 # run each case N times…
min_pass: 2 # …and require at least this many passes# 01-refund-policy/case.yaml — unknown fields are rejected at load
description: "Support agent quotes the refund window"
tags: [smoke, support]
attachments: [policy.pdf] # uploaded first, passed with the question
question: "How many days do customers have to request a refund?"
expect:
answer:
contains: ["30 days"] # case-insensitive substrings
not_contains: ["I don't know"]
json: # parse the answer as JSON, assert per field
status: ok # bare scalar = equality (gjson paths)
sources: { min: 1 } # retrieval sources returned
tools:
called: [lookup_policy]
judge:
rubric: "States the 30-day window and cites the policy document."
min_score: 0.7 # judge scores 0–1; 0.7 is the default
limits:
max_seconds: 60
max_total_tokens: 8000 # v1 target only
golden: false # compare against the recorded golden.jsonCI/CD and regression safety
Machine-readable output, exit codes a pipeline can gate on, and two ways to catch a regression.
Output is --json or --junit report.xml, so any pipeline can gate on it. The exit code is 0 when every case passed, 1 on failures and 2 on a configuration error. A suite path that is not a directory, or a min_score outside 0–1, fails validation before any case runs.
bench record and --update exit 0 even when assertions fail, because their job is to write snapshots rather than to judge. Gate on a plain run.
Golden files and baselines are different mechanisms. bench record writes a golden.json beside each case: a snapshot of the answer, compared on later runs by the golden-snapshot assertion. Separately, every run is kept under ~/.docsgpt/bench/<suite>/, and --baseline last diffs this run against the previous one: regressions, fixes, and latency and token drift.
- 01
Scaffold the suite
bench init writes a bench.yaml and a first case directory to edit.
docsgpt-cli bench init my-suite
- 02
Write cases
One directory per question: the question, its attachments, and the expect block that defines a good answer.
01-refund-policy/case.yaml
- 03
Record golden answers
Snapshot today's answers so later runs can diff the answer text itself, not only pass and fail.
docsgpt-cli bench record → golden.json
- 04
Run on every change
Every prompt edit, source change or model swap runs the suite in CI against the staging instance.
docsgpt-cli bench ./bench --junit report.xml
- 05
Diff against the previous run
See which cases changed status and where latency and tokens drifted, not just which failed.
--baseline last
- 06
Gate the deploy
JUnit or JSON output and exit codes block the merge when a case regresses.
exit 0 pass · 1 failures · 2 config error
Flaky answers, subsets and A/B runs
An eval suite is only a gate if a passing run means something and a failing run is cheap to read.
Selection
Run a subset
-k matches case names and descriptions; --tags selects the tags a case declares. A smoke subset on every push, the full suite nightly.
-k refund · --tags smoke
Feedback
Fail fast
Stop at the first failing case instead of paying for the rest of the run — useful on a pull-request check where the first regression is enough.
--fail-fast
Throughput
Concurrency and timeouts
Cases run in parallel, two at a time by default, with a per-case timeout of 120 seconds. Both are suite defaults and command-line flags.
concurrency: 2 · timeout: 120s
Comparison
A/B two agents
Run one suite against two agents in the same invocation and compare them case by case — a prompt rewrite, a retrieval change or a model swap.
--vs candidate-agent
Secrets
Commit the suite, not the key
${VAR} in bench.yaml and case.yaml resolves from the shell environment, then the suite's .env, then ./.env. An unset variable fails the load, so a CI job never benchmarks against an empty key.
agent: ${DOCSGPT_BENCH_KEY}
A --vs run compares two agents on one suite, pass rate per assertion type. Your own cases decide the real numbers.
Pairing an A/B run with repeat and min_pass is what makes the comparison mean something. A five-point gap on single samples is noise; a five-point gap on three-of-five passes is a signal.
Content assertions passedA 92% · B 84%
Source assertions passedA 88% · B 79%
Judge rubric ≥ 0.70A 76% · B 71%
A = pinned agent, B = candidate. Example values.
How it runs privately
docsgpt-cli is a Go binary that runs on an engineer's machine or a CI runner and talks only to the instance URL you configure. On Managed, On-premises and Air-gapped deployments benchmark traffic stays inside your network, and the judge agent can be backed by a local model so no answer leaves it to be scored. Run history and golden files stay on the machine that ran the suite. On Cloud only the URL differs.
Benchmarking is part of the open-source CLI in every deployment posture, so there is nothing extra to buy. See pricing. If you are building in-house, the eval harness is usually the part nobody budgeted for.
Example
A team runs a policy assistant for 400 staff. Its suite directory holds 40 case directories drawn from real tickets, each asserting at least one cited source and, for the 12 numeric answers, the exact figure; the four cases about the staff handbook keep the PDF beside the case.yaml as an attachment. Every prompt edit and every model swap runs docsgpt-cli bench in CI with --tags smoke on each push and the full suite nightly; a drop in source assertions blocks the merge, and a judge score under 0.70 on the long-form cases opens a review.
Frequently asked questions
How to evaluate RAG performance?
- Write a case for each question your users ask and assert on what a good answer must contain: the right facts or JSON fields present, at least one cited source, tools called when they should be, and latency and tokens within limits. Add an LLM-as-judge rubric with a minimum score for the qualities an exact match cannot express, then run the suite on every change.
How do I test an AI agent before deployment?
- Point a suite directory at the staging instance and run
docsgpt-cli bench ./bench, then gate the release on the JUnit report. Record golden answers withbench recordso later runs diff the answer text, and setrepeatandmin_passso a single flaky sample does not fail the build. Can I run RAG evals in CI?
- Yes. docsgpt-cli is a single Go binary;
--jsonand--junit report.xmlgive machine-readable output, and the exit code is 0 when every case passed, 1 on failures and 2 on a configuration error. Gate on a plain run:bench recordand--updateexit 0 even when assertions fail, because their job is to write snapshots. How do I compare two models on my own data?
- Run the suite against both agents in one invocation with
--vs, or pin the current model, run the suite, then run the candidate and diff it with--baseline last. The numbers come from your own cases on your own documents. How do I keep API keys out of a committed benchmark suite?
- Write
agent: ${DOCSGPT_BENCH_KEY}inbench.yamlinstead of the key itself. Values resolve from the shell environment first, then the suite's.env, then./.env, so the suite is safe to commit and the secret stays in the CI runner.
See DocsGPT on your documents
A 30-minute demo of test-driven agent quality, or self-host and run your first suite today.