What this page covers

Nimble publishes results from four benchmarks: DRACO, SimpleQA, four vertical benchmarks, and a web extraction benchmark. The results page carries the leaderboards; this page documents how each number is produced and what it does and does not establish. The benchmarks sit at different levels of independence, and reading them correctly starts there.

Nimble is a measured vendor in all four. DRACO's task set and rubrics and SimpleQA's questions are third-party work; the vertical question sets are Nimble-authored; the extraction harness, datasets, and scoring are open so the run can be reproduced. The DRACO and SimpleQA harnesses are published under Apache-2.0 at github.com/Nimbleway/Public-benchmarks, so every number in those two tables can be regenerated from code.

DRACO: task set, rubrics, and scoring

DRACO (Deep Research Accuracy, Completeness, and Objectivity) was published by Perplexity AI (paper, dataset on Hugging Face). Its 100 tasks are sampled from a de-identified corpus of real deep-research requests, then filtered and augmented so each is open-ended, complex, and objectively evaluable. The set spans 10 domains at 10 tasks each (finance, shopping and product comparison, academic, technology, general knowledge, UX design, law, medicine, needle-in-a-haystack, personalized assistant) and requires sources from 40 countries across five regions, which penalizes providers that only index the easy, English-language, US-centric web.

Rubric construction

Each task has its own rubric written by domain experts: attorneys, physicians, financial analysts, engineers, designers. The process: 26 domain experts drafted rubrics with LLM assistance at 45 to 60 minutes each; a second expert peer-reviewed each rubric; saturation testing sent any task scoring above 90% back to be made harder (about a 45% revision rate); final QA returned roughly 10% for rework.

Rubric axes and weights

Every criterion is judged MET or UNMET and carries a weight. Rubrics average 39.3 criteria per task.

The -500 penalties apply to actively harmful guidance in medical tasks; other domains use -10 to -25. Because negative weights are large and positive weights are small, a single confidently wrong claim can wipe out several correct ones. High scores require retrieval that is complete and correct, not verbose.

Scoring formula

raw   = Σ ( weight_i × verdict_i )      // verdict ∈ {MET = 1, UNMET = 0}
score = max( 0, min( 1, raw / Σ positive_weights ) ) × 100%

The DRACO paper's own grading used Gemini 3-Pro as judge, with GPT-5.2 and Sonnet-4.5 as robustness checks and five grading runs per task (reported standard deviations of ±0.2 to 0.4%).

How Nimble's run was judged

Nimble's published run used its open-source DRACO harness, executed on July 29, 2026, with anthropic:claude-sonnet-5 as judge and three samples per criterion. The design choices:

• One criterion per judge call, binary verdict only. The judge returns MET or UNMET plus a one-sentence justification, never a score or a weighing of criteria. The formula above is applied in code, so the final number is reproducible from the stored verdicts.

• Three independent samples per criterion, averaged. This reduces per-criterion judge noise. Criteria where the samples disagree are tracked, which flags ambiguous rubric items.

• Unparseable verdicts count as UNMET, biasing scores down rather than granting free credit, and are flagged in the output.

• The overall score is computed over the flat list of all criteria, not as an average of the four axis scores, which would silently re-weight the rubric.

• Identical presentation for every system. Each system's own bibliography is stripped and replaced with one bibliography in one format, built from its structured citations, so no adapter can win on citation formatting.

• Failures are scored, not dropped. A system call that times out or errors is recorded as 0.0. Excluding it would reward a system for failing on its hardest items. Interrupted runs resume and retry only items that errored or are missing.

• Byte-identical rubrics. The dataset is fetched from Hugging Face at run time and cached locally, so every system is scored against the same copy.

What Nimble's DRACO run establishes, and what it doesn't

• All eight vendor configurations answered the identical 100 tasks and were graded on identical criteria, in an evaluation run executed by Nimble. Nimble is one of the vendors measured.

• Two axes are reported per vendor: overall score and factual accuracy. The published breakdown does not separate breadth/depth, presentation, and citation quality per vendor.

• Cost per task is average spend as billed where the vendor reports it, otherwise derived from published pricing, so it is not strictly like-for-like across vendors, and it moves with pricing and task mix. Latency is average wall-clock time per task, including vendor queue time, and moves with load.

• Effort tiers are a dial, not a ranking: nimble/high and nimble/x-high are one system at two retrieval depths, as are the Exa and Parallel pairs.

• The DRACO paper notes absolute score magnitudes shift across judge models; relative ordering is the meaningful signal. Nimble's run used a different judge (Claude Sonnet 5) than the paper (Gemini 3-Pro), so absolute scores are not directly comparable with the paper's figures. DRACO's stated limits: single-turn only, static task set, text-only outputs, English-language evaluation despite global topics, and augmentation that may over-specify tasks. At 10 tasks per domain, domain-level differences are directional.

• The paper's own strongest published baseline scores 70.5%; it ran on a different judge, harness, and date than Nimble's run, so treat it as calibration, not a head-to-head.

• Parallel receives one instruction the other systems do not. Its Task API is not a report product: with no output specification it answers with a single short value, which a research rubric scores near zero. The harness therefore requests a comprehensive research report with inline citations. That is configuration parity, not a hint about the answer, but it is disclosed so Parallel's numbers are read with it in mind.

• Rubric criteria can go stale. DRACO tasks have factual answers that drift as the world changes.

SimpleQA: search API harness

SimpleQA is OpenAI's short-form factuality dataset (openai/simple-evals): questions with a single, verifiable answer. Nimble's SimpleQA harness uses it to compare web search APIs on retrieval quality.

Pipeline

For each question and each provider lane:

1. Query the provider's /search endpoint once, timing the round trip.

2. Extract chunks: normalize the response into a ranked list of URL and text.

3. Synthesize one answer from the top chunks with a single fixed model (gpt-4o), identical for every lane.

4. Grade the answer against SimpleQA's gold target with the official classifier (correct, incorrect, or not attempted), using the upstream grader prompt verbatim.

5. Score retrieval two further ways: URL-based NDCG, recall, and MRR against the gold source URL, and LLM-judged chunk relevance (UMBRELA).

6. Test significance: a paired t-test of every lane against a baseline lane, Bonferroni-corrected.

Nothing is cached between runs, and no lane sees another lane's output. Because every lane feeds the same synthesizer and grader, the accuracy column measures how well each vendor's retrieval supports one answer model. It is not a measure of any vendor's own answer API, and no lane is asked for a server-side answer.

Run configuration

• Sample: 500 questions drawn with the same seeded sampler as simple-evals (--limit 500 --random-state 0), so the identical subset can be drawn on any machine.

• Lanes: nine lanes across six vendors. Where a vendor sells a speed-versus-quality tier (Exa type, Parallel mode, Tavily search_depth), both ends run as separate lanes. Nimble ran at search_depth=standard. Brave runs with search operators off, because its default reads SimpleQA's quoted titles as required exact matches and empties about 12% of the dataset. Firecrawl returns snippets only, to stay comparable with the other result-list lanes.

• Provenance: every published figure is read from the run's analyzed_results.csv, not entered by hand.

Metric definitions

• Accuracy counts a non-answer as a miss. A request that reached the provider but returned nothing usable (a refusal or zero results) scores 0 for that question. Only rows whose request never landed are excluded, and those are reported as a failure rate, which is why some lanes show n below 500. Upstream simple-evals reports not-attempted as a separate bucket and divides by attempts only, so this figure is deliberately stricter than a published SimpleQA accuracy and should not be quoted against one.

• Latency is the provider round trip at p50 under load, timed inside the client-side rate limiter so the harness's own throttling and retry backoff are excluded. Each lane runs at whatever concurrency its rate limit permits, so these are loaded response times, not idle single-request latencies.

• Cost is each vendor's published list price per query for the lane's tier (snapshot of September 17, 2026), not a measured bill. It excludes overage beyond 10 results, Exa's page summaries and deep tiers, and volume discounts. Firecrawl, which is credit-metered, is normalized from its Standard plan.

Limitations, stated plainly

• The harness is Nimble-built and Nimble-run, though the dataset and grader are OpenAI's and every lane runs through the same code path.

• Accuracy reflects one synthesizer (gpt-4o). Changing the synthesis model moves every lane, so results are only comparable within a run.

• 500 questions is a slice of the full set. Small gaps, such as Nimble's 95.4% against Parallel basic's 95.8%, are within noise.

• List prices go stale, and the cost estimate slightly miscounts in both directions: retries are not charged, while failed requests are, even though most vendors don't bill them.

• Each vendor's terms of service govern publication of performance comparisons; anyone rerunning the harness is responsible for complying with them.

Vertical benchmarks: three-run design

The vertical benchmarks measure the repeat-use case DRACO deliberately doesn't: 100 domain-specific questions per vertical (GTM, company research, market analysis, social media monitoring), answered by Nimble, Exa, and Parallel, three times each, 36 runs in total. Accuracy is the reported metric.

Held constant: the same 100 questions per domain for every vendor; three runs per vendor per domain; no reconfiguration between runs for anyone; accuracy scored the same way across all vendors.

Deliberately not held constant: prior exposure to the domain. That is the variable under test. By run 3, Nimble has a private index and a memory of which retrieval paths worked; Exa and Parallel start from zero each time. Nimble's improvement comes from the harness learning, not from a human tuning it between runs.

Limitations, stated plainly

• The question sets are Nimble-authored, so domain selection and question phrasing are not neutral. This is a materially weaker independence setup than DRACO.

• Nimble ran all three vendors; competitor configurations, effort settings, and prompts were chosen by Nimble.

• Three runs is a small sample. Competitor run-to-run noise spans roughly ±0.4 points, so single-run differences under half a point are not meaningful.

• Run order is confounded with learning by design. The rising curve is consistent with memory accumulating, but three sequential runs cannot rule out other order effects. The mechanism is the claim; the curve is evidence, not proof.

• Competitors were not given a comparable memory layer; they don't have one to give. That is the product difference under test, but it means this is not an apples-to-apples test of cold-query retrieval.

• Nimble does not win run 1 everywhere: in GTM its first run trails Exa by 1.1 points. One-shot workloads should be judged on run 1.

Web extraction benchmark: dataset, ground truth, and composite scoring

Five engines (Nimble, Firecrawl, Exa, Parallel, Tavily) fetched the same URL corpus; returned content is graded locally and deterministically against ground truth, with no LLM judge.

The dataset

• 5,856 unique URLs across 2,107 domains, zero repeats: a 5k open-web core plus a ~19% Google block (search-result pages and Google-hosted content) added to stress search-result extraction.

• Verticals, so coverage is transparent: news/sports/entertainment 20.3%, web search/SERP 19.5%, e-commerce and reviews 13.3%, government and academic 11.3%, travel/real estate/local 8.8%, developer and docs 7.6%, reference and community 6.3%, finance/business/jobs 5.6%, health and medical 3.9%, general/other 3.3%.

• Render routing: 4,359 URLs served by the fast default path, 1,497 JS-heavy URLs routed to full render. Each engine fetched every URL identically.

• Data hygiene: 104 URLs excluded (11 dead for every engine, 93 returning server-side 5xx errors), plus 71 redirect-only URLs unextractable by every engine. 5,856 valid URLs were scored.

Ground truth and the two success definitions

Ground truth per URL is a verbatim excerpt confirmed in at least two independent extractors: real page text, not a paraphrase. Two success definitions are reported because "returned bytes" is not success:

• Raw fetch success: returned any non-empty content with a 2xx status. This rewards returning anything, including error pages and anti-bot screens.

• Useful success: returned content with completeness ≥ 0.3 against ground truth, excluding non-content responses.

Completeness is the fraction of the page's real content actually captured (anchor recall over successful fetches). F1 is token recall times best-window precision against the ground-truth passage, so it measures exact-content fidelity.

The composite formula

Score = 0.40 × Completeness

      + 0.30 × Success

      + 0.20 × F1

      + 0.05 × Latency        // exp( -p95_seconds / 10 )

      + 0.05 × Throughput     // min( avg_bytes_per_fetch / 100KB, 1.0 )

Quality outweighs speed by design: a fast extractor that returns blank pages should not win. Weights are configurable per run and disclosed with each report.

Limitations, stated plainly

• Config parity: extraction results depend on each engine's configuration. A like-for-like comparison should run every engine with comparable settings and per-URL effort.

• Ground truth is enriched from fetched content; the two-engine consensus reduces single-engine bias but does not fully remove it.

• Excluded rows include URLs where Nimble returned a server-side 5xx error, which raises Nimble's measured success rate. This is disclosed rather than hidden.

• Google SERP ground truth is refreshed from a contemporaneous fetch because results rotate frequently.

• Cost figures are list-price estimates per 1,000 successful extractions, not contract pricing; current rates are at nimbleway.com/pricing.

The standing recommendation

A benchmark written by someone who has never seen your product tells you less than one built from your own traffic. That applies to Nimble's benchmarks too. The four-step template for running your own evaluation is on the results page, and the developer docs cover the endpoints needed to wire one up.

Related pages

• Benchmark results: the full DRACO, SimpleQA, vertical, and extraction leaderboards, including where competitors win.

• Public-benchmarks on GitHub: the open-source DRACO and SimpleQA harnesses.

• Reliability and scale: the measured reliability numbers and plan-level SLAs.

• Web Search Agents, pricing, and the announcement post (July 29, 2026).