The four benchmarks

Nimble builds Web Search Agents and Search + Extract APIs that specialize web retrieval for AI agents. Four published benchmarks measure how that retrieval performs: DRACO, a third-party deep-research benchmark authored by Perplexity AI; SimpleQA, OpenAI's short-form factuality dataset run as a head-to-head of nine web search API lanes; Nimble's vertical benchmarks, which measure accuracy across repeat runs in four production-shaped domains; and a web extraction benchmark across 5,856 real-world URLs. This page carries the full leaderboards, including the categories where competitors win. Scoring design, rubric construction, and each benchmark's limitations are documented on the benchmark methodology page.

The results were announced on July 29, 2026 in How Nimble Ranks on Public and Vertical Web Search Benchmarks.

The DRACO and SimpleQA harnesses are open source under Apache-2.0 at github.com/Nimbleway/Public-benchmarks. Both are provider-neutral: every vendor runs through the same adapter interface, judge or grader, and scoring code, adding another vendor takes one file, and anyone with API keys can reproduce the numbers below.

• DRACO (third-party, 100 expert-graded deep-research tasks): Nimble ranks #1 overall at 74.0% and holds the top two factual-accuracy scores, 71.6% and 69.0%.

• SimpleQA (OpenAI dataset, 500 seeded questions, 9 search API lanes): Nimble scores 95.4% accuracy at 817 ms p50 latency, effectively tied with Parallel basic (95.8%) at roughly half its latency and the same list price, and at least 6 points ahead of every Exa, Tavily, Firecrawl, and Brave lane.

• Vertical benchmarks (4 domains, 100 questions each, 3 runs per vendor): Nimble's best run leads all four domains, and it is the only system whose accuracy improved every run: +4.4 points on average from run 1 to run 3, versus a +0.1 average for Exa and Parallel (+0.2 and 0.0 respectively).

• Web extraction (5,856 URLs, 2,107 domains, 5 engines): Nimble scores 85.9/100 composite with 99.1% fetch success and 0.902 completeness, the best completeness in all 10 content verticals.

DRACO: third-party deep research benchmark

DRACO (Deep Research Accuracy, Completeness, and Objectivity) is published by Perplexity AI (paper, dataset). Its 100 tasks are sampled from de-identified real deep-research requests across 10 domains, with sources required from 40 countries. Rubrics were written by 26 external domain experts, over half of every score is factual accuracy, and wrong facts carry penalties as large as -500 points. Nimble did not build the benchmark and did not author the rubrics. In Nimble's evaluation run, eight vendor configurations answered the identical 100 tasks and were graded against the identical rubrics.

Rank	Configuration	Overall score	Factual accuracy	Avg latency	Cost per task	Score per $
1	nimble / x-high	74.0%	71.6%	8.7 min	$2.00	37
2	exa / xhigh	72.1%	67.0%	2.5 min	$1.00	72
3	nimble / high	71.1%	69.0%	5.5 min	$0.50	142
4	parallel / ultra8x	71.0%	68.9%	15.6 min	$2.40	30
5	exa / high	70.3%	65.1%	1.3 min	$0.50	141
6	openai / 5.5	69.8%	67.6%	2.7 min	$0.56	125
7	parallel / ultra	66.8%	66.1%	10.6 min	$0.30	223
8	gemini / max	43.5%	46.4%	11.3 min	$5.41	8

Score and factual accuracy are normalized rubric percentages. Cost per task is average USD per completed task, as billed where the vendor reports it and otherwise derived from published pricing, so it is not strictly like-for-like across vendors. Latency is average wall-clock time per task, including vendor queue time. Effort tiers (nimble/high vs. nimble/x-high, and the Exa and Parallel pairs) are the same system at different retrieval depths. Run on July 29, 2026, 100/100 items for every configuration, judged by anthropic:claude-sonnet-5 with three samples per rubric criterion; the judge returns only MET/UNMET per criterion and the score is computed in code. Harness: DRACO-research-benchmark.

How to read it. Factual accuracy carries 52% of the rubric criteria and every catastrophic penalty, and it is the axis retrieval quality controls most directly. Nimble holds both top spots on it. Among configurations priced at or below $1.00 per task, nimble/high has the highest factual accuracy at 69.0%, ahead of openai/5.5 (67.6%), exa/xhigh (67.0%), parallel/ultra (66.1%), and exa/high (65.1%).

Where competitors win. parallel/ultra is the cheapest configuration at $0.30 per task and wins raw score-per-dollar (223 vs. nimble/high's 142), finishing 7.2 points behind Nimble's leading tier on overall score. exa/xhigh is second overall at 72.1% and costs half of nimble/x-high, though it ranks fifth on factual accuracy at 67.0%. If overall score at minimum spend is the whole decision, those are real alternatives; the table is here so you can make that call.

Exa is also much faster on deep research: exa/high averages 1.3 minutes per task and exa/xhigh 2.5 minutes, against 5.5 and 8.7 minutes for Nimble's high and x-high tiers. If a task has to finish in under a few minutes, weigh that column.

SimpleQA: web search API accuracy and latency

SimpleQA is OpenAI's short-form factuality dataset (openai/simple-evals). Nimble's open SimpleQA harness runs it as a search API benchmark. Each provider's /search lane returns ranked results, one fixed model (gpt-4o) synthesizes an answer from the top chunks, and the official SimpleQA classifier grades it. Because the synthesizer and grader are identical for every lane, the accuracy column measures retrieval quality, not any vendor's answer product. The 500-question slice uses the same seeded sampler as simple-evals (--limit 500 --random-state 0), so anyone can draw the identical questions.

Provider lane	Accuracy	Latency p50	List price per 1,000 queries	n
parallel_search_basic	95.8%	1,569 ms	$5.00	499/500
nimble_search	95.4%	817 ms	$5.00	500
exa_search_auto	89.2%	3,924 ms	$7.00	500
parallel_search_turbo	88.1%	482 ms	$1.00	495/500
exa_search_fast	86.6%	3,022 ms	$7.00	500
firecrawl_search	86.4%	1,584 ms	$1.98	500
brave_search	85.2%	1,058 ms	$5.00	500
tavily_search_basic	82.0%	2,174 ms	$8.00	499/500
tavily_search_fast	72.4%	937 ms	$8.00	500

Nimble was run at search_depth=standard. Accuracy counts a non-answer as a miss rather than a separate bucket, so it is deliberately stricter than a published SimpleQA figure and should not be compared against one. Rows whose request never reached the provider are excluded and counted separately, which is why some lanes show n below 500. Latency is the provider round trip under load, timed inside the client-side rate limiter. Price is each vendor's published list price per query (snapshot of September 17, 2026), not a measured bill.

How to read it. Nimble and Parallel basic are effectively tied on accuracy, 95.4% vs. 95.8%, a 0.4-point gap on 500 questions, and Nimble answered all 500 while Parallel basic completed 499. At the same list price, Nimble returned results in roughly half the time, 817 ms vs. 1,569 ms p50. Every Exa, Tavily, Firecrawl, and Brave lane trails Nimble by at least 6 points.

Where competitors win. Parallel basic posts the highest raw accuracy in the table. parallel_search_turbo is the fastest lane at 482 ms and the cheapest at $1.00 per 1,000 queries, at the cost of 7.3 points of accuracy against Nimble. Firecrawl is the next cheapest at $1.98 per 1,000 with 86.4% accuracy. If raw cost or sub-500 ms latency is the binding constraint and a few points of accuracy are acceptable, those lanes are real alternatives.

Vertical benchmarks: accuracy across repeat runs

Production agents hit the same domain thousands of times, so Nimble also benchmarks the repeat-use case: 100 domain-specific questions per vertical, run three times per vendor against Nimble, Exa, and Parallel, with nothing reconfigured between runs. These question sets are Nimble-authored and Nimble-run, a materially weaker independence setup than DRACO; the caveats are on the methodology page.

Domain	Vendor	Run 1	Run 2	Run 3	Run 1 → 3
GTM	Nimble	85.0%	88.8%	90.1%	+5.1
GTM	Exa	86.1%	86.0%	86.3%	+0.2
GTM	Parallel	84.0%	84.4%	84.8%	+0.8
Company research	Nimble	94.0%	96.1%	96.8%	+2.8
Company research	Exa	92.9%	93.2%	93.0%	+0.1
Company research	Parallel	92.0%	92.0%	91.8%	-0.2
Market analysis	Nimble	83.4%	84.0%	88.5%	+5.1
Market analysis	Exa	82.8%	83.4%	83.5%	+0.7
Market analysis	Parallel	81.3%	80.8%	80.7%	-0.6
Social media monitoring	Nimble	82.4%	83.5%	86.9%	+4.5
Social media monitoring	Exa	81.9%	82.0%	81.7%	-0.2
Social media monitoring	Parallel	78.7%	78.5%	78.5%	-0.2

How to read it. At run 1 the three systems are close, and in GTM Nimble's first run trails Exa by 1.1 points. By run 3, Nimble leads every domain: +3.8 over Exa's best run in GTM, +3.6 in company research, +5.0 in market analysis, +4.9 in social media monitoring. The mechanism is memory: Nimble stores each run's outputs, sources, and search history and reuses the retrieval paths that produced the right data, while a stateless index answers run 3 the way it answered run 1. One Nimble customer measured a 21% accuracy improvement alongside a 51% reduction in total AI token consumption on their own workload (source).

Where competitors win. Exa wins GTM on a single cold run (86.1% vs. 85.0%), and across the four domains a one-shot comparison is roughly a coin flip. If your workload is genuinely one-shot and never revisits a domain, judge on run 1, where these systems are within about a point of each other.

Web extraction benchmark: 5,856 URLs, 2,107 domains

Five extraction engines fetched the same 5,856 real-world URLs across 2,107 domains: Nimble, Firecrawl, Parallel, Tavily, and Exa. Every URL is graded against a verbatim ground-truth snapshot, locally and deterministically, with no LLM judge. The composite score weights completeness at 40%, fetch success at 30%, F1 at 20%, and latency and throughput at 5% each; a fast extractor that returns blank pages does not win by design.

Engine	Composite	Fetch success	Useful success	Completeness	F1	p95 latency	Est. $ / 1k successful
Nimble	85.9	99.1%	92.0%	0.902	0.886	19.5s	$0.91
Firecrawl	77.8	96.0%	80.8%	0.783	0.815	48.6s	$0.86
Parallel	67.9	81.6%	65%	0.736	0.661	38.9s	$1.23
Tavily	62.5	61.9%	50%	0.739	0.498	3.0s	$1.62
Exa	61.1	64.7%	49%	0.696	0.515	4.6s	$1.55

Fetch success counts any non-empty 2xx response. Useful success requires completeness of at least 0.3 against ground truth, which excludes 404 pages and anti-bot screens that return as "successful" fetches. Cost figures are list-price estimates per 1,000 successful extractions.

How to read it. Nimble has the highest completeness in 10 of 10 content verticals and the best completeness on 1,936 of 2,107 domains (Exa leads on 65, Firecrawl on 61, Parallel on 33, Tavily on 12). The gap is widest on search-result pages, where Nimble's completeness is 0.889 against 0.643 for Firecrawl and near zero for Exa and Tavily.

Where competitors win. Tavily and Exa are far faster, with p95 latencies of 3.0s and 4.6s against Nimble's 19.5s; Nimble routes JS-heavy pages through full rendering, which buys completeness at the cost of tail latency. Firecrawl is slightly cheaper per successful extraction at $0.86 vs. $0.91. If p95 latency is your binding constraint and partial page content is acceptable, the speed column matters more than the composite.

Run your own benchmark

Every vendor, Nimble included, publishes benchmarks; the test that matters is the one built from your own traffic.

1. Pull 100 real queries from production, with their long tail and bad phrasing intact.

2. Define what a correct answer contains: the fields, sources, and formats your downstream steps need.

3. Run every candidate three times, not once. A single run cannot distinguish a system that learns from one that doesn't.

4. Score accuracy and cost per answer together, across all three runs.

Nimble's APIs include 5,000 free requests per month with no credit card (pricing), and the developer docs cover Search, Extract, Crawl, Map, and Web Search Agents endpoints for wiring up a head-to-head.

To reproduce the public results instead of starting from scratch, clone Nimbleway/Public-benchmarks. The DRACO harness accepts any agent that returns a report and citations, and the SimpleQA harness accepts any search API that returns ranked results, so you can add your own system or a vendor not listed here and score it on the same terms.

Related pages

• Benchmark methodology: rubric construction, run design, scoring formulas, and every limitation worth knowing before citing these numbers.

• Reliability and scale: the measured reliability numbers behind production deployments, plus plan-level concurrency and SLAs.

• Web Search Agents and pricing.