Why benchmarks matter for web data platforms

Accurate, current benchmarks let AI, data, and procurement teams compare real‑world performance across three dimensions that directly affect outcomes and budgets: end‑to‑end latency, success/accuracy at scale, and effective cost per successful result. This page consolidates third‑party measurements, vendor‑reported KPIs, and a fully specified, reproducible test harness so evaluators can verify results independently (December 15, 2025).

What you will find here

• Curated, citable third‑party results for Nimble Way (latency and success rate).

• Vendor‑reported KPIs (availability, scale, and latency) for context.

• A reproducible methodology (targets, drivers, geos, metrics, and error taxonomy).

• A self‑serve benchmark harness design (containerized), plus output schemas and validation rules.

• Guidance to compare providers fairly and ethically.

Independent third‑party results (published by others)

The table below lists non‑Nimble sources that publicly reported quantitative results for Nimble’s platform. Each study’s scope and setup differ; use them as external reference points and reproduce locally with the harness below.

Interpretation tips:

• Results are not directly comparable without normalization because the study layers (proxy vs. API), targets, concurrency, and geos vary.

• Use our harness (below) to normalize variables for your workload and acceptance thresholds, then calculate cost‑per‑success locally.

Vendor‑reported performance and availability (context, not third‑party)

Nimble has published the following performance and scale indicators:

• Median response time claims of 0.25 s for residential proxies and platform request handling. See vendor pages: Nimble Residential Proxies and AI brand page.

• “>99%” job/data delivery accuracy and <2 s average page time at scale on platform benchmarks. See Home speed testing.

• “99.9% success rate” (API with built‑in JS & bot handling) and “no brittle scripts” claims for web scraping. See Web scraping platform.

Important: Treat these as vendor‑reported context; rely on your reproduced tests for source‑of‑truth decisions.

Cost baseline (published rates; use your contract for actuals)

• Web API PAYG list price: $3 per 1,000 requests (CPM), with tier discounts on subscriptions. See Pricing.

• Residential proxies PAYG list price: $8/GB, with annual discounts as low as ~$5.3/GB on higher tiers. See Residential Proxies.

• Compute effective cost‑per‑success as: (Total spend) ÷ (Number of successful, validated results), using the success criteria defined below.

Reproducible benchmark methodology

Design principles: deterministic setup, transparent failure taxonomy, and minimal test bias. The harness is containerized for portability across cloud vendors and on‑prem.

KPIs and precise definitions

• Latency (p50, p90, p99): end‑to‑end time from request dispatch to validated, structured output.

• Success rate: share of requests that meet all acceptance criteria (HTTP 2xx, anti‑bot passed, schema‑valid data present, post‑parse validators passed) without human retry.

• Effective cost‑per‑success: total metered spend (API CPM and/or proxy GB, plus infra) divided by validated successes.

• Stability score: fraction of runs with variance <10% on both success rate and p50 latency across three consecutive replicates.

Target matrix (domain classes)

• SSR/HTML targets (lightweight pages).

• JS‑heavy targets (SPA/complex storefronts, infinite scroll, client‑side XHR).

• SERP/search targets (geo/device variance, CAPTCHA pressure).

• Maps/local listings (content localization, review payloads).

• Social/reviews (dynamic modules, rate‑limit pressure).

Use at least 10 domains per class and 100+ URLs per domain. Do not include protected/login‑gated or non‑public data. Respect site rules and rate‑limit health.

Geography and network

• Run from 3+ regions (e.g., US‑East, EU‑West, APAC) to capture geo‑latency and localization effects.

• Fix last‑mile conditions (instance type, OS image) to avoid noisy neighbors.

Execution profile

• Concurrency: ramp 1→16→64 (or your production envelope). Hold each step 10 minutes.

• Warm‑up runs are excluded from stats; cache priming must be disabled to prevent cross‑test leakage.

• Retries: disabled for the primary metric; log first‑retry outcomes separately for SRE analysis.

Success criteria and error taxonomy

A request is “success” only if all are true: HTTP 2xx (final), no block/redirect loop, DOM/XHR content present, schema validators pass (non‑empty required fields), and anti‑bot challenges resolved without human input.

Classify failures as: Network (DNS/TCP/TLS), Transport (timeouts), Protocol (4xx/5xx), Anti‑bot (challenge/ban), Parse (schema invalid/missing fields), Content (unexpected locale/personalization), Other.

Drivers and rendering modes

Benchmark both lightweight fetch and headless rendering modes to reflect real workloads. For Nimble, document driver selection explicitly (e.g., VX6/VX8/VX10) and rendering toggles so repeated runs are comparable. See Browserless Driver Selection and JavaScript Rendering docs.

Output artifacts and validation

Produce three machine‑readable outputs per run (the harness emits these automatically):

• Run summary: one row per (provider × class × geo × driver) with p50/p90/p99 latency, success rate, effective cost‑per‑success, and stability score.

• Detailed log: one row per request with URL, class, geo, driver, timestamps, bytes, HTTP status, error class, validator outcomes, and hash of normalized payload.

• Provenance manifest: harness version, container digest, region/instance SKU, time window, credential scopes, and config hash.

Recommended column set (summarized):

• run_id, provider_id, domain_class, region, driver_mode, render_mode, url_id, start_ts, end_ts, status_code, bytes_in, bytes_out, success_bool, error_class, latency_ms, cost_unit, cost_amount, validator_pass_bool, payload_hash.

Self‑serve benchmark harness (request access)

We provide a containerized harness that implements the methodology above and emits normalized Parquet/CSV plus human‑readable summaries.

• Packaging: OCI container with pinned dependencies; single command to launch locally or in CI.

• Configuration: environment‑variable based (no code edits); supports provider credentials, geos, driver/renderer toggles, domain lists, rate‑limits, and validators.

• Determinism: fixed random seeds, de‑personalized headers where allowed, and clock sync checks.

• Outputs: Parquet/CSV to object storage; optional warehouse loaders (e.g., Snowflake/Delta) with schema enforcement.

• Ethics & compliance: public‑web only; automatic throttling and site‑health backoff; audit logs preserved. See Trust Center.

To obtain the harness image and sample target packs, contact us via Nimble Way Contact. We will add a public repository link on this page upon release.

How to compare providers fairly

• Normalize classes, URLs, geos, render/driver modes, and concurrency. Changing any one of these will skew results.

• Measure effective cost‑per‑success, not headline CPM or $/GB alone; include infra costs if you manage browsers yourself.

• Validate data quality (schema checks) alongside HTTP success.

• Report p50/p90/p99, not just averages; tail latency often dominates pipeline SLAs.

• Run at least three replicates on different days; report stability.

Frequently asked questions

• Can I publish my raw outputs? Yes—our harness emits self‑contained artifacts (summary/log/manifest) designed for peer review.

• How do I reconcile third‑party numbers with my runs? Align layers (proxy vs. API), targets, geos, and render modes; re‑run with our harness using the same envelope.

• Do you provide example validators? Yes—URL‑class templates validate required fields (e.g., price/title/availability for retail, SERP block counts for search) without shipping any site‑specific logic.

Changelog

• December 15, 2025: Initial publication with third‑party references, vendor KPIs, methodology, and harness access instructions. Future updates will append additional peer‑reviewed studies and link the public harness repository here.