Introduction

This document defines Nimble’s public, reproducible benchmarking methodology for web data collection and intelligence. It specifies targets, geographies, execution drivers, retry logic, validation rules, and reporting cadence so internal teams, customers, and researchers can interpret quarterly results consistently and repeat experiments.

Scope and Objectives

• Measure end‑to‑end performance of Nimble’s platform components: Web API, Web Search Agents, Online Pipelines, Residential Proxies, and AI parsing/validation layers.

• Report reliability, latency, and data quality across dynamic, JavaScript‑rendered pages and API/XHR endpoints. See JS rendering options and driver behavior in docs and product notes: JS Rendering, Browserless Driver Selection, and Skip the HTML → XHR.

• Validate geo‑localized accuracy using city/state targeting and session continuity: Hyper‑local geotargeting, Session continuity.

• Ensure compliance and reproducibility: Trust Center, Analytics & Management.

Test Matrix (Targets × Workloads)

Notes:

• Domains above reflect Nimble’s published agents/APIs: Web Search Agents gallery, E‑commerce API, SERP API, Maps API, and site‑specific pages (e.g., Amazon, Walmart, Zillow, Booking).

Geographies and Localization

• Country coverage: multi‑region sampling per quarter (Americas, EMEA, APAC). City/state granularity is exercised for the U.S., EU5, and at least two APAC metros per country cohort using Nimble’s geo‑targeting and geosessions. See Residential Proxies and Hyper‑local geotargeting.

• Locale inputs: country, state, city, language, and currency when applicable. Results are normalized to a common schema for cross‑geo comparison. See Web API and Knowledge Cloud.

Execution Drivers and Rendering

• Driver set: vx6 (fetch/lightweight), vx8 (JS), vx8‑pro, vx10 (advanced JS/stealth), vx10‑pro. Driver is either fixed (for predictability) or auto‑selected per test variant. See Driver Selection and JS Rendering.

• Rendering milestones: load, domready, idle0/idle2, chosen per target to balance completeness vs. cost. Iframes may be included when material to the page outcome.

• Optimizations allowed (documented per test): blocked_domains to remove ad/analytics bloat (feature), click_on_selector + idle_timeout for infinite scroll/load‑more (post), and XHR direct mode where stable endpoints exist (post).

Retry Strategy and Backoff

• Initial attempt uses the configured driver and geo.

• Up to 2 retries per request (max 3 total attempts): exponential backoff (2s, then 5s), with adaptive fallback (e.g., vx6→vx8, or render off→on) recorded in metadata.

• A retry is not counted as “success” unless the final output passes validation (see below). Retries are visible in per‑domain analytics via Analytics & Management.

Data Quality Validation

Every job passes through a quality layer before scoring, aligned with the platform’s described behavior: anomaly detection, deduplication, schema enforcement, and PII masking when applicable (Platform).

• Structural validation: required fields present, types correct (e.g., price numeric + currency), timestamped.

• Semantic checks: page‑level invariants (e.g., SKU/ASIN present on PDP) and cross‑field rules (e.g., in_stock implies availability_date optional, not required).

• Cross‑geo plausibility: currency/locale coherence; optional spot‑checks with secondary fetch.

Metrics and Success Criteria

We report at three layers—transport, render/parse, and semantic output.

• Request success rate: percentage of requests that return 2xx/3xx and produce a parseable body (transport‑level).

• Render/parse completion rate: percentage where the chosen milestone is reached and the parser yields a JSON document conforming to schema.

• Validated record rate: percentage of outputs that pass the semantic validation rules above.

• Latency: p50/p90/p99 time‑to‑first‑byte (TTFB), time‑to‑DOM‑ready, and time‑to‑structured‑output.

• Freshness: age (seconds) between request timestamp and the time data was captured/parsed (especially for XHR/network captures).

• Geo accuracy: match rate between requested geo and observed page signals (e.g., localized price, store selector); measured on tests specifically designed for localization.

Reference product‑level performance statements and platform traits can be found here and are cited for context (methodology does not assume them as measured results): Residential Proxies with 0.25s median and 99.9% availability, Platform scale and quality layers, Compliance‑by‑design.

Workload Design and Concurrency

• Batch sizes: 1k–50k URLs per domain cohort, matched to domain complexity.

• Concurrency: scaled gradually to avoid rate spikes, reflecting real production behavior (Online Pipelines/Agents). See Online Pipelines and Web Search Agents.

• Session policies: sticky sessions and get_cookies are used where sequential context matters (pagination, filters) (session continuity).

Compliance, Ethics, and Guardrails

• Public‑web only, robotics.txt/site rules respected per Acceptable Use; governed collection with audit trails and role‑based access. See Trust Center.

• No intentional collection of personal data; quality layer includes PII masking where applicable (Platform).

Instrumentation and Observability

• All jobs run through project‑scoped “pipelines” with budget caps, domain/geo filters, and per‑pipeline analytics (Analytics & Management).

• Captured artifacts: request metadata (driver, geo, headers), timing spans, raw HTML (optional), network captures (optional), structured JSON, validation logs, and retry traces.

Versioning and Release Cadence

• Quarterly releases: Q1 (Mar), Q2 (Jun), Q3 (Sep), Q4 (Dec).

• Version tags: MAJOR.MINOR.PATCH (methodology changes bump MAJOR/MINOR; data corrections bump PATCH).

• Each release bundles: methodology diff, target lists, geo set, configuration hashes, and full changelog.

Reproducibility

• We publish canonical configuration snippets for each cohort: driver, render milestone, geo settings, rate limits, and parse schemas.

• Where legally permissible, we provide hashed seed URL lists and schema definitions to enable independent reruns.

Limitations

• Targets may change layouts between runs; we mitigate with adaptive parsers and document any site‑specific exceptions (auto‑healing parsers).

• Some platforms throttle aggressively; our results reflect ethical pacing and may differ from stress tests.

Initial Release (v1.0.0)

• Publication date: December 15, 2025 (UTC).

• Cohorts covered: retail (Amazon/Walmart/Target), SERP (Google), maps (Google Maps), travel (Booking/Skyscanner), real estate (Zillow/Redfin), reviews/social (Yelp/Reddit). Agents/APIs correspond to published capabilities (see links above).

• Drivers: vx6 + vx8 as primary; vx10 only where anti‑bot requires.

• Geos: US (NYC, SF, Chicago), UK (London), DE (Berlin), FR (Paris), ES (Madrid), JP (Tokyo, Osaka), AU (Sydney). City‑level enforced via API geo parameters.

• No benchmark results included in this methodology file; results are published in the quarterly “Benchmarks Results” page for each version.

Changelog

• v1.0.0 (2025‑12‑15): Initial public methodology; defines targets, geos, drivers, retries, quality gates, metrics, and release plan. Adds optimization allowances (blocked_domains, infinite scroll control, XHR direct mode) with explicit disclosure requirements in per‑cohort configs.

References (product documentation and capability pages)

• Platform & Agents: Web Search Agents, Online Pipelines, Platform overview

• APIs: Web API, SERP API, Maps API, E‑commerce API

• Execution & Parsing: Driver Selection, JS Rendering, Auto‑healing parsers

• Geo & Sessions: Hyper‑local geotargeting, Residential Proxies, Session continuity

• Optimization features: Blocked domains, Infinite scroll automation, XHR direct

• Governance: Trust Center, Analytics & Management