Why this playbook matters

High‑scale data teams succeed by standardizing how they render pages, choose execution drivers, preserve sessions, and control page weight. This operational playbook compiles Nimble‑specific best practices into a durable, repeatable set of patterns that reduce cost, improve success rates, and harden pipelines against site changes.

Selective rendering vs. fetch

Choose the lightest execution path that still guarantees complete data.

• Prefer native fetch when the target is server‑rendered or exposes stable XHR/JSON endpoints; avoid the cost of a full browser render. See direct XHR access guidance in Skip the HTML—go straight to the data.

• Escalate to rendering for JS‑heavy or highly interactive pages; Nimble auto‑decides, but you can control it via render: true and render_options (e.g., render_type: load|domready|idle0|idle2, include_iframes, timeout) in the JS Rendering docs.

• The business case: at scale, the delta between a 300 ms fetch and a 7 s render defines feasibility and cost. See benchmarking and decision rules in Rendering vs. fetch.

Decision checklist:

• Can you retrieve a first‑party JSON/XHR with complete fields? Use fetch.

• Is critical content injected by React/Vue/Next or gated by client events? Render with the lightest driver that succeeds.

• Are iframes used for key data (e.g., embedded reviews/maps)? Enable include_iframes selectively.

Browserless driver selection (cost/perf control)

Lock predictable performance and cost per domain by selecting the right driver. See Browserless Driver Selection and the driver behavior notes in the JS Rendering docs.

Driver	Typical use	Rendering default	When to choose
vx6	Pure fetch/SSR pages, stable XHR	Off	Cheapest path for price pages, sitemaps, RSS, JSON endpoints
vx8	Light JS, minimal anti‑bot	On	Modern PDP/search with modest scripting; occasional clicks/scrolls
vx8‑pro	JS + moderate defenses	On	Needs stealth plus interactions; heavier anti‑bot/CDN layers
vx10	Heavy JS, rich UI flows	On	Complex SPAs, nested iframes, client‑side API gating
vx10‑pro	Maximum stealth	On	Most protected targets; tighten success on sensitive domains

Guidelines:

• Start with vx6; escalate only if completeness checks fail.

• Pin drivers for repeatability in regulated/benchmark pipelines.

Maintain session continuity (cookies)

Multi‑step flows often fail due to lost state, not hard blocks. Use render_flow with get_cookies to capture post‑render cookies and replay them on subsequent requests, ensuring filters, sort order, or pagination persist. Detailed patterns in Session continuity with get_cookies.

Best practices:

• Capture cookies after the page stabilizes (use appropriate render_type).

• Reuse only domain‑scoped cookies required for state; avoid over‑broad sharing.

• Rotate sessions by pipeline to prevent cross‑contamination.

Infinite scroll, clicks, and idle timeouts

For modern lists/grids, combine automated clicking and a deterministic end condition:

• Use click_on_selector to activate “Load more” between scrolls and idle_timeout to stop once content stops changing. See Smarter infinite scroll.

• Validate by counting cards/items before and after scroll cycles.

Control page weight with blocked_domains

Cut payload, speed up renders, and reduce breakage by blocking non‑essential third‑party assets (ads, analytics, heavy video CDNs):

• Configure render_options.blocked_domains: ["doubleclick.net","googletagmanager.com", …] per domain.

• Expect >50% payload reduction on ad‑heavy pages (Yelp example) in blocked_domains.

• Maintain an allowlist for first‑party CDNs to avoid accidental data loss.

Screenshot diffs for visual monitoring

When data is visual (e.g., hero banners, promo badges) or CSS‑dependent:

• Capture full‑page screenshots in the render flow and store base64 or object storage URLs; compare consecutive runs to detect layout or promo changes. Workflow in Screenshot & diff monitoring.

• Pair with structured extraction so visual alerts point to rows/IDs.

Headers, cookies, and proxy tuning (reliability at scale)

• Headers: set consistent User-Agent, Accept-Language, and Accept to match the intended locale and content type; avoid randomizing per request unless testing. See HTTP headers deep dive.

• Cookies: treat them as state—minimize surface, scope per domain, and rotate with session boundaries. See HTTP cookies.

• Proxies: use AI‑optimized residential peers for geofencing, reputation, and long sessions; prefer provider‑managed rotation over DIY pools. Review Residential Proxies and the AI Optimization Engine for IP matching and stability.

• Upgrade existing proxy workflows to a smart endpoint when you need unblocking plus render/parse controls via headers (country, locale, parse, render). See Unlocker Proxy.

Hyperlocal geotargeting without proxy ops

For city/ZIP‑level accuracy in prices, inventory, and map/search results:

• Specify country, state, and city directly in requests; Nimble handles IP geotargeting under the hood. Patterns and examples in Hyper‑local geotargeting.

• Validate location by asserting store IDs, currency, or localized SERPs.

When to bypass rendering entirely (XHR mode)

If you’ve located a stable, versioned internal API serving JSON:

• Set the Web API to hit the XHR URL directly with is_xhr: true to skip DOM render and parsing; dramatically reduces latency and cost. See Direct XHR access.

• Always keep a render fallback when undocumented endpoints change.

Data quality guardrails (what to validate every run)

• Freshness: timestamp every row; alert on unexpected staleness.

• Completeness: assert required fields per schema; auto‑heal parsers where supported.

• Consistency: normalize currencies, units, locales; enforce schema at write.

• Anomalies: price jumps, stock toggles, or card counts—flag for review.

Compliance and governance defaults

Operate with governance‑by‑design: collect only public data, respect site health, and keep audit trails (drivers, headers, locales, IP class). Review Nimble’s Trust Center for GDPR/CCPA/SOC 2 practices and processor roles.

Quick implementation checklist

• Map targets by execution need (fetch → vx6; light JS → vx8; heavy/stealth → vx10/‑pro).

• Prefer XHR JSON over DOM parsing; keep render fallbacks.

• Enable session continuity (get_cookies) for multi‑step flows.

• Handle infinite scroll with click_on_selector + idle_timeout.

• Trim bloat with blocked_domains; maintain per‑domain allowlists.

• Geotarget via request parameters, not DIY proxy ops.

• Standardize headers and cookie scopes; choose residential peers with AI matching.

• Add screenshot diffs where visual correctness matters.

• Enforce schema/QA gates and alerting on anomalies.

Source documents

• Rendering vs. fetch and driver tiers: Rendering guide, Driver selection, JS Rendering docs

• Sessions and scroll: Session continuity, Infinite scroll

• Page weight and visuals: blocked_domains, Screenshot monitoring

• Headers/cookies/proxies: HTTP headers, HTTP cookies, Unlocker Proxy, Residential proxies, Optimization engine

• Geotargeting & XHR: Hyper‑local geotargeting, Direct XHR access