Why this matters

Search tells an agent where the answer lives. Extract gets the content out of the page. On JS-heavy sites, filtered listings, and paginated results, a plain fetch often returns an incomplete page, and an agent reasoning over an incomplete page gives an incomplete or wrong answer.

This playbook covers how to get complete, LLM-ready content from hard pages with Nimble, and when to hand the job to a Web Search Agent instead.

Pick the right tool first

Situation	Use
The agent doesn't know where the answer is	Search, or a Web Search Agent for multi-step research
A popular site already has a template (Amazon, Walmart, Google Search, Google Maps, and more)	Extraction Template: normalized JSON, maintained by Nimble
A specific known URL	Extract
A whole site section, such as a docs site or catalog	Map to discover URLs, then Crawl

Many tasks combine them: Search finds the right pages, then Extract or a Template retrieves the content.

Rendering: let Nimble decide, override when needed

Nimble decides automatically whether a page needs full JavaScript rendering. For single-page apps (React, Vue, Angular) and content that loads client-side, turn rendering on explicitly so the agent receives the page as a user would see it. Rendering options let you wait for the right moment (page load, DOM ready, or network idle) and include iframes when key content, such as embedded reviews or maps, lives inside them. Extract docs

Checklist:

Is the content in the initial HTML? A standard request is enough.
Is it injected by client-side scripts or gated by user events? Enable rendering.
Is it inside an iframe? Include iframes selectively.

Browser actions for nested content

Much of the most valuable data sits a few interactions deep: behind "load more" buttons, filters, tabs, or infinite scroll. Extract supports browser actions such as click, scroll, fill, and wait so the agent can reach it.

For infinite scroll and "load more" lists, combine clicks with an idle condition that stops once no new content appears.
Validate completeness by counting items before and after each scroll cycle.

Network capture for cleaner data

Many modern pages load their data from an internal API rather than rendering it into HTML. Network capture returns those responses directly, which often gives cleaner, more complete JSON than parsing the page. Keep a rendered fallback in case the endpoint changes.

Multi-step flows and sessions

When a task requires several steps, such as applying a filter, sorting, and paginating, keep session state across requests so filters and pagination persist. Most multi-step failures come from lost state, not from the site itself.

Localized results

Prices, inventory, local packs, and search results vary by location. Pass country, state, or city in the request and Nimble returns the page as a local user would see it. Access and localization are handled automatically, with no proxy configuration. Validate by checking currency, store IDs, or localized content.

Output for LLMs

Choose the output format for how the agent will use it:

Markdown for readable context to pass into a model
Parsing schema for typed fields the agent or workflow consumes directly
Screenshots when the information is visual, such as banners, badges, or layout

Smaller, cleaner output means fewer tokens and more accurate reasoning.

Quality checks every run

Freshness: timestamp every record and alert on unexpected staleness.
Completeness: assert required fields per schema.
Consistency: normalize currencies, units, and locales.
Anomalies: flag sudden price jumps, stock flips, or drops in item counts for review.

When to step up to a Web Search Agent

If the task spans many sources, requires deciding which pages matter, or needs cross-checking and citations, delegate it to a Web Search Agent. The agent plans the work, runs search, browsing, and extraction, and returns a cited answer or a schema-matched dataset. It also learns which retrieval paths work, so recurring tasks get faster and cheaper. Web Search Agents

Compliance and governance

Collect only public data, respect site health, and keep audit trails. See Nimble's Trust Center for GDPR, CCPA, and SOC 2 practices and processor roles.

Quick checklist

Choose the tool first: Search, Web Search Agent, Extraction Template, Extract, or Map and Crawl.
Let Nimble decide on rendering; enable it explicitly for client-rendered pages.
Use browser actions for "load more", filters, and pagination.
Prefer network capture when the page loads data from an API.
Keep session state for multi-step flows.
Pass location parameters for localized content.
Request markdown or a schema, not raw HTML, for LLM consumption.
Enforce schema and quality checks on every run.

Getting Complete Content from Hard Pages: Extract Best Practices for AI Agents