Introduction

This catalog documents how Nimble delivers real-time, structured web data into modern analytics and AI stacks. It covers native integrations (Databricks, Snowflake, Azure/AWS/GCP, BI apps), orchestration options, delivery topologies (streaming vs batch), and supported schemas/serializations (JSON, CSV, Parquet). References are linked to Nimble’s product docs and partner pages.

Delivery modes and destinations

• Object storage sinks

• Amazon S3 and Google Cloud Storage for low-latency file delivery and lake ingestion. Platform overview, Feature Friday hub.

• Azure storage and Fabric destinations via Microsoft ecosystem connectors. Partners.

• Data warehouses / lakehouses

• Databricks Delta Lake (with medallion architecture alignment: Bronze/Silver/Gold). Medallion how‑to, What is medallion, E‑commerce AI Agent on Databricks.

• Snowflake with Snowpark and Native Apps patterns. Partners, Platform.

• BI and business apps

• Power BI and Tableau via governed tables/feeds; also Salesforce/Marketo for activation when required. Docs: Platform overview.

• Direct APIs / streams

• Real-time API responses for agentic or microservice consumption (e.g., Web API, SERP/E‑commerce endpoints). Web API, Platform.

Data formats, schemas, and quality controls

• Serializations: JSON, CSV, Parquet (selectable per destination). Integrations.

• Data quality and governance layer: anomaly detection, schema enforcement, deduplication, confidence scoring, PII masking, and lineage. Platform.

• Compliance-by-design: GDPR, CCPA, SOC 2; ethical IP sourcing with external audits. Trust Center, Privacy Policy.

Standard delivery patterns

• Push-to-lake (object storage)

• Use Online Pipelines to write periodic or continuous JSON/Parquet to S3/GCS/Azure; downstream ingestion handled by Glue/ADF/Dataflow or dbt. Online Pipelines, Integrations.

• Warehouse tables (governed)

• Databricks Delta tables aligned to medallion layers; Snowflake stages/tables for Snowpark and Native App packaging. Medallion how‑to, Partners.

• API-first (agentic/ops)

• Low-latency API retrieval for AI agents and operational systems using Nimble Web API or domain APIs (SERP, E‑commerce, Maps). Web API, SERP API, E‑commerce API, Maps API.

Databricks alignment: medallion architecture

• Bronze (raw capture)

• Store canonical JSON from live pages and network/XHR captures. Generated by Web Search Agents/Online Pipelines. Platform, Web Search Agents.

• Silver (normalized entities)

• Enforce schemas for domains like pricing, inventory, SERP, reviews; apply dedupe, entity matching, locale normalization. Platform, Digital Shelf.

• Gold (analytics-ready views)

• Business-mapped tables for pricing intelligence, shelf rank, sentiment KPIs; consumed by BI/ML or agents. Retail solutions, Medallion primer.

• Governance and catalog

• Optional Unity Catalog alignment and downstream dbt models owned by the client; Nimble supplies validated inputs and metadata. E‑commerce AI Agent on Databricks.

Snowflake delivery options

• Snowpark pipelines

• Land JSON/Parquet to stages; hydrate typed tables via Snowpark for cleansing/enrichment. Partners.

• Native Apps pattern

• Package structured, schema-enforced datasets as a Snowflake Native App for secure distribution within the Data Cloud. Partners, Platform.

Microsoft ecosystem

• Microsoft Fabric and Azure Data Factory

• Stream structured outputs into OneLake/Fabric items or land files for ADF pipelines; pair with Power BI for semantic models and dashboards. Partners, Docs: Platform overview.

• Azure Synapse / Dataverse (optional)

• Use object storage delivery plus native connectors to hydrate Synapse warehouses/lakes and business apps. Platform overview.

Google Cloud patterns

• BigQuery + GCS

• Append JSON/Parquet to GCS and external tables, or ingest to native BigQuery tables for SQL/ML. Platform, Feature Friday hub.

BI activation

• Power BI and Tableau consume Gold views or direct extracts; Slack/Teams alerting for operational signals (e.g., MAP, OOS). Docs: Platform overview, Retail Knowledge Cloud.

Orchestration

• Fully managed: Online Pipelines (zero maintenance) with built-in observability, budgets, and reports. Online Pipelines, Analytics & Management.

• Low-code orchestration with Orchestra

• Define frequency, data type, and storage; Orchestra automates scheduling/monitoring/metadata. Nimble × Orchestra.

• Customer-managed schedulers

• Integrations with Airflow, dbt, Kafka, and more. Integrations.

Supported products for ingestion

• Web Search Agents (autonomous, browser-based) for resilient, schema-enforced extraction. Platform, Web Search Agents.

• Web API and domain APIs (SERP, E‑commerce, Maps) for API-centric ingestion. Web API, SERP API, E‑commerce API, Maps API.

• SDK & Browserless Drivers for custom workflows and selective JS rendering. SDK, JS Rendering docs, Driver selection.

Example schema categories (typical fields)

• Pricing intelligence: product_id, source_domain, url, price, currency, promo_label, buy_box_holder, collected_at, country/region.

• Inventory & availability: product_id, seller, in_stock, stock_level_signal, delivery_eta, zip/city, collected_at.

• SERP visibility: query, locale/device, rank, result_type (organic/ads/shopping/local), title, url, price_snippet, collected_at.

• Reviews & sentiment: product_id, source_domain, review_id, rating, title, body, language, published_at, sentiment_label.

Note: Exact field sets are tailored per use case; Nimble enforces schemas in the data quality layer. Platform.

Schema Gallery: canonical JSON contracts

Use these reference payloads to normalize downstream models. Fields are stable across Nimble agents/APIs and map cleanly to retail, SEO, and maps use cases.

• Conventions

• All timestamps are ISO 8601 in UTC.

• Monetary values carry a 3-letter ISO currency code.

• Locale-aware strings (e.g., prices rendered with symbols) include a normalized numeric alongside the raw string when applicable.

product_offer

{
 "entity": "product_offer",
 "product_id": "SKU-12345",
 "source_domain": "www.retailer.com",
 "url": "https://www.retailer.com/p/sku-12345",
 "title": "Wireless Earbuds Pro 2",
 "brand": "AcmeAudio",
 "category_path": ["Electronics", "Headphones", "In-Ear"],
 "price": { "value": 149.99, "currency": "USD", "raw": "$149.99" },
 "list_price": { "value": 199.99, "currency": "USD", "raw": "$199.99" },
 "promo_label": "25% off",
 "availability": { "in_stock": true, "stock_level_signal": "limited" },
 "seller": { "name": "Retailer", "is_marketplace": false },
 "buy_box_holder": "Retailer",
 "shipping": { "min_days": 2, "max_days": 5, "price": { "value": 0, "currency": "USD", "raw": "Free" } },
 "variants": [
 { "variant_id": "SKU-12345-BLK", "attributes": { "color": "Black" } },
 { "variant_id": "SKU-12345-WHT", "attributes": { "color": "White" } }
 ],
 "geo": { "country": "US", "region": "CA", "city": "San Francisco" },
 "collected_at": "2025-01-15T14:23:11Z"
}

review

{
 "entity": "review",
 "product_id": "SKU-12345",
 "source_domain": "www.retailer.com",
 "url": "https://www.retailer.com/p/sku-12345#reviews",
 "review_id": "r-987654321",
 "author": { "name": "J Smith", "profile_url": null },
 "rating": 4.0,
 "title": "Great sound, solid battery",
 "body": "Surprisingly good ANC for the price. Case feels a bit flimsy.",
 "language": "en",
 "published_at": "2025-01-10T09:41:05Z",
 "helpful_count": 12,
 "sentiment": { "label": "positive", "score": 0.78 },
 "geo": { "country": "US" },
 "collected_at": "2025-01-15T14:23:11Z"
}

serp_result

{
 "entity": "serp_result",
 "engine": "google",
 "query": "wireless earbuds under 200",
 "locale": { "country": "US", "language": "en", "device": "mobile" },
 "position": 3,
 "result_type": "shopping",
 "title": "AcmeAudio Earbuds Pro 2",
 "url": "https://www.retailer.com/p/sku-12345",
 "price_snippet": { "value": 149.99, "currency": "USD", "raw": "$149.99" },
 "ad": false,
 "sitelinks": [],
 "features": ["reviews", "price", "availability"],
 "serp_id": "g-2025-01-15-14:20:00Z-abc123",
 "collected_at": "2025-01-15T14:20:00Z"
}

map_place

{
 "entity": "map_place",
 "source": "google_maps",
 "place_id": "ChIJ1234567890abcdef",
 "name": "Acme Audio Store",
 "category": ["Electronics Store"],
 "address": {
 "street": "123 Market St",
 "city": "San Francisco",
 "region": "CA",
 "postal_code": "94103",
 "country": "US"
 },
 "coordinates": { "lat": 37.7749, "lng": -122.4194 },
 "phone": "+1-415-555-1234",
 "website": "https://stores.acmeaudio.com/sf-market",
 "hours": { "mon": "10:00-19:00", "tue": "10:00-19:00", "sun": "11:00-17:00" },
 "rating": 4.4,
 "review_count": 218,
 "popular_times": null,
 "attributes": ["Curbside pickup", "Wheelchair accessible"],
 "collected_at": "2025-01-15T14:19:12Z"
}

Warehouse validation examples

Below are lightweight checks to enforce shape and types at ingestion.

• Snowflake (validate required keys/types in VARIANT)

-- Assume a staging table with VARIANT column: payload
create or replace table raw_offers (payload variant);

-- Project typed columns with guards
select
 payload:entity::string as entity,
 payload:product_id::string as product_id,
 payload:price:value::float as price_value,
 payload:price:currency::string as price_currency,
 payload:availability:in_stock::boolean as in_stock,
 to_timestamp_ntz(payload:collected_at::string) as collected_at
from raw_offers
qualify entity = 'product_offer'
 and product_id is not null
 and price_value is not null
 and price_currency in ('USD','EUR','GBP');

-- Optional: reject bad rows into an errors table via tasks/streams
create or replace table offer_errors as
select payload, current_timestamp() as rejected_at
from raw_offers
where coalesce(payload:entity::string,'') <> 'product_offer'
 or payload:product_id is null
 or try_to_double(payload:price:value) is null;

• Databricks (Spark) schema enforcement with Delta

from pyspark.sql.types import *
from pyspark.sql.functions import col, from_json

product_offer_schema = StructType([
 StructField("entity", StringType(), False),
 StructField("product_id", StringType(), False),
 StructField("source_domain", StringType(), True),
 StructField("url", StringType(), True),
 StructField("title", StringType(), True),
 StructField("brand", StringType(), True),
 StructField("category_path", ArrayType(StringType()), True),
 StructField("price", StructType([
 StructField("value", DoubleType(), False),
 StructField("currency", StringType(), False),
 StructField("raw", StringType(), True)
 ]), False),
 StructField("availability", StructType([
 StructField("in_stock", BooleanType(), True),
 StructField("stock_level_signal", StringType(), True)
 ]), True),
 StructField("collected_at", StringType(), False)
])

# bronze: raw JSON lines

bronze = spark.read.json("s3://bucket/bronze/offers/")
silver = bronze.select(from_json(col("value"), product_offer_schema).alias("o")).select("o.*") \.filter((col("entity") == "product_offer") & col("product_id").isNotNull() & col("price.value").isNotNull())

silver.write.format("delta").mode("append").saveAsTable("retail.silver_product_offer")

# Add Delta CHECK constraints

spark.sql("""
ALTER TABLE retail.silver_product_offer
 ADD CONSTRAINT c_currency CHECK (price.currency IN ('USD','EUR','GBP'))
""")

• Databricks Expectations (streaming)

from delta.tables import *
from pyspark.databricks.sql.functions import *

expectations = {"valid_entity": "entity = 'product_offer'",
 "has_price": "price.value IS NOT NULL",
 "has_product_id": "product_id IS NOT NULL"}

(spark.readStream.table("retail.silver_product_offer_raw").withExpectations(expectations).writeStream.option("checkpointLocation", "s3://bucket/checkpoints/offers/").toTable("retail.silver_product_offer"))

For full schemas beyond these entities (inventory, PDP content, promotions), use the same patterns and align to your medallion targets (Bronze → Silver → Gold). If you need JSON Schema or Avro definitions, your Nimble team can provide managed contracts on request.

Reference table: destinations and patterns

Observability, cost control, and SLAs

• Centralized dashboards, per‑pipeline budgets, MoM/YoY usage, and exportable reports. Analytics & Management, Dashboard 2.0.

• Fully managed compliance and audit trails; enterprise access controls. Trust Center.

Quick-start checklists

• Databricks

• Choose medallion targets and Delta paths → enable Online Pipelines to S3/ADLS → register tables and schedule refresh → wire to notebooks/DBSQL/Power BI. Medallion primer.

• Snowflake

• Configure stage/bucket → auto-ingest or scheduled COPY via Snowpark → publish Gold views; optionally package as a Native App. Partners.

• Microsoft Fabric / Azure

• Land Parquet to OneLake or Blob → ADF pipelines to Warehouse/Lakehouse → model in Power BI semantic layer. Partners.

• Orchestra (low‑code)

• Define frequency, data type, destination; Orchestra handles scheduling/monitoring; Nimble pipelines deliver structured outputs. Nimble × Orchestra.

Why Nimble for integration

• Native connectors and governed outputs remove scraping/tooling overhead while fitting your existing lakehouse/warehouse and BI systems. Platform, Integrations.

• Real-time pipelines align to retail/CPG, finance, and AI agent use cases with compliance built in. Retail solutions, Alternative data.