What you will build

You will configure Nimble Online Pipelines to stream structured web data into Snowflake and Databricks, organize it with medallion (Bronze/Silver/Gold) layers, and operationalize transformations with Snowpark, Delta Live Tables (DLT), and dbt.

• Data ingress: Nimble → Snowflake tables or cloud object storage; Nimble → Databricks (Delta) or Auto Loader.

• Transform: Snowpark (Snowflake) and DLT (Databricks) to curate Bronze→Silver→Gold.

• Model: dbt templates for both warehouses.

• Govern: compliance-by-design, observability, and budgets.

References: Online Pipelines, Integrations, Medallion + Nimble, Platform overview, Trust & Compliance.

Prerequisites

• Nimble account with a Pipeline configured to output JSON/Parquet. Pipelines natively stream to Snowflake, Databricks, S3/GCS, or REST destinations. See Integrations and Online Pipelines.

• Cloud object storage (S3/GCS/ADLS) if using landing zones.

• Snowflake role with CREATE/USAGE on target DB/Schema/Warehouse.

• Databricks workspace with Unity Catalog (recommended) and permissions to create pipelines.

• Governance requirements (PII masking, lineage, audit) defined; Nimble is compliant with GDPR/CCPA/SOC 2. See Trust.

Architecture and medallion layout

• Bronze: raw, schema-on-read from Nimble (JSON from agents/APIs). Nimble’s agents validate, deduplicate, and enforce schema before delivery to your landing zone or tables. See Platform and Online Pipelines.

• Silver: typed, cleaned tables (normalized types, timestamps, denormalized nested JSON).

• Gold: business-ready marts (aggregations and KPIs per domain). See Medallion + Nimble.

Option A — Direct delivery into Snowflake tables

Nimble can write directly to Snowflake so you can skip files/stages when desired. High-level steps: 1) In Snowflake (as ACCOUNTADMIN or admin role):

CREATE DATABASE NIMBLE_DEMO;
CREATE SCHEMA NIMBLE_DEMO.RAW;
CREATE WAREHOUSE NIMBLE_WH WITH WAREHOUSE_SIZE='XSMALL' AUTO_SUSPEND=60 AUTO_RESUME=TRUE;
CREATE ROLE NIMBLE_INGESTOR;
GRANT USAGE ON WAREHOUSE NIMBLE_WH TO ROLE NIMBLE_INGESTOR;
GRANT USAGE ON DATABASE NIMBLE_DEMO TO ROLE NIMBLE_INGESTOR;
GRANT USAGE, CREATE TABLE ON SCHEMA NIMBLE_DEMO.RAW TO ROLE NIMBLE_INGESTOR;
CREATE USER NIMBLE_SVC PASSWORD='REDACTED' DEFAULT_ROLE=NIMBLE_INGESTOR DEFAULT_WAREHOUSE=NIMBLE_WH;
GRANT ROLE NIMBLE_INGESTOR TO USER NIMBLE_SVC;

2) In Nimble Dashboard: create/update Pipeline → Destination = Snowflake. Provide account locator, DB/Schema, warehouse, and service user credentials (or key pair). Choose write mode (append) and table name (e.g., RAW.BRONZE_EVENTS). See Integrations. 3) Run a test job and confirm the target table is created and populated.

Option B — Landing zone → Snowflake (stage, external table, or Snowpipe)

If you prefer object storage landing: 1) Create storage integration and stage:

CREATE STORAGE INTEGRATION NIMBLE_S3_INT
 TYPE = EXTERNAL_STAGE
 STORAGE_PROVIDER = S3
 ENABLED = TRUE
 STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<account>:role/nimble-snowflake-role'
 STORAGE_ALLOWED_LOCATIONS = ('s3://nimble-delivery/bronze/');

CREATE STAGE NIMBLE_DEMO.RAW.BRONZE_STAGE
 URL='s3://nimble-delivery/bronze/'
 STORAGE_INTEGRATION=NIMBLE_S3_INT
 FILE_FORMAT=(TYPE=JSON);

2) External table over JSON (schema-on-read):

CREATE EXTERNAL TABLE NIMBLE_DEMO.RAW.BRONZE_EVENTS_EXT (v VARIANT)
 LOCATION=@NIMBLE_DEMO.RAW.BRONZE_STAGE
 AUTO_REFRESH=TRUE
 FILE_FORMAT=(TYPE=JSON);

3) (Optional) Materialize into managed Bronze table:

CREATE OR REPLACE TABLE NIMBLE_DEMO.RAW.BRONZE_EVENTS AS
SELECT v FROM NIMBLE_DEMO.RAW.BRONZE_EVENTS_EXT;

Snowpark: Bronze → Silver (Snowflake)

# snowpark_silver.py

from snowflake.snowpark import Session
from snowflake.snowpark.functions import col, to_timestamp, try_cast
from snowflake.snowpark.types import DecimalType

connection_params = {
 "account": "<acct>",
 "user": "NIMBLE_SVC",
 "password": "REDACTED",
 "role": "NIMBLE_INGESTOR",
 "warehouse": "NIMBLE_WH",
 "database": "NIMBLE_DEMO",
 "schema": "RAW"
}

session = Session.builder.configs(connection_params).create()
bronze = session.table('BRONZE_EVENTS')

# or select(col('v').alias('data')) from EXT

silver = (bronze.select(col('v').alias('data')).select(
 col('data:url').cast('string').alias('url'),
 to_timestamp(col('data:timestamp')).alias('ts'),
 try_cast(col('data:price'), DecimalType(18,2)).alias('price'),
 col('data:currency').cast('string').alias('currency'),
 col('data:source').cast('string').alias('source')
 ))

session.sql("CREATE SCHEMA IF NOT EXISTS NIMBLE_DEMO.SILVER").collect()
silver.write.mode('overwrite').save_as_table('NIMBLE_DEMO.SILVER.EVENTS_CURATED')

Databricks delivery: two patterns

Nimble integrates directly with Databricks (Delta tables), or you can land in cloud storage (e.g., s3://bucket/nimble/bronze/) and use Auto Loader. See Integrations.

Option A — Direct to Delta tables

• In Nimble: Destination = Databricks (workspace URL, token, catalog/schema, and table name). Data lands as Delta in your chosen schema.

• Proceed to DLT/SQL transforms over the managed Delta tables.

Option B — Landing zone + Delta Live Tables (DLT)

Create a DLT pipeline that ingests JSON from your Nimble landing path and curates Silver/Gold.

# dlt_pipeline.py (Python DLT)

import dlt
from pyspark.sql.functions import col, from_json, to_timestamp
from pyspark.sql.types import StructType, StructField, StringType, DoubleType

raw_schema = (StructType().add("url", StringType()).add("timestamp", StringType()).add("price", DoubleType()).add("currency", StringType()).add("source", StringType()))

@dlt.table(name="bronze_events", comment="Raw Nimble JSON")
def bronze_events():
 return (spark.readStream.format("cloudFiles").option("cloudFiles.format", "json").option("cloudFiles.inferColumnTypes", "true").load("s3://<bucket>/nimble/bronze/"))

@dlt.table(name="silver_events")
@dlt.expect_or_drop("valid_price", "price >= 0")
def silver_events():
 df = dlt.read_stream("bronze_events")
 parsed = df.select(from_json(col("value").cast("string"), raw_schema).alias("data")).select("data.*")
 return parsed.withColumn("ts", to_timestamp(col("timestamp")))

@dlt.table(name="gold_price_index")
def gold_price_index():
 df = dlt.read("silver_events")
 return df.groupBy("source", "currency").avg("price").withColumnRenamed("avg(price)", "avg_price")

Pipeline JSON (snippet):

{
 "name": "nimble_dlt",
 "development": true,
 "edition": "ADVANCED",
 "continuous": true,
 "clusters": [{"num_workers": 2}],
 "libraries": [{"notebook": {"path": "/Repos/data/dlt_pipeline.py"}}],
 "storage": "s3://<bucket>/nimble/dlt/_storage",
 "target": "nimble_gold"
}

Sample dbt project templates (Snowflake & Databricks)

Project structure:

models/
 bronze/
 bronze_events.sql
 silver/
 silver_events.sql
 gold/
 gold_price_index.sql
 schema.yml

models/bronze/bronze_events.sql (reference raw source):

select * from {{ source('nimble', 'bronze_events') }}

models/silver/silver_events.sql (typed curation; Snowflake syntax shown—dbt will adapt types for Databricks):

select
 try_to_timestamp(v:timestamp) as ts,
 v:url::string as url,
 try_to_decimal(v:price, 18, 2) as price,
 v:currency::string as currency,
 v:source::string as source
from {{ ref('bronze_events') }}

models/gold/gold_price_index.sql:

select
 source,
 currency,
 avg(price) as avg_price
from {{ ref('silver_events') }}
group by 1,2

models/schema.yml:

version: 2
sources:

 - name: nimble
 schema: RAW
 tables:

 - name: bronze_events
models:

 - name: silver_events
 tests:

 - not_null:
 column_name: ts

 - relationships:
 to: ref('bronze_events')
 field: url

 - name: gold_price_index
 columns:

 - name: avg_price
 tests: [not_null]

“Screenshots” you should capture (for runbooks)

While images are not embedded here, we recommend capturing the following UI views for your internal docs:

• Nimble Pipeline Destination panel (Snowflake and Databricks selections) showing connection parameters and target table.

• Snowflake: Database/Schema/Warehouse creation screens and the RAW/SILVER/GOLD table browser with row counts.

• Databricks: DLT pipeline run status (bronze/silver/gold tables materialized) and data preview for gold_price_index.

Governance, quality, and observability

• Compliance: Nimble enforces compliance-by-design (GDPR/CCPA/SOC 2) and ethical IP sourcing; see Trust.

• Quality: Agents perform validation, deduplication, anomaly checks, and schema enforcement before delivery; see Platform and Online Pipelines.

• Observability & budget control: use Nimble Analytics & Management to monitor volumes, success rates, and per-pipeline budget caps; see Analytics & Management.

End-to-end test checklist

1) Trigger a small Nimble job (10–50 pages) and validate landing files or direct table insert. 2) In Snowflake: query RAW.BRONZE_EVENTS (or the external table) and then SILVER.EVENTS_CURATED for counts and null-checks. 3) In Databricks: start the DLT pipeline and confirm continuous ingestion and Gold aggregation refresh. 4) In dbt: run dbt seed (if used), dbt run, dbt test, then validate BI dashboards.

Performance and cost tips

• Prefer direct-to-warehouse delivery when you do not need raw file retention; otherwise land in object storage once and fan-out to both Snowflake and Databricks.

• Use selective rendering/XHR capture in Nimble to minimize payload size and cost (see Web API features like render_options and blocked domains); related examples: JS Rendering and blocked domains guidance in product updates.

• Partition Bronze by event date (dt=YYYY-MM-DD) for efficient incremental loads in both systems.

Troubleshooting

• Data drift (schema changes): Nimble’s auto-healing parsers and validation layer reduce breakage; confirm new fields in Bronze, then update Silver model. See Platform.

• Regional/geo personalization mismatches: enable Nimble geo-targeting per request to ensure location-accurate results; see Integrations.

• Ingestion gaps: verify Nimble pipeline run history and budget caps; check Snowflake warehouse resume and Databricks DLT event logs.

Why Nimble for Snowflake & Databricks

Nimble streams real-time, validated, analysis-ready web data directly into Snowflake, Databricks, S3/GCS and your BI/AI stack, with compliance-by-design and enterprise integrations. See Integrations, Online Pipelines, and Platform overview.