DATA LAKE PIPELINE

Bronze / Silver / Gold architecture with Delta Lake. Drop files into data/raw_intake/ and run the pipeline. ACID transactions, time travel, full audit trail. Portable on thumbdrive — no Spark required.

DuckDB Size

Tables

Total Rows

Uncompressed Est.

Last Sweep

▶ Run Sweep

Click for commands

WEEKLY DATA SWEEP

Run from the SAS project root directory:

python scripts/weekly_sweep.py

Report only — shows what sources are due for refresh

python scripts/weekly_sweep.py --run

Fetch all due sources, update registry dates, regenerate DB stats

python scripts/weekly_sweep.py --force

Re-fetch everything regardless of schedule

After sweep: python scripts/lake_pipeline.py full to push new files through Bronze/Silver/Gold

RAW INTAKE

files waiting

▶

BRONZE

raw tables

▶

SILVER

cleaned tables

▶

GOLD

in DuckDB

TABLE STATUS

Table	Bronze	Silver	Gold	Rows	Version	Flags

ARCHITECTURE

BRONZE — Delta Lake

Raw data, as-received. Schema-on-read. Every file versioned with ACID transactions. Full audit trail (source file, timestamp). Time travel to any previous version. We never lose the original.

SILVER — Delta Lake

Cleaned: nulls handled, duplicates removed, whitespace trimmed. Typed: dates parsed, column names standardized. Validated: quality flags surfaced. Rollback-ready if cleaning was wrong. User reviews and approves before promotion.

GOLD — DuckDB

Analytics-ready. Powers all 4 dashboard tabs. Fast columnar queries. Single portable file (sas.duckdb). Also exported as Parquet for interoperability. What the dashboards see.

PIPELINE COMMANDS

python scripts/lake_pipeline.py full

Run entire pipeline: ingest -> clean -> load to DuckDB

python scripts/lake_pipeline.py ingest

Bronze: ingest files from data/raw_intake/

python scripts/lake_pipeline.py promote-all

Silver: clean and validate all Bronze tables

python scripts/lake_pipeline.py gold-all

Gold: push all Silver tables to DuckDB

python scripts/lake_pipeline.py status

Show full lake status with row counts and versions

python scripts/lake_pipeline.py rollback <table> <version>

Time travel: restore a table to any previous version

python scripts/lake_pipeline.py history <table>

Show version history for a table

DATA SOURCES & REFRESH SCHEDULE

Source	Provider	Coverage	Frequency	Refresh	Key?	Last Run	Action

QUICK START

Step 1: Drop CSV, Excel, JSON, or Parquet files into data/raw_intake/
Step 2: Run python scripts/lake_pipeline.py full
Step 3: Check this page for status, quality flags, and row counts
Step 4: Data is now in DuckDB and powering all dashboards

If something went wrong: python scripts/lake_pipeline.py rollback <table> <version>
Bronze layer always preserves the original. Silver can be re-promoted. Gold can be re-pushed.

Technology: delta-rs (standalone Delta Lake, no Spark/JVM). DuckDB (embedded OLAP). All portable on thumbdrive. | Audit: Every pipeline action logged to lake/logs/pipeline_log.jsonl with timestamps. | 72-Hour Assignment: Alaska drops files -> you drop them in raw_intake/ -> run full pipeline -> minutes later everything is profiled, cleaned, and in DuckDB.