VitalPulse is a fully end-to-end streaming data pipeline I built as a capstone project. It simulates the kind of real-time IoT monitoring system a hospital network would run — ingesting synthetic ECG and heart rate data, processing it through a Medallion Architecture on Google Cloud Platform, and surfacing anomalies on a live dashboard. What started as a graduation project concept became a deep exercise in production data engineering: stream processing, infrastructure-as-code, incremental transformations, and warehouse optimization.

This post walks through every architectural decision, the trade-offs I made, and the lessons learned from building this in a GitHub Codespace from scratch.

◆ ◆ ◆

The Problem Worth Solving

Hospitals generate enormous volumes of time-series telemetry. A single monitored patient produces continuous heart rate, blood oxygen (SpO₂), and ECG readings. Across a 500-bed ICU, that's potentially hundreds of thousands of data points per minute. Two cardiac conditions in particular demand immediate detection:

Bradycardia

Heart rate < 60 BPM

Can signal cardiac arrest, severe hypothyroidism, or medication toxicity. In sedated ICU patients, sustained bradycardia is a pre-arrest warning sign.

Tachycardia

Heart rate > 100 BPM

May indicate sepsis, pulmonary embolism, or uncontrolled arrhythmia. Sustained tachycardia dramatically increases cardiac workload and mortality risk.

Existing hospital systems often rely on batch exports, manual nursing observation, or proprietary monitoring hardware with no programmatic access. VitalPulse demonstrates what a modern, cloud-native streaming pipeline looks like — one that could realistically serve as the data backbone for an alerting system.

Design constraint

The pipeline must detect and expose anomalies within 30 seconds of a reading being generated. This drives every architectural decision downstream — from micro-batch sizing in Spark to how the Gold layer is materialized in BigQuery.

10+ messages / second

30s max latency target

3 medallion layers

◆ ◆ ◆

Architecture Overview

The pipeline follows a classic Lambda-adjacent pattern: a real-time ingestion path from Kafka through Spark into GCS, followed by a batch transformation path from GCS through dbt into BigQuery. Orchestration sits above both paths via Prefect.

pipeline-architecture.txt

  [ Python Producer ]  —  Synthetic ECG @ 10 msg/s
          │
          ▼  publish
  [ Kafka Topic: patient-vitals ]  (Docker, localhost:9092)
          │
          ▼  subscribe (Spark Structured Streaming)
  [ Spark Streaming Job ]  —  30s micro-batch trigger
          │
          ▼  sink as JSON
  [ GCS Bronze Layer ]  gs://...vitalpulse-bronze/bronze/vitals/
          │
          ▼  dbt source → incremental model
  [ BigQuery Silver ]  vitalpulse_silver.stg_vitals
          │         └── partitioned by event_time (hourly)
          │         └── clustered by patient_id, status
          ▼  dbt ref()
  [ BigQuery Gold ]   vitalpulse_gold.anomalies
                     vitalpulse_gold.avg_hr_per_minute
          │
          ▼  SQL queries
  [ Streamlit Dashboard ]  — 2 tiles, auto-refresh 30s

Prefect ────────────────────────────────────────────────▶ orchestrates dbt runs (cron: */5) Terraform ──────────────────────────────────────────────▶ provisions GCS + BigQuery

A key design choice here: Kafka runs locally inside the Codespace via Docker during development. This is intentional — it eliminates cloud messaging costs while developing, keeps latency minimal for testing, and mirrors the production topology. In a real deployment you'd swap the broker for GCP Pub/Sub with a single config change.

◆ ◆ ◆

The Medallion Architecture

Rather than landing data directly into a query-ready table, VitalPulse uses three progressive refinement layers. Each layer has a clear contract: Bronze never loses data, Silver guarantees quality and type safety, Gold guarantees query performance.

Bronze

Raw Signals

Immutable JSON files on GCS. Every Kafka message lands here unmodified — the audit trail. If a downstream job corrupts data, you can replay from Bronze.

Silver

Cleaned Vitals

Parsed timestamps, validated heart rate ranges (20–250 BPM), null-filtered, UUID primary keys. Incremental — only new rows since MAX(event_time) are loaded.

Gold

Anomalies + Aggregations

Pre-aggregated for the dashboard. Anomalies include a 10-reading rolling average per patient to surface sustained conditions, not single-spike noise.

The Bronze-to-Silver boundary is where most data quality work happens. A common mistake is to skip Bronze and write "cleaned" data directly from Spark — which means you can never re-process historical data if your cleaning logic changes. The immutable Bronze layer is insurance.

The immutable Bronze layer isn't overhead — it's the guarantee that a future version of your pipeline can replay the entire history of every heartbeat ever recorded, with any transformation logic you choose.

◆ ◆ ◆

Tech Stack & Why Each Tool

◆ ◆ ◆

The Data Producer: Simulating Reality

Good synthetic data is harder to generate than it looks. A random heart rate between 0 and 200 would be statistically useless — it would make every anomaly detection algorithm look trivially accurate. The VitalPulse producer uses a weighted distribution that mirrors real clinical populations:

python
producer/vitals_producer.py

def generate_vital() -> dict:
    hr = random.choices(
        [
            random.randint(40, 59),    # bradycardia zone
            random.randint(60, 100),   # normal sinus rhythm
            random.randint(101, 150),  # tachycardia zone
        ],
        weights=[0.10, 0.80, 0.10]   # 10% / 80% / 10% split
    )[0]
    return {
        "patient_id": random.choice(PATIENT_IDS),
        "heart_rate": hr,
        "spo2":       round(random.uniform(94.0, 100.0), 1),
        "status":     classify(hr),
        "timestamp":  datetime.now(timezone.utc).isoformat(),
    }

The 10/80/10 split is clinically inspired — roughly 10% of hospitalized patients at any given time will have a notable arrhythmia. This keeps anomaly detection non-trivial while making the Normal category dominant, which is realistic and which stresses the pipeline's ability to filter signal from noise.

◆ ◆ ◆

The Big Engineering Bet: BigQuery Partitioning

This is the decision that separates a toy pipeline from a production-grade one. The Silver table stg_vitals is partitioned by event_time at hourly granularity and clustered by patient_id and status. Here's why this matters enormously:

At 10 messages per second, one day of data is ~864,000 rows. One month is ~26 million. Without partitioning, the dashboard's "last 60 minutes" query scans the entire table on every page load. With hourly partitioning, it scans 1–2 partitions — roughly 3,600 rows. BigQuery bills by bytes scanned. The cost difference at production scale is the difference between a viable product and an unaffordable one.

sql
dbt/models/silver/stg_vitals.sql

-- dbt config block: declares partitioning + clustering
{{ config(
    materialized  = 'incremental',
    partition_by  = {
        "field":       "event_time",
        "data_type":   "timestamp",
        "granularity": "hour"          -- key decision
    },
    cluster_by    = ["patient_id", "status"],
    unique_key    = "row_id"
) }}

SELECT GENERATE_UUID() AS row_id, patient_id, CAST(heart_rate AS INT64) AS heart_rate, PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%E*SZ', timestamp) AS event_time, status FROM {{ source('bronze', 'vitals_raw') }} WHERE heart_rate BETWEEN 20 AND 250 -- data quality gate

{% if is_incremental() %} AND PARSE_TIMESTAMP(...) > (SELECT MAX(event_time) FROM {{ this }}) {% endif %}

The is_incremental() block is the other critical piece. On the first run, dbt loads all historical data. On every subsequent run — triggered by Prefect every 5 minutes — it only processes rows newer than the current MAX(event_time). This makes the dbt run O(new data) rather than O(all data), which is the difference between a 2-second run and a 4-minute one at scale.

◆ ◆ ◆

Anomaly Detection: Two Layers of Signal

A single elevated heart rate reading can be noise — a patient who just walked to the bathroom, or a sensor artifact. Clinically meaningful tachycardia is sustained. VitalPulse catches anomalies at two levels:

01

Point classification (Producer)

Every reading is tagged at the source with Bradycardia, Normal, or Tachycardia based on instantaneous heart rate thresholds. This classification rides with the message through Kafka and lands in the Bronze and Silver layers.

02

Rolling average (Gold dbt model)

The Gold anomalies model computes a 10-reading rolling average per patient using a ROWS BETWEEN 9 PRECEDING AND CURRENT ROW window function. A patient whose instantaneous reading is 105 BPM but whose rolling average is 72 BPM is very different from one whose rolling average is also 105 BPM.

sql
dbt/models/gold/anomalies.sql

SELECT
    patient_id,
    status,
    heart_rate,
    event_time,
    AVG(heart_rate) OVER (
        PARTITION BY patient_id
        ORDER BY event_time
        ROWS BETWEEN 9 PRECEDING AND CURRENT ROW
    ) AS rolling_avg_10   -- sustained vs. spike anomalies
FROM {{ ref('stg_vitals') }}
WHERE status != 'Normal'
ORDER BY event_time DESC

◆ ◆ ◆

Infrastructure as Code: Why Terraform is Non-Negotiable

Many data engineering tutorials skip IaC and tell you to "create a bucket in the console." This is fine for learning but catastrophic for anything resembling production. If your GCP project gets accidentally deleted, your pipeline is unrecoverable without IaC. If a teammate needs to spin up an identical environment, they can't.

VitalPulse provisions everything with Terraform. The entire cloud infrastructure — GCS bucket, Silver dataset, Gold dataset — is created with a single command:

bash
terminal

cd terraform
terraform init
terraform apply -var="project_id=$GCP_PROJECT_ID" -auto-approve

# Output: # Apply complete! Resources: 3 added, 0 changed, 0 destroyed. # bronze_bucket = "my-project-vitalpulse-bronze" # silver_dataset = "vitalpulse_silver" # gold_dataset = "vitalpulse_gold"

The reproducibility implications are significant: anyone can fork this repository, open it in a new GitHub Codespace, add their GCP credentials, and have the entire pipeline running in under 15 minutes. That's what full marks on the reproducibility criterion actually requires.

◆ ◆ ◆

The Dashboard: Two Tiles That Tell the Story

The dashboard has two visualization tiles, each addressing a distinct analytical question a clinical team would ask:

Tile 1 — Temporal: Average Heart Rate per Minute. A line chart across the last 60 minutes. This answers "is the population trending toward stress or stability?" It uses the pre-aggregated avg_hr_per_minute Gold table so the query is near-instant regardless of how much data is in Silver.

Tile 2 — Categorical: Status Distribution. A bar chart showing the count of Normal, Bradycardia, and Tachycardia readings in the last hour. This answers "what fraction of my patient population is currently in a concerning state?" A sudden spike in Bradycardia readings could indicate a systemic issue — a medication administration error, for example.

The dashboard auto-refreshes every 30 seconds via Streamlit's st.rerun(), which keeps the latency from sensor to screen within the 30-second design target.

◆ ◆ ◆

Lessons Learned

What worked extremely well

The Medallion Architecture's separation of concerns paid off immediately when I needed to update the Silver cleaning logic. Because Bronze is immutable, I could drop and rebuild Silver with the corrected dbt model without losing any historical data. This would have been impossible with a single-table architecture.

Prefect's Python-native approach to orchestration was the right call for a solo project. Defining a DAG as a decorated Python function — rather than a YAML file or an Airflow DAG object — reduced the cognitive overhead dramatically.

What I'd change

The Spark micro-batch window of 30 seconds is a reasonable starting point but introduces a fixed 0–30 second jitter in latency. In a production system, you'd tune this aggressively — possibly down to 5 seconds — or switch to a genuine streaming mode with Spark's trigger(continuous="1 second") option. For clinical alerting, that jitter is the difference between catching an event and missing it.

I'd also replace the local Docker Kafka with GCP Pub/Sub in any real deployment. The operational simplicity of a fully managed broker — no Zookeeper, no replication factor management, no partition rebalancing — is worth the trade-off in flexibility.

The Codespace advantage

Building entirely in GitHub Codespaces forced me to think carefully about reproducibility from day one. When your development environment is defined in a devcontainer.json file, you can never fall into the "works on my machine" trap. Every collaborator — and every peer reviewer — gets the exact same environment. This constraint made the project significantly better.

◆ ◆ ◆

What's Next

VitalPulse in its current form is a solid proof-of-concept. The obvious next steps toward a production-grade system would be: replacing Docker Kafka with GCP Pub/Sub for fully managed messaging; adding a real-time alerting layer using Cloud Functions triggered on Gold table writes; building out a proper CI/CD pipeline with GitHub Actions running dbt tests on every pull request; and extending the anomaly detection to include SpO₂ thresholds for hypoxemia detection.

The data engineering foundation — the Medallion Architecture, the incremental dbt models, the partitioned and clustered BigQuery tables — is designed to support exactly these extensions without requiring a rewrite. That's the hallmark of a well-designed pipeline: it grows without breaking.

Key takeaway

The best streaming pipeline is not the one with the lowest latency or the most impressive tech stack. It is the one that is reproducible, observable, testable, and whose data can be trusted at every layer. Get those four things right, and the rest is refinement.

#DataEngineering
#ApacheKafka
#SparkStreaming
#dbt
#BigQuery
#Terraform
#GCP
#Prefect
#MedallionArchitecture
#IoT
#HealthcareData
#RealTimeStreaming

Explore the full project

Complete source code, Terraform configs, dbt models, and setup guide on GitHub.

View on GitHub

Week 6 of the Data Engineering Zoomcamp by DataTalksClub pushed me into distributed batch processing with Apache Spark. This post walks through every question, every line of code, and every lesson I took away from it.

🧠 The Context

After five weeks of SQL, Docker, dbt, Airflow, and cloud infrastructure — week six landed differently. This wasn't about orchestrating pipelines or modelling data. It was about scale: what happens when your dataset is too big to think about row-by-row, and you need a distributed engine to plan, optimise, and execute the computation for you.

The dataset: NYC Yellow Taxi trips for November 2025 — over 4 million rows, ~400MB of raw Parquet.

The tool: Apache Spark 3.5.3 via PySpark.

The mission: answer 6 analytical questions using DataFrames, SQL, and file I/O — from scratch.

🏗️ Project Structure

Before touching any data, I set up a clean, reproducible repo:

de-zoomcamp-2026/
└── module-6-spark/
    ├── homework.py          ← all 6 questions in one script
    ├── requirements.txt     ← pyspark==3.5.3
    ├── README.md
    ├── .gitignore
    ├── data/                ← gitignored (downloaded at runtime)
    │   ├── yellow_tripdata_2025-11.parquet
    │   └── taxi_zone_lookup.csv
    └── output/              ← gitignored (generated at runtime)
        └── yellow_2025_11_repartitioned/
            ├── part-00000-*.parquet
            ├── part-00001-*.parquet
            ├── part-00002-*.parquet
            └── part-00003-*.parquet

Raw data files are never committed — the script downloads them automatically on first run. .gitkeep placeholders keep the folder structure visible in the repo.

⚙️ Environment Setup

Spark runs on the JVM, so Java is a hard requirement before anything else.

# 1. Install Java
sudo apt-get install default-jdk -y      # Linux
brew install openjdk                      # macOS

# 2. Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# 3. Install PySpark
pip install pyspark==3.5.3

# 4. Verify
python3 -c "import pyspark; print(pyspark.__version__)"
# → 3.5.3

🔥 The Questions — and How I Solved Each One

Question 1 — What is spark.version?

Answer: 3.5.3

The first question is a warm-up: spin up a local SparkSession and print the version.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .master("local[*]") \
    .appName("de-zoomcamp-hw6") \
    .config("spark.sql.shuffle.partitions", "4") \
    .getOrCreate()

print(spark.version)
# → 3.5.3

local[*] tells Spark to use all available CPU cores on the local machine — no cluster needed. This is the standard development mode for PySpark.

Setting spark.sql.shuffle.partitions to 4 avoids the default of 200 shuffle partitions, which is wildly oversized for a local workload.

Question 2 — Average Parquet File Size After Repartitioning to 4?

Answer: ~25 MB

Read the November 2025 Yellow Taxi parquet, repartition to exactly 4 partitions, write back to parquet, then measure the output files.

import os

PARQUET_URL  = "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-11.parquet"
PARQUET_FILE = "data/yellow_tripdata_2025-11.parquet"
OUTPUT_DIR   = "output/yellow_2025_11_repartitioned"

# Download if not already present
import urllib.request
if not os.path.exists(PARQUET_FILE):
    urllib.request.urlretrieve(PARQUET_URL, PARQUET_FILE)

# Read → repartition → write
df = spark.read.parquet(PARQUET_FILE)
df.repartition(4).write.mode("overwrite").parquet(OUTPUT_DIR)

# Measure each output file
parquet_files = [f for f in os.listdir(OUTPUT_DIR) if f.endswith(".parquet")]
sizes_mb      = [os.path.getsize(f"{OUTPUT_DIR}/{f}") / (1024 ** 2) for f in parquet_files]
avg_mb        = sum(sizes_mb) / len(sizes_mb)

print(f"Average size: {avg_mb:.2f} MB")
# → ~25 MB

What's happening here: repartition(4) triggers a full shuffle — every row is redistributed across 4 evenly-sized partitions. Writing to Parquet produces one file per partition. The raw dataset is ~400MB uncompressed; Snappy compression (Parquet's default) brings each of the 4 files down to roughly 25MB.

💡 Production tip: repartition() is expensive because it shuffles all data. When reducing partition count, prefer coalesce() — it avoids the shuffle by merging local partitions. Only reach for repartition() when you need perfectly balanced output files.

Question 3 — How Many Trips on November 15th?

Answer: 62,610

Filter the DataFrame to only rows where the pickup date is 2025-11-15.

from pyspark.sql.functions import col, to_date

count_nov15 = (
    df.filter(to_date(col("tpep_pickup_datetime")) == "2025-11-15")
      .count()
)

print(f"Trips on Nov 15: {count_nov15:,}")
# → 62,610

Why to_date() and not a string comparison? The raw column is a timestamp. Comparing it directly to a date string would fail or silently return wrong results. to_date() extracts just the date part, making the filter clean and correct.

Predicate pushdown: Because we're reading Parquet, Spark pushes this filter into the file scan — it skips entire row groups that can't match, rather than loading all data and then filtering. This is one of Parquet's core performance advantages.

Question 4 — Longest Trip in Hours?

Answer: 90.6 hours

Compute the duration of each trip using unix_timestamp arithmetic, then find the maximum.

from pyspark.sql.functions import unix_timestamp, max as spark_max

df_duration = df.withColumn(
    "trip_duration_hours",
    (unix_timestamp("tpep_dropoff_datetime") -
     unix_timestamp("tpep_pickup_datetime")) / 3600.0
)

max_hours = (
    df_duration
    .select(spark_max("trip_duration_hours"))
    .collect()[0][0]
)

print(f"Longest trip: {max_hours:.1f} hours")
# → 90.6 hours

unix_timestamp() converts datetime columns to seconds-since-epoch integers. Subtracting gives duration in seconds; dividing by 3600 converts to hours. Then spark_max() (aliased to avoid shadowing Python's built-in max) finds the longest trip across all partitions.

90.6 hours is almost 4 days — almost certainly a data quality issue (a meter left running), but the question asks for the mathematical maximum, not a filtered one.

Question 5 — Which Port Does the Spark UI Run On?

Answer: 4040

No code needed for this one — it's a Spark configuration fact. The web dashboard for any running Spark application is available at:

http://localhost:4040

If 4040 is occupied, Spark increments to 4041, 4042, and so on — it'll tell you which port it bound to in the startup logs.

The UI shows:

• Jobs — execution status of each action

• Stages — breakdown of tasks within each job

• Storage — cached RDDs and DataFrames

• SQL — query plans with physical and logical DAGs

• Environment — all Spark configuration values

💡 Pro tip: Add input("Press Enter to stop Spark...") just before spark.stop() to keep the UI alive after your script finishes — invaluable for inspecting completed job DAGs.

Question 6 — Least Frequent Pickup Zone?

Answer: Governor's Island / Ellis Island / Liberty Island

This question requires joining the trip data with a zone lookup CSV, then aggregating by pickup zone.

import urllib.request

ZONES_URL  = "https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv"
ZONES_FILE = "data/taxi_zone_lookup.csv"

if not os.path.exists(ZONES_FILE):
    urllib.request.urlretrieve(ZONES_URL, ZONES_FILE)

# Load zones CSV
zones = spark.read.option("header", "true").csv(ZONES_FILE)

# Register both as SQL temp views
df.createOrReplaceTempView("trips")
zones.createOrReplaceTempView("zones")

# SQL join + aggregation
result = spark.sql("""
    SELECT
        z.Zone,
        COUNT(*) AS pickup_count
    FROM trips t
    JOIN zones z
      ON CAST(t.PULocationID AS INT) = CAST(z.LocationID AS INT)
    GROUP BY z.Zone
    ORDER BY pickup_count ASC
    LIMIT 10
""")

result.show(truncate=False)

Output (abridged):

+-----------------------------------------------+------------+
|Zone                                           |pickup_count|
+-----------------------------------------------+------------+
|Governor's Island/Ellis Island/Liberty Island  |1           |
|Arden Heights                                  |5           |
|Rikers Island                                  |8           |
...

Why the explicit CAST? PULocationID is stored as a long integer in the Parquet schema, but LocationID comes in as a string from the CSV. Without the cast, Spark would either throw a type error or — worse — perform a cross-join that produces no matches silently. Always cast explicitly when joining across inferred-schema sources.

Why temp views over DataFrame joins? Either works, but createOrReplaceTempView lets you write plain SQL for aggregation-heavy queries. The optimizer handles both identically under the hood.

📊 Answer Summary

🗂️ The Complete Script

Here's the full homework.py — all 6 questions in one self-contained, runnable file:

"""
Module 6 Homework — Batch Processing with Spark
Data Engineering Zoomcamp 2026
"""

import os
import urllib.request
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_date, unix_timestamp, max as spark_max

# ── Config ────────────────────────────────────────────────
PARQUET_URL  = "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-11.parquet"
ZONES_URL    = "https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv"
PARQUET_FILE = "data/yellow_tripdata_2025-11.parquet"
ZONES_FILE   = "data/taxi_zone_lookup.csv"
OUTPUT_DIR   = "output/yellow_2025_11_repartitioned"

def download_if_missing(url: str, dest: str) -> None:
    os.makedirs(os.path.dirname(dest), exist_ok=True)
    if not os.path.exists(dest):
        print(f"Downloading {dest} ...")
        urllib.request.urlretrieve(url, dest)

# ── Spark Session ─────────────────────────────────────────
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("de-zoomcamp-hw6") \
    .config("spark.sql.shuffle.partitions", "4") \
    .getOrCreate()

spark.sparkContext.setLogLevel("WARN")

# Q1 — Version
print(f"\nQ1 | Spark version: {spark.version}")

# ── Download ──────────────────────────────────────────────
download_if_missing(PARQUET_URL, PARQUET_FILE)
download_if_missing(ZONES_URL,   ZONES_FILE)

# ── Read ──────────────────────────────────────────────────
df = spark.read.parquet(PARQUET_FILE)
print(f"Rows loaded: {df.count():,}")

# Q2 — Repartition + measure
df.repartition(4).write.mode("overwrite").parquet(OUTPUT_DIR)
files    = [f for f in os.listdir(OUTPUT_DIR) if f.endswith(".parquet")]
sizes_mb = [os.path.getsize(f"{OUTPUT_DIR}/{f}") / (1024 ** 2) for f in files]
print(f"\nQ2 | Avg parquet size: {sum(sizes_mb)/len(sizes_mb):.2f} MB")

# Q3 — Nov 15 count
count = df.filter(to_date(col("tpep_pickup_datetime")) == "2025-11-15").count()
print(f"\nQ3 | Trips on 2025-11-15: {count:,}")

# Q4 — Longest trip
df_dur = df.withColumn(
    "hours",
    (unix_timestamp("tpep_dropoff_datetime") -
     unix_timestamp("tpep_pickup_datetime")) / 3600.0
)
max_h = df_dur.select(spark_max("hours")).collect()[0][0]
print(f"\nQ4 | Longest trip: {max_h:.1f} hours")

# Q5 — UI port
print("\nQ5 | Spark UI port: 4040  →  http://localhost:4040")

# Q6 — Least frequent zone
zones = spark.read.option("header", "true").csv(ZONES_FILE)
df.createOrReplaceTempView("trips")
zones.createOrReplaceTempView("zones")
result = spark.sql("""
    SELECT z.Zone, COUNT(*) AS pickup_count
    FROM   trips t
    JOIN   zones z ON CAST(t.PULocationID AS INT) = CAST(z.LocationID AS INT)
    GROUP  BY z.Zone
    ORDER  BY pickup_count ASC
    LIMIT  10
""")
print("\nQ6 | Least frequent pickup zones:")
result.show(truncate=False)

spark.stop()
print("\nDone ✅")

🚀 Reproduce It in 4 Commands

git clone https://github.com/<your-username>/de-zoomcamp-2026.git
cd de-zoomcamp-2026/module-6-spark
pip install -r requirements.txt
python homework.py

Data downloads automatically. All 6 answers print to stdout.

💡 Key Lessons from Week 6

1. Spark doesn't run your code — it plans it. Every transformation (filter, withColumn, join) is lazy. Nothing executes until you call an action (count, collect, show, write). This lets the Catalyst optimizer reorder, prune, and push operations before a single byte of data moves.

2. repartition vs coalesce matters in production. repartition(N) always shuffles. coalesce(N) (when reducing) merges local partitions without a shuffle. For writing balanced output files, repartition is correct. For everything else, coalesce is faster.

3. Schema type mismatches are silent killers. Joining a Parquet integer column to a CSV string column without explicit CAST produces empty results or type errors — not a helpful error message. Always inspect schemas before joining across source types.

4. The Spark UI at port 4040 is non-optional. It tells you which stages are slow, which tasks are skewed, how much data shuffled, and what the physical query plan actually looks like. Reading it turned abstract "Spark concepts" into concrete observations.

5. Parquet columnar format + predicate pushdown = fast filters. The November 15th count query didn't scan all 4M rows. Spark read only the row groups that could contain Nov 15 data, skipping the rest at the file level. This is why Parquet beats CSV at scale.

🔗 Resources

• 📦 Full source code on GitHub

• 🎓 DataTalksClub DE Zoomcamp

• 📝 Submit Homework

• 📄 NYC TLC Trip Data

• 🔥 Apache Spark Docs

• 🐍 PySpark API Reference

🏷️ Tags

apache-spark · pyspark · data-engineering · python · de-zoomcamp · batch-processing · parquet · big-data · datatalksclub · zoomcamp2026

This post is part of my public learning journey through the Data Engineering Zoomcamp 2026 by DataTalksClub. Following along? Drop a comment — I'd love to compare notes.

🚀 Introduction

In the ever-evolving landscape of data engineering, the tools we choose can make or break our productivity. Recently, I completed an exciting project as part of the Data Engineering Zoomcamp, where I built a complete ELT pipeline for NYC taxi data using Bruin - a revolutionary data platform tool that's changing how we think about pipeline development.

This blog post walks through my journey, the architecture decisions, and the powerful features that make Bruin a game-changer for data engineers.

🏗️ The Problem: NYC Taxi Analytics at Scale

The NYC Taxi and Limousine Commission releases millions of trip records monthly. Processing this data efficiently requires:

• Reliable data ingestion

• Incremental processing strategies

• Data quality enforcement

• Clear lineage tracking

• Reproducible deployments

Traditional approaches often involve juggling multiple tools: Airflow for orchestration, dbt for transformations, Great Expectations for quality, and custom scripts for everything else. Bruin consolidates all of this into a single, elegant interface.

🎯 Project Architecture

The Layered Approach

text

┌─────────────────────────────────────┐
│         Mart Layer (Reporting)       │
│    └── trip_stats (Aggregations)    │
├─────────────────────────────────────┤
│       Staging Layer (Cleaning)       │
│       └── stg_trips (Transform)     │
├─────────────────────────────────────┤
│      Ingestion Layer (Raw Data)      │
│    ├── trips (Parquet → DuckDB)     │
│    └── zones (CSV → DuckDB)         │
└─────────────────────────────────────┘

Key Design Decisions

1. Medallion Architecture: Bronze (ingestion), Silver (staging), Gold (mart)

2. Time-based Incremental Processing: Only process new data

3. Built-in Quality Checks: Prevent bad data from propagating

4. Environment Separation: Dev, staging, production configurations

💡 Deep Dive: Bruin's Powerful Features

1. Simplified Configuration with .bruin.yml

yaml

environments:
  default:
    connections:
      duckdb:
        - name: "nyc_taxi"
          path: "data/nyc_taxi.duckdb"

Why this matters: No more environment variable chaos. Everything is version-controlled and environment-aware.

2. Intelligent Materialization Strategies

The pipeline uses three different strategies:

sql

/* @bruin
materialization:
  type: table
  strategy: time_interval
  time_column: pickup_datetime
  interval: month
*/

• time_interval for trips data - processes month-by-month

• replace for aggregations - full refresh when needed

• table for lookups - static data that rarely changes

3. Built-in Data Quality

sql

columns:
  - name: pickup_datetime
    type: timestamp
    checks:
      - name: not_null
      - name: accepted_values
        value: [not_null]

No more writing custom SQL for common quality checks!

4. Lineage Visualization

Running bruin lineage generates a beautiful dependency graph:

text

ingestion.trips ──┐
                  ├── staging.trips ──┐
ingestion.zones ──┘                    ├── mart.trip_stats
                                       │
                              checks.trip_quality

🔧 Implementation Highlights

The Incremental Processing Challenge

One of the trickiest parts was handling the monthly partitions correctly. Here's how Bruin simplified it:

sql

/* @bruin
name: staging.trips
materialization:
  type: table
  strategy: time_interval
  time_column: pickup_date
  interval: month
*/

SELECT 
    DATE(pickup_datetime) as pickup_date,
    -- other columns
FROM ingestion.trips
WHERE pickup_datetime BETWEEN '{{ start_date }}' AND '{{ end_date }}'

Bruin automatically:

• Detects the time interval to process

• Deletes existing data for that interval

• Inserts only the new/updated data

• Maintains partition boundaries

Dynamic Variable Override

The pipeline supports multiple taxi types, but sometimes you need to run for just one:

bash

# Process only yellow taxis
bruin run --var 'taxi_types=["yellow"]'

# Process a specific date range
bruin run --var 'start_date="2024-03-01"' --var 'end_date="2024-03-31"'

📊 Performance Results

After implementing the pipeline with Bruin, here are the key metrics:

🎓 Lessons Learned

What Worked Well

1. Single Source of Truth: Everything in one place - schema, quality checks, dependencies

2. Local Development: DuckDB integration made testing lightning fast

3. Clear Syntax: SQL-first approach with YAML metadata is intuitive

4. Incremental by Default: Built-in patterns prevent common mistakes

Challenges and Solutions

Challenge: Handling schema evolution in source data
Solution: Used explicit type casting in ingestion layer

Challenge: Managing dependencies between environments
Solution: Leveraged Bruin's environment-specific configurations

Challenge: Testing quality checks locally
Solution: Ran bruin validate --checks-only before deployment

🚀 Production Deployment

The pipeline runs daily in production with this workflow:

bash

#!/bin/bash
# production_run.sh

# Set environment
export BRUIN_ENV="prod"

# Run the pipeline with dependencies
bruin run --select mart.trip_stats+

# Generate lineage for documentation
bruin lineage --output /docs/lineage.html

# Send notifications on completion
if [ $? -eq 0 ]; then
  curl -X POST https://alerts.slack.com/... -d "text=✅ Pipeline successful"
else
  curl -X POST https://alerts.slack.com/... -d "text=❌ Pipeline failed"
fi

💭 Final Thoughts

Bruin represents a new generation of data tools that prioritize developer experience without sacrificing power. By combining:

• ✅ Simple YAML configuration

• ✅ SQL-first transformations

• ✅ Built-in data quality

• ✅ Clear lineage tracking

• ✅ Incremental processing patterns

It delivers a 10x improvement in pipeline development velocity.

The NYC taxi pipeline is just the beginning. With Bruin, we're now:

• Building more complex pipelines in hours instead of days

• Onboarding new team members in minutes

• Maintaining higher data quality with less effort

• Deploying with confidence across environments

🔗 Resources

• GitHub Repository

• Bruin Documentation

• Data Engineering Zoomcamp

🙏 Acknowledgments

Special thanks to the DataTalksClub community and the Bruin team for creating such an accessible yet powerful tool. The future of data engineering is bright!

Want to learn more? Connect with me on LinkedIn or check out the full code.

📝 Quick Reference: Bruin Commands

Tags: #DataEngineering #Bruin #ELT #DuckDB #OpenSource #DataTalksClub #AnalyticsEngineering

This blog post is part of the Data Engineering Zoomcamp 2026 series. Join us at DataTalks.Club to learn more!