Real-Time Streaming Data Pipeline

A solo data engineering project that ingests live cryptocurrency trade data from the Binance WebSocket API, streams it through Kafka, processes it with Spark Structured Streaming, lands it as Parquet in a MinIO (S3-compatible) data lake, orchestrates maintenance with Airflow, and serves it through Spark SQL to Superset dashboards.

Architecture

Binance WebSocket (BTC/USDT live trades)
        |
        v
  Kafka Producer (Python, confluent-kafka)
        |
        v
   Kafka Topic: raw_trades (3 partitions)
        |
        v
Spark Structured Streaming
   - parses JSON, casts types
   - Sink 1: raw trade events -> Parquet (partitioned by date)
   - Sink 2: 1-minute OHLC aggregation (tumbling window) -> Parquet
        |
        v
MinIO (S3-compatible data lake)
   s3a://streaming-lake/raw_trades
   s3a://streaming-lake/ohlc_1min
        |
        +-----------------------------+
        |                             |
        v                             v
Airflow (containerized)        Spark Thrift Server
  - parquet_compaction DAG       - exposes SQL tables over JDBC/Hive
  - runs every 10 minutes        - reads raw_trades + ohlc_1min_compacted
  - writes to a separate                |
    ohlc_1min_compacted path            v
    (never overwrites the        Superset (containerized)
    live streaming sink)           - live tick-level BTC price chart
                                    - 1-minute OHLC chart
                                    - dashboard auto-refreshes every 30s

Why this design

• Kafka decouples ingestion from processing. The producer doesn't know or care what consumes its data; Spark could be swapped for another consumer without touching the producer.
• Two independent streaming sinks off one parsed stream demonstrate both raw event storage (for reprocessing/audit) and real-time aggregation (for analytics), a common pattern in production streaming systems.
• MinIO instead of real S3 keeps the whole stack local and free to run, while using the same S3A APIs that would work unchanged against real AWS S3.
• Airflow orchestrates the batch/maintenance layer, not the stream itself. Streaming jobs are long-running processes; Airflow's role here is compaction of small Parquet files, run on a schedule, completely separate from the always-on streaming job.
• Compaction writes to a separate output path (ohlc_1min_compacted), never the live streaming sink. Structured Streaming maintains a _spark_metadata log to guarantee exactly-once output; a batch job overwriting that same path directly corrupts the log. This was discovered and fixed during development (see Lessons Learned below) and shapes the bronze/silver-style separation in the architecture.
• Spark Thrift Server, not a full Trino+Hive Metastore setup, for the query layer. It reuses the same Spark install already proven out in the streaming job, avoids standing up an extra metastore service, and still gives a real JDBC/Hive-compatible SQL interface that Superset (or any BI tool) can connect to like a normal database.

Tech stack

Component	Tool
Message broker	Apache Kafka (KRaft mode)
Stream processing	Spark Structured Streaming (PySpark 3.5.3)
Object storage	MinIO (S3-compatible)
File format	Parquet
Kafka producer client	confluent-kafka (librdkafka)
Orchestration	Apache Airflow 2.10.3 (containerized, custom image w/ Java + PySpark)
Query engine	Spark Thrift Server (JDBC/Hive protocol)
Dashboards	Apache Superset (containerized)

Project structure

producer/
  binance_producer.py       # Streams live BTC/USDT trades from Binance -> Kafka
  requirements.txt
spark/
  streaming_job.py          # Kafka -> Parquet on MinIO (raw + OHLC aggregation)
  compact_parquet.py        # Compacts small OHLC Parquet files into ohlc_1min_compacted
  verify_read.py            # Utility script to read back and sanity-check output
  start_thrift_server.sh    # Starts the Spark Thrift Server (SQL over the lake)
  sql/
    create_tables.sql       # Registers raw_trades + ohlc_1min_compacted as SQL tables
  requirements.txt
airflow/
  Dockerfile                # Custom Airflow image with Java + PySpark baked in
  dags/
    compaction_dag.py       # Triggers spark/compact_parquet.py every 10 minutes
superset/
  Dockerfile                # Custom Superset image with Hive/Postgres drivers
  superset_config.py        # Points Superset at its Postgres metadata DB
docker-compose.yml           # Kafka, Kafka UI, MinIO, Airflow, Superset, Postgres (x2)

Running it locally

Prerequisites: Docker Desktop, Python 3.11+, Java 17 (for PySpark), Homebrew (for librdkafka on Mac).

1. Start the infrastructure

docker compose up -d

This brings up Kafka, Kafka UI, MinIO, Postgres (x2, for Airflow and Superset), Airflow (webserver + scheduler), and Superset. First run will build the custom Airflow and Superset images, which takes a few minutes.

2. Create the Kafka topic (first run only)

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --create --topic raw_trades \
  --bootstrap-server localhost:9092 \
  --partitions 3 --replication-factor 1

3. Create the MinIO bucket

Create a streaming-lake bucket at http://localhost:9001 (login minioadmin / minioadmin).

4. Set up the Python environment

python3 -m venv venv
source venv/bin/activate
pip install -r producer/requirements.txt
pip install -r spark/requirements.txt

5. Start the producer (Terminal 1)

python producer/binance_producer.py

6. Start the Spark streaming job (Terminal 2)

spark-submit \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.3,org.apache.hadoop:hadoop-aws:3.3.4 \
  spark/streaming_job.py

7. Start the Spark Thrift Server (Terminal 3)

export SPARK_HOME=<path-to-venv>/lib/python3.14/site-packages/pyspark
./spark/start_thrift_server.sh

Then register the SQL tables (one-time, or after recreating the lake):

$SPARK_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -f spark/sql/create_tables.sql

8. Explore

• Kafka UI: http://localhost:8081 — inspect topics and messages
• MinIO Console: http://localhost:9001 — browse the raw Parquet files
• Airflow: http://localhost:8082 (admin / admin) — view/trigger the parquet_compaction DAG
• Superset: http://localhost:8088 (admin / admin) — the dashboard

To connect Superset to the Thrift Server: add a database with SQLAlchemy URI hive://host.docker.internal:10000/default (not localhost — Superset runs in Docker and needs the host-machine bridge address).

Status

• Kafka + MinIO infrastructure (Docker Compose)
• Live Binance trade producer
• Spark Structured Streaming: raw events + 1-minute OHLC aggregation to Parquet
• Airflow orchestration: containerized, custom image, Parquet compaction DAG every 10 minutes
• Spark Thrift Server + Superset dashboards with live and periodic charts, 30s auto-refresh
• Failure handling, monitoring, checkpointing hardening
• Demo video / GIF

Sample output

Raw trade event:

{
  "symbol": "BTCUSDT",
  "trade_id": 6490112780,
  "price": 62337.48,
  "quantity": 0.00341,
  "trade_time": 1783532490176,
  "is_buyer_maker": false,
  "ingested_at": "2026-07-08T17:41:30.022805+00:00"
}

1-minute OHLC candle: window_start: 2026-07-08 22:41:00 window_end: 2026-07-08 22:42:00 symbol: BTCUSDT open: 62334.0 high: 62344.0 low: 62290.0 close: 62290.0 volume: 37.09975

Lessons learned

A few non-obvious things this project surfaced, worth knowing if you're building something similar:

• Never batch-write into a live Structured Streaming sink's exact output path. Spark Structured Streaming maintains a _spark_metadata log to guarantee exactly-once semantics on its output directory. A separate batch job (like a compaction script) that overwrites that same path directly will desync the metadata log from the actual files on disk, breaking the streaming job. The fix: compact into a separate output location, never the original sink.
• Reading and overwriting the same path in one Spark job is a race condition. Spark's overwrite mode deletes the target directory's contents as part of the write commit, but reads are lazy — so a plain df = spark.read.parquet(path); df.write.mode("overwrite").parquet(path) can delete files before they've actually been read. Fix: .cache() and materialize (.count()) the DataFrame before writing.
• Bitnami's free Docker Hub images for Kafka were deprecated mid-project (moved behind a paid tier). The official apache/kafka image, which now ships with built-in KRaft support, was a clean drop-in replacement.
• Superset's official Docker image uses uv (not pip) to manage its virtualenv. Installing extra Python packages (e.g., Hive/Postgres drivers) requires uv pip install --python /app/.venv/bin/python3 ... rather than plain pip install, since the system pip binary doesn't target the app's actual runtime environment.
• Containers need host.docker.internal, not localhost, to reach services running on the host machine. This came up connecting Superset (in Docker) to the Spark Thrift Server (running directly on the host).