04-analytics-engineering

Module 4: Analytics Engineering

Goal: Transforming the data loaded in DWH into Analytical Views developing a dbt project.

Prerequisites

The prerequisites depend on which setup path you choose:

For Cloud Setup (BigQuery):

• Completed Module 3: Data Warehouse with:
  • A GCP project with BigQuery enabled
  • Service account with BigQuery permissions
  • NYC taxi data loaded into BigQuery (yellow and green taxi data for 2019-2020)

For Local Setup (DuckDB):

• No prerequisites! The local setup guide will walk you through downloading and loading the data.

Note

This module focuses on yellow and green taxi data (2019-2020). While Module 3 may have included FHV data, it is not used in this dbt project.

Setting up your environment

Choose your setup path:

🏠 Local Setup

• Stack: DuckDB + dbt Core
• Cost: Free
• → Get Started

☁️ Cloud Setup

• Stack: BigQuery + dbt Cloud
• Cost: Free tier available (dbt Cloud Developer), BigQuery costs vary
• Requires: Completed Module 3 with BigQuery data
• → Get Started

Content

Introduction to Analytics Engineering

Introduction to data modeling

What is dbt?

Differences between dbt Core and dbt Cloud

Project Setup

Alternative A	Alternative B
BigQuery + dbt Platform	DuckDB + dbt core

dbt Course

dbt Project Structure	dbt Sources	dbt Models	Seeds and Macros

dbt Tests	Documentation	dbt Packages	dbt Commands

Troubleshooting

• DuckDB Troubleshooting Guide — If you're getting OOM errors during dbt build with DuckDB

Extra resources

Note

If you find the videos above overwhelming, we recommend completing the dbt Fundamentals course and then rewatching the module. It provides a solid foundation for all the key concepts you need in this module.

SQL refresher

The homework for this module focuses heavily on window functions and CTEs. If you need a refresher on these topics, you can refer to these notes.

• SQL refresher

Homework

• 2026 Homework

Community notes

Did you take notes? You can share them here

• Slides used in previous years
• Notes by Alvaro Navas
• Sandy's DE learning blog
• Notes by Victor Padilha
• Marcos Torregrosa's blog (spanish)
• Notes by froukje
• Notes by Alain Boisvert
• Setting up Prefect with dbt by Vera
• Blog by Xia He-Bleinagel
• Setting up DBT with BigQuery by Tofag
• Blog post by Dewi Oktaviani
• Notes from Vincenzo Galante
• Notes from Balaji
• Notes by Linda
• 2024 - Videos transcript week4
• Blog Post by Jonah Oliver
• 2025 Notes by Manuel Guerra
• 2025 Notes by Horeb SEIDOU
• 2025 Notes by Daniel Lachner
• 2026 Notes by Sharad K. Gupta
• Analytical Engineering overview
• 2026 Notes about dbt | dbt + Duckdb setup using Docker by Khanh Nguyen
• Add your notes here (above this line)