Automated Data Pipelines with sqlite, sqlalchemy and airflow

---

Overview

This repo contains code for automating data pipelines and data engineering tasks. The purpose of this repo is to build automated data pipelines that -

• Query data from REST APIs using requests.
• Using json and pandas tp clean and transform the raw data.
• Storing data into relational databases using sqlite and sqlalchemy.
• Update databases on a schedule using cron and airflow.

This repo also contains notes and code from the Real Python SQLite and SQLAlchemy course.

---

Data Sources

• Iowa Environmental Mesonet
• NASDAQ100 from the St. Louis Federal Reserve (FRED)