Youtube API Project

📊 YouTube API – Data Warehouse & Analytics Solution

This repository demonstrates a complete data pipeline that extracts data from the YouTube Data API, models it using the Medallion Architecture, and delivers business-ready insights via Grafana dashboards.

📦 Project Summary

This project implements a modern analytics pipeline with:

• Medallion Architecture: Structured into Bronze, Silver, and Gold layers for scalable data processing.
• ETL Workflows: Automated extraction, transformation, and loading using Apache Airflow.
• Data Modeling: Dimensional modeling in PostgreSQL for optimized querying.
• Dashboards: Real-time reporting using Grafana, powered by SQL.

🧰 Tech Stack

• PostgreSQL – Central data warehouse
• Apache Airflow – Workflow orchestration
• Grafana – Real-time data visualization
• Linux VM – Compute environment for pipeline execution
• Python – API ingestion & transformation logic

🎯 Project Objectives

Build a production-ready analytics solution to analyze YouTube channel and video performance:

• Source structured data from the YouTube Data API
• Clean, validate, and model for business intelligence
• Persist historical metrics (views, likes, etc.) for trend analysis
• Deliver actionable insights via dashboards and SQL queries

🗃️ Data Architecture (Medallion Model)

This project follows a Bronze → Silver → Gold pipeline:

🔹 Bronze Layer

Raw ingestion from the YouTube API (JSON format)

🔸 Silver Layer

Cleaned, validated, and structured data (see data flow and model below)

• Data Flow
  

• Data Model
  

🟡 Gold Layer

Aggregated data used to generate KPIs and dashboards in Grafana

• Visualization Sample

📈 BI Use Cases

Dashboards and SQL queries answer key questions such as:

• What are the top-performing videos per channel?
• How is each channel performing over time?
• What are the daily trends for views and engagement?

📁 Repository Structure

├── README.md
├── channel_lists.py
├── channel_overview.py
├── channel_videos.py
├── __pycache__/                        # Compiled Python files
├── project_files/
│   ├── Architecture/                  # Draw.io and PNG files for architecture
│   └── ddl_update_scripts/           # SQL DDLs and procedures
│       ├── dim_channels.sql
│       ├── dim_videos.sql
│       ├── fct_subscribers_views_video_count.sql
│       └── fct_video_statistics.sql
└── requirements.txt                   # Python dependencies

🔗 Access the Code

Browse the full codebase here