#dataengineering Development Tutorials & Tips | Roast Dev

100daysofcode 100daysofdevops 100pay 10mistakes 10yearworkanniversary 11 11labs 11tly 11ty 127001

Building Automated Data Reports from Supabase with GitHub Actions and R Markdown

Imagine being able to generate professional data reports automatically every week without lifting a finger. In this guide, I'll walk you through creating an automated reporting system that connects to...

05.04.2025 0 Read More

Distributed Model Serving Patterns

Intro The goal of every company is to make money and AI Models are seen as an integral part of the business. As machine learning models move from experimentation to production, serving them b...

05.04.2025 0 Read More

Time of YAML/JSON for data engineer

If you're tracking trends of data engineering, you could find out that there's a big shift in tools we use. Recently, Java(+Scala) and Python have been a go-to languages for data engineers, and it's s...

06.04.2025 0 Read More

Stock Data Extraction Using Apache Kafka

Overview This project utilizes Apache Kafka to extract stock data from the Polygon.io API and stores it in an Apache Cassandra database. It leverages Python for implementation and Confluent for Kafka ...

06.04.2025 0 Read More

🚀 Building an ETL Pipeline with Python to Scrape Internship Jobs and Load into Excel

Have you ever needed up-to-date job listings but struggled to find one clean source? In this project, I built a full ETL (Extract, Transform, Load) pipeline using Python to scrape internship job data ...

06.04.2025 0 Read More

Understanding Data Pipelines: The Backbone of Modern Data Systems

In today’s data-driven world, organizations are collecting vast amounts of data from various sources — websites, applications, sensors, APIs, and more. But raw data is rarely useful on its own. It...

06.04.2025 0 Read More

Building an Automated Weather Data Pipeline with Apache Kafka and Cassandra

This article creates an end-to-end weather data pipeline that collects weather data from multiple African cities, processes it through Apache Kafka, and stores it in a Cassandra database for analysis....

07.04.2025 0 Read More

How to treat secure data on lakehouse

In the modern data stack, the lakehouse has emerged as a hybrid solution that combines the scalability of a data lake with the transactional power of a data warehouse. But with this convergence comes ...

08.04.2025 0 Read More

Getting Values from Multiple Format Strings to Multiple Records — From SQL to SPL #16

Problem description & analysis: There are four strings separated into two parts by vertical bar, namely: ID|DUE_DATE, 80781|2026–12–01, VARCHAR2|DATE, |yyyy-mm-dd.Task: Now we need to...

08.04.2025 0 Read More

InsightFlow Part 1: Building an Integrated Retail & Economic Data Pipeline - Project Introduction

Introduction I'm thrilled to begin documenting my journey building "InsightFlow" - an end-to-end data engineering project that I've designed to showcase real-world data engineering skills whi...

09.04.2025 0 Read More

Building Automated Data Reports from Supabase with GitHub Actions and R Markdown

Distributed Model Serving Patterns

Time of YAML/JSON for data engineer

Stock Data Extraction Using Apache Kafka

🚀 Building an ETL Pipeline with Python to Scrape Internship Jobs and Load into Excel

Understanding Data Pipelines: The Backbone of Modern Data Systems

Building an Automated Weather Data Pipeline with Apache Kafka and Cassandra

How to treat secure data on lakehouse

Getting Values from Multiple Format Strings to Multiple Records — From SQL to SPL #16

InsightFlow Part 1: Building an Integrated Retail & Economic Data Pipeline - Project Introduction

#reading

#popular