#dataengineering Development Tutorials & Tips | Roast Dev

100daysofcode 100daysofdevops 100pay 10mistakes 10yearworkanniversary 11 11labs 11tly 11ty 127001

InsightFlow Part 3: Building the Data Ingestion Layer with AWS Batch

InsightFlow GitHub RepoIn this post, we’ll explore how the data ingestion layer for the InsightFlow project was built using AWS Batch. This layer is responsible for fetching raw data from public sou...

28.04.2025 0 Read More

InsightFlow Part 4: Data Exploration & Understanding the Datasets

InsightFlow GitHub RepoBefore diving into building any data pipeline, a crucial first step is Data Exploration and Understanding. This involves getting familiar with the source datasets, their structu...

28.04.2025 0 Read More

InsightFlow Part 5: Designing the Data Model & Schema with dbt for InsightFlow

InsightFlow GitHub RepoIn this post, we’ll dive into how the data model and schema for the InsightFlow project were designed using dbt (Data Build Tool). This layer is critical for transforming raw ...

28.04.2025 0 Read More

InsightFlow Part 6: Implementing ETL Processes with AWS Glue for InsightFlow

InsightFlow GitHub RepoIn this post, we’ll explore how AWS Glue was used to implement the ETL (Extract, Transform, Load) processes for the InsightFlow project. AWS Glue provides a serverless, fully ...

28.04.2025 0 Read More

InsightFlow Part 7: Data Quality Implementation & Best Practices for InsightFlow

InsightFlow GitHub RepoIn this post, we’ll explore how data quality was implemented in the InsightFlow project and share best practices for ensuring reliable and accurate data pipelines. Data qualit...

28.04.2025 0 Read More

InsightFlow Part 8: Setting Up AWS Athena for Data Analysis in InsightFlow

InsightFlow GitHub RepoIn this post, we’ll explore how Amazon Athena was set up for querying and analyzing data in the InsightFlow project. Athena is a serverless, interactive query service that all...

28.04.2025 0 Read More

InsightFlow Part 9: Workflow Orchestration with Kestra

9. Workflow Orchestration with Kestra In modern data engineering, orchestrating workflows is a critical component of building reliable, scalable, and automated data pipelines. For the Insight...

29.04.2025 0 Read More

#34 50 Advanced SQL Queries Every Developer Should Know

SQL is a powerful language for managing and querying relational databases. While basic queries like SELECT, INSERT, UPDATE, and DELETE are essential, mastering advanced SQL techniques can significantl...

29.04.2025 0 Read More

Personal Picks: Data Product News (April 30, 2025)

Modern Data Stack Updates: April 2025 ※This article is an English translation of my original Japanese post.As a consultant specializing in the Modern Data Stack, I've noticed that there's a...

29.04.2025 0 Read More

Building and Deploying My First Python ETL Package to PyPI

Introduction Creating your own Python package is one of the most satisfying steps as a developer/engineer. For me, it all started with a goal to simplify ETL (Extract, Transform, Load) operat...

30.04.2025 0 Read More

InsightFlow Part 3: Building the Data Ingestion Layer with AWS Batch

InsightFlow Part 4: Data Exploration & Understanding the Datasets

InsightFlow Part 5: Designing the Data Model & Schema with dbt for InsightFlow

InsightFlow Part 6: Implementing ETL Processes with AWS Glue for InsightFlow

InsightFlow Part 7: Data Quality Implementation & Best Practices for InsightFlow

InsightFlow Part 8: Setting Up AWS Athena for Data Analysis in InsightFlow

InsightFlow Part 9: Workflow Orchestration with Kestra

#34 50 Advanced SQL Queries Every Developer Should Know

Personal Picks: Data Product News (April 30, 2025)

Building and Deploying My First Python ETL Package to PyPI

#reading

#popular