#dezoomcamp Development Tutorials & Tips | Roast Dev

Real-Time Data Processing with PyFlink and Redpanda

Just leveled up my #DataEngineering skills by building real-time data pipelines with PyFlink and Redpanda! 🚀 Discovered how session windows can reveal hidden patterns in NYC taxi data that batch pr...

17.03.2025 0 Read More

Study Notes 6.1-2: Introduction to Stream Processing

1. Overview of Stream Processing Definition: Continuous, real-time processing of data as it arrives. Importance: Enables real-time analytics, event-driven applications, and prompt reaction...

18.03.2025 0 Read More

Study Notes 6.3-4: What is Kafka & Confluent Cloud

1. Introduction to Kafka in Stream Processing Context of Stream Processing: Stream processing involves continuously handling data as it flows from producers to consumers. Kafka is positio...

18.03.2025 0 Read More

Study Notes 6.5-6: Kafka Producer, Consumer & Configuration

1. Overview of Kafka Producer & Consumer Objective: Learn how to produce and consume messages in Apache Kafka programmatically using Java. Context: The video demonstrates both produc...

18.03.2025 0 Read More

Study Notes 6.7-10: Kafka Stream Basics, JOIN, Testing & Windowing

1. Overview Kafka Streams Basics Objective:Learn the fundamental building blocks of Kafka Streams by examining a simple example that: Reads messages from a Kafka topic. Processes them by grou...

18.03.2025 0 Read More

Study Notes 6.11-12: Kafka ksqlDB, Connect & Schema Registry

1. Overview of Kafka ksqlDB & Kafka Connect ksqlDB: ksqlDB is Kafka’s SQL-based stream processing engine. It allows you to perform real-time analysis and ad hoc queries on Kafka topics ...

18.03.2025 0 Read More

Study Notes 6.13-14: Kafka Streaming with Python & PySpark Structured Streaming with Kafka

1. Overview of Kafka Streaming with Python Purpose & Context: This session demonstrates how to implement Kafka streaming applications using Python, mirroring the producer–consumer examp...

18.03.2025 0 Read More

Study Notes 6.15: Kafka & Flink Streaming

1. Introduction to Kafka Streaming with PyFlink Streaming Data Processing: Involves the continuous ingestion, processing, and movement of data in real time. Critical for use cases where i...

18.03.2025 0 Read More

InsightFlow Part 1: Building an Integrated Retail & Economic Data Pipeline - Project Introduction

Introduction I'm thrilled to begin documenting my journey building "InsightFlow" - an end-to-end data engineering project that I've designed to showcase real-world data engineering skills whi...

09.04.2025 0 Read More

InsightFlow Part 2: Setting Up the Cloud Infrastructure with Terraform

In this post, I’ll walk you through how I set up the cloud infrastructure for my project, InsightFlow, using Terraform. InsightFlow is a data engineering project that integrates retail and fuel pric...

28.04.2025 0 Read More

Real-Time Data Processing with PyFlink and Redpanda

Study Notes 6.1-2: Introduction to Stream Processing

Study Notes 6.3-4: What is Kafka & Confluent Cloud

Study Notes 6.5-6: Kafka Producer, Consumer & Configuration

Study Notes 6.7-10: Kafka Stream Basics, JOIN, Testing & Windowing

Study Notes 6.11-12: Kafka ksqlDB, Connect & Schema Registry

Study Notes 6.13-14: Kafka Streaming with Python & PySpark Structured Streaming with Kafka

Study Notes 6.15: Kafka & Flink Streaming

InsightFlow Part 1: Building an Integrated Retail & Economic Data Pipeline - Project Introduction

InsightFlow Part 2: Setting Up the Cloud Infrastructure with Terraform

#reading

#popular