#dataengineering Development Tutorials & Tips | Roast Dev

100daysofcode 100daysofdevops 100pay 10mistakes 10yearworkanniversary 11 11labs 11tly 11ty 127001

Real-Time Data Processing with PyFlink and Redpanda

Just leveled up my #DataEngineering skills by building real-time data pipelines with PyFlink and Redpanda! 🚀 Discovered how session windows can reveal hidden patterns in NYC taxi data that batch pr...

17.03.2025 0 Read More

Study Notes 6.1-2: Introduction to Stream Processing

1. Overview of Stream Processing Definition: Continuous, real-time processing of data as it arrives. Importance: Enables real-time analytics, event-driven applications, and prompt reaction...

18.03.2025 0 Read More

Study Notes 6.3-4: What is Kafka & Confluent Cloud

1. Introduction to Kafka in Stream Processing Context of Stream Processing: Stream processing involves continuously handling data as it flows from producers to consumers. Kafka is positio...

18.03.2025 0 Read More

Study Notes 6.5-6: Kafka Producer, Consumer & Configuration

1. Overview of Kafka Producer & Consumer Objective: Learn how to produce and consume messages in Apache Kafka programmatically using Java. Context: The video demonstrates both produc...

18.03.2025 0 Read More

Study Notes 6.7-10: Kafka Stream Basics, JOIN, Testing & Windowing

1. Overview Kafka Streams Basics Objective:Learn the fundamental building blocks of Kafka Streams by examining a simple example that: Reads messages from a Kafka topic. Processes them by grou...

18.03.2025 0 Read More

Study Notes 6.11-12: Kafka ksqlDB, Connect & Schema Registry

1. Overview of Kafka ksqlDB & Kafka Connect ksqlDB: ksqlDB is Kafka’s SQL-based stream processing engine. It allows you to perform real-time analysis and ad hoc queries on Kafka topics ...

18.03.2025 0 Read More

Study Notes 6.13-14: Kafka Streaming with Python & PySpark Structured Streaming with Kafka

1. Overview of Kafka Streaming with Python Purpose & Context: This session demonstrates how to implement Kafka streaming applications using Python, mirroring the producer–consumer examp...

18.03.2025 0 Read More

Study Notes 6.15: Kafka & Flink Streaming

1. Introduction to Kafka Streaming with PyFlink Streaming Data Processing: Involves the continuous ingestion, processing, and movement of data in real time. Critical for use cases where i...

18.03.2025 0 Read More

Calculate the Hierarchy of Recursive References — From SQL to SPL #11

Problem description & analysis: A certain table in the MS SQL database has a multi-level self-association structure, where the second field parent node ID is a foreign key pointing to the...

18.03.2025 0 Read More

Data architecture stack

Hey all,My current stack is data coming as an eventing system to Pub/Sub (GCP). This contains streaming event data at scale of thousands every 5 minutes.I also have an intelligence table in BigQuery w...

19.03.2025 0 Read More

Real-Time Data Processing with PyFlink and Redpanda

Study Notes 6.1-2: Introduction to Stream Processing

Study Notes 6.3-4: What is Kafka & Confluent Cloud

Study Notes 6.5-6: Kafka Producer, Consumer & Configuration

Study Notes 6.7-10: Kafka Stream Basics, JOIN, Testing & Windowing

Study Notes 6.11-12: Kafka ksqlDB, Connect & Schema Registry

Study Notes 6.13-14: Kafka Streaming with Python & PySpark Structured Streaming with Kafka

Study Notes 6.15: Kafka & Flink Streaming

Calculate the Hierarchy of Recursive References — From SQL to SPL #11

Data architecture stack

#reading

#popular