#pyspark Development Tutorials & Tips | Roast Dev

PySpark & Jupyter Notebooks Deployed On Kubernetes

PySpark is an interface for Apache Spark in Python. It not only allows you to write Spark applications using Python APIs, but also provides the PySpark shell for interactively analyzing your data in a...

05.05.2025 0 Read More

How to be Test Driven with Spark: Chapter 5: Leverage spark in a container

This goal of this tutorial is to provide a way to easily be test driven with spark on your local setup without using cloud resources.This is a series of tutorials and the initial chapters can be found...

15.03.2025 0 Read More

Study Notes 6.13-14: Kafka Streaming with Python & PySpark Structured Streaming with Kafka

1. Overview of Kafka Streaming with Python Purpose & Context: This session demonstrates how to implement Kafka streaming applications using Python, mirroring the producer–consumer examp...

18.03.2025 0 Read More

Apache Pyspark

It is a fast and general-purpose distributed computing system for big data processing. It provides an in-memory computation model, which significantly improves performance over traditional disk-based ...

31.03.2025 0 Read More

Feature Engineering para Embeddings com SparkML e MLFlow no Databricks Experiments

Hoje resolvi relembrar alguns conceitos de machine learning e entre eles a parte de vetorização de categorias para ter um dataset mais apto para deep learning (Redes neurais). Portanto neste artigo ...

06.04.2025 0 Read More

Weekly Updates - Apr 14, 2025

@here Hi developers! 👋 There is something happening every week at Couchbase, here are a few things you might have missed recently. 🎇 Couchbase with Apache Spark just got better! - The Couchba...

14.04.2025 0 Read More

Adding Audit Columns to Existing Tables: Comparing Approaches for Large Datasets

Adding Audit Columns to Existing Tables: Comparing Approaches for Large Datasets Introduction In data engineering, adding audit columns like bd_insert_dtm and bd_updated_dtm to track...

16.04.2025 0 Read More