Splittable DoFns in Python: A Hands-On Workshop

This repository contains the code samples and exercises used for the workshop "Splittable DoFns in Python" at Beam Summit 2022.

🎥 Resources

• Workshop Recording: Watch on YouTube
• Presentation Slides: Download Slides (PDF)
• Official Session Page: Beam Summit 2022 Session

🌿 Repository Structure

This repository is organized into two main branches:

• main: Template branch containing the exercises. Start here and write your code.
• solution: Reference branch containing the complete solutions. Use this to verify your work.

---

🛠️ Setup & Dependencies

1. Python Version: Ensure you have Python 3.10, 3.11, 3.12, or 3.13 installed.
2. Install Requirements:

   pip install -r requirements.txt

3. Kafka (Optional): To run the Kafka examples, you will need a running Kafka cluster. Instructions to set up a local Kafka cluster using Minikube are provided below.

---

🚀 Running the Synthetic Pipelines

These pipelines use synthetic data generators to demonstrate Splittable DoFns without external dependencies.

1. Batch Pipeline

Demonstrates reading files in parallel by chunks.

• Run command:

  python my_batch_pipeline.py

• Implementation file: mydofns/synthetic_sdfn_batch.py

2. Streaming Pipeline

Demonstrates a streaming source with multiple partitions.

• Run command:

  python my_streaming_synth_pipeline.py

• Implementation file: mydofns/synthetic_sdfn_streaming.py
• Note: You can configure the number of partitions (default is 4) in mydofns/synthetic_sdfn_streaming.py (around line 62).

---

🎡 Running the Kafka Pipelines

To run these examples, you need to set up a Kafka cluster and populate a topic.

🐳 Running Kafka Locally with Minikube

Follow these steps to set up Kafka in a local Minikube cluster:

1. Install Minikube: Follow the Minikube Start Guide.
2. Configure Access: Set up an alias for convenience:

   alias k=kubectl

3. Create Namespace:

   k create namespace kafka

4. Install Strimzi Kafka Operator:

   k create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

5. Deploy Ephemeral Kafka Cluster:

   k apply -f manifests/kafka-cluster.yaml -n kafka

6. Retrieve Kafka Bootstrap Server Details:
   • Get Node Port:

     k get service my-cluster-kafka-external-bootstrap -o=jsonpath='{.spec.ports[0].nodePort}{"\n"}' -n kafka

   • Get Minikube IP:

     k get node minikube -o=jsonpath='{range .status.addresses[*]}{.type}{"\t"}{.address}{"\n"}'

   • Define the bootstrap server environment variable (replace with your IP and Port):

     export BOOTSTRAP="<MINIKUBE_IP>:<NODE_PORT>"

📝 Topic Creation & Data Population

Use the helper script kafka_single_client.py to manage the Kafka topic:

1. Create Topic:

   python kafka_single_client.py --bootstrap $BOOTSTRAP --create

2. Produce Test Data:

   python kafka_single_client.py --bootstrap $BOOTSTRAP

3. Verify Data (Consume):

   python kafka_single_client.py --consumer --bootstrap $BOOTSTRAP

🏃 Running the Kafka Pipeline

• Run command:

  python my_streaming_kafka_pipeline.py --bootstrap $BOOTSTRAP

• Implementation file: mydofns/kafka_sdfn_streaming.py
• Note: Ensure the partition count matches the one used in kafka_single_client.py.