Apache Kafka Setup with Docker + Python Producer & Consumer

This project sets up Apache Kafka using Docker containers and demonstrates a Producer-Consumer workflow in Python.
The producer (producer.py) generates transaction data, while the consumer (app.py) consumes the data from a Kafka topic.

📌 Prerequisites

Before you start, ensure you have the following installed:

• Docker
• Docker Compose
• Python 3.x
• Python Kafka client library:

pip install kafka-python

📂 Project Structure

.
├── docker-compose.yml   # Docker Compose file for Kafka + ZooKeeper
├── producer.py          # Kafka producer that generates transactions
├── app.py               # Kafka consumer that reads transactions
└── README.md            # Setup guide

🚀 Kafka Setup Instructions

1. Navigate to Project Directory

Make sure you are in the folder that contains the docker-compose.yml file:

cd /path/to/your/project

2. Start Kafka & ZooKeeper Containers

docker-compose up -d

• -d runs the containers in the background.

• This starts:

  • ZooKeeper (required by Kafka)
  • Kafka Broker

3. Verify Running Containers

docker ps

You should see containers for kafka and zookeeper.

📡 Kafka Topic Setup

1. Access Kafka Container

docker exec -it <kafka-container-name> bash

2. Create a Topic

bin/kafka-topics.sh --create \
  --topic transactions \
  --bootstrap-server localhost:9092 \
  --partitions 1 \
  --replication-factor 1

3. List Topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

You should see transactions in the list.

📝 Running Producer & Consumer

1. Run Producer (producer.py)

This script generates transaction data and sends it to the Kafka topic transactions.

python producer.py

You should see messages being sent to Kafka.

2. Run Consumer (app.py)

This script consumes messages from the transactions topic and displays/uses them.

python app.py

You should see messages being received in real time.

🛑 Stopping Services

Stop Kafka and ZooKeeper:

docker-compose down

Restart later with:

docker-compose up -d

⚙️ Notes

• Kafka default ports:

  • 9092 → Kafka broker
  • 2181 → ZooKeeper

• Both producer.py and app.py are configured to use the topic transactions.
  If you change the topic name or Kafka connection settings, update the scripts accordingly.

• Logs can be checked using:

docker logs <kafka-container-name>

✅ With this setup, you can:

• Run Kafka & ZooKeeper inside Docker
• Produce transaction data via producer.py
• Consume transaction data via app.py