Kafka on Kubernetes

Bits you need to run Kafka cluster on Kubernetes. It is based on Kafka
version 0.9.x.

Kafka requires access to a zookeeper cluster. We run zookeeper in Kubernetes
too, take a look here: https://github.com/UKHomeOffice/docker-zookeeper.

Configuration

• ZOOKEEPER_CONNECT - a comma separated list of zookeeper node:port pairs.
  Default: localhost:2181.
• BROKER_ID - Kafka broker.id. Default: unset. If unset, Kafka will get
  zookeeper to allocate one.
• ADVERTISED_HOSTNAME - what address to advertise to other brokers and
  producers/consumers. Default: unset. If unset, Kafka uses the hostname.
• NUM_PARTITIONS - Number of partitions by default. Default: 1.
• DEFAULT_REPLICATION_FACTOR - Default replication factor. Default: unset.

Deployment

By default, if you don't specify any parameters, kafka will start in
a single broker mode.

Deploying onto a Kubernetes cluster is fairly easy. There are example
kubernetes controller and service files in kube/ directory.

Deploy Services

There is no strict ordering how you deploy the resources, let's start with
services first:

$ kubectl create -f kube/kafka-service.yaml
$ kubectl create -f kube/kafka-1-service.yaml
$ kubectl create -f kube/kafka-2-service.yaml
$ kubectl create -f kube/kafka-3-service.yaml

Deploy Replication Controllers

$ kubectl create -f kube/kafka-1-controller.yaml
$ kubectl create -f kube/kafka-2-controller.yaml
$ kubectl create -f kube/kafka-3-controller.yaml

Known Caveats

By default there is no data persistence. So be aware that if you delete more
than one replication controller or more than one pod, you may end up losing
data.