This document presents the DC/OS method of installing Kafka with Portworx. Please refer to the Kafka and Zookeeper with Portworx on Kubernetes page if you are running Portworx on Kubernetes.

This guide will help you to install the Kafka service on your DCOS cluster backed by Portworx volumes for persistent storage.

The source code for these services can be found here: Portworx DCOS-Commons Frameworks

Note: This framework is only supported directly by Portworx, Inc.. Please contact support@portworx.com directly for any support issues related with using this framework.

Please make sure you have installed Portworx on DCOS before proceeding further.

Finding the service in the DCOS Universe/Catalog

You should see the Kafka service available in your universe as Portworx-Kafka

Kafka-PX in DCOS Universe


Default Install

If you want to use the defaults, you can now run the dcos command to install the service

 dcos package install --yes portworx-kafka

You can also click on the “Install” button on the WebUI next to the service and then click “Install Package”.

Advanced Install

If you want to modify the default, click on the “Install” button next to the package on the DCOS UI and then click on “Advanced Installation”

Here you have the option to change the service name, volume name, volume size, and provide any additional options that you want to pass to the docker volume driver. You can also configure other Kafka related parameters on this page including the number of broker nodes.

Kafka-PX install options

Click on “Review and Install” and then “Install” to start the installation of the service.

Install Status

Once you have started the install you can go to the Services page to monitor the status of the installation.

Kafka-PX on services page

If you click on the Kafka service you should be able to look at the status of the nodes being created.

Kafka-PX install started

When the Scheduler service as well as all the Kafka services are in Running (green) status, you should be ready to start using the Kafka service.

Kafka-PX install finished

If you check your Portworx cluster, you should see multiple volumes that were automatically created using the options provided during install, one for each of the Brokers.

Kafka-PX volumes

If you run the “dcos service” command you should see the portworx-kafka service in ACTIVE state with 3 running tasks

 dcos service
 NAME                           HOST             ACTIVE  TASKS  CPU   MEM      DISK   ID
 portworx-kafka      ip-10-0-3-116.ec2.internal   True     3    3.0  6144.0    0.0    66d598b0-2f90-4d0a-9567-8468a9979190-0038
 marathon                      True     2    2.0  2048.0    0.0    66d598b0-2f90-4d0a-9567-8468a9979190-0001
 metronome                     True     0    0.0   0.0      0.0    66d598b0-2f90-4d0a-9567-8468a9979190-0000
 portworx            ip-10-0-1-127.ec2.internal   True     4    3.3  4096.0    25.0   66d598b0-2f90-4d0a-9567-8468a9979190-0031
 portworx            ip-10-0-2-42.ec2.internal    True     3    1.2  3168.0  12288.0  66d598b0-2f90-4d0a-9567-8468a9979190-0032

Verify Setup

From the DCOS client; install the new command for portworx-kafka

  dcos package install portworx-kafka --cli

Find out all the kafka broker endpoints

  dcos portworx-kafka endpoints broker
   "address": [
  "dns": [
  "vip": "broker.portworx-kafka.l4lb.thisdcos.directory:9092"

Find out the zookeeper endpoint for the created kafka service

 dcos portworx-kafka endpoints zookeeper

Create a topic, from the DCOS client use dcos command to create a test topic test-one with replication set to three

dcos portworx-kafka topic create test-one --partitions 1 --replication 3
    "message": "Output: Created topic \"test-one\".\n"

Connect to the master node:

 dcos node ssh --master-proxy --leader

and launch a kafka client container:

docker run -it mesosphere/kafka-client

Produce a message and send it to all kafka brokers:

 echo "Hello, World." | ./kafka-console-producer.sh --broker-list kafka-2-broker.portworx-kafka.mesos:1025,kafka-0-broker.portworx-kafka.mesos:1025,kafka-1-broker.portworx-kafka.mesos:1029 --topic test-one

Consume the message

 ./kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-portworx-kafka --topic test-one --from-beginning
 Hello, World.

Last edited: Friday, Oct 28, 2022