Mastering Kafka: Essential UIs, Best Practices, Use Cases, and Certifications

In this article, I decided to compile some important points about Kafka, such as Kafka itself, graphical user interface (UI) administration tools, material for Devs / QA, best practices for optimization, use cases, books, and certifications. The goal is to provide some tips without going into detail on each item.

Kafka + UI

To set up a quick environment for development, you can use Docker. Below is the link to the Docker Compose file to install and configure Kafka, Zookeeper, and the visual UI for Kafka.

Kafka + Zookeeper is the Kafka server where the broker handles all orchestration, and Zookeeper stores Kafka data. More details can be found on the Apache Kafka website. Kafka UI is a project developed by Provectus, which can be found here. It's important to note that this UI is more developer-focused and not intended for production.

Best Practices for Optimization

A very comprehensive article with various tips for optimizing Kafka, worth reading.

Use Cases

• Activity tracking: to record user activities, for example, this was LinkedIn's case where Kafka originated;

• Messaging: an application can send messages for users to receive notifications, for instance;

• Metrics and Logs: collecting metrics and logs from applications and sending them to a database;

• Commit log: listening to changes happening in the database, serving as replication and monitoring.

• Stream processing: transforming large volumes of data.

For Devs

Confluent is a platform that offers services and content for Kafka and also has several free courses, a good reference.

Another very interesting site is the Kafka Docker Playground, which has various information, such as connectors, security, Confluent Cloud examples, etc.

For QA

I found this material interesting as it has information related to Kafka focused on test engineers (QA).

Book

This book is a very comprehensive guide, explaining various topics about Kafka, and it is free on the Confluent website.

Certifications

Confluent is a scalable, reliable, and secure event streaming platform developed by the creators of Apache Kafka. Confluent offers several certifications.

• Confluent Certified Developer for Apache Kafka (CCDAK) is aimed at developers and solution architects. It covers Confluent and Apache Kafka® with a particular focus on the platform knowledge necessary to develop applications that work with Kafka.

• Confluent Certified Administrator for Apache Kafka® (CCAAK) is aimed at those who manage Kafka cluster environments. It covers the most critical work activities that an Apache Kafka administrator performs, from configuration and deployment to monitoring, management, and supporting Kafka clusters.