How Can I Create My Own Rhel-based Kafka (4.0.0) Image And Modify The Server.properties At Startup With Docker-compose.yml

by ADMIN 123 views

In this comprehensive guide, we will explore the process of creating a custom Red Hat Enterprise Linux (RHEL)-based Kafka 4.0.0 image and modifying the server.properties file at startup using Docker Compose. The challenge we aim to address is Kafka's inherent behavior of not directly utilizing environment variables passed in from Docker Compose, instead relying on its default server.properties configuration. This article will provide a detailed walkthrough of how to overcome this limitation and tailor your Kafka configuration to your specific needs.

Understanding the Challenge: Kafka and Environment Variables

Kafka, by default, is designed to load its configuration from the server.properties file. While this approach works well in many scenarios, it poses a challenge when deploying Kafka in a containerized environment like Docker. The typical expectation is that environment variables defined in Docker Compose or the Dockerfile can be used to configure the application. However, Kafka does not natively support this mechanism. This means that any configurations you attempt to pass via environment variables will be ignored, and Kafka will revert to its default settings.

This behavior can be particularly problematic when you need to configure Kafka dynamically, such as setting broker IDs, listeners, or ZooKeeper connection strings based on the environment in which the container is running. Manually modifying the server.properties file each time you deploy a new container is not a scalable or maintainable solution. Therefore, a more robust approach is needed to ensure that Kafka can be configured dynamically using environment variables.

To effectively manage Kafka's configuration in a Docker environment, we need to implement a strategy that allows us to modify the server.properties file at startup based on the environment variables provided. This involves creating a custom Docker image that includes a script to read these environment variables and update the server.properties file accordingly. This approach ensures that Kafka is configured correctly each time the container is started, without requiring manual intervention.

Prerequisites

Before we begin, ensure you have the following prerequisites in place:

  • Docker: Docker installed and running on your system.
  • Docker Compose: Docker Compose installed on your system.
  • Red Hat Enterprise Linux (RHEL) Subscription: A valid RHEL subscription to access the necessary packages.
  • Basic Docker and Docker Compose Knowledge: Familiarity with Docker concepts and Docker Compose syntax.
  • Text Editor: A text editor for creating and modifying files.

Having these prerequisites in place will ensure a smooth process as we proceed with creating the custom Kafka image and configuring it with Docker Compose.

Step 1: Create a Dockerfile

The first step is to create a Dockerfile that will serve as the blueprint for our custom Kafka image. This Dockerfile will include the necessary instructions to install Kafka, configure it, and set up the environment for running Kafka within a container. We will start with a RHEL base image and then add the required dependencies and configurations.

Create a new directory for your Kafka project and create a file named Dockerfile inside it. Add the following content to the Dockerfile:

FROM registry.access.redhat.com/ubi8/ubi-init:latest

ENV KAFKA_VERSION=4.0.0
ENV SCALA_VERSION=2.13
ENV KAFKA_DOWNLOAD_URL=https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz



RUN yum -y update && 

yum -y install java-11-openjdk-devel wget tar gzip vim && 

yum clean all



RUN wget {KAFKA_DOWNLOAD_URL} -O /tmp/kafka.tgz && \
    tar -xzf /tmp/kafka.tgz -C /opt && \
    mv /opt/kafka_{SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka && 

rm /tmp/kafka.tgz



ENV KAFKA_HOME=/opt/kafka
ENV PATH=PATH:PATH:KAFKA_HOME/bin



RUN mkdir -p /var/lib/kafka/data



COPY docker-entrypoint.sh /



RUN chmod +x /docker-entrypoint.sh



EXPOSE 9092



ENTRYPOINT ["/docker-entrypoint.sh"]

Explanation of the Dockerfile:

  • FROM registry.access.redhat.com/ubi8/ubi-init:latest: This line specifies the base image for our Docker image, which is the RHEL 8 Universal Base Image with systemd support. This ensures that we have a stable and reliable foundation for our Kafka installation.
  • ENV KAFKA_VERSION=4.0.0, ENV SCALA_VERSION=2.13, ENV KAFKA_DOWNLOAD_URL=...: These lines define environment variables that specify the Kafka version, Scala version, and the URL to download Kafka. Using environment variables makes it easier to update the Kafka version in the future without modifying the Dockerfile.
  • RUN yum -y update && yum -y install java-11-openjdk-devel wget tar gzip vim && yum clean all: This line updates the package repositories and installs the necessary packages, including Java 11, wget, tar, gzip, and vim. Java 11 is required to run Kafka, and the other tools are used for downloading and extracting Kafka.
  • RUN wget ${KAFKA_DOWNLOAD_URL} -O /tmp/kafka.tgz && tar -xzf /tmp/kafka.tgz -C /opt && mv /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka && rm /tmp/kafka.tgz: This line downloads the Kafka distribution, extracts it to the /opt directory, and then renames the extracted directory to /opt/kafka. The downloaded archive is then removed to save space.
  • ENV KAFKA_HOME=/opt/kafka, ENV PATH=$PATH:$KAFKA_HOME/bin: These lines set the KAFKA_HOME environment variable and add the Kafka binaries to the system's PATH. This allows us to run Kafka commands from anywhere in the container.
  • RUN mkdir -p /var/lib/kafka/data: This line creates the directory where Kafka will store its data.
  • COPY docker-entrypoint.sh /: This line copies a script named docker-entrypoint.sh from the current directory to the root directory of the container. This script will be responsible for modifying the server.properties file and starting Kafka.
  • RUN chmod +x /docker-entrypoint.sh: This line makes the docker-entrypoint.sh script executable.
  • EXPOSE 9092: This line exposes port 9092, which is the default port Kafka uses for communication.
  • `ENTRYPOINT [