Kafka Message Broker Setup

The heart of Cloud Builder is the message broker. The messaging system is the foundation of the Cloud Builder API. All data send and read is validated against a Cerberus based schema definition. The used schema implements the API for the service. Only successfully validated data will be handled.

Note

There are some messaging services available. Currently Cloud Builder implements support for the Kafka message broker only. Thus the following information is specific to Apache Kafka

For Cloud Builder it is immaterial from where it can access the kafka service. Because of this reason it can be a self manufactored local solution, an offer by a managed service provider like Aiven or a cloud service offering like Amazon MSK. For good performance it is recommended to place the kafka service near to the Cloud Builder services with regards to the network and region.

Note

The current implementation of kafka in Cloud Builder assumes the most simple access method to the kafka service. This requires kafka to run in the same network from which cb- tools and services are used. If Cloud Builder gains some attraction this will be extended such that the config file to store the access credentials can also work with SSL and certificate based information.

As the focus of this documentation is to run Cloud Builder in the Amazon cloud, the promotion for kafka is set to Amazon MSK. A good start to get kafka running in Amazon can be found here:

• https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html

Once a kafka message broker is running in some way shape or form the next step in setting up the Cloud Builder build cluster is the launch of the control plane. Learn how to start and configure the control plane next: Create and Install the Control Plane