Cloud Builder Documentation¶
Note
This documentation covers Cloud Builder 1.0.7- A package and appliance build backend primarily used as IaaS system. The main use case is to extend/adapt a base Linux distribution to personal needs, e.g for embedded devices.
Building Software Packages and Appliances as a Service¶
The Cloud Builder project provides a collection of services which
allows to stand up a package and appliance building backend to
build software packages as well as system images through a
messaging API. Primarily rpm and deb package formats
are supported as well as any image type supported by the
KIWI appliance builder.
Cloud Builder is designed to run in cloud environments and this
documentation describes how to run Cloud Builder in
AWS (Amazon Web Services). However, in general the environment
to run the Cloud Builder services is not restricted to a specific
cloud framework.
Why¶
If small services for dedicated tasks that can scale with the amount of tasks and use cases is preferred over a monolithic server. If easy integration of packaging services with other services is a priority. If building in an isolated private network should be possible by utilizing cloud services. If build power, network bandwidth and storage capacity should be a scalable factor in the cloud. If managing package sources and build metadata should be done completely in git. If one ore more of this aspects are a matching criteria, this project might be interesting.
Architecture¶
The Cloud Builder architecture consists out of three major components:
- The Sources:
The package and image sources and all metadata to build them with Cloud Builder must be stored in a git repository from which the Cloud Builder services fetches information. The idea behind this is that package sources and metadata is treated like code and git is great in managing code. The history in Git allows to track any information about changes in the life time of a package or image. For details how to setup the source repo see: Cloud Builder Source Setup
- The Messaging API:
Cloud Builder services uses the Apache Kafka message broker to communicate for request and response information. From a Cloud Builder implementation perspective other brokers could be added via a new interface class. However, it depends on the broker features if it makes sense to use this message broker together with Cloud Builder. Apache Kafka was choosen because it allows for the Publish/Subscribe mode and for the Shared Queue mode in a nice way. Especially The Shared Queue mode in combination with the topic partition setup is used to distribute build requests across the available runner instances. For details how to setup the messaging API see: Kafka Message Broker Setup
- The Cloud Builder Services:
The Cloud Builder services introduced in Overview Cloud Builder Services implements the actual functionality to allow building a package, or fetching information about a package, or building repos from packages, and so on. The services runs on one or more machines representing a certain functionality like a
runnerto build packages and images or acollectorto create repositories from build results. The collection of all machines represents the Cloud Builder cluster.
The following diagram shows how the Cloud Builder services play together and provides a visual overview about the system.
Cloud Builder Architecture¶
Note
The minimal infrastructure to build packages with Cloud Builder requires:
A git repo with the package sources. There is the cloud-builder-packages example repo
A Kafka message broker. The focus will be on Amzon MSK
A runner instance with the cb-info and cb-scheduler services running.
As the idea of Cloud Builder is to come up with a scalable system that is configurable to the needs of the user, some of the services are optional on top of the minimal infrastructure.