Every pod on one screen.

From the installer to a cluster

The installer has a page for the cluster: how many control planes and workers, how big each is, and its addresses, with a 3x3 starter that fills them in. At the first boot kbx builds it as one job you can follow in the Jobs window, each phase read back before the next starts.

1. PreflightRefuses before anything is touched: the hypervisor answers, the box has the memory, the node image and the stack are what they claim.
2. The goldenThe node network, then the node image the appliance carries, installed once into a machine and sealed as a template.
3. The nodesEach a clone of the golden with its seed, started and found ready: its agent, cloud-init done, containerd running.
4. The meshWireGuard between every pair of nodes, every handshake read back.
5. The first control planeIts API at the cluster's own address, read back through it.
6. The joinsEvery worker, then every other control plane, through that address.
7. The add-onsCilium until every node is Ready, kube-vip, MetalLB and its pool, metrics-server, the ZFS volume driver, Tetragon and its policies.
8. ReadyCoreDNS up and the API answering at its address: then, and only then, the record says ready.

A node that does not come up fails the job, and the job removes what it made. The record is written as the build goes, so a build that stopped says what it left.

Fly through your pods

Tap Super and every pod in the cluster is a card, its containers stacked on it like sheets of paper, the frontmost with its image and whether it is ready. Super+Tab lays the namespaces out as a wall, each with its pods stacked, the ones that are not well on top. Enter goes into a namespace, and into a pod; Escape comes back out.

The cards are asked of kbxd every five seconds while the view is open. A read that fails says so over the last good one, and never empties the wall.

A pod's log, and why it ended

The Pods window and the console list every pod with its node, its state, its containers ready, its restarts and what it uses, and what is wrong with it in a few words. In the console, open one for each container's state and how its last run ended, with the reason and the exit code, then the last lines of its log, and what it wrote before its last restart.

Deleting a pod is how a pod is restarted: what owns it makes it again, and the job says which it was.

Every flow, the dropped first

Cilium is the cluster's network: eBPF programs in each node's kernel, in place of kube-proxy. Hubble records the flows through it: who spoke to whom, on which port, and whether it was forwarded or dropped. The Flows window shows the last thousand, the dropped ones first with Hubble's reason, and the counts by verdict above them.

Every process, on every node

Tetragon watches each node's kernel through eBPF: every program a workload starts and every one that ends, and whatever its tracing policies ask after. Two come with the cluster: every write to the files that say who may log in, in a pod or on a node, and every zfs, zpool and zdb that anything but the kubelet runs on a node.

The Security window lists the last thousand events, a policy's first, each with the pod or the host it ran in, the node, the program and its arguments, and what the hook saw. On the bench, a shell in a pod appending to its own /etc/passwd was reported; the read just before it was not.

Nodes are machines, a click away

The Nodes menu has a row for each node: its state, its roles and its share of CPU and memory, with cordon, uncordon and drain as its buttons. Drain moves a node's pods to the others before you work on it; a pod on a ZFS volume, which lives on its node, waits for it to come back. Add a worker, or two control planes so etcd keeps an odd count: one click, and it starts at once.

When the box restarts, the control planes start first and the workers once the API answers, so the workers' pods start with an API to reach.

Everything it runs, carried

The cluster is built from a lock: Kubernetes and every add-on at an exact version, every image by its digest, twenty-eight in all, carried inside the appliance and put on every node. A node missing one of them fails its health check. Nothing comes from a registry at the build, so a box with no internet builds the same cluster.

Every volume is a ZFS dataset on its node's pool, by one of two storage classes: zfs, the default, and zfs-db for a database: 16K records, synchronous writes logged for latency, lz4, no access times.

Ready means every check

The cluster is ready when its checks say so, not when its build ends:

And every pack it rolled out, whole and answering. A check that cannot run counts as failed, never as passed. Each is shown red by its own fault in the tests, and on the bench moving the pool and stopping CoreDNS turned theirs red. An upgrade of the box is judged by them too, once the cluster has had time to settle.

Your workload, carried in

A pack is a workload with every image it needs inside it. One rollout proves the pack whole, puts each image into every node that lacks it from the pack itself, applies its manifests, and reads it back: every deployment available, its service given an address from the pool, and that address answering.

The first pack comes in the image: a small page that says which pod answered it, so each refresh shows the load balancer at work. On the bench it rolled out in six seconds, its image into all six nodes, nothing pulled. Your own packs, built on a workstation and signed with your key, are next.

Where it stands

kbx is in testing, on the beta line only: its installer is below. Every picture on this page is from the bench, a cluster of three control planes and six workers on one box. The installer runs from memory before it installs, so it needs a machine with about 8 GiB of memory or more; on less it stops and says how much.

What one box cannot do, said plainly: three control planes on one box survive a node failing, a control plane being replaced, and a bad deploy. They do not survive the box itself. That is the appliance's own job, with its boot environments; a cluster across two boxes comes later.

Still to come: