README

What does the command give a host of GitHub Actions runners, and what does it refuse?

Binds: architecture, code

The package builds the firecracker-runners-for-github command.

Who it is for

The command is for the operator of a Linux host with KVM that runs the jobs of GitHub Actions. The operator wants each job in a Firecracker VM of its own, so that no job sees the files of another.

serve reads one config file

firecracker-runners-for-github serve <config> reads a TOML file such as this:

[github]
config_url = "https://github.com/loadbudget"
app_id = "123456"
installation_id = 7890123
private_key_file = "/etc/loadbudget/github-app/private-key.pem"

[host]
firecracker_path = "/usr/local/bin/firecracker"
kernel_image_path = "/var/lib/firecracker-runners-for-github/vmlinux"
root_disk_path = "/var/lib/firecracker-runners-for-github/runner.ext4"
bridge = "fcbr0"
vm_network = "10.65.0.0/24"
state_dir = "/var/lib/firecracker-runners-for-github/vms"
control_socket = "/run/firecracker-runners-for-github.sock"
idle_timeout_minutes = 10
job_timeout_minutes = 1440

[[scale_set]]
name = "lb-minisforum-heavy"
labels = ["linux-heavy"]
vcpu_count = 8
mem_size_mib = 16384
root_disk_size_gib = 64
max_runners = 2

The command refuses a file with a key that it does not know. It also refuses a missing value, a relative path, or two scale sets of one name. It refuses a vm_network with fewer VM addresses than the sum of max_runners. It then prints each problem and exits with code 1. A wrong command line makes it exit with code 2.

drain <config> and status <config> read the same file, and they talk to serve over the unix socket at control_socket. Only root can connect to it.

serve keeps one VM for each assigned job

For each [[scale_set]], serve finds the runner scale set of that name in the runner group “Default”. When the scale set is missing, serve creates it. It sets the labels of the scale set to labels. It deletes no scale set.

For each job that GitHub assigns to a scale set, serve registers a runner and boots one VM, up to max_runners. The runner takes the name of the scale set and a random suffix, such as lb-minisforum-heavy-k3fqx2ab. When the VM stops, serve deletes its files and its TAP device, and removes its runner. When a VM stops before its job starts, the job stays assigned, and serve boots another VM for it at its next poll of GitHub.

Each VM is a child process of serve. At its start, serve deletes each file in state_dir and each TAP device that a previous run left.

The reaper stops a VM that waits or runs too long

Once a minute, serve checks each VM:

• A VM whose runner took no job within idle_timeout_minutes: serve removes its runner, and then stops it.
• A VM that has run longer than job_timeout_minutes: serve stops it, with its job.

GitHub refuses to remove a runner that holds a job. So where a job reached the runner first, its VM keeps running.

drain lets each job end, and starts no VM

drain <config> makes each scale set start no VM, and stops each VM whose runner holds no job. Then drain prints the status. A VM that runs a job runs to its end. The drain lasts until serve stops.

status <config> prints one JSON object:

{ "draining": true, "busy": 1, "idle": 0 }

busy counts the VMs whose runner took a job, and idle counts the others. A runner is busy from the message of its started job, or from the refusal of its removal, whichever comes first.

SIGTERM and SIGINT make serve drain. When each VM has stopped, serve exits. A kill of serve stops each VM, and the job in it fails.

The host gives the bridge, the kernel and the root disk

The host must give these before serve starts:

• KVM, and root rights for serve, which makes TAP devices.
• The bridge bridge, with the first address of vm_network, and NAT to the internet.
• The Firecracker binary, the kernel image and the root disk at their paths.

Each VM gets its own copy of the root disk, of root_disk_size_gib. serve refuses a root disk larger than that size.

The root disk reads its runner from the metadata service

The root disk boots as /dev/vda, and it needs no /boot/efi. The kernel argument ip= sets the address of eth0, and the address of the bridge as its gateway. So the root disk needs no DHCP client. It must grow its file system to the size of the disk at boot.

The metadata service of Firecracker (MMDS version 2) holds this document:

{ "runner": { "name": "lb-minisforum-heavy-k3fqx2ab", "jit_config": "..." } }

The root disk must start the runner with run.sh --jitconfig and the value of /runner/jit_config. After the job, it must run reboot, and not poweroff.

The log of serve holds each line of the serial console of each VM.