NATIVELINK

How do Caddy and the NativeLink scheduler admit a call and send an action to a worker?

Binds: architecture, code

Caddy admits a request that carries a bearer token or a worker certificate

Caddy serves nativelink.loadbudget.com over TLS. A certificate from letsencrypt.org comes from the HTTP-01 challenge on port 80. The Cloudflare record is DNS only, so the challenge and the gRPC stream reach the host unchanged.

Two tokens exist. Caddy sends each token to its own listener of NativeLink. Any other request gets status 401. Caddy checks no method. After a rotation, Caddy also admits the previous value of each token. infra/docs/operations/SECRETS.md states the rotation.

nativelink-tokens keeps each token in its own file under /etc/nativelink-tokens, readable by root alone. It writes a route for each token into /home/nl-server/caddy/tokens/routes.caddy. The Caddyfile imports it. Without a token, the file holds no route. Every request then gets status 401. Caddy mounts the directory and not the file. A file that a rename replaces reaches the container through a directory mount alone.

A NativeLink worker off the host shows a client certificate. kmacmini and lb-minisforum each run such a NativeLink worker. The play infra/ansible/nativelink-worker-certs.yml writes the trust pool on the host. The pool holds the certificate of each one. lb secret rotate runs the play. Caddy mounts the directory and not the file, as with the token routes. Caddy verifies a certificate against the pool, and it sends a verified client to the workers listener. A client with no certificate must show one of the two tokens.

The matcher reads {http.request.tls.client.certificate_der_base64}. Without a certificate, {tls_client_fingerprint} still holds a value: the SHA-256 of an empty input. A matcher on the fingerprint then admits every client to the workers listener.

With client authentication on, Caddy refuses a request whose Host differs from the server name of the TLS handshake. It answers status 421.

Each route holds its tokens as text. Inside a header matcher, the placeholder form {env.NAME} does not expand, and every request then gets status 401.

The scheduler sends an action to a worker by its platform

infra/ansible/roles/nativelink/files/server.json5 matches OSFamily, ISA and cpu_arch exactly. Each worker states its values in its own configuration. The worker on the host states OSFamily linux. So an action that names another OSFamily reaches that worker never.

The scheduler also matches macos_build and xcode_build exactly. They name the builds of macOS and of Xcode of the Mac worker, so that an update of either gives each Apple action a new key. The worker on the host states neither, and no Linux action names them.

The scheduler matches distribution exactly too. It names the Ubuntu release of the image of the Linux worker, such as ubuntu26.04. A new release gives each Linux action a new key. A new build of the image on the same release keeps the key, because an update within a release of Ubuntu keeps its ABI. The Mac worker does not state it.

The scheduler matches pool exactly too. The simulator NativeLink worker of kmacmini states pool simulator, and no other worker states pool. So a UI test action that names it runs on that worker alone. infra/docs/development/MAC_WORKER.md, section “The simulator NL worker runs the UI tests in a macOS job VM”, states why.

A property that an action does not name restricts nothing. So an action with an empty platform matches every worker. A key that the scheduler does not know, such as cache-silo-key, also restricts nothing, because no worker states it.

Each action reserves the CPUs that its platform names. The scheduler reads cpu_count as a minimum. While an action runs, the scheduler subtracts the cpu_count of the action from the count of its worker. It adds the count back after the worker uploads the outputs. Both platforms of bazel/platforms and the default properties of .bazelrc name cpu_count 1000. A target whose action runs more than one process names that count in its own exec_properties. An action that names no cpu_count reserves nothing.

cpu_count counts thousandths of a CPU. Each worker advertises its CPUs times 1000, and each action asks for 1000 for each CPU. The scheduler learns CPU in that unit alone. resource_sketch.rs in nativelink-scheduler/src of NativeLink v1.8.3 stores it as cpu_time_ms * 1000 / wall_time_ms. It reserves that number in cpu_count with no change of unit. With whole CPUs, an action that used half a CPU asked for about 500 CPUs. No worker had that many, so the scheduler refused the action.

The worker on the host advertises twice its CPUs. worker_cpu_overcommit in infra/ansible/roles/nativelink-worker/defaults/main.yml sets the factor. Each action reserves the 95th percentile of its average CPU plus 20%, so the reservations add up to more than the actions use. With the factor at 1, the worker left about half of its CPUs idle while actions waited. --cpus still caps the container, so a full worker shares CPU time and kills no action. Memory takes no factor.

The scheduler raises each reservation to what the worker measured. The historical_resource scheduler of server.json5 learns from the workers. Each worker samples the processes of each action every 250 ms. The scheduler keeps the 95th percentile of each kind of action, by target, mnemonic and command, for 7 days. It reserves that value plus 20%, in cpu_count and memory_kb. It never lowers a value that the client sent. SCHEDULER_SKETCH_STORE keeps what it learned across a restart. The 95th percentile, and the defaults of 20% and 7 days, come from HistoricalResourceSpec in nativelink-config/src/schedulers.rs of NativeLink v1.8.3. The 250 ms is RESOURCE_USAGE_SAMPLE_INTERVAL in nativelink-worker/src/running_actions_manager.rs of the same version.

An action with no history reserves no memory. The worker on the host takes any number of actions, because max_inflight_tasks is 0. So the reservations alone bound how many run at once. The worker kills an action that grows past its memory reservation plus 20%. The scheduler then queues it again at the next step of ladder_kb in server.json5. Past the top step, the action reserves the whole NativeLink worker. When the kills pass max_steps of MemoryEscalationSpec, the scheduler fails the action. A learned reservation can sit far below what the action needs. Doubling from there can spend every step and still fall short, and the first step of the ladder starts above it. An action with no history carries no memory_kb, so the NativeLink worker does not kill it. A burst of such actions can fill the memory of the container, and the kernel then kills inside it. That happens on a cold cache.

A worker short of free memory takes no action that reserves more. Each worker reports its free memory with each keepalive. With live_memory_veto, the scheduler skips a worker that reports less free memory than the memory_kb of the action. It catches an action that reserved too little and still runs.

An action that times out runs once. A NativeLink worker reports a timeout as an error with DeadlineExceeded. The scheduler counts that error, and an internal error of the action, against max_job_retries, then queues the action again. server.json5 sets max_job_retries to 0, so the scheduler completes the action with the first such error. The default is 3. A lost NL worker and an unreachable store spend budgets of their own, so they still queue the action again. nativelink-config/src/schedulers.rs and nativelink-scheduler/src/simple_scheduler_state_manager.rs, at the NativeLink version in infra/nativelink-worker/Containerfile, state both.

The server pushes its metrics to an Alloy beside it

NativeLink pushes its metrics, logs and traces over OTLP to the address in NL_OTEL_ENDPOINT. It serves no /metrics, so no agent can scrape it. The exporter speaks gRPC alone, and the Prometheus agent of lb-hetzner02 takes no OTLP over gRPC. web/apps/docs/content/docs/operate/observability.mdx and nativelink-util/src/telemetry.rs of NativeLink state both, at the version in infra/nativelink-worker/Containerfile.

Alloy runs as a container of nl-server on network lb. From that network, the server reaches no port of lb-hetzner02, host.containers.internal included. alloy.container holds the fixed address 10.89.0.200, outside IPRange of lb.network. The unit of the server names that address in NL_OTEL_ENDPOINT.

infra/ansible/roles/nativelink/templates/alloy.alloy.j2 holds the pipeline:

The scheduler counts each completed action with its digest. Each action then makes a series of its own. So Alloy deletes the digest before the push.

An endpoint that refuses stops no server. NativeLink starts, and it answers /status, while nothing listens at the endpoint. Its metrics reach Alloy once Alloy starts. An endpoint that is no URL with a scheme, a host and a port stops the server at its start. So does a host name that does not resolve, so the unit names an address.

Alloy and the server run only where the metrics token exists. The role nativelink reads push.json and the token under /etc/loadbudget/grafana-cloud/. Until a rotation writes both, no Alloy runs, and the unit of the server sets no NL_OTEL_ENDPOINT. Alloy reads a copy of the token, /home/nl-server/alloy/token, that only nl-server reads. The role writes it at each deploy, and infra/ansible/metrics-token.yml writes it at each rotation.