Runs

A run is one step on one tree of the code: a command, the paths that it reads (its inputs) and the steps that must pass first (its needs). The design is #32 and #64. Buck2 with NativeLink as its cache runs inside a step (#97); this page covers the queue, the record and the runner.

Where each part lives

Part	Store	Why
The queue of the steps	NATS JetStream, stream DESK_RUNS, subject desk.runs.<repo id>	A work queue: a message stays until a runner acknowledges it, and a delivery that is not acknowledged within the ack wait goes to the next runner.
The record of each run	Postgres, table runs	What ran, its status, its attempts, who ran it, its result and the key of its log.
The log of a run	The object store (S3), key repo-<id>/logs/<row id>	Bytes.

The decision of the owner of 2026-10-17 on #32 sets this split: NATS is the queue, and Postgres keeps the record.

The run id

The run id is the SHA-256 of the command, the tree id of the inputs and the sorted run ids of the needs (runIdOf in src/server/runs.ts). A change in a file that the inputs do not name keeps the id. A request for a step whose run is queued or running gets that run, and nothing new is queued. A step that ran before runs again; the build inside it (Buck2) takes the results that did not change from NativeLink, the cache of #97, so a run of the same code takes seconds.

The queue

src/server/run-queue.ts has two queues with the same rules:

• NatsQueue: the stream is a work queue on file storage, at most 512 MiB (1.5 GiB with 3 replicas). When it is full, a new step is refused, and its run ends as an error with 503. Each repository has one durable pull consumer, runners-<repo id>, with explicit acknowledgement and an ack wait of 60 s. The message id is run-<row id>, so JetStream drops a second publish of the same run within its duplicate window.
• MemoryQueue: the queue in the memory of one server process. A server with no NATS_URL uses it, and the tests use it.

With NATS_URL, the server connects the queue in the background and tries again, from 1 s up to every 30 s, until it works (ConnectingQueue). Until then the server serves its code, a new step answers 503 with its record as an error, and a claim finds no step; the log names each failed attempt. On 2026-10-17 a start that waited on NATS, while NATS loaded the new account of desk, ended with "timeout" and the server was down for about 17 minutes.

At each start the server publishes its queued runs again (with NATS, when the queue connects), so a crash between the record and the publish loses no step; with the memory queue it publishes the running runs too. A second message of a run is skipped: the record keeps the sequence of the message that it runs (delivery_seq), and a claim takes a run only when it is queued or when the delivery is of that same message.

A claim takes the next delivery. The run record then holds a lease token and the address of the delivery ($JS.ACK.…). A heartbeat sends +WPI to that address and the ack wait starts again; a completion sends +ACK. Any server process can do this, because the address is in Postgres. A new delivery of the same step replaces the lease, so the runner before it gets 409 on its next call. A step that comes a fourth time ends as error and gets +TERM.

The routes

All routes are under /v1/runs/<owner>/<repo>/ and need Authorization: Bearer <token>.

Route	Right	Answer
POST steps {"name","command","inputs","needs","timeoutMs","tree"}	merge	201 with the new run, or 200 with the run of the same id that waits. 503 when the queue refuses the step; the record then ends as error.
POST claim {"waitMs"}	run	{"run": <claim> | null}. The server waits up to waitMs (at most 20 s) for a step.
POST runs/<id>/heartbeat {"lease"}	run	409 when the lease is lost.
POST runs/<id>/complete {"lease","reason","exitCode","durationMs","log","error"}	run	The record. 409 when the lease is lost.
GET runs/<id>	read	The record.
GET runs/<id>/log	read	The log.
GET landings/<seq>	read	The loop of the landing seq of the log: checks (the newest run of each check), images (the builds, and the promotions of the images whose tags wait for a check) and deploys of its tree, each with name, run, status, error, artifact, durationMs, startedAt and finishedAt. A time is null while the run waits or runs. 404 when the log has no such entry.

desk runs reads these routes, so an agent needs no call of the API by hand:

• desk runs lists the newest runs, and desk runs checks the newest run of each check of main.
• desk runs view <id> prints one run, and desk runs log <id> [--tail n] prints its log.
• desk runs landing <seq> [--wait] prints the loop of a landing, with the time of each run and, when every run ended, the time from the landing to its last step. With --wait it prints each change of a run, until no run waits or runs for 15 s.
• desk land --wait does the same for the landing that it made. The exit code is 0 when every run of the landing passed (the checks, the images, their promotions and the deploys), and 1 otherwise. A failed check ends the wait at once.
• In a work folder, these commands use its server, repository and token. Elsewhere, --repo owner/name uses the server of ~/.config/desk/policy.json and DESK_TOKEN.

A step runs its command on the runners, so queueing one needs merge, the right that lands code.

The right run also reads the objects of the code (GET /v1/code/<owner>/<repo>/objects/…), because a runner writes the tree of each step. It reads nothing else of the repository.

The checks after a landing

A proposal runs the same checks on its own tree when it is created, and shows their results before the merge (docs/server.md, "Proposals, reviews and checks").

When a change lands on main, the server reads .desk/config.json of the new main and queues one step for each entry of validation (queueChecks in src/server/runs.ts):

• The command of the step is setup, then command, in the folder cwd: cd '<cwd>' && <setup> && <command>.
• paths says when a check runs, as for desk validate: the landing queues the check when it changed a file that paths match (against the tree before the landing), and always when the check has no paths. A glob with no / matches the file name in any folder, so /package.json names the one at the root and package.json matches ui/package.json too.
• The input of the run is the whole tree, because a command reads files outside its paths (the tests of desk read server/migrations). The build inside the check (Buck2) takes from NativeLink the actions whose inputs did not change.
• The time limit is timeoutMinutes, 30 by default, at most 120.
• stage says when the check runs: main (the default) on main after the landing, or land before it (the next section). The checks after a landing skip the land checks.
• secrets maps a variable of the command to a secret of the runner (#131): {"NODE_AUTH_TOKEN": "npmtoken"} gives the command the runner's DESK_BUILD_SECRET_NPMTOKEN as NODE_AUTH_TOKEN, for example to publish a package. Only the check on main after a landing gets them. A proposal, a bisect and desk validate run the same command without them, and a land check cannot have them. The runner replaces each value with [secret] in the log, and a runner without the secret ends the run as an error that names it.
• A check that the queue refuses names its error in the answer, and the other checks are still queued.

A check can read other repositories of the server. checkouts of .desk/config.json lists them. An example is [{"repo": "acme/lib", "path": "../lib"}] for an app that builds acme/lib by path. The rules:

1. The landing puts the tree of main of each one at that moment into the run record (extra_trees, migration 008_run_checkouts.sql) and into the run id. So a new main of acme/lib gives a new run.
2. The runner writes each tree at its path next to the checkout. It reads the objects with its right run on that repository.
3. A path must stay in the scratch folder of the run, and away from its home and tmp.
4. The author of the landing needs the right read on each repository of checkouts, because the log of a check shows what it read.
5. A repository with no main, or one that the author cannot read, gives each check an error. The landing stands.
6. The log of a run needs the right read on the repository of the run and on each repository of its checkouts.

The answer of the landing lists the checks (checks: name, run, status, and whether it was queued now); desk land prints them. A fault while the checks are queued, for example a config that is not JSON, does not undo the landing: the answer names it in checksError, and the server log names it too. A tree with no config queues nothing.

The budget of a landing (#143)

The owner's budget: a change is validated, built and released in under two minutes, and under one minute in the usual case. Measured from desk change and desk runs view:

Landing	Change	Landed	Check	Image	Last push
255, server code	11:08:16.9	11:08:25.0	root 71.4 s	three images, 7.6–8.4 s each, in parallel	11:09:46.7
256, UI and docs	09:24:02.0	09:24:13.6	root 2.7 s (cache)	desk-ui 68.3 s	09:25:33.7

The images no longer wait for the checks (#145). These are the landings before and after that change, measured with desk runs landing <seq>:

Landing	Change	Checks	Images	Promotions	Landing to last step
260	the times of desk runs landing	docs 55 ms, root 124 s	desk-ui 61 s and the three server and runner images 7.7–8.2 s, after the check	—	172 s
261	the promotion itself	docs 45 ms, root 116 s	desk-ui 64 s and the three server and runner images 8.2–9.2 s, after the check	—	130 s
262	the promotion as a step of the landing	root 85 s	desk-server 8.2 s, desk-runner-rust 8.3 s, desk-runner-base 7.7 s, beside the check	1.3 s each	90 s
263	a page of docs/	docs 43 ms	desk-ui 46 s, beside the check	—	48 s

A landing waits for its check: an image adds its promotion, not its build. The build of an image with needs runs beside the check, and its tags follow in a promotion once the check passed (1.3 s on landing 262). So a server landing is the time of root plus about a second, and the repositories over 60 s of #145 are the ones whose check or image is long.

What a change runs:

• docs/** and README.md: the docs check (scripts/check-docs.mjs), in about a second. They are not a path of root, so a docs landing no longer queues the Buck2 suite. Since the UI serves the docs at /docs (#144), a docs change also builds the desk-ui image.
• ui/**: the desk-ui image. scripts/image-ui.sh runs next build (which type checks the app) and npm test, and keeps the Turbopack cache and tsconfig.tsbuildinfo in desk-ui-build next to the npm cache of the runner, so a build that changed a few files reuses the rest. On 2026-11-02 a warm local build took 5.3 s where a cold one took 40 s.
• src/**, test/**, skills/** and the other paths of root: the Buck2 check, and the images that need it build beside it. Their tags follow in a promotion once the check passed (#145), so the check is not on the path of the image and a check that failed publishes no tag.
• The check of root lasts as long as its slowest action. test/server-history.test.mjs runs one test in each action since #143 (it was two groups of 5 and 4, and the group of 5 held landing 255 for 57 s). A PGlite file groups its tests by 5, because each process pays about 4 CPU-seconds to start PGlite.

The roll of the image is Keel and the Deployment, outside this repository; it adds about a minute.

The checks before a landing: stage: "land" (#66)

A check with "stage": "land" in validation runs before main moves, on the tree that would land: the change itself, or the merge of the change onto a main that moved. Keep this stage for fast checks, such as a type check; the full tests run after the landing.

1. POST /v1/code/<owner>/<repo>/land runs the gate (landGate in src/server/runs.ts) right before main moves. The land checks come from the config of main, as desk merge reads the config of the base branch, so a change cannot switch off or edit the gate that judges it. The first landing of a repository has no main, so its own config holds the gate. When the config of main does not load, only a change whose own config loads lands: it repairs main, and its own config judges it. A land check whose paths match a change against the tree of main is due. With no due check, the gate passes and reads no checkout.
2. A check whose run id passed before passes at once. A check with no run is queued, and the answer is 202 with status: "checking" and the runs. A run that ended with an error is queued again, up to 3 times.
3. When every due check passed, main moves to that tree, and the checks after the landing are queued.
4. When one failed, the answer is 422 with status: "refused", and main does not move.
5. desk land asks again every 2 seconds while the answer is checking, for at most 2 hours, and prints the runs once. If main moves in between, the tree that would land changes, and its checks run again.

A revert of desk (next section) lands with no gate, because it restores the tree from before the broken change.

Land first, test after: bisect and revert (#66)

A change lands with no wait, and its checks run on main after it. The server records the landing of each check run (landing_checks, migration 009_landing_checks.sql). When a run of a check ends, the server takes the next step for that check (src/server/bisect.ts):

1. The newest result of the check decides. If it passed, nothing happens.
2. If it failed, the server finds the newest landing where the check passed. If no landing passed it, nothing happens.
3. That pass and the first failure after it mark a range. If the failure is the next landing after the pass, that landing broke the check.
4. If not, the server runs the check on the landing in the middle of the range. It uses the config of that landing and ignores the paths of the check.
5. The run in the middle reads the same other repositories as the failed run. A landing whose config has no such check counts as passed.
6. Each run that ends takes the next step, so the range halves until one landing is left.
7. The server reverts that landing with a new change of the author desk on the main of now. The change takes back the files of that landing and keeps the landings after it.
8. The revert runs the check again. The issue of the broken change (its --intent) gets a message: the check, the landing, the revert and the run of the failure.

A landing is reverted once (reverts, keyed by the entry of the log). The bisect stops, and nothing is reverted, in these cases:

• The landing is a revert of desk, because its revert lands the broken change again.
• The revert conflicts with main. The row is conflict, and the issue tells the author to revert by hand.
• A run in the range ended with an error 3 times (CHECK_ERRORS), or hit its time limit. Those results say nothing about the code. The server log names the run.

A check of a landing that ends with an error (a fault of the runners or of the build infrastructure, such as exit 75) or on its time limit runs again on the same tree, with the same run id, until it has ended in one of these ways 3 times. Block-builder run 415 waited 30 min, until its time limit, on a cache pod that was Pending. The landing then points to the new run, so the bisect, desk runs landing and the images follow it. Axon run 333 ended with "broken pipe" when a rollout restarted the NativeLink scheduler; the next run passes, with no step by hand.

• The passing run and the failed run read different trees of the other repositories. One of those repositories can be the cause. The server log names both runs.

GET /v1/runs/<owner>/<repo>/reverts lists the reverts, newest first, with the right read.

A server that stops in the middle of a revert leaves it open. Every server, at its start and then every 5 minutes, finishes each open revert that is older than 5 minutes. A revert or a check that fails there is logged, and the others go on. It also takes the bisect steps that no server took. A live replica works on a younger revert, so a second replica leaves it alone.

The live proof on hotmun (2026-10-22): log entry 95 added a test that fails, and log entry 96 changed only this page. The result of the proof is in #66.

The check of desk: Buck2 with NativeLink (#97)

The check of desk is sh scripts/check.sh. It runs buck2 build //:check, with these actions (BUCK, build/defs.bzl):

• //:node_modules: npm ci --include=dev from package.json and package-lock.json.
• //:dist: npm run build from the sources.
• One action for each test, with only the files of dist/ that its file reaches (build/test_deps.bzl, made by scripts/buck-test-deps.mjs). A file whose tests share state (a before or after hook, a top-level let) runs as one action, unless it says // buck: each test makes its own state. A named test must run exactly once.
• Each test action gets only the data roots of BUCK (DATA_ROOTS) that its test file or its helpers name as a path (TEST_DATA in build/test_deps.bzl); a test that runs desk install or desk update gets every root. Before, each test took all data files, so a change of .desk/config.json ran about 160 of 370 actions again on each landing (runs 143 to 163). A root that a test reads and the map leaves out is missing in the folder of the action, so the test fails.
• //:test-deps-fresh: fails when build/test_deps.bzl is out of date.

On a runner, DESK_CACHE_ADDRESS names NativeLink, and check.sh writes it into .buckconfig.local, with the versions of node and npm, which go into the key of every action. An action whose inputs did not change comes from NativeLink and does not run. Every action copies its inputs into a folder of DESK_ACTION_TMP (the TMPDIR of the run), outside the checkout.

A machine on the mesh reads the results of the runners and writes none: DESK_CACHE_ADDRESS=grpc://cache.int.hotmun.com:443 DESK_CACHE_TLS=true DESK_CACHE_UPLOAD=false sh scripts/check.sh. The private Traefik of hotmun (traefik-internal, VIP 10.200.0.210, which the vRack and the NetBird mesh reach) serves that host with TLS and sends it to the Service desk-nativelink-mesh: a second listener of NativeLink with a read-only action cache and a content store that checks each digest. DESK_CACHE_UPLOAD=false writes cache_uploads = false into .buckconfig.local, and platforms/defs.bzl then turns off the uploads.

Measured on 2026-10-17 on a machine with 16 cores and a local NativeLink, from an empty buck-out:

Case	Time	Actions from the cache
No cache	204 s	0 of 39 (one action for each test file)
The same code	1.4 s	276 of 276
One test file changed	16.7 s	all but that test
src/server/merge.ts changed	231 s	134 of 276; the slowest action was test-cli, 133 s

The time of a changed check is the time of its slowest test and of the queue of its actions on the cores of the runner.

Remote execution on all the nodes (#97)

The actions of a check that the cache does not hold run on the workers of NativeLink, not only on the runner that took the check. The runner has DESK_RUNNER_RE (the scheduler, grpc://desk-nativelink-scheduler:50051 on hotmun) and gives it to each step as DESK_RE_ADDRESS; scripts/check.sh then sets desk.remote_exec, and the platform of Buck2 (platforms/defs.bzl, and buck/rust/platforms.defs.bzl for the Rust repositories) sends every action to the scheduler with the properties OSFamily=linux and container-image=desk-runner.

• The scheduler is one pod: its queue of actions is in memory. Its content store and action cache are the NativeLink cache (desk-nativelink, S3 and Dragonfly), through grpc stores.
• The workers are a DaemonSet, one pod on each node (desk-re-worker), on the image of the runners: Node, the Rust toolchain, git and the static nativelink binary (docker/base/runner-rust.Dockerfile). A worker runs 10 actions at once, and has the grace period of the runners.
• A worker gives the inputs of an action read-only, so an action makes its copies writable (chmod -R u+w in BUCK and build/defs.bzl).
• The first check of a repository with remote execution uploads all the inputs of its actions, also node_modules, to the cache; OVH S3 answers such a burst with "SlowDown", so the S3 stores of the cache retry each write with a backoff (munhq/k3s-gpu#199). Later checks upload only what changed.
• An error of the infrastructure of Buck2 (exit 2, 4, 5, 11 or 12) makes scripts/check.sh exit 75 (EXIT_INFRA): the run is an error, and desk starts no bisect and reverts nothing.
• Without DESK_RE_ADDRESS, DESK_CACHE_ADDRESS is the cache and the actions run on the machine of the check.
• The pods of desk have a CPU request and no CPU limit (munhq/k3s-gpu#217, #218). A worker or a runner takes the idle CPU of its node. When the node is busy, the requests set the share of each pod, so the databases of the node keep theirs. On 2026-10-25 the namespace desk peaked at 19 cores in 6 hours, with an average of 0.9 cores for the three workers.

Images: BuildKit builds with the landing, and a promotion applies the tags (#67, #145)

images in .desk/config.json lists the images of a repository. Each entry has name, image (the name with no tag), dockerfile (default Dockerfile), context (default .), needs (names of checks of stage main), tags, paths, changeArg, secrets, target (the stage of a multi-stage Dockerfile; without it, the last stage) and timeoutMinutes.

1. After a landing, and after each run that ends, the server looks at the head of main (advanceImages, src/server/images.ts). An image is due when its landing is on main, when no check of needs failed on that landing, and when its paths match (step 4). A check that the landing did not queue counts as passed, as in the bisect.
2. A build starts with the landing, and not after its checks: the check and the build wait in the queue side by side, so the check is not on the critical path of the image (#145). A check that failed blocks the build of that landing.
3. A build is a run of kind image (migration 011_images.sql). Its run id holds the tree and the image spec. The whole tree is its input.
4. With paths, the build starts only when a file that they match changed since the last image that was pushed.
5. Only the head of main is built, and one build of an image waits or runs at a time (a unique index). A landing during a build is built when that build ends, so an older landing never replaces the tag of a newer one.
6. A build that failed is not queued again by itself, and a build that ended with an error is queued again 3 times. A new landing builds again.
7. A runner with BuildKit takes the run, writes the tree, and runs buildctl (src/server/image-build.ts). An image with needs and tags pushes only image:<change id>: its own tags (main, latest) come in a promotion (step 9). An image with no check, or with no tags, pushes image:<change id> and each of tags in the build. changeArg gives the change id to the build as a build argument. Each id of secrets (for example npmtoken) is a BuildKit secret: the runner writes the value of its variable DESK_BUILD_SECRET_<ID> to a file of the scratch folder and passes --secret id=<id>,src=<file>. The Dockerfile reads it with RUN --mount=type=secret,id=<id>. A runner that does not have the variable ends the run with an error that names it, and the server queues the build again for another runner.
8. The digest of the push is the artifact of the run.
9. A promotion is a run of kind promote (migration 031_promote.sql, src/server/promote-run.ts). It starts when each check of needs passed on the landing, copies the manifest of the pushed digest to each tag of the step with the registry API, and tells Keel of each tag. The manifest does not change, so no layer moves: the three promotions of landing 262 took 1.3 s each. A tag never names a change whose check did not pass, and a check that failed leaves the tag of the change in the registry, as the record of what the build pushed. The run waits in the lane runs, as a check does; a runner without a registry login ends it with an error that names the login, and the server queues it again 3 times. One promotion of an image on one tree waits or runs at a time (a unique index).
10. A row of deploys records the push: change, image, digest. The row is written for the build of an image that does not promote, and for the promotion of one that does. A Deployment that follows a tag (:latest) rolls through Keel; a Deployment that runs a digest rolls through a target of deploys (below), and that target takes the digest of the promotion, so a Deployment never rolls a change that a check has not passed.

On landing 262 of munhq/desk (2026-11-02) the root check took 85 s, the three builds of the server and the two runners 7.7 to 8.3 s each beside it, and the three promotions 1.3 s each after it: 90 s from the landing to the last step. The check is what a landing waits for, and the image adds only its promotion.

Only the config makes an image run: POST steps refuses to set one. A runner that has no DESK_BUILDKIT_HOST or no login ends an image run with an error that names the missing setting.

Variable of the runner	Use
DESK_BUILDKIT_HOST	The BuildKit daemon, for example tcp://buildkitd:1234.
DESK_KEEL_WEBHOOK	The native webhook of Keel, for example http://keel-metrics.keel.svc.cluster.local:9300/v1/webhooks/native. After a push, the runner sends {"name": image, "tag": tag} for each tag that the build pushed, and after a promotion for each tag that it set, as the image workflows of GitHub Actions did: Keel cannot poll ghcr.io/munhq. A failed call is a line of the run log.
DESK_REGISTRY_USER, DESK_REGISTRY_TOKEN	The login of the registry of the images. Both or none. The runner writes them into a Docker config file in the scratch folder of the run. They are in no command line and not in the environment of the command.
DESK_BUILD_SECRET_<ID>	The value of the build secret <id> of an image step, for example DESK_BUILD_SECRET_NPMTOKEN for npmtoken. It goes to a file of the run for BuildKit, and to no command line and no environment of a command.

Tested on 2026-10-21 against BuildKit v0.34.1 and registry:2 on one machine: DESK_TEST_BUILDKIT_HOST=tcp://127.0.0.1:1299 DESK_TEST_REGISTRY=127.0.0.1:5077 node --test test/server-images.test.mjs builds a tree, and both tags in the registry have the digest of the run.

Build arguments, other repositories, and builds on their landings

• args gives a Dockerfile build fixed build arguments, for example {"VITE_CLOUD": "1"}. They come before changeArg, sorted by name, and are part of the run id.
• An image build reads the checkouts of the repository, as a check does: the runner writes each one at its path. A path inside the checkout (for example product) puts that repository in the BuildKit context. The trees of the checkouts are part of the run id.
• A landing in a repository advances the images of each other repository whose checkouts name it (advanceDependentImages). The image of chatrecall.dev, which munhq/chat-recall-site builds from munhq/chat-recall, is built again when the product lands.

Images with a base and no Dockerfile (#116)

An image step with base (an image pinned by its digest, registry/name@sha256:<hex>) and build (a command) has no Dockerfile and needs no BuildKit. The runner runs build in the checkout with the variables of a check (a Buck2 build takes its actions from NativeLink) and DESK_IMAGE_ROOT, a folder that build fills with the files of the image. The runner makes one layer of that folder (src/server/image-assemble.ts): a tar with the names sorted, the time 0 and the owner 0, compressed with no name and no time, so the same files give the same layer. Then it pushes with the registry API (src/server/oci.ts): the layers of the base (mounted from the repository of the base on the same registry, copied from another registry), the layer, the config of the base with the layer, a history entry, the label org.opencontainers.image.revision and the settings entrypoint, cmd, env, user and workdir, then the manifest under each tag. The digest of the manifest is the artifact of the run. target and secrets go with a Dockerfile only.

Own base images: BuildKit for one build (#116)

An image step with a Dockerfile builds an own base image (the owner's decision on #116: public bases as they are where they fit, own bases built only when their Dockerfile changes). With DESK_BASE_LANE=true on the server, its run waits in the lane base (<stream>.<repo id>.base, the consumer base-builders-<repo id>). A runner with DESK_RUNNER_LANE=base and no DESK_BUILDKIT_HOST runs buildctl-daemonless.sh: BuildKit starts for the one build and ends with it, with BUILDKITD_FLAGS=--oci-worker-no-process-sandbox. The cache of the build is in the registry (--import-cache and --export-cache type=registry,ref=<image>:buildcache,mode=max), so a base that did not change is built from the cache: on 2026-10-24 a fresh BuildKit built a Debian base with apt-get in 31 s, then the same base in 4 s from the registry cache. The pod needs the settings of rootless BuildKit (seccomp and AppArmor unconfined).

A runner of the pool asks GET /v1/runner/next?lane=runs|base&waitMs=<n> for the repository of the oldest run that waits in its lane, among the repositories where it has the right run, and claims there. The server reads the waiting runs in Postgres every 0.5 s for at most 20 s (index runs_queued, migration 025_runs_queued.sql); NATS still delivers each run. Before this, a runner polled the repositories one after another, so a new run waited up to 20 s for its turn: the images of desk waited 16 s each on 2026-10-25. A server without the route answers 404, and the runner then takes the repositories in turn.

A runner of the lane base serves the repositories with base set (GET /v1/runner/repos?lane=base). When the server queues a Dockerfile image of a repository that is not in that lane, it puts the repository in the lane and gives the runners the right run (src/server/lanes.ts), so the build never waits for a lane that no runner serves. On 2026-10-25 the images of axon waited for 3 hours: its move did not set the lane.

An image step with a base gets the secrets that it names as files: the runner writes each one to a file of the run and gives build its path in DESK_BUILD_SECRET_<ID>_FILE. build gets the landed change in DESK_IMAGE_REVISION, for a binary that stamps its commit; changeArg names the variable of the image that holds it.

Deploys: the Deployments follow the pushed digests (#115)

deploys in .desk/config.json lists the targets. Each entry has name, namespace, deployments (names, or withContainer with an optional except), containers (a container name and the name of an image of images) and timeoutMinutes (default 15).

1. After each run that ends, the server looks at each target (advanceDeploys, src/server/deploys.ts). A target waits while a build or a promotion of one of its images is queued, runs or waits for a check, so one landing rolls each Deployment once.
2. The deploy run sets each container to the newest digest that a build of its image pushed, or the digest that its promotion set (#145). It is a run of kind deploy (migration 016_deploys.sql); its run id is the target and the images, so the same digests are never deployed twice, and one deploy of a target waits or runs at a time.
3. A deploy run waits in its own lane of the queue (<stream>.<repo id>.deploy, the consumer deployers-<repo id>). Only a runner with DESK_RUNNER_LANE=deploy takes it. That runner runs no code of the repository, so the token of its service account is never in a pod that runs a check: the sandbox of a check does not isolate files (src/server/sandbox.ts).
4. The runner (src/server/deploy-run.ts):
   • reads the Deployments of the namespace, and the containers whose image is not the one of the run;
   • starts one pod on each node that runs a pod of such a Deployment, with each new image and a command that does not exist: the kubelet pulls each image, the pod runs nothing, and the runner deletes it. A pull that fails changes no Deployment;
   • with canary (container, command, timeoutMinutes, default 5), starts one pod with the pod template of the first Deployment that changes, on the node of its pod and with its volumes, with only the container container on its new image and command as its command, no probe, no port, no init container, and TMPDIR in an empty folder. A pod that does not succeed in time changes no Deployment, and its last 40 log lines are in the run log. munbot runs munbot db check-migrations: the migrations of the new image on a VACUUM INTO copy of the tenant database (2026-10-23, an image without migration 0111 stopped six tenants);
   • patches each Deployment once with all its new images (application/strategic-merge-patch+json), so a Deployment that recreates its pod stops once, and not during a pull;
   • waits until each rollout has the new generation, all pods updated and available, or its time limit ends. A Deployment whose new pod is in CrashLoopBackOff fails at once; a Deployment that fails or does not end in time gets its old images back in one patch (on 2026-10-23 six tenants crashed on a new image and waited for a rollback by hand).
5. A deploy that passed writes a row of deploys for each Deployment that it set: the change of the newest landing whose image it set, <namespace>/<Deployment>, and container=<image>@<digest>.

The service account of the deploy runner needs a Role in the namespace of the target: get, list, patch on deployments, get, list, create, delete on pods, get on pods/log, and list on events. The runner starts only with that token (KUBERNETES_SERVICE_HOST, /var/run/secrets/kubernetes.io/serviceaccount).

Releases: a new version is tagged after its checks

release in .desk/config.json makes the server record the release of a landing whose version is new:

"release": { "versionFile": "packages/cli/package.json", "tagPrefix": "v", "needs": ["unit"] }

1. versionFile is a package.json (its version), a Cargo.toml (the version of [package]) or a text file that holds only the version.
2. tagPrefix defaults to v. needs defaults to every check of validation that runs on main.
3. After each landing and after each run that ends, advanceRelease (src/server/auto-release.ts) reads the version at the head of main. When no release has the tag, and each check of needs passed on that landing, it records the release with the section ## [<version>] of CHANGELOG.md as its notes. The author is desk.
4. The mirror pushes the tag to the copy at its next round. A repository that publishes from GitHub workflows on a tag (desk repo move --keep-actions) releases with no step of a person.

A landing with the same version records nothing, and a check that failed blocks the release of that landing. test/server-auto-release.test.mjs covers the config, the version files, the notes and the landings.

The runner

Two images run a runner. ghcr.io/munhq/desk-server (the stage runtime of the Dockerfile) has Node, git, Buck2 and buildctl. ghcr.io/munhq/desk-runner-rust (the stage runtime-rust, on Debian trixie for the Python 3.12 of the Buck2 prelude) adds gcc, g++, lld, python3 and the Rust toolchain of munbot (nightly-2026-10-23, rustc 1.101.0-nightly (7189d39d9 2026-10-22)), installed with a rustup that is checked by its hash. A repository with another toolchain sets the build argument RUST_TOOLCHAIN. The check of munbot passed in it on 2026-10-23: cold, with no cache, 6 min 44 s for 3,234 actions.

desk runner <server> <owner/repo> [--token-env NAME] [--dir DIR] [--once] and node dist/server/runner-main.js (the pod, with the variables below) take a step, write its tree into an empty folder, run the command with sh -c in that folder, and report the exit code, the time and the log. The same program runs as a pod on hotmun and on any other machine with Node 20 or later, started on demand (decision of the owner, 2026-10-17).

Variable	Use
DESK_SERVER	The address of the server. Required.
DESK_REPOS	The repositories, owner/repo, separated by commas. The runners of hotmun are one shared pool: each runner claims from each repository in turn, and each claim waits its share of DESK_RUNNER_WAIT_MS (at least 2 s), so a step of any repository is taken within about one wait. The identity of the runner needs the right run on each repository.
DESK_REPO	One repository, owner/repo. With neither DESK_REPOS nor DESK_REPO, a runner of the lane runs or base reads the repositories of its lane from the server every minute (GET /v1/runner/repos, #125), so a moved repository needs no change of the deployment. A deploy runner needs DESK_REPO.
DESK_RUNNER_TOKEN	The token of an identity with the right run. Required.
DESK_RUNNER_DIR	The folder of the scratch folders of the runs. Required.
DESK_RUNNER_NPM_CACHE	The npm cache of the commands; default DESK_RUNNER_DIR/npm-cache.
DESK_RUNNER_WAIT_MS	How long one claim waits on the server; default and most 20000.
DESK_RUNNER_LOG_BYTES	The largest log of a run; default 2 MiB.
DESK_RUNNER_CACHE	The remote cache of the build, grpc://host:port (NativeLink); each step gets it as DESK_CACHE_ADDRESS. desk runner --cache sets it.
DESK_RUNNER_LANE	runs (default) or deploy. A deploy runner takes only the deploy runs (#115) and needs the service account token of its pod.

Before the command, the runner writes the code tree of the run into an empty checkout. The objects of the tree stay in a cache on the volume of the runner (objects/, at most 4 GiB; src/server/object-cache.ts): an object id is the SHA-256 of its bytes, so the runner checks each object that it reads, and fetches only the objects that the cache does not hold, 16 at a time. Without the cache, the 2,047 files of munbot took 26 s of each run.

A command that exits with 75 (EX_TEMPFAIL) reports that its infrastructure failed, not the code that it checks: the run ends as an error, and a bisect stops at it and reverts nothing, as it does for a time limit. The check of munbot exits with 75 when Buck2 reports an infrastructure error (exit 2), for example a remote cache that does not answer; a compile error (exit 3) or a failed test (exit 32) stays a failure.

The sandbox (src/server/sandbox.ts) is the process level only: the command runs in its own process group, a time limit or a log limit kills the group, and the command gets only PATH, HOME, TMPDIR, LANG, CI, the npm cache and the toolchain folders of the image (RUSTUP_HOME, CARGO_HOME). A command with a new HOME and no RUSTUP_HOME made rustup install the toolchain of the repository again in each run. It does not isolate the network or the file system outside the scratch folder. A runner runs only the code of a repository that the owner trusts.

The run history in ClickHouse (#65)

Postgres keeps the record of a run while it waits and runs. When a run ends, the server writes its history to ClickHouse every 10 seconds (src/server/run-history.ts) and marks the run with history_at. A ClickHouse that does not answer stops neither the server nor the runs: the pass is in the log, and the next pass tries again. The tables are ReplacingMergeTree on their sorting key, so a row that a pass writes twice stays one row (read with FINAL).

Table	One row for	Columns
steps	each run that ended: a check, a bisect run, an image build, a promotion or a deploy	the repository, the run, the name, the kind, the status and the reason, the exit code, the runner, the landing (seq, change) and whether it was a bisect run, the times of the queue, the start and the end, and wait_ms
tests	each test that a step reported	the run, the step, the test, the status, the time, and whether the cache gave it
runs	each landing whose checks all ended: the checks of a landing as one run	the landing, the count of checks and of failed checks, the status, and the time from the first queue to the last end

A step reports its tests in the file DESK_TEST_REPORT that the runner gives it: one JSON object for each line, {"test", "status", "durationMs", "cached"}, the status passed, failed or skipped. The runner sends at most 20,000 tests, and names in the log the lines that are not a test. Postgres holds them in runs.tests until they are in ClickHouse. The check of desk writes one line for each Buck2 test action, from buck2 log what-ran (scripts/test-report.mjs): failed when what-ran --failed lists it, cached when the executor is Cache.

The cache of a step is NativeLink (#97): the key of an action is the hash of its command, its environment (with the versions of the tools) and its declared inputs.

The tables are made at the start of the server, ON CLUSTER of DESK_HISTORY_CLUSTER on the storage policy of DESK_HISTORY_STORAGE_POLICY. On hotmun the policy hot_cold of the cluster has the volume hot on the local disk and the volume cold on the S3 disk (the prefix tiered/ of the bucket hotmun-gra1-clickhouse-cold). The TTL of each table follows the settings at each start:

• DESK_HISTORY_HOT_DAYS: a part moves to the volume cold after these days (TTL toDateTime(finished_at) + INTERVAL <n> DAY TO VOLUME 'cold'). The table stays one table, and a query reads both volumes.
• DESK_HISTORY_RETENTION_DAYS: a row is deleted after these days (… DELETE). 0, the default, deletes nothing, as git keeps its history (the owner, 2026-10-17).

A part on the S3 volume is not a backup, because a dropped table removes its parts. The backup of the cluster runs every day to the bucket hotmun-gra1-clickhouse-backups (clickhouse-backup, chart clickhouse of munhq/k3s-gpu); it holds the database desk with the others.

scripts/test-run-history.sh runs test/server-run-history.test.mjs against ClickHouse 26.8.27.9 in Docker, the version of hotmun, with a policy hot_cold of two local disks.

The server settings

Variable	Use
DESK_RUNNER_TOKEN	The token of the runners of the cluster (Vault managed/desk/runner). The server keeps the identity desk-runner on it; a new token replaces the old one.
DESK_RUNNER_REPOS	The repositories of those runners, owner/repo separated by commas. desk-runner gets the right run on each that exists; the log names a missing one.
NATS_URL	The NATS of the queue with its login, nats://user:pass@host:4222; the Secret desk-nats of the platform-data registry gives it. Without it the queue is in memory.
DESK_NATS_STREAM	The stream; default DESK_RUNS.
DESK_NATS_REPLICAS	The replicas of the stream, 1 to 5; default 1.
CLICKHOUSE_URL, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_DATABASE	The ClickHouse of the run history; the Secret desk-clickhouse of the registry gives them. Without CLICKHOUSE_URL the history stays in Postgres.
DESK_HISTORY_CLUSTER, DESK_HISTORY_STORAGE_POLICY	The cluster of the replicated tables and their storage policy (main and hot_cold on hotmun).
DESK_HISTORY_HOT_DAYS, DESK_HISTORY_RETENTION_DAYS	The days before a part moves to the volume cold, and before a row is deleted; 0 is no rule.

Tests

test/server-runs.test.mjs runs the routes and the runner against PGlite and the memory queue: the run id, the needs, a step that runs again after it passed, two runners on one step, a step that comes back after the ack wait, a second message of a run, the publish again at a start, a refused publish, the end after three attempts, a claim that waits for a step, the rights, the outcomes of the runner (pass, fail, timeout, output limit, a link out of the checkout), a change that lands and runs node --test on its tree, the checks that a landing queues from .desk/config.json (only for changed paths, with the whole tree as input), one refused check among others, a bad config, the right merge for a step, the identity desk-runner, a restart with the memory queue, and desk runner --once. With DESK_TEST_NATS_URL set, one more test runs the claim, the heartbeats, the return of a step of a silent runner and the acknowledgement on a real NATS JetStream; on 2026-10-17 it passed on nats:2.15.0-alpine, the version of hotmun.