Lowest median time
404 s
Vitko Runners
GitHub-hosted (ubuntu-latest): 884 s
Slowest median
884 s
GitHub-hosted runners (ubuntu-latest, 2 vCPU)
2.2× the lowest
Lowest cost, managed
$0.014
Vitko Runners
mean per job
Failed runs
1
GitHub-hosted larger runner (4 vCPU) 1
of 270 measured runs
Median time
Seconds from start to finish · lower is betterTable
Ordered by median time. Self-hosted options are priced all-in at 100,000 jobs a month. Green marks the lowest median time and the lowest cost among the managed runners.| Configuration | Median | p95 | Queue p50 | Queue p95 | Cost | OK |
|---|---|---|---|---|---|---|
| Vitko Runners | 404 s | 441 s | 1.5 s | 2 s | $0.014 | 30 of 30 |
| Vitko Runners (warm starts + test splitting) | 405 s | 425 s | 2 s | 2 s | $0.014 | 30 of 30 |
| Blacksmith | 448 s | 748 s | 8 s | 10 s | $0.034 | 30 of 30 |
| GitHub-hosted larger runner (4 vCPU) | 476 s | 511 s | 3 s | 3.6 s | $0.099 | 29 of 30 |
| Self-hosted runners on AWS (one VM per job) | 512 s | 598 s | 52 s | 93.3 s | $0.049 | 30 of 30 |
| Self-hosted runners on Google Cloud | 581 s | 625 s | 1 s | 2 s | $0.064 | 30 of 30 |
| RunsOn | 623 s | 662 s | 18 s | 23.1 s | $0.010 | 30 of 30 |
| Ubicloud | 682 s | 881 s | 18.5 s | 26 s | $0.015 | 30 of 30 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 884 s | 934 s | 3 s | 4 s | $0.087 | 30 of 30 |
The job
- Project: openai/codex at
rust-v0.159.3(01fc69f4026735edfdf6789820549727a4867b11), licence Apache-2.0. Fetched at run time, not vendored. - Upstream job:
.github/workflows/rust-ci-full.yml, job lint_build (ubuntu-24.04, x86_64-unknown-linux-gnu, dev). - Benchmark workflow:
.github/workflows/wl-codex-lint.yml.
Portability edits (identical for every provider)
- Upstream runs this on its own larger runner group; here it runs on the provider under test.
- Two extra apt packages (libglib2.0-dev, libdbus-1-dev) that upstream's runner image has.
codex-voice-hostis excluded (needs GStreamer 1.28+, absent from stock Ubuntu 24.04).- No
-D warnings, so the job ends green; the compile work is the same. - Needs more than 8 GB of memory without swap (GitHub's runner has 4 GB of swap). Providers without swap run it at their next memory size, priced there.
- The
kernel.unprivileged_userns_clonesysctl in upstream's setup action runs only where the kernel has it.
suite/codex-lint holds the workflow and notes.