Lowest median time
140 s
Vitko Runners (warm starts + test splitting)
GitHub-hosted (ubuntu-latest): 368 s
Slowest median
368 s
GitHub-hosted runners (ubuntu-latest, 2 vCPU)
2.6× the lowest
Lowest cost, managed
$0.0078
Ubicloud
mean per job
Failed runs
0
no failed runs
of 270 measured runs
Median time
Seconds from start to finish · lower is betterTable
Ordered by median time. Self-hosted options are priced all-in at 100,000 jobs a month. Green marks the lowest median time and the lowest cost among the managed runners.| Configuration | Median | p95 | Queue p50 | Queue p95 | Cost | OK |
|---|---|---|---|---|---|---|
| Vitko Runners (warm starts + test splitting) | 140 s | 145 s | 1 s | 2 s | $0.012 | 30 of 30 |
| GitHub-hosted larger runner (4 vCPU) | 192 s | 198 s | 3 s | 3 s | $0.045 | 30 of 30 |
| Vitko Runners | 272 s | 326 s | 1 s | 2 s | $0.0097 | 30 of 30 |
| Blacksmith | 286 s | 323 s | 8 s | 9.5 s | $0.020 | 30 of 30 |
| Self-hosted runners on Google Cloud | 310 s | 326 s | 1 s | 2 s | $0.017 | 30 of 30 |
| RunsOn | 327 s | 334 s | 18 s | 21 s | $0.0041 | 30 of 30 |
| Self-hosted runners on AWS (one VM per job) | 357 s | 398 s | 68 s | 104.7 s | $0.014 | 30 of 30 |
| Ubicloud | 359 s | 429 s | 20 s | 25.1 s | $0.0078 | 30 of 30 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 368 s | 382 s | 3 s | 3.5 s | $0.039 | 30 of 30 |
The job
- Project: cli/cli at
v2.102.0(fc4b137cdef0a6bd28fd461b7cf9c84a5812a8cd), licence MIT. Fetched at run time, not vendored. - Upstream job:
.github/workflows/go.yml, job build (ubuntu-latest). - Benchmark workflow:
.github/workflows/wl-cli.yml.
Portability edits (identical for every provider)
- One matrix cell (Linux).
GOFLAGS=-count=1: the same commit runs every time, so Go's test-result cache would skip every test. Module and build caches are kept.- Track B (split tests) only: the attestation tests that call the public Sigstore services (the integration-tagged tests under
pkg/cmd/attestation:TestLiveSigstoreVerifier*,TestVerifyAttestations,TestVerifyIntegration*,TestNewInspectCmd_PrintOutputJSONFormat,TestChooseVerifier*) are skipped in the split step and run unsplit in the following step of the same job, because split parts have no network. The job runs the same tests as the unsplit arms.
suite/cli holds the workflow and notes.