Lowest median time
97 s
Blacksmith
GitHub-hosted (ubuntu-latest): 259 s
Slowest median
259 s
GitHub-hosted runners (ubuntu-latest, 2 vCPU)
2.7× the lowest
Lowest cost, managed
$0.052
Vitko Runners (warm starts + test splitting)
for all 20 jobs
Failed runs
0
no failed runs
of 270 measured runs
Median time
Seconds from start to finish · lower is betterTable
Ordered by median time. Time and cost cover all 20 jobs; queue time is per job. Self-hosted options are priced all-in at 100,000 jobs a month. Green marks the lowest median time and the lowest cost among the managed runners.| Configuration | Median | p95 | Queue p50 | Queue p95 | Cost | OK |
|---|---|---|---|---|---|---|
| Blacksmith | 97 s | 190 s | 9 s | 16 s | $0.166 | 30 of 30 |
| Vitko Runners (warm starts + test splitting) | 131 s | 191 s | 17.3 s | 115.1 s | $0.052 | 30 of 30 |
| Self-hosted runners on Google Cloud | 153 s | 165 s | 2 s | 2.5 s | $0.135 | 30 of 30 |
| GitHub-hosted larger runner (4 vCPU) | 154 s | 179 s | 3 s | 38.6 s | $0.636 | 30 of 30 |
| Vitko Runners | 160 s | 372 s | 18 s | 294.6 s | $0.056 | 30 of 30 |
| Self-hosted runners on AWS (one VM per job) | 161 s | 182 s | 65.3 s | 92.6 s | $0.111 | 30 of 30 |
| RunsOn | 177 s | 340 s | 22.5 s | 193.6 s | $0.051 | 30 of 30 |
| Ubicloud | 203 s | 271 s | 18 s | 127.5 s | $0.062 | 30 of 30 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 259 s | 283 s | 3 s | 31.3 s | $0.500 | 30 of 30 |
The job
- Project: psf/black at
26.5.1(87928e6d6761a4a6d22250e1fee5601b3998086e), licence MIT. Fetched at run time, not vendored. - Upstream job:
.github/workflows/test.yml, job test (3.13, ubuntu-latest). - Benchmark workflow:
.github/workflows/wl-black.yml.
Portability edits (identical for every provider)
- One matrix cell (Python 3.13, Linux); Coveralls upload runs only in upstream's repository.
- Dispatched 20 times at once: the measured time is from the first job created to the last job finished.
suite/burst holds the workflow and notes.