Lowest median time
11 s
Self-hosted runners on Google Cloud
GitHub-hosted (ubuntu-latest): 23 s
Slowest median
70 s
Self-hosted runners on AWS (one VM per job)
6.4× the lowest
Lowest cost, managed
$0.0004
Vitko Runners (warm starts + test splitting)
mean per job
Failed runs
0
no failed runs
of 270 measured runs
Median time
Seconds from start to finish · lower is betterTable
Ordered by median time. Self-hosted options are priced all-in at 100,000 jobs a month. Green marks the lowest median time and the lowest cost among the managed runners.| Configuration | Median | p95 | Queue p50 | Queue p95 | Cost | OK |
|---|---|---|---|---|---|---|
| Self-hosted runners on Google Cloud | 11 s | 32 s | 1 s | 2 s | $0.0011 | 30 of 30 |
| Vitko Runners (warm starts + test splitting) | 13 s | 14 s | 1 s | 2 s | $0.0004 | 30 of 30 |
| Vitko Runners | 16 s | 20 s | 1 s | 5.1 s | $0.0005 | 30 of 30 |
| GitHub-hosted larger runner (4 vCPU) | 21 s | 24 s | 3 s | 3 s | $0.012 | 30 of 30 |
| Blacksmith | 22 s | 36 s | 8 s | 10 s | $0.0040 | 30 of 30 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 23 s | 28 s | 3 s | 3 s | $0.0060 | 30 of 30 |
| Ubicloud | 36 s | 46 s | 18 s | 23.5 s | $0.0013 | 30 of 30 |
| RunsOn | 38 s | 40 s | 19 s | 21 s | $0.0014 | 30 of 30 |
| Self-hosted runners on AWS (one VM per job) | 70 s | 134 s | 50 s | 106.1 s | $0.0036 | 30 of 30 |
The job
- Project: pallets/flask at
3.1.3(22d924701a6ae2e4cd01e9a15bbaf3946094af65), licence BSD-3-Clause. Fetched at run time, not vendored. - Upstream job:
.github/workflows/tests.yaml, job tests (Python 3.13). - Benchmark workflow:
.github/workflows/wl-flask.yml.
Portability edits (identical for every provider)
- One matrix cell (Python 3.13).
suite/flask holds the workflow and notes.