Lowest median time
69 s
Vitko Runners (warm starts + test splitting)
GitHub-hosted (ubuntu-latest): 179 s
Slowest median
179 s
GitHub-hosted runners (ubuntu-latest, 2 vCPU)
2.6× the lowest
Lowest cost, managed
$0.0022
Vitko Runners (warm starts + test splitting)
mean per job
Failed runs
0
no failed runs
of 270 measured runs
Median time
Seconds from start to finish · lower is betterTable
Ordered by median time. Self-hosted options are priced all-in at 100,000 jobs a month. Green marks the lowest median time and the lowest cost among the managed runners.| Configuration | Median | p95 | Queue p50 | Queue p95 | Cost | OK |
|---|---|---|---|---|---|---|
| Vitko Runners (warm starts + test splitting) | 69 s | 119 s | 2.5 s | 47.9 s | $0.0022 | 30 of 30 |
| Blacksmith | 103 s | 145 s | 8 s | 11 s | $0.0087 | 30 of 30 |
| Vitko Runners | 105 s | 140 s | 2 s | 37.8 s | $0.0034 | 30 of 30 |
| Self-hosted runners on Google Cloud | 119 s | 129 s | 2 s | 2.5 s | $0.0065 | 30 of 30 |
| RunsOn | 147 s | 152 s | 18.5 s | 22 s | $0.0024 | 30 of 30 |
| GitHub-hosted larger runner (4 vCPU) | 154 s | 173 s | 3 s | 4 s | $0.033 | 30 of 30 |
| Self-hosted runners on AWS (one VM per job) | 154 s | 184 s | 49.5 s | 67.8 s | $0.0064 | 30 of 30 |
| Ubicloud | 159 s | 204 s | 18.5 s | 27.6 s | $0.0037 | 30 of 30 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 179 s | 196 s | 3 s | 4 s | $0.020 | 30 of 30 |
The job
- Project: pydantic/pydantic at
v2.13.5(001dea020e0809844e5b17666432c9135a976f46), licence MIT. Fetched at run time, not vendored. - Upstream job:
.github/workflows/ci.yml, job Test ubuntu-latest / 3.13. - Benchmark workflow:
.github/workflows/wl-pydantic.yml.
Portability edits (identical for every provider)
- One matrix cell (Linux, Python 3.13).
- The coverage artifact upload is omitted (it feeds a separate combine job).
ulimit -s 16384before the tests: GitHub's runner image sets a 16 MiB stack limit, stock Ubuntu 8 MiB, where a recursion test overflows.
suite/pydantic holds the workflow and notes.