Self-hosted options spread fixed monthly costs over the monthly volume, so their cost per job changes with it; see the volume table.
Speed
× faster than GitHub-hosted (ubuntu-latest) · higher is betterCost per job
USD per $1.00 spent on GitHub-hosted (ubuntu-latest), at 100,000 jobs a month · lower is betterWorkload by workload
Vitko Runners had the lowest median time of the two on 10 of 12 workloads, Self-hosted runners on Google Cloud on 1.| Workload | Vitko Runners (warm starts + test splitting) | Vitko Runners | Self-hosted runners on Google Cloud | Lowest median |
|---|---|---|---|---|
| Burst: 20 jobs at once | 131 s | 160 s | 153 s | Vitko Runners (warm starts + test splitting) |
| cli | 140 s | 272 s | 310 s | Vitko Runners (warm starts + test splitting) |
| codex-lint | 405 s | 404 s | 581 s | Vitko Runners |
| container | 178 s | 179 s | 178 s | Vitko Runners (warm starts + test splitting) |
| fd | 45 s | 49 s | 64 s | Vitko Runners (warm starts + test splitting) |
| flask | 13 s | 16 s | 11 s | Self-hosted runners on Google Cloud |
| guava | 651 s | 655 s | 820 s | Vitko Runners (warm starts + test splitting) |
| monorepo | 172 s | 180 s | 278 s | Vitko Runners (warm starts + test splitting) |
| pydantic | 69 s | 105 s | 119 s | Vitko Runners (warm starts + test splitting) |
| ts | 243 s | 372 s | 533 s | Vitko Runners (warm starts + test splitting) |
| vite | 199 s | 350 s | 401 s | Vitko Runners (warm starts + test splitting) |
| vue | 32 s | 63 s | 76 s | Vitko Runners (warm starts + test splitting) |
Cost by volume
Mean cost per job in USD, from the benchmark’s sensitivity results| Volume | Vitko Runners (warm starts + test splitting) | Vitko Runners | Self-hosted runners on Google Cloud |
|---|---|---|---|
| 100,000 jobs a month | $0.158 | $0.157 | $0.409 |
| 10,000 jobs a month | $0.158 | $0.157 | $0.409 |
| 1,000 jobs a month | $0.158 | $0.157 | $0.409 |
Failed runs
Counted, never replaced- Vitko Runners (warm starts + test splitting)
- 3 of 360 measured runs (0.8%)
- Vitko Runners
- 0 of 360 measured runs (0.0%)
- Self-hosted runners on Google Cloud
- 1 of 360 measured runs (0.3%)
Each configuration page lists where its failures happened. All losses.
All results for this pair
One line per configuration pairing, generated from the benchmark data- Vitko Runners (warm starts + test splitting) ran jobs 1.5× faster than Self-hosted runners on Google Cloud (95% interval 1.5× to 1.6×); Vitko Runners (warm starts + test splitting) cost 61% less per job (at 100,000 jobs a month).
- Vitko Runners ran jobs 1.2× faster than Self-hosted runners on Google Cloud (95% interval 1.1× to 1.2×); Vitko Runners cost 62% less per job (at 100,000 jobs a month).