Self-hosted options spread fixed monthly costs over the monthly volume, so their cost per job changes with it; see the volume table.
Speed
× faster than GitHub-hosted (ubuntu-latest) · higher is betterCost per job
USD per $1.00 spent on GitHub-hosted (ubuntu-latest), at 100,000 jobs a month · lower is betterWorkload by workload
Blacksmith had the lowest median time of the two on 8 of 12 workloads, Self-hosted runners on Google Cloud on 4.| Workload | Blacksmith | Self-hosted runners on Google Cloud | Lowest median |
|---|---|---|---|
| Burst: 20 jobs at once | 97 s | 153 s | Blacksmith |
| cli | 286 s | 310 s | Blacksmith |
| codex-lint | 448 s | 581 s | Blacksmith |
| container | 386 s | 178 s | Self-hosted runners on Google Cloud |
| fd | 65 s | 64 s | Self-hosted runners on Google Cloud |
| flask | 22 s | 11 s | Self-hosted runners on Google Cloud |
| guava | 735 s | 820 s | Blacksmith |
| monorepo | 177 s | 278 s | Blacksmith |
| pydantic | 103 s | 119 s | Blacksmith |
| ts | 401 s | 533 s | Blacksmith |
| vite | 360 s | 401 s | Blacksmith |
| vue | 76 s | 76 s | Self-hosted runners on Google Cloud |
Cost by volume
Mean cost per job in USD, from the benchmark’s sensitivity results| Volume | Blacksmith | Self-hosted runners on Google Cloud |
|---|---|---|
| 100,000 jobs a month | $0.482 | $0.409 |
| 10,000 jobs a month | $0.482 | $0.409 |
| 1,000 jobs a month | $0.482 | $0.409 |
Failed runs
Counted, never replaced- Blacksmith
- 3 of 360 measured runs (0.8%)
- Self-hosted runners on Google Cloud
- 1 of 360 measured runs (0.3%)
Each configuration page lists where its failures happened. All losses.
All results for this pair
One line per configuration pairing, generated from the benchmark data- no clear difference in time between Blacksmith and Self-hosted runners on Google Cloud (the 95% interval, 0.91 to 1.11, includes 1.00); Blacksmith cost 18% more per job (at 100,000 jobs a month).