Short answer
In CI Bench v1, every configuration completed at least 357 of 360 measured runs: failed runs were Vitko Runners (warm starts + test splitting) 3, Vitko Runners 0, GitHub-hosted larger runner (4 vCPU) 3, Blacksmith 3, Self-hosted runners on Google Cloud 1, RunsOn 1, Self-hosted runners on AWS (one VM per job) 0, Ubicloud 1, GitHub-hosted runners (ubuntu-latest, 2 vCPU) 3. Failures are counted and published, never replaced. It is one result set of 360 runs per configuration over three sessions, so it says little about rare outages; each failed run is listed in the changelog. Maintained by Vitko, Inc., which sells one of the products measured.
Failed runs by configuration
| Configuration | Failed runs | Share |
|---|---|---|
| Vitko RunnersVitko | 0 of 360 | 0.0% |
| Self-hosted runners on AWS (one VM per job) | 0 of 360 | 0.0% |
| Self-hosted runners on Google Cloud | 1 of 360 | 0.3% |
| RunsOn | 1 of 360 | 0.3% |
| Ubicloud | 1 of 360 | 0.3% |
| Vitko Runners (warm starts + test splitting)Vitko | 3 of 360 | 0.8% |
| GitHub-hosted larger runner (4 vCPU) | 3 of 360 | 0.8% |
| Blacksmith | 3 of 360 | 0.8% |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 3 of 360 | 0.8% |
Every failed run, with where it happened and why where known, is in the changelog. Small counts: a difference of one or two runs is not evidence of a reliability gap.
More: all questions answered · 12 options compared · ranked by speed · ranked by cost per job · burst workload.
Method: how the runs were made; data: results.json; every raw record: the repository. Maintained by Vitko, Inc., which sells one of the products measured. Page dated October 28, 2026.