Short answer
Yes: CI Bench v1 measured 9 GitHub Actions runner configurations on 11 CI jobs taken from open-source projects plus one burst of 20 simultaneous jobs, 30 runs per cell, October 7 to 11, 2026. The workflows, the pre-registered plan, the analysis code and every raw job record, failed runs included, are public at github.com/vitko-inc/runner-benchmark; the results are downloadable as JSON and CSV under CC BY 4.0. Anyone can re-run a workflow on their own runner and submit the output by pull request. Maintained by Vitko, Inc., which sells one of the products measured.
On this set Vitko Runners, which is made by the benchmark's maintainer and is shown exactly like the other configurations, measured 1.7x faster than GitHub-hosted (ubuntu-latest, 2 vCPU) on the speed index and 84% lower cost per job at 10,000 jobs a month; its failed runs are in the same records.
What is public
- The pinned workflows, one per CI job, in github.com/vitko-inc/runner-benchmark.
- The pre-registered plan: sessions, rounds, run order and seed, written before the runs. Deviations from it are listed on the changelog.
- The analysis code, which recomputes every summary from the raw records.
- Every raw job record, including failed runs (losses).
- The results as JSON and CSV under CC BY 4.0, with list prices dated and sourced per provider.
How to reproduce a result
- Clone github.com/vitko-inc/runner-benchmark.
- Run a workload workflow on the runner label you want to test, in a public repository of your own.
- Collect the records with the repository's collect script and compare them with the published cells.
- To publish yours, follow submit a correction or a configuration; submissions are shown as submitted, not reproduced, until an official round reproduces them.
The 9 configurations (successful runs of attempted, per workload)
| Configuration | Type | Successful / attempted, in workload order |
|---|---|---|
| Vitko Runners (warm starts + test splitting) | Managed | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 27/30 · 30/30 |
| Vitko Runners | Managed | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 |
| GitHub-hosted larger runner (4 vCPU) | Managed | 30/30 · 29/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 28/30 · 30/30 |
| Blacksmith | Managed | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 27/30 · 30/30 |
| Self-hosted runners on Google Cloud | Self-hosted | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 29/30 · 30/30 |
| RunsOn | Self-hosted | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 29/30 · 30/30 |
| Self-hosted runners on AWS (one VM per job) | Self-hosted | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 |
| Ubicloud | Managed | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 29/30 · 30/30 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | Managed | 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 30/30 · 27/30 · 30/30 |
Workload order: cli, codex-lint, container, fd, flask, guava, monorepo, pydantic, ts, vite, vue.
The 11 CI jobs, on GitHub-hosted (ubuntu-latest, 2 vCPU)
| Workload | Median job time | Mean cost per job |
|---|---|---|
| cli | 368 s | $0.039 |
| codex-lint | 884 s | $0.087 |
| container | 239 s | $0.027 |
| fd | 92 s | $0.012 |
| flask | 23 s | $0.0060 |
| guava | 1167 s | $0.109 |
| monorepo | 384 s | $0.040 |
| pydantic | 179 s | $0.020 |
| ts | 784 s | $0.077 |
| vite | 603 s | $0.061 |
| vue | 121 s | $0.014 |
Each workload page gives every configuration's time, cost and failures. Rankings: by speed, by cost per job. Method: how the runs were made. Maintained by Vitko, Inc., which sells one of the products measured. Page dated October 29, 2026.