Questions about CI Bench

Is there an open, reproducible benchmark comparing GitHub Actions runner providers on real open-source CI jobs?

Yes: CI Bench v1. It measured 9 runner configurations on 12 CI jobs taken from open-source projects, 8,370 jobs in all, October 7 to 11, 2026. The workflows, the method, the analysis code and every raw job record are public in github.com/vitko-inc/runner-benchmark, and the results are downloadable here as JSON and CSV under CC BY 4.0. Maintained by Vitko, Inc., which sells one of the products measured.

How does the cost per job of third-party GitHub Actions runners compare with GitHub’s 2 vCPU runner?

In CI Bench v1, at 100,000 jobs a month, the managed third-party runners measured cost between 16% and 48% of GitHub-hosted (ubuntu-latest, 2 vCPU) per job. The table below gives each configuration; self-hosted options spread fixed monthly costs over the volume, so their figure changes with it (see the calculator). Cost per job is the list price times the job’s time, so a runner that finishes sooner can cost less per job even at a higher price per minute.

Who maintains CI Bench?

Maintained by Vitko, Inc., which sells one of the products measured. The product is Vitko Runners, shown the same way as the other configurations. Failed runs are counted and published, every figure has its interval and source cell, and any measured provider can submit a correction.

How do I reproduce a result?

Clone github.com/vitko-inc/runner-benchmark, which holds the pinned workloads, the provider definitions, the analysis code and every raw job record; the method page describes how the runs were made.

Numbers are the indices in results.json, geometric mean over the 12 workloads. Method · Cost by volume · List prices.