CI Bench

Head to head · CI Bench v1

GitHub-hosted larger runner (4 vCPU) vs GitHub-hosted (ubuntu-latest)

GitHub-hosted larger runner (4 vCPU) ran jobs 1.6× faster than GitHub-hosted runners (ubuntu-latest, 2 vCPU) (95% interval 1.5× to 1.6×); GitHub-hosted larger runner (4 vCPU) cost 39% more per job (at 100,000 jobs a month).

Speed

× faster than GitHub-hosted (ubuntu-latest) · higher is better
GitHub-hosted larger runner (4 vCPU)
1.6×
GitHub-hosted (ubuntu-latest)
1.0×

Cost per job

USD per $1.00 spent on GitHub-hosted (ubuntu-latest), at 100,000 jobs a month · lower is better
GitHub-hosted larger runner (4 vCPU)
$1.39
GitHub-hosted (ubuntu-latest)
$1.00

Workload by workload

GitHub-hosted larger runner (4 vCPU) had the lowest median time of the two on 12 of 12 workloads, GitHub-hosted (ubuntu-latest) on 0.
WorkloadGitHub-hosted larger runner (4 vCPU)GitHub-hosted runners (ubuntu-latest, 2 vCPU)Lowest median
Burst: 20 jobs at once154 s259 sGitHub-hosted larger runner (4 vCPU)
cli192 s368 sGitHub-hosted larger runner (4 vCPU)
codex-lint476 s884 sGitHub-hosted larger runner (4 vCPU)
container185 s239 sGitHub-hosted larger runner (4 vCPU)
fd66 s92 sGitHub-hosted larger runner (4 vCPU)
flask21 s23 sGitHub-hosted larger runner (4 vCPU)
guava946 s1167 sGitHub-hosted larger runner (4 vCPU)
monorepo220 s384 sGitHub-hosted larger runner (4 vCPU)
pydantic154 s179 sGitHub-hosted larger runner (4 vCPU)
ts435 s784 sGitHub-hosted larger runner (4 vCPU)
vite288 s603 sGitHub-hosted larger runner (4 vCPU)
vue67 s121 sGitHub-hosted larger runner (4 vCPU)

Cost by volume

Mean cost per job in USD, from the benchmark’s sensitivity results
VolumeGitHub-hosted larger runner (4 vCPU)GitHub-hosted runners (ubuntu-latest, 2 vCPU)
100,000 jobs a month$1.387$1.000
10,000 jobs a month$1.387$1.000
1,000 jobs a month$1.387$1.000

Failed runs

Counted, never replaced
GitHub-hosted larger runner (4 vCPU)
3 of 360 measured runs (0.8%)
GitHub-hosted runners (ubuntu-latest, 2 vCPU)
3 of 360 measured runs (0.8%)

Each configuration page lists where its failures happened. All losses.

All results for this pair

One line per configuration pairing, generated from the benchmark data