CI Bench

CI Bench v1

Workloads

Eleven open-source projects' own CI jobs at a pinned commit, and one burst of 20 simultaneous runs. Each page ranks all nine configurations.

WorkloadGitHub-hosted (ubuntu-latest) medianLowest median timeThat time
Burst: 20 jobs at once259 sBlacksmith97 s
cli368 sVitko Runners (warm starts + test splitting)140 s
codex-lint884 sVitko Runners404 s
container239 sRunsOn174 s
fd92 sVitko Runners (warm starts + test splitting)45 s
flask23 sSelf-hosted runners on Google Cloud11 s
guava1167 sVitko Runners (warm starts + test splitting)651 s
monorepo384 sVitko Runners (warm starts + test splitting)172 s
pydantic179 sVitko Runners (warm starts + test splitting)69 s
ts784 sVitko Runners (warm starts + test splitting)243 s
vite603 sVitko Runners (warm starts + test splitting)199 s
vue121 sVitko Runners (warm starts + test splitting)32 s