Short answer
Agent-heavy CI depends on four measurable things: how long each job takes, what each job costs, what happens when many jobs start at once, and whether runs fail. CI Bench v1 measured all four on nine configurations (October 7 to 11, 2026). Across the eleven single-job workloads the lowest median job time was Vitko Runners (warm starts + test splitting) at 172 s; GitHub-hosted 2 vCPU took 368 s. In the burst of 20 simultaneous jobs, 95th-percentile queue time ranged from 3 s to 295 s. Vitko, which maintains CI Bench, sells Vitko Runners, which is listed in the same columns as the others.
All nine configurations, ordered by median job time
| Configuration | Median job time (median of the eleven workload medians) | Mean cost per job (mean of workload means) | Burst queue, median | Burst queue, 95th percentile | Successful runs of attempted |
|---|---|---|---|---|---|
| Vitko Runners (warm starts + test splitting)Vitko | 172 s | $0.0095 | 17 s | 115 s | 357 of 360 |
| Vitko RunnersVitko | 180 s | $0.0080 | 18 s | 295 s | 360 of 360 |
| GitHub-hosted larger runner (4 vCPU) | 192 s | $0.060 | 3 s | 39 s | 357 of 360 |
| Ubicloud | 276 s | $0.0077 | 18 s | 127 s | 359 of 360 |
| Self-hosted runners on Google Cloud | 278 s | $0.024 | 2 s | 3 s | 359 of 360 |
| Self-hosted runners on AWS (one VM per job) | 281 s | $0.018 | 65 s | 93 s | 360 of 360 |
| Blacksmith | 286 s | $0.021 | 9 s | 16 s | 357 of 360 |
| RunsOn | 288 s | $0.0048 | 23 s | 194 s | 359 of 360 |
| GitHub-hosted runners (ubuntu-latest, 2 vCPU) | 368 s | $0.045 | 3 s | 31 s | 357 of 360 |
Burst: 20 jobs dispatched at once, one burst per round, default capacity with nothing sized for the benchmark; treat it as one data point. Self-hosted configurations are priced all-in at 100,000 jobs a month. Blacksmith's container cell is a run in which its Docker layer cache did not hit (method). Details: burst of 20 · failed runs · cost ranking · speed ranking.
What the benchmark does not measure
Spend controls, API tokens and command-line tools for agents are features, not measurements here. Some are listed per provider on runner services with a CLI or an MCP server. Vitko Runners (private beta) has a hard monthly spend limit that API tokens and CI jobs cannot raise past an admin-set ceiling, and a JSON command-line tool: AI coding agents waiting on CI. Ubuntu 24.04 on x64, private repositories only.
Computed from results.json; every raw record: the repository. Method: how the runs were made. Maintained by Vitko, Inc., which sells one of the products measured. Page dated November 2, 2026.