Quick answer
Vitko Runners (warm starts + test splitting) had the highest measured speed (2.3× GitHub-hosted); RunsOn had the lowest cost per job at 100,000 jobs a month (88% lower), and Vitko Runners the lowest among the managed runners. Self-hosted options cost less at high volume and take operations work. Every entry says where another option leads.
| Rank | Option | Type | Speed vs GitHub-hosted | Cost per job at 10,000 jobs a month | List $ a minute | Status |
|---|---|---|---|---|---|---|
| 1 | Vitko Runners (warm starts + test splitting)Vitko | Managed | 2.3x faster | 84% lower | $0.002 | Measured |
| 2 | Vitko RunnersVitko | Managed | 1.7x faster | 84% lower | $0.002 | Measured |
| 3 | GitHub-hosted larger runner (4 vCPU) | Managed | 1.6x faster | 39% higher | $0.012 | Measured |
| 4 | Blacksmith | Managed | 1.6x faster | 52% lower | $0.004 | Measured |
| 5 | Self-hosted runners on Google Cloud | Self-hosted | 1.5x faster | 59% lower | cloud bill | Measured |
| 6 | RunsOn | Self-hosted | 1.2x faster | 75% lower | n/a | Measured |
| 7 | Self-hosted runners on AWS (one VM per job) | Self-hosted | 1.2x faster | 60% lower | cloud bill | Measured |
| 8 | Ubicloud | Managed | 1.1x faster | 83% lower | $0.00125 | Measured |
| 9 | GitHub-hosted runners (ubuntu-latest, 2 vCPU) | Managed | baseline | baseline | $0.006 | Measured |
| – | Depot | Managed | not measured in v1 | not measured in v1 | $0.006 | Not measured in v1 |
| – | Namespace | Managed | not measured in v1 | not measured in v1 | unit minutes | Not measured in v1 |
| – | WarpBuild | Managed | not measured in v1 | not measured in v1 | $0.004 | Not measured in v1 |
How this was done
Speed is the inverse of the time index and cost is the cost index, each the geometric mean over 12 workloads against GitHub-hosted (ubuntu-latest), from results.json. Ranked by speed; cost per job uses list prices dated 2026-10-05. Workflows, method and every raw job record: the repository, method. Prices for the options not measured are copied from each provider's own page on 2026-10-24. We operated the self-hosted options. Maintained by Vitko, Inc., which sells one of the products measured.
The 9 measured options
1. Vitko Runners (warm starts + test splitting) Vitko
- Verdict
- 2.3x faster, 84% lower cost per job: 1st of 9 by speed; 3rd of 9 by cost per job at 100,000 jobs a month. Where others lead: RunsOn cost less per job at 100,000 jobs a month.
- Best for
- private-repository teams on Ubuntu x64 that want the lowest measured time and cost per job among the managed runners and can add the warm-start and test-splitting steps
- Measured
- speed 2.3× GitHub-hosted (ubuntu-latest); cost per job 84% lower at 100,000 jobs a month, 84% lower at 10,000, 84% lower at 1,000; lowest median time on cli, fd, guava, monorepo, pydantic, ts, vite, vue; 3 of 360 measured runs failed. All workloads.
- Pricing
- $0.002 a minute, per second, no per-job minimum; source, read 2026-10-24
- Limits
- Private beta; Ubuntu 24.04 on x64 only, private repositories only, United States region; no Arm, Windows, macOS or GPU. Warm starts and test splitting need workflow steps. Run by the benchmark maintainer.
2. Vitko Runners Vitko
- Verdict
- 1.7x faster, 84% lower cost per job: 2nd of 9 by speed; 2nd of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- private-repository teams on Ubuntu x64 that want to change one runs-on line and nothing else
- Measured
- speed 1.7× GitHub-hosted (ubuntu-latest); cost per job 84% lower at 100,000 jobs a month, 84% lower at 10,000, 84% lower at 1,000; lowest median time on codex-lint; 0 of 360 measured runs failed. All workloads.
- Pricing
- $0.002 a minute, per second, no per-job minimum; source, read 2026-10-24
- Limits
- Private beta; Ubuntu 24.04 on x64 only, private repositories only, United States region; no Arm, Windows, macOS or GPU. Run by the benchmark maintainer.
3. GitHub-hosted larger runner (4 vCPU)
- Verdict
- 1.6x faster, 39% higher cost per job: 3rd of 9 by speed; 9th of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- jobs that need 4 vCPU and 16 GB on GitHub-hosted infrastructure
- Measured
- speed 1.6× GitHub-hosted (ubuntu-latest); cost per job 39% higher at 100,000 jobs a month, 39% higher at 10,000, 39% higher at 1,000; lowest median time on none of the workloads; 3 of 360 measured runs failed. All workloads.
- Pricing
- $0.012 a minute, per minute; source, read 2026-10-24
- Limits
- Higher list price per minute; larger runner, always charged.
4. Blacksmith
- Verdict
- 1.6x faster, 52% lower cost per job: 4th of 9 by speed; 7th of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- teams that want a managed runner with a low median time on short, bursty and monorepo-style jobs
- Measured
- speed 1.6× GitHub-hosted (ubuntu-latest); cost per job 52% lower at 100,000 jobs a month, 52% lower at 10,000, 52% lower at 1,000; lowest median time on Burst: 20 jobs at once; 3 of 360 measured runs failed. All workloads.
- Pricing
- $0.004 a minute, per minute; source, read 2026-10-24
- Limits
- Measured at 2 vCPU on the configuration and prices listed on its runner page; cost per job at list prices dated 2026-10-05.
5. Self-hosted runners on Google Cloud
- Verdict
- 1.5x faster, 59% lower cost per job: 5th of 9 by speed; 6th of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- teams that already operate Google Cloud and want pooled VMs under their control
- Measured
- speed 1.5× GitHub-hosted (ubuntu-latest); cost per job 59% lower at 100,000 jobs a month, 59% lower at 10,000, 59% lower at 1,000; lowest median time on flask; 1 of 360 measured runs failed. All workloads.
- Pricing
- All-in cloud cost, measured per job (cost rules)
- Limits
- We operated it for the benchmark; operations time is not in the cost figures. Fixed monthly costs spread over the volume.
6. RunsOn
- Verdict
- 1.2x faster, 88% lower cost per job: 6th of 9 by speed; 1st of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster.
- Best for
- teams with high job volumes that are happy to run runners in their own AWS account
- Measured
- speed 1.2× GitHub-hosted (ubuntu-latest); cost per job 88% lower at 100,000 jobs a month, 75% lower at 10,000, 43% higher at 1,000; lowest median time on container; 1 of 360 measured runs failed. All workloads.
- Pricing
- All-in cloud cost; RunsOn does not meter minutes (source, read 2026-10-24)
- Limits
- Runs in your AWS account: you operate it and pay the cloud bill; we operated it for the benchmark.
7. Self-hosted runners on AWS (one VM per job)
- Verdict
- 1.2x faster, 62% lower cost per job: 7th of 9 by speed; 5th of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- teams that already operate AWS and want one VM per job under their control
- Measured
- speed 1.2× GitHub-hosted (ubuntu-latest); cost per job 62% lower at 100,000 jobs a month, 60% lower at 10,000, 49% lower at 1,000; lowest median time on none of the workloads; 0 of 360 measured runs failed. All workloads.
- Pricing
- All-in cloud cost, measured per job (cost rules)
- Limits
- We operated it for the benchmark; operations time is not in the cost figures. Fixed monthly costs spread over the volume.
8. Ubicloud
- Verdict
- 1.1x faster, 83% lower cost per job: 8th of 9 by speed; 4th of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- teams that want a low per-minute list price from a managed service
- Measured
- speed 1.1× GitHub-hosted (ubuntu-latest); cost per job 83% lower at 100,000 jobs a month, 83% lower at 10,000, 83% lower at 1,000; lowest median time on none of the workloads; 1 of 360 measured runs failed. All workloads.
- Pricing
- $0.00125 a minute, per minute (resource use calculated at per-minute granularity, billed monthly); source, read 2026-10-24
- Limits
- Measured at the standard 2 vCPU tier; a premium tier is listed at a higher price and was not measured.
9. GitHub-hosted runners (ubuntu-latest, 2 vCPU)
- Verdict
- baseline, baseline cost: 9th of 9 by speed; 8th of 9 by cost per job at 100,000 jobs a month. Where others lead: Vitko Runners (warm starts + test splitting) was faster; RunsOn cost less per job at 100,000 jobs a month.
- Best for
- no setup: it is the reference every other result is compared with
- Measured
- speed baseline; cost per job baseline; lowest median time on none of the workloads; 3 of 360 measured runs failed. All workloads.
- Pricing
- $0.006 a minute, per minute; source, read 2026-10-24
- Limits
- The baseline for every comparison; included minutes depend on your plan, and the list price applies to private repositories.
Three more, not measured in v1
Depot not measured in v1
- Status
- Not measured in v2: Depot runners were assigned but did not start jobs for our account during the test window (checks on 2026-10-01 and 2026-10-24).
- Pricing
- $0.006 a minute; Per second; included: 2,000 GitHub Actions minutes a month on the Developer plan; 20,000 on Startup; fees: Developer plan $20 a month; Startup plan $200 a month. source, read 2026-10-24
- Measured numbers
- None: not measured in CI Bench v1, so no speed or cost-per-job figure is given. Provider page.
Namespace not measured in v1
- Status
- Not yet measured; joins in v2.
- Pricing
- Not stated as a per-minute figure; Unit minutes; included: Team plan: 100,000 unit minutes; Business: 250,000; Developer: pay as you go; 30-day free trial that includes managed GitHub runners; fees: Developer: none (pay as you go); Team $100 a month; Business $250 a month. source, read 2026-10-24
- Measured numbers
- None: not measured in CI Bench v1, so no speed or cost-per-job figure is given. Provider page.
WarpBuild not measured in v1
- Status
- Not benchmarked WarpBuild's Terms of Service (§9, Prohibited Uses) prohibit “Use the Service to compete with Argonaut HQ, Inc.” As the maker of a competing runner service, we don't run it. Terms read 2026-10-24.
- Pricing
- $0.004 a minute; Per minute ("CI runners are billed on a per-minute basis"); included: A free usage or credit amount is mentioned; the amount is not on the pricing page text read; fees: None stated; add-ons priced separately; bring-your-own-cloud runners $0.002 per minute. source, read 2026-10-24
- Measured numbers
- None: not measured in CI Bench v1, so no speed or cost-per-job figure is given. Provider page.
More: speed ranking · cost ranking · alternatives to GitHub-hosted · self-hosted vs managed · free minutes · list prices · price per minute, 2 and 4 vCPU · Docker builds · CPU-heavy jobs · spot instances · security questions · ARC vs managed · self-hosted platform fee · cut the Actions bill · CLI and MCP · RunsOn vs Blacksmith vs Depot vs Namespace · WarpBuild vs Ubicloud vs BuildJet · alternatives to Depot, WarpBuild, Namespace, BuildJet. Maintained by Vitko, Inc., which sells one of the products measured. Page generated October 27, 2026.