Method

Design

Pre-registration

Fixed before the sessions ran (tag prereg-v1). Later changes are dated in the changelog.

Replication

30 measured blocks per configuration and workload. Run timeout 2 hours. Failures count; none is replaced unless the benchmark's own harness failed.

Workloads

11 projects' own CI jobs at a pinned commit, with portability edits identical for every provider, and one burst of 20 simultaneous runs. Workload pages.

What changes

Mostly the runner. Accelerators a provider documents and that were used are on its runner page. Vitko Runners with warm starts and split tests also uses its provider’s recommended options and is labelled as such.

Times

GitHub’s jobs API, whole seconds, one clock for every provider. Queue is started_at minus created_at; run is completed_at minus started_at.

Names on this site

Display names are short; the exact configuration measured is on each runner page and in the repository's provider files.

Reading the numbers

Speed is how many times faster the median job finished than on GitHub-hosted (ubuntu-latest). Cost is what the same jobs cost, in dollars per $1.00 spent on GitHub-hosted (ubuntu-latest). Both are averages over the twelve workloads and carry 95% intervals (open “Indexes and 95% intervals” on the leaderboard and on each runner page). An average can hide a workload where a configuration is behind, so the leaderboard heatmap, the workload pages and the head-to-head pages show each workload. Measurements were taken October 7 to 11, 2026, with the list prices dated 2026-10-05; the separate list-prices page was read on a later date. All dates are in the footer of every page.

Indices

Per workload, a configuration's median wall time and mean cost divided by GitHub-hosted (ubuntu-latest)'s. Time and cost index are the geometric means over the 12 workloads, equal weight. The site shows speed as 1 divided by the time index, and cost as the difference from 1. 95% intervals: block bootstrap, 10,000 replicates, resampling sessions then rounds. A cell needs at least 90% successful runs to enter the indices. Pairwise ratios, leave-one-workload-out and per-session indices are in results.json.

Cost rules

Managed

List price on 2026-10-05 under the provider's billing rule (per second, or each job rounded up to a minute). Sources: price file.

Self-hosted

RunsOn, AWS EC2 and the Google Cloud pool are priced all-in: everything the jobs use in the account, measured per job, plus fixed monthly costs over 100,000 jobs a month. Sensitivity at 10,000 and 1,000. The pool is modelled at 50% utilization.

Excluded

Queue time, free minutes, plan fees, discounts.

Split tests

Only the parts are billed while a split step runs.

Vitko capacity

Both Vitko configurations ran at default capacity: two warm standby runners per pool, others created on demand. Nothing was sized for the benchmark.

Limits

• 2 vCPU class plus one 4 vCPU reference; Linux x64; one region per provider; public repositories. No GPU, macOS, Windows or Arm.
• Maintained by Vitko, Inc., which sells one of the products measured. We operated the self-hosted options with the settings in providers/.
• The warm-start configuration ran on a Vitko host build that predates a later cleanup of restored setups (see the deviations in the changelog).
• One result set. Prices change; each figure shows its price date.

Reproduce

See the result set README. Raw job records: results/v1/raw.