Blacksmith vs GitHub-hosted (ubuntu-latest): speed and cost on 12 real CI workloads

Speed

× faster than GitHub-hosted (ubuntu-latest) · higher is better

Blacksmith

1.6×

GitHub-hosted (ubuntu-latest)

1.0×

USD per $1.00 spent on GitHub-hosted (ubuntu-latest), at 100,000 jobs a month · lower is better

Blacksmith

$0.48

GitHub-hosted (ubuntu-latest)

$1.00

Blacksmith had the lowest median time of the two on 11 of 12 workloads, GitHub-hosted (ubuntu-latest) on 1.

Workload	Blacksmith	GitHub-hosted runners (ubuntu-latest, 2 vCPU)	Lowest median
Burst: 20 jobs at once	97 s	259 s	Blacksmith
cli	286 s	368 s	Blacksmith
codex-lint	448 s	884 s	Blacksmith
container	386 s	239 s	GitHub-hosted runners (ubuntu-latest, 2 vCPU)
fd	65 s	92 s	Blacksmith
flask	22 s	23 s	Blacksmith
guava	735 s	1167 s	Blacksmith
monorepo	177 s	384 s	Blacksmith
pydantic	103 s	179 s	Blacksmith
ts	401 s	784 s	Blacksmith
vite	360 s	603 s	Blacksmith
vue	76 s	121 s	Blacksmith

Mean cost per job in USD, from the benchmark’s sensitivity results

Volume	Blacksmith	GitHub-hosted runners (ubuntu-latest, 2 vCPU)
100,000 jobs a month	$0.482	$1.000
10,000 jobs a month	$0.482	$1.000
1,000 jobs a month	$0.482	$1.000

Counted, never replaced

Blacksmith: 3 of 360 measured runs (0.8%)
GitHub-hosted runners (ubuntu-latest, 2 vCPU): 3 of 360 measured runs (0.8%)

Each configuration page lists where its failures happened. All losses.

One line per configuration pairing, generated from the benchmark data

Blacksmith ran jobs 1.6× faster than GitHub-hosted runners (ubuntu-latest, 2 vCPU) (95% interval 1.3× to 1.7×); Blacksmith cost 52% less per job (at 100,000 jobs a month).