Blacksmith vs GitHub-hosted larger runner (4 vCPU): speed and cost on 12 real CI workloads

Speed

× faster than GitHub-hosted (ubuntu-latest) · higher is better

Blacksmith

1.6×

GitHub-hosted larger runner (4 vCPU)

1.6×

GitHub-hosted (ubuntu-latest)

1.0×

USD per $1.00 spent on GitHub-hosted (ubuntu-latest), at 100,000 jobs a month · lower is better

Blacksmith

$0.48

GitHub-hosted larger runner (4 vCPU)

$1.39

GitHub-hosted (ubuntu-latest)

$1.00

Blacksmith had the lowest median time of the two on 7 of 12 workloads, GitHub-hosted larger runner (4 vCPU) on 5.

Workload	Blacksmith	GitHub-hosted larger runner (4 vCPU)	Lowest median
Burst: 20 jobs at once	97 s	154 s	Blacksmith
cli	286 s	192 s	GitHub-hosted larger runner (4 vCPU)
codex-lint	448 s	476 s	Blacksmith
container	386 s	185 s	GitHub-hosted larger runner (4 vCPU)
fd	65 s	66 s	Blacksmith
flask	22 s	21 s	GitHub-hosted larger runner (4 vCPU)
guava	735 s	946 s	Blacksmith
monorepo	177 s	220 s	Blacksmith
pydantic	103 s	154 s	Blacksmith
ts	401 s	435 s	Blacksmith
vite	360 s	288 s	GitHub-hosted larger runner (4 vCPU)
vue	76 s	67 s	GitHub-hosted larger runner (4 vCPU)

Mean cost per job in USD, from the benchmark’s sensitivity results

Volume	Blacksmith	GitHub-hosted larger runner (4 vCPU)
100,000 jobs a month	$0.482	$1.387
10,000 jobs a month	$0.482	$1.387
1,000 jobs a month	$0.482	$1.387

Counted, never replaced

Each configuration page lists where its failures happened. All losses.

One line per configuration pairing, generated from the benchmark data

no clear difference in time between Blacksmith and GitHub-hosted larger runner (4 vCPU) (the 95% interval, 0.94 to 1.15, includes 1.00); Blacksmith cost 65% less per job (at 100,000 jobs a month).