GitHub Actions Runner Cost Per Job: CI Bench Results
Compare GitHub Actions runner costs per job across workloads. See CI Bench v1 results, billing trade-offs and how to estimate your own monthly spend.
What Does a GitHub Actions Job Really Cost? Comparing Runner Options
A runner's advertised price per minute doesn't tell you what a finished CI job will cost. Runtime matters, sure - but so does billing granularity. If one provider rounds a 61-second job up to two minutes and another bills by the second, the "cheaper" rate may turn into the more expensive outcome. And even if pricing is the same, different hardware can change how long the same workload takes.
CI Bench v1 tries to get at that problem by running twelve real open-source CI jobs on nine runner configurations and comparing the price of each completed run. In its published comparison, at 100,000 jobs per month, several alternatives come out substantially less expensive per job than a GitHub-hosted 2 vCPU runner. Those numbers are helpful - but they're also tied to the benchmark workloads, the pricing assumptions, and which operating costs the comparison includes.
This article walks through what the results show, why "cost per job" is usually a better yardstick than list price alone, and how to use the benchmark without treating it like a universal bill forecast. It also calls out a caveat: the benchmark maintainer founded a company whose runner product is included in the measurements.
Why cost per job is more useful than price per minute
Runner price is an input. The question a team actually has to answer is: what does it cost to get a successful job through the pipeline? That depends on at least three things: the provider's rate, how long your job takes on that hardware, and how the provider bills.
Take two runners with different advertised rates. One finishes faster but bills in whole-minute increments. Another takes longer but bills by the second. Which one ends up cheaper depends on your job's actual duration - and, importantly, where it lands relative to the rounding boundary. A job that finishes just after a minute can cost materially more than one that finishes just before it. If you compare only per-minute or hourly list prices, you hide that effect.
Hardware can push things around too. A faster runner can reduce elapsed time, but it may also cost more per minute. It might still be worth it if it shortens a critical path, improves concurrency, or gives developers feedback sooner. It's not automatically a win, though - the real question is whether the speed translates into value for the workloads you run.
The job, not the runner, is the unit of comparison
Cost per job combines price and runtime. It's a practical unit for something concrete - like compiling a project or running tests - and it makes it easier to estimate monthly spend once you know roughly how many jobs of each type you run.
That said, it's not the whole story. One job might be a quick lint check; another might compile a large codebase. If you only look at a blended average, you can miss those workload differences. CI Bench's approach - measuring individual workloads - helps here, but your pipeline may still behave differently.
What cost per job leaves out
The benchmark comparison can also exclude costs that show up in real operations. For example, self-hosting can involve cloud setup, ongoing maintenance, security controls, and staff time. If those aren't included, then you're looking at the runner job charge - not the full cost of running a CI platform.
That matters most when you compare managed services to systems you operate yourself. Treat cost per job as a decision aid, not a complete financial model. The practical goal is to see what's included, what's left out, and whether the benchmark's workloads resemble yours.
What CI Bench v1 found
CI Bench v1 reports results for twelve open-source CI jobs across nine runner configurations. Its headline comparison uses a GitHub-hosted 2 vCPU runner as the baseline and models 100,000 jobs per month. Under that scenario, the published cost differences are:
| Runner option | Reported cost difference vs. GitHub-hosted 2 vCPU |
|---|---|
| RunsOn | 88% lower |
| Vitko Runners | 84% lower |
| Ubicloud | 83% lower |
| Blacksmith | 52% lower |
| GitHub-hosted 4 vCPU | 39% higher |
These are benchmark results for a stated scenario, not guarantees for every repository or every monthly invoice. For example, the RunsOn estimate is for self-hosting in your own AWS account and excludes setup and ongoing operating work. So it's a useful way to compare infrastructure pricing, but you'd need to add operational costs yourself if you want a like-for-like total cost comparison versus a managed runner.
Faster is not always cheaper
CI Bench v1 reports that the GitHub-hosted 4 vCPU runner costs 39% more per job than the 2 vCPU baseline, even though it runs 1.6 times as fast. That trade-off isn't something you can boil down to a single "better" answer.
If shorter CI feedback matters to your release process, the faster runner may be worth paying for. If the job isn't on a critical path and the team is focused on direct spend, extra speed might not buy you much.
The real question is what the saved time changes in your workflow. Does it shorten the developer feedback loop? Reduce queue pressure? Help the team deploy more often? Or does it just finish a background task sooner? The answer depends on how the workload fits into your product and engineering process.
Read the percentages as scenario comparisons
The 100,000-jobs-per-month assumption matters because some options include fixed monthly costs that get spread across job volume. As volume changes, the effective cost per job changes too. A result shown at 100,000 jobs shouldn't be blindly reused to forecast cost at 5,000 jobs - or at a million.
CI Bench also provides workload-level results (with intervals), a full ranking, provider list prices (with sources and read dates), and raw records. Those details matter especially when the headline ranking is close, or when one workload matches your repository more closely than the overall mix. Treat the percentage as a starting point for digging - not as the final word.
How the benchmark measured cost
The benchmark's core idea is simple: run the same set of twelve real open-source CI jobs across nine configurations, measure runtime, and apply provider pricing to each run. That's more useful than comparing pricing cards in isolation because the same workload can take different amounts of time on different machines.
Billing granularity is part of the calculation. The source notes that GitHub rounds every job up to a whole minute, while Vitko Runners bills per second. That difference can hit short jobs hardest. If a brief task gets billed as a full minute, its effective cost can jump compared with a per-second model - even if the nominal per-minute rate looks similar. The actual impact depends on each job's runtime and the provider's pricing rules.
Check the workload and measurement uncertainty
A benchmark is only useful when its workloads look like yours. Twelve open-source CI jobs are real data across multiple tasks, but they can't cover every language, build system, dependency layout, or test suite. A large monorepo with heavy caching may behave differently than a smaller project that rebuilds dependencies on every run.
CI Bench publishes intervals alongside workload-level costs. That's important because measured runtimes can vary. When you're comparing options, look at the spread across runs and at the individual workloads - not only the single average or overall ranking.
Check what the price includes
The benchmark depends on provider pricing data and the date when the pricing was read. Plans change, and prices change, so any comparison is time-bound. Before making a purchasing decision, check the current price list and confirm exactly how the provider bills runtime, storage, concurrency, and other usage.
The source also points to an important commercial relationship: the CI Bench maintainer founded Vitko, which sells one of the measured products (Vitko Runners), currently in private beta. That doesn't automatically invalidate the benchmark, but it's relevant context. The benchmark's published method, raw records, and pricing sources give you something concrete to inspect, and the right next step is to validate results against your own jobs.
How to apply the results to your CI budget
Use the benchmark as a shortlist and a set of questions, not as a direct forecast. Start with your own job mix. A service that looks cheap for one type of workload may be a poor fit for another. Then match the comparison to the volume you expect and the operational model you're willing to run.
A practical comparison workflow
- Group your jobs by workload. Separate builds, tests, linting, and other tasks that differ materially. If a job meaningfully affects runtime or spend, don't hide it inside an average.
- Estimate monthly volume by group. Count how often each job runs, including pull requests, pushes, scheduled workflows, and retries. The benchmark's 100,000-job scenario isn't a default assumption for most teams.
- Measure representative runs. Run your jobs on the candidate configurations under comparable conditions. Keep workflow, dependencies, and caching behavior consistent where possible; otherwise you may end up measuring the change in setup rather than the runner.
- Apply the real billing rules. Include rounding, minimum charges, and the pricing tier that applies to you. Per-minute list prices aren't enough when jobs are short or providers bill with different increments.
- Add operating costs. For self-hosted runners, include infrastructure setup and ongoing work such as upgrades, access management, monitoring, and incident response. If you omit those, make it explicit that you're looking at infrastructure cost - not full cost of ownership.
- Evaluate speed as a separate outcome. Track runtime and cost together. Decide whether faster feedback (or less queueing) is worth the difference for the jobs that benefit.
Use volume scenarios rather than one monthly number
Fixed monthly costs don't behave like per-job charges. The source notes that self-hosted options spread fixed monthly costs across volume, so the effective cost per job changes as usage changes. At lower volume, fixed costs are divided over fewer runs; at higher volume, they get diluted more. One point estimate can hide how sensitive the outcome is.
Build at least three scenarios: a normal month, a lower-volume month, and a growth case. For each, estimate direct runner charges and - where relevant - operational costs. This helps you see whether a low-cost option stays low when usage dips, or whether capacity and fixed costs make it less attractive than expected.
Make the decision against your constraints
If your main goal is to reduce direct spend, compare cost per successful job using your real workload and your real volume. If delivery speed is the constraint, compare the value of a shorter feedback loop against the price difference. And if your team doesn't want to run infrastructure, include the engineering time and reliability responsibilities that a self-hosted option shifts onto you.
The benchmark can't make that choice for you. What it can do is narrow the list to options worth testing and show where list price is a misleading comparison. Your own measurements close the gap between a public benchmark and the system you'll actually need to support.
Conclusion
CI runner costs come from what gets billed after a job completes, not just the price printed on a provider plan. Runtime, billing increments, monthly volume, and operational responsibilities all affect the effective cost per job. CI Bench v1 shows those differences across twelve real open-source jobs, with substantial savings for several options versus a GitHub-hosted 2 vCPU baseline at 100,000 jobs per month.
Use the results as a shortlist, not as a final procurement answer. Review the workload-level measurements and current pricing, include any costs the benchmark leaves out, and test your own CI jobs at realistic volume. The best runner is the one that fits your cost, speed, and operating constraints - not necessarily the one with the lowest advertised rate.