How Much Does an A/B Test Cost? The 33 Hours, Itemized

WRITTEN BY

Gary Culaev, Founder and CEO, Weblics Consulting

How Much Does an A/B Test Cost? The 33 Hours Behind One Test

Most quotes price a test as though it were a headline swap. A real one is 33 hours of work, and the hours are where the money is. Here is each one, what it comes to at $65 an hour, and the number that actually belongs in your budget, which is the price of a winner rather than the price of a test.

One A/B test takes about 33 hours of work. At $65 an hour that is about $2,145 per test, and about $7,100 to get one that wins. At the median winning lift, a winner pays for itself in about a month.
‍
Every number below comes with its assumption next to it. Swap in your own and the math still works.

In this guide

Why the price of a test is the wrong number
The 33 hours, one by one
What the 33 hours do not cover
Total due: the price of a winner
The same receipt at your traffic
Red flags in a testing quote
FAQ

Why the Price of a Test Is the Wrong Number

About one A/B test in three produces a statistically significant winner, so you pay for roughly three tests to get one change worth shipping.
‍
That figure comes from Georgi Georgiev, the statistician behind Analytics-Toolkit, who analyzed 1,001 A/B tests in 2022. 33.5% ended with a significant winner. The median lift across all tests was 0.08%, close to nothing. Among the winners, the median was 7.5%. He notes the set leans toward experienced practitioners, so a newer program shouldn't expect better odds.
‍
Yes, that means two tests in three don't win. A test that doesn't win isn't wasted, though. It's what you pay to avoid shipping a change that would have cost you. We ran one ourselves: pinning reviews on a product page looked like an obvious improvement, came out about 5% worse, not a significant difference, and never went live.
‍
So price the thing you're actually buying, which is a winner. When someone shows you a winning test, ask how many tests it took to find it.

The 33 Hours, One by One

One test on a mid-size Shopify store takes about 33 hours of work: 6 hours of pre-analysis, 8 of design, 14 of development and 5 of QA. At $65 an hour that's $2,145.
‍
The assumptions behind the hours: a store doing about $300,000 a month on 150,000 sessions, testing a template every order passes through, one test a month. The rate is ours. What the hours are spent on doesn't change with the rate, which is why the breakdown is the useful part.
‍
Pre-analysis, 6 hours. Reading the analytics, the session recordings and the reviews until there's a hypothesis worth 27 more hours. Build the wrong hypothesis here and everything after it is wasted, however well it's executed. This is also where someone works out how big a lift your traffic can actually confirm, before anyone draws anything.
‍
Design, 8 hours. Two to three variations, each on desktop and mobile, with developer notes and a round of client approval. A variant designed only for desktop is a test of your desktop traffic, which on most Shopify stores is the smaller half.
‍
Development, 14 hours. The largest line, and the one most quotes pretend isn't there. This is rebuilding a section properly inside your theme, not recoloring a button. Invesp, which has run a CRO agency since 2006, puts a typical test at 10 to 30 development hours and a complex one at 40 to 80, so 14 sits at the low end of normal.
‍
QA, 5 hours, across three layers. Devices and browsers, then the real use cases a shopper puts the page through, then live checks once the test is running. A variant that breaks on one phone model is a test of the bug, not of the idea.

Where the 33 hours in one A/B test go: 6 pre-analysis, 8 design, 14 development, 5 QA, $2,145 at $65 an hour

That is the honest shape of a test. It is also why a $10,000 quote for a title change should end the conversation: the hours aren't there. The number moves with the rate, not with the work, and the work is the part you can check.
‍
One more thing the shape explains. Big structural changes and small binary tweaks take wildly different hours, so pricing them the same way is a choice that favors whoever is selling. We run the heavy changes first, the ones that move profitability, and bill the small binary tests by the hour once the heavy lifting is done.

What the 33 Hours Do Not Cover

Three costs sit outside the hours: the testing tool, the traffic a losing variant burns, and the work of shipping a winner into your theme for good.
‍
The testing tool: about $300 a month. Assumption: a Shopify-native tool on a mid tier, which is the honest middle of a wide range. Published prices in October 2026: Intelligems starts at $69 a month for a single module at the lowest order tier and scales with your order volume, and its all-modules Unlimited plan is listed at $1,279 a month on a three-month commitment, $2,894 at the standard rate. Shoplift is $99 a month on Core and $699 on its managed plan, priced by monthly visitors. Convert is $399 to $599 a month, or $299 to $420 paid annually, and scales with tested users. Invesp puts the wider market at $200 to $1,500. The entry price is the one vendors quote and the one almost nobody pays, because every plan scales with the orders or visitors you already have. You pay this whether or not you run a test that month. Often left off the quote: whose name the account is in. If it's yours, your test history stays when the vendor leaves.
‍
Traffic on the losing variant: up to $2,500. Illustration: if the tested page drives $100,000 a month in revenue, a third of the store in our assumptions, and the variant is 5% worse, half your traffic loses 5% for four weeks. Nobody invoices this, and you pay it anyway. Stopping clear losers early helps, and 90% of the tests in Georgiev's dataset used sequential methods that allow it.
‍
Shipping the winner: 10 hours, $650. The variant that ran the test is built to run a test. Building it into your theme properly, so it survives the next app install and the next theme update, is a separate job. You only pay this one when you win.

Total Due: The Price of a Winner

At one winner in three, a winner costs about $7,100: three tests at $2,145, plus $650 to ship the one that won.
‍
Here's the number to bring to the budget meeting. If that winner lifts conversion on the $100,000 page by the median 7.5%, it adds about $7,500 a month in revenue, or about $3,375 of gross profit at a 45% margin. Against $7,100 that's payback in about a month on revenue and a little over two on profit, and for as long as the change stays live, it keeps paying.
‍
Winning estimates tend to run high, so run it at half. At a 3.75% lift the same winner pays back in about two months on revenue and about four on profit. Still inside a quarter.
‍
Now picture the cheap version. A store pays $99 a month for a testing app, runs ten tests a year without anyone checking whether its traffic can detect the lifts it's looking for, and ends the year with $1,188 spent and no answers. It spent far less and bought nothing.
‍
The same division works on any quote, because the hours don't change, only the rate does. Our hiring comparison prices the same work at market rates: about $4,950 a test on the hourly freelance path, about $6,300 through an agency retainer, about $16,950 for a full-time hire running one test a month. Multiply each by three and the price of a winner on that path is roughly $15,000, $19,000 and $51,000.
‍
It's also why we price some of our own work the way we do. On Weblics' pay-per-positive-test plan, a test that doesn't win costs the client nothing, and a winner is billed at a fixed rate plus 50%. The client pays for winners, and we carry the two in three.

The Same Receipt at Your Traffic

Traffic decides how small a win your test can see, and that decides whether the receipt is worth paying at all.
‍
The lift figures below are for a four-week test at a 2% conversion rate, 95% confidence and 80% power, with every session entering the test. The math is in when to start CRO.

What a four-week test can confirm and what a winner costs at each traffic tier, from under 50,000 sessions to 200,000 and up

Under 50,000 sessions, most winners are too small for a test to confirm in any reasonable time. The median winning lift in Georgiev's data is 7.5%, well under what a test at that traffic can see. Put the budget into fixes and research first, and do you need a CRO agency yet? covers what to do instead.
‍
Above 200,000 sessions the fixed costs spread, because the same 33 hours buy a result you can read sooner and act on across more traffic. Bigger brands are moving that way: in Convert's 2025 survey of agency leaders, 87% of large agencies said clients were bringing more experimentation in-house.

Red Flags in a Testing Quote

The biggest red flag is a quote that prices tests and never mentions winners. A test is only worth what it can find.
‍
• It prices a headline swap and a rebuilt section the same way. Those are not the same number of hours, and a quote that can't tell them apart isn't costing the work.
‍
• It promises a number of tests a month without saying what size of lift each one can detect at your traffic.
‍
• It quotes a win rate without saying what counts as a win. A statistically significant winner is a different thing from a test that moved in the right direction, and among experienced practitioners about one test in three produces the first kind.
‍
• There's no line for QA, or for shipping the winner into the theme.
‍
• The case studies show only winners. Ask how many tests each one took.
‍
• The testing tool account is in the vendor's name.
‍
A budget exists to buy things that keep working after you've paid for them. Ad spend is gone by the end of the month. A winning test keeps converting every visitor you send it, including the ones you're already paying to bring in. That's what the money is for. Our CRO ROI calculation takes the same numbers the rest of the way.
‍
The cheapest way to lower the price of a winner is to raise the share of tests that win, which means spending the first six hours properly. Our product page optimization checklist is the list we start from at Weblics on a new store. If you'd like the cost of a winner worked out on your traffic, send us your store.
‍
You're not buying tests. You're buying winners, at about three tests apiece.