A Bergen County manufacturer came to us last quarter with a 14-page “CRO report” from their previous agency. Heatmaps. Session recordings. A scroll-depth pie chart. A $4,800 invoice. Three months of “ongoing optimization.” Their conversion rate had moved from 1.4% to 1.5%. Inside the margin of error. They thought they were getting CRO. They were getting theater.
This is the most common scam in digital marketing right now. It’s not malicious — most agencies running these “CRO” retainers genuinely believe they’re doing real work. They’re not. Watching session recordings is interesting. It’s also not conversion rate optimization.
What CRO actually is
Real CRO is a statistical discipline. It has four non-negotiable components, and if your agency isn’t doing all four, you’re paying for theater:
- A hypothesis based on data. Not vibes. Not “this button feels small.” A specific testable claim: “Changing the headline from X to Y will increase form completions by ≥15%.”
- An A/B test framework. Two variants, traffic split 50/50, served randomly to comparable audiences. Google Optimize is dead — use VWO, Convert, or AB Tasty.
- Statistical significance math. A sample-size calculation BEFORE the test runs. A confidence threshold (≥95%) BEFORE you call a winner.
- An honest stopping rule. Tests run until they hit the calculated sample size, full stop. Not “we saw a lift on day 4 so we shipped it.”
Why heatmaps and session recordings aren’t CRO
Heatmaps are great. We use them. They’re a diagnostic tool — they help you form hypotheses about what to test. But “the heatmap shows people don’t scroll to the form” is not an optimization. It’s an observation. Turning that observation into a 15% lift in form completions requires running a test, measuring the result with statistical rigor, and shipping only the winner.
Most CRO agencies stop at the diagnostic. They send you a deck of heatmaps and recordings, make some recommendations, you implement them, and… nobody measures the actual lift. Or worse, they measure it for one week, see a 2% bump (which is noise), and claim victory.
The sample size math nobody runs
Here’s why most “CRO wins” are fake. If your site converts at 2% and gets 1,000 visitors a week, you need roughly 17,000 visitors PER VARIANT to detect a 20% lift with 95% confidence. That’s 34,000 total visitors. That’s eight months of traffic for a typical Bergen County B2B site.
If your CRO agency is “running tests” on a site that gets 4,000 visitors a month and “calling winners” after two weeks, they are running coin flips. The “lift” they’re showing you is statistical noise. They’d get the same result by changing nothing and waiting.
How to spot a real CRO operator
Five questions to ask before you sign a CRO retainer:
- “What’s the minimum monthly traffic you need to detect a 15% lift in 8 weeks?” If they can’t answer in numbers, walk away.
- “What’s the confidence threshold you use to call a winner?” Should be 95% or higher. “We look at trends” is not an answer.
- “How many tests have you run that came back null or negative?” A real CRO operator has plenty. Most tests fail. If they only show winners, they’re cherry-picking.
- “What testing platform do you use?” VWO, Convert, AB Tasty, Optimizely, or custom server-side. “Hotjar” is not a testing platform — it’s a recording tool.
- “Can I see the raw test data, not just the summary deck?” Variant traffic, conversion counts, p-values. If they hesitate, they don’t have it.
When CRO is wrong for your site
If your site gets under 5,000 unique visitors per month, you cannot run statistically valid A/B tests. Period. The math doesn’t work. Anyone selling you “CRO” at that traffic level is selling you heatmaps and vibes. Spend that $2,000-$5,000/month on traffic acquisition or qualitative research (real customer interviews) instead. Come back to CRO when you have the volume.
How AJD handles this
We turn down CRO retainers when the traffic math doesn’t support real testing. For sites that do qualify, we run structured A/B tests with pre-registered sample sizes and 95% confidence thresholds — typically 2-4 tests per quarter, $1,800/month, with the raw test data shared in your dashboard. Whether you work with us or not, ask your current CRO agency for sample-size math on their last three tests. If they can’t produce it in an hour, you’re paying for theater.
Want us to review your current CRO program and tell you what’s real and what’s noise? Book Free Discovery Call →
