When Testing is Bad

As a marketer, you’re used to hearing the caveat “you should test to see…” after every marketing tip you’ve ever received. Well, after reading Tim Ash’s article on Website Magazine, I’m convinced there ARE times testing does more harm than good.

Between that advice and the proliferation of low-cost testing tools available, it’s no wonder that testing has become the latest digital marketing craze.

It may come as a surprise then, that testing is NOT always the quickest or most effective path to higher conversions.

Here are some situations in which testing can do more harm than good.

Unstable Traffic

Under ideal conditions, your landing page should be the only thing that changes during your test. Otherwise, it is difficult to attribute any outcome to specific variables being tested. And one of the biggest external factors muddying up the testing waters can be traffic instability.

It may be converting better in spite of that big, ugly button — not because of it.

In other words, if traffic to your website isn’t consistent, you can’t attribute that big, ugly green button to higher conversions. It may be converting better in spite of that button.

Too Few Conversions

Testing is based on statistical probability. Probability is math that requires a very large (they call it ‘statistically significant’) sample size. When you don’t have enough conversions to reach statistical significance, the true probability of conversion for each page remains unknown.

Flip a coin 5 times and it comes up heads every time. Does that mean heads is more likely? Nope. Your sample is too small.

To use a classic statistics example… Flip a coin and you always have a 50% chance it’ll come up heads. However, you can flip a coin 5 times and it continually come up tails. Does that mean tails is statistically more common? Nope. It means your sample size is too small.

An (in)famous experiment was conducted by POW while in captivity (nothing else to do) where he flipped a coin thousands of times to see if it’s really 50/50. Even though he had long stretches when it came up heads or tails, on average it was always 50% heads and 50% tails as long as you flipped it enough.

That said, you may be thinking all you have to do is run the test long enough and you’ll get the numbers you need. The problem with that is polluting your data with other variables, such as seasonal traffic spikes, changes in the competitive landscape or other external factors that could affect your conversion rate.

Lipstick on a Pig

If you have serious problems with your website design, if your checkout flow is cumbersome, or your architecture is confusing and non-intuitive, testing won’t do you much good. Testing works well for optimizing individual pages and elements, but without taking a more holistic approach to fixing your site, you run the risk of morphing it into an unusable Frankenstein-like mess.

All that testing might just be lipstick on a pig. Maybe it’s time for an overhaul that includes a more holistic approach.

So No More Testing?

There’s no question that website testing is an important element of conversion rate optimization, but it isn’t the ONLY element.

If your site has very low conversions it may be a result of fundamental flaws that should be corrected before testing is considered. Once you have a usable and well-designed site with a steady flow of traffic, then you can begin the ongoing process of testing, fine-tuning and continually optimizing for greater success.

Lead Gen Trends
When Testing is Bad
5 - 6 votes

About the Author

Justin Downey
27 articles
About / Bio
President of Dallas-based digital ad agency, JDM Digital, Justin has decades of B2B experience in all things digital. His company, JDM, specializes in branding, design, web development, copywriting and more 'weird and wonderful' things.
Custom Photo 1
JDM Digital
Hottest Articles
New Contributors
  • Todd Riddle
  • Brian
  • Joe Pulizzi
Recent Articles

You might also like...

Cliqued Web App Review

Cliqued: Web App Review

Read More →