Between that advice and the proliferation of low-cost testing tools available, it’s no wonder that testing has become the latest digital marketing craze.
It may come as a surprise then, that testing is NOT always the quickest or most effective path to higher conversions.
Here are some situations in which testing can do more harm than good.
Under ideal conditions, your landing page should be the only thing that changes during your test. Otherwise, it is difficult to attribute any outcome to specific variables being tested. And one of the biggest external factors muddying up the testing waters can be traffic instability.
In other words, if traffic to your website isn’t consistent, you can’t attribute that big, ugly green button to higher conversions. It may be converting better in spite of that button.
Testing is based on statistical probability. Probability is math that requires a very large (they call it ‘statistically significant’) sample size. When you don’t have enough conversions to reach statistical significance, the true probability of conversion for each page remains unknown.
To use a classic statistics example… Flip a coin and you always have a 50% chance it’ll come up heads. However, you can flip a coin 5 times and it continually come up tails. Does that mean tails is statistically more common? Nope. It means your sample size is too small.
An (in)famous experiment was conducted by POW while in captivity (nothing else to do) where he flipped a coin thousands of times to see if it’s really 50/50. Even though he had long stretches when it came up heads or tails, on average it was always 50% heads and 50% tails as long as you flipped it enough.
That said, you may be thinking all you have to do is run the test long enough and you’ll get the numbers you need. The problem with that is polluting your data with other variables, such as seasonal traffic spikes, changes in the competitive landscape or other external factors that could affect your conversion rate.
If you have serious problems with your website design, if your checkout flow is cumbersome, or your architecture is confusing and non-intuitive, testing won’t do you much good. Testing works well for optimizing individual pages and elements, but without taking a more holistic approach to fixing your site, you run the risk of morphing it into an unusable Frankenstein-like mess.
All that testing might just be lipstick on a pig. Maybe it’s time for an overhaul that includes a more holistic approach.
There’s no question that website testing is an important element of conversion rate optimization, but it isn’t the ONLY element.
If your site has very low conversions it may be a result of fundamental flaws that should be corrected before testing is considered. Once you have a usable and well-designed site with a steady flow of traffic, then you can begin the ongoing process of testing, fine-tuning and continually optimizing for greater success.