With the advent of more and more software, split testing has become quite a hot topic. For the last few years, it seems almost every online marketing blog you read has at least a few sections dedicated to "split testing."
But, there's a big problem these companies aren't really telling you:
Most small to medium sized businesses are absolutely wasting their time and resources on split testing.
Split testing is generally a really bad idea for new business owners, but before we dig into that let me explain what split testing is:
Split testing, otherwise known as "A/B testing", is the practice of setting up two variations of the same page on a website (generally a landing page, the home page, or another important sales page on the website) and sending equal amounts of traffic to those pages until one of them is deemed the "winner." This winner is then used as the sole page for that section of the website.
Change a couple of headlines and BOOM! Conversions go through the roof. Move your contact form to the right side, and you see a 50% increase in conversions on the landing page. That's the kind of "results" most companies expect to see in split testing.
However, it's rarely the case that these results occur. In fact, more often than not, you'll find a split tested page has almost the exact same conversion rate. Anyone who regularly runs split tests will tell you that huge wins like that generally don't happen.
Let me use an example:
Say you're selling hot tubs. You can't decide which photo you should use on your landing page. One of the photos is a tasteful picture of a woman in a bikini inside one of your hot tubs, and the other is a picture of a family enjoying their new hot tub.
You're torn between the two photos, so you create a split test to see which of the two page variations convert better.
Here's what you may expect to happen:
Photo of the woman in the bikini: "40% conversion rate"
Photo of family in hot tub: "54% conversion rate"
Here's what will actually happen:
Photo of the woman in the bikini: "41% conversion rate"
Photo of family in hot tub: "42% conversion rate"
What does this data tell you? Specifically, what does the second set of data tell you? Does it tell you the family in the hot tub should be the right picture?
No, it's actually telling you that you need to run about 20 more split tests to properly optimize the page. In fact, most professional split tests I've been a part of are between 20 - 40 tests on a single page. Even when you do get a "grand slam" like the expectation example (where one conversion rate was 14% higher) over time, that conversion rate will slowly fall back to the "average" conversion rate your website typically sees.
This is not to mention the fact that an accurate split test generally needs tens of thousands of people (or more) in order to produce the results necessary to make effective decisions with the data.
So what's to be done?
A lot of companies (especially web based companies) waste time in split testing. Instead of focusing on increasing your conversion rate, focus on increasing the amount of visitors to your specific landing page. For example:
Imagine you send 1000 people to your landing page last week. You saw a conversion rate of 1.5%, or 15 people out of that 1000 that completed the desired action such as an email signup or free download. Instead of spending the rest of the month testing out different options for the landing page, focus on getting 10,000 through to it. Those numbers are more important to your success, and more telling when used for analysis later.
So, When Should You Split Test?
Simply put, before you have the proof of concept.
Don't waste a lot of time on the "little" details like different subject lines, different blog titles, and variations of button color. Instead, only split test the most important points of your website. This could include:
- Your main sales page
- Your landing page
- Your first variations of sales copy and brand copy (this means, your first set of mass emails, your first set of customer focused emails, and so on).
Further, anything that needs to ensure people will buy what you're selling should be split tested. The ultimate goal here is obtaining proof of concept, or, evidence that your idea actually works.
Expanding on the above example, what if the 1000 people who visited your website last week gave you a true conversion rate of 0%? None of the 1000 people bought from you, or filled out a contact/lead form. Your 1.5% that "converted" stuck with their freebie, and moved along with no intention of ever purchasing from you.
This means you don't have proof of concept, and whatever you're doing needs to be tested more effectively. Maybe your landing page copy was off. Perhaps you put a picture of a woman that was way too scantily clad in her bikini-and you offended your customers.
However, once you do have proof of concept-you need to drop split testing and focus on attracting more eyeballs to the proof of concept. If you spend too much time "tweaking the system" to make it work more efficiently, you'll find you haven't spent nearly as much time attracting new and returning customers to that system.
To sum up:
Use split testing strategically, and only on elements of your website that will directly effect your profits (or other critical goals) to ensure you're not wasting valuable time.
Don't expect wild swings and "home runs" in your data, especially over time.
Once you have proof of concept, get as many people visiting that site as possible, as quickly as possible.