A/B testing is simple. It utilizes multivariant testing in order to test a statistical hypothesis. This type of test is also referred to, colloquially, as a split test or a bucket test. It is often used to test the effectiveness of two or more marketing campaigns in comparison with each other.
An example would be that of an A/B test to look at two different Google Adwords advertisements and to see which one results in better conversions. It could also be used to test different landing pages on a website, to measure audience behavior or the outcomes of audience engagement with the site.
In two words? A/B testing exists for you to build a bulletproof marketing strategy. With right approach, you can increase conversions by a huge number. However, before embarking on a quest of As and Bs it is important to understand what exactly demands your attention. Today, we'll talk just about that - even if A/B testing is simple, the details are not.
Things to Know Prior to Conducting the Test
Here are a few things you should know before you decide to run an A/B test.
Avoid assumptions
A/B testing is primarily meant to test two different marketing messages or styles. This requires that you do not make assumptions about the preference of your audience. Do not allow the message itself to discriminate based on things like income, age, or gender. And definitely do not believe yourself. You can say that green color is always preferrable for a CTA button, because green is inviting (read more on the topic here). Is it true? Well, it is, but not always.
Here's an example. When Performable tested two variations of their CTA button color (one green, the other red) they found that the red button outperformed the green by 21%.
In case of SuperFastBusiness, when they changed the color of the CTA from yellow to purple there was a 50% increase in signups.
So which color is better - red or purple?
The answer is neither. It could be green, red, purple or yellow. It depends. In case of Superfastbusiness, the yellow CTA button merged in with the overall look of the site. The purple button brought in the much needed variation. Similarly, Perfomable website featured a lot of green already and the addition of red color stands out.
Just because the color red works for someone it doesn't mean that it'd work for you as well. Don't go about A/B testing based on assumptions.
Single variable changes
Do not test two unknown variables against each other. Always test a controlled variable against an unknown variable in order to determine the effectiveness of each.
According to WhichTestWon:
Test thoroughly
It is extremely easy to assume a specific outcome has been reached. The truth is, until the proper statistical analysis has been done on the outcome of the test itself, there is no clear way to tell which outcome was more favorable.
Sample size
Have a large sample size. A small sample size can lead to statistical bias which can cause your results to be skewed. Optimizely has a free sample size calculator that you can use for determining the correct sample size for testing.
Prepare for unknowns
One interesting fact about running an analysis of this type is that you may find out that the surveys you have conducted with your clients or customers in the past do not match how they behave at the moment. This is something that is somewhat common and it is important not to discount information simply because it does not seem to "add up".
Define metrics
You should have a clear idea of what a "success" is prior to beginning the test. If you do not have that idea in mind, then you will have no way of determining what the results of your test mean. What makes one variable more successful than another? Is it more conversions, more opt-ins, more sales or something else?
Start big
Your test should be used on things that will have a significant impact on what you are doing. Do not get caught up in the minutiae of testing little things that bear little to no significance. Be focused.
A/B testing is relatively a simple thing to do with a little bit of training. It can provide valuable information about your business and your marketing efforts and can have a large impact on growth. It is, therefore, a good way to gauge the effectiveness of your marketing campaigns. What do you think?