We A/B Tested Our Email Subject Lines For Months: Here's What We Learned

Posted on August 27th 2014

We A/B Tested Our Email Subject Lines For Months: Here's What We Learned

Every week, we send out the Content Marketing Update email. Before I send it out to the entire list, I always set up an A/B test on subject lines to see which subject line is preferred.

We’re a little competitive here, and sometimes it becomes a battle between Garrett’s blog post or my blog post being the winning email. That’s silly, of course (though admittedly I still cheer and point it out when I win), but after doing A/B testing for many months, we have some interesting data to sift through.

 

Our Email A/B Test Details

Let’s take a look and see if we can learn a bit about what makes for great email open rates, based on the data from the past three months.

To start, let me tell you about the data you see here, and the A/B test we ran. Like any A/B test, you are focusing on only one thing to test, despite there being other options. There are also two key variables that would obviously affect the outcome and data.

1. Who got the email, and when.

We sent the email out to 10% of our email list at 10 am CST, letting the test run for one hour before sending the winning email out to the rest. Of that 10% that receives the email, on average 10% actually opened the email.

  • Test recipients outside of our timezone might not have had the chance to open the email.
  • An hour-long test isn’t that long. A longer test might have changed the data as more recipients had a chance to participate.

2. Arbitrary decisions on featured posts.

The email subject line did not always match the headline of the post it referred to.

  • Sometimes they were made shorter (more about email subject length later in this post).
  • I chose from the newest blog posts that week and decided which ones to feature in this test.
  • I used two different featured posts with two different areas of interest. You could use the same post with two different subject line for a “pure” test where topic preference would not interfere.

3. What you choose to measure.

This is quite a bit to consider when it comes to A/B testing your email. You might focus on the subject line (which is what we did here) and build data on character count, how the subject line is phrased, and the EMV of the subject line.

There are other things to test besides subject line, though.

We use Campaign Monitor and some of the other options for A/B testing they offer include adjusting who the email comes from (we use Garrett’s email because we think a real person is better than info@yourcompany.com), and the email content.

a/b email testing

3 Months Of Data On Email Subject Lines

In this chart, you can see the email subject lines used. The green row is the winner, the subject line with the most opens. You can also see:

  • How many people unsubscribed (unsubs) after receiving an email with that subject
  • The emotional value (EMV) of that email subject line
  • How many characters (chrctr) made up the total subject line
  • How many social shares the actual blog post received on the blog

What does this data tell us?


Does having a number in the subject line help?

In the sample above, nine of the email subject lines contained a number. Five of them won. However, in two cases, both A and B had a numbered headline. So, taking them out of the pool means that there are only five true examples of a numbered and un-numbered headline pitted against each other. Of those five, three won.

After seeing this, I may be more aware of creating a numbered and non-numbered headline to see which performs better over a longer period of time. In looking further back in our A/B test emails, numbered headlines performed slightly better, but not significant enough to say that they are a sure thing.

Do longer or shorter subject lines do better?

There are a few different theories about how long a subject line should be. MailChimp has done some great research on what makes email attractive to recipients, and a general rule of thumb seems to be about 50 characters.

In our sample, the character count looked like this:

Character CountTotal UsesWins
20-291 
30-3961
40-49113
50+62

This is a pretty small sample, but what this data shows is that shorter subject lines were less successful. Mid 40′s and up worked best..

In looking at this table, I’m thinking I could be writing longer subject lines. Ours tend to hover around the mid-40′s in character count. What I also found interesting is that in half of these A/B tests, I tended to create subject lines that were of about the same character count instead of varying the length of the two competing against each other. Often, our blog posts have longer headlines and in future tests, we could be using that length in our favor.

Which emails had the biggest and least differences in open rate?

In the June 10 email, the spread between the two subject lines was the largest, at 27. The winner, “A No-Fail Method For Writing Your Blog Posts” beat out “50+ Places To Repurpose Your Content.”

What is odd about that is that this was the same week where the winning subject line had an EMV of 0%, while the losing subject line pulled in a fair 33% EMV.

My theory is that the subject matter (how to write blog posts) with the promise (no-fail) was simply too good to pass up. Bloggers are looking for help on blogging, period. In any other week, the 50+ post would have been a strong contender.

How does the EMV play into email subject lines?

We’d seen such dramatic results in how EMV (the emotional value) affected social shares of a post. I expected to see the same here, with email subject lines. In this sample of 12 tests, 67% of the winners had the higher EMV value. Emotional headlines AND emotional email subject lines are your best bet.

But does the EMV also affect if recipients choose to unsubscribe to the email? Let’s look.

In the sample of 12, one was a draw (winner and loser had same amount of unsubscribes). Of the remaining 11, the higher EMV also had more unsubscribes.

Perhaps it’s good to remember that emotional reaction works both ways: it causes us to click and share, but it can also cause us to unsubscribe.

Does asking a question make an email more attractive?

I always think that a question is a good way to get people’s attention. MailChimp’s research suggests that is true. However, in our small sample (which is clearly not definitive), that wasn’t the case.

We had two email subject lines with direct questions. Both of them failed to win. However, they both had the lower EMV of the two email subject line options. You can’t get past the importance of how you word your subject line, even if it’s as powerful as a question.

A question alone won’t cut it. An emotional question probably will.

Do winning subject lines also translate into higher post shares?

You’ll notice a few posts with significantly higher shares that have an asterisk. These posts were included in Buffer’s suggested shares and received a great deal of attention. I took them off the table for our purposes here, and used the remaining 10 email subject line tests.

Of the remaining 10, four of the winning subject lines were blog posts that received more shares, while six of the losing subject lines were blog posts that received more shares. And remember, the email subject lines don’t mirror the exact headline of the post, nor do all of the social messages that go out have the same headline, either.

So in other words, not terribly conclusive, except perhaps for one minor note: the wording you use to promote your post in headline, social media, and email really does matter. It’s not enough to write a fantastic post about a subject people are really interested in if you present it to them in a way they are uninterested in.

What should you learn from our email subject line test results?

What works for our email list and readers might have a vague application to your own, but in all likelihood, it doesn’t mean a hill of beans for you. The best thing you can learn from what we are starting to discover is the process of what we did, not the results.

Your results will probably be different, and blindly accepting someone else’s content marketing tips and data as the solution to your own content is not a good idea.

In the never-ending struggle to find a way to get readers to click and read your content, testing your email subject lines (just as you test headlines) is important. Not only for that specific email, ensuring that your best email goes out to your list, but also to learn about what attracts your list the best.

Julie Neidlinger

Julie Neidlinger

Content Crafter, CoSchedule

Julie has been blogging on her Lone Prairie blog for over a decade. She is also writes for the popular CoSchedule blog, a social media editorial calendar for WordPress.

See Full Profile >