Testing on a small emailing list with A/B tests

It’s important for you to tweak and test your emailings until you’ve come up with the perfect document. But how to do so if you have a small emailing list where split-run emailings have little meaning?

If you have a small emailing list, let’s say less than 5,000 recipients, split-run emailings have little use. After all, the test groups are so small that it’s impossible to create a representative sample group, making the results unreliable.

But that doesn’t mean it’s impossible to test on a small emailing list.

A/B test

In the world of email marketing split-run and A/B test are often used in the same context. Contrary to popular belief however, there is a difference between the two.

• When conducting a split-run test, you send two versions of the same emailing to a small part of the send list. The best performing version of the two will then be sent to the rest of the group. 

• With an A/B test, you try out version A first, and send out version B a week later for example. Comparing the results of these mailings, you decide which version to use towards your target group.

As mentioned earlier, when it comes to small emailing lists, a split-run mailing is less likely to give you reliable results. A/B tests however are a suitable alternative.

Create similar circumstances

If you use A/B tests to compare two different versions of emailings, it is of course of great importance to create similar circumstances.

To give you an example:

• You send version A on a Monday, resulting in a 55 percent open rate
• You send version B on a Tuesday, resulting in 75 percent open rate

While one version might have scored better than the other, you still don’t know if it was document B that caused the difference, or that Tuesday is just a better day to send an email than a Monday.