a healthcare marketers guide to email A/B testing
Email A/B testing allows marketers to build and send more impactful emails to increase results
What is Email A/B Testing?
Email A/B testing is a method used to test two versions of a marketing asset to determine which version creates a better audience response. For example, if you are sending a monthly wellness newsletter and you want to include only the most engaging hero image, you can use an email A/B test. To run this A/B test, half of your abbreviated test audience will receive the version with ‘image A’, and half the audience will receive the version with ‘image B’. The version with the best performance is deemed the winner and sent to the full target audience for the broader send.
Why A/B Test?
Email A/B testing is helpful when framing and developing your marketing strategy. While basic reporting data like opens, clicks, likes, and impressions provide insight into how recipients engage with your messages, these metrics in a vacuum cannot help you determine how to maximize the performance of a single campaign. Email A/B testing enables building and sending better, more compelling messages to your audience.
It is often easy to lose sight of your healthcare organization's goals when developing and executing a particular send. Email A/B testing is a great tool to help you and your team hit your KPIs for a specific campaign.
Configuring an Email A/B Test
To get the most insight out of email A/B testing, only a single part of your email should be altered while the rest stays the same. For example, if you are A/B testing the subject line for your monthly newsletter send, you might include the following two variations:
Subject line A: “Is your immune system ready for Fall?”
Subject line B: “Open 365, Even on Holidays.”
In this case, you want to test which subject line is more effective in encouraging recipients to open the email. It’s important to only change the subject line; otherwise, you would not be able to determine what part of the email most directly affected its open rate. Isolate one variable and hold the others constant.
Selecting an Audience Size
An important element to consider when determining what to test is the size of your test audience. When A/B testing an email item where the winner is determined by clicks (e.g. one content block vs. another), you will need a larger sample size because clicks are far rarer than opens. As of 2020, the average click rate for healthcare emails was 3.01% compared to the open rate at 26.88% (source).
Test: Content Areas
Winner Metric: Clicks
Sample Size: 300
In this test, though group A outperformed group B, a margin of 2 clicks is not enough to determine a winner. It's very possible group A’s content area is superior, but the only real way to confirm that theory is by testing it on a larger sample size. If you have questions regarding appropriate sample size, confidence level, or margin of error, consider using a sample size calculator when determining if a campaign is well suited for an A/B test.
Test: Subject Lines
Winner Metric: Opens
Sample Size: 5,000
In this example, the sample size is large enough to illustrate that group B engaged more with their subject line than group A. The remainder of your audience should receive the email with the subject line sent to group B.
Email Elements to A/B Test
Content Areas
Test two different images or two sets of copy located in the same content area. When doing so, make sure the rest of the layout remains the same for emails A and B.
Related metrics: Clicks, Conversion
Send Time
Email metrics can be affected by the timing of when a consumer receives an email. For example, you can try sending an email to one half of your audience at 4 pm and send it to the other half at 8 am. Use these insights to help with similar sends in the future.
Related Metrics: Opens, Clicks
From Name
Testing ‘from name’ will allow each test group to see different versions of who the email is from. For example, one group will see “Socratic Health” and the other will receive “Shannon from Socratic Health”. Again, whichever option gets higher engagement will be sent to the remaining audience.
Related Metrics: Opens, Unsubscribes
More Elements to Test
Beyond what we’ve detailed above, there are other areas you can test, as well:
- ‘Preview text’ can often entice a recipient to make an open decision when scrolling through their inbox. Use open rates to determine a winner.
- Testing message length can determine how much text and content to include in your email.
- Experiment with different verbiage, color, and location of your CTAs to find the most effective option to send to recipients.
- Moving around content areas within the message can change how recipients interact with the email. Test which email layout is the best performing option.
- Testing the number of CTAs present in your message can determine whether consumers prefer one or multiple redirections to your website.
