How to Do Newsletter A/B Testing With AfterShip Email
Marketing can unlock endless opportunities if you figure out which strategies are the best for your business. With AfterShip Email, you can do newsletter A/B testing to improve email open and click rates. This help guide covers everything you need to know.
You can create a new email, use a draft, or use an email you’ve already sent. If you choose to use an email already sent, a copy will be created for you to edit and prepare for A/B testing. It will not affect the emails already sent, so be careful to avoid sending users the same email twice.
Log in to your AfterShip Email account and go to the Emails section
Decide whether you want to do A/B testing with a new email, a draft, or a sent email. Once decided, click Create A/B test and then choose whether you want to test the email content or send time
Choose email content if you want to check what kind of email subject, sender info, content, and style gets a better response from recipients.
Choose send time if you want to know what would be the best time and date to send your newsletters.
Recipient distribution: Divide the recipients between version A and version B. The winning version will be automatically sent to the remaining recipients after the test ends
You can also do a full split A/B test with no remaining recipients and no winning version, i.e. 50% will receive version A and 50% will receive version B.
Winning metric: Choose the metric which determines the winner between version A and version B. You can choose either open rate or click rate. (only for non-full split A/B test)
Test duration: Decide how long you wish the A/B test to run. Once the test ends, the winning version will be picked based on the winning metric. (only for non-full split A/B test)
Low sample size version: If you have less than 1,000 email recipients for an A/B test, you can conduct a full split with no winner as normal. But if you choose a non-full split with a winner, the A/B will not run, and the low sample size version will be sent to all recipients. Basically, you need at least 1,000 recipients to calculate a winner
Fallback version: Choose which version will be sent to the remaining contacts if the test result is inconclusive
To perform A/B testing based on the send time, you must conduct a full split with all recipients split between versions A and B, and no winning version. This is because the send times will have passed when the test ends, and so no winning version can be sent at the exact time on the very same day.
You just need to set the Date, Time, and Timezone for both version A and version B. Once done, click Create
Once your newsletter is sent based on your A/B test settings, click the three horizontal dots (⋯) and click View report to check how your newsletter has performed.
If you have an odd number of recipients, the extra recipient will always be sent to version A
The fallback version will be triggered in two scenarios:
When there is no winner. For example, both version A and B have the same open rate
Statistical significance is low
Switch from version B to version A if you want to make changes to any of these fields: “Campaign name,” “Send to,” and “Follow-up email”
Just visit the Newsletters section and use the filter to find which emails have run an A/B test.
Or you can simply scroll down to your newsletter list, as we put an A/B test badge on all the newsletters that have been used for A/B testing
To check A/B testing info, just click on the dropdown arrow and get all the details like this:
Yes, just open your newsletter to check the A/B test settings that were set earlier.
For any other questions, please contact our chat support team
Steps to perform newsletter A/B testing with AfterShip Email
You can create a new email, use a draft, or use an email you’ve already sent. If you choose to use an email already sent, a copy will be created for you to edit and prepare for A/B testing. It will not affect the emails already sent, so be careful to avoid sending users the same email twice.
Log in to your AfterShip Email account and go to the Emails section
Decide whether you want to do A/B testing with a new email, a draft, or a sent email. Once decided, click Create A/B test and then choose whether you want to test the email content or send time
Choose email content if you want to check what kind of email subject, sender info, content, and style gets a better response from recipients.
Choose send time if you want to know what would be the best time and date to send your newsletters.
Scenario 1: Testing email content
Recipient distribution: Divide the recipients between version A and version B. The winning version will be automatically sent to the remaining recipients after the test ends
You can also do a full split A/B test with no remaining recipients and no winning version, i.e. 50% will receive version A and 50% will receive version B.
Winning metric: Choose the metric which determines the winner between version A and version B. You can choose either open rate or click rate. (only for non-full split A/B test)
Test duration: Decide how long you wish the A/B test to run. Once the test ends, the winning version will be picked based on the winning metric. (only for non-full split A/B test)
Low sample size version: If you have less than 1,000 email recipients for an A/B test, you can conduct a full split with no winner as normal. But if you choose a non-full split with a winner, the A/B will not run, and the low sample size version will be sent to all recipients. Basically, you need at least 1,000 recipients to calculate a winner
Fallback version: Choose which version will be sent to the remaining contacts if the test result is inconclusive
Scenario 2: Testing send time
To perform A/B testing based on the send time, you must conduct a full split with all recipients split between versions A and B, and no winning version. This is because the send times will have passed when the test ends, and so no winning version can be sent at the exact time on the very same day.
You just need to set the Date, Time, and Timezone for both version A and version B. Once done, click Create
Once your newsletter is sent based on your A/B test settings, click the three horizontal dots (⋯) and click View report to check how your newsletter has performed.
Important info to note:
If you have an odd number of recipients, the extra recipient will always be sent to version A
The fallback version will be triggered in two scenarios:
When there is no winner. For example, both version A and B have the same open rate
Statistical significance is low
Switch from version B to version A if you want to make changes to any of these fields: “Campaign name,” “Send to,” and “Follow-up email”
How will I know which emails have run an A/B test?
Just visit the Newsletters section and use the filter to find which emails have run an A/B test.
Or you can simply scroll down to your newsletter list, as we put an A/B test badge on all the newsletters that have been used for A/B testing
To check A/B testing info, just click on the dropdown arrow and get all the details like this:
👀 Is there a way to check my A/B test settings once the test is completed?
Yes, just open your newsletter to check the A/B test settings that were set earlier.
For any other questions, please contact our chat support team
Updated on: 05/07/2023
Thank you!