We talk a lot about the great benefits of implementing testing strategies with our email efforts. I’ve made multiple references in various posts, and so far have had two dedicated posts on the topic: A/B Subject Line Testing and Email Marketing Testing . Marketers (not just email) LOVE testing. Combine with some juicy analytics and we’re entertained for a long time.

Nothing has changed with the fact that testing should be implemented and will inevitably improve your email marketing efforts. I do feel, however, that there can be a risk of over-testing, or ‘testing burnout,’ if you will. This mostly comes down to available resources a company has and the expected return on marginal testing programs. If we put 10X more resources into testing and only realize an x% increase in sales, our testing program could be producing negative returns.

This has a more costly effect on smaller companies, as the scale of return is much smaller. [The marginal return from an x% increase in open or click rate is much larger for a big company with a list size of 5,000,000 vs. a small company with a list size of 5,000.]

For that small company, testing is indeed important. They need to make improvements to their email marketing efforts and increase sales just like every other company. But at some point a negative return is realized.

Example

• Let’s say a small e-commerce site selling backpacks has a list size of 15,000. Their monthly promotional email brings in $1,350 on average. [25% open, 12% click, 5% conversion, $60 average order]
• This company creates a testing plan that will require an additional 3 hours per month of company resources.
• After the test, they increase their metrics to: [28% open, 15% click, 5% conversion, $60 average order]. In this case we see a $540 increase in revenue. Perhaps a decent result for the small e-tailer. Their gross testing return was $180/hour.
• Now let’s say they create a testing plan that is more robust and requires an additional 30 hours of company resources each month. (compared to no testing)
• After the new testing program, they increase their metrics to: [30% open, 19% click, 6% conversion, $62 average order]. In this case we see an $1,830 increase in revenue. Their gross testing return was $61/hour. Depending on their margins and several other unknown factors, this may or may not be a profitable scenario

These same metrics with a big competitor would have a much different result. e.g. If another company had a list size of 750,000, their marginal return on the last scenario would be $91,530 with a $3,051/hour testing return. Probably quite favorable.

So where do you find that testing equilibrium? It comes down to the unique situation of each business. Regardless of size, start small with your testing program and work up from there. Pay close attention not only to the increased results of your tests, but the amount of resources that go into your various testing programs. Time for different content, designers, approval, segmentation, deployment, review & analysis etc, can add up when you are introducing complex testing strategies.

In the end, you should test to see what testing portfolio is optimal for your email efforts.

Cheers,

Forest Bronzan 

Questions or thoughts? Leave a comment or feel free to shoot me an email