What the M&M Duels can teach us about A/B Testing

By May 9, 2019
Share this:

A few years ago, there was a funny story going around the Internet by a guy who said he would find the M&M champion in each bag through what he called M&M Duels.

He would dump out his entire bag of M&Ms, select two at random, and crush each of them between his thumb and index finger. He would eat the loser and set the winner aside. He did this for all the M&Ms until he had reduced the number by half. Then he would run through them all again in the same way, eating the loser, setting the winner aside.

He did this over and over, until he found the one M&M champion. This one he would send back to M&M Mars with a note that said, “Please use this M&M for breeding purposes.”

Albeit a ridiculous, and cavity-inducing mission, it is a perfect metaphor for proper A/B testing that marketers should be doing on most of their email marketing, plus social and Google ads.

Marketing Combaaaaat!

Think of A/B testing as marketing duels. You squeeze two elements of a marketing message and see which one gets the higher number of clicks or reads. Set the winner aside, discard the losing element, and test two new ones. Do this over and over until you find the best elements that yields the greatest results. That’s the combination you throw your marketing budget behind.

But you need to follow the right methods for proper A/B testing otherwise you could skew your results and not find the one true champion.

Single Variable Testing

You should only test a single variable at a time. If you’re running a display ad, you’ll have several elements you could test, including the headline, the graphic, the body copy, and the call to action (CTA). (There are more than that, but we’ll focus on these.)

One rookie mistake is that marketers will test Headline A against Headline B, and at the same time, test Graphic A against Graphic B. When you get the final results, which one was the winner? Was it Headline A? Or did people only click it because they liked Graphic A? Would Graphic B perform higher if it had been paired with Headline A? There are too many variables in play to know which variable was actually the cause of the higher clicks. Down the rabbit hole we go.

Keep It Moving

Another common mistake is to run the winning element through a variety of combinations with each of the remaining elements: Headline A with Graphic A, then with Copy A, then with CTA A, then pairing it with each of the B elements. You would then pair off Headline B against each element, then start with the Graphics, Copy, and CTAs, pairing each of them with each element.

This might help you find the absolute best combination, but it would take months to complete and you would waste a lot of time that could be spent actually pushing the ad or email to the largest audience.

Stats Don’t Lie

A good test always has a large enough sample size. If you are testing email subject lines on 14 people, the results won’t tell you much. If your email send is targeted at 1,200 people, you’ll want to consider running a 10/10/80 test, where 10% (120) will receive Email A, another 10% (120) will receive Email B, and the remaining 80% (780) will receive the winning email. Pro tip: you will want to wait at least 24 hours for the first 20% to deliver so you receive conclusive results before sending to the remaining list.

Are you implementing A/B testing in your marketing efforts?

Share this:
s