When A/B Tests Fail: Why It’s Not a Loss and What Smart Marketers Do Next

A/B testing is a great tool to prove a point, lead you to solutions you may never think about and use data to optimize and learn rather than just using your intuition to decide what works best.  Many platforms now include it as part of their services, so usually you just need to pick an audience, pick the content and/or data you want to test and then you are ready to start testing.  Even if it’s not simple as that, it’s still not that hard to create in a journey tool once you have a template ready to use.

A lot of times, we know what tests are going to work and do them just to prove a point.  But what happens if a test fails in your hypothesis or set up?  Do you consider it a waste of your time?

The reality is that failures are so valuable in all aspects of life and lead us to learning and optimizing anything.  I usually take that failed test and do an optimization on it rather than just walk away.  It’s also good to know that it didn’t work before putting all your resources into something you might have otherwise implemented.

What Constitutes an A/B Test ‘Failure’?

There are many way that an A/B test may fail. It may have no or low statistical significance.  Many platform tools will give you insight into statistical significance.  If you don’t have that and need help, I’d recommend checking in with an Analyst at your company.

If you’ve formed a hypothesis (either loosely or formal), you may just not expect the results that the testing provided, this may be considered a ‘failure’ to you.

For me, the most common ‘failure’ has been a negative lift in conversions or engagement.  I generally am relying on the data to tell me how to move forward.  It is always a bit disappointing  when this happens even if you know you will learn from it.

You may also have operational issues like broken links, poor segmentation or data tracking errors.  Make note of these to make sure you learn not to miss it in future testing.

Why A/B Test Failures Are Incredibly Valuable

You learn what doesn’t work which will narrow your focus.  You should be constantly optimizing from your A/B testing, but a failure really helps you rule out what you just tested and focus on a different direction.

It uncovers customer behaviors or preferences you may not have anticipated. In the past, I had been absolutely sure providing additional sale items would be the way to go (it wasn’t!) as past testing had shown customers really liked assortment.  The fact that it wasn’t the way to go, helped me narrow in on less content & really improving that.

Failures help prevent costly rollouts of bad ideas.  When I do have a failure, it does help me assess if I should go in a completely opposite direction or just optimize what did not work in this test.

A/B tests help us to make decisions based on data.  In the past, I’ve ensured that tests are statistically sound, properly segmented, and aligned with clear hypotheses. A/B testing drives a culture of experimentation, where we use data to learn quickly, reduce risk, and make informed product decisions.  It is a great tool for innovation and continuous improvement across the customer experience.

What to Do After a Failed Test

• Diagnose: Was it truly a failure, or was there a setup or execution flaw?  When you first start testing, you learn a lot about execution flaws, which allows you to create a checklist of how to set up more efficiently for the future.  Aside from that, it may truly be a failure, but just knowing what type of fail it was will help you with the rest of the steps.

• Analyze: Take a close look at the data to look for micro-insights.  This could be device types, time of day, outlying segment behavior, category affinity, etc.  Make sure to look at a variety of metrics like click rate, conversion rate, AOV, and RPE.  If a website test, there would be additional metrics like bounce rate, average time on page, or behavior flow.

• Document: Note results clearly for future reference.  Be sure to note the type of test (SL, content, segment, landing page), give a summary of the test, hypothesis, analytics, and your learnings.

• Communicate: Use your documentation to share findings with your team to prevent repeated mistakes. This is where you can use a test that you considered a failure to learn.  It’s especially helpful to get your teams perspective on it and brainstorm what you test next based on the results.

  
  

How to Reframe Failures as Part of a Testing Culture

Encourage curiosity over perfection.  I do this by keeping an excel list of my tests where I can easily see an overview and hypothesis of each test.  Next, I start thinking of more questions that I want to answer.  Any test is a good place to start!

Celebrate learnings, not just wins.  As part of your testing and summary metrics, include your learnings.  I think it’s easy to just include the analysis of the test and which element won, but you want to look at learnings from all tests, not just the tests that gave you expected results.

Create a “test-and-learn” roadmap that embraces iteration.  It’s great to document all of your tests with hypothesis, analysis, outcome & learnings, but make sure to look forward as well with a roadmap.

Examples of Famous A/B Test Failures That Led to Big Wins

Bing’s Slight Ad Color Change (Source: AI Overview)

• The Test: Microsoft’s Bing changed the color of ad titles to a slightly different shade.

• The “Failure”: The change was so small it seemed insignificant and was easy to dismiss.

• The Win: That tiny adjustment reportedly increased Bing’s ad revenue by $80 million annually, proving small changes can have massive business impact — even when they seem like non-events in testing.

  
  

SimCity 5 pre-order discount (Source: AI Overview)

• The hypothesis: Offering a discount on a future purchase would incentivize pre-orders for SimCity 5.

• The unexpected result: Removing the discount and simply offering the game boosted sales by over 40%.

• Learning: Understanding customer motivation is crucial. SimCity enthusiasts were more interested in playing the game itself than in a future discount on potentially unrelated products.

• The Win: This understanding led to a significant increase in pre-order sales and helped the company realize the value of focusing on the core product's appeal. 

  
  

Keep on Testing

Make sure to reiterate. The only true A/B test failure is one you don’t learn from.   I hope this insight has helped you appreciate what you considered a failure.  Keep testing, learning, and adapting!