3 Reasons To Retest Your Incrementality Experiments

So, you’ve designed the perfect marketing experiment, collected your data, and interpreted your results. Your next question might be, “how long are these results relevant?”

The results from a single experiment may not withstand the test of time. Say, for example, you tested the incrementality of your Pinterest ads and found they drove no lift for your business. But now, you are in the peak season for your business, those test results may not apply. Or, maybe you ran an experiment in a period of unusually high economic turbulence that’s since slowed down. Countless factors can make your perfectly executed experiment’s results less durable. 

And even if there are no extenuating circumstances to consider, the world of marketing is constantly evolving. Paid marketing is a significant investment, so you want to make sure your campaigns are working as you expect. One channel working 10% better or 10% worse can be the difference between you making your quarter or not. That’s why it's best practice to periodically retest. 

Certain circumstances, like changing your business strategy or marketing plans, getting unexpected experiment results, and the passing of time are all good signals that it’s time to revisit your experiment.

Let’s dive into each:

‍

1. When Your Strategy Changes Significantly 

‍

Your business is constantly changing, whether you’re shifting budget due to seasonality or launching new products. By retesting your big bets, you can quickly ensure that your marketing is best set up to drive business results.

Re-test your experiments when you experience a change that could impact your marketing strategy, like: 

• Increasing or decreasing spend on a channel
• Adopting a new marketing channel or new campaign tactic
• Changing campaign setups
• Introducing a new product or service
• Focusing on a new customer segment

Retest to understand how these new factors impact performance and measure how effective they are to your business’s bottom line. 

‍

2. When the Original Results Were Unexpected 

‍

If you get a surprising result in an experiment, you can consider rerunning the test before making any drastic changes to your strategy. This “double-checking” allows you to rule out any flukes or outside influences your experiment may not have accounted for and it gives you peace of mind before making a large change to your strategy.  

For example, maybe you test a channel that you’ve invested in for a long time and the results come back with no lift. Turning off this channel entirely would be a massive strategy shift so it might be worth running the test once more so you can feel confident in your decision before turning the channel off entirely.

Retesting can help clear up unusual results like these. And if a retest comes back with the same results, you can confidently trust and act upon the confirmed data.

‍

3. On a Set Schedule, Even if Nothing Has Changed

‍

Perhaps the safest answer on when to retest is on a set schedule. You’ll catch any variations in performance, even if there aren’t external factors to your knowledge that could have influenced the campaign. 

Ad platforms constantly tweak and optimize their algorithms, and since these businesses closely guard their algorithms, there’s no way to correct for the changes being made. As a marketer, you’re given a moving target – with little information on what direction it’s moving in.

You may think retesting isn’t necessary if you haven’t changed your strategy on those platforms. But since the AI charged with distributing your ads changes, what might have performed well on a platform a few months ago won’t necessarily continue to do so. Retesting gives you the chance to make tweaks and ride the wave of algorithm changes.

Retesting is especially helpful for navigating ad platforms’ auction-based systems. Auction algorithms vary constantly based on who and how many people are bidding, the highest bid price, clickthrough rates, and much more. 

Even if you haven’t recently changed your bidding strategy, a competitor may have changed theirs. Their changes could result in them winning the auction over an ad of yours that was previously performing well. This variance in the market makes retesting a good idea.

We recommend retesting tactics on the channels you invest most in at least once a quarter. This cadence gives you enough time in between tests to utilize your results, but is frequent enough to give you confidence in the channel.

It's often tough to gather conversion data on a quarterly basis for offline channels, like direct mail, TV or billboards. We recommend annual retesting for harder to measure channels like these.

‍

Make Retesting Easy with Haus 

‍

Retesting your marketing experiments is important--but on your own, it's also time-consuming and expensive. That’s where Haus comes in. 

Our state-of-the-art platform allows you to design your experiment and run it right within the product. We make sure your controls are airtight, your experiment is powered well for accurate results, and analyze your results for you. The best part? Our experts can get you results in weeks instead of months, making retesting much easier and faster.

‍

Request a demo today.

‍