Matched Market Tests Don't Cut It: Why Haus Uses Synthetic Control in Incrementality Experiments

The first rule of good incrementality test design: Make sure your control group is very similar to your treatment group. The more characteristically similar these groups are, the more precisely you can measure the impact of your marketing. 

Unfortunately, this is where too many incrementality platforms cut corners. Instead of embracing the math to produce very similar testing groups, they wing it with a matched-market test (MMT). A good example of “matched markets” could be Detroit and Milwaukee. Both cities have comparable populations and sorta-similar income profiles. If “close enough” is the goal, MMTs will do. But acting on “close enough” data could lead to flawed strategy and inefficient ad spend.

That’s why Haus takes test design a step further. Forget MMTs — a more exact regional match for Detroit might combine aspects of multiple untreated regions. Say, 70% of Milwaukee, 20% of Philadelphia, and 10% of St. Louis. (St. Milwaukadelphia? We’ll workshop it.) 

This Frankenstein’s monster of metro areas is known as a “synthetic control group.” Instead of cherry-picking one similar region, a handful of markets make up your control group. Haus uses the whole country's data, rather than just a little bit of it. Then you can adjust the weighting of these markets to better resemble the treatment group, as below:

Haus always uses synthetic control methodology. While many MMT providers just pick from a small fraction of designated marketing areas (DMAs), Haus uses all 210 DMAs when designing tests. This comprehensive sample produces test results that are less error-prone and more statistically precise compared to MMTs. Then we clean the data in these groups to reduce outliers and add yet another layer of precision.

It’s easy to tell you that. But let’s run an experiment and show you how Haus’ synthetic control groups offer more precise incrementality testing than platforms using MMTs.

Putting synthetic controls and MMTs to the test

To show how Haus’ synthetic control methods outshine MMTs, we ran a simulation using actual revenues from a representative Haus customer’s geo-based holdout experiment. 

First we pulled out a six-month pre-treatment period where the customer’s control group and treatment group hadn’t received the marketing intervention in question. 

Next, we computed estimated MMTs using this six-month dataset. Sparing you the in-the-weeds data science, we generated a large dataset using one-to-one matching of similar DMAs. We calculated the variance in marketing lift estimates generated by this data set, which helped us assess just how much of a match these “matched markets” were.

Then we did the same thing using synthetic control methods. We generated a large dataset of treatment groups and control groups using synthetic controls (i.e. weighted combos of DMAs rather than one-to-one matching). And again, we calculated variance in estimated marketing lift to determine how comparable these treatment groups and control groups were. 

What did we find? Haus’ synthetic control methods produced results that were 4X more precise than those produced by MMTs. In short, Haus creates synthetic markets that are more comparable — ensuring more precise insight into marketing performance. 

4x precision matters in real-life media spend decisions

Imagine this: A brand runs a geo-based experiment to test the impact of YouTube ads on sales. The experiment delivers an incrementality lift of +5%, but how much confidence can they have in that result?

Scenario 1: Matched Market Test (low precision)

The confidence interval (range of possible true values) is -3% to +13%. The wide range makes it hard to determine if the campaign actually drove growth or if the results are just noise. How do they know for sure? Do they rerun the experiment multiple times? Do they take a swing in the dark? The team hesitates to scale the campaign because they lack confidence in its effectiveness.

Scenario 2: Haus Synthetic Control (4x higher precision)

The confidence interval is much tighter: +3% to +7%. Now, the team can be confident that YouTube drove a positive lift. With this clarity, they reallocate budget, doubling down on YouTube and optimizing their strategy.

Why synthetic control groups are more precise

A city has lots of noise — literally and figuratively. Comparing regions one-to-one (a la MMTs) puts you at risk for outlier events in that control region. This introduces noise as you compare marketing outcomes to your treatment region. But when the treatment and control groups consist of multiple “spliced together” regions, more noise is averaged out.

Haus’ synthetic control method accounts for regional variation, which helps brands better understand how a nationwide marketing initiative might perform. For instance, comparing marketing outcomes in Detroit versus Milwaukee could help you predict how a campaign will do in Grand Rapids. But comparing Detroit to a combination of far-flung American cities helps you better predict how a nationwide rollout might perform. 

Beyond synthetic control groups, Haus takes additional data-cleaning steps to soften noise, like accounting for other outliers and tracking seasonality trends. 

"When you’re working with MMTs, an anomaly like a natural disaster or supply-chain issue mid-experiment would be catastrophic to your results,” says Tyler Horner, Haus Analytics Lead. “But with Haus, it's simply not something you have to worry about."

Wondering how a change in strategy is affecting your marketing’s ability to break through and drive conversions? Our bench of PhD causal inference experts builds the synthetic control groups that get you closer — offering a more representative picture of your marketing that you can quickly act on. 

Sometimes all it takes is a team with a little extra muscle – and a genuine interest in solving hard problems thoughtfully.