Run Cleaner, More Accurate Holdout Tests with Haus Commuting Zones

At the heart of any trustworthy incrementality experiment is strong test design. And when it comes to testing internationally – especially in densely populated areas – experiment design becomes especially consequential.

Whether you refer to these experiments as incrementality tests, holdout tests, GeoLift tests, lift tests, or something else altogether (“regional test and control experiments” is kinda wordy, we admit), one of the most important things you can do as a steward of your paid media budget is invest in incrementality testing that’s robust enough to bet your business on.

Let’s get fundamental for a second: To run an incrementality test, a brand’s ad platform needs to be able to target customers at a granular geo-level such as DMA, Postal Code, or NUTS2.  

But – and this is where things get extra interesting – to run a robust GeoLift test, you want to ensure that each region comprises an area where people tend to live, work, and play. This limits the possibility of the much-maligned "spillover effect.”

What are spillover effects in test design?

Spillover effects occur when a potential customer moves between two different geo-regions, exposing them to both the test and control groups. It doesn’t take a PhD econometrician to figure out how this might lead to unreliable test results. Just imagine a vaccine trial where some test subjects receive the vaccine and the placebo. You’d have no way of knowing what actually drove an outcome.

In the scenario where every participant received the vaccine and placebo, the efficacy would show zero. That’s the impact of spillover: it underestimates lift, lowers iROAS, and increases CPIA estimates. This means you’re underestimating the impact of marketing, which could lead to abandoning effective strategies. 

But spillover effects must be pretty rare, right? Not necessarily. If you’re geo-segmenting a bustling, densely populated city like London by postal code area, you’ll find that individuals move between multiple areas all within the same day. A Shoreditch resident in your holdout group might take the Tube over to Kensington and see the very ad you were “holding them out” from — completely spoiling your robust holdout test.

How does Haus minimize spillover effects in incrementality testing?

If you’re using out-of-the-box region definitions in your geo-testing, like postal code areas, you’re likely to have extreme spillover effects – and thus not-so-reliable incrementality results.

Luckily, Haus has created Haus Commuting Zones, which solve the issue of spillover effects. Based on open source mobility data collected from mobile phone GPS data, Haus Commuting Zones serve up a proprietary methodology for geo-segmenting a country so brands can have confidence in their test design and results from the moment they set up their incrementality experiment in the Haus app. 

“The clusters work well for GeoLift because there is minimal spillover between regions,” says Haus’ Head of Product Dylan Daniels. “They are slightly modified to ensure easy implementation in ad platforms."

When a brand creates a test in the Haus app, Haus Commuting Zones are automatically applied based on the test design – so as a growth marketer, you never have to worry or wonder if you’re setting up the test “right.” Just a few quick selections and you’re off to the incrementality races.

Which brands can benefit from Haus Commuting Zones?

Any brands running tests in Mexico, Canada, Great Britain, and Australia can automatically apply Haus Commuting Zones in the experiment design flow. For enterprise brands that are marketing internationally, Haus Commuting Zones offer a powerful way to add more rigor to your testing program. Given the extensive research put into building these proprietary zones, building something similar in-house would be incredibly time-intensive.

But it’s not just global enterprises that stand to benefit: Brands that truly prioritize rigor and accuracy in test design can apply Haus Commuting Zones to minimize spillover effects. 

What are some other ways Haus designs more accurate holdout tests?

It starts with rigorous randomization (say that five times fast). Haus ensures your randomized control group and treatment group are characteristically very similar by using synthetic controls. Unlike matched market tests that cherry-pick one region similar to your control, a synthetic control group comprises a handful of markets that are then re-weighted to better resemble the treatment group. This leads to fewer errors and more statistical precision. 

Plus Haus’ Science team takes additional data-cleaning steps to soften noise, like accounting for other outliers in your data and closely tracking seasonality trends. And once you receive results, outputs are transparent and interpretable, with clarity around test power and confidence intervals.

Minimize spillover, isolate business impact

It’s all about isolating variables so that you can accurately tie a change in strategy to a change in outcomes. You might not have even thought about the possibility of customers “spilling over” between control and treatment regions, confusing your results. And if not, that’s okay — because Haus already thought of it for you.