Retail media has transformed e-commerce by allowing brands to advertise their products directly to consumers right at the digital point of sale. However, in a complex advertising marketplace, testing a new feature requires more than a conventional user-level A/B test.

Most A/B tests rest on a simple assumption: what User A does doesn't affect User B. In a standard consumer app, that's mostly true: you randomize users, show them different variants, and compare the results.

Retail media, however, operates slightly differently. It's a two-sided marketplace where advertisers compete for visibility via ad placements, customers see the results, and the whole system runs on a finite resource: advertisers' budgets. When experiment variants draw from the same campaign budgets, the statistical validity of the test is compromised. The control and treatment groups lose their independence, entering a cycle where they actively engage in Cannibalization Bias against one another.

In this blog post, we describe how we used Budget Split and Orthogonal Concurrency to address this challenge within our experimentation framework, and how these capabilities improved our testing between 2023 and 2025.

SUTVA Violations and Budget Cannibalization

In a standard A/B test, the validity of your results depends on the Stable Unit Treatment Value Assumption (SUTVA). The core pillar of SUTVA is no interference between the variants of the experiment, meaning the treatment assigned to one user should not affect the outcome for another user. In an advertising marketplace, this assumption is easily violated.

Consider a scenario where a novel, more effective bidding algorithm (Variant A) is evaluated against the baseline system (Variant B). An issue emerges when both variations pull resources from a single, shared wallet. Variant A's superior efficiency allows it to secure more auction wins, depleting the budget at an accelerated rate. This rapid consumption inadvertently restricts Variant B's access to funding, artificially depressing its performance metrics. This phenomenon, termed Cannibalization Bias, dilutes the true average treatment effect and skews the experiment's results.

Creating Parallel Universes with "Budget Split"

To solve this, we introduced a capability called Budget Split, a mechanism for allocating separate budget pools to each experiment variant.

Conceptually, Budget Split creates isolated universes for each variant. Instead of having Treatment and Control fight over a single pool of money, we divide the campaign's budget upstream, proportional to the traffic split.

How it works under the hood:

• Traffic Randomization: We assign users to Treatment or Control.
• Budget Partitioning: We split every participating ad campaign into virtual "sub-campaigns".
• Isolated Auctioning: Treatment users only trigger bids from the Treatment sub-campaign budget, while Control users only trigger bids from the Control sub-campaign budget.

By effectively creating two identical, independent marketplaces, we block the confounding path of budget consumption. If the Treatment outperforms the Control, we know it is because of the feature we built, not because it cannibalized the other variant's budget.

Engineering Orthogonal Concurrency

The ability to execute concurrent experiments without overlapping interference remains a technical challenge in online experimentation, a challenge that becomes more difficult in a Budget Split architecture.

To solve our concurrency problem, we adopted Orthogonal Concurrency. In this model, if two experiments (Experiment 1 and Experiment 2, each with A/B variants) are running simultaneously, a campaign's budget is split into four orthogonal buckets: 1A2A, 1A2B, 1B2A, and 1B2B.

Considerations

In a complex, automated marketplace, absolute isolation is nearly impossible. However, Budget Split and Orthogonal Concurrency have brought the system closer to satisfying the no-interference conditions required by SUTVA.

Despite splitting budgets, we actively monitor for residual SUTVA violations:

• Algorithmic Campaign Steering: Mechanisms like daily budget shifts and max bid adjustments optimize based on the combined performance of both variants. If the treatment affects run rates or return on ad spend (ROAS), it indirectly shifts the budget or bids for the control group, creating bias.
• Pre-Experiment Machine Learning Effects: Our pCTR (predicted click-through rate) and pCVR (predicted conversion rate) models train daily on historical data (using the last 7 and 28 days, respectively). Although our architecture supports separate training pipelines for ad personalization ML models, pre-experiment event distributions (typically reflecting control groups) are still incorporated into the treatment group's data, thereby influencing its training data distribution.
• Manual Advertising Operations Interventions: Advertising operations staff may manually adjust budgets or bids across markets based on aggregate campaign performance, introducing human-in-the-loop interference.
• Timing & Right-Censoring: Campaigns that extend beyond the experiment window can create right-censoring selection effects that skew revenue interpretations.

When A/B testing is not possible, or if we cannot guarantee isolation, we pivot to causal inference methodologies (like non-experimental counterfactual analysis) to inform our decisions.

The Impact: Scaling Innovation and Concurrency

Implementing Budget Split was as much about statistical accuracy as it was about unlocking engineering velocity and fostering a culture of experimentation.

Before this architecture, running experiments was flaky and had to be done sequentially. We faced both a reliability problem and a "concurrency problem": we could not run multiple tests at the same time without them fighting for the same traffic and budget.

This increased the speed and reliability of our experimentation process. ZMS experimentation scaled from 8 experiments without Budget Split in 2023 to more than 60 experiments with the isolation properties described above by 2025, increasing experiment volume while enabling more frequent and more reliable decision-making.

For our Product Managers, Product Analysts, and Applied Scientists, Budget Split is a prerequisite for trust. It empowers us to accurately gauge causal impact, fail faster when hypotheses don't pan out, and innovate continuously.

We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a Software Engineer!