Tracer Bullets to Production: Taking Commerce Learning to Rank Online for its First A/B Test

Taking a machine learning model from architecture diagrams to live traffic is where theory meets reality. The challenge is to move quickly without introducing unnecessary risk. For Learning to Rank (LTR) for Commerce, we used tracer bullets and shadow testing to validate the production path, uncover issues early, and build confidence before launching our first A/B test.

This is the third post in our Learning to Rank for Commerce series. The first introduced the challenges of learning to rank in a multi-tenant environment. The second explored the architecture we designed to separate retrieval from ranking. Now, we turn to the journey of taking that architecture live, refining it, and ultimately demonstrating its impact through a revenue-winning A/B test.

Our first A/B test was, in a naive sense, a failure: no statistically significant lift on conversions or customer revenue, and only a slight improvement on click rank. But this null result wasn’t a wasted step – it was the fastest way to get the key learnings we needed, and it set up the revenue-positive win we’ll cover in a future follow-up to this post. In today’s post, we discuss the road to the first A/B test, and how we got there without harming customers or shoppers in the process.

We deliberately did not elongate the offline study phase before moving into online A/B testing. Spending too long offline keeps you in a local optimum – it delays the surprise rather than avoiding it. And the biggest unknown at this stage isn’t whether the model is good enough. It’s whether the offline definition of “good enough” means anything once real users are involved. It’s one thing driving changes in a lab to your favourite offline ranking metric, but it’s quite another convincing real shoppers buy products in the real world. This gap can be huge, and ultimately a model that shines in the lab but makes no impact to human behaviour is a bad model. We deliberately designed our approach to optimise for measuring online value as quickly as possible.

But we couldn’t be reckless either. We had a fresh new system – multiple components not yet launched at scale – and a ranking intervention whose behaviour across our production stacks we hadn’t yet characterised. So what did we do? Careful haste toward going online. This post is the story of going from experimental offline notebooks to an online null result, and why that null result was exactly what we needed. It enabled the conversion and revenue success we achieved later.

Where We Left Off

Our second post ended with a clean separation: first-stage retrieval handles recall, a second-stage reranker handles precision.

A system design was in place. The offline metrics looked promising. But an offline-only system cannot easily resolve a key uncertainty: how will end users (i.e., shoppers) perceive and react to the new ordering? We can’t perfectly predict user behaviour, especially given strong biasing effects (see the field of Unbiased Learning to Rank for further info).

Two risks sat in front of us, corresponding to the two primary sources of uncertainty for this project:

1. System risk – we were introducing a new reranking component into the live query path. Any latency spike, failure mode, or unexpected interaction with existing infrastructure could degrade the experience for shoppers.
2. Modelling risk – the ML model now owns the final ordering of results. If its notion of relevance diverges from what shoppers actually want, we’ve actively made search worse.

The question this post answers: how do you take a working notebook (offline experimentation in Jupyter) to live traffic without harming customers or shoppers? Haste and safety are required. This is an exercise in managing risk and blast radius.

What’s new in the architecture. To ground the blast-radius discussion that follows, here are the net-new or changed components on the query-time path (for a fuller picture of this architecture, see Post 2) – each one a place where production risk lives.

• (a) ML reranker serving service – new. A web service that receives candidate documents and returns a reordered list. If it fails, we fall back to the baseline ranking. This may compromise A/B test’s correctness. If it slows (while within precautionary timeout limits), shoppers may notice.
• (b) Index calling out to an ML service at query time – new, and high-stakes. The Index is the backbone of our live search.
• (c) Search API passing augmented configuration to the Index – these changes are more routine for us and lower risk, but still part of the picture. The Search API lives on the query path between end user and index, and an integration issue here could silently route traffic to the wrong behaviour.

Each of the following sections addresses how we de-risked these components, layer by layer, before asking real shoppers to be our judges.

The Two-Systems Problem

The model that worked in the notebook is not the same as the model in production. This is not a deployment hygiene point – it’s a fundamental gap across three dimensions.

The environment gap. A notebook is a closed, synchronous, in-memory world. Production is a distributed system with real latency budgets, partial failures, and cascading dependencies across components. The notebook never had to care about any of that; production cannot ignore it. At the scale and variety of customers we serve, this is not a theoretical concern.

The data gap. Feature pipelines, retrieval sets, and request-time data are all subtly different between training time and serving time. Any drift is a silent bias on every prediction. At the time, we had a known subtle difference between product grouping in our training data (in our ranking logs) versus the grouping applied at serving time – a potential skew in the candidate set the model sees. Production is also inherently time-dependent: catalogues are updated, user context evolves, customers’ business rules get tweaked. Our offline analysis approximates these dynamics by reconstructing the past and backtesting, but the approximation is never perfect.

The evaluation gap. Offline, we compute NDCG against relevance judgements derived from behavioural interaction signals. Online, we measure indicators like click rank, click through rate (CTR), conversion rate, and – most importantly – revenue per visitor. These are not the same question. A model can look great on offline relevance and do nothing for revenue. The offline eval is a hypothesis about correlation with online impact – not evidence of it. This is something teams working in relevance optimisation encounter frequently: Mercari’s search team went through multiple rounds of A/B tests where offline improvements did not translate to business KPIs, before eventually achieving a statistically significant shift. This is sometimes called the offline-online correlation gap, and often solved the same way – by shipping safely and iterating quickly.

Why does this matter for our A/B test? If the two systems aren’t equivalent, the experiment isn’t really testing what we think it is. And if the offline eval doesn’t correlate with the online metrics that matter, we can iterate offline indefinitely and never learn whether the model works. Both are reasons to get online fast – but carefully.

Managing Risk: The Blast Radius of a Change

Taking a new system online entails some level of risk – each change we make has the potential to have unwanted side-effects. However, not all risks are equal, and a useful framework for thinking about production impact is to consider the “blast radius” of a change.

Load testing in staging – zero blast radius on customers and shoppers. At Coveo, each environment (e.g., staging vs production) is isolated, with no shared resources. No production workload is affected. There is still impact if something goes wrong – other Coveans share the staging environment, and our path-to-production is sacred – but the blast radius doesn’t reach customers or shoppers.

Shadow testing in production – low blast radius, but not zero. Shadow traffic exercises production infrastructure. The blast radius depends on where in the stack the replication happens – that choice determines which production workloads absorb additional load. The majority of our systems auto-scale, but it is still need to do our due diligence on components that may be affected. Getting this right was one of the more consequential architectural decisions we made; we cover the specifics shortly.

A/B test – intentionally affects end users. The blast radius is the size of the traffic split. If the model degrades experience, the whole split feels it. And, as with shadow testing, the variant traffic will exercise novel systems – if they cannot cope, it puts our availability and latency SLOs at risk.

Tracer Bullets, Not Throwaway Prototypes

The tracer bullet metaphor has a long history in software engineering. When faced with a large, complex, and intricate system, we want to cut a thin end-to-end implementation path through the whole stack. This gets us to something running quickly, that we can safely test with users.

It’s not a throwaway prototype – instead, the code is real (fully tested and built to last), the infrastructure is real, the data path is real. You can test something real – such as testing a product feature with a customer. Or, in our case, test the practical impact of an ML model that improves the ranking of results.

Time to value is always on our minds when building ML-driven products. In this case, our focus is specifically on time to insight: The tracer bullet’s job is to get real signal from production as fast as possible, so that every subsequent iteration is informed rather than guessed. With conventional product features, we often talk about testing a “minimum viable product” (MVP). For projects like learning to rank, we’re focused on the “minimum viable model” (MVM). This is our working hypothesis for a model that will have a positive impact – as validated through offline experimentation – while minimising time to insight.

The specific insight we were racing toward: does the offline evaluation correlate with online impact? Plus, is the architecture sound, and does the infrastructure hold up under real traffic? If those things are true, we can iterate with confidence – the concept is proven, and we might even be ready to product-ise this for all customers. If the online impact doesn’t materialise – we’ve learned that the offline metric needs refining before we can trust it to predict what matters, and that our model needs more work. Either way, you’ve just bought yourself the most important piece of information available.

This raises a question of what to build and what to defer. The distinction that helped us: quality is not the same as grade. Grade is richness of features; quality is fitness for purpose. A low-grade solution can be perfectly high quality.

What we compromised on (grade): feature engineering, model sophistication, handling of customer edge cases we didn’t need to support immediately.

What we didn’t compromise on (quality): SLOs, observability, testing, reliability, code correctness on the serving path.

Why? A tracer bullet that doesn’t fly straight teaches you nothing. If the observability is missing, you don’t know your system is operational. If the serving path is flaky, your A/B analysis will be flawed. If the online model isn’t the model and system we designed, we’re testing the wrong thing. The basics are non-negotiable precisely because the tracer bullet exists to generate trustworthy signal.

Load Testing: A Step Before Shadow Testing

Load testing (including stress, spike, and soak tests) gives you confidence in latency and capacity under realistic traffic volumes – zero blast radius on customers, since it runs entirely in a separate staging environment. We replayed server-side events against a Coveo-owned test organisation, at target queries per second (QPS): representative of real usage patterns, more trustworthy than hand-curated query sets, and reusable across projects.

We have two rounds of tests: one where the system under test was specifically the ML reranking service, and another cross-system test that targetted Search API. The latter exercises the multiple new components involved, and reveals potential bottlenecks.

What it validated: that the new ML reranking component – introducing a new inter-service network call from Index to ML service under production query rates – could handle production-scale load within our latency SLOs, and that compute requirements were appropriate.

Load testing answers the system risk (will it hold?). It doesn’t answer the modelling risk – our load tests run against Coveo-owned data, not real customer traffic. For that, we need shadow testing: the customer never sees the candidate ranking, and we control the blast radius precisely.

Shadow Testing

Shadow testing has two distinct components: the traffic infrastructure, and the tests we run on it.

The shadow traffic

At request time, the production ranking is served to the user as normal. In parallel, a portion of traffic is replicated and the candidate ranking is computed on a separate path – its response is never returned to end users. This is sometimes called a shadow model; more generally, we’re making a shadow intervention on the ranking path.

The architectural decision that mattered most: where to replicate. We had three candidate points – the query pipeline (Search API), the Index, or the ML service. We chose the Index layer.

How it works: live Index instances receive traffic and serve the production ranking. They also replicate a portion of that traffic to separate shadow Index instances. The shadow Index does the heavy work – processing the query against the candidate model, calling the ML service, computing the candidate ranking, and logging the results. The live Index absorbs a small replication overhead, but the full shadow workload runs on its own dedicated instance.

Why this matters for blast radius: if we’d replicated at Search API level, both live and shadow queries would hit the same Index instances – doubling query-processing load on a core component. By replicating at the Index layer and routing shadow traffic to a dedicated instance, the live Index stays insulated. Even in the extreme case of a crash, only the shadow Index is affected. Shadow instances never serve live traffic.

The shadow test

We have shadow traffic flowing, but no user labels – results were never shown to users, so there’s no click or conversion signal. What can we test?

“Your ranking is just maths buried in your infrastructure.”
– Doug Turnbull, at MICES 2024

Even if the model is correct in a notebook, we’ve now mapped it onto a distributed system. The shadow test verifies both operational characteristics and ranking behaviour.

Operational checks: latency, availability, error rates. The shadow path exercises the same production infrastructure under real traffic patterns – standard, but important to confirm.

Quality checks – change magnitude. These are diagnostic metrics (sometimes called “debug metrics”, as per Kohavi et al. 2020). They don’t tell you whether the model is better – they tell you whether it’s doing something, and how much. No reranking, no change – if the reranker produces no rank shift, there can be no downstream impact.

Some basics that we can look at include:

• Rank shift at k: how much re-ordering does the candidate introduce vs the baseline? Measures like Kendall tau and rank-biased overlap.
• Churn at k: how many results in the top-k baseline are absent from the top-k candidate? A simple Jaccard coefficient.

We can get even more sophisticated with these, for example checking characteristics of the products that were promoted. Although they’re “unsupervised” (no quality proxy labels involved), measures like these are useful – and can be extended to include content features, activity aggregates (CTR change), and more. We can also reproduce our offline studies (using supervised performance measures) using these online predictions, as an additional step to identify any offline-online skew.

We break these metrics down into various query cohorts; e.g., by query length and query frequency (head / torso / tail). We’re looking for a Goldilocks zone: enough reranking to be worth running an A/B test, but not so much that it triggers concern before we understand why.

LGTM@k – eyeballing results (a qualitative “looks good to me”). This is a repeat of the same kind of qualitative inspection we do during offline analysis: randomly sample queries (biased toward those with large rank shifts), and use side-by-side comparison tooling to inspect candidate vs baseline on real production results. It’s worth repeating here because production infrastructure can diverge from the offline environment – and in our case, we had some known potential causes of skew. Quantitative checks and qualitative inspection are complementary – you learn things from looking at specific results that an aggregate metric will not surface.

By the time shadow testing is done, we’ve run our system on real live queries, exercising the production stack end-to-end. That verified latencies, confirmed the serving path holds under real traffic, and caught integration issues that no amount of offline testing would surface. All our monitoring dashboards have been proven too – the exact surfaces we’ll rely on once we launch the A/B test.

The Outcome of This First Test: A Safe and Measurable Null Result, Delivered Quickly

With the shadow phase complete, we launched an A/B test – canary first, then ramping to full traffic. We’ll cover A/B testing methodology in detail in the next post;here, we focusonthe outcome and what it meant.

This first test yielded a null result for the key success metrics; i.e., no statistically significant change to conversion rate or revenue per visitor. It’s not the “product acceptance” outcome we wanted – but that’s far-fetched to expect at this stage.

But we did see a promising signal: a consistent and statistically significant improvement in click rank. Relevant products were being pulled earlier up the page, while CTR and conversion rate remained unchanged. The model was re-ranking in the right direction – and removing friction for shoppers – but not yet moving the needle on their decision to purchase.

Conclusions

In this post, we described our approach to rapidly and safely evaluate the impact of a new ML model on user behaviour. In ML ranking, the first online experiment is often a null result – but it’s usually the most fruitful stage of a project, where the biggest course-corrections can be made.

Because the infrastructure was proven solid in every prior ring, this is a null result we can trust – and learn from. For us, reaching this milestone was essential:

• The serving path is proven end-to-end. The tracer bullet has given us a thin-slice of a vertically integrated solution – one we can now extend with confidence, and that we won’t need to completely throw away.
• We now have a concrete online dataset – users’ behaviour change with respect to our model. We can perform post-test analysis and decide where to invest in extending the “grade” of the solution, guided by real online impact.
• We have a baseline to iterate from. And the directional signal on click rank tells us this is worth pursuing.

As our next post will highlight, this null result on the primary metrics wasn’t a dead end – it was an invitation to iterate. It gave us precious data from which to make informed decisions in the next phase of experimentation, from revising our evaluation suite, to extending the feature set.

Who’s Been Inspiring Us

Here are many people we’re thankful to have learnt from through their writing and presentations:

• The Pragmatic Programmer (1999) by David Thomas and Andrew Hunt: The canonical reference for the tracer bullet metaphor, and “Good Enough Software” (quality vs grade).
• Martin Fowler’s “Is High Quality Software Worth the Cost?” (2019): The economic argument for internal quality.
• “Quality” and “grade” – As applied in the context of project management.
• Etsy’s Real-Time Personalized Ranking in E-commerce Search paper from SIG eCom 2021: A valuable demonstration of the relationship between LTR’s offline NDCG gains and online conversion uplifts.
• The Mercari Search team and their multi-year journey with LTR for commerce, especially The Journey to Machine-Learned Re-ranking by Alexander Zagniotov and Richard Calland (2023) and Robust AI Search Ranking for Radical C2C Marketplace Growth by Teo Narboneta Zosa and Chingis Oinar (at Berlin Buzzwords 2024).
• Planning of e-commerce search relevance work by Doug Turnbull (at MICES 2024): What teams should really be optimising for when they bring commerce experiments to production infrastructure.
• Danilo Sato (et al.) on Continuous Delivery for Machine Learning (2019): How shadow testing fits within the variety of deployment strategies for ML.
• Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang, and Ya Xu (2020): The definitive guide to A/B testing, including the taxonomy of goal metrics, guardrail metrics, and diagnostic metrics.
• Spotify Engineering’s Risk-aware Product Decisions in A/B Tests with Multiple Metrics (2024): A compelling framework for how to formulate product decisions as A/B tests, including guidance on experiment design and statistical evaluation.
• James Rubinstein’s A/B Testing Search: Thinking Like a Scientist (2020): On the experimental mindset for search and relevance improvement.

This article is part of our Learning to Rank for Commerce series, following the journey from defining the multi-tenant ranking challenge and separating retrieval from ranking to taking the model into production and testing it’s impact.