What happened when we adapted agentic “autoresearch” to production applied science.
There is an irresistible simplicity to Andrej Karpathy’s autoresearch. Give an AI agent a Machine Learning (ML) model, a measurable objective and a way to run experiments. Let it change the code, evaluate the result and repeat. Instead of helping with one isolated task at a time, the agent now participates in the scientific feedback loop itself, autonomously working towards an improved model.
Applied-science teams rarely have a shortage of ideas. The scarce resource is usually the time required to turn each idea into trustworthy evidence. We wanted to know whether we could adapt autoresearch to help us improve Search Page Optimizer (SPO), Coveo’s learning-to-rank system for commerce. Our reality is rather more complex than the idealised lab setting autoresearch was born into. Our experiments are not five-minute local training runs against a single unambiguous score. Instead they involve production-quality code, remote infrastructure, protected data, careful experiment design and hours of execution.
So we built something deliberately less autonomous: semi-autoresearch.
Ideas are cheap, but time is limited
SPO reranks commerce search results using models trained separately on each customer’s search interactions. We have written previously about the challenge of building one ranking recipe that works across very different commerce businesses and about where that reranker fits in a production search stack.
By the time this project began, SPO was working towards Beta. The core system worked, the model was good, the production path was taking shape, and many of the most obvious model improvements had already been explored. However, we still had plenty of ideas on how to improve things further: alternative learning algorithms; different losses and labels; sampling strategies; bias corrections; hyper-parameters and new features.
The problem we had was not deciding whether any individual idea sounded reasonable. Most of them did. The problem was that implementing, testing, running, analysing and documenting every reasonable idea would take time away from our primary objective at this stage: getting SPO ready for customers.
Adding the “Semi” to Autoresearch
The appeal of autoresearch was that we might be able to farm out that backlog and let an agent work through it. The design was close to what we needed, but not identical, so we adapted it into a deliberately supervised loop: the “semi” in semi-autoresearch. We organised the work as a repeating cycle of five stages, which follows the normal process we would work through for any idea.
Hypothesise → implement → execute → analyse → document → repeat
We divided the responsibility for different stages of this workflow between human experts and agents.
The agents read our accumulated research backlog, proposed candidate hypotheses, implemented approved code changes and tests, prepared experiment configurations, analysed aggregate results, drafted experiment journals and updated the research record ready for the next cycle.
Humans remained responsible for choosing which hypothesis to pursue, approving the experimental method, triggering the execution of the experiment and deciding whether the evidence justified accepting or rejecting the result.
This separation of responsibilities was defined so that humans remained accountable for the highest leverage and most complicated decisions, and to insulate agents from unfettered access to sensitive data.
We built robustness into the system by requiring agent driven review after each critical juncture, leveraging a technique we termed “persona reviews”: several parallel LLMs, each instructed with a different perspective, produce independent verdicts on the quality of a given stage. Proceeding to the next stage required convincing this panel of experts.
Not Instant Research, but Much Less Human Effort
Our first iteration began as a single skill outlining the entire process that the agent would refer back to over the course of an experiment. This workflow was productive almost straight away: the agent was able to reason over our accumulated research backlog, propose logical hypotheses the human could select from, produce code that ran and write ups that looked plausible.
It did not make experiments instantaneous. Reasoning, coding and remote evaluation still took hours. Crucially however, we did observe a large change in active human effort. Several experiments could progress during a normal day because the agent handled much of the implementation, configuration, first-pass analysis, review and documentation. Human attention concentrated on selecting the idea, checking the method, authorising the run and interpreting the evidence.
That made breadth-first exploration practical. We initially prioritised cheap, contained changes that could reuse the same training data and were quickly able to explore dozens of interesting ideas. Most of those experiments did not improve the model, which was a useful result: a well-supported null result still closes a door and prevents a plausible idea from returning every few months in a new outfit.
Within the region we explored most thoroughly, the pattern was consistent with our original suspicion: gains were generally incremental. We found a few small, credible improvements, but no magical configuration that transformed the model overnight.
Faster Experiments Also Produce Faster Mistakes
However, everything was not as idyllic as it first seemed. The clearest problems appeared when we moved from contained model changes into new feature construction. Human review caught that several of our experiments were flawed.
Some features were constructed in a way that allowed an observation to contribute to its own historical aggregate. Others used information from later in time or had very different coverage in training and evaluation data. In some cases, the mere presence of a feature revealed that an example had appeared in the training process, even if the feature value itself looked harmless.
This type of problem is known as information leakage: the model has access during training or evaluation to information that would not have been fairly available when making a real prediction.
In other work, we saw a result that appeared to have the wrong sign and where the agent had added a plausible explanation for how the sign had been reversed upstream. The explanation looked like diligence, so the question temporarily closed. A later check showed that no such reversal existed. An unexplained anomaly would have kept us curious; a coherent explanation made us less likely to look.
In all of these cases the errors were insidious: the code still ran; the metrics still moved and the reports looked like good science at a surface level. But the results weren’t trustworthy. Moreover, as soon as results were recorded in our ongoing research log, future iterations treated them as fact, meaning mistakes were beginning to compound over time.
Complexity × Verifiability
For an individual experiment, the most useful way we found to predict reliability was to consider two dimensions: how complex is the change, and how easy is it to verify?
Agents excel on well defined tasks with clear, verifiable success criteria: switching a learning algorithm; changing a hyper-parameter or running a clean loss-function ablation has a clear treatment, a clear baseline and an executable result. These tasks still require care, but less domain knowledge and expert craft is hidden inside the implementation.
Novel feature construction was different. A new feature can be syntactically correct and statistically predictive while being temporally impossible, unavailable during serving or useful only because of an accidental difference between training and test coverage. Verifying it requires understanding the data-generating process and instrumenting the feature so that its behaviour can be confirmed, rather than computing an average blindly and accepting that a test passed or a training metric improved.
That well-factored tasks with verifiable success criteria produce better agentic work is not a new observation. What a research loop adds is a sharper question about which kind of verifiability matters. It is not whether the code is correct; it is whether the comparison is. A passing test suite tells you nothing about whether a feature could have existed at the moment of prediction.
Improving the Process
There was promise here: many of the experiments were good and the reduction in cycle time was real; so we invested in improving the process.
The first improvement was to split one broad research capability into focused phases. Hypothesis generation, implementation, execution, analysis and documentation each needed different context and different success criteria. Carrying everything into every phase made the instructions bloated and encouraged concerns to blur together; it also caused the agents to drift in quality over long sessions. To address this we broke our single skill into five focused stages with defined success criteria for completion.
The second improvement was to connect those phases with explicit artefacts. A methodology review produced an implementation checklist. Code review produced an evaluation watchlist describing suspicious patterns to look for. Analysis ended with a structured verdict rather than an unqualified narrative. Checks for self-inclusion, temporal validity, coverage symmetry and serving-time feasibility became part of the feature workflow. Tracking the number of related comparisons became part of evaluating a newly selected winner.
A third improvement was to add more integration points where the workflow could define and evaluate additional checks: for example, comparing the distributions of new features at training and serving time. These kinds of integration meant that our analysis agent could quantitatively verify that the implementations were as expected, rather than assuming everything had gone to plan and then glossing over anything unexplained with convincing prose.
Finally we improved the quality of our agentic self review steps. We did this by adding in new personas targeted at our observed failure modes: for example a specialist in experimental design to catch feature building or experiment design flaws; and by ensuring that the review was based on verifiable facts, data and statistics rather than a review of a written document. We were optimistic about the impact of this step, as a blind re-review of our historical experiments correctly spotted both the flawed experiments we had already identified and a few more.
From Offline Candidate to Online Test
We’ve simplified things a bit here, but after several iterations of this program we were reasonably confident in its quality. The clearest result to survive the tightened process was a model algorithm change: replacing our incumbent approach with CatBoost using its YetiRank objective. This had produced a small but credible offline improvement, which survived all of our cross checks and repeated consistently over many experiments. We were confident enough to take this model to the real test: online with real users. Thankfully our methods made this simple: the agents had already implemented the feature to a high quality, so the process of deploying this to production was trivial. As for the final online results, well they were modest, but we did observe a small effect in some of our metrics. That is consistent with the realm of marginal improvement we believed ourselves to be in. Crucially, this small gain is not something the team would have had time to explore themselves – and we rejected dozens of other ideas along the way.
The whole process felt a little like having five interns. They are smart and motivated, but lack some of the depth of experience needed to apply this in practice. A great deal could happen in parallel, but someone experienced still needed to decide what work was useful and review what came back. You become the supervisor, and supervision becomes the new constraint.
If You Want to Try This
If you wanted to try something similar our experience can be condensed into a few practical principles:
- Write durable context. Treat research logs and handover artefacts as part of the system.
- Decompose long workflows. Give each phase the context and definition of done it actually needs. Focus.
- Automate the volume work. Save human attention for methodology, judgement and consequential decisions.
- Prefer contained, verifiable experiments. Complexity is safer when failure is easy to detect.
- Make important conventions executable. A check that runs is stronger than another agent rereading plausible prose.
- Scale scrutiny with throughput. More experiments need stronger controls over leakage, comparability and result selection.
Autoresearch asks what happens when an agent can participate in the entire experimental loop. Semi-autoresearch gave us more experimental capacity without removing the scientist from the loop. It moved their attention from producing configurations, reports and documentation to deciding whether an experiment deserved to run and whether its result deserved to be believed. The bottleneck moved from execution to methodology: which is where we should have been spending our attention anyway.