ENGINEERING
Autoresearch At Listen
How we applied the autoresearch pattern on four different levels of ambiguity, from objective metrics to preference-based judgments.
AI coding agents have become surprisingly capable at improving well-defined metrics by iterating on entire codebases. This enables a simple but effective pattern for improving products for your customers: Define a meaningful eval metric that captures product quality and have a coding agent improve it autonomously. Karpathy’s autoresearch went viral for optimizing a single-file LLM’s validation loss with this technique. We found that autoresearch yields impressive results on a wide range of tasks, including query time optimization, agent output formatting, agent tool use streamlining, report generation, and emotion extraction. In this article, we share our takeaways from autoresearch on our AI-enabled research platform.
What is needed for autoresearch?
A clear metric to improve is THE central requirement to set up an autoresearch loop from a researcher’s perspective. We need to provide a way for the coding agent to know whether a change was good or not. Iteration speed is crucial: Just like for human researchers, the faster you get feedback, the faster you can iterate to a great solution. A way to break down the optimization metric into its parts and to debug also helps. For example, when optimizing the speed of an API endpoint, a trace of which function takes how long makes optimization a lot more useful.
An agent orchestrator is the other part. We need to track versions with results, invoke the evaluation pipeline, feed the results to the coding agent, let the coding agent create a new version of the code, and repeat. This is mostly an infrastructure challenge.
So how do we apply this to Listen Labs?
Listen Labs is building a human-preference model that companies and AI agents query to predict what people think, want, and decide by running thousands of AI-moderated interviews. We successfully applied the autoresearch pattern on four different levels of ambiguity, from objective metrics to preference-based judgments.
Level 1: Objective metrics
Query timing is a classic example. The user experience on our web app drastically depends on how fast we manage to load data. This is a classic autoresearch problem: For data loading bottlenecks, have a coding agent optimize the average query time. Orchestration is remarkably simple: write a local script that times the query in question. Then, prompt a local Claude Code to iterate on the query and call the timing script until the latency drops below a certain threshold.
This approach tended to work out of the box. While the results were pretty impressive (speedups of >2x), some surprising tradeoffs arose:
Tradeoff | What it means |
|---|---|
Dev vs prod timing | The timing script has to target the prod database to yield meaningful results. Ideally, we would time each iteration from our prod environment. Unfortunately, this is significantly more work to set up and reduces iteration time. We therefore resort to timing the query speed from the local machine to the prod database. While the local network connection is significantly slower, the locally-optimized queries tended to mirror the performance benefits on prod. However, to squeeze the last percentage points of performance out of the optimization, the timing script would have to run on a realistic prod environment |
Speedup vs maintainability | While a coding agent in a loop proved to consistently find significant performance boosts, this sometimes comes at the cost of more complex code. To keep our codebase maintainable, we cannot blindly accept the fastest version, but sometimes have to push back on certain optimizations or have a manual cleanup pass. |
Speedup vs manual setup effort | We observed the speedup to be highest when we gave the most context on timing, e.g. exact traces of parallel queries. However, feeding this to the agent requires more manual setup, partly defeating the promise of automatic improvements without any human input required. |
Level 2: Tasks with a Ground Truth
Emotion detection from video is critical for our Emotional Intelligence feature. It is a pipeline that takes a video and identifies the emotions the person in that video feel. We use a benchmark based on a human-labelled dataset, choosing the F-beta score as the core metric to boost. Just like for latency-optimization, our coding agent is quick to find significant improvements. However, whether the changes are positive is not as clear as for the previous case:
Point | What it means |
|---|---|
Relevance of the metric | It is unclear how well the F-beta score represents the perceived quality of emotion extraction. Leaving all things equal, a higher F-beta is better. However, what if the model finds a way to boost the score slightly but shifts failure cases from one category to another? In these cases, human judgement is required to weigh the gains and losses. It is a good indicator but not necessarily a clean sweep. |
Reward hacks | You may think (or hope?) that coding agents have judgement in their changes. Think again. Our emotion detection pipeline predicts both which emotions are present and when they are present. However, the benchmark only tests classification prediction, not timestamp prediction. The coding agent saw an opportunity to boost classification performance: Simply remove the timestamp prediction, and the F-beta score will jump by a few percentage points. Did it know that timestamp prediction was needed in the codebase? Yes! Did that stop it? No! Its task was only to optimize the F-beta score, and nothing else. Takeaway: Have a good look at the side effects of your “optimization” before merging! Or maybe another agent can do it… |
Level 3: Tasks with Explicit Evaluation Criteria
Research Agent allows our clients to analyze their study data through simple prompts. It can search responses, quantify results, create charts, presentations, and video clips, and all claims link back to the source. So how do we measure the quality of agent responses? The open-ended nature of agent responses makes this non-obvious; there is no clear ground truth for how an agent should respond.
However, it is easy to express criteria that the agent response should fulfill. For example, we want agent responses to answer user’s requests. We want the response to be concise. We want it to reference a source for each of its claims. And, most importantly, we want the response to be accurate. What that means differs on a case-by-case basis.
Evaluating the criteria above requires judgement. We employ the LLM-as-a-judge pattern: Give a chat trace, combined with extra context on the study and referenced data to an LLM ask it to output a judgement (pass/fail) and a justification for each criterion. An eval set, consisting of a large set of (prompt, criteria) pairs, measures how well the agent follows design criteria. Find two exemplary eval cases in the table below.
Input prompt to research agent | Evaluation criteria |
|---|---|
“Create a short video clip of a female respondent who is not happy with a feature of the product” | • Must respond with video clip |
“Visualize how the problems people have with our product differ between genders” | • Must respond with chart |
As the main criterion, we ask a coding agent to boost the pass rate. We orchestrate the autoresearch loop with Claude Code in-a-loop running in a cloud environment. We set up an eval script that Claude can run whenever it wants to try a new version. The script persists eval results, including judgements and traces, so that the coding agent can investigate failed test set cases and make targeted edits.
Autoresearch eliminated several classes of undesired behavior almost entirely. For example, research agent should never respond with raw numbers when quantifying results, but instead use references that we can update when new data comes in. Before autoresearch, research agent often refused to use our reference syntax, undermining trust in the integrity of our data. The autoresearch loop found that drastically cut the system prompt and clarifying the syntax pushes this failure mode close to zero.
However, with less clarity and bigger scale, more things can go wrong:
Problem | What it means |
|---|---|
Orchestration problems | A single research agent turn can take a minute and hundreds of megabytes of network traffic, and turns nest, forcing sequential execution. Across hundreds of eval cases, with rate limits on top, one run takes about two hours and 32GB of RAM to parallelize properly. Not something you run locally, and cloud environments choke on it too. Iteration slows, and the approach gets less useful. We never found a good fix. Ideally your metric is fast and cheap. |
Isolation | Ideally, your coding agent cannot write to your production database. We resorted to a local copy of the database, containing all data required for the eval cases. While safer, this adds significant infrastructure overhead. |
LLM judges can be wrong | After a few iterations, we found that a large fraction of the failure cases are false positives. The only reliable way around this we found is to tune the judge manually. Or who knows, maybe the next big thing will be a judge for LLM judges? In any case, tuning the judge leads to the next problem. |
Changes to the judge break comparability | Let’s assume we fix a bug in the judge that eliminates some false positive cases. Your eval scores go up. However, the comparison with all prior eval runs are no longer fair now. This has two downsides: First, we do not know anymore whether a feature branch is better than main. This can be solved relatively easily be re-evaluating main with the new judge. Second, when tracking response quality over weeks or months, the trend is no longer as meaningful. The clean fix is to re-judge all prior evals. Unfortunately, this is computationally expensive and potentially infeasible if newer judge versions are no longer compatible with old agent versions. |
Features not covered in evals degrade | Research Agent has a wide range of features, each of which requires its specific instructions to work. For instance, it can create many different types of charts. However, if one of the chart types is not covered in the evals, the coding agent will be default see cutting its prompting as a chance to streamline the system prompt. We found two possible solutions to this problem:
|
Scope of edits | Unless explicitly prompted against, the coding agent may suggest changing the evals to boost performance. |
Convenient benefits: A large part of the criteria are not specific to a specific prompt. This allows us to track response quality over time by evaluating the default criteria against each of the live research agent requests. So why don’t we simply give the judge feedback to research agent and have it iterate on its response up to perfection? The reason is latency. We want to stream the response to the user, and an extra round trip to a judge would make research agent unbearably slow. However, in async tasks, we have no such constraints, largely solving issues that can be described with a clear criterion. As a result, we can lift the bar on quality expectations, requiring us to push on the ultimate level: judgement.
Level 4: Tasks that require judgement
Imagine you want to find the most interesting insights from a large set of interviews and condense them into a report. How would you judge the quality of these reports? You may care about criteria such as the coherence of the story, the quality of the insights, or the clarity of the language. A judgement does not clearly map to pass or fail; instead, the performance is on a continuous scale. Unfortunately, we found LLM judges to be remarkably bad at judging on a Likert scale. Try asking your favorite assistant how they like your outfit, and no matter what you wear, you are very likely to get an 8 out of 10. Asking to find the best out of four options gives clear signal.
So how can we get around this? While an absolute rating is most useful, a relative ranking between two versions of the codebase can already help when deciding whether a change is good or not. In other words: Try a change on a feature branch, compare it to the baseline, and accept if better. Having an agent iterate on this works surprisingly well. However, there are some caveats:
Complex orchestration: Comparing two versions of the codebase head-to-head requires operating with both of them at the same time. Instead of simply computing two scores, we need to create a worktree for each version of the code, have an LLM judge compare the results, and iterate.
Cyclic preferences: There is no reason to believe that LLM judges cannot like report A better than B, B better than C, and C better than A. In theory, this could lead to a cycle in the autoresearch loop.
No quantifications, no tracking over time: Relative comparisons yield only one piece of information: Is version A or B better. There is no direct way of comparing version A and C, even if B and C have already been compared. We do not have an absolute quality score to track over time.
Order matters: LLMs are not invariant to changing the order of their inputs. We circumvent this by running the judge with both possible orders and calling it a tie if e.g. the first report always wins.
Our takeaways
Iterating on a codebase to improve a well-defined metric is mostly a solved problem. Autoresearch proved itself as an effective way to improve a diverse range of product features at Listen Labs. The more straight-forward the metric to improve, the simpler the setup. However, humans are still needed for three key parts of the process:
Defining a meaningful metric to optimize. Depending on the level of ambiguity, this may be as easy as measuring a query time or as hard as building an eval system. The quality and coverage of the metric is integral for the quality of the outcome. Only what is covered in the metric will be optimized.
Providing the environment a coding agent easily iterate in. Sometimes, a local dev server and Claude Code are enough, sometimes, a dedicated cloud environment with an elaborate data-copy and eval scripts are required. In all cases, iteration speed crucially depends on how well the agent can inspect failure cases.
Judging whether the changes are net-positive. It is easy to say that the eval metric should simply cover every part of the product. In practice, this is rarely feasible. Real-world products like the Listen platform have so many implicit tradeoffs that covering all of them in a single metric seems implausible. When asked to optimize a metric, agents go from “helpful-assistant”-mode to “radical-optimizer”-mode. Only one metric matters, and anything not needed or strictly prompted to keep will be considered for sacrifice if it improves the metric. Given that coding agents became remarkably reliable for most tasks, you may be caught off-guard by the changes a coding agent in optimizer mode may suggest you confidently.
Written by Ole Petersen