A real-world telecom case study offers a more useful test of autonomous machine-learning research than another closed benchmark: can coding agents search an open-ended engineering space, run experiments and discover a competitive system without a researcher choosing every step?
Researchers from Georgetown University and Nokia Bell Labs tried exactly that on telecom ticket retrieval, using Cursor Composer 2.5, Claude Sonnet 5 and a locally hosted GPT-OSS 120B model inside autonomous research loops. Their best agent-designed retriever reached Recall@1 of 0.343, about 90% of an unpublished human-built internal system at 0.380. The autonomous campaign took 10 weeks end to end and one GPU-week of compute; the authors compare that with roughly 10 months spent developing the human systems.
The result is promising, but the paper is more valuable for what did not automate cleanly. Agents repeatedly needed a carefully engineered harness, a deterministic outer loop and human intervention when orchestration failed. They were good at deep parameter search inside a defined space, but poor at inventing the higher-level techniques that gave the human system its edge.
The agents got close, but did not rediscover the human recipe
The study uses a real internal telecom corpus with about 250,000 incident tickets, 204,000 fault analyses and 89,000 technical analyses. Evaluation used 7,600 held-out queries and 1,200 held-out gold resolution documents competing against the full corpus.
The strongest human internal system achieved Recall@1 of 0.380. It combined techniques including LLM-assisted data augmentation and re-ranking. The best autonomous result, generated by a single Cursor loop after 17 unique experiments, reached 0.3433. A single uninformed Claude campaign reached 0.3378, while an uninformed local GPT-OSS campaign reached 0.3353.
Those numbers matter because they weaken a common assumption: a stronger underlying model or a more elaborate multi-agent structure necessarily produces better autonomous research. In this experiment, the top results clustered closely despite very different agent choices. Multi-agent campaigns did not outperform the strongest single-agent loops.
The authors say the agents tended to discover conventional optimization moves—different pair sampling, sequence lengths, training epochs and other hyperparameters—rather than the more creative structural ideas used by human researchers. They did not independently converge on the internal system's re-ranking, data augmentation and model-ensembling strategy.
The hidden product was the harness
The study's most transferable contribution may be operational rather than algorithmic.
Simply telling an agent to keep experimenting was unreliable. Python heartbeat controllers also stalled or produced redundant work. The researchers eventually moved continuity outside the agent: a deterministic shell loop repeatedly resumed the same session and instructed it to continue improving Recall@1.
The harness also documented the dataset, task, available hardware and optimization target. Even then, open-weight models struggled with sub-agent orchestration. The paper reports GPT-OSS 120B successfully spawning sub-agents only once across dozens of attempts despite explicit instructions and an extension; Qwen Coder 30B attempts also failed to orchestrate reliably. Commercial agents handled sub-agent spawning more consistently.
This changes the interpretation of “autonomous research.” The autonomy did not come from releasing a model into an unconstrained laboratory. It came from surrounding an agent with a narrow objective, instrumented experiments, version control, a predefined search space and a deterministic process supervisor.
That is closer to an automated optimization service than a replacement research scientist—and that distinction is useful.
The economics are interesting, but not apples-to-apples
The authors estimate a 10–20 experiment campaign with Cursor Composer 2.5 at roughly $150–$200 in API charges. Equivalent Claude Sonnet 5 campaigns cost about two to three times as much, while the open-weight GPT-OSS runs avoided API fees by using local infrastructure.
But those numbers exclude repository setup and harness-development costs. They also do not price researcher supervision, failed runs, GPU hardware, engineering of the evaluation pipeline or the prior human work required to define the problem and build the comparison systems.
Likewise, “10 weeks versus 10 months” is not a controlled productivity experiment. The autonomous effort was conducted after the problem, data and human baselines existed, and it optimized against a known metric. The human system also achieved the higher score and included ideas the agents failed to invent.
The defensible conclusion is narrower: once the problem and measurement loop already exist, autonomous agents can compress a substantial amount of iterative experimentation at relatively low marginal API cost.
Model choice mattered less than search-space design
One of the paper's sharpest results is that agent brand did not determine the ceiling.
The researchers expected Claude to outperform Cursor and Cursor to outperform GPT-OSS, and expected multi-agent structures to beat single-agent loops. The campaigns did not support those expectations. Several single-agent runs converged in a narrow Recall@1 band around 0.30–0.34, while some multi-agent campaigns performed worse despite running many more experiments.
What mattered more was how the search was framed. Without explicit guidance, agents defaulted to familiar hyperparameter tuning. When researchers exposed variables for representation, data generation and architecture, exploration broadened—but largely by toggling options that humans had already encoded.
For teams building autonomous R&D systems, that suggests a practical design principle: invest first in the experiment contract. Define what can change, what must remain invariant, how results are validated, how failures roll back, and how the process resumes after interruption. A more capable model cannot compensate for an underspecified research environment.
Autonomy currently looks strongest as directed search
This paper does not demonstrate general scientific autonomy. It reports one industrial retrieval problem, one organization’s internal dataset, an unpublished internal state-of-the-art baseline and a relatively small number of campaigns. The data cannot be independently inspected, and the work is an arXiv preprint rather than independently verified peer-reviewed evidence.
Still, the case study is valuable precisely because it shows both capability and friction in the same system. Agents can run enough disciplined experiments to approach a strong human baseline. They can do so with modest marginal model costs. But they still need humans to design the environment, stabilize execution and supply the kind of structural intuition that moves beyond local search.
The more realistic near-term model for autonomous research is therefore not “AI scientist replaces the lab.” It is a governed research loop in which humans define the objective and boundaries, agents explore aggressively inside them, and humans intervene when the search needs a new idea rather than another trial.
That is less dramatic than full autonomy. It is also much closer to something engineering teams can deploy today.