Case studies
RefactorPlatform: an open-source coding agent evaluation harness for refactoring tasks
The challenge
In 2025, we partnered with Tipico Group, Germany’s leading provider of sports betting and online gaming, during Pi School of AI session 17. Our task was to build AI agents that could help clean up and restructure their codebase of over 2 million lines. This experience revealed to our team a gap in the market: the lack of a reliable coding agent evaluation harness for refactoring tasks. Why is this important? Because code restructuring is one of the most expensive and error-prone tasks an engineering team faces, and although AI coding agents promise to change that, most companies have no way to know whether their agents are performing optimally or even up to the job. The lack of a controlled, reproducible approach to measure agent accuracy causes poor design choices to go undetected and can silently erode output quality.
Our solution
We created RefactorPlatform with this gap in mind: an open-source evaluation harness that provides companies with a controlled environment to assess the performance of their AI coding agents. It allows teams to test one variable at a time, such as which AI model to use, whether the agent can search the codebase for relevant code, whether it can split work across helper agents, and how detailed its instructions are. Every test run is logged in detail, and results are fully exportable, making evaluations easy to review, repeat, and audit. RefactorPlatform is open-source, meaning any team can use it to get objective, reproducible answers about how their AI agents are performing.
The results
We tested the framework across 100 real-world tasks involving complex, multi-file code restructuring tasks. Here’s what we found:
- How you feed information to the AI matters as much as which AI you use. Giving the model structured, meaningful chunks of code rather than arbitrary, fixed-size chunks improved accuracy by 25–30 percentage points. Companies deploying AI coding agents without this consideration are leaving significant performance gains on the table.
- More complexity doesn’t mean better results. A single agent that can search the code effectively (86% accuracy) outperformed a setup that splits the work across helper agents (66%). Before investing in a more elaborate architecture, it’s worth knowing whether the added complexity is actually delivering value.
- You don’t have to trade cost for accuracy. Smarter information retrieval improved accuracy without increasing the cost per successful task across all four AI models we tested, meaning the business case is just as strong as the technical one.
Find us at EMNLP
Sébastien Bratières, our Director of AI, and Vijayasri Iyer, Senior Deep Learning Scientist, will present RefactorPlatform at the 31st Empirical Methods in Natural Language Processing EMNLP 2026 conference in Budapest on the System Demonstration Track. We are truly excited about this opportunity, as EMNLP is one of the leading conferences in natural language processing, and acceptance was especially selective this year.
Meet the RefactorPlatform team
- Sébastien Bratières, our Director of AI, with 16+ years of experience in language processing and a PhD in probabilistic machine learning.
- Vijayasri Iyer, Senior Deep Learning Scientist and alumna of the School of AI currently working on EVE.
- Mann Acharya, AI Researcher and Engineer, currently working with Pi School on applied AI projects, most recently diffusion models and agentic AI.
- Drish Mali, AI Researcher and Consultant who worked with Pi School on applied AI projects, focusing on agentic RAG architectures and repository-scale multi-agent workflows.
- Aziz Ben Amor, AI Researcher and Software Engineer, currently working with Pi School on applied AI projects.
Under the leadership of Sébastien Bratières and Vijayasri Iyer, our team participated in the School of AI 2025 and co-authored the RefactorPlatform coding agent evaluation harness.
Explore the project:
- Full project page: https://pischool.github.io/refactor-platform/
- Paper: https://arxiv.org/abs/2609.04898