What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests Anonymous Anonymous Institution

Citation

Dovhoshliubnyi l, Soroush N, Sami A & Brownlee A (2026) What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests Anonymous Anonymous Institution. In: Search-Based Software Engineering. 18th Symposium on Search-Based Software Engineering 2026 (SSBSE 2026) Challenge Track, Montreal, Canada, 05.07.2026-06.07.2026. https://doi.org/10.1007/978-3-032-30699-9_12

Abstract
AI coding agents are black boxes: we cannot inspect how they generate code, but we can inspect what they change. This distinction matters for search-based software engineering (SBSE), where techniques such as genetic improvement depend on mutation operators that reflect how code is actually transformed. Of the 33,596 agent PRs in the AIDev dataset, less than 400 target performance (fewer than 1%), making each successful case a valuable window into otherwise opaque agent behaviour. We classify 1,254 performance-relevant diff hunks from 216 of these PRs, spanning five agent systems, against the 18-category syntactic mutation taxonomy of Even-Mendoza et al. (2025) using an LLM-as-a-judge pipeline. Three categories dominate: name modification (36.9%), object creation (26.3%), and type change (22.6%), a profile strikingly different from prior genetic improvement corpora where no change accounted for 84%. Each agent commits to a distinctive mutation vocabulary, and each performance strategy activates a largely disjoint category subset. Agent identity and target strategy are therefore informative priors that narrow the effective SBSE operator space from 18 categories to a handful per context. Replication package: https://anonymous.4open.science/r/ssbse-challenge-2026-710C/

Keywords
mutation testing; AI agents; empirical study; search-based software engineering; performance optimization