Angry Developers: Clumsy AI Threatens Open-Source Code

An In-Depth Study on the Impact of "AI Slop"

A recent qualitative study has highlighted the growing frustration among developers regarding the poor quality of code generated by artificial intelligence, often referred to as "AI slop." This situation is described as a "tragedy of the commons," where individual benefits gained through AI come at the expense of the open-source community. Researchers from the universities of Heidelberg, Melbourne, and Singapore Management University analyzed 1,154 publications from 15 discussion threads on Reddit and Hacker News. The study focused on discussions containing the term "AI slop," which skewed the results towards a negative perception of AI. Positive or neutral experiences were not taken into account, thus limiting the scope of the conclusions to a critical subgroup.

From this data, the researchers developed a codebook comprising 15 categories divided into three thematic clusters: Review Friction, Quality Degradation, and Forces and Consequences. These categories help structure developers' critiques and better understand the issues related to the use of AI in software development.

The Consequences of Increased Individual Productivity

The study emphasizes that "AI slop" is perceived as a tragedy of the commons. Developers and companies leverage AI to boost their productivity, but this comes at the expense of reviewers and the community. Codebases accumulate technical debt, knowledge resources degrade, and trust in collaborative development erodes. The problem is particularly acute in the open-source domain, where resources are maintained by volunteers. Concrete examples include the curl project, which had to shut down its bug bounty program due to AI-generated vulnerability reports, and similar issues have been observed with Apache Log4j 2 and the Godot game engine.

Developers have also reported having AI workflows imposed by management. In some cases, senior executives have directly integrated AI results to solve technical problems without thorough consultation with the team, exacerbating tensions and frustrations.

The Burden on Code Reviewers

Developers also criticize the increased burden on those who must review the code generated by AI. One developer noted that while development time has been reduced, the team must spend more time on reviews, which is not seen as a benefit. One team reported receiving 30 pull requests per day with only six reviewers available. Reviewers feel transformed into unpaid "prompt engineers," tasked with understanding and evaluating AI's sloppy code. They have developed heuristics to identify AI-generated code, such as the presence of emojis in comments or step-by-step comment patterns.

Trust in collaborative processes is also undermined. One developer described a pull request from an AI agent, expressing skepticism about the reliability of the code: "I don't know how you could trust anything at any given moment. No real understanding of what it's doing, it's just guessing."

Strategies to Counter Negative Effects

In light of these challenges, the study documents countermeasures implemented by developers:

• Limit the size of pull requests containing AI-generated code to fewer than 500 lines of code.
• Impose self-reviews before peer reviews.
• Implement synchronous code walkthroughs and double reviews with external teams.
• Tie developers' accountability to performance evaluations.

The researchers recommend that tool developers focus more on code verification and review rather than generation. Team leaders should reassess productivity evaluation criteria to include review costs. Finally, educational institutions should limit the use of AI tools in early courses to foster the development of fundamental skills among students. Producing results alone is not proof of competence, they assert.