“AI” automated PR reviews mostly useless junk

The team that makes Cockpit, the popular server dashboard software, decided to see if they could improve their PR review processes by adding “AI” into the mix. They decided to test both sourcey.ai and GitHub Copilot PR reviews, and their conclusions are damning.

About half of the AI reviews were noise, a quarter bikeshedding. The rest consisted of about 50% useful little hints and 50% outright wrong comments. Last week we reviewed all our experiences in the team and eventually decided to switch off sourcery.ai again. Instead, we will explicitly ask for Copilot reviews for PRs where the human deems it potentially useful.

This outcome reflects my personal experience with using GitHub Copilot in vim for about 1.5 years – it’s a poisoned gift. Most often it just figured out the correct sequence of ), ], and } to close, or automatically generating debug print statements – for that “typing helper” work it was actually quite nice. But for anything more nontrivial, I found it took me more time to validate the code and fix the numerous big and subtle errors than it saved me.
↫ Martin Pitt

“AI” companies and other proponents of “AI” keep telling us that these tools will save us time and makes things easier, but every time someone actually sits down and does the work of testing “AI” tools out in the field, the end results are almost always the same: they just don’t deliver the time savings and other advantages we’re being promised, and more often than not, they just create more work for people instead of less. Add in the financial costs of using and running these tools, as well as the energy they consume, and the conclusion is clear.

When the lack of effectiveness of “AI” tools our in the real world is brought up, proponents inevitably resort to “yes it sucks now, but just you wait on the next version!” Then that next version comes, people test it out in the field again, and it’s still useless, and those same proponents again resort to “yes it sucks now, but just you wait on the next version!”, like a broken record. We’re several years into the hype, and that mythical “next version” still isn’t here.

We’re several years into the “AI” hype, and I still have seen no evidence it’s not a dead end and a massive con.