The reality of AI-centric coding

When Amazon Web Services CEO Matt Garman told his developers in June that they may no longer be in the coding business in as little as two years, it was a heads up to change-resistant coders that they had better embrace the new AI-centric reality.

But it also forced serious conversations among executives about what generative artificial intelligence (GenAI) coding environments would realistically look like. 

Garman’s recorded comments, which have since been confirmed by Amazon, were first reported by Business Insider. 

“If you go forward 24 months from now, or some amount of time — I can’t exactly predict where it is — it’s possible that most developers are not coding,” Garman was quoted as saying. “It just means that each of us has to get more in tune with what our customers need and what the actual end thing is that we’re going to try to go build, because that’s going to be more and more of what the work is, as opposed to sitting down and actually writing code.”

A different world

But managing code generated by human programmers will be vastly different than managing GenAI-generated code. Dev Nag, CEO of SaaS firm QueryPal, said that he has been working with GenAI coding efforts and that many enterprise IT executives are not prepared for how different it is.

“It made tons of weird mistakes, like an alien from another planet. The code misbehaves in a way that human developers don’t do,” Nag said. “It’s like an alien intelligence that does not think like we do, and it goes in weird directions.”

Of even greater concern, Nag said, is that GenAI coding mechanisms will follow the rules to a certain degree, but it will also get creative in finding ways to circumvent those rules. “AI will find a pathological way to game the system.”

Humans are still essential

Another programming specialist, Veny Moola, CEO of application development firm Fleet Studio, said that GenAI hallucinations are major problems.

“Hallucination-laden software shouldn’t be given direct control over executables, and that’s precisely why the role of software engineers remains critical. When AI systems, prone to hallucinations, are involved in generating code or making decisions, it is essential for human engineers to validate, test, and monitor these outputs before they are allowed to interact with or control executable processes,” Moola said.

“Engineers are responsible for implementing rigorous checks and balances, such as automated testing, code reviews, and safety protocols, to catch and correct potential errors introduced by AI. In high-stakes environments where software has direct control over critical systems, human oversight is crucial to ensure that the AI’s outputs are reliable and safe. Without these safeguards, the risk of deploying AI-generated software without proper vetting could lead to catastrophic failures, which is why software engineers must remain actively involved in the development and deployment process.”

GenAI-centric programming also lacks the kind of shared coding context that is in the heads of just about all human developers. “It needs far more overhead because it is missing the shared organizational context. We are not used to having to mention everything,” said Nag, adding that he estimates that context-related issues “will be magnified 100 times with GenAI coding.”

New testing requirements

A critical part of coding oversight will involve testing, but Nag stressed that enterprises will have to create entirely new approaches to testing. Today, most enterprise programming teams are trusted to create homegrown apps. The apps will certainly be tested for functionality, but many today forego a rigorous line-by-line examination looking for backdoors or other problematic elements. 

In a GenAI coding environment, “testing processes will have to change vastly, changing from simply functionality testing to far, far deeper automated testing of everything, including having every app subjected to automated penetration testing,” Nag said.

Generative AI limitations

Another change that enterprise development managers will have to accept is that moving from small coding projects where AI is acting as a programming assistant to giving GenAI far more coding authority is a massive leap.

“Although AI is able to solve many college problem sets and handle small-to-medium snippets of code generation, it still struggles with complex logic, large code bases, and especially novel problems without precedent in the training data. Hallucinations and errors remain significant issues that require expert engineering oversight and correction,” Nag said. “These tools are far better at quick prototypes from scratch rather than iterating large applications, which is the bulk of engineering. Much of the context that drives large applications doesn’t actually exist in the code base at all.”

Tom Taulli, who has authored multiple AI programming books, including this year’s AI-Assisted Programming: Better Planning, Coding, Testing, and Deployment, agreed that the move to great GenAI coding efforts will catch most enterprises off guard.

“These tools will mean a change in traditional workflows, approaches, and mindset. Consider that they are pretrained, so they are often not updated for the latest frameworks and libraries. Another issue is the context window. Code bases can be massive. But even the most sophisticated LLMs cannot handle the huge amount of code files in the prompts,” Taulli said. “Many of these tools are mostly trained on public code repositories. But there is lots of legacy code that is incredibly important and quite sparse for the training datasets, like COBOL.”

In his own GenAI coding efforts, Taulli said he has seen a lot of unorthodox coding behaviors. He also pointed to published reports of many different GenAI-coding challenges. 

“For example, you can ask these LLMs to create code and they sometimes make up a framework, or an imaginary library or module, to do what you want it to do,” Taulli said, explaining that the LLMs were not creating a new framework as much as pretending that they created it, but they actually didn’t.

That is the sort of behavior that LLM coding will do that almost no human programmer would even consider doing, Taulli noted, adding, “unless (the human coder) is insane, they are not going to make up, create out of thin air, an imaginary library or module.”

When that happens, it can be easy to detect, if someone looks for it. “If I try to pip install it, you can find that there’s nothing there. If it hallucinates, the IDE and compiler give you an error,” Taulli said.

The coder’s new role

Nag said that he thinks Garman’s comments are correct in that enterprise programming efforts are going to need to make the AI move, even if it causes a lot of painful changes along the way. Eventually, Nag said, enterprise development efforts will be better for the change.

“[The coder’s] role is as much about understanding the larger context and business goals as it is about crafting a particular syntax or debugging an error code. The rise of digital photography didn’t destroy the photography profession. It democratized it and created explosive demand for new specialties like mobile filters and retouching,” Nag said. “The rise of spreadsheets didn’t make accountants obsolete. It gave them the leverage to focus on financial analysis, strategy, and more complex corner cases. AI in software engineering will likewise democratize application development while creating new roles around prompt engineering, AI-assisted development, and expanding the scope of what’s possible with software.”

Security concerns

Another coding specialist, Ashley Rose, CEO at SaaS vendor Living Security, said there are also security concerns if more coding influence gets turned over to GenAI.

“Think about how our internet infrastructure has evolved. It was originally designed to be decentralized, but over time, key services have become concentrated around major providers like Cloudflare and AWS. This centralization has introduced new points of vulnerability, where an outage or attack on one provider can ripple out to impact large portions of the web,” Rose said.

“The same could happen with AI in coding. As more companies integrate AI models into their development processes, these models could become high-value targets for attackers. Compromised models could introduce hidden vulnerabilities that slip through standard security checks. Given the widespread use of these models, the impact could be devastating, potentially affecting thousands of applications across different organizations.”