How to succeed (or fail) with AI-driven development

Artificial intelligence (AI) continues to permeate seemingly every aspect of business, including software development. AI-augmented development involves using generative AI to support various stages of the software development lifecycle, including design, testing, and deployment. Introducing AI-powered tools into the development process is intended to increase developer productivity by automating certain tasks. It can also enhance the quality of code and speed up the development lifecycle, so development teams can bring products to users more quickly.

AI-augmented development is on the rise, according to industry research. An May 2025 report by market intelligence and advisory firm QKS Group forecasts that the global AI-augmented software development market will expand at a compound annual growth rate of 33 percent through 2030.

“In an era where speed, innovation, and adaptability define competitive advantage, AI-augmented software development is rapidly becoming a transformative force for enterprises,” the report says. “By embedding AI into every stage of the software development lifecycle, from code generation and testing to debugging and deployment, organizations across industries like finance, healthcare, retail, telecom, and manufacturing are redefining how software is built, optimized, and scaled.”

Deploying AI-augmented development tools and processes comes with both risks and rewards. For tech leaders and software developers, it is vital to understand both.

Risks of AI-augmented software development

Risks of relying too heavily on AI for software development include bias in the data used to train models, cybersecurity threats, and unchecked errors in AI-generated code. We asked a range of experts what they’ve found most challenging about integrating AI in the software development lifecycle and how they’ve managed those challenges.

Bias in the models

Bias in the data used to feed models has long been an issue for AI, and AI-augmented development is no exception.

“Because AI is trained on human-coded data, it can replicate and amplify existing biases,” says Ja-Naé Duane, faculty and academic director of the Master’s Program in Innovation Management and Entrepreneurship at Brown University School of Engineering. “Without deliberate oversight and diverse perspectives in design and testing, we risk embedding exclusion into the systems we build,” she says.

Most Loved Workplace, a provider of workplace certifications, uses machine learning to analyze employee sentiment. But early on, it saw signs that its models were misreading certain emotional tones or cultural language differences.

“We had to retrain the models, labeling according to our own researched models, and using humans in the loop to test for bias,” says Louis Carter, founder of the company and an organizational psychologist.

“Our internal team did a lot of work to do so, and we created a gaming platform for everyone to label and add in their own interpretation of bias,” Carter says. “We improved the [BERT language model], developing our own construct for identifying emotions and sentiment. If we hadn’t caught it, the results would have misled users and hurt the product’s credibility.”

Intellectual property (IP) infringement

The use of AI-augmented development and possible IP infringement can raise complex legal issues, especially within the area of copyright. Because AI models can be trained using enormous datasets, including some copyrighted content, they can generate outputs that closely resemble or infringe upon existing copyrighted material. This can lead to lawsuits.

“The current uncertainty around how these models do or don’t infringe on intellectual property rights is absolutely still a risk,” says Joseph Mudrak, a software engineer at product design company Priority Designs. “OpenAI and Meta, for example, are both subjects of ongoing court cases regarding the sources of the data fed into those models.”

The American Bar Association notes that as the use of generative AI grows rapidly, “so have cases brought against generative AI tools for infringement of copyright and other intellectual property rights, which may establish notable legal precedents in this area.”

“Most generally available AI-augmented development systems are trained on large swaths of data, and it’s not particularly clear where that data comes from,” says Kirk Sigmon, a partner at law firm Banner & Witcoff Ltd. Sigmon specializes in AI and does coding and development work on the side. “Code is protectable by copyright, meaning that it is very possible that AI-augmented development systems could output copyright-infringing code,” Sigmon says.

Cybersecurity issues

AI-augmented development introduces potential cybersecurity risks such as insecure code generation. If they are trained on datasets with flawed or insecure examples, AI models can generate code containing common vulnerabilities such as SQL injection or cross-site scripting attacks.

AI-generated code could also inadvertently include sensitive data such as customer information or user passwords, exposing it to potential attackers. Training models on sensitive data might lead to unintentional exposure of this data in the generated code.

“From a privacy and cybersecurity standpoint, unvalidated AI-generated code can introduce serious vulnerabilities into the software supply chain,” says Maryam Meseha, founding partner and co-chair of privacy and data protection at law firm Pierson Ferdinand LLP.

“We’ve seen companies unknowingly ship features that carried embedded security flaws, simply because the code ‘looked right’ or passed surface-level tests,” Meseha says. “The cost of retroactively fixing these issues, or worse, dealing with a data breach, far outweighs the initial speed gains.”

False confidence

There might be a tendency for development teams and leaders to assume that AI will get it right almost all the time because they believe automation removes the problem of human error. This false confidence can lead to problems.

“AI-augmented approaches, particularly those using generative AI, are inherently prone to mistakes,” says Ipek Ozkaya, technical director of engineering intelligent software systems at the Carnegie Mellon University Software Engineering Institute.

“If AI-augmented software development workflows are not designed to prevent, recognize, correct, and account for these mistakes, they are likely to become nightmares down the line, amounting to unmanageable technical debt,” Ozkaya says.

Most Loved Workplace, which uses tools such as Claude Code, Sentry, and custom AI models for emotion and sentiment analysis in its platform, has experienced false confidence with AI-augmented development.

“Claude and other tools sound right even when they’re dead wrong,” Carter says. “One piece of output missed a major edge case in a logic loop. It passed initial testing but broke once real users hit it. Now, everything AI touches goes through multiple human checks.”

The company has had developers submit code from Claude that looked solid at first but failed under load, Carter says. “When I asked why they made certain choices, they couldn’t explain it—it came straight from the tool,” he says. “Since then, we’ve made it clear: If you can’t explain it, don’t ship it.”

Rewards of AI-augmented software development

While increased productivity and cost-effectiveness garner the most attention from business leaders, tech leaders and developers are finding that AI supports developer learning and skills development, prevents burnout, and may make software development more sustainable as a career.

Speed without burnout

It’s no surprise, given the pressure to deliver quality software at a rapid pace, that many developers experience burnout. A 2024 study by Kickstand Research, based on a survey of more than 600 full-time professionals in software engineering, found that nearly two-thirds of respondents (65 percent) experienced burnout in the past year.

The report, conducted on behalf of Jellyfish, a provider of an engineering management platform, indicated that the problem was particularly acute for short-staffed engineers and leaders overseeing large organizations. Of respondents at companies with more than 500 people in their engineering organization, 85 percent of managers and 92 percent of executives said they were experiencing burnout.

Deploying AI-augmented development tools can help address the issue by automating tasks and increasing productivity.

“Claude Code has helped us move faster without overwhelming the team,” Carter says. “One of our junior developers hit a wall building a complex rules engine. He used Claude to map out the logic and get unstuck. What would’ve taken half a day took about an hour. It saved time and boosted his confidence.”

Cleaner code and fewer bugs

AI-augmented development can lead to fewer bugs and improved code quality. This is because AI tools can handle tasks such as code analysis, bug detection, and automated testing. They can help identify possible errors and suggest enhancements.

“We use Sentry to catch issues early, and Claude to clean up and comment the code before anything ships,” Carter says. “Claude is a great way of cleaning up messy code.”

Commenting, or adding notes and reasoning behind what code is doing and what it is intended to accomplish, makes it easy for everyone to understand, Carter says. This is especially helpful for programmers whose second language is English, “because there are a lot of misunderstandings that can happen.”

Most Loved Workplace is running sentiment and emotion scoring in its human resources SaaS application Workplacely, used for certifying companies. “AI helps us test edge cases faster and flag inconsistencies in model outputs before they go live,” Carter says.

“My favorite way to use AI-augmented development systems is to use them to help me bugfix,” Sigmon says. “AI systems have already saved me a few times when, late at night, I struggled to find some small typo in code, or struggled to figure out some complex interrelationship between different signaling systems.”

Cost-effectiveness and increased productivity

AI-augmented development systems can be cost-effective, particularly over time due to increased efficiency and productivity, the automation of tasks, reduced errors, and shorter development lifecycles.

“Using AI-augmented development systems can save money because you can hire fewer developers,” Sigmon says. “That said, it comes with some caveats. For instance, if the world pivots to only hiring senior developers and relies on AI for ‘easy’ work, then we’ll never have the opportunity to train junior developers to become those senior developers in the future.”

AI “can automate routine coding tasks and surface bugs, as well as optimize performance, dramatically reducing development time and cost,” Duane says.

“For example, tools like GitHub Copilot have been shown to significantly cut time-to-deploy by offering developers real-time code suggestions,” Duane says. “In several organizations I work with, teams have reported up to a 35 percent acceleration in release cycles, allowing them to move from planning to prototyping at unprecedented speed.”

Upskilling on the fly

The skills shortage is one of the biggest hurdles for organizations and their development operations. AI-powered tools can help developers learn new skills organically in the development process.

“I’ve seen junior team members start thinking like senior engineers much faster,” Carter says. “One in particular used to lean on me for direction constantly. Now, with Claude, he tests ideas, reviews structure, and comes to me with smarter questions. It’s changed how we work.”

AI is lowering the barrier to entry for individuals without formal programming training by enabling no-code and low-code platforms, Duane says. “This transformation aligns with our vision of inclusive innovation ecosystems,” she says.

For instance, platforms such as Bubble and Zapier enable entrepreneurs, educators, and others without technical backgrounds to build and automate without writing a single line of code, Duane says. “As a result, millions of new voices can now participate in shaping digital solutions, voices that would have previously been left out,” she says.

Further reading:

Why AI projects fail, and how developers can help them succeed

11 open source AI projects that developers will love

6 generative AI Python projects to run now