AI chatbots deliver minimal productivity gains, study finds

A new study by economists analyzing a wide range of jobs and employees found that AI chatbots do not markedly improve an organization’s productivity or efficiency.

At the same time, AI is being adopted faster than any technology in history, according to industry analysts, economists, and technology historians.

Generative AI (genAI) has seen the fastest uptake. OpenAI’s ChatGPT, for example, reached 100 million users just two months after launch in November 2022 — faster than TikTok, Instagram, or any other app in history.

Yet, the actual gains from genAI’s adoption can be difficult to calculate across the broader labor market, according to the study, which was released in May by the National Bureau of Economic Research. The researchers, who explored a broad swath of employees and tasks, found that despite heavy investment by organizations, AI chatbots have had modest impact on worker producivity, saving users just 2.8% of work hours on average. And those productivity gains rarely led to higher pay (only 3% to 7% improvement).

But not all was negative. The researchers tempered their findings somewhat by stating that AI chatbots save time for 64% to 90% of users, but effects on work quality and satisfaction vary. The chatbots have also created new tasks for 8.4% of workers, even among non-users.

The report noted that past randomized control trials of AI use have documented much greater productivity gains, often exceeding 15%. Yet “it remains unclear how these translate into earnings and hours, as high-quality microdata on such outcomes is rarely available,” the researchers wrote.

And those trials have typically focused on the types of jobs that are likely see the biggest productivity gains from AI tools, in environments where AI chatbot use is encouraged and fully supported by employers. Looking at a wider range of occupations in less optimized settings revealed more modest gains.

“These findings caution against directly extrapolating productivity gains from controlled experiments to the broader economy,” the report said.

The researchers used two large surveys from Denmark covering 25,000 workers in 7,000 workplaces across 11 occupations. The job titles included accountants, customer support specialists, financial advisors, HR professionals, software developers, IT support specialists, and marketing and legal professionals. The report noted that Danish workers are very similar to those in the US in terms of genAI adoption, job flexibility, and salary negotiations.

While the study found widespread use of AI, with employers promoting its adoption through in-house models and training, the technology had little economic impact on those workers.

“AI chatbots have had no significant impact on earnings or recorded hours in any occupation,” the report stated. “Modest productivity gains (average time savings of 3%), combined with weak wage pass-through, help explain these limited labor market effects.”

The impact of genAI on the labor market is still unclear due to three key reasons: firms may not be fully integrating AI tools, chatbot benefits vary by task and can turn negative, and there’s limited data linking productivity gains to pay or work hours, according to the report.

Employers are driving AI chatbot adoption through encouragement, in-house models (38%), and training (30% of workers). Those efforts nearly doubled usage from 47% to 83%, the researchers found. Importantly, AI chatbot benefits are 10% to 40% greater when employers promote their use, highlighting the value of firm-led support, the researchers found.

In search of the elusive ROI

In addition to the Denmark study, the results of a recent IBM survey of 2,000 CEOs found that only 25% of AI projects meet ROI expectations. It also showed that 64% of respondents invest in AI mainly to avoid falling behind competitors.

New IBM research shows that AI strategy is becoming more focused, with only 6% of organizations using it ad hoc — down from 19% last year.

Manish Goyal, vice president and senior partner for global AI and analytics at IBM Consulting, said there are three best practices that organizations sometimes forget when deploying AI. First, align AI use with key priorities. Second, build strong technical foundations. Third, manage change and develop skills to drive adoption and growth.

“Clients have been seeing the most ROI when applying AI to scaled horizontal processes like software development, customer service, marketing, operations, IT, etc., where there is typically a human-in-the-loop,” Goyal said in an email reply to Computerworld. “From an industry perspective, we see telco, retail, and banking clients using AI successfully in areas like customer service.”

Last year, Gartner Research said genAI was slipping into the “Trough of Disillusionment” due to a mismatch between high expectations and reality, enterprise challenges in maturing data engineering and AI governance, and the intangible ROI related to many genAI initiatives.

Gartner’s trough of disillusionment describes a time when interest in a hot new technology wanes as experiments and implementations fail to deliver on the initial hype.

“We do believe that much ‘conventional AI’ has started climbing the slope out of the trough. GenAI is still falling into the trough. ‘Agentic AI’ is higher — near the peak,” Whit Andrews, a Gartner distinguished vice president analyst, wrote in an email response to Computerworld.

Gartner’s own research showed that some forms of AI have had a marked impact on worker efficiency. For example, a recent Gartner survey revealed AI save those using it an “average of 5.4 hours a week.” That’s about 12% of the work week, according to Andrews, which means Gartner’s survey is within a margin of error of the Denmark study. The differences between the two surveys, Andrews said, could also be due to differing methodologies.

“We do see workers telling us that they are saving time with AI,” Andrews said.

Yet, Gartner’s research also found teams that implement traditional AI and generative genAI are not significantlymore likely to report high productivity gains than teams that implement other technologies, “such as robotic process automation (RPA) or blockchain.”“While time savings from AI adoption are significant (5.4 hours per week per person), more than two-thirds of that time is redeployed toward non-value-adding activities,” Gartner’s report stated.

As genAI in its many forms begins to lose favor in organizations, enterprise focus on ROI will likely spur the adoption of autonomous AI in the form of agents — something with a more solid potential for productivity and efficiency gains, according to Gartner.

AI agents hold far more potential than chatbots

More than 500 tech executives surveyed by Ernst & Young in its latest Technology Pulse Poll said AI agents will make up the majority of upcoming AI deployments. The survey revealed that 48% are already adopting or fully deploying AI agents, and half of those leaders say that more than 50% of AI deployments will be autonomous in their company in the next 24 months.

An AI agent is a software program that collects data and uses the data to perform self-determined tasks to meet predetermined goals. For example, an AI agent could act as a customer care representative and automatically ask the customer different questions, look up information in internal documents, and respond with a solution. Based on the customer responses, it determines if it can resolve the query itself or pass it on to a human.

The results of a newly released survey from content management provider Box revealed a clear division between organizations that have strategically embraced AI and those that remain in beginning stages of adoption.

Box’s survey of 1,300 IT leaders found that early adopters of AI agents are seeing major productivity gains as the tech becomes key to their workforce.

“Early AI [agent] adopters are seeing significant ROI, with leading-edge companies measuring 37% productivity improvements on average,” Box’s study stated.

AI agents range from simple to advanced. Some 87% of organizations are using agents, including 41% using them for fully autonomous tasks, according to Box. While advanced agents boost efficiency and innovation, most companies are still adjusting and training employees to bridge the AI skills gap.

“I think the real question when it comes to something as dimensionally powerful as AI is: Do you want to save time in doing what you already do, or do you want to do more of it, or do you want to do it better, or do you want to change the way your industry functions?” said Gartner’s Andrews. “How do you want to mix that up?”

For example, focusing on productivity gains alone can be short-sighted. “When companies isolate their focus to efficiency, they sentence themselves to a diminishing significance that fails to increase their significance to their customers,” Andrews said.