Evaluating AI agents? Early adopters outline practical challenges

Non-tech companies trying out AI agents in the field said the technology still has a lot of  challenges to overcome before it can be used practically. 

Some of the early uses of agentic AI are in the area of customer service, which was part of a panel discussion at Nvidia’s GTC conference, held last week in San Jose.

The main problem is humans still prefer talking to humans because they don’t trust machines, said Cameron Davies, chief data officer at Yum Brands, which includes brands such as Pizza Hut, Taco Bell and KFC.

The company hopes to get to 100% digital ordering in five years, with agentic AI playing a role, Davies said.

Specifically, Yum Brands wants to automate orders at drive-throughs, Davies said. One of the most challenging service jobs is running drive-through windows — to take, fill and cash out orders — and AI agents could lighten that load. 

But the main issue isn’t technological, Davies said. “What’s the greatest challenge in putting this into place, is that nobody wants to talk to a machine right now. And you have to ask yourself why is that the case?” 

Yum Brands is also eyeing AI agents to reduce the “cognitive” load on human servers and employees. Scripted agents can do the role of “upselling and asking about charity donations,” Davies said.

“You do these things, then that person can now focus on being happier, making sure the order is right, getting the change right, etc.,” Davies said, adding that Yum Brands has been testing agents for back-office and HR functions, with mixed results.

Beyond trusting AI systems, compliance and accuracy are concerns for Craig Daniels, the head of Mayo Clinic’s Smart Hospital and Unbound Project.

Healthcare, by its nature, tends to lag other industries in adopting technology; Daniels is looking at the progress of AI agents at companies like Yum Brands to see what works.

For Mayo, the challenge is for doctors and patients to gain a high level of trust in AI to assist in diagnosis and treatment. Then they can consider the role of agents in helping doctors and patients.

Patients need to trust AI models just as they trust MRI machines without having to worry about the underlying technology, Daniels said.

Mayo Clinic is creating its own data platform with anonymous patient information gathered from 61 different healthcare organizations in four continents. For example, one Mayo AI system trained on 7 million electrocardiograms can detect and diagnose heart failure.

“There’s a point at which we have to trust the model works,” Daniels said. “It generates novel insights and we’ve researched and we trust that and we’re using it with the human to make the final decision. That’s a wonderful advancement.”

The US Food and Drug Administration (FDA) will ultimately regulate AI in healthcare through “medical-as-a-software device,” which will require proven research, Daniels said.

More than in other industries, Mayo will require a variety of guardrails be in place to deliver trustworthy results as there’s no allowance for AI or agents to hallucinate. Many panelists also mentioned they were still in the process of figuring out those guardrails. “We want to be safe,” Daniels said.

The panelists said that while customer service and chatbots aren’t new, they’re being revisited in light of the arrival of AI agents.

Agents are expected to grow more human-like in reaction and voice, and they’re more flexible. Agents don’t follow scripts like chatbots, and depending on the customer, they can connect to other agents to better serve customers.

“I can make it talk like a person. I can change dialects. I can do those things, and I want to control it,” Yum Brands’ Davies said.

Agents won’t fully replace people any time soon, as human ingenuity is still required in many areas — to supervise AI agents, modify them to be more effective, and be able to verify AI output and results.

U.S. Bank’s first AI agents are augmenting human knowledge to service customers.

Human agents for banking, mortgage or investments are subject matter experts that need information at their fingertips; this is where the bank is testing AI agents, said Sumitri Kolavennu, head of AI research and senior vice president at U.S. Bank.

“Keeping the human-in-the-loop in many [regulated] industries is … really paramount,” Kolavennu said. “We love the advantages and autonomy aspects of AI agents, but we do want to keep the human in the loop.”

U.S. Bank is testing the effectiveness of agents by seeing how quickly problems are resolved.

“The biggest thing we are seeing is resolution on first call. When you call, you don’t want the agent to say, ‘Let me call you back tomorrow’ or something like that. Those are some of the things that we are seeing [AI agents] being able to do,” Kolavennu said.