The crisis of AI’s hidden costs

The AI gold rush is creating an unexpected problem: massive waste in resource provisioning. Cloud cost overruns inevitably lead to a lack of true ROI for these systems. Many enterprises spend $2 to get $1 of benefits. That dog won’t hunt.

Recent data shows a staggering reality of organizations hemorrhaging money through overprovisioned cloud resources, with only 13% of provisioned CPUs and 20% of memory being utilized.

Let me paint you a picture of what keeps CFOs up at night. Imagine walking into a massive data center where 87% of the computers sit there, humming away, doing nothing. Sounds crazy, right? That’s exactly what’s happening in your cloud environment. If you manage a typical enterprise cloud computing operation, you are wasting money. It’s not rare to see companies spend $1 million monthly on cloud resources, with 75% to 80% of that amount going right out the window. It’s no mystery what this means for your bottom line.

If you’re running 1,000 high-performance instances and each costs a buck an hour, that’s $720,000 a month, but you are probably only using about $93,600 worth of computing. The rest is just expensive digital decoration. But wait, it gets worse. You’re not just wasting money on unused compute and storage; you’re also paying for cooling, power, management, and software licenses for capacity that is sitting there collecting digital dust. Cloud providers are not charities. They’re passing that cost on to you. That wasted capital could be funding innovation, driving competitive advantage, or just making your shareholders happier. Overprovisioning is masking more profound problems in your architecture.

I’ve been in this game long enough to know that cloud computing is supposed to be your competitive advantage, not your financial anchor. Right now, for most enterprises, it’s the latter. Until enterprises get serious about tackling this waste, the promise of cloud economics will remain just that—a promise.

Numbers don’t lie, but people lie about numbers

In 2023 alone, cloud providers deployed 878,000 accelerators that generated seven million GPU hours of work, resulting in approximately $5.8 billion in revenue. These numbers mask a troubling inefficiency. The revenue figures would significantly increase if these resources were utilized more effectively.

AI workloads have exponentially worsened this problem. Cloud providers are racing to deploy tens of thousands of GPUs and AI accelerators, yet evidence suggests most of these processors are being underutilized. Consider AWS’s UltraScale clusters. Each comprises 20,000 Nvidia H100 GPUs, which could theoretically generate $6.5 billion annually at full utilization but aren’t coming close to that figure.

Organizations typically overprovision cloud resources by one-third more than what they actually use. More than half of organizations cite a lack of visibility into cloud usage as the primary culprit behind this wasteful behavior. This problem is further compounded by the AI boom, which has driven data center component revenues to record highs, growing 127% yearly to $54 billion.

Avoiding AI-driven cloud waste

Smart enterprises aren’t just hoping the problem will disappear; they’re taking action. Here’s my advice:

Double down on real-time monitoring. Don’t rely solely on the basic tools offered by your cloud provider; they won’t give you the immediate cost visibility you need. Instead, invest in third-party solutions that provide a clear, up-to-the-minute picture of your resource utilization. Focus on power-hungry GPUs running AI workloads.

Optimize your resource allocation. Rather than spinning up more instances, consider rightsizing. Modern instance types offered by public cloud providers can give you more bang for your buck.

Use AI to manage your cloud resources. Predictive analytics can help you scale up or down based on demand, ensuring you’re not paying for idle resources.

Don’t just focus on instance sizes. Be strategic and look at the bigger picture. Evaluate reserved instances and savings plans to balance cost and performance. Remember, you might have workloads running on large instances when fewer, more efficient ones could do the job better.

Regularly audit your GPU utilization. Consistently low CPU and memory utilization rates are red flags. Monitor the gap between what you provision and what you use, especially for AI workloads.

Cloud resource inflation isn’t just about costs. It’s about efficiency and sustainability, including processes and best practices. Organizations need to take a hard look at their cloud resource allocation strategies, especially as AI workloads become more prevalent. The key is to balance having enough resources to handle peak demands while avoiding the trap of expensive overprovisioning.

Yes, this sounds like common sense, but I’m getting a call a week from boards of directors, CFOs, and CIOs upset about budget-blowing cost overruns, often caused by cloud admins, cloud architects, or other IT staff who don’t even understand that they have a problem. That’s unacceptable.