The paradox of chaos engineering

Chaos engineering involves stress-testing systems by simulating real-world adversities, such as cyberattacks and internal failures. By creating controlled chaos, organizations hope to prepare their infrastructure for unforeseen incidents and minimize potential downtime. Observing how a system withstands these disturbances can pinpoint hidden vulnerabilities that traditional testing might overlook. Is this a cutting-edge methodology organizations need to bolster their defenses against ever-evolving cyber threats, or is it a dangerous distraction?

Although chaos engineering offers potential insights into system robustness, enterprises must scrutinize its demands on resources, the risks it introduces, and its alignment with broader strategic goals. Understanding these factors is crucial to deciding whether chaos engineering should be a focal area or a supportive tool within an enterprise’s technological strategy. Each enterprise must determine how closely to follow this technological evolution and how long to wait for their technology provider to offer solutions.

The high cost of oops

In their most recent quarterly analysis of cybersecurity threats, cloud computing security company Cloudflare reported a 65% increase in distributed denial of service (DDoS) attacks in the third quarter of 2023 compared to the previous quarter. According to Cloudflare, the second quarter of 2024 had four million DDoS attacks.

Companies using cloud-based software are vulnerable to outages, in addition to DDoS and other deliberate attacks. These are typical problems, mostly related to humans operating clouds, but some are caused by connection problems due to physical server failures or cyberattacks.

On July 19, 2004, CrowdStrike’s Falcon sensor caused Windows hosts connected to the Microsoft Azure cloud computing system to crash. As you may recall, this caused a global IT outage. The Falcon sensor, designed to prevent cyber-related attacks, was not compromised by a cyberattack, but by a technical issue with an update.

This was a wakeup call for several reasons:

Most enterprises began to realize how vulnerable they are. Productivity could stop because of a stupid mistake.

The total cost of this event was much higher than most enterprises expected. It also had a bigger-than-anticipated impact on soft issues such as public relations and customer relationships.

The clear lesson is that the greatest risk comes from people, not technology.

Chaos engineering advantages

Let’s say that a major e-commerce company implements chaos engineering to examine its cloud system resilience during peak shopping seasons. They use a chaos engineering tool to simulate increased traffic loads that mimic Black Friday conditions. The team deliberately introduces latency and random server shutdowns to observe how the system responds under stress.

During these tests, they discover bottlenecks in their database architecture that traditional testing should have noticed. With real-time metrics, they quickly implement adaptive strategies like autoscaling server resources and optimizing database queries. By continuously iterating these chaos experiments, the e-commerce platform not only withstands simulated pressures but enhances its ability to automatically adjust to unexpected spikes. This ensures, or should ensure, a seamless customer experience during critical sales periods. This proactive approach transforms potential chaos into an opportunity for strengthening infrastructure resilience. At least, that’s the idea.

Chaos engineering drawbacks

Despite its benefits, chaos engineering poses significant challenges and questions for enterprises:

Resource intensiveness. Implementing chaos engineering requires substantial investments in the right tools, skilled personnel, and time to effectively simulate and analyze scenarios. This can strain budgets and divert attention from core business objectives.

Operational risks. Intentionally introducing faults carries inherent risks. Enterprises must be cautious, as these practices can disrupt services, affect performance, and create unwanted side effects that might result in customer dissatisfaction or financial loss.

Focus shift. Chaos engineering might distract from more strategic initiatives. Enterprises often prioritize straightforward ROI-based projects that contribute directly to growth. Engaging extensively in chaos engineering could detract from pursuing innovations or operational improvements that show immediate benefits.

Complexity management. As enterprises grow, their systems become more complex. Chaos engineering requires a deep understanding of interdependencies within systems. Managing this complexity is challenging and might deter companies from effectively applying chaos principles.

A balanced approach

This article is not a sales pitch for chaos engineering. I’m looking at it through the lens of enterprise IT, which may view chaos engineering as yet another rabbit to follow down a hole.

Chaos engineering offers a proactive defense mechanism against system vulnerabilities, but enterprises must weigh its risks against their strategic goals. Investing heavily in chaos engineering might be justified for some, particularly in sectors where uptime and reliability are crucial. However, others might be better served by focusing on improvements in cybersecurity standards, infrastructure updates, and talent acquisition.

Also, what will the cloud providers offer? Many enterprises get into public clouds because they want to shift some of the work to the providers, including reliability engineering. Sometimes, the shared responsibility model is too focused on the desire of the cloud providers rather than their tenants. You may need to step it up, cloud providers.

Ultimately, enterprises should consider how chaos engineering fits into their broader IT strategy. By integrating elements that align with their objectives rather than adopting chaos engineering wholesale, companies can benefit from the insights without being sidetracked from their core missions. As with any innovation, the key is judicious application.