What Is Chaos Engineering?
Your customers, clients, visitors, and internal employees rely on your systems to be available, stable, and performant. In a perfect world, applications and services would never go down. In reality, outages happen, dependencies fail, cloud regions have issues, traffic spikes unexpectedly, and small weaknesses can turn into larger incidents.
Chaos engineering, also called chaos testing, is a disciplined approach to testing how systems respond to unexpected failures. Instead of waiting for an outage to expose weaknesses, teams intentionally introduce controlled failure conditions to learn how the system behaves and how resilient it really is.
With the adoption of DevOps practices, more organizations have added chaos testing to their reliability workflows. Whether chaos engineering is owned by a dedicated team, platform engineering, or site reliability engineers (SREs), the goal is the same: uncover hidden weaknesses before they affect users.
Chaos Engineering vs. Performance Testing
Like stress testing or load testing, chaos engineering helps teams identify breaking points and failure behavior. The difference is the type of failure being studied.
Performance testing focuses on how a system behaves under traffic, transactions, concurrency, and load. Chaos engineering focuses on how a system behaves when something goes wrong, such as a service failure, network delay, database issue, dependency outage, region failure, or instance termination.
The two practices are closely related. A system may perform well during a normal load test but fail when a downstream service slows down, a cache becomes unavailable, or a region loses capacity. Combining performance testing with chaos engineering gives teams a better view of how applications behave under both high demand and partial failure.
How Chaos Engineering Started
Chaos engineering is closely associated with Netflix. As Netflix moved from traditional data centers to the cloud, the company needed a way to prove that its systems could survive failures in a distributed environment. Cloud migration introduced benefits such as scalability and flexibility, but it also created new failure modes around instances, zones, regions, databases, networking, and autoscaling.
Netflix developed tools and practices to intentionally test failure scenarios before real outages exposed them. One of the best-known examples was Chaos Monkey, a tool designed to terminate instances so engineering teams could confirm that services recovered properly and users were not significantly affected. From this idea, the broader practice of chaos engineering grew.
Chaos Engineering Principles and Steps
Chaos engineering does not mean breaking things randomly. A good chaos experiment is planned, controlled, measured, and reversible. The purpose is to learn how the system behaves under failure and reduce risk before users are impacted.
Step 1: Create a Hypothesis
Start with an assumption about how the system should respond when a failure condition is introduced. For example, you may expect that if one service instance goes down, traffic will be routed to healthy instances and error rates will remain within an acceptable range. This is also where you define the metrics to measure, such as error rate, latency, throughput, recovery time, saturation, and user-impact signals.
Step 2: Identify Variables and Anticipate Effects
Decide which failure condition to introduce and what effect it may have. For example, what happens if a server crashes, a dependency slows down, a database connection pool fills up, or there is a sudden increase in traffic? Teams should define the expected impact before the experiment starts.
Step 3: Run the Experiment Safely
Chaos experiments are often most valuable in production because that is where real dependencies, traffic, and user behavior exist. However, production experiments need safeguards. Teams should limit the blast radius, choose a low-risk window when appropriate, notify the right people, monitor closely, and have a rollback or stop condition ready.
Chaos experiments can also be run in staging or pre-production environments, especially when teams are new to the practice. These environments may not fully match production, but they can still help validate tooling, processes, and recovery behavior before larger experiments are attempted.
Step 4: Measure the Impact
After the experiment, compare the results with the original hypothesis. Did the system behave as expected? Were error rates, response times, and recovery times acceptable? Was the blast radius too large or too small? Did the experiment uncover additional issues that need investigation?
The value of chaos engineering comes from the learning that follows. Teams should document what happened, fix weaknesses, update runbooks, improve alerts, and repeat the experiment when needed.
Chaos Engineering Tools
Chaos Monkey is one of the best-known chaos engineering tools. It was created at Netflix to randomly terminate instances so teams could confirm that services were resilient to instance failure. This may sound painful, but it pushed teams to build systems that could survive real infrastructure problems.
Proactively running chaos experiments can help teams build more reliable systems and improve the user experience. Real production environments are not controlled labs. Dependencies fail, networks slow down, instances disappear, and users still expect the application to work.
Chaos Monkey was once part of a broader Netflix toolset called the Simian Army. Some of those tools have since been retired, replaced, or absorbed into other platforms, but the concepts remain useful for understanding how chaos engineering evolved.
Chaos Monkey
Chaos Monkey was designed to terminate service instances so teams could verify that applications handled instance loss without major user impact.
Chaos Kong
Chaos Kong was designed to simulate a larger AWS region failure and observe whether the system could recover by shifting traffic to another region without unacceptable performance degradation.
Conformity Monkey
Conformity Monkey checked whether instances followed predefined rules. Instances that did not conform could be reported to the owning team for correction.
Chaos Gorilla
Chaos Gorilla simulated the failure of an entire availability zone. The goal was to test whether services, load balancers, and failover systems could keep the application available during a large-scale cloud infrastructure issue.
Latency Monkey
Latency Monkey introduced network delays or failures to help teams understand how services and dependencies responded when latency increased. This helped expose timeout, retry, fallback, and dependency-chain problems.
Doctor Monkey
Doctor Monkey monitored instance health, such as CPU load, memory, and resource usage. It could report unhealthy instances and remove them from service when needed.
10-18 Monkey
10-18 Monkey focused on localization and internationalization risks. For a global service, region-specific language, configuration, character set, and performance issues can affect reliability and user experience.
Janitor Monkey
Janitor Monkey identified unused cloud resources, such as groups, volumes, or instances, and flagged them for cleanup. Over time, this type of function was replaced by other internal tools and cloud resource management practices.
Modern Chaos Engineering Examples
Today, chaos engineering can include many types of experiments, such as:
- Terminating instances, containers, or pods
- Adding latency between services
- Simulating packet loss or network partitioning
- Disabling a dependency or third-party API
- Restricting CPU, memory, disk, or network resources
- Testing database failover or replica lag
- Introducing DNS, CDN, or load balancer issues
- Testing autoscaling behavior during partial failure
- Combining traffic load with dependency failure
These experiments should be tied to a clear business or reliability question. For example: Can checkout continue if the recommendations service fails? Can the application handle peak load if one availability zone is unavailable? Do retries make a dependency outage worse? Does the user see a graceful error or a broken page?
Conclusion: Chaos Engineering Principles, Examples, and Tools
Chaos engineering has grown into a mature reliability practice with open-source and commercial tools such as Litmus Chaos, Gremlin, Chaos Mesh, and others. It is no longer only for large technology companies. Banks, ecommerce companies, healthcare organizations, government agencies, educational institutions, and SaaS providers all depend on distributed systems where failures can be hard to predict.
As systems become more complex, teams need proactive ways to test resilience. Performance testing and chaos engineering both help teams learn how systems behave under stress. Performance testing shows what happens under traffic. Chaos engineering shows what happens under failure. Together, they help teams build more reliable applications and protect the user experience.