Today, users are offered such a level of variety in applications that even 5 seconds delay in website load time and response time higher than eight seconds can cause you a major loss. For instance, in 2013, only a 5 minute downtime of Google made the company lose almost $545,000. Similarly, in January 2016, a software update caused the Nest thermostat to stop working and left many users cold, who later ranted about the problem on social media. Also, it is not long ago that Amazon Web Service outage caused the companies’ sales loss worth $1100 per second.
Let us have a look at canary testing, its meaning, benefits, and downsides, so we can justify its significance.
From setup to test execution in just minutes.
What is Canary Testing?
Canary testing, canary release, and canary deployment are the terms used alternatively for a technique that enables continuous delivery while keeping the risk minimum. The process is to roll out a software update to a selected group of users for testing. The method got its name from canary birds which were used as toxic gas detectors in the coal mines in the past. Canaries are more sensitive to toxic gases than humans therefore, they immediately fall sick or die at low exposure levels. This helped the miners to evacuate the mines and escape the danger before it was too late.
Rollout Strategy
Similarly, in canary testing, the canaries are a small group of users who experience the latest software updates, and their feedback helps the development team decide if the new release should be made available to all the users or the changes should be rolled back quickly. Canary testing is an application of parallel change that is done on a separate production environment to which only a few selected users are routed, while the majority is kept on the old production environment. This migration of users lasts until all the users have been shifted to the new version. This is an incremental approach that supports innovation and reduces the risk of disruption at a mass level.
Rollback Strategy
If the new version proves to be extremely erroneous and poses a great threat to the user experience, instead of reversing the changes in the production environment; the users are redirected to the stable old version. Meanwhile, the development team strives for bug fixing.
Who are the Selected Users?
The next question is who shall be the users impacted by your software update? Well, there are different ways to decide upon the canary users, such as:
- They could be the in-house or remotely distributed testing teams
- From a specific region, in case of geographically distributed users
- The single brand first, in case of multiple brands
The best application of canary testing that can be cited is Facebook. Facebook works with a “multiple canary” strategy. Only internal employees are exposed first to the new changes with all Feature Toggles turned on to detect the issues early.
What are the Benefits of Canary Testing
You cannot measure the benefits of something unless you have something else to compare it with. So, yes, the canary testing is often confused with A/B testing and blue-green testing. However, it’s the best of both worlds. Let’s see why.
A/B Testing
By implementation, A/B testing is similar to canary testing. 50 percent of the users are provided an upgraded version of the application with new feature “B.” The remaining 50 percent of users keep using the baseline feature “A.” The feature release results are then monitored to observe the results against the expectations. However, the focus of A/B is to see how useful a feature is for customers. While canary testing is carried out for risk mitigation.
Blue-Green Deployment
Blue-green deployment is focused on eliminating downtime and disruption for end users. The changes are made on an idle production environment to which all the users are migrated from old production, once testing is completed. However, in canary testing, the volume of impacted users is kept minimum at first.
We can conclude that canary testing is the best of both worlds i.e, A/B testing, and blue-green deployment. The benefits of the canary deployment approach can be listed down to:
- It aids innovation, expedites the agile development, and allows easy roll out of new features.
- It ensures zero downtime.
- It allows for a quick roll back of the changes in case of negative results.
- It allows a high level of configuration for the conditions to determine which traffic is sent to the new release, before releasing it to the entire user base.
- It allows multiple teams to simultaneously test individual micro-services on production.
- Unlike blue-green deployment, resources expansion is not required to test an additional environment. Because per micro-service resource strategy is adopted for testing.
- The behavior of the new version can be monitored by slowly ramping up the load on production, instead of setting up a whole new capacity testing environment.
- It allows the deployment of multiple versions in production. This provides the team with desired freedom, flexibility, and velocity to test the new release.
Canary Deployment and LoadView
You can achieve the best results for performance testing without compromising the user experience of the masses. All you need is the practice of canary deployment and an excellent testing tool that supports metrics for performance testing. Using the canary approach, execution of performance test cases can be automated with the help of an advanced solution like LoadView. This might sound confusing and complex. But let’s try to understand it with the help of an example. For instance, Netflix, a single-page application (SPA), develops a new feature and releases it to the canary users (7,50,000 i.e, 5 percent of 15 million; based on 2020 stats).
Now, no DevOps team can ever monitor such a massive number of users manually. Thus, LoadView acts like a magic wand, as it empowers the DevOps and the testing teams to spin up hundreds to thousands of concurrent users to load and stress test against their applications, and see how their applications perform under that applied load.
Furthermore, what makes LoadView, with its EveryStep Web Recorder scripting tool, a compelling choice is that it supports load testing SPAs. SPA logic relies on JavaScript technology which is not supported by many tools, like JMeter for example. JMeter only works at the protocol level, and not the browser, so a lot of the client-side performance is left out of the performance testing results. With the EveryStep recorder, creating user scenarios and paths is easy and utilizes real browsers. Simply record the path and set up your test scenario. It’s really that easy.
Challenges of Canary Deployment/Testing
Every coin has two sides. Some would say that canary deployment/testing methodology has its cons, but it’s more appropriate to state the following mentioned points as its challenges.
- DevOps teams have to manage multiple versions of the software in parallel. Therefore, it is advised to keep the number of concurrent versions to a minimum.
- You do not have complete control over the new version when it is upgraded if your solution software is installed on users’ devices.
- Database management becomes difficult during canary releases.
- The implementation of canary tests demands substantial effort for effective and robust monitoring of the infrastructure and application.
- The monitoring of each incremental release takes time and might affect the target release date. Canary tests can take up to several hours.
Conclusion: Canary Testing
To conclude it all, it can be deduced that the canary deployment/testing approach can be your next best friend in risk mitigation when it comes to doing performance testing of your software’s new version with actual users without any downtime. As Ian Molyneaux stated in his book, The Art of Application Performance Testing: From Strategy to Tools, “If an end-user perceives bad performance from your website, her next click will likely be on your-competition.com.” Don’t let that happen. Make sure your clients and customers are getting a great user experience.