The Importance of ROI on Load Testing

This is the first piece of our ROI on load testing series. The series consists of:

  1. Importance of ROI on Load Testing
  2. How to Calculate ROI on Load Testing
  3. ROI on Load Testing of an On-Demand vs On-Premise Platform

Companies focus resources on performance tuning and optimization resulting in high spends on tools (such as Selenium), engineering, and operational efforts. If the wrong approach is followed, pitfalls are exposed, and timelines are put at risk. However, if the proper approach is followed, there are excellent opportunities for cost reduction and financial gains.

What are the Cost Drivers?

Load and performance testing are engineering activities that require planning, smart decisions, and skilled engineers to make it a success. If we look at the expenditures, there are three drivers involved:

  1. Efforts related to load and performance testing tools.There are open-source solutions which are typically free, on-premises web-based platforms with an initial annual license fee, and on-demand services with a pay-as-you-go charging model.
  2. The cost of developing a load and performance testing strategy. Do you have performance requirements in place, and do you validate those already during Dev and QA stages? Late involvement will produce other results than early validation of performance requirements.
  3. Costs for your load injection infrastructure. Imagine that you simulate 1,000 virtual browser-based user load which click through their day-to-day business transactions. A single browser session often requires 1GB RAM and one core. For this given scenario, you will eventually need 100 load injection machines in your data center.

What are the Profits?

There are many advantages of performance engineering outperforming the costs involved. For instance, a large player in the e-commerce business has shown that a minor 100 ms push in speed resulted in significant growth in sales. Research has shown that 50 percent of users won’t use a web service with load times of more than four seconds. Response times of eight seconds or more result in frustration and loss of attention span, which is critical for your service applications.

Why Should we Calculate the ROI of Load Testing?

Profitability is critical for every business and most organizations make careful calculations before they invest in new products or projects. There are short-term and long-term investments, but after some time, the benefits should be higher than the efforts. If we look to load and performance testing suites, it makes sense to compare available solutions carefully before you make your decision.

Obviously, you will put the platforms which satisfy your requirements, such as mobile testing, real browser testing, or browser cache simulation feature support on your short list of options. However, be careful when it comes to ROI calculation of those suites because this varies between locally deployed and cloud-based platforms.

Operational efforts and maintenance costs can be enormous cost drivers. Imagine that you are going to simulate a 50,000 concurrent user load test with a browser-based user simulation. For this test setting you will eventually need 5,000 load injection machines. The deployment of load injection software and the hardware and patch management, can lead to high efforts if you decide to use an on-premises load testing solution.

Choosing the best load testing suite is not only a feature-driven undertaking. Naturally, performance testing is a risk mitigation activity, with the objective to reduce operational loss due to serious outages or slowdowns. If your investment in a load testing solution outweigh your costs of performance problems, it makes less sense to consider load and performance tests as a risk mitigation.

Forward-thinking is a critical skill which helps to address minor issues before they turn into huge problem spots. Those who oversee testing and monitoring platforms are especially challenged because they need to provide flexible solutions for demanding activities. Even if you use an established testing and monitoring product, the chances are high that you’ll miss opportunities if you don’t consider evaluating competing solutions from time to time.

Keep in mind that load and performance testing is a journey and high initial expenditures are a killer for your long-term return on investment. In our next piece on this topic, we’ll explain the essential steps to calculating ROI on load testing.