Learn about Load Testing

The Importance of ROI on Load Testing


This is the first piece of our ROI on load testing series. The series consists of:

  1. Importance of ROI on Load Testing
  2. How to Calculate ROI on Load Testing
  3. ROI on Load Testing of an On-Demand vs On-Premise Platform

Companies focus resources on performance tuning and optimization resulting in high spends on tools, engineering, and operational efforts.  If the wrong approach is followed, pitfalls are exposed, and timelines are put at risk.  However, if the proper approach is followed, there are excellent opportunities for cost reduction and financial gains.

What are the Cost Drivers?

Load and performance testing is an engineering activity and requires planning, smart decisions, and skilled engineers to make it a success. If we look at the expenditures, there are three drivers involved:

  1. Efforts related to load and performance testing tools. There are open-source solutions which are typically free, on-premise based platforms with an initial and annual license fee, and on-demand services with a pay-as-you-go charging model.
  2. The cost of developing a load and performance testing strategy. Do you have performance requirements in place and do you validate those already during DEV and QA stages? Late involvement will produce other results than early validation of performance requirements.
  3. Costs for your load injection infrastructure. Imagine that you simulate 1000 virtual browser-based user load which click through their day-to-day business transactions. A single browser session often requires 1 GB RAM and 1 Core. For this given scenario you will need eventually 100 load injection machines in your data center.

What are the Profits?

There are many advantages of performance engineering outperforming the costs involved. For instance, a large player in the e-commerce business has shown that a minor 100 ms push in speed resulted in significant growth in sales. Research has shown that one in two users won’t use a web service with load times of more than 4 seconds. Response times of 8 seconds or more result in frustration and loss of attention span which is critical also for your service applications.

Why Should we Calculate the ROI of Load Testing?

Profitability is critical for every business and most organizations make careful calculations before they invest in new products or projects. There are short-term and long-term investments but after some time the benefits should be higher than the efforts. If we look to load and performance testing suites it makes much sense to compare available solutions carefully before you take your decision.

Obviously, you will put those platforms on your short list which satisfy your requirements such as mobile testing, real browser testing or browser cache simulation feature support. Be careful though when it comes to ROI calculation of those suites because this varies between locally deployed and cloud based platforms.

Operational efforts and maintenance costs can be enormous cost drivers. Imagine that you are going to simulate a 50,000 concurrent user load test with a browser based user simulation. For this test setting you will need eventually 5000 load injection machines. The deployment of load injection software, and the hardware and patch management can lead to high efforts if you decide to use an on-premise load testing solution.

Choosing the best load testing suite is not only a feature driven undertaking. Naturally, performance testing is a risk mitigation activity with the objective to reduce operational loss due to serious outages or slowdowns. If your investment in a load testing solution outweigh your costs of performance problems, it makes less sense to consider load and performance tests as a risk mitigation.

Forward-thinking is a critical skill which helps to address minor issues before they turn into huge problem spots. Those who oversee testing and monitoring platforms are especially challenged because they need to provide flexible solutions for demanding activities. Even if you use an established testing and monitoring products, the chance is high that you miss opportunities if you don’t consider evaluating competing solutions from time to time.

Keep in mind that load and performance testing is a journey and high initial expenditures are a killer for your long-term return on investment. In our next piece on this topic, we’ll explain the essential steps to calculating ROI on load testing.