Learn about Load Testing

ROI Comparison: Cloud vs On-Premise Load Testing Tools


What is Return on Investment (ROI)?

Return on investment is according to Investopedia a popular metric because of its versatility and simplicity. The simple formula for ROI = (gain from investment – cost of investment) / costs of investment. In this formula, gain from investment refers to the realized savings.

ROI Performance

Businesses today live and die by user experience. Due to the increasing complexity of their services and the pressure to deploy quickly, some companies fail to take time to ensure that their applications deliver acceptable response times under actual and future growth load patterns.

However, some IT leaders learned from previous mistakes and integrated performance engineering successfully in their DevOps chain. They start early during design and development stages with performance analysis, automate those tasks, review the results and eliminate hotspots in the construction phase. In this piece, we will demonstrate how you can calculate the return on performance investments of an on-premise and on-demand based load testing platform.

Costs of Performance Testing

Responsive applications, which are robust and deliver acceptable response times under load spike situations, are not free. Things would be easy if there were be a performance feature that you could simply purchase, integrate it, and suddenly all your hotspots have been eliminated. However, the reality is that there is no permanent cure for bad performance. Organizations hire engineers, buy tools and set up performance testing platforms on their infrastructure.

Professional businesses rely on commercial load testing suites. They pay an initial license fee and yearly maintenance charges to get the required support and product updates. Load injection requires powerful and exclusive load agent machines, which leads to additional annual costs. In recent years SaaS-based load testing platforms have appeared on the market. Businesses realized that these SaaS products are cheaper and easier for load testing because there are no initial fees and they just charge for the actual usage.

But, there is one more often ignored cost driver in the performance testing space. Your engineers will identify long running requests, applications that crash, memory leaks or other defects. The analysis, implementation of fixes and deployment of the improved code will cause additional work for your engineering team. Research has shown that late detected issues are much more expensive. A defect, which occurs in production, leads to a factor of 1000 or higher efforts than on development stages.

Savings of Performance Testing

Reputational gains, trust in IT, and more time for other tasks due to less troubleshooting are benefits of load and performance tests but it’s almost impossible to calculate their financial impact. We all appreciate fast loading web pages and assume that those providers manage their IT services properly but the cost impact would be very hard to calculate. The same principle is valid for reputational wins or time, which could be spent on other tasks. We will not consider these factors when we calculate the return on performance investments.

On the plus side are increases in revenue and earnings due to performance problems that can be detected ahead of time. Amazon, for instance, has demonstrated that a 100 ms speed improvement pushed their sales by 1 %. When users experience slow loading web pages, they often abandon and spend their money on a similar website from one of the competitors. Load and performance tests help companies to fix such slowdowns early in the lifecycle. Those fixes on pre-production stages are easier to implement and result in additional savings.

Calculation of ROI Performance

Have you ever calculated the real costs and savings of your load and performance testing services? Here’s how this may when comparing an on-premise based and SaaS-based load testing solution.

Take note of the following key figures in this example:

  • 1000 concurrent user load testing tool license
  • 120 test executions per year
  • 800 http based and 200 real browser based user in each test run
  • 58 load injection machines
  • $6000 annual costs for each load injection server
  • 120 performance defects per year
  • 20% fewer defects on production due to load testing
  • 2 performance engineers earning $60,000 annually

Not considered in this calculation are

  • Increases in sales due to less abandon rates
  • Efforts to fix identified performance defects

ROI of On-Premise Based Load and Performance Testing Platforms

Companies deploy on-premise load testing suites on dedicated servers hosted in their datacenters. There are some open-source solutions with limited user simulation features on the market but professional businesses typically rely on commercial platforms. The vendor of this load testing suite charges an initial license fee of $300,000 and an annual maintenance fee of $60,000.

Two engineers are responsible for load and performance testing in this company, and they identify 120 performance defects per year. The company is using 58 dedicated load injection machines which result in additional expenses of $6,000 annually per machine. On the savings side, there are 20% fewer defects on production. Further aspects such as the impact on revenue or less rework activities will be not considered due to simplicity.

The chart below shows how expenditures, earnings and the corresponding ROI progresses over a period of 5 years.

This calculation shows that an organization, which relies on an on-premise load and performance-testing platform, realizes a negative -58 % return on their load and performance testing investments after five years.

ROI of SaaS On-Demand Based Load and Performance Testing Platforms

These types of load and performance testing suites are fully cloud-based. The provider hosts the load agent machines and takes care for operation and maintenance of both, the load testing software and the underlying infrastructure. Organizations just pay for permanent storage of their test results and the actual usage of virtual user minutes. Due to less maintenance efforts for a SaaS-based solution the head count in the load and performance testing team can be reduced to four.

On the savings side, there are 20% fewer defects on production. Further aspects such as the impact on revenue will be not considered due to simplicity.  The chart below stresses the costs, savings and ROI of an on-demand based load and performance-testing platform over a period of five years.

This calculation shows that organizations, which rely on SaaS-based load and performance testing platforms, realize a positive 12 % return on load and performance testing investments.

Comparison of SaaS On-Demand VS On-Premise

What are the ROI-impacting factors and at what point is an on-premise based load-testing suite cheaper than a SaaS based platform?

SaaS based load testing providers charge a fee per virtual user minute. The number of test executions impacts the ROI of SaaS solutions heavily. Locally deployed, on-premise load and performance testing suites don’t have this cost driver because they have a static license fee independent of how many tests they execute.

At 120 test executions per year SaaS solutions delivered a ROI of 12% while on-premise based platforms fall behind with -58% ROI. If this company decides to execute 600 load tests annually, SaaS based load and performance testing platforms reach an equivalently ROI as on-premise-based solutions. Above 600 load-test executions on-premise-based platforms are cheaper than full SaaS based platforms.

The chart below shows the development of ROI on performance investments after a period of five years for companies that execute between 120 to 600 load tests annually.


Investing in SaaS-based load and performance testing suites outperforms on-premise-based platforms.

The ROI of SaaS load testing platforms is already positive in the first year while on-premise based solutions deliver a negative ROI during the whole period. One main reason for better ROI of SaaS based solutions is that there are no licensing costs, no maintenance efforts, and no internal infrastructures involved.

All-dominating factors are the number of test executions or the test duration because they are cost driver for SaaS based load and performance testing solutions. If the given company decides to execute 5 x more or longer load tests the ROI of both, on-premise and on-demand load testing suites, would reach the same level. On-demand based load testing platforms are the cheaper option if a company executes more than 600 load tests per year.