ROI Comparison: Cloud vs On-Premises Load Testing Tools

ROI Performance

Businesses today live and die by user experience. Due to the increasing complexity of their services and the pressure to deploy quickly, some companies fail to take time to ensure that their applications deliver acceptable response times under actual and future growth load patterns.

However, some IT leaders have learned from previous mistakes and integrated performance engineering successfully in their DevOps chain. They start early during the design and development stages with performance analysis, automate those tasks, review the results, and eliminate hotspots in the construction phase. This article demonstrates how you can calculate the return on performance investments of an on-premises and on-demand cloud-based load testing platform.

Costs of Performance Testing

Responsive applications, which are robust and deliver acceptable response times under load spike situations, are not free. Things would be easy if there were a performance feature that you could simply purchase, integrate, and suddenly eliminate all your hotspots. However, the reality is that there is no permanent cure for bad performance. Organizations must hire engineers, buy tools, and set up performance testing platforms on their infrastructure in order to continually address performance issues.

Professional businesses rely on commercial load testing suites. They pay an initial license fee and yearly maintenance charges to get the required support and product updates. Load injection requires powerful and exclusive load agent machines, which leads to additional annual costs. In recent years SaaS-based load testing platforms have appeared on the market. Businesses realized that these SaaS products are cheaper and easier for load testing because there are no initial fees and they are only charged for the actual usage of these on-demand platforms.

However, there is one cost driver in the performance testing space that is often ignored. Engineers will identify long running requests, applications that crash, memory leaks, and other defects. The analysis, implementation of fixes, and deployment of the improved code will cause additional work for your engineering team. Research has proven that the later issues are detected, the more expensive they are to fix. A defect which occurs in production leads to a factor of 1,000 percent higher efforts to remedy than in development stages.

Savings of Performance Testing

Reputational gains, increased trust in IT, and more available time for other tasks are benefits of load and performance tests, but it is difficulty to calculate their true financial impact. As such, we will not consider these factors when we calculate the return on performance investments.

Additional benefits include increases in revenue and earnings due to performance problems that can be detected ahead of time. Amazon, for instance, has demonstrated that a 100ms speed improvement increased their sales by 1 percent. When users experience slow loading web pages, they often abandon and spend their money on a competitor’s website. Load and performance tests help companies fix such slowdowns early in the lifecycle, protecting their online revenue streams. Those fixes on pre-production stages are easier to implement and result in additional savings for the organization.

Calculation of ROI Performance

Have you ever calculated the real costs and savings of your load and performance testing services?
Here’s how this may look when comparing an on-premises and SaaS-based load testing solution.

Take note of the following key figures in this example:

  • 1,000 concurrent user load testing tool license
  • 120 test executions per year
  • 800 HTTP based and 200 real browser based users in each test run
  • 58 load injection machines
  • $6,000 annual costs for each load injection server
  • 120 performance defects per year
  • 20% fewer defects on production due to load testing
  • 2 performance engineers earning $60,000 annually

Not considered in this calculation are:

  • Increases in sales due to less abandon rates
  • Efforts to fix identified performance defects
  • Additional compensation benefits and incurred costs of the two performance engineers

ROI of On-Premises Load and Performance Testing Platforms

Companies deploy on-premises load testing suites on dedicated servers hosted in their datacenters. There are some open-source solutions with limited user simulation features on the market, but professional businesses typically rely on commercial platforms. The vendor of this load testing suite charges an initial license fee of $300,000 and an annual maintenance fee of $60,000.

Two engineers are responsible for load and performance testing in this company, and they identify 120 performance defects per year. The company is using 58 dedicated load injection machines which result in additional expenses of $6,000 annually per machine. On the savings side, there are 20 percent fewer defects on production. Further aspects, such as the impact on revenue or less rework activities, will be not considered in this scenario for the sake of simplicity.

The chart below shows how expenditures, earnings, and the corresponding ROI progresses over a period of five years.

ROI Performance

This calculation shows that an organization which relies on an on-premises load and performance-testing platform realizes a negative -58% return on their load and performance testing investments after five years.

ROI of SaaS On-Demand Cloud-Based Load and Performance Testing Platforms

SaaS-based load and performance testing suites are fully cloud-based. The provider hosts the load agent machines and takes care for operation and maintenance of both the load testing software and the underlying infrastructure. Organizations only pay for permanent storage of their test results and the actual usage of virtual user minutes. Due to less maintenance efforts for a SaaS-based solution, the head count in the organization’s load and performance testing team can be reduced to four.

On the savings side there are 20 percent fewer defects on production. Further aspects, such as the impact on revenue, will be not considered in this example for the sake of simplicity. The chart below stresses the costs, savings, and ROI of an on-demand cloud-based load and performance testing platform over a period of five years.

ROI Performance1

This calculation shows that organizations which rely on SaaS-based load and performance testing platforms realize a 12 percent return on performance testing investments.

Comparison of SaaS On-Demand VS On-Premises Platforms

What are the ROI-impacting factors and at what point is an on-premises load-testing suite cheaper than a SaaS-based platform?

SaaS-based load testing providers charge a fee per virtual user minute. The number of test executions impacts the ROI of SaaS solutions heavily. Locally deployed on-premises load and performance testing suites don’t have this cost driver because they have a static license fee independent of how many tests they execute.

At 120 test executions per year, SaaS solutions deliver an ROI of 12 percent, while on-premises platforms fall behind with a -58 percent ROI. If this company decides to execute 600 load tests annually, SaaS-based load and performance testing platforms will reach an equivalent ROI to that of on-premises solutions. Increasing beyond 600 load test executions will see on-premises platforms yielding higher ROI than full SaaS-based platforms.

The chart below shows the development of ROI on performance investments after a period of five years for companies that execute between 120 to 600 load tests annually.

ROI Performance2

Conclusion

Investing in SaaS-based load and performance testing suites outperforms on-premises platforms in most scenarios.

The ROI of SaaS-based load testing platforms is already positive in the first year, while on-premises solutions deliver a negative ROI during the five-year test period. One main reason for better ROI of SaaS-based solutions is that there are no licensing costs, no maintenance efforts, and no internal infrastructures involved.

The decisive factor is the number of test executions (or the average test duration), as they are the primary cost driver for SaaS-based load and performance testing solutions. If the organization decides to execute 5x more (or longer) load tests, the ROI of both on-premises and on-demand load testing suites would reach the same level. On-demand cloud-based load testing platforms are the more economical option if a company executes less than 600 load tests per year.