What Makes an Outstanding Load Testing Platform

Successful companies understand that the speed of their business applications is a top priority because frustrated customers will quickly move away to a competitor’s website. Every lost customer has an impact on financial revenue. Therefore, businesses have started to improve reliability of their applications and spend more money on load testing and application monitoring. The former helps to eliminate hotspots, from pre-deployment into production. The latter is mainly used to identify outages or slowdowns to isolate the root cause of issues during production stages.

In this piece, we will shine a light on load testing platforms, give some differentiators, and describe why you should use LoadView in your next performance engineering assignment.

 

Different Types of Load Testing Solutions

Almost 30 years ago, the first web pioneers created load simulation platforms. At this time, web pages were simple, and content was mainly static. With the rise of technology, many services today are dynamic and constantly changing. Competition is high, and businesses try to hold existing customers, or gain new customers, with even better services. One opportunity to improve service quality is to provide responsive and reliable applications.

In recent years new load and performance testing tool solutions have appeared in this growing market. Pioneers such as JMeter and LoadRunner were deployed inside company’s local network on their machines. With the rise of cloud computing, some of them extended their services to SaaS or on-demand load testing platforms.

On-Premises

There are open source and commercial on-premises load testing solutions, which are deployed on a customer’s local infrastructure. While open source tools are free of charge, commercial solutions charge an initial license fee and deliver the software. The customer deploys updates on their local testing servers themselves. If some load testing script or execution challenges occur, a support specialist can be involved to investigate and fix the identified issue.

On-Demand or SaaS

Maintaining a local load testing infrastructure can be challenging. Switching to a cloud-based offering avoids the pain of operating local load testing suites. The benefit is that there are no maintenance tasks or fees involved, and the client pays for the required service.

How Does Load Simulation Work?

Manual tests are designed to detect functional issues, such as a feature that is not working as intended. These types of tests are not feasible for testing how your new application behaves under production-like load conditions. Performance testing helps to validate the reliability of your new system under actual and future growth load volumes. In contrary to manual testing, load testing is a highly technical task and often requires experienced specialists.

The basic steps of a load test setting include the following:

  1. Calculate the load pattern
  2. Identify the relevant user interactions
  3. Automate the testing scripts
  4. Configure the test scenario such as regular and peak load
  5. Assign the load injection machines
  6. Execute the tests
  7. Analyze the results
  8. Fix defects and repeat the test until hotspots have been fixed

All on-premises or on-demand performance testing platforms support the mentioned steps to some extent. Open-source solutions provide limited user simulation techniques. Some new on-demand and SaaS-based providers try to convince new customers with extremely high free user simulation opportunities. You can waste your money with both solutions and it is crucial to evaluate each carefully before you make your decision. In the next few sections, we will give you some guidance around how to select the best load testing suite.

 

The Importance of Self-Monitoring During Load Generation

Many pop-up load testing platforms offer simulation of thousands of virtual users for free, but what can you expect from their proposals? Do you think that they will provide their powerful, cloud-based load injection machines for free? Obviously, this is not the case and they just promise something which won’t work at all.

Based on our experience with hundreds of performance testing projects, an accurate and reliable load testing farm can do three main things:

  1. Monitoring of the load on load injection machines
  2. Inform the user about actual health of load generators during a test run
  3. Never share load generators for multiple tests at the same point in time

Aggressive competitors in the performance testing business often share load injectors across many tests and simulate a high number of virtual users on a single machine. The consequence is that response times are far too high, and the expected load pattern can’t be achieved at all. From a customer’s perspective, a few thousands concurrent users for free is very attractive. If you look carefully and think about this offer, you will come to realize that there are some pitfalls involved.

The first being that you can’t trust response times captured on an overloaded load test injector. Simulated users on such machines will fight for CPU and memory resources. Those servers will start paging, the processor queue will increase, and the user experience metrics be negatively impacted.

Secondly, due to the high load on the injection machine, the requests will arrive delayed at your application under test, which leads to a wrong load pattern. Your system being monitored will eventually easily handle the low number of requests.

Finally, you will start tuning the system in a wrong way because the incorrect response times and the low number of requests will point you into the wrong direction. After tuning and re-running those invalid load tests with totally overloaded load injection machines, you will still experience the same issues.

To sum up this section, it’s highly recommended to use a load testing solution such as LoadView, which monitors its load agents permanently during a test execution, provides health monitoring charts while the test is executed, and never shares load generators with other tests.

 

What Makes a Good Load-Testing Platform?

It’s safe to assume that many companies are not happy with their load-testing suite because they are too expensive, usability is poor, maintenance efforts are high, and reusing automated testing scripts is extremely limited.

Based on our experience, an outstanding performance-testing platform fulfills the following key requirements:

  • Ease of use: Create a script and run a load test within several hours
  • Technology support: Support capture and replay of the most popular browsers, mobile devices, web2.0 techniques, Java, Citrix, SAP, and .NET-based applications
  • Realistic user simulation: Provides protocol, headless, and full browser-based user simulation techniques
  • Reusability: Use automated user interactions for performance test and application monitoring on production stages
  • Support community: Knowledge base and professional support provides guidance to fix automation or tooling issues
  • Maintainability: Close to zero tool maintenance effort
  • Costs: Pay by use is the preferred charging model
  • Self-monitoring: Distribute load across injection machines and built-in features to avoid overloaded machines falsifying load test results

 

LoadView – An Even Better Load Testing Suite

Why should businesses setup and maintain their own performance testing suite? Obviously, this is a waste of time and money because the effort to install the load injection software often exceeds the time required for test automation, execution, and results analysis. LoadView is a fully SaaS-based service hosted in the cloud. There is no need to deploy your private load generation machine. You can create your account and immediately start with test automation and execution.

The majority of performance testing solutions are complicated. It will take you ages to understand how they work, and without experienced engineers, you might be lost. LoadView is everything but complicated. It’s fully web-based UI and intuitive design lets you create test scripts and execute tests within minutes. No coding skills are required to automate complex user interactions.

LoadView addresses the problem of unrealistic load patterns with its flexible user simulation techniques, from protocol to real browser-based. Furthermore, LoadView calibrates your automated scripts, distributes those across load generation machines, and automatically avoids situations where an overloaded injection machine falsifies your response times.

Execution of load and performance tests during development and QA stages is highly recommended and gives you confidence that your new application can handle production user and transaction volumes within expected boundaries. Nevertheless, user, content, and data volumes will change over time and can lead to serious reliability issues. Therefore, performance is more a journey than a destination and you should permanently monitor end-to-end response times and availability on production stages. In a best-case scenario, you can reuse your load testing assets for application monitoring at production. LoadView provides this flexibility and comes up with a closed loop testing approach, enabling you to reuse testing scripts for production monitoring.

 

Load Testing Platform Evaluation Sheet

There are hundreds of load testing solutions on the market today. Companies often have no idea where to begin or what criteria should be used to make their decision, therefore, we’ve created a matrix to assist in the decision making process.

  1. Answer this form for each load testing solution on your short list
  2. Select the appropriate rating for each criterion
  • ++ means that it completely fulfills the criteria
  • + means that this is a good fit
  • – means that it somewhat meets the criteria
  • — means that it doesn’t meet the criteria in any way
  1. Calculate the score
  • Each x in ++ column is a 3
  • Each x in + column is a 2
  • Each x in – column is a 1
  • Each x in – column is a 0

The matrix below shows the rating for LoadView. Its score of 23 (24 would be the max value) is extremely high and demonstrates that LoadView is a perfect match for forward-thinking and cost aware companies.

 

Criteria Rating of LoadView
 ++  +  –  —
Ease of use x      
Technology support   x    
User simulation x      
Reusability x      
Support community x      
Maintainability x      
Self monitoring x      
Costs x      
  21 2 0  
Total Score 23

Don’t waste your money—replace your underperforming load testing solution with LoadView. Get started today and get a load test running in minutes!