Load Simulation & Performance Test Types

Slow loading or non-responsive web pages have a direct impact on financial revenue. Ensuring that you set up your performance tests with the right load simulation is critical. If not done correctly, it could have disastrous effects. Frustrated users will most likely abandon whatever it was that they were doing and not return. Even if you and your team has resolved this issue, it is likely too late. They have probably already found an alternative solution. And once they have, it is unlikely they will switch back.

Not only do you have to worry about lost revenue, but other factors, such as consumer confidence, brand integrity, etc., are probably more detrimental to the business. The longer the issue lasts, the harder it is to regain consumer’s confidence. Not only that, the return on the investments you made into your product, teams, and organization is going to take longer to realize, if at all. Therefore, performance testing (and monitoring) has become a fundamental and critical part of the software and application development chain. The need for a solution that can properly and accurately execute performance tests closely to the user’s experience is in high demand.

Performance testing platforms provide a broad range of load simulation methods such as HTTP/S, headless, and real browser-based testing. In this article, we will outline the main aspects of each, followed by a comparison matrix that you can use for choosing an appropriate simulation approach.

HTTP-Based Load Simulation

In the early days of our digital age, HTTP-based testing was very popular. With the rise of rich web client technology, this approach has become increasingly outdated (and performance testing with tools such as JMeter has become a bit outdated as well). A typical HTTP-based test driver executes service requests and parses responses. Modern web 2.0 applications consist of many client-side scripts, which are totally ignored and not measured in this type of test execution. Due to a shortage of client-side generated session IDs, complex use cases cannot be simulated at the protocol level.

Due to their request and response-based nature, the overhead on the load injection machine is very low. A typical load test server can simulate up to 800 simultaneous sessions. Complex protocol-based use cases can be difficult to implement. A performance engineer needs to deal with cookies, session IDs, and other dynamic parameters. Depending on the technology of the system being tested, some web form names often change once a new version has been deployed, which will cause the HTTP-based script to fail.

The sample script below showcases the very technical nature of those scripts. If a technical attribute of your application changes, you must rewrite the whole script, which is easier said than done.


//sample protocol level script
transaction TMain
var
hContext: number;
begin
WebPageUrl("https://lab3/st/", "Greetings");
WebPageStoreContext(hContext);
WebPageLink("Join the experience!", " - New Visitor");
WebPageSubmit("Continue", CONTINUE001, "Main menu");
WebPageLink("Products", "ShopIt - Products");
WebPageLink(NULL, "ShopIt - Product", 3);
WebPageSubmit("Search", SEARCH001, " - Search", 0, NULL, hContext);
end TMain;

dclform
CONTINUE001:
“name”            := “jack”,
“New-Name-Button” := “Continue”;

SEARCH001:
“search”          := “boot”;

After all, protocol level scripts are good for component level tests in continuous integration environments, and due to their low footprint on load injection machines, are the perfect setting for stress tests.

Headless Browser-Based Load Simulation
With the rise of web 2.0 technologies the testing business was faced with serious challenges. Rich browser applications could no longer be tested or simulated at the protocol level due to missing client-side logic during script replay. Therefore, several headless browsers were introduced, such as HtmlUnit, PhantomJS, or SlimerJS. They are often built on top of WebKit, the engine behind Chrome and Safari.

Headless browsers have all the advantages of real browsers and they run faster without the heavy GUI. Many test automation and performance testing platforms are using headless browsers as they allow realistic user simulation.

Some providers have built their own headless browser engines but have run into maintenance pitfalls because they must keep up with new browser versions. In this situation, it is highly recommended to use any freely available headless browser kits because there is a broad community which works on improvements.

The simulation of client-side rendering is not for free. A typical load injection server can simulate up to eight simultaneous headless browser sessions.

Test script implementation and customization is not too difficult. If you have basic developer skills, you will be able to create simple scripts. Not all headless browsers provide visual replay features and without visual replay, script debugging, and error analysis can become very tricky.

In the sample script below a simple post request is executed. You need Java programming skills to customize such test scripts.


//sample phantomjs script
"use strict";
var page = require('webpage').create(),
server = 'https://posttestserver.com/post.php?dump',
data = 'universe=expanding&answer=42';

page.open(server, ‘post’, data, function (status) {
if (status !== ‘success’) {
console.log(‘Unable to post!’);
} else {
console.log(page.content);
}

phantom.exit();
});

With this consideration in mind, headless browsers are the way to go if you have programming skills and you are using a solution that allows visual script replay.

 

Real Browser-Based Load Simulation

Web 2.0 applications are full of JavaScript, Flash, AJAX, CSS, etc. Without a full browser, it is not possible to measure the actual end-to-end response times of the entire application or web page. Real browser-based performance testing allows you to verify site functionality and speed as perceived by the end user, which as we discussed above, is where it really counts.

A typical real browser performance test solution collects loading times of images, JavaScript, CSS, and more. Often, they provide waterfall charts, which visualize the load time of those components.

The footprint of a real browser-based browser is slightly higher. Because headless browser simulation does not deliver a 100 percent realistic response times, it is fair to say that real browser-based simulation should be the preferred method for testing.

Implementation and maintenance of test scripts is easy because user actions are directly reflected, and the visual replay makes debugging easy. In the sample script below, the browser opens a URL, inserts the user’s password, and clicks the login button.


//sample real browser-based script
transaction TMain
begin
BrowserStart(BROWSER_MODE_DEFAULT, 800, 600);

//navigate to the login site
BrowserNavigate(“https://demo.com/TestSite/LoginForm.html”);
//set the authentication for the secure site
BrowserSetText(“//INPUT[@name=’user’]”, “User1”);
BrowserSetPassword(“//INPUT[@name=’pwd’]”,     “Password1”);
//Login
BrowserClick(“//INPUT[@value=’Login’]”, BUTTON_Left);
end TMain;

After all, real browser simulation is useful for realistic end-to-end load tests. However, don’t use it for stress testing because the footprint on the load injection server is too high.

 

Different Types of Load and Performance Tests

Component Speed Tests

In recent years, software development methods have moved in the agile direction. Short release sprints are essential. Developers and test engineers automate their quality assurance and performance checks. Typically, they implement service-based performance tests (and sometimes API tests) on protocol level, or they simulate real browser-based performance checks to compare end-to-end response times with agreed performance boundaries.

Objectives of Component Speed Tests:

  • Verify and repeat input/output behavior.
  • Automated interface and end-to-end performance checks.
  • Comparing response times with agreed thresholds.

Load Tests

Load tests the ideal setting when it comes to verification of non-functional requirements. One reason is that the response times can be verified under reproducible conditions. Another being that those tests allow verification of response time thresholds. Realistic response time measurement is essential in load test scenarios. Therefore, test engineers use headless or real browser-based user simulation for their load test settings.

Objectives of Load Tests:

  • Reproducible load simulation.
  • Verification of response time thresholds.
  • Identifying bottlenecks under production-like load conditions.
  • Creating realistic end-to-end test scenarios.

Stress Testing

If you must prove the reliability of your application under peak load conditions, run a stress test to determine the breaking point of your system.

Objectives of Stress Testing:

  • Prove scalability and stability at peak traffic conditions.
  • Observe how system fails and comes back online.
  • Reproducibility is not important.

 

Comparison

Obviously, there are good reasons and situations for protocol, headless, and real browser-based user simulations. The matrix below provides some guidance on when to choose the appropriate testing approach.

Criteria HTTP Headless Browser Real Browser 
User simulation (1) No client-side rendering (2) Some client-side elements are simulated (3) Real user simulation
Script implementation and customization (1) Difficult when web sites are complex (2) Developer skills required to build robust scripts (3) Simple scripts, easy to customize
Script replay (1) Low level analysis required (2) Depending on used engine is visual replay possible (3) You see what you get
Script maintainability (1) Programming skills required (2) Errors in not rendered sections are tricky to solve (3) Easy because you see failures during replay
Multi Browser Support (1) Some tools emulate web browsers, but this is not comparable (2) Yes, but some elements are often missing (2) Some support other versions and different browsers
Footprint on load injection machine (3) Low, up to 800 sessions per server (2) Medium, up to 8 sessions per server (1) High, up to 6 sessions per server
Recommended for DevOps (2) Depends on actual test scenario (1) No, often complex scripts (3) Yes, easy to use and realistic figures
Recommended for Load Tests (1) No, client-side processing is skipped out (2) Yes, better than HTTP simulation (3) Yes, realistic user simulation
Recommended for Stress Tests (3) Yes, because there is a low overhead on load generator machine (2) No, overhead on load generator machine is too high (1) No, highest overhead on load generator machine
Costs (3) Low (2) Medium (1) High
Total Score 17 19 24

Hopefully, this article on load simulation and performance testing types has given you some additional insight into your performance tests. If you have any questions about running load and stress tests, or the LoadView solution in general, contact our team or sign up for a demo with one of our performance engineers.