Clicky

 

Learn about Load Testing

Load Simulation & Performance Test Types

 



Slow loading or non-responsive web pages have an impact on financial revenue because frustrated users will most likely not return once the issue has been resolved.  Therefore, performance testing has become a fundamental part of the development chain and the demand is still growing.

Performance testing platforms provide a broad range of load simulation methods such as HTTP, headless, and real browser-based. In this paper, we will outline the main aspects of those, followed by a comparison matrix, which you can use for choosing an appropriate simulation approach.

HTTP-Based Load Simulation

In the early days of our digital age, HTTP-based testing was very popular. With the rise of rich web client technology, this approach has become increasingly outdated. A typical HTTP-based test driver executes service requests and parses responses. Modern web 2.0 applications consist of many client-side scripts, which are totally ignored and not measured, in this type of test execution. In worst case complex use cases cannot be simulated on protocol level due to a shortage of client-side generated ids.

Due to their request and response based nature the overhead on the load injection machine is very low. A typical load test server can simulate up to 800 simultaneous sessions. Complex protocol based use cases can be difficult to implement. A performance engineer needs to deal with cookies, session IDs, and other dynamic parameters. Depending on the used technology of the system being tested, some web form names often change once a new version have been deployed which will cause the HTTP-based script to fail.

The sample script below is a showcase for the very technical nature of those scripts. If a technical attribute of your application changes, you have to rewrite the whole script.


//sample protocol level script
transaction TMain
var
hContext: number;
begin
WebPageUrl("http://lab3/st/", "Greetings");
WebPageStoreContext(hContext);
WebPageLink("Join the experience!", " - New Visitor");
WebPageSubmit("Continue", CONTINUE001, "Main menu");
WebPageLink("Products", "ShopIt - Products");
WebPageLink(NULL, "ShopIt - Product", 3);
WebPageSubmit("Search", SEARCH001, " - Search", 0, NULL, hContext);
end TMain;

dclform
CONTINUE001:
“name”            := “jack”,
“New-Name-Button” := “Continue”;

SEARCH001:
“search”          := “boot”;

After all, protocol level scripts are good for component level tests in continuous integration environments and the perfect setting for stress test due to their low footprint on load injection machines.

Headless Browser-Based Load Simulation
With the rise of web 2.0 technologies the testing business was faced with serious challenges. Rich browser applications could no longer be tested or simulated on the protocol level due to the missing client-side logic during script replay.  Therefore, several headless browsers have been introduced such as HtmlUnit, PhantomJS or SlimerJS. They are often built on top of WebKit, the engine behind Chrome and Safari.

Headless Browsers have all the advantages of real Browsers, and they run faster without the heavy GUI. Many test-automation and performance testing platforms are using headless browsers as they allow realistic user simulation.

Some tool providers have built their headless browser engines and have run into maintenance pitfalls because they have to keep the pace with new browser versions. It’s highly recommended using free available headless browser kit because there is a broad community which works on improvements.

The simulation of client-side rendering is not for free. A typical load injection server can simulate up to 8 simultaneous headless browser sessions.

Test script implementation and customization is not too difficult. If you have basic developer skills you will be able to create simple scripts. Not all headless-browsers provide visual replay features and without visual replay script debugging and error analysis can become very tricky.

In the sample script below a simple post request is executed. You need Java programming skills to customize such test scripts.


//sample phantomjs script
"use strict";
var page = require('webpage').create(),
server = 'http://posttestserver.com/post.php?dump',
data = 'universe=expanding&answer=42';

page.open(server, ‘post’, data, function (status) {
if (status !== ‘success’) {
console.log(‘Unable to post!’);
} else {
console.log(page.content);
}

phantom.exit();
});

Considering this, headless browsers are the way to go if you have programming skills and you use a solution, which allows visual script replay.

Real Browser-Based Load Simulation

Web 2.0 applications are full of JavaScript, Flash, Ajax and CSS. Without a full browser, it’s not possible to measure the actual end-to-end response times of the whole web page. Real browser-based performance testing allows you to verify the site’s functionality and speed as perceived by the end-user.

A typical real browser performance test solution collects loading times of images, JavaScript, CSS and more. Often, they provide waterfall charts, which visualize the load time of those components.

The footprint of a real browser-based browser is slightly higher. Because headless browser simulation does not deliver 100% realistic response times it’s fair to say that real browser based simulation should be the preferred method for testing.

Implementation and maintenance of test scripts is easy because user actions are directly reflected and the visual replay makes debugging easy. In the sample script below the browser opens a URL, inserts user and password and clicks the login button.


//sample real browser-based script
transaction TMain
begin
BrowserStart(BROWSER_MODE_DEFAULT, 800, 600);

//navigate to the login site
BrowserNavigate(“http://demo.com/TestSite/LoginForm.html”);
//set the authentication for the secure site
BrowserSetText(“//INPUT[@name=’user’]”, “User1”);
BrowserSetPassword(“//INPUT[@name=’pwd’]”,     “Password1”);
//Login
BrowserClick(“//INPUT[@value=’Login’]”, BUTTON_Left);
end TMain;

After all, real browser simulation is useful for realistic end-to-end load tests. However, don’t use it for stress testing because the footprint on the load injection server is too high.

Different Types of Load and Performance Tests

Component Speed Tests

In recent years software development methods have moved in the agile direction. Short release sprints are essential. Developers and test engineers automate their quality assurance and performance checks. Typically, they implement service based performance tests on protocol level or they simulate real browser based performance checks to compare end-to-end response times with agreed performance boundaries.

Objectives

  • Repeatability
  • Automated interface and end-to-end performance checks
  • Comparing response times with agreed thresholds

Load Tests

For several reasons are load tests the ideal setting when it comes to verification of non-functional requirements. One reason is that the response times can be verified under reproducible conditions. Another being that those tests allow verification of response time thresholds. Realistic response time measurement is essential in load test scenarios. Therefore, test engineers use headless or real browser based user simulation for their load test settings.

Objectives

  • Reproducible load simulation
  • Verification of response time thresholds
  • Identifying bottlenecks under production like load conditions
  • Creating realistic end-to-end test scenarios

Stress Testing

If you have to proof the reliability of your application under peak load conditions, run a stress test to determine the breaking point of your system.

Objectives

  • Prove scalability and stability
  • Simulate peak load conditions
  • Reproducibility is not important

Comparison

Obviously, there are good reasons for protocol, headless or real browser based user simulation. The matrix below provides some guidance to choose the appropriate approach.

Criteria HTTP Headless Browser Real Browser 
User simulation (1) No client-side rendering (2) Some client-side elements are simulated (3) Real user simulation
Script implementation and customization (1) Difficult when web sites are complex (2) Developer skills required to build robust scripts (3) Simple scripts, easy to customize
Script replay (1) Low level analysis required (2) Depending on used engine is visual replay possible (3) You see what you get
Script maintainability (1) Programming skills required (2) Errors in not rendered sections are tricky to solve (3) Easy because you see failures during replay
Multi Browser Support (1) Some tools emulate web browsers, but this is not comparable (2) Yes, but some elements are often missing (2) Some support other versions and different browsers
Footprint on load injection machine (3) Low, up to 800 sessions per server (2) Medium, up to 8 sessions per server (1) High, up to 6 sessions per server
Recommended for DevOps (2) Depends on actual test scenario (1) No, often complex scripts (3) Yes, easy to use and realistic figures
Recommended for Load Tests (1) No, client-side processing is skipped out (2) Yes, better than HTTP simulation (3) Yes, realistic user simulation
Recommended for Stress Tests (3) Yes, because there is a low overhead on load generator machine (2) No, overhead on load generator machine is too high (1) No, highest overhead on load generator machine
Costs (3) Low (2) Medium (1) High
Total Score 17 19 24

 


Login