Goal-Based Performance Testing with LoadView

Performance tests mainly verify the speed and reliability of an application and are divided into load tests (goal based) and stress tests.  Since the rise of agile development methods, being able to reproduce load testing results has become a top priority.

What Are Reasons for Stress Testing?

The easiest way to configure performance tests is by allowing a number of users to iteratively execute test scripts. This type of test is called a stress test and is often used to identify breaking points in business applications. The drawback is that the actual response time of your application is mainly responsible for the simulated load and you cannot control the actual throughput. In scalability tests, this is not an issue, but for a performance comparison between different releases it can be problematic.

What Are the Reasons for Goal-Based Performance Tests?

The major benefit of this type of test is that it allows speed measurements under realistic, reproducible conditions and throughput boundaries. Goal-based performance tests are often used to validate service level agreements in production-like environments.

Consider the following situation:

Let’s assume that 20 concurrent users will create 2,000 transactions in your new CRM application per hour. You are tasked with creating a performance test which verifies that the response time of eight seconds for this application could be achieved in the next four releases. A stress test will most likely not allow the exact simulation of the expected throughput in your four release performance tests because you cannot assume that the response time will stay the same as in your actual release.

Solution:

  1. Add ThinkTimes to your scripts
  2. Find the baseline and single user script runtime to get the session time
  3. Configure the workload including, max users, goal-based transaction rate, and goal-based transaction time
  4. Run the goal-based performance test to simulate the expected load
  5. Review the test report and verify if the application under test was able to handle the load within the agreed response time boundaries
  6. Repeat this test run in your next four releases and verify if your application was able to hold throughput and response time thresholds

Tips for Configuring the EveryStep Tool

ThinkTime (required)

Create new keywords in The EveryStep Web Recorder (ThinkTimes) or re-use existing keywords

Ensure allowed values are floating points 0.0 – 999.99

Ensure users can add ThinkTimes manually to scripts

Remember that ThinkTimes are waiting times and added by EveryStep Web Recorder automatically during the recording of user actions.

There can be N ThinkTimes in one script

ThinkTimes are ignored in single script test runs

ThinkTimes will be used in Calibration / Get Baseline

ThinkTimes are not part of response time measurements

ThinkTimes are ignored in stress tests

User Concurrency (optional)

New “WaitFor (Number of users)” keyword in the EveryStep Web Recorder

This is a global waiting point which blocks simulated users at a certain point in the scripts until the expected number of users has reached this section of the script

Response Time Thresholds (optional)

New SetBoundary keyword in the EveryStep Web Recorder

Syntax: SetBoundary(Timername, Bound 1, Bound 2)

Baseline/Calibration (required)

LoadView executes a single user test run

ThinkTimes will be used as scripted

LoadView will calculate the session time

Session Time = script execution time + ThinkTime

Configure Workload/Execution Plan (required)

Customer sets ramp up time

Customer sets goal transaction rate

Customer sets goal session time

System calculates number of users

Customer decides whether to calculate response times during ramp up or not

Run Test (required)

LoadView runs the test according to configured workload/execution plan

LoadView collects response times of simulated scripts or transactions

LoadView dynamically adjusts ThinkTime to reach expected throughput, if application under test slows down ThinkTimes are reduced. If ThinkTimes are zero and session time gets > goal session time, an error message will be raised which indicates that the expected throughput could not be reached.

LoadView calculates response times of actual transactions and timers without ThinkTimes.

Tips for Integrating with Dotcom-Monitor

EveryStep Web Recorder

Introduce new ThinkTime keywords

Ignore ThinkTime during single user test runs

Add ThinkTime during script recording

Introduce new WaitFor (Number user) keyword

Introduce new SetBoundary (TimerName, B1, B2) keyword

WaitFor keyword must be added manually to the created scripts

Use SetBoundary keyword

Calibration/Get Baseline

Calculate session time during calibration

Execution Plan/Workload

Option 1:

Add a new workload configuration feature

Replace Execution plan with Workload feature

Create Workload configuration dialog to support stress test, transaction goal and other types

Specify the ramp-up time

Check the box for calculating response times during ramp-up (yes/no)

Option 2:

Use the enhanced execution plan configuration feature

Select test type (stress, goal-based)

Set the transaction goal details

Specify the ramp-up time

Check the box for calculating response times during ramp-up (yes/no)

Run Test

Calculate actual script execution time/session time

Dynamically adjust ThinkTimes based on actual session time

Raise warning if expected throughput could not be reached

Report

Configure a section for response time, actual vs thresholds per timer

Configure a section for throughput, actual vs. expected

What are the User Inputs?

ThinkTimes (Floating point, >0)

Goal Transactions per hour (Integer)

Max number of users (Integer)

Ramp up time (minutes)

Calculate response time during ramp up (Yes / No)

What is “Baseline?”

A single user execution of the device or script. Think times are used in the parameter. The script execution time is calculated and stored as session time. Additional details are also calculated such as required execution resources.

How Can You Dynamically Adjust the Load Test If Transaction Speed Changes on the Target System?

Calculate session time during calibration

Use ThinkTimes to reach the requested goal session time

Recalculate actual session time during test execution

Dynamically adjust ThinkTimes depending on actual session time

Log error message if script execution time is > goal session time

Specify the number of max users in the workload calculation

What is the WaitFor Keyword?

This helps to simulate complex user scenarios such as concurrency situations. It’s very useful if you have to test if certain functionality works correctly if x number of users are accessing a resource at the same time.

What is the SetBoundary Keyword?

SLAs specify response time boundaries for user actions such as searching for a customer. The SetBoundary keyword helps you to verify the actual speed of a certain action or timer. If the allowed boundary is violated an error message appears and will be logged in the test report. SLA verification is much easier with this new SetBoundary keyword

What Should Your Goals Be for Your Load Test?

  • 100 percent comparable performance tests across different releases/executions
  • Feature for simulation of regular or peak load patterns
  • Confidence that the system being tested can handle the expected load within the agreed boundaries
  • Focusing performance optimization on user actions that violated agreed boundary

What Type of Reports Should You Configure?

  • Create reports that are similar to your current reports
  • Avg, Min, Max, Stddev, Percentile response times
  • Transactions ok, Transactions failed
  • Error rate
  • All response times should be without the ThinkTimes

Limitations

High goal session times can lead to session timeouts and false-positives. Consider situations where your web session timeout is very low, such as 10 minutes. If you run a goal-based performance test with a simple script that executes a login and a customer search, you should place the ThinkTime between login and search. In this hypothetical situation, the session time should be 10 seconds the goal session time 700 seconds. In this scenario the ThinkTime will be higher than the session timeout of your application under test. All your simulated transactions will fail because your script will run in the web session timeout and will not be able to perform the customer search.

What Will Happen If We Don’t Reach the Goal?

If the response time of an application under load testing slows down and the session time gets higher than the goal session time the expected transaction rate can’t be reached.

LoadView watches actual session time during test execution and adjusts ThinkTimes in order to reach expected goal-based transaction rate.

LoadView also displays error message in the monitoring screen if session time gets higher than goal session time.

LoadView also continues with the test execution if the goal transaction rate can’t be achieved. In this situation, the test run will be marked with the failed state. The test would report clearly show that expected throughput could not be reached due to slow down in response times and would indicate which devices are affected.

What Does a Sample Goal Based Workload Look Like?

Script / Device ST (sec)

Not editable

GST (sec)

User Input

TPH

User Input

User

Not editable

Search_User 25 10 500 72
Inser_User 25 60 1000 216

Ramp up time:                                               15 minutes

Measure response times during Ramp up: Yes / No
ST:       Session Time

GST:    Goal Session Time

TPH:    Transactions per hour

User:   Calculated by LoadView (3600 / TPH ) * GST = 72