Performance tests mainly verify the speed and reliability of an application and are divided into load tests (goal based) and stress tests. Since the rise of agile development methods, being able to reproduce load testing results has become a top priority.
What Are Reasons for Stress Testing?
The easiest way to configure performance tests is by allowing a number of users to iteratively execute test scripts. This type of test is called a stress test and is often used to identify breaking points in business applications. The drawback is that the actual response time of your application is mainly responsible for the simulated load and you cannot control the actual throughput. In scalability tests, this is not an issue, but for a performance comparison between different releases it can be problematic.
What Are the Reasons for Goal-Based Performance Tests?
The major benefit of this type of test is that it allows speed measurements under realistic, reproducible conditions and throughput boundaries. Goal-based performance tests are often used to validate service level agreements in production-like environments.
Consider the following situation:
Let’s assume that 20 concurrent users will create 2,000 transactions in your new CRM application per hour. You are tasked with creating a performance test which verifies that the response time of eight seconds for this application could be achieved in the next four releases. A stress test will most likely not allow the exact simulation of the expected throughput in your four release performance tests because you cannot assume that the response time will stay the same as in your actual release.
Solution:
- Add ThinkTimes to your scripts
- Find the baseline and single user script runtime to get the session time
- Configure the workload including, max users, goal-based transaction rate, and goal-based transaction time
- Run the goal-based performance test to simulate the expected load
- Review the test report and verify if the application under test was able to handle the load within the agreed response time boundaries
- Repeat this test run in your next four releases and verify if your application was able to hold throughput and response time thresholds
Tips for Configuring the EveryStep Tool
ThinkTime (required)
Create new keywords in The EveryStep Web Recorder (ThinkTimes) or re-use existing keywords
Ensure allowed values are floating points 0.0 – 999.99
Ensure users can add ThinkTimes manually to scripts
Remember that ThinkTimes are waiting times and added by EveryStep Web Recorder automatically during the recording of user actions.
There can be N ThinkTimes in one script
ThinkTimes are ignored in single script test runs
ThinkTimes will be used in Calibration / Get Baseline
ThinkTimes are not part of response time measurements
ThinkTimes are ignored in stress tests
User Concurrency (optional)
New “WaitFor (Number of users)” keyword in the EveryStep Web Recorder
This is a global waiting point which blocks simulated users at a certain point in the scripts until the expected number of users has reached this section of the script
Response Time Thresholds (optional)
New SetBoundary keyword in the EveryStep Web Recorder
Syntax: SetBoundary(Timername, Bound 1, Bound 2)
Baseline/Calibration (required)
LoadView executes a single user test run
ThinkTimes will be used as scripted
LoadView will calculate the session time
Session Time = script execution time + ThinkTime
Configure Workload/Execution Plan (required)
Customer sets ramp up time
Customer sets goal transaction rate
Customer sets goal session time
System calculates number of users
Customer decides whether to calculate response times during ramp up or not
Run Test (required)
LoadView runs the test according to configured workload/execution plan
LoadView collects response times of simulated scripts or transactions
LoadView dynamically adjusts ThinkTime to reach expected throughput, if application under test slows down ThinkTimes are reduced. If ThinkTimes are zero and session time gets > goal session time, an error message will be raised which indicates that the expected throughput could not be reached.
LoadView calculates response times of actual transactions and timers without ThinkTimes.
Tips for Integrating with Dotcom-Monitor
EveryStep Web Recorder
Introduce new ThinkTime keywords
Ignore ThinkTime during single user test runs
Add ThinkTime during script recording
Introduce new WaitFor (Number user) keyword
Introduce new SetBoundary (TimerName, B1, B2) keyword
WaitFor keyword must be added manually to the created scripts
Use SetBoundary keyword
Calibration/Get Baseline
Calculate session time during calibration
Execution Plan/Workload
Option 1:
Add a new workload configuration feature
Replace Execution plan with Workload feature
Create Workload configuration dialog to support stress test, transaction goal and other types
Specify the ramp-up time
Check the box for calculating response times during ramp-up (yes/no)
Option 2:
Use the enhanced execution plan configuration feature
Select test type (stress, goal-based)
Set the transaction goal details
Specify the ramp-up time
Check the box for calculating response times during ramp-up (yes/no)
Run Test
Calculate actual script execution time/session time
Dynamically adjust ThinkTimes based on actual session time
Raise warning if expected throughput could not be reached
Report
Configure a section for response time, actual vs thresholds per timer
Configure a section for throughput, actual vs. expected
What are the User Inputs?
ThinkTimes (Floating point, >0)
Goal Transactions per hour (Integer)
Max number of users (Integer)
Ramp up time (minutes)
Calculate response time during ramp up (Yes / No)
What is “Baseline?”
A single user execution of the device or script. Think times are used in the parameter. The script execution time is calculated and stored as session time. Additional details are also calculated such as required execution resources.
How Can You Dynamically Adjust the Load Test If Transaction Speed Changes on the Target System?
Calculate session time during calibration
Use ThinkTimes to reach the requested goal session time
Recalculate actual session time during test execution
Dynamically adjust ThinkTimes depending on actual session time
Log error message if script execution time is > goal session time
Specify the number of max users in the workload calculation
What is the WaitFor Keyword?
This helps to simulate complex user scenarios such as concurrency situations. It’s very useful if you have to test if certain functionality works correctly if x number of users are accessing a resource at the same time.
What is the SetBoundary Keyword?
SLAs specify response time boundaries for user actions such as searching for a customer. The SetBoundary keyword helps you to verify the actual speed of a certain action or timer. If the allowed boundary is violated an error message appears and will be logged in the test report. SLA verification is much easier with this new SetBoundary keyword
What Should Your Goals Be for Your Load Test?
- 100 percent comparable performance tests across different releases/executions
- Feature for simulation of regular or peak load patterns
- Confidence that the system being tested can handle the expected load within the agreed boundaries
- Focusing performance optimization on user actions that violated agreed boundary
What Type of Reports Should You Configure?
- Create reports that are similar to your current reports
- Avg, Min, Max, Stddev, Percentile response times
- Transactions ok, Transactions failed
- Error rate
- All response times should be without the ThinkTimes
Limitations
High goal session times can lead to session timeouts and false-positives. Consider situations where your web session timeout is very low, such as 10 minutes. If you run a goal-based performance test with a simple script that executes a login and a customer search, you should place the ThinkTime between login and search. In this hypothetical situation, the session time should be 10 seconds the goal session time 700 seconds. In this scenario the ThinkTime will be higher than the session timeout of your application under test. All your simulated transactions will fail because your script will run in the web session timeout and will not be able to perform the customer search.
What Will Happen If We Don’t Reach the Goal?
If the response time of an application under load testing slows down and the session time gets higher than the goal session time the expected transaction rate can’t be reached.
LoadView watches actual session time during test execution and adjusts ThinkTimes in order to reach expected goal-based transaction rate.
LoadView also displays error message in the monitoring screen if session time gets higher than goal session time.
LoadView also continues with the test execution if the goal transaction rate can’t be achieved. In this situation, the test run will be marked with the failed state. The test would report clearly show that expected throughput could not be reached due to slow down in response times and would indicate which devices are affected.
What Does a Sample Goal Based Workload Look Like?
Script / Device | ST (sec)
Not editable |
GST (sec)
User Input |
TPH
User Input |
User
Not editable |
Search_User | 25 | 10 | 500 | 72 |
Inser_User | 25 | 60 | 1000 | 216 |
Ramp up time: 15 minutes
Measure response times during Ramp up: Yes / No
ST: Session Time
GST: Goal Session Time
TPH: Transactions per hour
User: Calculated by LoadView (3600 / TPH ) * GST = 72