Clicky


Common Technical Questions when Load Testing



How do you ensure the cloud machines are not a bottleneck of testing?

When you calibrate a test, we calculate how many virtual users per machine can safely run without over-burdening the CPU of each load injector. This is how we arrive at the number of virtual users per load injector. Depending upon the task type, and whether the tasks use a real browser or not, there can sometimes be a small spike at the beginning of a task as a browser such as IE opens, clears cache and starts the script. Typically the average CPU usage is much lower than 100%, so we do allow you to increase the number of virtual users per machine to get a higher utilization of each machine, but if you change this number we cannot guarantee the cloud machines will not max out CPU usage and become a slight bottleneck of the test.



Where is the load generated from?

LoadView tests run in the cloud. When you set up a load test you can select from a number of cloud based locations from Google and Amazon around the world. As Google and Amazon expand their available locations we will continue to add these options to LoadView.



Can you Load Test complex web applications that require a login or use AJAX, Silverlight or Flash?

You can set up basic single page load tests or complex multistep scripts. Using the EveryStep script recorder you can simply point and click on a website, fill out forms, click buttons and navigate through the website. All user interactions can be recorded, including Ajax, html5, Flash or other Rich Internet Applications (RIAs).



Do you use real or simulated browser?

Both. You have the option of selecting simple GET requests or of using a real browser to actually download, render and interact with your website during a test. The differences may seem subtle, and a full browser may not always be required for every test, but particularly when a website has advanced user interfaces with RIA, rendering the test in a full browser will show a much different result, and be more representative of real users interacting with the website.



What is the User Behavior Delay?

User Behavior is how we define the delay between interactions. If you are trying to simulate real users on a web page, you may want them to pause for a random number of seconds when a user might stop to read text on a page before continuing. Setting this value close to zero may not emulate real browser behavior, but will place much higher load and stress on your system.



How can I simulate my real users?

Using EveryStep script recording tool you can point and click through your website just as a real user would. EveryStep records your actions and allows you to upload those recordings to be replayed by a virtually unlimited number of simultaneous users.



How can I create as much load on a server as possible?

It depends upon what type of load you are talking about. Using an HTTP/s type task you can easily send thousands of simultaneous GET requests to your webserver, and increase the number of simultaneous users as you wish. Setting the User Behavior delay to 0 will run the tasks as fast as possible, rather than introducing a delay to simulate users reading content on the page.



What data do I get back as a result of the test?

After the test runs, the results of every session that was run in the test is available in the test history. Graphs of test response times are generated as well as a variety of reports including total sessions generated, response times by location and by individual task.



What is the maximum number of users can you generate?

Because LoadView runs in the cloud, there is virtually no limit to the number of users that can be generated for a load test. The tests are theoretically limited by the total number of load injectors available from the cloud providers at any given point and also by the cost to lease those servers from providers such as Amazon and Google. Given enough advanced time to setup a test, you should be able to generate as much load as you need to complete a proper load test.



How can I ensure that there is no throttling or resource limitation on load generating side?

When you setup a load test, it is automatically calibrated with a machine in the cloud. This calibration sets the default limits of load that can be generated per load injector so that the load injector will not be a bottleneck.



Setting Up A Load Test


 

When running an HTTP test most platforms will introduce a delay in between checks. Typically, most services introduce a delay of 3 seconds to 10 seconds between requests. If you have a fast HTTP request, 1 user will consume 100% of the CPU to constantly hit that URL. So, to simulate 5000 real “simultaneous” users, the service would need to allocate 5000 CPUs. There is no way to pay for 5000 CPUs from the service provider’s side for the common price of $99 that you see advertised. This would run costs in the thousands from clouds providers like Amazon and Google.

So, in Dotcom-Monitor’s LoadView platform we decided to give the control to the user. We created a process of calibration, where the user can approximately estimate how much one load injector can handle in terms of true simultaneous users. The counter-intuitive thing is, that the slower the site is, the more users could be run on the same machine, because CPU is just waiting for the site to respond. The faster the site is, you need more CPUs to truly generate the load. Also, response time may change with the load. The higher the load, the higher response time, the less CPUs may be needed. So, calibration is not perfect, but gives our users a baseline to work from.

After calibration, when the user runs a test, we provide a CPU load chart. We are the only load testing service on the market that does this from what we know. Our users can review the chart and adjust their next test based on the results of CPU usage from the previous test.

We think of our service as a “platform” where users can configure tests exactly to their specifications, rather than a one-off tool with defaulted and unconfigurable testing parameters.

We also support delays within the testing sequence. If a user wants they can set a delay anywhere from 1 to 60 seconds between each user session ‘hit’.

For example, to run http://stress-test-html-target.s3-website-us-east-1.amazonaws.com/index.html with a 1000 user test, without delay, one would need 250 load injectors with a price of $4 per machine. In contrast, if you add just a 1 second delay between checks, you would only need 2 load injectors to generate the same 1000 users. So, the cost in this example would vary between $1,000 and $8, depending on what our user is trying to accomplish with their test.

This is one of the ways LoadView is set apart from other load testing solutions. Dotcom-Monitor offers a fully customizable service and puts the control in the hands of the user. If you have questions regarding pricing, calibration or HTTP load testing, feel free to contact us and speak to one of our service experts today.