Performance testing is a type of testing to evaluate the speed, responsiveness and stability of an application under load. The goal of the performance testing in Agile testing has been to ensure application behaves well under expected load and it aims to find performance related issues in application. It helps to identify loopholes and bottlenecks which may arise when the page or application is stressed out.
Types of Performance Testing
There are different types of performance testing, which can be employed depending on the projected business need.
Load testing checks the application’s ability to perform under-anticipated user loads. The objective is to identify performance bottlenecks.
Stress tests involve testing an application under extreme workloads to see how it handles high traffic or data processing. The objective is to identify the breaking point of an application.
One to make sure the software can handle the expected load over a long period of time.
Spike testing tests the software’s reaction to sudden large spikes in the generated load.
The objective is to check the application’s performance under varying database volumes.
The objective of scalability testing is to determine the effectiveness in “scaling up” to support an increase in the load. It helps to plan capacity additions to the existing system.
Picking the Right Performance Testing Tool
After we have decided the type of performance testing required for our platform, the most important question arises is which tools to use? There are tons of load test tools available in the market, few are open source and few are closed. But based on the requirement we would be looking below for the major tools which are popular, serve multiple purposes and help us to get the best results out of our performance test.
We have chosen to take a look at Gatling, JMeter and LoadView, which we consider to be the most popular and help us to run load tests for all different sets of applications like websites, internal/external web applications, and APIs.
What is Gatling?
Gatling is an open source load test tool based on Scala, Akka, and Netty. It helps us create and run high performance and easy maintainable load tests on local machines and cloud servers. Gatling can be used with multiple options, one of the major cases we can create an automated simulation for website performance testing, the script can be edited and can be added with multiple parameters, Gatling configuration can be edited with full freedom. It also provides excellent support for the HTTP protocol, which makes it one of the choice to test HTTP servers.
Advantages of Gatling
- Open-source. Gatling is an open source performance testing framework based on Scala, which has human readable DSL, which makes it easier to create and run our tests.
- Code Reusability and Versioning. We can introduce versioning to the Gatling source code, which enhances team collaboration and tracks historical changes easily.
- Multithreading. Gatling uses Netty framework and Akka toolkit which is based on the actor model that is distributed and fully asynchronous by design, which makes sure Gatling doesn’t allocate new thread for each user. It allows simulating a list of user journeys by using a single thread.
- CI/CD Integration. Gatling projects can be built using Maven, Gradle builders, which can be easily integrated into CI/CD pipelines out of box.
- Reporting. It provides easy integration with tools like real time tracking tools like NewRelic, Grafana, Taurus.
- Assertions. It also provides an integrated assertions API, which allows us to run different types of functional checks along with your performance testing.
Disadvantages of Gatling
- Protocol Support. Only supports HTTP, WebSockets, server-side events, and JMS (Java Message Service).
- Lots of Scripting. Gatling is purely code-based, so can be quite a learning curve to someone who isn’t familiar with Gatling.
- Lack of Support Material. Compared to a tool like JMeter, there is a smaller user base and not a lot of support documentation available.
- Test Execution Data. Response times can only be viewed upon test completion, requiring using a third-party solution.
What is JMeter?
JMeter is the Apache Foundation’s Java based open source performance test tool and one of the oldest performance tools. It measures the performance metrics for Web application, FTP application and HTTP back-end servers. JMeter can simulate a heavy load on a server by creating tons of virtual concurrent users to hit the web server.
Advantages of JMeter
- Open-source. JMeter is an open source software and can be downloaded free of cost. It is also a pure Java based application. The developer can use its source code, can modify and customize it as per their requirement.
- User-friendly. It also provides a comprehensive GUI, which helps to create test plans and add test scenarios with few clicks.
- Support. It supports non-functional tests such as Stress testing, Distributed testing and Web service testing.
- Comprehensive Documentation. As JMeter has been around for a long time, there is extensive documentation, which is easy to understand and setup tests from scratch.
- Recording. It allows us to watch and record our actions on browser and based on that create scripts, which can be rerun with a number of concurrent users.
- Reporting. It supports dashboard report generation, a couple of reports are generated which helps to understand execution results.
Disadvantages of JMeter
- Scripting. Requires someone with deep knowledge of JMeter elements and functionality.
- Support for Real Browsers. Does not render web elements in a real browser.
- Executing Large Load Tests. Requires extensive manual work to carry out tests with a large number of users.
What is LoadView?
LoadView is a fully-managed, cloud-based load testing tool that allows users to test websites, web applications, web servers, internal applications, and streaming media from real browsers. It lets us create test scenarios by recording complex web applications on the browser. We can run geo-distributed load tests via LoadView cloud servers with no hardware required from our end.
Advantages of LoadView
- Fully managed cloud testing. While using LoadView, we don’t require servers or machines to generate load tests, it provides full cloud support which saves internal infrastructure support
- EveryStep Web Recorder. It provides functionality to web record complex web application and supports over 40+ desktop/mobile browsers and devices, as well technologies used to create interactive content, such as Flash, HTML5, Java, PHP, Ruby, Silverlight, and more.
- Virtual Load Injectors. It utilizes Amazon Web Services servers located in over 15 zones throughout the United States, Canada, APAC, South America, and Europe, so you can easily carry out tests with hundreds or thousands of users.
- Distribute User Load. It provides access to geo-distributed loads depending on the majority of traffic on our website.
- Summary Report. It provides an extensive detailed overview of execution plans, transaction per minute, average response time and error discovery. It allows us to review the test history and details anytime.
- CI/CD Integration and Easy Proxy Setup. It provides support for easy integration with Jenkins and proxy setup to stress test internal applications which are not public.
- Integrated with Monitoring Platform. After load testing, users can choose to monitor their web pages, services, and applications in production and helps to check health status via easily understandable dashboards.
Below is a chart that outlines the basic feature differences between the tools we discussed. Depending on the use case and environment, all these tools have something to offer users when looking to carry out performance testing.
|Writing test scenarios||Requires knowledge of basic Scala and DSL language understanding.||GUI helps to create thread groups and write easy to use test scenarios.||A user friendly GUI helps to write test scenarios based on requirement.|
|Test runs||Load test can be run from your local machine or dedicated servers.||It requires local machine or dedicated servers for heavy traffic.||No hardware is required as it is browser-based, LoadView provides dedicated cloud server to run tests.|
|Geo-distributed Load Test||Requires setup of AWS or GCP servers on dedicated location to test.||Requires setup of AWS or GCP servers on dedicated location to test.||It provides feature to host load test from the desired location most traffic is expected from website.|
|Documentation and Support||Gatling requires familiarity with DSl and Scala to understand documentation better.||JMeter has the best documentation as it is the oldest and matured tool.||LoadView has an extensive set of documentation and provides 24/7 on call support and live demo.|
|Pricing||Open-source and free||Open-source and free||LoadView charges for the dedicated hosted servers used to run load tests.|
|Integration with CI/CD Jenkins||Yes||Yes||Yes|
Gatling vs. JMeter vs. LoadView: Conclusion
As we have seen on load testing tools, we know that there are a lot of load testing options to choose from. There are free load testing tools, paid load testing tools. With all of these choices, it can be hard to determine the best load testing tool for our specific situation. In this article, we took a look at the features that set each of these tools apart, their benefits, and differences, and explained CI/CD integrations, reporting tool support for each tool. While it’s true that we can use a free load testing tool, and it may be good enough to get the job done, but the benefits LoadView provides are immense. For a deeper dive into other load testing solutions compared to LoadView, please visit our Alternatives page.
Try LoadView for youself today. Sign up for the free trial and receive $20 in load testing credits. Or schedule a live demo with one of our performance engineers for a full walk through of the LoadView platform.