What is Shift Left Testing? And What Does It Mean for DevOps?
The shift left testing methodology is an approach to software testing that brings testing requirements earlier into the software development cycle.
The software development has seen quite the evolution over the last few decades. Today, performance engineers and testers can integrate any number of automation, machine learning, and AI tools into their testing. It wasn’t really that long ago when software testing was a side note. In fact, early software development didn’t even have a formal testing phase. The introduction of testing didn’t come about until software became more complex, where bugs and defects from the production environment began impacting the business. Developers carried out the testing themselves, not a separate team. This is how functional testing came into existence and became a part of the traditional waterfall model, consisting of separate stages (Analysis, Requirements, Design, Coding, Testing, Acceptance, Production, and Post-Production) that moved through a linear path, similar to an assembly line in a factory.
Waterfall Model by Paul Hoadley. Used under the Creative Commons License.
Once each stage was complete, it passed onto the next group and moved down the line. It looked good on paper, but QA didn’t test the code until after most of these stages were complete. Consequently, this meant that there was very little time left to make modifications to the code before production. If the issue or bug was severe enough, that meant scrapping or delaying the project. Moreover, this meant a potential huge loss for companies, depending on the significance of the software application.
Shift Left Testing – Laying the Foundation
Fortunately, the costly lessons learned from prior development mistakes helped usher in the shift left method. Organizations realized that identifying issues late in the game was not only too costly, but risky as well. A TechRepublic article from May 2019 reported that the average yearly cost to fix visual bugs in software is over $400,000. Not exactly chump change.
In addition, the industry as a whole was changing. There was a digital transformation that businesses had to contend with. Organizations were beginning to realize that they couldn’t rely on one to two release cycles a year anymore. Customers’ expectations and demands were high. Releasing a poorly designed product meant losing customers to the competition. Something had to change.
Organizations began breaking down the development cycle into smaller, manageable blocks and incorporating testing into each phase. This allowed for a more collaborative environment, giving teams the ability to identify issues sooner. For example, by better understanding the software requirements earlier in the process, testing teams could assist with developing better test cases to fix any potential bugs. This is also known as ‘failing early and often’ or ‘failing fast.’ Consideration for user scenarios and software behavior earlier in the process eliminates potential bugs and optimizes code. This allows for teams to deliver a consistent product.
Obviously, there are going to be times where testing just doesn’t make sense. For example, you can’t carry out GUI testing until you have a GUI. However, shift left is more of a mindset, not just a development process. Most importantly, teams should always be thinking about what would make a better user experience. Ultimately, it’s the user that’s going to have the end product in their hands. Anything that can improve an application before it’s pushed into production benefits everyone.
The Rise of Shift Left Testing and Agile and DevOps
Due to the rise in technological advances and focus on the digital experience, a paradigm shift occurred. Development and testing cycles became shorter and more frequent. This resulted in new features being brought to the market sooner. Most importantly, not only did this allow companies to stay competitive, it kept customers happy and engaged. For instance, mobile and web applications typically work within two-week release cycles. Some organizations release updates daily – or even hourly!
The focus for application and software development is to be fast, agile, and reduce risk. Organizations are meeting this challenge through Agile software development and DevOps practices. Agile software development is similar to the waterfall model; however, the key difference is that in a waterfall model, there is always a testing phase after the design phase. The Agile method breaks development into small iterations, called sprints, that last no longer that four weeks. Each sprint involves cross-functional team members working on all aspects of sprint, with testing completed in every iteration. This allows for more collaboration among team members, shorter feedback cycles, and a higher quality product.
And that’s another important factor about shift left testing. In traditional waterfall testing, responsibility for testing and managing the quality of the product rests on the QA team. In shift left testing and Agile environments, developers and testers carry out the actual testing, but the responsibility ultimately falls on everyone because of its collaborative, cross-functional approach during development. Today, there are four different approaches to shift left testing: Traditional, Incremental, Agile/DevOps, and Model-based shift left testing.
Types of Shift Left Testing
The traditional shift left testing, which most people think of, moves testing lower and slightly left of the classic V-model.
Traditional Shift Left Testing by Don Firesmith. Used under the Creative Commons License.
Incremental Shift Left Testing by Don Firesmith. Used under the Creative Commons License.
Incremental shift left testing is ideal for large, complex projects with hardware dependencies. By testing incrementally, it ensures each segment of the system is functional before moving forward to the next step.
Agile/DevOps shift left testing completed in short, iterative sprints and typically carried out in development testing, not development testing. That occurs once the system is placed into operation.
Agile/DevOps Shift Left Testing by Don Firesmith. Used under the Creative Commons License.
Model-based Shift Left Testing by Don Firesmith. Used under the Creative Commons License.
Model-based shift left testing sets out to resolve defects in the requirements stage, where the majority of defects are introduced. The previous shift left methods above begin testing in the development cycle. This allows testing to be completed as early as possible.
In recent years, the term DevOps has become more of a buzzword for marketing, software products, and even job descriptions. Simply put, DevOps is an alternate framework within the Agile approach that aims to bring together Development and Operations teams. Historically, each department had their respective, sometimes contradictory, goals. Developers create, design, and innovate. Operations teams focus on the infrastructure and “keeping the lights on” with ongoing monitoring to ensure availability. Similarly, DevOps was specifically created to bring about more collaboration, feedback, and agility between these departments. DevOps is also referred to when talking about CI/CD (Continuous Integration and Continuous Deployment) and automation, but the fact that an organization uses CI/CD and automation, does not automatically make them DevOps certified.
Load Testing Best Practices for DevOps Environments
If your teams follow an outdated load testing and monitoring approach, DevOps can quickly push you into a performance disaster. QA teams have to deal with frequent new release deployments. DevOps teams need an easy way to manage their tests, in addition to flexible monitoring, to provide adequate insights.
Continuous load testing.
There are too many changes that can affect the UI and crash your load test. The initial focus should be to execute fully automated API testing and schedule it to run on a daily basis. Once key business processes are stable, you can add additional end-to-end tests to your testing scenario
Share valuable insights.
Involve your DevOps team in result analysis activities. Don’t practice information hiding. Share all load testing (including Selenium load testing) and monitoring results with your engineers to nail down the cause of all issues.
Monitoring of all layers.
Shift left also means to have production-like monitoring on Development and QA stages. You won’t have time to continuously reproduce errors in agile development pipelines. Make sure that you collect front-end, back-end, and infrastructure utilization metrics 24/7 for non-production stages.
Baseline and benchmark.
Continuous load tests produce tons of data. Set baselines and use benchmarks to detect deviations early in the cycle. The earlier you come to know about slowdowns, the more time your Development team will have to fix such issues.
Integrating Performance Testing into the Shift Left Methodology
Today’s applications utilize multiple technologies, relying on vast networks of third-party providers and CDN’s. Add to the fact that users can access your applications from anywhere in the world, from any number of devices, all with varying connection speeds. That’s a lot to account for when trying to give users a great experience all the time. Response times, quality, and availability are critical factors that need attention before pushing applications into production.
Once in production, your application or site will need to stand up to hundreds, or even thousands of concurrent users. Small, incremental changes to code can affect performance, so the sooner you can find performance related bugs, the easier and less costly it will be to fix them. In most cases, teams should be able to fix any performance issues within a day or two. Again, it’s much easier to carry out those improvements then, versus discovering them pre-deployment.
Ideally, once code has gone through the necessary functional tests, with features vetted and signed-off on, teams should execute non-functional tests, such as stress and load testing, to see how well the functional test artifacts stand up to virtual users. LoadView plays an integral part of the shift left testing approach, allowing users to time, money, and deliver optimized code and applications.
The LoadView Platform: Cloud-based Load Testing in Real Browsers
The LoadView platform is a flexible load testing platform that can address the issue of ineffective load patterns, simulating everything from protocol-based tests to real browser-based tests. It is completely cloud-based, so there’s no need to setup and deploy any internal load injectors, manage third-party cloud accounts, or worry about hardware or software requirements. Performance testing usually requires additional infrastructure and resources that some organizations may not be able to support. LoadView manages this for you through the platform.
LoadView is ideal for testing code or web services early to help to benchmark performance characteristics, as it can easy spin up and simulate high levels of load on the backend from a single load injector, saving you time and money compared to other tools. This makes it ideal for testing Web API architectures like JSON, SOAP, and REST. In addition, non-functional tests typically require longer setup times and complex scripts that require developers and engineers to have working knowledge of specific programming languages. This can sometimes be difficult to automate as they tend to only work in a vendor-specific ecosystem. This is not the case with LoadView.
Scripting Made Easy: The EveryStep Web Recorder
LoadView utilizes an easy-to-use script recorder called the EveryStep Web Recorder. Users can easily create advanced scripting actions that simulate real user actions with your APIs, web applications, and websites. To validate the end-to-end response times for client-side applications, users can simply navigate through their test case and record every action. Understanding how much load an API or web-app can handle during the early development phase can assist developers in these critical areas:
- Determining response time baselines under specific user load numbers
- Identifying performance bottlenecks
- Finding upper limits of your current systems for capacity planning
- Analyzing server performance (CPU, memory, bandwidth, disk I/O) and database response times
LoadView: Flexibility to Test Real-World Scenarios
LoadView gives users the option to distribute user load between geographic locations based upon the percent of traffic to your site. For instance, if you know that a certain percentage of your customers and users come from a specific geographic location, you can select those specific areas to test. The platform utilizes Amazon Web Services (AWS) and Azure Cloud Services to generate virtual users. Furthermore, users can take their load testing one step further by customizing the type of load curve. This provides even greater flexibility for test engineers, depending on their unique scenario. LoadView users can choose from three different load curves:
Load Step Curve
- Begins with a pre-determined number of concurrent users and gradually increases the load to see how your website can handle a spike in traffic and compare it to expected traffic.
- This type of load curve is useful when you have already determined your throughput goal or number of visitors to your site during a fixed time interval. Great for validating SLA or non-functional requirements.
Dynamic Adjustable Curve
- This test allows users to adjust load while the test is running to uncover the performance limits of websites or server capacity.
After the tests have finished, LoadView automatically generates reports which assist teams in measuring the success of their KPIs (Key Performance Indicators), such as number of concurrent users, errors, average response times, CPU usage, throughput, latency, and more. These metrics are critical for developers, DevOps, and QA teams as they aid in uncovering real-world bottlenecks that could potentially affect end user performance.
After Shifting Left, Don’t Forget to Shift Right
With all the focus on shift left testing, it’s hard to remember that there’s another extremely important step in the process that gets less attention. After your application goes into production, you have to ensure that everything continues to run smoothly for users. Dotcom-Monitor offers multiple monitoring solutions to ensure your pages and applications continue to perform and function properly.
Web Services Monitoring
- Uptime and performance of web services, such as HTTP/S, SOAP/REST, and more
Web Page Monitoring
- Web page performance in real browsers, identifying slowest and fastest elements over time
Web Application Monitoring
- Monitor complex web applications, like AJAX, PHP, Ruby, Flash, and more
- Functionality and performance of Internet Servers and Protocols, such as HTTP/S, email, streaming media, VoIP, and more
These platforms allow users to set up ongoing monitoring based upon custom thresholds and can alert specific individuals or teams when errors occur so they can work on fixing issues before potentially impacting many more users.
Shift Left Testing: Good for Business and Peace of Mind
In conclusion, both the shift left and shift right concepts can be extremely valuable, not only just within the software development cycle, but within other departments and industries as well. For instance, Product Managers or Customer Experience Managers are ‘shifting left’ and are becoming more frequently engaged with actual customers to get continuous feedback. This allows organizations to become more agile and get closer to the source of information and feedback to better understand their customers. Just think about it. Wouldn’t you be more likely to work with someone, or continue to do business with a company that values your input? So, now when you hear someone say, “shift left” or “test early and often,” it’s not just a fancy buzzword that they’re throwing around. It’s a critical piece of the customer experience puzzle and one that you should always keep in the back of your mind. Not only will your users and customers be happy, you’ll gain efficiencies, achieve better results, and give you and your organization peace of mind.