How to start with regression testing
This was originally published on the Clarus website.
To help engage my audience, I first asked, “Who has a suite of automated regression tests?” One developer sheepishly put his hand up and indicated (to the surprise of his colleagues) that he had a small regression pack. I then asked, “Who has a set of regression tests – manual or otherwise?" Nobody raised their hands.
I'd like to say I'm surprised, but the truth is a lot of companies have never invested the time and/or money into developing a regression test suite.
There are pressures when developing a new product – attempting to be the first to market, having a limited development budget and the pressure to develop a ‘well-rounded offering’. This means testing is often squeezed out. Then, later in the product’s lifecycle, quality becomes a more important factor. Often, companies then have to rapidly develop a testing solution.
This led to “Where do I start?” Having only limited time, I couldn’t answer in detail so I’ll do it here.
In the real world, companies need to continue developing new features, bring on new clients and sell the product. ‘Taking time out’ to develop tests has to be balanced against the financial needs (and in some cases survival needs) of the company.
In an ideal world, if faced with the situation these companies are in, I would immediately stop developing new features and focus on building a regression pack.
The lack of a regression pack means that a company cannot be sure of the quality of each release. New features may break existing functionality. Potentially, each release would be accumulating additional defects, additional technical debt and adding to development and support costs. Therefore, the cost of building a regression pack could have its ROI calculated against these factors.
Ken Schwaber raises some interesting points about the cost of a lack of investment in quality:
- For every dollar of competitive advantage gained by cutting quality, it costs $4 to restore it
- Software is an organisational asset and decisions to cut quality must be made by executive management and reflected in the financial statements
In typical Schwaber style, it’s put bluntly but is very true.
‘Where to start?’ breaks into two relevant questions – ‘What to start testing?’ and ‘How to regression test?’
What to start testing?
With an existing product, no regression test pack and the need to continue producing functionality, one obvious approach is that with each piece of new functionality, you would start adding tests to cover that functionality. Assuming that the new functionality replaces existing functionality or at least that when add adding new functionality you would re-work a chunk of the old functionality. This approach would eventually lead to reasonable coverage – but, in my experience, you're not that lucky! Applications tend to grow in size instead, re-using rather than re-factoring existing code.
In order to get reasonable coverage, there is the need to add tests to areas of code not currently in development. Time (money) devoted to testing is limited, therefore it makes sense to spend this as prudently as possible. This leads onto the question ‘How do you select areas of existing code to create tests for?’
The simple answer would be to test the parts of the product that cause the most ‘pain’. Every time the development team fixes a defect, they should also create a test that captures that ‘class’ of defect. If it becomes obvious that an area of the product has a ‘cluster of defects’, then set aside additional effort to create a set of tests for this area. A more complicated answer would be to perform multi-criteria analysis of previous defects, code complexity, usage patterns, risk, age of code, development teams’ ‘gut feel’ and other factors to identify where tests would be most beneficial.
How to regression test?
A regression test is any test that is run to prove existing functionality has not been compromised. By its nature, a regression test will be run multiple times and should have behaviour that is well understood. This means regression tests are prime candidates for automation.
Often, when I'm asked ‘How do we regression test?’, the company is really asking me how we automate. While automation offers good ROI when compared with manual testing, it comes with substantial risk. Software automation projects often fail! When starting automation, the key messages are to start small and to choose which tests to automate wisely. Fortunately, it is often the ‘simple’ and repeated tests that will yield the best ROI.
When introducing automation, start with a small pilot project in a well-understood part of the product. Start by automating simple tests. Repetitive actions that take a long time to manually perform are good candidates for automation. Be prepared to throw away any and all the scripts written during the pilot project. The goals of the pilot project are to identify a set of tools that work with the application and for the team and to demonstrate the ROI to management/Product Owner.
Establishing regression testing is not an overnight activity but ,once it’s done, anyone undertaking it will be in a position to develop their products faster and with fewer overheads than before – and therefore out-manoeuvre their competition. Hard work, but it’s important to make a start.