In this ten part series we will be reviewing and offering solutions to the most common complaints from development to QA. We will provide real world examples from QA experts to illustrate the root cause of the challenges, and the real world techniques that are used that can help you avoid these pitfalls in your day to day efforts. Hopefully if you can employ these suggestions you’ll reduce the churn between teams and demonstrate a more collaborative path forward is possible with the right implementation.
Nothing frustrates an engineering team more than seeing the same issues over and over again, across builds, sprints and releases. Looking at the cause can uncover some understandable origins which need different solutions to solve the systemic problem.
As a product owner I get as frustrated as developers when encountering duplicate bugs. They bog down my sprint review and planning meetings and can even cause fights between development and QA in the middle of what should be a productive meeting.
– Product Manager
Causes of Duplicate Bugs in QA Testing
Lack of Centralized Bug Reporting and Tracking
The first challenge may be simply that multiple testers are reporting bugs separately and do not know when a bug has already been filed. Some teams even lack access to a central defect tracking tool such as JIRA, Bugzilla, Fogbugz, etc. In this case testers sometimes have to resort to reporting bugs via excel spreadsheets or google docs, which is in turn given to engineering later. Sometimes the QA lead will collate the separate bug reports and remove redundant bugs; however, duplicate bugs still slip through. This is surprisingly common, particularly with external QA teams.
Solution: Use an Integrated Test Case Management Tool
Integrate the bug tracking tool with a centralized test case management tool. Not all test management tools are created equally. To really avoid duplicate bugs you will want to review how the testing tool links the bugs to the test cases and vice versa. It is important that from the testing tool users are able to see which test cases have bugs associated with them. Many tools also provide a link to the bug so testers can review it in the bug tracker. QMetry Test Manager goes one step further by showing bugs that are associated with a test case before a tester can add a new bug so the tester can link an applicable existing bug before filing a new one.
For added security and improved workflow for enterprise users the Test Management tools should provide permissions to limit what team members can do both within the testing tool, and through integration with the bug tracking tool. For instance, QMetry provides a full set of permissions for test management, while maintaining only the defect tracking rights given to the QA testers by the defect tracker. This can provide full or very limited rights to one or the other system for both testers and developers.
Challenges Caused By Large Distributed QA Testing Teams
Even with a centralized bug tracking tool duplicate bugs are still far too common in large and distributed teams. This is often because multiple testers are often required to test the same or similar functions. A test case is often run through many different test scenarios, and against many different hardware and or software configurations. For instance an online retailer will want to test the checkout process with many different shopping cart statuses, payment methods, and user types, and against multiple different browsers. Add mobile checkout to the list and you have many permutations of the same basic test. It is often very difficult for one person to do all the testing, but if two people are testing the same function then how do they know when a bug has already been filed?
Solution: Improved Communication
I like how QMetry shows testers how which bugs are already associated with a test case before they can add a new test case. I can quickly link an existing bug and update the details rather than confusing the process- and irritating my developer- by adding a duplicate bug.
– QA Manager
The “simple” solution is use an agreed upon naming and tagging process to categorize the bugs and then have everyone search for the existing bugs. This sounds good in theory, but in practice is very difficult. Not only is there a productivity loss as users search for bugs, but it is still easy for things to slip through.
Over the year of working with and managing distributed QA teams we decided it was very important to improve the communication between team members to help avoid duplicate bugs. This is why the QMetry UX was designed to always show a tester what bugs are already associated with a test case before they can create a new issue. If the bug already exists the user can just link it. This not only reduces the time of filing the bug, but can also improve the bug reporting since the bug can be updated with the information from the new test. For instance a browser based bug that was previously reported on Firefox could be updated when it is found also on Chrome.
Bug Count as a Measure of Productivity and Value
Many QA teams measure productivity in terms of issues (bugs) found. This can lead to individual team members to feel compelled to file as many issues as they can to show they’re working hard and making progress throughout their testing. Even if this doesn’t describe your group’s mentality, it’s human nature to want to feel like you’ve accomplished something new each day even if what you’re seeing has been found in other environments or across other builds. If this is a driving force behind our assessment of personal value than we should take stock and consider the impact on our counterparts who, for most engineers, would prefer to be working on new areas of the product, coding new features and functions, not drowning in a sea of duplicate issues. Don’t get me wrong – unique issues are measurements of progress and value for any QA team, but a daily bug count, no matter the quality does more harm than it does good.
Prior to the launch of major release of a new product we decided to give it to another division’s QA group for testing. Since we wanted an independent assessment we didn’t monitor the format of the testing- boy was that a mistake!
The QA group held a “bugathon” in which all QA testers conducted ad hoc / exploratory testing and awards were given out for the most bugs. In the end over 500 bugs were filed in just a few hours. The majority were labelled using the default “Major” status. The CEO heard about the results and the hammer came down hard on development to fix the issues. It took multiple team members days to sift through all of the bugs only to find many bad bugs, and the majority were minor or trivial duplicates. The few truly new and major bugs were fixed, but the delay in finding them, and the aggravation from development soured the relationship with QA for months.
– Anonymous, Director of Quality Assurance
In my over ten years of working in and around QA with a wide range of projects I’ve seen some QA managers making the decision to have their testers write up everything they find as a new issue and assign it to either development or Product Management for prioritization. You can think of this as the “Bug everything and let God sort it out” methodology. Clearly, this can come across as a very hostile stance to take since it pushes additional tasks onto the other groups in the organization. Every QA group needs to have their own way to report on progress; however, the QA group’s objectives should be to show their value to the company without compromising the integrity of their partner groups.
Solution: Focus on Quality Testing And Reporting
The “simple” solution is to “Focus on quality not quantity”. Easy to say, but sometimes hard to do. As we mentioned, test engineers want to show they have accomplished something. After all, testers don’t get credit for not filing a bug- no matter how high quality their work is. This means that you need a system that makes linking a bug as easy as creating a new one, and a system that reports the number of linked bugs not just the quantity of bugs.
“If a bug is not unique to what’s already in the system, it should be linked to the existing bug.”
If your organization changes the focus to linking bugs rather than “bug everything” then you need to make sure reporting is also adjusted to show the link as equal to the creation of a new issue. The added advantage is that a single bug that is linked to multiple test cases and/or executions can be easily shown to be of elevated importance. In addition, maybe providing an update with the platform information and a log file for your configuration is typically good enough to help communicate to Engineering that the issue is reproducible across environments or releases and still needs to be addressed without adding to the ever growing list of issues to triage and fix.
The solution that QMetry provides is the ability to link existing bugs to new test runs. This doesn’t result in duplicate bugs that the developers will get upset with, but simply provides more details on what environments a bug fails in. If the bug already exists the user can quickly link it to the new test run, and if needed add additional details to the existing bug. The added details should actually help development resolve the issue faster rather than distract them. In this case both the developer and tester saves time and productivity increases.
Reports are available to show either the number of linked bugs, or the number of unique bugs depending on what you are looking for. Reports can also be filtered and sorted by any of the data fields making it easy to find bugs.
Search and Destroy
Encourage your team to practice better search (see: Research!) and discovery efforts before filing new issues. This can sometimes take a bit more time on the front end; however, in the long run the extra research prior to filing a bug reduces your work, your team’s workload, and ultimately time to market.
If you are working on a small team and have the luxury of working closely with your fellow test engineers, take the time to discuss an issue with the original reporter to make sure you’ve got something unique on your hands, or maybe their issue just needs a few more details and you found an identical bug. Many test management tools can help by proactively listing out similar issues or issues found in the same test section or test module, which can significantly cut down on the dreaded “dupes.”
Duplicate bugs is a deceptively difficult problem for many QA organizations. Too often duplicate issues distract development and clutter up the reporting. Unfortunately many team members downplay the problem by looking at each case individually rather looking at the cost of the problem over time. Using a test management tool and a set of simple QA best practices can reduce the problem and improve overall QA productivity.
Thanks for reading and keep an eye out for Part 2 where we will cover Critical Issues Found Late in Development. In that article we will uncover why many QA groups do not fin show-stopping issues until late in development and what QA best practices can be used to prevent that from happening.