QMetry Test Manager (QTM) allows QA organizations to manage and execute test cases in a variety of ways. We have created a couple of videos that highlight some of the main features related to executing Test Cases. You can read the below description of the user stories used in the video, or go straight to the videos at the bottom of the article.
The first video provides an introduction on how to do ad hoc testing of an individual test case, and how to setup formal test suites to validate user stories, or provide regression testing. While QTM allows users to get started fast by executing individual test cases to record exploratory testing, we recommend most testing to be done through a test suite.
User Story : As a tester, I want to be able to record my ad hoc testing so that I can record my exploratory testing and log bugs against the requirements.
The second video covers how executed against different builds to allow for continuous testing, bulk updating test executions / runs, and some basic reporting.
Organizing Test Cases For Testing Purposes
A test suite is a group of test cases that should be executed together. This allows a group of functional and UI tests to be organized to test a specific user story or workflow. Not only is it important to group test cases together to ensure that you fully validate a user story, but you often need to ensure that you execute the test cases in the proper order. QTM allows users to define an order for the test cases in a test suite so that you can create a logical flow.
User Story : As a tester I want to be able to organize my test cases into a logical user story in order to properly validate my requirements and report on my test coverage.
Using Test Suites as Templates for Further Testing
When you have your test cases in the order you want, you can copy the Test Suite. While the copied test suite is independent of the original, all of the test cases are not. They are linked to the test suite so that if changes are made to the test case the latest versions can be updated.
Note: QTM provides multiple options for test case version control including not updating a test suite if desired. This allows users to use older versions of a test case if needed.
Once a test suite has been copied you can choose to add or remove test cases and change the order of test cases. This allows you to create templates that can be reused across multiple sprints, releases, and editions of your product.
User Story : As a tester I want to be able to copy my past test suites so that I can use them as templates and save time on creating quality regression tests. User Story : As a QA Manager I want my testers to use the same test cases (e.g. not copied test cases) so that I can see the history of my common test cases, find patterns of regression, and prevent the filing of duplicate bugs.
Recording Environmental Factors
In addition to test cases we recommend you define a platform to execute the test suite. A platform can be many things from an operating system, browser, or device to a complex environment with multiple variations or configurations. For instance when testing QMetry we have platforms with a mix of browsers and operating systems. Mobile QA groups may define actual devices. Hardware OEMs usually have to deal with more complexity with hardware, software, and configuration settings. You can also choose to use “No Platform” if you are always testing in an unchanging environment and don’t need to report test results.
User Story : As a QA Manager I need to use platforms so that I can report on the test coverage of my product in a variety of environments in the field and avoid embarrassing failures due to coverage gaps.
If you are using the Enterprise addition you can actually enter run-time attributes like CPU utilization or applications. We will cover platforms in another demo so for now let’s move on to the main execution screen.
User Story : As a QA Manager I want to be able to record important environmental factors that might effect test results at run-time so that I can filter my reporting.
Highlights of The Execution Screen
The QTM execution screen looks similar to a spreadsheet. If you have a background in testing using Excel then the execution screen should look familiar; however, there are a few key advantages to our execution screen.
- If you use a lot of custom fields you may want to redesign the page to show different columns important to one project or another or remove unnecessary columns
- Unlike a spreadsheet you can also quickly close or expand the test case steps.
- By default if you update a test case the next test case will automatically open.
- You can also change an individual step status.
- You can insert actual results using rich text.
- When you find a bug you can easily add it to your bug tracking system- automatically importing all of the information from the test case into the bug.
- Before adding a new bug you can view existing bugs and just link to those to avoid duplicate bugs.
- You can bulk update steps and/or test cases.
QTM Uses A Continuous Testing Approach
QTM allows test suites to be run multiple times and updated incrementally. When users open the execution screen test cases will show their previous status. While you can run separate test runs of your test suite, QTM is designed to allow organizations to do continuous testing of new builds or drops from development. In fact, QTM’s support for testing across builds in a single test cycle, or sprint, is unmatched. When testers change the status of a test case the Latest Executed Build is automatically updated. So if yesterday they were testing a different build, last night a new build passed the smoke test and was accepted by QA, the tester can just continue testing from where they left off.
Note: QMetry automates our build process through CI and this is what we recommend for you; however, you can also manually change your build.
As users update test results they will see that the build is automatically applied. This allows users to quickly see which build was last executed for each test case. Users can also view their progress in the test suite which is helpful for those with large regression suites. And if users ever feel the need to reset and force a retest on a new build they can select all the test cases and do a bulk status change to not run.
User Story : As a QA Manager with limited time to test I want my testers to start testing ASAP and record the build they tested with so that later I can filter the test results by build and evaluate if any test cases need to be retest on the latest build.
Robust Reporting on Execution History
QTM records all executions and provides a robust set of standard and unique reports to help QA Managers evaluate the test progress and coverage. Usually you want to see the current status of the test cases in your test suite. This can be done in the test suite itself with a chart at the top; however, if a user has conducted multiple repetitive tests on many builds they can review the test suite execution history to view all testing; using build report you can view the testing across builds, and with the platform report you can see the testing across platforms.
These videos do not cover automation testing; however it should be noted that a QA organization can take a manual test suite and automate the test cases. Once automated the test suite can be executed with a click of the button, or scheduled to launch later. Teams using Continuous Integration can launch it via the CI tool of choice and bring back the results to QMetry.
There are many more powerful features that QMetry Test Manager provides to help organizations manage and report on testing results, but I hope this introduction has helped you understand some of the basics and depth of our Execution Screen.