Software testing metrics measure the effectiveness of your development and testing efforts. With automated testing becoming more commonplace and significant to business productivity, metrics provide insight into the status of automated testing and making further improvements.
Typically, automated testing metrics quantify the performance of the implemented automated test process. These include past, present and future efforts.
When we talk of KPIs and metrics, we need clearly defined goals. Once you establish the goals clearly, metrics should be able to evaluate some of the following things:
• Success rate for the deployment
• Total error rates for each phase
• Criteria for Automation improvement
• Technical debt
Let’s look at some of the test automation metrics:
Equivalent Manual Test Effort(EMTE)
As the name suggests it helps measure the benefits of running automated tests vis-à-vis manual tests. This is important for those running both manual and automated tests. And also for those who have recently transitioned to automation. Test case EMTE is the amount of effort you need to run the case manually.
Automation code coverage
With DevOps and continuous delivery guiding testing and automation efforts, it is important to validate the relevance of automation. For instance, how much of the total code is covered by automation? This quantifies the worth of your automation efforts.
Similarly, a closely tied-in metric is requirements coverage. It tracks the number of features tested and delivered that were covered by automation. Requirements coverage aims to show the percentage of tests that are aligned to user stories.
Test Effectiveness based on Defects
To understand whether your automation is running high value test cases – look at the number of defects. This helps define how good or bad a release is. And there are many approaches to measure this. Defect density being one of them.
Defect density is simply the number of known defects divided by the size of the application.
You can also measure test effectiveness using defect containment efficiency. Calculate by dividing the number of bugs found in testing by the total number of bugs(pre and post shipping), multiplied by 100.
The higher the percentage – the better the test set and its efficacy.
Do you measure the percentage of automation is accomplished during the sprint? And what percentage is later picked up by the automation team members. Getting this measure can improve the quality for several root causes.
This includes quality of backlog grooming and stability of use cases in the sprint. The level of detail within user stories that allows the parallel implementation of automation. The quality of the automation framework design and modularity. Both of these parameters impact the speed of automation. And last but not the least, the amount of API testing versus UI based testing.
This measures the total tests executed during a build. It is important to understand if the automated tests were up to the mark. Also, graphs and charts can show the total executions categorized as passed, failed, blocked, incomplete, etc for easy understanding of the status.
Ultimately, you can measure success based on the ability of automation to accomplish the goals you had set out. It is important to track progress continuously. A good metric should be simple, easy to measure and objective. Most importantly, it should help you identify areas for automation improvement.