If your organization performs software testing, then you probably use test data in some form. But is your process for producing, updating, and implementing that test data adding net value for the organization? Or is it a hidden roadblock to producing quality software?

Unfortunately, most organizations assume that simply having some sort of test data management (TDM) process in place is sufficient. True, doing some test data management is better than doing none at all. (And there are telltale signs that an organization needs test data management  to move forward.) But even with a test data management program in place, it’s important to set up appropriate metrics and KPIs to ensure that your test data management efforts are actually producing the kind of test data needed for optimal quality control.

Why Are Metrics Needed for Test Data Management?

Managing test data can be challenging, especially in large and complex software systems. Many TDM projects fail because the processes involved work at first, but erode over time. For example, the test data could lose its “freshness,” or the request process is not robust enough to create effective data sets.

This is why it is important to gain insight into the TDM process by collecting various metrics. Some of these can be captured using test data management tools, while others will require some customized reporting. But the more complete your picture of the test data management process is, the better your organization will be able to keep its testing process on-track and delivering according to schedule.

7 Key Test Data Management Metrics

Here, then, are seven key metrics to consider for tracking your test data management capabilities. These can be split into two categories: Metrics that measure the test data itself and its quality, and metrics for the testing process.

Metrics for Test Data Quality

Data completeness. Data completeness is a measure of how well test data covers scenarios from production domains. This can especially be a concern if test data is created via subsetting, or by creating synthetic data. Special cases exist in all data sets and workflows, and those cases need to be represented in test data. There also need to be appropriate boundary cases, null cases, and negative-path cases as well. Otherwise, testing will not be sufficient.

Data quality. While data completeness is a measure of which cases are covered, data quality is a measure of how well the test data respects the rules and constraints of the database schema, as well as the application being tested. In other words, it is a measure of how well the test data “matches” production data, which in turn affects how well the data will turn up bugs with consistency and reliability.

Data freshness (data age). Aspects of data sets change over time; using test data that accurately represents the freshest production data is thus crucial. For example, the demographics of clients in a client database might shift over time, or account activity might change as new interfaces and new products are introduced. The freshness of one’s test data can be measured in terms of the age of the data itself, or in terms of the rate at which new test data is generated (the refresh rate).

Metrics for Test Data Management Processes

Data security. To what degree do the processes for generating and using test data ensure the security of the original production data? How is test data itself kept secure and compliant? Data security metrics should give numeric proof that datasets are handled in such a way as to keep sensitive information secure and its use compliant with local laws and regulations.

Test cycle time. Test cycle time is the total time for a testing cycle to complete, from request to test data creation to actual testing and validation. The goal is to reduce test cycle time as much as possible without sacrificing quality—by introducing automation, for example.

Data request % completion. Are all requests for reliable test data being met? Data request % completion is the other side of the coin from test cycle time; while cycle time measures the average speed of provisioning, data request % completion measures how many requests are actually being met in a timely matter.

Test effectiveness. If all of the above metrics were to improve within an organization, then overall test effectiveness should improve as well. So, even though test effectiveness is a lagging indicator of the quality of test data and test data management, it is important to track as effectiveness is what will ultimately affect the bottom line.

Here, test effectiveness is simply a count of the number of bugs found during a period of testing, divided by the total bugs found (that is, both bugs found during testing and bugs found after shipping/deployment). For example, if all bugs are found during testing and none in production, testing effectiveness is 100%. If testing only reveals half of the bugs, with the other half discovered after deployment, testing effectiveness is 50%. The higher test effectiveness is, the better: Catching bugs in testing often makes remediation orders of magnitude cheaper than if caught in production.

How Mage Helps with Test Data Management

If you have a test data management strategy  in place, you’ve already taken the first step in the right direction. Now it is important to start collecting the right metrics, measuring your efforts, and bringing on board the right tools to improve the process.

Mage’s own test data management solution ensures the security and efficiency of your test data processes with secure provisioning of sensitive data across teams, wherever required and whenever needed. This tool allows organizations to rapidly provision high-quality test data, allowing your TDM efforts to scale while staying secure.

To learn more about how Mage can help solve your test data management challenges, contact us today to schedule a free demo.

BLOG LIBRARY >