Category: Blogs – Test Data Protection and Delivery

If your organization performs software testing, then you probably use test data in some form. But is your process for producing, updating, and implementing that test data adding net value for the organization? Or is it a hidden roadblock to producing quality software?

Unfortunately, most organizations assume that simply having some sort of Test Data Management (TDM) process in place is sufficient. True, doing some Test Data Management is better than doing none at all. (And there are telltale signs that an organization needs Test Data Management to move forward.) But even with a Test Data Management program in place, it’s important to set up appropriate metrics and KPIs to ensure that your Test Data Management efforts are actually producing the kind of test data needed for optimal quality control.

Why Are Metrics Needed for Test Data Management?

Managing test data can be challenging, especially in large and complex software systems. Many TDM projects fail because the processes involved work at first, but erode over time. For example, the test data could lose its “freshness,” or the request process is not robust enough to create effective data sets.

This is why it is important to gain insight into the TDM process by collecting various metrics. Some of these can be captured using Test Data Management tools, while others will require some customized reporting. But the more complete your picture of the Test Data Management process is, the better your organization will be able to keep its testing process on-track and delivering according to schedule.

7 Key Test Data Management Metrics

Here, then, are seven key metrics to consider for tracking your Test Data Management capabilities. These can be split into two categories: Metrics that measure the test data itself and its quality, and metrics for the testing process.

Metrics for Test Data Quality

Data completeness. Data completeness is a measure of how well test data covers scenarios from production domains. This can especially be a concern if test data is created via subsetting, or by creating synthetic data. Special cases exist in all data sets and workflows, and those cases need to be represented in test data. There also need to be appropriate boundary cases, null cases, and negative-path cases as well. Otherwise, testing will not be sufficient.

Data quality. While data completeness is a measure of which cases are covered, data quality is a measure of how well the test data respects the rules and constraints of the database schema, as well as the application being tested. In other words, it is a measure of how well the test data “matches” production data, which in turn affects how well the data will turn up bugs with consistency and reliability.

Data freshness (data age). Aspects of data sets change over time; using test data that accurately represents the freshest production data is thus crucial. For example, the demographics of clients in a client database might shift over time, or account activity might change as new interfaces and new products are introduced. The freshness of one’s test data can be measured in terms of the age of the data itself, or in terms of the rate at which new test data is generated (the refresh rate).

Metrics for Test Data Management Processes

Data security. To what degree do the processes for generating and using test data ensure the security of the original production data? How is test data itself kept secure and compliant? Data security metrics should give numeric proof that datasets are handled in such a way as to keep sensitive information secure and its use compliant with local laws and regulations.

Test cycle time. Test cycle time is the total time for a testing cycle to complete, from request to test data creation to actual testing and validation. The goal is to reduce test cycle time as much as possible without sacrificing quality—by introducing automation, for example.

Data request % completion. Are all requests for reliable test data being met? Data request % completion is the other side of the coin from test cycle time; while cycle time measures the average speed of provisioning, data request % completion measures how many requests are actually being met in a timely matter.

Test effectiveness. If all of the above metrics were to improve within an organization, then overall test effectiveness should improve as well. So, even though test effectiveness is a lagging indicator of the quality of test data and Test Data Management, it is important to track as effectiveness is what will ultimately affect the bottom line.

Here, test effectiveness is simply a count of the number of bugs found during a period of testing, divided by the total bugs found (that is, both bugs found during testing and bugs found after shipping/deployment). For example, if all bugs are found during testing and none in production, testing effectiveness is 100%. If testing only reveals half of the bugs, with the other half discovered after deployment, testing effectiveness is 50%. The higher test effectiveness is, the better: Catching bugs in testing often makes remediation orders of magnitude cheaper than if caught in production.

How Mage Data Helps with Test Data Management

If you have a Test Data Management strategy in place, you’ve already taken the first step in the right direction. Now it is important to start collecting the right metrics, measuring your efforts, and bringing on board the right tools to improve the process.

Mage’s Data own Test Data Management solution ensures the security and efficiency of your test data processes with secure provisioning of sensitive data across teams, wherever required and whenever needed. This tool allows organizations to rapidly provision high-quality test data, allowing your TDM efforts to scale while staying secure.

To learn more about how Mage Data can help solve your Test Data Management challenges, contact us today to schedule a free demo.

While managing test data may not be as exciting as what financial institutions can do with analytics, poorly managed test data holds an existential risk. Imagine a rare bug that failed to be uncovered during testing—something that reported a balance of zero for a small set of users. The consequences would be dire. Or, imagine testing done with live data that was not properly masked. The potential for a data breach would be astronomical.

The good news is that properly managed test data can help reduce and eliminate many of these issues before applications are rolled out to users, and so securely. Getting this process right is key to keeping development costs low and avoiding the pain of small bugs with massive consequences.

Best Practice #1: Understand Your Data Landscape.
A core principle of data privacy and security is that businesses should minimize their use of data to what is necessary. One of the primary uses for test data is likely to test user interfaces on public-facing applications. In this scenario, there is a lot of information that banks hold, like social security numbers, spending patterns, and credit histories, that won’t ever show up on the front end—and consequently won’t be relevant. While your live data is a key starting point for creating a high-quality test dataset, copying it wholesale will be a waste of resources and create more risk in the event of a leak or breach. By thoughtfully exploring the minimum amount of data necessary for a test, banks can reduce their risk and speed up testing time while reducing resource consumption.

On the other side of the coin, banks often face issues gathering enough of the right information for testing. Banks are especially at risk for holding data in profoundly non-centralized manners. Legacy systems, some decades old, might handle core business operations like payment processing, credit card rewards, and even the bank accounts themselves. When data is fragmented, collecting the necessary types and amounts of data for testing while respecting data privacy and security laws can be a significant challenge. Banks need a “census” of their data, cataloging what kinds of data they hold and where it is so that they can ensure their test data is as complete as possible and assembled in a legal manner.

Best Practice #2: Proactively Refresh Your Data
Like any business, banks are always developing new products and services. Test data management provides a means of stress testing these new offerings so that they perform as expected on launch. However, using test data that isn’t up-to-date can mean that products could pass testing, but fail in actual use. For example, a bank could be expanding into a new country. That would mean that its applications must support a new language and currency symbol. While those may seem like minor adjustments to make, if poorly implemented, they could create a terrible experience for your users in the new area. Without the right test data, the issue may not be identified beforehand.
Consequently, refreshing test data isn’t just about having the best picture of where your business currently is, though that is important. Instead, refreshing test data can help propel your business to where it’s going and help future-proof it against upcoming developments. Of course, it’s also important to update your data so that it remains current. If test data grows stale, it may become a poor reflection of your business, causing testing to miss critically important issues before changes are rolled out to the public.

Best Practice #3: Anonymize Your Data
One of the riskiest things a business could do is take its most sensitive data and give it to its most junior employee. Yet, developers need access to test data to validate their software solutions. In banking, giving employees access to customer information is not just a risk, but in a heavily regulated industry like finance, is probably illegal. Companies could use various techniques to help maintain compliance, such as static or dynamic data masking, anonymization, or pseudonymization.

Best Practice #4: Automate Test Data Management
Every day, banks and their customers generate billions of new data points. In an environment where massive change can occur relatively quickly, it can become impossible for humans to keep up with the necessary changes. This is true for test data management. One of the best ways to ensure that your test data is fresh enough for testing is to automate the creation and refreshing of test datasets. That can be a massive challenge at scale, but with the right tools, banks and other financial institutions can keep up with the pace of change.

How Mage Helps Banks with Test Data Management
Ultimately, Test Data Management is not just one process but many often-complex interlocking processes. Banks have to get all of these processes right to have effective test data and perform them securely under the watchful gaze of any number of regulators. In an environment where regulatory intervention can be devastating, not just any Test Data Management solution will do. With a deep data privacy and security background, Mage Data’s Test Data Management platform provides businesses of any size with the tools they need to tackle their thorniest data management issues. Mage Data’s platform can handle data at any scale and already supports multiple businesses with multibillion-dollar revenues. To learn more about what Mage Data’s Test Data Management solution can do for your business, contact Mage Data today for a free consultation.

Test data, and Test Data Management, remain a huge challenge for tech-driven companies around the globe. Having good test data is a boon to the organization: It helps support rigorous testing to ensure that software is stable and reliable, while also mitigating security risks. But having good test data is exactly the issue. The way in which test data is created and subsequently managed has a huge effect on testers’ ability to do their jobs well.

This is why many Test Data Management projects fail. Testers manage to create testing that is usable once, or perhaps a handful of times. But over time, problems accumulate. The data loses its freshness, for example, or the request process is not robust enough to create appropriate data sets.

While we cannot diagnose every unique issue that organizations face when it comes to Test Data Management, there are some common challenges that we’ve seen—challenges that routinely sink Test Data Management projects, making them less effective, more costly, and more likely to fail outright.

Here are the top six:

1: Lack of Buy-In

Test Data Management isn’t likely to be high on the list of priorities for any company. Too often, it’s an afterthought. There isn’t always an internal champion making the project a priority, let alone making the argument for investing in better tools.

When this lack of buy-in exists, especially among company leadership, a TDM project is liable to fail before it ever takes off. Thus, it is important to get stakeholders on board; they should understand why Test Data Management is important, and have some say into new projects from the start. This includes both company leadership and those who will be responsible for implementing the plan.

2: Lack of Standardization

One sign or outcome of a lack of buy-in is a lack of standardization. Answer these questions: Does your organization have a well-developed data dictionary? Where can that be found? Who created it? What does the data model look like? If the answers to these questions are not readily known by you or the team responsible for Test Data Management, chances are you don’t have robust standards in place.

Another manifestation of this problem is an absence of standard data request forms. This leads to data requests in different types of formats, as well as different tests, both of which ultimately lengthen testing cycles.

3: Older Data and/or Merged Data

Test data should cover a range of relevant test cases and legitimately resemble the data in your production environments. But data is not static; it tends to change over time. For example, the demographics of clients in a client database might shift over time, or account activity might change as new interfaces and new products are introduced. This means there is an expiration date on datasets, and yet outdated test data is routinely used.

This especially becomes a problem after an acquisition or merger. Each party will bring their own data to the new entity, and merging the datasets is a challenge in its own right. If the conversion is done badly, this can throw off the data sample.

4: Privacy and Safety Standards

Many applications traffic is sensitive data—banking apps, tax preparation software, HR software, shopping apps…the list goes on. These applications can reveal crucially important insights, but they also risk revealing sensitive personal and financial information.

This creates a trade-off between safety and relevance when it comes to test data. Using real-time data from production environments ensures relevance, but often at the cost of privacy. Using synthetic data ensures privacy but risks not having the proper relationships and, hence, not being relevant.

Using automated test data generation software can help the team to deal with this problem. This helps to create data with the relevant relations intact, but without revealing sensitive information. Masking and/or encryption should be part of this process.

5: Problems with Referential Integrity

Ideally, any set of test data should contain a representative cross-section of the data, maintaining a high degree of referential integrity. Again, this is easier to achieve with real-time data but much harder with synthetic test data.

So, when creating synthetic testing data, it is important to have a data model that accurately defines the relationships between key pieces of data. Primary keys must be properly linked, and data relationships should be based on well-defined business rules.

Sometimes a TDM project will fail because the test data does not have this kind of referential integrity, and the entire testing process becomes an exercise in demonstrating the adage: “garbage in, garbage out.”

6: Waterfall TDM in an Agile Development Environment

There are plenty of data management tools out there today…and many of them assume a more-or-less waterfall approach to development. In a waterfall approach, a “subset, mask, and copy” methodology usually ensures that test data is representative of live data, is small enough for efficient testing, and meets all data privacy requirements. With the testing cycle lasting weeks or months and known well in advance, it’s relatively easy to schedule data refreshes as needed to keep test data fresh.

Things are not so straightforward in an agile development environment. Besides integration challenges, commonly there also are timing challenges. Agile sprints tend to be much shorter than the traditional waterfall process, so the prep time for test data is dramatically shortened. The approach outlined above tends to impede operations by forcing a team to wait for test data and can create a backlog of completed, but untested, features waiting for deployment. Again, automation can help create test data on-the-fly and as needed.

Preventing Failure

Given that the above list represents the majority of reasons why TDM projects fail, we can reverse-engineer those reasons to put together a plan for TDM success:

1. Get buy-in early from both teams and leadership,
2. Start with appropriate standardization (including a data dictionary and current data model with appropriate referential integrity),
3. Make a plan to clean older, irrelevant data and convert any data from other sources,
4. Choose methods that will yield data with the appropriate trade-off between privacy/security and relevance (including using masking and encryption), and
5. Automate the process wherever possible.
   
   Mage Data can help with many of these steps. Our Test Data Management solution ensures the security and efficiency of your test data processes, allowing for the secure provisioning of sensitive data across teams, wherever required and whenever needed. Your teams will have a better experience with our ready-to-go, plug-and-play platform for TDM, irrespective of data store type. You can ask for a demo today.