Agile is a growing part of nearly every business’ software development process. Agile can better align teams with the most pressing customer issues, speed up development, and cut costs. However, like just-in-time manufacturing, Agile’s unique approach to development means that a delay in any part of the process can lead to a screeching halt across all of it. Testing software solutions, as the last step before deployment, is critical to ensuring that companies ship working software, as well as catching and resolving edge cases and bugs before the code goes live (and becomes far more expensive to fix). If test data management is handled poorly in an Agile environment, the entire process is at risk of breaking down.

Why Test Data Management is a Bigger Challenge in Agile

As companies produce and consume more and more data, managing your test data is an increasing challenge. The key to success in leveraging test data is the realization that, the more your test data represents your live data, the better it will be at helping you uncover bugs and edge cases before deployment. While using your live production data in your tests would resolve this issue, that approach has serious data privacy and security concerns (and may not be legal in some jurisdictions). At the same time, the larger your dataset, the slower your tests.

In a traditional waterfall approach to development, a “subset, mask, and copy” approach generally ensures that data is representative of your live data, small enough for efficient testing, and meets all data privacy requirements. With the testing cycle lasting weeks or months and known well in advance, it’s relatively easy to schedule data refreshes as needed to keep test data fresh.

Agile sprints tend to be much shorter than the traditional waterfall process, so the prep time for test data is dramatically shortened. A traditional subset, mask, copy approach could severely impede operations by forcing a team to wait on test data to start development. Even worse, it could create a backlog of completed but untested features waiting for deployment, which would require companies to keep teams from starting new stories or pull people off a project to fix bugs after testing is completed. Both hurt efficiency and prevent companies from fully implementing an Agile development process.

Best Practices for Effective Test Data Management in Agile

Unfortunately, there are no shortcuts to test data management in an agile system. You have to do everything you would have done in a traditional approach, but significantly speed up the process to ensure it’s never the bottleneck. Implementing this system can require a change in institutional thinking. Success in this area means finding new ways to integrate your testers and data managers into the development process and providing them with the tools they need to succeed in an Agile environment.

1. Integrate Data Managers into the Planning Process

No matter how efficient your test data managers are, creating the right dataset for testing for a particular customer story takes at least some time. Waiting until after the planning phase is over to inform your data team of the needed data will lead to delays just from the time needed to create a dataset. However, if more esoteric data is required, the delay could be much longer than typical. By integrating your data team into the planning phase, they can leverage their expertise to help identify potential areas of concern before the start of the development phase. They can also begin working on the test datasets before the start of development, potentially providing everything needed for development and testing on the first day of development.

2. Adopt Continuous Data Refreshing

At most companies, data managers support multiple teams. With different customer stories requiring different amounts of time to complete, the data team must be flexible and efficient to meet sometimes unpredictable deadlines. However, that doesn’t excuse them from ensuring that data is up to date, that it’s free of personally identifiable information, or that it’s subset correctly for the test.

The good news is that significant portions of this process can be automated with the right tools. Modern tools can identify PII in a dataset, enabling rapid, automated transformation of an insecure database into a secure one for testing. Plus, synthetic generation tools can help companies rapidly create great datasets for testing that include no reversible PII while maintaining important referential integrity. With these processes in place, testing teams will be better equipped to handle the pace of Agile while also spending more time on high-value planning operations rather than low-level data manipulation.

3. Create a Self-Service Portal

One thing that’s guaranteed to slow Agile teams down is a formal request process to access test data. While tracking who is accessing what data is important, access requests and tracking can largely be automated with today’s tools. This idea can be taken one step further by creating a self-service portal that includes basic datasets for common development scenarios. A self-service portal ensures that smaller teams or side projects can run meaningful tests without tying up your data manager’s resources. Just like with your primary testing datasets, these must be kept reasonably up to date, but automation can significantly help reduce this burden.

How Mage Helps with Agile Test Data Management

Agile is a process that can greatly speed up development and transform the delivery of new features to your customers. However, teams need more training and tooling to execute it effectively. Not all test data management solutions are up to handling an Agile approach to development. But, Mage’s test data management solution is, providing just about everything a company could want right out of the box, while providing flexible customization options to enable companies to build the test data pipeline that works best for their needs. Contact Mage today for a free demo and learn more about how Mage can help streamline your Agile test data management process.

BLOG LIBRARY >