Mage Data

Category: Blogs – Data Catalog

  • Healthcare Reinvented: Data Security Meets Compliance

    Healthcare Reinvented: Data Security Meets Compliance

    In today’s healthcare ecosystem, data is both an operational backbone and a compliance challenge. For organizations managing vast networks of primary care centers, protecting patient data while maintaining efficiency is a constant balancing act. As the healthcare industry becomes increasingly data-driven, the need to ensure security, consistency, and compliance across systems has never been more critical.

    Primary care organizations depend on sensitive clinical and claims data sourced from multiple payers. Each source typically arrives in a different format—creating integration hurdles and privacy risks. Manual processing not only slows operations but also increases the chance of human error and non-compliance with data protection mandates such as HIPAA.

    To overcome these challenges, one leading healthcare provider partnered with Mage Data, adopting its Test Data Management (TDM) 2.0 solution. The results transformed the organization’s ability to scale securely, protect patient information, and maintain regulatory confidence while delivering high-quality care to its patients.

    The organization faced multiple, interrelated data challenges typical of large-scale primary care environments:

    • Protecting Patient Privacy: Ensuring HIPAA compliance meant that no sensitive health data could be visible in development or test environments. Traditional anonymization processes were slow and prone to inconsistency.
    • Data Consistency Across Systems: Patient identifiers such as names, IDs, and dates needed to remain accurate and consistent across applications and databases to preserve reporting integrity.
    • Operational Inefficiency: Teams spent valuable time manually processing payer files in multiple formats, introducing risk and slowing development cycles.
    • Scaling with Growth: With over 50 payer file formats and new ones continuously added, the organization struggled to maintain standardization and automation.

    These pain points created a clear need for an automated, compliant, and scalable Test Data Management framework.

    Mage Data implemented its TDM 2.0 solution to address the organization’s end-to-end data management and privacy challenges. The deployment focused on automation, privacy assurance, and operational scalability.

    1. Automated Anonymization

    Mage Data automated the anonymization of all payer files before they entered non-production environments. This ensured that developers and testers never had access to real patient data, while still being able to work with datasets that mirrored production in structure and behavior. The result was full compliance with HIPAA and other healthcare data protection requirements.

    1. NLP-Based Masking for Unstructured Text

    To mitigate the risk of identifiers embedded in free-text fields—such as medical notes or descriptions—Mage Data integrated Natural Language Processing (NLP)-based masking. This advanced capability identified and anonymized hidden personal data, ensuring that no sensitive information was exposed inadvertently.

    1. Dynamic Templates and Continuous Automation

    Mage Data introduced dynamic templates that automatically adapted to new or changing file types from different payers. These templates, combined with continuous automation through scheduled jobs, detected, masked, and routed new files into development systems—quarantining unsupported formats until validated. This approach reduced manual effort, improved accuracy, and allowed the organization to support rapid expansion without re-engineering its data pipelines.

    The adoption of Mage Data’s TDM 2.0 delivered measurable improvements across compliance, efficiency, and operational governance:

    • Regulatory Compliance Assured: The organization successfully eliminated the risk of HIPAA violations in non-production environments.
    • Faster Development Cycles: Developers gained access to compliant, production-like data in hours instead of days—accelerating release cycles and integration efforts.
    • Consistency at Scale: Mage Data ensured that identifiers such as patient names, IDs, and dates remained synchronized across systems, maintaining the accuracy of analytics and reports.
    • Operational Efficiency: Manual discovery and masking processes were replaced by automated, rule-driven workflows—freeing technical teams to focus on higher-value work.
    • Future-Ready Scalability: The solution’s adaptable framework was designed to seamlessly extend to new data formats, applications, and business units as the organization grew nationwide.

    Through this transformation, Mage Data enabled the healthcare provider to turn data protection from a compliance burden into a strategic advantage, empowering its teams to innovate faster while safeguarding patient trust.

    In conclusion, Mage Data delivers a comprehensive, multi-layered data security framework that protects sensitive information throughout its entire lifecycle. The first step begins with data classification and discovery, enabling organizations to locate and identify sensitive data across environments. This is followed by data cataloging and lineage tracking, offering a clear, traceable view of how sensitive data flows across systems. In non-production environments, Mage Data applies static data masking (SDM) to generate realistic yet de-identified datasets, ensuring safe and effective use for testing and development. In production, a Zero Trust model is enforced through dynamic data masking (DDM), database firewalls, and continuous monitoring—providing real-time access control and proactive threat detection. This layered security approach not only supports regulatory compliance with standards such as GDPR, HIPAA, and PCI-DSS but also minimizes risk while preserving data usability. By integrating these capabilities into a unified platform, Mage Data empowers organizations to safeguard their data with confidence—ensuring privacy, compliance, and long-term operational resilience.

  • Reimagining Test Data: Secure-by-Design Database Virtualization

    Reimagining Test Data: Secure-by-Design Database Virtualization

    Enterprises today are operating in an era of unprecedented data velocity and complexity. The demand for rapid software delivery, continuous testing, and seamless data availability has never been greater. At the same time, organizations face growing scrutiny from regulators, customers, and auditors to safeguard sensitive data across every environment—production, test, or development.

    This dual mandate of speed and security is reshaping enterprise data strategies. As hybrid and multi-cloud infrastructures expand, teams struggle to provision synchronized, compliant, and cost-efficient test environments fast enough to keep up with DevOps cycles. The challenge lies not only in how fast data can move, but in how securely it can be replicated, masked, and managed.

    Database virtualization was designed to solve two of the biggest challenges in Test Data Management—time and cost. Instead of creating multiple full physical copies of production databases, virtualization allows teams to provision lightweight, reusable database instances that share a common data image. This drastically reduces storage requirements and accelerates environment creation, enabling developers and QA teams to work in parallel without waiting for lengthy data refresh cycles. By abstracting data from its underlying infrastructure, database virtualization improves agility, simplifies DevOps workflows, and enhances scalability across hybrid and multi-cloud environments. In short, it brings speed and efficiency to an otherwise resource-heavy process—freeing enterprises to innovate faster.

    Database virtualization was introduced to address inefficiencies in provisioning and environment management. It promised faster test data creation by abstracting databases from their underlying infrastructure. But for many enterprises, traditional approaches have failed to evolve alongside modern data governance and privacy demands.

    Typical pain points include:

    • Storage-Heavy Architectures: Conventional virtualization still relies on partial or full data copies, consuming vast amounts of storage.
    • Slow, Manual Refresh Cycles: Database provisioning often depends on DBAs, leading to delays, inconsistent refreshes, and limited automation.
    • Fragmented Data Privacy Controls: Sensitive data frequently leaves production unprotected, exposing organizations to compliance violations.
    • Limited Integration: Many solutions don’t integrate natively with CI/CD or hybrid infrastructures, making automated delivery pipelines cumbersome.
    • Rising Infrastructure Costs: With exponential data growth, managing physical and virtual copies across clouds and data centers drives up operational expenses.

    The result is an environment that might be faster than before—but still insecure, complex, and costly. To thrive in the AI and automation era, enterprises need secure-by-design virtualization that embeds compliance and efficiency at its core.

    Modern data-driven enterprises require database virtualization that does more than accelerate. It must automate security, enforce privacy, and scale seamlessly across any infrastructure—cloud, hybrid, or on-premises.

    This is where Mage Data’s Database Virtualization (DBV) sets a new benchmark. Unlike traditional tools that treat masking and governance as secondary layers, Mage Data Database Virtualization builds them directly into the virtualization process. Every virtual database created is masked, compliant, and policy-governed by default—ensuring that sensitive information never leaves production unprotected.

    Database Virtualization lightweight, flexible architecture enables teams to provision virtual databases in minutes, without duplicating full datasets or requiring specialized hardware. It’s a unified solution that accelerates innovation while maintaining uncompromising data privacy and compliance.

    1. Instant, Secure Provisioning
      Create lightweight, refreshable copies of production databases on demand. Developers and QA teams can access ready-to-use environments instantly, reducing cycle times from days to minutes.
    2. Built-In Data Privacy and Compliance
      Policy-driven masking ensures that sensitive data remains protected during every clone or refresh. Mage Data Database Virtualization is compliance-ready with frameworks like GDPR, HIPAA, and PCI-DSS, ensuring enterprises maintain regulatory integrity across all environments.
    3. Lightweight, Flexible Architecture
      With no proprietary dependencies or hardware requirements, Database Virtualization integrates effortlessly into existing IT ecosystems. It supports on-premises, cloud, and hybrid infrastructures, enabling consistent management across environments.
    4. CI/CD and DevOps Integration
      DBV integrates natively with Jenkins, GitHub Actions, and other automation tools, empowering continuous provisioning within DevOps pipelines.
    5. Cost and Operational Efficiency
      By eliminating full physical copies, enterprises achieve up to 99% storage savings and dramatically reduce infrastructure, cooling, and licensing costs. Automated refreshes and rollbacks further cut
      manual DBA effort.
    6. Time Travel and Branching (Planned)
      Upcoming capabilities will allow enterprises to rewind databases or create parallel branches, enabling faster debugging and parallel testing workflows.

    The AI-driven enterprise depends on speed—but the right kind of speed: one that doesn’t compromise security or compliance. Mage Data Database Virtualization delivers precisely that. By uniting instant provisioning, storage efficiency, and embedded privacy, it transforms database virtualization from a performance tool into a strategic enabler of governance, innovation, and trust.

    As enterprises evolve to meet the demands of accelerating development, they must modernize their entire approach to data handling—adapting for an AI era where agility, accountability, and assurance must coexist seamlessly.

    Mage Data’s Database Virtualization stands out as the foundation for secure digital transformation—enabling enterprises to accelerate innovation while ensuring privacy and compliance by design.

  • Building Trust in AI: Strengthening Data Protection with Mage Data

    Building Trust in AI: Strengthening Data Protection with Mage Data

    Artificial Intelligence is transforming how organizations analyze, process, and leverage data. Yet, with this transformation comes a new level of responsibility. AI systems depend on vast amounts of sensitive information — personal data, intellectual property, and proprietary business assets — all of which must be handled securely and ethically.

    Across industries, organizations are facing a growing challenge: how to innovate responsibly without compromising privacy or compliance. The European Commission’s General-Purpose AI Code of Practice (GPAI Code), developed under the EU AI Act, provides a structured framework for achieving this balance. It defines clear obligations for AI model providers under Articles 53 and 55, focusing on three key pillars — Safety and Security, Copyright Compliance, and Transparency.

    However, implementing these requirements within complex data ecosystems is not simple. Traditional compliance approaches often rely on manual audits, disjointed tools, and lengthy implementation cycles. Enterprises need a scalable, automated, and auditable framework that bridges the gap between regulatory expectations and real-world data management practices.

    Mage Data Solutions provides that bridge. Its unified data protection platform enables organizations to operate compliance efficiently — automating discovery, masking, monitoring, and lifecycle governance — while maintaining data utility and accelerating AI innovation.

    The GPAI Code establishes a practical model for aligning AI system development with responsible data governance. It is centered around three pillars that define how providers must build and manage AI systems.

    1. Safety and Security
      Organizations must assess and mitigate systemic risks, secure AI model parameters through encryption, protect against insider threats, and enforce multi-factor authentication across access points.
    2. Copyright Compliance
      Data sources used in AI training must respect intellectual property rights, including automated compliance with robots.txt directives and digital rights management. Systems must prevent the generation of copyrighted content.
    3. Transparency and Documentation
      Providers must document their data governance frameworks, model training methods, and decision-making logic. This transparency ensures accountability and allows regulators and stakeholders to verify compliance.

    These pillars form the foundation of the EU’s AI governance model. For enterprises, they serve as both a compliance obligation and a blueprint for building AI systems that are ethical, explainable, and secure.

    Mage Data’s platform directly maps its data protection capabilities to the GPAI Code’s requirements, allowing organizations to implement compliance controls across the full AI lifecycle — from data ingestion to production monitoring.

    GPAI Requirement

    Mage Data Capability

    Compliance Outcome

    Safety & Security (Article 53)

    Sensitive Data Discovery

    Automatically identifies and classifies sensitive information across structured and unstructured datasets, ensuring visibility into data sources before training begins.

    Safety & Security (Article 53)

    Static Data Masking (SDM)

    Anonymizes training data using over 60 proven masking techniques, ensuring AI models are trained on de-identified yet fully functional datasets.

    Safety & Security (Article 53)

    Dynamic Data Masking (DDM)

    Enforces real-time, role-based access controls in production systems, aligning with Zero Trust security principles and protecting live data during AI operations.

    Copyright Compliance (Article 55)

    Data Lifecycle Management

    Automates data retention, archival, and deletion processes, ensuring compliance with intellectual property and “right to be forgotten” requirements.

    Transparency & Documentation (Article 55)

    Database Activity Monitoring

    Tracks every access to sensitive data, generates audit-ready logs, and produces compliance reports for regulatory or internal review.

    Transparency & Accountability

    Unified Compliance Dashboard

    Provides centralized oversight for CISOs, compliance teams, and DPOs to manage policies, monitor controls, and evidence compliance in real time.

    By aligning these modules to the AI Code’s compliance pillars, Mage Data helps enterprises demonstrate accountability, ensure privacy, and maintain operational efficiency.

    Mage Data enables enterprises to transform data protection from a compliance requirement into a strategic capability. The platform’s architecture supports high-scale, multi-environment deployments while maintaining governance consistency across systems.

    Key advantages include:

    • Accelerated Compliance: Achieve AI Act alignment faster than traditional, fragmented methods.
    • Integrated Governance: Replace multiple point solutions with a unified, policy-driven platform.
    • Reduced Risk: Automated workflows minimize human error and prevent data exposure.
    • Proven Scalability: Secures over 2.5 billion data rows and processes millions of sensitive transactions daily.
    • Regulatory Readiness: Preconfigured for GDPR, CCPA, HIPAA, PCI-DSS, and EU AI Act compliance.

    This integrated approach enables security and compliance leaders to build AI systems that are both trustworthy and operationally efficient — ensuring every stage of the data lifecycle is protected and auditable.

    Mage Data provides a clear, step-by-step plan:

    This structured approach takes the guesswork out of compliance and ensures organizations are always audit-ready

    The deadlines for AI Act compliance are approaching quickly. Delaying compliance not only increases costs but also exposes organizations to risks such as:

    • Regulatory penalties that impact global revenue.
    • Data breaches harm brand trust.
    • Missed opportunities, as competitors who comply early gain a reputation for trustworthy, responsible AI.

    By starting today, enterprises can turn compliance from a burden into a competitive advantage.

    The General-Purpose AI Code of Practice sets high standards but meeting them doesn’t have to be slow or costly. With Mage Data’s proven platform, organizations can achieve compliance in weeks, not years — all while protecting sensitive data, reducing risks, and supporting innovation.

    AI is the future. With Mage Data, enterprises can embrace it responsibly, securely, and confidently.

    Ready to get started? Contact Mage Data for a free compliance assessment and see how we can help your organization stay ahead of the curve.

  • What is Considered Sensitive Data Under the GDPR?

    What is Considered Sensitive Data Under the GDPR?

    There are many different kinds of personal information that a company might store in the course of creating and maintaining user accounts: names, residential addresses, payment information, government ID numbers, and more. Obviously, companies have a vested interest in keeping this sensitive data safe, as data breaches can be both costly and embarrassing.

    What counts as private or sensitive data—and what sorts of responsibility companies have to protect such data—changed with the passage of the General Data Protection Regulation (GDPR) by the European Union. (The GDPR is a component of the EU’s privacy law and human rights law relevant to Article 8 of the Charter of Fundamental Rights of the European Union.) The GDPR is proving to be both expansive in what it covers and strict in what it requires of entities holding user data, and the fines levied for non-compliance can sometimes be harsh.

    The European Union’s own GDPR website has a good overview of what the regulation is, along with overviews of its many parts and guidelines for compliance. But one of the stickier points of this regulation is what is considered “sensitive data,” and how this might differ from personal data, which is at the core of the GDPR. Sensitive data forms a special protected category of data, and companies must take steps to find it using appropriate sensitive data discovery tools

    The GDPR Protects Personal Data

    At the heart of the GDPR is the concept of personal data. Personal data includes any information which can be linked to an identified or identifiable person. Examples of such information includes things like:

    • Names
    • Identification numbers.
    • Location data—this includes anything that can confirm your physical presence somewhere, such as security footage, fingerprints, etc.
    • Any data which represents physical, physiological, genetic, mental, commercial, cultural, or social identity.
    • Identifiers which are assigned to a person—telephone numbers, credit card numbers, account data, license plates, customer numbers, email addresses, and so on.
    • Subjective information such as opinions, judgments, or estimates—for example, an assessment of creditworthiness or review of work performance by an employer.

    It is important to note that some kinds of data might not successfully identify a person unless used with other data. For example, a common name like “James Smith” might apply to many people, and so would not pick out a single individual. But combining that name with an email address further narrows things down to a particular company and identifier; together, the name and email are personal information. Likewise, things like gender, ZIP Code, or date of birth would be non-sensitive, non-personal information unless combined with other information to identify someone. Hackers and bad actors will often use disparate pieces of data to identify individuals, so all potential personal information should be handled cautiously.

    That said, some personal information is also considered sensitive information; the GDPR discourages collecting, storing, processing, or displaying this information except under special circumstances—and in those cases, extra security measures are needed.

    Sensitive Information Under the GDPR

    Sensitive data under the GDPR (sometimes referred to as “sensitive personal data”) includes:

    • Any personal data revealing racial or ethnic origin, political opinions, or religious or philosophical beliefs;
    • Trade union membership;
    • Genetic data;
    • Biometric data used to identify a person;
    • Health-related data; and
    • Data concerning a person’s sex life or sexual orientation.

    According to Article 9 paragraph 1 of the GDPR, these kinds of information cannot be processed except for special cases as outlined in paragraph 2. This includes gathering and storing such data in the first place.

    Application of the GDPR: Does it Affect Your Organization?

    In short, yes, the GDPR is relevant even for companies operating largely outside of the European Union. The goal of the GDPR is to protect data belonging to EU citizens and residents; it categorizes many of its provisions as a right that people have. Thus, anyone handling data about EU residents is subject to GDPR regulations, independent of their location.

    For example, if you have a company in the U.S. with a website, and said website is accessed and used by citizens residing in the European Union, and part of that use is creating accounts which process and store user data, then your company must comply with the GDPR. (This is referred to as the “extra-territorial effect.”)

    Even more alarming is the fact that sensitive data might exist within an organization without its being aware of the scope and extent of that data’s existence. Consider:

    In short, no company should assume that it has a handle on sensitive data until it can verify the location of all sensitive personal data using a robust sensitive data discovery procedure.

    Data Subject Requests, The Right to Be Forgotten, and Data Minimization

    Processing sensitive information becomes an especially challenging conundrum when it comes to Data Subject Requests (DSRs). Such requests can include things like the Right to be Forgotten: The right that individuals have to request that information about them be deleted if they choose. According to the GDPR (and many other data protection regulations), organizations receiving requests from individuals have a limited and specific time period for honoring such requests.

    Most organizations will honor these requests simply by deleting the relevant information. But this approach runs into two problems.

    First, redundant copies of data often exist in complex environments—for example, the same personal information might appear in a testing environment, a production environment, and a marketing analytics database. Without robust sensitive data discovery, it’s possible that an individual isn’t really “forgotten” by the system after all.

    Second, there is the issue of database integrity. Deleting data might remove important bits of information, such as transaction histories. This can make it incredibly difficult to keep audit trails or maintain accurate data analytics. Companies that acquire sensitive information, then, would do better finding ways to minimize this data, rather than delete it completely.

    If you would like to learn more about data minimization, sensitive data discovery, or GDPR compliance in general, feel free to browse our articles or contact a compliance expert. In the meantime, our case study of a Swiss Bank also highlights how cross-border data-sharing can be accomplished while maintaining compliance with the GDPR.

     

  • What to Look for in a Sensitive Data Discovery Tool

    What to Look for in a Sensitive Data Discovery Tool

    Selecting the right sensitive data discovery tool for your organization can be challenging. Part of the difficulty lies in the fact that you will only get a feel for how effective your choice is after purchasing and implementing it. However, there are things you can do to help maximize your return on investment before you buy by focusing your attention on the right candidates. By selecting your finalists based on their ability to execute on the best practices for sensitive data discovery, you can significantly increase the odds that your final choice is a good fit for your needs.

    Best Practices for Sensitive Data Discovery

    Of course, you can’t effectively select for the best practices in sensitive data discovery without a deep understanding of what they are and how they impact your business. While any number of approaches could be considered “best practices,” here are four that we believe are the most impactful when implementing a new sensitive data discovery system.

    Maximize Automation

    While more automation is almost always good, when it comes to sensitive data discovery, there’s a big difference between increasing automation and maximizing automation. In an ideal world, your data team would configure the rules for detecting personally identifiable information once and then spend their time on higher-value activities like monitoring and reporting. But there’s more to automation than just data types. Is the reporting automated? Does the system work well with the system that handles “right to be forgotten” requests? Any human-driven process is likely to fail when scaled up to millions or billions of data points. Success in this area means finding a solution that maximizes automation and minimizes the burden on your team.

    Merge Classification and Discovery

    Data must be classified before its insights can be unlocked. Despite its similarities to data discovery, data classification is sometimes handled by a different department with different tools. A potential downside of that approach is that a key stakeholder gets a report from each department and asks why the numbers don’t match. As a result, your team is forced to spend time reconciling the different tools’ output—which is not a great place to expend resources. An easy way to fix this problem is to use a single tool to perform both processes. If that’s not a viable approach, ensuring the tools are integrated to produce the same results can be a great way to ensure that your company has a unified and consistent view of its data.

    Develop a Multi-Channel Approach

    One trap that companies sometimes fall into is believing that the discovery process is over once data from outside the company is identified and appropriately secured on the company network. This approach neglects one of the biggest sources of risk when it comes to data: your employees. Are you monitoring your employee endpoints like laptops, desktops etc. for personally identifiable information? If so, are you able to manually or automatically remedy the situation? You won’t always be able to stop employees from making risky moves with data. However, with a multi-channel approach to sensitive data discovery, you can monitor the situation and develop procedures to limit the damage.

    Create Regular Risk Assessments

    Identifying your sensitive data is only the first step in the process. To understand your company’s overall risk, you must deeply understand the relative risk that each piece of sensitive information holds. For example, data moved across borders holds significantly more risk than that in long-term cold storage. Databases that hold customer information inherently have more risk than those holding only corporate information. To meaningfully prioritize your efforts in securing data and optimizing your processes, you need regular risk assessments. At scale, this can be difficult to do on your own—so your sensitive data discovery software either needs to do it for you or have a robust integration with a program that can.

    Choosing the Right Sensitive Data Discovery Software

    While there are many possible ways to select sensitive data discovery tool , the best practices we’ve covered offer a good starting place for most businesses. Remember that the features that one software package has vs. another is not necessarily as important as how those features support your business objectives. Maximizing automation, merging discovery and classification, developing a multi-channel approach, and creating regular risk assessments all have relatively little to do with the actual mechanics of data discovery—but they can all make a huge difference when building a healthy, secure company. There are a lot of different sensitive data discovery solutions that can solve your immediate problem. However, they may not do it in a way that holistically improves your business.

    Another important point is that data discovery is the first step in the data lifecycle that runs all the way to retirement. You could use a different tool for each stage of the process, but the end result would be a system with multiple independent parts that may or may not work well together. Ideally, you would be able to handle data throughout the lifecycle in one application. That’s where Mage Data comes in.

    How Mage Data Helps with Sensitive Data Discovery

    Mage Data’s approach to data security begins with robust data discovery through its patented Mage Sensitive Data Discovery tool, which is powered by artificial intelligence and natural language processing. It can identify more than 80 data types right out-of-the-box and can be configured for as many custom data types as you need.

    But that’s only the start of the process. Mage’s Data Masking solutions provide powerful static and dynamic masking options to keep data safe during the middle of its lifecycle, and Data Minimization tool helps companies handle data access and erasure requests and create robust data retention rules. Having an integrated platform that handles all aspects of data security and privacy can save you money and be far simpler to operate than having different platforms for different operations. We believe that it shouldn’t matter if you’re a small business or enterprise – your data solutions should just work. To learn more about how Mage Data can help you with sensitive data discovery, schedule a demo today.

  • Your Data Protection Journey Should Start with True Data Discovery

    Your Data Protection Journey Should Start with True Data Discovery

    One of the key services we offer here at Mage Data is robust data discovery, and for a good reason. We have found that too many organizations try to unearth sensitive data simply through using regular expressions in their database—and they inevitably come to us when those efforts fail.

    So, for those organizations out there starting their data protection journeys, please learn from the lessons of others. You can save a lot of time and expense by using a robust data discovery tool. While it might be tempting to try a DIY solution, those likely will be inadequate for some well-known reasons.

    Regular Expressions: The Basics

    Regex, or regular expressions, are simply sequences of characters that specify a search pattern. With them, a user can specify rules that determine which search strings the system should return. For example, they could be used to search a database to find:

    • All records with a last name
    • All records with a last name of “Smith”
    • All records with a last name beginning with “S”
    • All records with a last name beginning with “S” except those that are “Smith”
    • …and so on.

    Naturally, these are not limited to names. Regular expressions can be used to query phone numbers, Social Security numbers, email addresses, credit card numbers…any data that is stored in a database.

    This is why they sometimes get used in data discovery. Take Social Security numbers. Knowing the format of Social Security numbers, one could use a regular expression to find all such numbers in a data set and flag them for data masking. That will hide sensitive information, but only if you can catch all (and only) Social Security numbers.

    So regular expressions can be thought of as their own kind of programming language, albeit a very specialized language with a very limited number of operators. That simplicity makes them powerful, but also limited in many ways.

    Using Regular Expressions (Regex) to Find Sensitive Data Patterns

    If regex can be used to find any arbitrary string of data, how could it possibly be limited? There are two ways.

    First, the user that created the regex query has to know what they are doing in order to capture all the relevant data. Take U.S. Social Security numbers (SSNs). One might be tempted to simply create a regular expression to capture any nine-digit number. But what if some of the SSNs have dashes, and others do not? And what if some are invalid as SSNs—for example, there are no legitimate SSNs that begin with the digits 666 or 989. The person who creates the regular expression will need to take into account all the different formats and combinations possible in the data.

    That leads to the second problem: False positives. It is possible that other forms of data can have a format similar to an SSN, once one takes into account all the different variations. For example, a telephone number that is missing a digit, or national I.D. numbers, can also fit the pattern of an SSN. This will lead to many pieces of data being flagged as sensitive, when they really are not.

    But why are false positives a bad thing? If the organization is finding all of the sensitive data, what does it matter if some non-sensitive data is flagged as sensitive, too?

    False Positives, Human Intervention, and the Investment of Time

    In data discovery, one wants to decrease the number of false positives as much as possible. Too many false positives will overwhelm the search as your team loses the ability to discern actual sensitive data from false positives that live in your data set.

    In fact, most organizations require additional human intervention to sift through data and identify false positives from actual hits. This additional human effort takes time.

    For example, suppose that a search using regex takes a full day to find the appropriate data. That might seem pretty speedy, but the human effort to weed out false positives might take 10 days thereafter. The data is ready, then, in 11 days.

    Compare this to our more robust data discovery that uses artificial intelligence and natural language processing that can understand the context surrounding a piece of data, as well as discover sensitive data in unstructured fields. The process sounds like it will take longer—five days instead of one day with regex! But when the data discovery process is done, the human team will need only a single day to sort out the data and find the few remaining false positives. A process that, overall, used to take 11 days now takes only six.

    MethodInitial scan (hypothetical)Human review (hypothetical)Total time
    Regular expressions (regex)1 day10 days11 days
    Mage Data Discovery5 days1 day6 days
    Difference4 days (4X increase)9 days (9X decrease)5 days saved

    In short, there is a direct correlation between the quality of the data discovery process and the time spent refining the data and weeding out false positives.

    Other Issues with Regex

    While time is certainly a factor, there are other issues with homebrew regex as well:

    Lost distinctions. Regex cannot make distinctions that are not already spelled out. For example, it might be able to return possible credit card numbers, but it won’t specify whether they are Visa or Mastercard numbers. Yet there are plenty of reasons why one might want to know this—for example, appropriately masking data for further analytics.

    Bad with unstructured data. Similarly, regex does not do well with unstructured data or data that does not have appropriate context. For example, there might be a mountain of sensitive data sitting within your email system, but regex would do a poor job at uncovering it, as there is not appropriate context.

    Where, not how. Regex can find sensitive data and show a user where that data “lives,” but it cannot uncover where the sensitive data came from, nor how it is flowing through the organization, unless that information is contained within the data, too (which it probably isn’t). More robust discovery tools can uncover this flow of data to anticipate future sources of sensitive data.

    Skip the Homegrown Scripts

    When sensitive data and compliance become an issue for an organization, it is all too common to bring more people onto a team to write scripts using regex for data discovery. The more complex and multi-dimensional the data, the more likely those homegrown scripts will fail.

    Whether this has happened to your organization, or you are still considering the possibility, know that there is another alternative.

    While contextual data discovery might sound costly and time-consuming at first, it will, in the end, save the entire organization time and money.

    If you want to find out more, contact us today.