Hello!
Last month, we launched Data Security and Privacy for AI: five capabilities that protect sensitive data as it moves into AI tools, models and agents. September showed why it matters.
A few weeks later, OpenAI disclosed that some of its own research agents had moved data out of internal training systems, including images from ChatGPT users. Around the same time, a server flaw at Gyazo exposed text that had been pulled out of millions of screenshots, the kind of data most inventories never list.
Two different companies, but they leave behind one question for every company reading this: do you know what sensitive data sits in your systems and pipelines, and what can reach it?
That’s where we start this month.
- When the helpers went off-script: OpenAI agents posted ChatGPT users’ images from its training pool
- Screenshots remember more than people think: a Gyazo server flaw exposed 23.6 million user records
- Two more from this month: Revolut and CenterPoint Energy
- Clean data in, safe model out: how Training Data Guardrails keeps sensitive values out of the model
When the Helpers Went Off-Script
OpenAI agents posted ChatGPT users’ images from its training pool
During a review that followed the Hugging Face incident earlier this year, OpenAI found that some of its research agents had sent data from internal training and testing systems to outside services. On September 25, it disclosed that this included 53 images from ChatGPT users who had not opted out of model training, posted to image-hosting sites as unlisted links. Enterprise and business data is excluded from its training by default. OpenAI says it has taken down most of them and is working on the rest.
Source: BleepingComputer
How to keep this out of your own pipeline
- If your teams train or fine-tune models, treat the training pool like any other data store. Anything that can read it can also move it.
- Scan training and evaluation sets for personal and confidential data before anything reads them.
- Mask or remove sensitive values before data enters the pool, so there’s far less to lose if a pipeline, agent or evaluator reaches it later.
- Limit what AI agents and test harnesses can reach, and keep a record of what they did with it.
Training Data Guardrails | AI Development Guardrails
Screenshots Remember More Than People Think
Gyazo server flaw exposes 23.6 million user records
On September 11, attackers exploited a server vulnerability at screenshot tool Gyazo and took about 23.6 million user records, including names, email addresses, password hashes, session IDs and integration tokens. They also took 490 million image metadata records with upload IP addresses, location data and text Gyazo had extracted from screenshots using OCR. Gyazo fixed the flaw the next day, notified affected users and brought in outside investigators. It hasn’t said which system or environment the data came from.
Source: BleepingComputer
How to shrink what an attacker can take
- Map where customer data lives, including the copies that pile up around it: OCR text, logs, analytics extracts. OCR output is machine-generated data, the same kind of derived copy AI pipelines create, and the first to drop off an inventory.
- Keep real data out of places that don’t need it by masking it in development, test and analytics environments.
- Put a database firewall in front of production so unexpected connections and bulk reads get blocked or flagged.
Data Discovery & Classification | Static Data Masking | Database Firewall
Two More From This Month
- A convincing email opened the vault at Revolut. An attacker posing as a government agency asked for customer data and received it, including ID documents and transaction history. BleepingComputer
- CenterPoint Energy’s API had no speed limit. 7.49 million customer records were pulled through a public API with no rate limiting. BleepingComputer
Clean Data In, Safe Model Out: Training Data Guardrails
The OpenAI story is the problem Mage Data’s Training Data Guardrails was built for: sensitive values sitting in training data that more and more systems can read. Once sensitive data is inside a model, you can’t reliably take it out.
- How it works: Training Data Guardrails scans training and fine-tuning data, from database tables to PDFs and call transcripts, and masks personal and confidential values before the model sees them.
- Why it matters: The masked data stays realistic, so the model still learns what it should.
It’s one of five capabilities in Data Security and Privacy for AI. For the other four (AI Usage Guardrails, Dynamic Data Masking for AI, AI Development Guardrails and Activity Monitoring for AI), our one-minute overview videos are the quickest way in. We’ll take a closer look at each in upcoming issues.
See how Training Data Guardrails works | Book a 30-minute demo
Short Reads
-
Every AI Model Leaves Copies of Production Behind (9-minute read)
New this month. Finding and masking sensitive values before training starts, and what OWASP, NIST and Article 10 of the EU AI Act actually ask for. -
Why Your AI Strategy Has a Data Security Problem (5-minute read)
Missed it in August? Where the copies of production data go, why runtime guardrails alone can’t catch them, and how to protect data before training starts.