IEEE report on synthetic data: privacy risks, metrics, and governance.

The review below is part summary, part author's notes on the points that deserve the most attention. The full report is available from IEEE.¹

What is synthetic data

Synthetic data is generated by training a machine-learning model on a real, potentially sensitive, dataset and then sampling new artificial records from it. Unlike traditional data protection methods such as pseudonymization or k-anonymity, synthetic records have no determinable one-to-one mapping back to real individuals. This property makes synthetic data appealing as a form of privacy-enhancing technology.

However, many ways remain in which synthetic data can still present privacy risks. These include leaking general facts about rare categories or value ranges, memorization or overfitting of the generative model, and dataset characteristics such as small size and outliers.

When generating synthetic data, two properties are always in tension:

• Utility: how usefully the synthetic data reproduces the original's patterns.
• Privacy: how well it avoids leaking information about real subjects.

Increasing privacy often means reducing utility (and other properties such as fairness are also affected), so each use case requires tuning the privacy-utility tradeoff. Common use cases include internal and external data sharing, data augmentation, reducing bias, balancing minority classes, software testing, cloud migration, and data retention after originals must be deleted. Each of these requires a different balance between privacy and utility.

While many well-established methods are available to measure utility, measuring privacy risks is more complicated and requires dedicated algorithms. Even when a suitable way of measuring residual risk is available, deciding whether that risk is low enough remains a challenge.

Synthetic data vs. anonymous data

Whether a particular synthetic dataset qualifies as anonymous under GDPR or a comparable regulation depends on the nature of the data, the applicable legal framework, the use case, the access controls in place, and the residual risk.

What is emerging as best practice is an adversarial evaluation approach. Rather than relying on structural properties of the synthesis model or on theoretical privacy guarantees, adversarial evaluation simulates attacks on the synthetic data to measure the actual information leaked about individuals in the original dataset. Adversarial evaluations, as opposed to more traditional distance-based metrics such as measuring similarity between original and synthetic records, are more interpretable: they provide metrics that are closer to regulatory requirements.

The Article 29 Working Party's 2014 opinion on anonymization² establishes three kinds of risks as essential to evaluating anonymization:

• Singling out: isolating a unique record.
• Linkability: connecting records of the same person across datasets.
• Attribute inference: deducing an unknown sensitive attribute.

A thorough empirical analysis using adversarial, legally aligned methods can identify which risks are dominant and inform the most effective mitigation strategy. Best practices for empirical risk assessment include: repeating evaluations and aggregating results; evaluating the actual dataset that will be released, not a subsample; not assuming results generalize to other datasets; and documenting assumptions, parameters, model cards, and versions for traceability and auditing.

Empirical risk measures provide the necessary ingredient to decide whether a particular synthesis configuration reduces risk to an acceptable level. Which level of residual privacy risk is acceptable cannot be established a priori. A suitable threshold must reflect data sensitivity (stricter for health and special-category data), the surrounding governance and access controls, and must be revisited over time.

Several mitigations can reduce risk, including privacy-preserving machine learning (differentially private training), regularization to combat overfitting and memorization, and combining synthesis with Statistical Disclosure Control (SDC) techniques like redaction, masking, generalization, and pseudonymization of direct identifiers.

Synthetic data lifecycle and governance

Synthetic data should fall into existing data-governance processes rather than a wholly separate regime. Overly burdensome rules may discourage organizations from using synthetic data and push them toward alternatives with weaker privacy properties. A workable governance process should include a streamlined review path for synthetic data that offsets the cost of generation and evaluation, rather than imposing a full data management review on every synthesis project regardless of risk level.

From a governance perspective, a synthetic data project starts when the source data is ingested, and the privacy obligations that govern the source data apply throughout the synthesis process.

The pre-synthesis phase is perhaps the most critical. It is then that the source data is assessed, permitted uses are established, synthesis parameters are defined, and the intended use case is documented. Lawful basis for processing, data lineage, permitted purposes, and feature selection are all determined at this stage, before a single synthetic record is generated.

After that, the generative model is trained and the synthetic data is created. Synthetic data should be tracked and tagged with metadata: source, technique, privacy and utility configurations, intended use case, and validity period, as well as being assigned the correct access-control rules.

Vendors of synthetic data generation platforms add a further complication. Organizations that procure synthesis capabilities externally are reliant on vendor methodologies to determine the privacy properties of the output. The governance framework must account for this: vendor capabilities should be assessed as part of third-party risk management, and responsibilities must be formalized through appropriate data processing agreements.

The case for a formal standard

The report recommends that the IEEE pursue a formal standard for structured synthetic data privacy, issued through a standard project authorization request. This would establish agreed-upon definitions, minimum evaluation requirements, and a shared vocabulary that organizations, regulators, and vendors can reference in contracts, compliance documentation, and regulatory submissions.

In the absence of a standard, the report itself provides what is currently available: a defensible methodology and a set of best practices grounded in the current state of research and regulatory guidance. For organizations deploying synthetic data in regulated industries, it is a useful reference for internal governance decisions and for explaining to regulators what evaluation was performed and why.