Privacy risks evaluation at scale: NIST's Collaborative Research Cycle

The NIST Collaborative Research Cycle (CRC) is a benchmarking initiative that ran from 2023 to 2025.¹ It provided curated datasets and invited researchers and practitioners to submit de-identified versions for evaluation. The first iteration of the CRC collected over 350 de-identified versions of datasets derived from the 2019 American Community Survey, produced by more than a dozen synthesis libraries. This gave us a rare opportunity: a controlled, comparative evaluation of actual privacy risks across a large number of algorithms and datasets, at a scale that individual practitioners rarely have access to.

Co-authored with Nicola Vitacolonna, the study was presented at the Privacy Data Management Conference in December 2023.² We applied Anonymeter to evaluate the privacy risks of 167 dataset-algorithm combinations across 12 synthesis libraries. The datasets cover three geographic subsets of the ACS: Massachusetts (7,634 records), Texas (9,276 records), and a National sample (27,254 records). Read the paper.

Risk levels across algorithms and datasets

The distribution across risk types is consistent with the framework established in the Anonymeter paper.³ Linkability is substantially lower than inference and singling out across all datasets and algorithms. Synthesis breaks the one-to-one mapping between individuals and records, which structurally weakens linkage attacks. Inference and singling out risks do not depend on record-to-record matching and persist as the dominant residual threats.

Across algorithms, the variance is significant. Sub-sampling, which releases a fixed percentage of original records rather than synthesizing new ones, produces the highest measured risks: approximately 30 percent, consistent with the theoretical expectation for a 40 percent release rate. While some synthesis algorithms show relatively high risk levels around 10 percent, the majority clusters between 2 and 5 percent.

The practical implication is direct: algorithm choice matters, and it can matter by an order of magnitude. Choosing between two plausible synthesis options for the same dataset without empirical evaluation is choosing blind on a decision that has real privacy consequences.

The aggregated risks are different for the different datasets, even when accounting for differences in sample size. This suggests that some groups are harder to protect than others.

Feature selection also matters. Restricting datasets to demographic-focused features substantially reduces linkability risk. For the National dataset, the same restriction increases inference and singling out risk. This means that data minimization principles must still be followed. There is no universal configuration that minimizes all risk types simultaneously, which reinforces the case for empirical evaluation tailored to each use case.

Privacy-utility trade-off and differential privacy

The CRC submissions included utility evaluations from SDNIST, using a k-marginal metric that measures the divergence between real and synthetic data distributions across multidimensional feature combinations. Mapping privacy risks against utility scores reveals a trade-off, but not a simple one.

Some libraries achieve both reasonable utility and relatively low privacy risk. Others reduce risk primarily by sacrificing utility. Sub-sampling sits at the extreme: high risk, high utility, since the data is real and the utility is trivially preserved. The optimal point in the privacy-utility trade-off is use-case specific and requires knowing where each algorithm, and its synthesis parameters, lands on both dimensions for the specific data in question.

For submissions that use differential privacy (DP), we measured the correlation between the theoretical privacy budget (epsilon) and the empirical risks from Anonymeter. For tight epsilon values below approximately 5, empirical risks are low and increase slowly. As the DP guarantee loosens above an epsilon of 10, Anonymeter detects substantially higher risks. The degree of correlation varies across libraries: for RSynthpop, the relationship is strong and empirical risks climb steeply with epsilon; for other libraries, the correlation is weaker and the DP budget is a less reliable predictor of actual behavior.

This matters for how DP guarantees are used in practice. A theoretical epsilon value bounds information leakage under worst-case conditions. The empirical risk evaluation by Anonymeter reflects what is measured in a specific deployment, for a specific generated dataset.

What this means for procurement and deployment

The key conclusion from this study is that synthetic data is not a uniform category with a single privacy level. Residual risks depend on the synthesis algorithm, its configuration, the nature and size of the source data, and the features included in the dataset.

A couple of percentage points of residual risk represents the current state of the art for well-configured synthesis algorithms on real-world datasets. Whether that level is acceptable depends on the use case, the regulatory context, and the organizational controls applied to restrict access to the synthetic data. It cannot be assumed to be zero, and it cannot be claimed to be zero without empirical evaluation of the specific dataset and configuration.

The diversity of results across the CRC submissions is itself a finding. Organizations that select synthesis algorithms based on vendor claims or theoretical guarantees alone are missing the variance that is observable only through evaluation.