DP fine-tuning leaks when data matches pretrain
New benchmarking tests whether differential privacy during Large Language Model adaptation actually protects data. It finds leakage increases when adaptation data matches pretraining distribution; even at ε=8, membership inference remains strong. LoRA reduces risk for out-of-distribution data; prefix tuning leaks canaries. Attackers with matched shadow models gain significant advantage.
Enterprises increasingly slap differential privacy (DP) on their Large Language Model (LLM) fine-tuning and hope it seals the data. This benchmark checks whether that holds once you account for what the base model learned in pretraining. Short version: leakage depends less on DP theory and more on how close your adaptation data looks to the pretrain set.
How the attacks land
The authors run state-of-the-art membership inference and canary extraction against DP adaptations across open Pile-trained families. Robust Membership Inference Attack (RMIA) works offline: train a shadow model that shares the same pretrained base, score candidate samples, then compare confidence patterns to decide who was in the adaptation set. When the attacker can match the base model, RMIA bites hard; power drops fast without that shadow.
They sweep the adaptation data from exact overlaps, through in-distribution (IID) splits, to out-of-distribution (OOD) sets. Distributional closeness is the big lever. IID samples leak about as much as overlapped ones, even when there is no literal duplicate. With a typical DP budget (for example ε=8), IID adaptations still show substantial attack success, with RMIA AUC roughly 0.7–0.9. OOD is safer, not safe.
Method matters
They compare full fine-tuning, last-layer head tuning, parameter-efficient LoRA, and prefix tuning under DP-SGD where applicable. LoRA gives the strongest empirical protection on OOD data while keeping utility competitive. Head and full fine-tuning are more exposed. Prefix tuning stands out for canary exposure and k-token extractability, though in some high-privacy regimes it can reduce memorisation of pretraining data via its input-space perturbation.
Privacy accounting also matters. The empirical bar for “this actually resists attacks” is tight: results indicate ε ≤ 0.1 is needed to rein in leakage. Moderate ε leaves meaningful signal for both membership inference and canary-based extraction. Computationally, head tuning is fastest, LoRA is a practical middle ground, and full fine-tuning is the most expensive.
Two angles linger for enterprise teams. First, attacker capability is realistic: if an adversary can obtain the same open pretrained base, RMIA becomes a strong audit and attack tool. Second, most risk lives in distributional similarity; your adaptation choices only help so much if your data sits close to the pretrain pile. The work closes by sketching a broader, pretrain-to-adapt privacy assessment framework. Closed models were out of scope, so the picture there is still fuzzy.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models
🔍 ShortSpan Analysis of the Paper
Problem
This paper examines whether differential privacy (DP) applied during adaptation of pretrained large language models (LLMs) actually prevents leakage of adaptation data in practice. Pretraining can create overlaps and interdependencies with downstream data that may undermine DP guarantees, yet systematic empirical evaluation across realistic pretrain–adapt relationships is lacking. The work seeks to quantify membership and extraction risks after DP adaptations and to clarify how data distribution, adaptation method and attacker knowledge affect real-world privacy.
Approach
The authors build a benchmark that applies state‑of‑the‑art attacks to DP adaptations of multiple open models trained on the Pile and related open models. They vary the adaptation data distribution along a spectrum from exact overlap with pretraining data, through in‑distribution (IID) validation splits, to out‑of‑distribution (OOD) datasets. Adaptation methods evaluated include full fine‑tuning, last‑layer (head) fine‑tuning, parameter‑efficient fine‑tuning (LoRA), and prefix tuning, each trained with DP‑SGD where applicable; DP prompting and in‑context approaches are also discussed. Attacks comprise the Robust Membership Inference Attack (RMIA) in an offline setting, reference and Min‑K% baselines, and canary insertion for data‑extraction exposure and k‑token extractability. Metrics include AUC for membership inference, exposure scores for canaries, and utility proxies such as validation loss, perplexity and Rouge‑1.
Key Findings
- Distributional closeness strongly drives practical leakage: adaptation data closer to the pretraining distribution has higher membership‑inference vulnerability even without direct overlap; IID validation samples leak as much as overlapping training data.
- Under moderate DP budgets (example ε=8) IID adaptations still show substantial attack success (RMIA AUC typically between roughly 0.7 and 0.9), whereas OOD adaptations are less vulnerable though not immune.
- LoRA (a PEFT method) consistently yields the best empirical privacy‑utility trade‑offs, providing relatively low MIA AUC for OOD data while keeping competitive utility; head and full fine‑tuning are often more vulnerable.
- Prefix tuning is among the most vulnerable to canary exposure and extraction attacks but can reduce memorization of pretraining data in some high‑privacy settings, likely owing to its input‑space perturbation.
- Attacker knowledge matters: RMIA is most powerful when the adversary has a shadow model sharing the same pretrained base; effectiveness drops rapidly when the attacker lacks closely matched shadow models.
- Practical protection requires tight privacy bounds: empirical results indicate ε ≤ 0.1 is needed to limit attacks effectively; moderate ε values can still permit meaningful leakage.
- Computational trade‑offs were measured: head fine‑tuning is fastest, LoRA is moderately efficient, and full fine‑tuning is most expensive, making LoRA attractive for sensitive adaptations.
Limitations
The benchmark focuses on open models and Pile‑trained families; closed large models could not be evaluated due to API and dataset opacity. The work centres on auditing adaptation stages and post‑adaptation effects on pretraining memorization rather than proposing new mitigation algorithms. Some evaluations depend on available validation splits and model families, limiting generalisability to undisclosed pretraining data or closed architectures.
Implications
An attacker can exploit distributional similarity to the pretrained corpus to improve membership inference and exfiltrate inserted secrets even when DP is applied during adaptation; having access to the same pretrained base as a shadow model greatly increases attack power. In practice, an adversary could focus on adapting attack strategies to exploit IID‑like targets or use canary prefixes to test memorization. The results suggest that attackers do not need exact overlap to succeed and that choosing adaptation method and a tight privacy budget are actionable levers for attackers to assess and amplify leakage risks.
