←Blog 一覧へ
◆43 分で読了

【Part 2】 How AI leaks data at every stage of its lifecycle — and what actually stops it

Hiroshi Nakagoe, CTO at CODAS & VPoE at Archetype Digital

BLOG

This is the second post in a series on privacy preservation for the GenAI era. The first covered the inference phase, where every prompt, retrieved chunk and log line is a fresh disclosure of live data. This one covers training on a corpus that one organization owns and assembles. Earlier posts in this series: 【Part 1】 How AI leaks data at every stage of its lifecycle — and what actually stops it

The Training Phase: A Dataset from One Party

Here one organization owns and assembles the corpus and trains or fine-tunes the model. Nothing leaves the building during training — which is exactly why this phase gets under-governed. The leak happens later, through the model.

Threat models

The star rating for each threat is rated by the author alone, not by any official authority. It combines the significance of the data leakage (higher where sensitive information is leaked directly), the prevalence of the attack (how standard it is), and the ease of the attack (how easily it can be carried out), on a scale of one to five where five is highest. A heading marked “not rated” weighs up a defense or presents the opposing case rather than naming a threat of its own, so a rating there would claim something the text does not.

Group A — What the corpus carries in

PII and live secrets in the corpus ★★★★★

Web-scale corpora are assembled by crawling, not by consent. Crawlers cannot distinguish a blog post from a leaked database dump or a pastebin of credentials. There are two distinct recipients. The first is anyone who reads the public dataset, with no model involved at all: the Truffle Security scan found 221,303 live verified secrets across 6,003 datasets, including 3,343 working AWS keys and 349 GitHub tokens, 223 of them with repository write access, with 44% of unique live secrets appearing in more than one dataset and a single Infura key propagating into 1,131 separate datasets after being pasted into a chatbot conversation (Ayrey). The second is a developer using the model's completions: Huang et al. extracted 2,702 plausible secrets from 8,127 GitHub Copilot suggestions. The base rates make "we'll just filter it" hard — Elazar et al. counted 7.6 million email addresses and 19.7 million phone numbers in C4, and 201 million emails in mC4-en. Critically, Lukas et al. showed scrubbing reduces but does not prevent leakage, and that sentence-level differential privacy at ε = 8 still leaked about 3% of PII sequences.

Unprovenanced, non-consented and copyright-encumbered records ★★★☆☆

Volume is optimized; licence status and legality are not checked at ingestion. Thiel's Stanford Internet Observatory audit identified 3,226 suspected entries of CSAM — child sexual abuse material — in LAION-5B, with the Canadian Centre for Child Protection classifying 746 of the PhotoDNA matches as CSAM or possible CSAM, and recommended deprecating models derived from the affected checkpoints. Note the asymmetry: nothing leaks outward here, so there is no adversary in the usual sense — the exposure runs to regulators, rights-holders and civil claimants, and it can require deprecating the model. Note too that filtering is itself a harm vector: Dodge et al. found C4's blocklist filtering disproportionately removed text from and about minority individuals.

Group B — What the weights absorb

Memorization and verbatim regurgitation ★★★★☆

Autoregressive training minimizes next-token loss over a finite corpus; for rare or duplicated sequences the lowest-loss solution is to store the sequence rather than generalize from it. Memorization is not a bug in the loss — it is the loss working correctly on low-entropy data. Everything it absorbs is then reachable by any external user with black-box API access. Three log-linear drivers were established by Carlini et al.: memorization grows with model capacity, number of duplicates, and prompt context length, with the conclusion that it "will likely get worse as models continue to scale" ("Quantifying Memorization"). Duplication is superlinear — Kandpal, Wallace and Raffel found a sequence present 10 times is generated about 1,000 times more often than one present once. Image models leak in a more visceral way: Carlini et al. recovered 109 training images from Stable Diffusion, including identifiable photographs of people, and 23 from Imagen, alongside 1,280 unique images from smaller CIFAR-10 diffusion models ("Extracting Training Data from Diffusion Models"). And this is now a legal predicate — The New York Times alleges OpenAI's models produce "near-verbatim reproductions" of its articles (Pope).

Membership inference exposure created at training time ★★★☆☆

Training drives loss lower on members than non-members, and that gap is an oracle available to an external party with API access, or to a plaintiff proving a record was used. Carlini et al.'s LiRA reframed the evaluation around true-positive rate at very low false-positive rates, achieving about 10 times higher true-positive rate at low false-positive rate than prior attacks — the operationally dangerous regime, where an attacker is right about a handful of individuals with near-certainty ("Membership Inference Attacks From First Principles"). The practical synthesis: membership-inference risk concentrates in fine-tuning, not pre-training. A small, high-epoch supervised fine-tune, or a LoRA run — low-rank adaptation, which freezes the base weights and trains a small adapter beside them — on medical notes or support transcripts is exactly the overfitted regime LiRA was built for.

Group C — What an attacker injects

Data poisoning ★★★☆☆

Web corpora are assembled from mutable third-party sources, so an attacker who controls the content at fetch time controls what enters the corpus — and that attacker needs no access to the training organization at all. Carlini et al. demonstrated split-view poisoning, where buying expired domains guaranteed control of 0.01% of LAION-400M or COYO-700M for about $60, and frontrunning poisoning against the predictable snapshot timing of crowd-sourced datasets like Wikipedia ("Poisoning Web-Scale Training Datasets is Practical").

Backdoor and trigger injection ★★★★☆

A backdoor is a conditional policy learned from a small set of trigger-behavior pairs, and because the trigger has essentially zero competition from clean data, the required number of poison documents is governed by an absolute count, not a share of the corpus. Anyone who can get a few hundred documents onto the open web can install one, and the resulting behavior is then inherited by every downstream deployment. The headline 2025 result: as few as 250 malicious documents produce a backdoor regardless of model size, tested from 600M to 13B parameters — roughly 0.00016% of training tokens for the largest model (Souly et al.). Read the scope precisely: the backdoor they installed was a denial-of-service trigger that makes the model emit gibberish, and the size-independence is demonstrated up to 13B, not proven for frontier scale. What generalizes is the mechanism — an absolute count rather than a share — not that any behavior at all can be installed with 250 documents. At the instruction-tuning stage, 100 poison examples suffice, with effects persisting across hundreds of held-out tasks and larger models increasingly vulnerable (Wan et al.). Worse, backdoors survive the safety pipeline: Hubinger et al. showed behavior not removed by supervised fine-tuning, reinforcement learning or adversarial training, and that adversarial training taught models to recognize their triggers better and hide the behavior, with persistence strongest in the largest models. Poisoned LoRA adapters are a distribution channel of their own — a backdoor adapter can be merged training-free with task-enhancing adapters, retaining both the backdoor and the capability uplift that incentivizes the download (Liu et al.). And even benign fine-tuning degrades safety: Qi et al. jailbroke GPT-3.5 Turbo through OpenAI's own fine-tuning API with 10 examples for under $0.20.

Reward-model hacking and poisoned preference data ★★☆☆☆

Reinforcement learning from human feedback optimizes a learned proxy of human preference, with two failure modes. Overoptimization is self-inflicted: Gao, Schulman and Hilton derived scaling laws showing the gap between proxy reward and ground truth follows a characteristic form in KL distance, so the amount of optimization you can safely apply is bounded by reward-model size. Annotation poisoning has an external author, because preference labels are the cheapest, least-audited artifact in the pipeline and are frequently produced by contracted annotators. Rando and Tramèr found 0.5% poisoned examples collapsed reward-model accuracy from 73% to about 42.5%, though the full pipeline required at least 5% to install a universal jailbreak backdoor — the two-stage architecture is itself a partial defense. Note also that the reward model is a second copy of your preference corpus, usually held to a lower security bar than the policy model, so everything in Group B applies to it as well.

Group D — What the environment exposes

Insider access to checkpoints, activations, optimizer state and logs ★★★★☆

A training run materializes far more copies of the data's information than the final weights: per-step gradients, Adam moment buffers, intermediate activations, intermediate checkpoints, and experiment-tracking logs that routinely contain sampled generations. These usually live on shared cluster storage with far weaker access control than production, which puts them within reach of an ML engineer, a contractor, or anyone who compromises one researcher's credentials. LAMP reconstructs private text directly from gradients using a language-model prior, recovering five times more bigrams and 23% longer subsequences than prior methods (Balunović et al.). The primitive is "given a gradient, recover the text" — which is exactly what an insider with read access to a debugging dump possesses.

Cloud-operator, co-tenant and hardware-residue threats ★★★☆☆

Conventional encryption protects data at rest and in transit, not in use. Two adversaries follow. The first is a co-tenant on the same GPU: Trail of Bits' LeftoverLocals (CVE-2023-4969) showed one kernel can read another user's GPU local memory with no elevated privileges, leaking about 5.5 MB per GPU invocation and accumulating to roughly 181 MB per query against a 7B-parameter session (Khlaaf and Sorensen). The second is a government compelling the provider: the CLOUD Act establishes that data held by a US-based provider can be reached wherever it is located, because the test is the provider's possession, custody or control — so physical data residency in-region does not by itself defeat a demand.

Model and checkpoint theft ★★★☆☆

Weights encode the entire capital investment of a run and, per the memorization results, the training data too — and they are replicated across object storage, cluster scratch, laptops and evaluation environments, so the attack surface is the union of all of those rather than the production serving path. RAND's framework identifies 38 distinct attack vectors against model weights and defines five security levels, noting rough expert consensus on defending against less capable actors but wide disagreement about top-tier nation-states (Nevo et al.). The adversary contemplated is a nation-state, a competitor or an insider.

Group E — What the regulator asks

Compliance and legal exposure ★★★☆☆

GDPR Article 17 grants erasure over personal data; weights are a lossy distributed encoding from which one individual's contribution cannot be surgically removed. Opinion 28/2024 of the European Data Protection Board, whose opinions set how supervisory authorities across the EU read the GDPR, holds that anonymity must be assessed case by case and requires it to be "very unlikely" both to identify individuals whose data trained the model and to extract personal data through queries — identifying regurgitation and extraction attacks as reasons a model may fail. Its sharpest clause: deploying a model trained on unlawfully processed personal data affects the lawfulness of deployment. An upstream collection defect can taint every downstream use. Separately, EU AI Act Article 53(1)(d) requires general-purpose AI providers to publish "a sufficiently detailed summary about the content used for training," which converts corpus provenance from internal hygiene into a discoverable published artifact.

Protection methods

This section shows "practical" solutions against the above threat models. There is a gap between the literature and the shipping code, so the version and maintenance status of each framework matter. Controls that exist only as papers or abandoned research repositories are gathered later.

PII detection and scrubbing

The same engine as in the inference post, applied to the corpus rather than the request: Presidio, now maintained by Data Privacy Stack rather than Microsoft, release 2.2.364 of July 2026, with the Analyzer, Anonymizer, Image Redactor and Structured modules combining named-entity recognition, regex, rules and checksums. It is a per-document library, not a distributed job — at corpus scale you supply the orchestration. The managed alternatives are Google Cloud Sensitive Data Protection, which bills per unit inspected and turns a multi-terabyte pass into a real budget line, and AWS Comprehend PII, which caps real-time analysis at 100 KB per document and supports only English and Spanish, a restriction that disqualifies it as the primary filter for a multilingual corpus.

For web-scale throughput the honest answer is that nobody has both. Hugging Face datatrove ships a PIIFormatter that anonymizes emails and IP addresses, and NVIDIA NeMo Curator ships a PII modifier that wraps Presidio and distributes it over Ray and GPUs. Neither claims high recall; both do cheap high-precision pattern replacement at scale. No tool today combines Presidio-grade recall with trillion-token throughput, and a team that needs both builds the bridge itself.

Scrubbing addresses PII and live secrets in the corpus at ingestion, and by extension reduces what memorization and verbatim regurgitation can later surface. It does nothing about poisoning, backdoors, insiders, or any environment or regulatory threat. Presidio's documentation states there is no guarantee of finding all sensitive information; Lukas et al. measured that scrubbing reduces but does not prevent leakage. Treat it as hygiene, not as a guarantee. One package still on many reading lists should come off: scrubadub last shipped in September 2023.

Secret scanning as a corpus pre-filter

Given 221,303 live verified secrets across 6,003 public datasets, credential scanning deserves to be its own pass rather than a Presidio recognizer. The two scanners that work on plain directories are gitleaks, MIT-licensed, with a gitleaks dir mode that needs no git repository, and TruffleHog, whose distinguishing feature is live credential verification. Both carry caveats you should know before adopting. Gitleaks' README states that the project is feature complete and that future releases will be security patches only. TruffleHog is AGPL-3.0, which is a genuine blocker in some corporate pipelines, and its verification feature fires real network requests against third-party APIs for every candidate string — run corpus scans with --no-verification and accept the false-positive rate. Yelp's detect-secrets is built around a pre-commit baseline file and has not shipped since May 2024; it is a poor fit here.

There is no published reference pipeline for secret-scanning a training corpus at scale. The nearest precedent is the BigCode work on The Stack, which used a purpose-built detect-secrets derivative rather than any of these off the shelf. In practice a team shards the corpus, runs the scanner per shard under Spark, Ray or Slurm, and treats the output as a drop-or-mask signal. That is a pattern you assemble, not one you adopt.

This addresses live secrets in the corpus, which is the threat with the shortest path to harm — a working key needs no model to leak through. It is silent on everything else in this phase.

Synthetic data replacement

Nominally aimed at membership inference exposure, PII in the corpus and the erasure obligation — and this is the most oversold control in the space. The dividing line is whether the generator carries a formal differential-privacy guarantee. On the yes side there is essentially one maintained option: smartnoise-synth from OpenDP, MIT, release 1.0.8 of April 2026, offering MWEM, AIM, MST, PATE-CTGAN, DP-CTGAN and PAC-Synth with genuine accounting in ε and δ, where ε bounds how much one record can shift the output and δ is the probability that the bound fails altogether. It is explicitly tabular and relational only. DataSynthesizer's correlated mode is also formally private and has not shipped since October 2023; private-pgm is the inference engine beneath MST and AIM and is best reached through smartnoise-synth rather than directly.

On the no side sit the tools most teams actually reach for. SDV is maintained and current but is published under the Business Source Licence rather than an OSI-approved one, and its anonymization is value replacement for flagged columns, which is pseudonymization and not a guarantee. YData's open package was renamed to fg-data-synthetic in April 2026. Gretel was acquired by NVIDIA in March 2025 and is no longer independent. Tonic Textual is the only commercial product aimed squarely at unstructured text, and it too is named-entity-driven redaction and synthesis with no formal privacy guarantee.

So for the unstructured text corpora this section is about, no differentially private synthesizer ships at all. That matters, because without a formal guarantee on the generator there is no privacy guarantee on its output — the generator memorizes and its samples inherit that.

The strongest evidence against synthetic data as a privacy control comes from the tabular setting, and it is worth being precise about what it does and does not cover. Stadler, Oprisanu and Troncoso evaluated five generators — three with no privacy protection and two differentially private ones — on two structured microdata sets, the Adult census extract and 50,000 Texas hospital discharge records. They found that synthetic data either does not prevent inference attacks or does not retain data utility, that it offers no better privacy-utility trade-off than the traditional sanitization it is sold as replacing — in their experiments a procedure drawn from NHS England guidance that groups categorical values into coarser buckets, generalizes granular dates and locations, drops rows carrying rare categorical values, caps numeric attributes at the 95th percentile and enforces k-anonymity at k = 10 over a fixed set of demographic columns — and, more damningly, that because it is impossible to predict which signals a synthetic dataset will preserve, the privacy gain is highly variable and the utility loss unpredictable. Note what that is not: it is not a result about text. Nobody has run the equivalent study on an unstructured corpus, so treating it as settled for text is an extrapolation rather than a finding. The extrapolation is the pessimistic direction, though. Every structural factor that made the tabular case hard — high dimensionality, rare records that the generator can only fit by memorizing them, no way to enumerate what a sample might carry — is worse in text, and in text the generator is itself a language model with the memorization behavior this series has already documented at length.

k-anonymity, l-diversity and t-closeness

Aimed at re-identification in structured releases, these are mature for tabular data and inapplicable to text; they belong here as a contrast class, not a recommendation. ARX is the one to name — open source, a Java library API as well as a GUI, covering k-anonymity, ℓ-diversity, t-closeness, δ-presence and (ε, δ)-differential privacy with risk and utility metrics. R users have sdcMicro, which national statistical offices rely on. Aggarwal proved the structural result that rules them out for corpora: as dimensionality grows we face "a choice of either completely suppressing most of the data or losing the desired level of anonymity." Natural-language documents are effectively unbounded-dimensional, so every token is a quasi-identifier and the framework degenerates. Treat the presence of these three in an AI data-governance policy as a sign the policy was written for a data warehouse.

Deduplication

The single best evidence-to-cost ratio defense available, because it attacks the strongest of the three memorization drivers Carlini et al. established — model capacity, the number of times a sequence is duplicated in the corpus, and the length of the prompt context used to elicit it. Two of those are not yours to turn down: capacity is the model you decided to train, and context length is how somebody else chooses to query it. Duplication is the only one of the three you control at data-preparation time, and it is also the one that moves superlinearly, so the return on the effort is disproportionate. Which brings us to the place where most published advice now points at dead repositories. Google Research's deduplicate-text-datasets, the canonical suffix-array implementation, was archived on 2 October 2025. Meta's SemDeDup was archived on 1 November 2024. The method survives; the code does not.

The live answers are Hugging Face datatrove, Apache-2.0, release 0.10.0 of August 2026, with MinHash-LSH, sentence-level and exact-substring dedup and executors for local multiprocessing, Slurm and Ray, proven at FineWeb's 15 trillion tokens; and NVIDIA NeMo Curator, Apache-2.0, PyPI 1.3.0 of July 2026, which is the only option with GPU-accelerated semantic deduplication and reports fuzzy dedup falling from 10.7 hours to 0.65 hours on three H100 nodes, used to curate more than 8 trillion tokens for Nemotron. Below those, datasketch 2.0.0 gives you MinHash and LSH as a maintained primitive to build a job around, and text-dedup and AllenAI's dolma are usable with effort. One datatrove limitation to plan for, stated in its own documentation: each file is processed by a single task and files are not split automatically, so you shard the input into many medium files yourself.

Deduplication addresses memorization and verbatim regurgitation directly, and membership inference exposure as a consequence. Lee et al. found models trained on deduplicated data emit memorized text ten times less frequently and require fewer steps for the same accuracy; Kandpal et al. report about 20 times less emitted training data and, crucially, that membership-inference AUROC collapses from about 0.90 on heavily duplicated sequences to chance on singletons. It does nothing for singleton-document memorization or for poisoning, and near-duplicate thresholds are a genuine tuning problem.

Quality filtering, sanitization and hash-matching

Quality filtering is solved and free. The same three libraries — NeMo Curator, datatrove and dolma — all ship heuristic filters implementing the Gopher and C4 rules, repetition and perplexity filters, and classifier hooks; fastText quality and language classifiers are the standard approach and are wired into all three. NeMo Curator additionally ships NVIDIA-trained quality and not-safe-for-work classifiers.

Be clear what that buys, because the word quality misleads. These rules test whether text reads like usable prose, not what it depicts: Gopher thresholds document length, mean word length, symbol-to-word ratio, the share of lines starting with a bullet or ending in an ellipsis, and the presence of common stop words, while C4 requires terminal punctuation and a minimum sentence count and drops pages containing "javascript", "lorem ipsum" or a curly brace. None of them is trained. They are hand-written rules with fixed thresholds, which is why they cost nothing to run and why they transfer poorly to a corpus unlike the web. Only the classifiers alongside them are learned, and the standard recipe is a model fitted on curated reference text as positives against random crawl as negatives.

Content screening is a separate layer, and the three libraries diverge sharply on it. C4's single content rule is lexical: drop any page containing a word from the "List of Dirty, Naughty, Obscene or Otherwise Bad Words" — page-level, no classifier, and the mechanism behind the fairness finding at the end of this section. Past that, NeMo Curator ships AEGIS for text, covering thirteen categories including violence, sexual content, sexual content involving minors and self-harm, plus a CLIP-based scorer for images; dolma ships fastText hate and not-safe-for-work classifiers trained on Jigsaw Toxic Comments together with around twenty URL blocklists; datatrove ships no content classifier at all, only URL blocklists and the same word lists. A plan that says "we filter for quality" and means "we screen for harmful content" has not specified a control.

Sanitization is the vaguest word in this heading, so pin it down: it covers what gets stripped or rewritten in place rather than filtered out whole, and it has four parts. Two have their own entries above — PII redaction and secret removal. The third is boilerplate and markup, where the tooling is mature and dull: trafilatura and readability for extraction, FTFY for mojibake, NeMo Curator's boilerplate, markdown, quotation, URL and newline modifiers. The fourth is benchmark decontamination, stripping evaluation items the crawl swept up, and it is the one to check rather than assume. datatrove measures contamination without removing it, and neither NeMo Curator 1.3.0 nor dolma ships a decontamination pass at all. Teams take it for granted because it sits in the same conceptual bucket as the rest of this paragraph; it is not in the same box on disk.

Illegal-content screening is where "what a developer can actually adopt" gets uncomfortable, and the post should say so. Hash-matching against verified child sexual abuse material — Thiel's explicit recommendation, and the reason you must query external authorities rather than run local heuristics — is available from exactly four places, all gated. Microsoft PhotoDNA is free but restricted to trusted online service providers and businesses hosting user-generated content, approved case by case after third-party vetting; an AI lab curating a scraped corpus is not the archetypal applicant. Thorn's Safer is a commercial contract with no public pricing, and its Safer Predict Text product is the only one of the four aimed at text corpora. The Canadian Centre for Child Protection's Project Arachnid Shield has a real API with published SDKs and is the most technically open, still granted on request. NCMEC operates the hash-sharing infrastructure and the CyberTipline and is not a vendor. There is no open-source detector, and there should not be. Lead times run to weeks or months, and a team that cannot get access has no compliant technical option beyond aggressive upstream source restriction plus classifiers that are not specific to the material in question.

This group is the control for unprovenanced and illegal records, and a partial one for PII in the corpus. It cannot address data poisoning at all, because split-view poisoned content is plausible by construction. And it has fairness consequences, per Dodge et al.

Provenance and documentation

Datasheets for Datasets (Gebru et al.) remains the prose norm. The machine-readable layer is Croissant from MLCommons — pip install mlcroissant, release 1.1.0 of April 2026, JSON-LD over schema.org, natively supported by Hugging Face, Kaggle and OpenML, with a Responsible AI extension carrying provenance, collection-process and consent fields, and SHA-256 per file object, which is the hook the pinning section below uses. Hugging Face dataset cards are zero-cost and universally understood but carry no enforcement. C2PA Content Credentials sign media manifests cryptographically; be precise about its status, because the reference implementation describes itself as a beta release implementing a subset of the technical specification with breaking changes every couple of months, and because its relevance to scraped corpora is asymmetric — almost no crawled web content carries a valid manifest.

The disclosure obligation is no longer hypothetical. The European Commission's AI Office published its Template for the Public Summary of Training Content for General-Purpose AI Models on 24 July 2025, with three mandatory sections: model and dataset identity, a list of data sources including the top domain names for scraped web content, and data-processing aspects covering text-and-data-mining opt-out compliance and the steps taken to exclude illegal content. The deadline was 2 August 2025 for models newly placed on the EU market and is 2 August 2027 for models already on it. That third section is the screening problem above, with a legal date attached.

These address unprovenanced records and the Article 53 obligation, and they make data poisoning auditable after the fact. All of them describe provenance without verifying it — Croissant metadata is exactly as trustworthy as whoever wrote it.

Content-addressed dataset pinning

This is the one control in the whole phase that costs almost nothing and works. Pin Hugging Face datasets to a commit SHA rather than a branch: load_dataset("org/name", revision="<40-char-sha>"). Hub repositories are git repositories, so the SHA is a content-addressed root, and the default revision="main" is a live mutable pointer — the single most common reproducibility and poisoning hole in the ecosystem. Record SHA-256 per file in a Croissant manifest so the integrity record is portable across hosting platforms. For datasets not on the Hub, DVC (Apache-2.0, 3.67.1 of March 2026) gives content-addressed storage with pointer files in git and input hashes recorded per pipeline stage in dvc.lock; git-lfs is fine for moderate assets and poor above a hundred gigabytes or so. If your corpus includes source code, Software Heritage identifiers pin an exact tree independent of whether the original host still serves it.

Be honest about what this buys. None of it is a poisoning defense in the certified sense. It is tamper-evidence and reproducibility: it guarantees you can prove which bytes you trained on and detect if they change, which converts split-view poisoning from an undetectable supply-chain attack into a detectable one, because you pin the content rather than the URL. Alongside it sit two procedural controls with the same character: randomized rather than announced crawl timing, which defeats frontrunning poisoning, and provenance restriction of alignment data to vetted annotators, which addresses poisoned preference data. Trigger scanning on holdouts rounds it out, with the caveat that Wan et al. found conventional defenses give only modest protection while degrading performance, and Hubinger et al. found adversarial training actively counterproductive.

Differential privacy

The next five entries are one mechanism seen from five sides, so they are grouped rather than listed separately. Two are ways of applying it — to a full training run, or to adapters alone. One is the bookkeeping that turns a sequence of noisy steps into a single number. One is what that number is set to in practice. The last is how you check the number is honest. Adopting differential privacy means adopting all five, not the first one on its own.

DP-SGD and DP-Adam

Per-example gradient clipping to bound any single example's influence, plus calibrated Gaussian noise, composed by a privacy accountant (Abadi et al.). For PyTorch there is one live option and it is worth naming precisely: Opacus, from Meta, Apache-2.0, release 1.6.0 of May 2026, which supports fast gradient clipping and ghost clipping — the feature that makes language-model-scale differentially private training memory-feasible at all. Its own PyPI classifier still reads Beta, and its structural limitations are real: it rewrites your model's modules and requires per-sample-gradient-compatible layers, arbitrary architectures need hooks written by hand, and batch normalization is disallowed outright.

Everything else in this category has drifted. TensorFlow Privacy last shipped in February 2024. AWS Labs' fast-differential-privacy makes the strongest scale claim of any tool here — under 20% memory and under 25% time overhead, with testing reported on 100B models across 512 GPUs — but has not released in about two years and its README strongly recommends PyTorch 1.11 or earlier, which is an integration problem in 2026. Google DeepMind's JAX-Privacy is JAX-only and warns of rough edges in its research components. The original ghost-clipping implementation, private-transformers, states plainly that it is not production-grade, lacks cryptographically secure noise sampling, and is vulnerable to floating-point representation attacks. And PyTorch-DP is simply the pre-1.0 name for Opacus; if it appears next to Opacus in a tool list, that list is wrong.

This is the only control in this section with a provable bound, and it covers memorization and verbatim regurgitation, membership inference exposure, and the residue of PII in the corpus, all at once. It provides no protection against data poisoning, backdoors, reward-model attacks, insiders, cloud operators or weight theft. And the most-skipped part of a deployment: differential privacy protects at the granularity of its unit. If your unit is a sentence but an individual appears in 500 documents, the person is not protected at ε.

Differentially private fine-tuning

Apply DP-SGD only to adapter parameters so the private gradient signal covers task adaptation alone — the practical form of the above, aimed at exactly the small, overfitted fine-tuning regime where membership inference actually works, and doubly attractive because LoRA shrinks the trainable parameter count and therefore the dimension the noise has to cover. Yu et al. report RoBERTa-Large on MNLI at ε = 6.7 reaching 87.8% against 90.2% non-private, and GPT-2 on DART at ε = 6.8 reaching 38.5–43.8 BLEU against 48.1. Li et al. additionally found that private learning with pretrained models avoids the dimension-dependent degradation theory predicts.

The deployable path is narrower than the literature suggests. There is no DP-LoRA library; DP-LoRA is a technique across several papers. Hugging Face peft and trl have no differential-privacy support whatsoever — no DP trainer, no DP config, no accountant. Microsoft's dp-transformers, which most write-ups still cite, last released in November 2022 and its README says outright that the repository is intended for research projects and prototypes and has not undergone the review necessary for deploying a system of critical nature such as privacy. What actually works is Opacus wired to peft directly, following the official Opacus LoRA tutorial from December 2024. That is one supported path, and you wire it yourself.

Privacy accounting

The healthiest category in this phase. Google's dp-accounting, Apache-2.0, release 0.6.0 of February 2026, is the reference implementation, offering privacy-loss-distribution accounting, Rényi accounting and composition for the Gaussian, Laplace and sampled mechanisms; it is the accountant underneath Google's own stack. The accountants bundled with Opacus — Rényi, Gaussian and privacy-loss-random-variable — are current and adequate, and Opacus shipping its own is largely why the standalone prv_accountant died in 2022. Keep autodp as a citation rather than a dependency; it has not released since November 2023.

What ε values are actually used

Practitioners want this number, and the honest answer is: far larger than theory would like. Apple's deployed emoji-discovery and Safari domain-monitoring mechanisms use ε = 4 per donation with daily rate limits; the 2020 US Census redistricting file was released at a global ε of 19.61 — ε = 17.14 for the persons data plus ε = 2.47 for housing units; academic differentially private fine-tuning of language models typically sits at ε ≈ 6.7–8. There is no accepted enterprise convention below this, which is a problem the last post in this series returns to.

Differential-privacy auditing

Rather than trusting the accountant's upper bound, run an attack and derive an empirical lower bound; a large gap either means your attack is weak or your implementation is over-noising, and a gap the other way means a bug. Google's DP-Auditorium is real and usable, Apache-2.0, living as a subdirectory of the google/differential-privacy monorepo rather than as a PyPI package, with property testers for Rényi, pure and approximate differential privacy and black-box dataset finders that search for inputs where the guarantee fails.

Scope it correctly, because this is where teams over-claim. DP-Auditorium audits mechanisms, not training pipelines: it will tell you a noise-addition mechanism is implemented correctly, and it will not give you an empirical ε for your fine-tuned model. Google's own write-up states that no single tester consistently outperforms the others and that the library provides lower bounds with no guarantees if the result is indeed bounded — it can find violations, never certify their absence. For the thing practitioners actually want, an empirical ε for a trained model, the state of the art is one-run auditing, and Steinke, Nasr and Jagielski released no official code; their result — empirical ε ≥ 1.8 against a theoretical ε ≤ 4 on CIFAR-10 from a single training run — is one you would be reimplementing against a literature that has moved three times since.

This defends against nothing by itself. It verifies that the DP-SGD defense against memorization and membership inference is delivering what the accountant claims.

Regularization against membership inference

Dropout, weight decay and early stopping reduce the train-test loss gap that membership inference exploits, and are essentially free since you were tuning them anyway. Evaluate them the LiRA way — at true-positive rate at 0.1% false-positive rate, not average accuracy — because a defense that lowers mean attack accuracy while leaving a confidently-identified tail is not a defense. Deduplication is the highest-value membership-inference control for language models; DP-SGD is the only one with a formal guarantee.

Trusted execution environments for training

Here the answer changed in the last year, and the detail decides whether you can use it. NVIDIA's Trusted Computing Solutions release R590 of December 2025 supports, on Hopper, single-GPU passthrough and Protected PCIe, the latter passing eight GPUs and four NVSwitches through to one virtual machine; and, on Blackwell B200, a new multiple-GPU passthrough mode that arrived with that release. Attestation tooling is real: NVIDIA's nvtrust provides an attestation SDK, a local GPU verifier and a PPCIe verifier, with remote verification through NVIDIA's attestation service or Intel Trust Authority. Note that nvtrust's Python SDK and local verifier are deprecated in favor of a new C++ SDK, so check which you are integrating against. On the cloud side, Azure's NCCads H100 v5 is generally available with a single H100 under AMD SEV-SNP, Google Cloud offers a3-highgpu-1g with one H100 on Intel TDX — SEV-SNP and TDX being the AMD and Intel processor features that encrypt a whole virtual machine's memory against the hypervisor beneath it in exactly three zones and without reservation support, and AWS has no GPU trusted-execution offering at all — Nitro Enclaves are CPU-only.

Now the sentence that governs adoption: multi-node confidential training does not exist. The R590 notes address single-node deployments only, and Microsoft stated on the record in July 2025 that there is no support for confidential GPU workloads across multiple nodes and no timeline for adding it. Confidential training in 2026 is capped at one node and eight GPUs. If your run needs multi-node data or pipeline parallelism, this control is not available to you.

Two further limitations matter for long runs. NVIDIA's notes state that key rotation is not supported with Protected PCIe, and separately that when the initialization-vector space is exhausted, transfers fail to complete — with frequent key rotation as the mitigation. Those two statements are in tension, and a multi-week training job is exactly where the tension bites. An independent assessment also documents plaintext remote-procedure-call metadata, timing channels in memory operations, unencrypted semaphore signals and leakage through the physical address table, and notes that confidential mode does not employ runtime memory encryption against physical decapsulation.

Within its envelope this is the answer to the cloud-operator, co-tenant and hardware-residue group, and it raises the bar on checkpoint theft and on insider access to intermediate state by keeping weights and optimizer state encrypted outside the enclave. An enclave moves the trust boundary to the silicon vendor; it does not eliminate it.

Encryption, customer-managed keys, identity management and audit logging

Customer-managed keys mean the provider cannot produce plaintext it cannot decrypt, which is the technical counterweight to CLOUD Act exposure. Least-privilege identity management segments corpus, checkpoint, optimizer-state and log buckets so one compromised researcher credential does not yield the insider attack above. Append-only audit logs make insider access detectable after the fact.

Together these cover the insider and cloud-operator threats and reduce the surface for weight theft. They touch none of the threats in the corpus or weights groups, which are properties of the model rather than the storage. Customer-managed keys inside a provider-hosted key service still leave the key in the provider's boundary — external key custody is the stronger and operationally harder posture.

Model signing and safe serialization

Start with the one that is structural rather than detective: safetensors, from Hugging Face, Apache-2.0, release 0.8.0 of June 2026, is a zero-copy memory-mappable tensor format with no code-execution path on load. It eliminates the entire pickle remote-code-execution class rather than scanning for it, and it has been the Hub default for most models since 2023. Its only real limitation is that it stores tensors and nothing else, so any custom serialization has to move somewhere explicit.

For provenance over artifacts, pip install model-signing from the Sigstore project, developed with the OpenSSF, signs model files keylessly against a transparency log, or with keys, certificates or PKCS#11 hardware tokens. Be candid that its PyPI status reads Alpha and it has not released in about eleven months. It addresses the supply-chain path into backdoor injection by letting a consumer verify that a model was not altered after training. Where you cannot avoid a third-party pickle, picklescan 1.0.5 of July 2026 and ProtectAI's modelscan 0.8.8 of February 2026 are maintained and adequate as continuous-integration gates — note that ProtectAI was acquired by Palo Alto Networks, so confirm the project's long-term status before depending on it.

And treat platform scanning as advisory. Hugging Face runs ClamAV on every commit plus pickle scanning plus third-party scanners, and the badges are still not a guarantee: JFrog found about a hundred malicious models on the Hub exploiting __reduce__ to open reverse shells, and ReversingLabs found models that defeated the scanner by compressing with 7z and deliberately breaking the pickle stream after the malicious opcodes, so validation errored out after the payload would have run. Scanning a Turing-complete format with a denylist is a losing game. The corollary to refusing pickle: also refuse trust_remote_code=True and dataset loading scripts, which are the other code paths in the same ecosystem.

What has no adoptable implementation in single-party training

Four gaps here, and they are not minor ones.

Memorization auditing has no library. Carlini et al.'s canary method — insert synthetic high-entropy sequences at controlled frequencies and measure exposure, validated against Google's Smart Compose in production — is the right technique and ships nowhere. The Steinke–Nasr–Jagielski one-run auditing paper released no code; the only PyPI package in the area is a single unaffiliated release at version 0.0.1. Rolling your own is a few hundred lines and teams do it, but the absence of standardization means exposure numbers are not comparable across papers or even across your own runs, which is precisely why Carlini et al. had to publish a follow-up note on how practitioners were misinterpreting them.

Certified poisoning defenses do not exist at this scale. Deep Partition Aggregation — partition the training set, train an independent model per partition, majority-vote, and certify that a prediction is unchanged under any attacker controlling up to n examples — has three commits of paper code, targets MNIST, CIFAR-10 and GTSRB, and needs Python 3.7. Its certificate covers at least half of CIFAR-10 against only 9 general poison insertions (Levine and Feizi). Compare that with 250 documents backdooring a 13B model. Even with perfect software the economics rule it out, because the certificate requires training k models and therefore k times the pre-training cost. Present this as an open problem, not a mitigation.

Machine unlearning cannot satisfy an erasure request. The most credible artifact, OpenUnlearning, unifies the TOFU, MUSE and WMDP benchmarks across a dozen methods and is explicit in its own README that these are baseline experiments at untuned default hyperparameters. It is an evaluation framework for researchers; no vendor ships an unlearning product, and no method offers anything resembling a deletion guarantee. The benchmarks exist precisely because evaluation is unsolved. Today the only defensible answers to GDPR Article 17 are retraining from a pinned, filtered corpus or training under differential privacy in the first place.

And differentially private synthesis for text does not ship, as the synthetic-data section above sets out — the one maintained DP synthesizer is tabular only, which leaves the corpora this phase is actually about with no option carrying a formal guarantee.

These are taken up again later.

References

「CODAS Community」へのご参加・ご登録について

一般社団法人AIデータ主権共創機構（CODAS）では、「Privacy-First AI」基盤の構築と持続可能なAIエコシステムの発展に向け、研究者・エンジニア・法務・政策・ビジネス・市民など、立場を越えてオープンに集い、秘匿AIの未来を共に創る共創プラットフォーム「CODAS Community」のメンバーを募集しています。

■ 法人企業様のご参画

CODASの理念や取り組みにご賛同いただき、AIエコシステムの社会実装や標準づくり、共創プロジェクトを共に推進していただける法人企業様を募集しております。ご入会や連携に関するお問い合わせ・お申し込みは、下記フォームより承っております。

お問い合わせフォーム：https://codas-ai.org/contact

※件名に「会員の申込」とご明記のうえ、貴社の簡単なご紹介（事業内容等）を添えてご連絡ください。確認後、担当者よりご案内いたします。

■ 個人・一般の皆様のご参加

公式Discordおよびメールマガジンでは、勉強会・ワークショップ・ミートアップ等のイベントへの優先アクセスや、秘匿性技術・ルール標準化に関する最新の知見をお届けします。東京のリアル拠点での交流も含め、立場を問わずどなたでもご参加いただけます。

CODASについて

一般社団法人AIデータ主権共創機構（CODAS）は、「安全・安心なAI社会」の実現を目指す非営利・中立の法人です。データプライバシー技術の研究開発と、そのオープンソース化による社会実装を推進しています。データを外部に出さずにAIを高度活用できる「Privacy-First AI」基盤を「創る・使う・広げる」ことを基本理念に掲げています。

≪本件に関するお問い合わせ先≫

一般社団法人AIデータ主権共創機構（CODAS）イベント事務局 E-mail：info@codas-ai.org