AI in Finance #003 — Can We Really Delete Data from an LLM?
Originally published in Weekend Reads on LinkedIn · Subscribe
I confess, the title is misleading, but I wanted you to give this article a chance. This article highlights one of the biggest challenges faced by organizations in implementing AI-based solutions. It also explores the challenges faced by various governments in drafting AI policies and acts.
Creating your own LLM and running it is not possible for organizations and only a few giants can afford it. That leaves the rest of them to use the LLMs created by others. Most of the organizations use LLMs and fine-tune them on their own data before using them.
Before we explore the two demons - Leakage & Bias that fine-tuning conjures if not done properly, it is important to understand how LLMs store data vs how traditional DBs store data. A traditional DB stores data in tables. You can write a query to find the data that you need and you will be able to extract the exact data required. You can modify this data, delete it, or do whatever you want; you have 100% control. LLMs, however, store data in weights, which are essentially strings of numbers. LLMs of different magnitude can store weights ranging from millions to trillions. When you fine-tune a model, say on your customer data, it stores everything as numbers. That's what makes correcting the behaviour of an LLM difficult once it has learned something.
Bottom line — when you fine-tune a model on your data, that data doesn't sit in a database the model reads. It becomes part of the model's weights. And weights don't forget on request.
In 2021, researchers led by Nicholas Carlini showed that GPT-2 had memorized and would reproduce real personal information from its training data — full names, email addresses, phone numbers — when prompted the right way. In 2023, the same line of work produced a startling result against a production system: a team from Google DeepMind and several universities found that simply asking ChatGPT to repeat a word forever caused it to break out of chatbot behaviour and emit verbatim training data — including personal contact details — at a rate roughly 150 times higher than normal, recovering thousands of unique memorized examples for about $200 in queries. Their conclusion was: current alignment techniques do not eliminate memorization.
Those are examples from pre-training on public data. Fine-tuning is worse in one specific way. When you fine-tune, the memorized material is your own confidential data — customer records, claims, clinical notes, financial histories — and it's typically a small, high-signal dataset, which is exactly the condition under which models memorize most aggressively. A model fine-tuned on your support tickets might reproduce a customer's exact complaint, verbatim, in response to a related query from a different user.
And you don't even need verbatim regurgitation to have a problem. A subtler attack, membership inference, lets someone determine whether a specific person's record was in your training set — without extracting the text itself. For a bank or hospital, "was this individual in the dataset you fine-tuned your default-prediction model on?" can itself be sensitive. Academic work has repeatedly demonstrated membership-inference susceptibility in clinical language models and, more recently, against LoRA fine-tuned models specifically — the cheap, popular fine-tuning method most enterprises actually use.
Studies on bias
- Fine-tuning can inject bias even into a "fair" model (Zhang & Zhou, 2024).
The researchers ran a clean experiment. They took two base models — one deliberately built to be unfair, and one built with a fairness constraint baked in — and fine-tuned both on a new task. Bias showed up after fine-tuning in both cases, including the model that started out fair. Their conclusion is the one every team should read before they start: bias can be introduced by the fine-tuning step itself, regardless of whether the base model was built with fairness in mind. In plain terms — starting from a "safe" foundation model does not save you. The risk enters at the step you control.
- The tuning meant to make models "safer" made them more biased (Itzhak et al., 2024, published in TACL).
Instruction-tuning and RLHF are the exact techniques used to align models with human intent and make them more helpful and safe. The researchers compared several base models (GPT-3, Mistral, T5 families) against their instruction-tuned versions and measured decision-making biases. The base models showed minimal to no bias; the tuned versions showed significantly higher bias across most categories. The uncomfortable takeaway: the fine-tuning step marketed as making a model safer measurably made it more biased on these axes. "We aligned it" is not the same as "we de-biased it."
- Bias compounds with each round of fine-tuning (Wang et al., 2024).
Researchers fine-tuned GPT-2 repeatedly and measured political bias after each cycle. Even when they started from unbiased data, the model grew steadily more skewed with every round of fine-tuning — bias didn't stay flat, it accumulated. They tested three fixes; two helped, one didn't — meaning the problem is manageable, but only if you deliberately measure and correct for it. Left alone, fine-tuning doesn't just risk bias once; it lets it build.
Notice what all three share: the base model was fine, and the fine-tuning step is where the bias entered — silently, by default, unless someone deliberately tested for it and corrected it. The model isn't malicious. The harm is simply the default outcome of an unguarded fine-tune — which is exactly why the control has to be built in and data must be prepared cautiously for fine-tuning.
Two examples that demonstrate the leakage problem in fine-tuned models:
-
The fine-tuned support bot that surfaces a prior customer's data. Security researchers have repeatedly shown that a support assistant fine-tuned on a company's ticket history can surface a previous customer's email or account number when a new user phrases a question the right (wrong) way — and that trying to stop it with a prompt instruction like "never reveal customer data" is not a control, it's a suggestion a motivated user talks around within a few turns. This is the single most relevant pattern for BFSI, because the fine-tuning corpus is the customer base.
-
Clinical language models leaking patient data. In the healthcare-and-finance-adjacent literature — the closest well-studied analogue to regulated financial data — models fine-tuned on sensitive records have been shown to be vulnerable to membership-inference and extraction, and researchers note that de-identifying the text beforehand doesn't fully solve it, because anonymized data can often be re-identified using outside information. That last point matters: "we removed the names" is not the same as "the data is safe."
Traditional acts/laws/regulations for privacy and data cannot be directly adopted for AI systems. Though governments across geographies are trying to come up with their own versions of policies to regulate AI systems, the next few lines shed light on some of the challenges they are facing.
GDPR Article 17 gives individuals the right to erasure — you must delete a person's personal data on request, without undue delay. India's DPDP Act carries an equivalent right to erasure and correction. Now ask the uncomfortable question: how do you delete one person from a model's weights?
You largely can't. As one 2026 privacy-infrastructure analysis put it, the right to erasure is fundamentally in tension with how gradient descent works — a single training record's influence is smeared across billions of parameters and cannot be cleanly isolated and removed. Delete the source document all you like; the gradient updates it caused remain baked in. The EU's own data-protection board (EDPB), in its January 2025 work on data-subject rights and AI, acknowledges the same thing: honouring erasure "requires reversing the memorisation of personal data by the model," which in practice means techniques akin to retraining or "machine unlearning" — costly, imperfect, and still an open research problem.
So a firm that fine-tunes a third-party model on non-anonymized customer data has, in effect, created an asset it may be legally unable to fully comply with the law about. That's not a hypothetical the regulators haven't noticed — it's the exact tension SEBI's Regulation 16C points at when it makes you responsible for your AI's outputs, and that GDPR/DPDP point at on the data underneath.
What should have been done — the controls
None of this means "don't fine-tune." It means treat your fine-tuning corpus as a governed, versioned, access-controlled asset, not a one-time input you pour into a model and forget. Concretely, and in the order that matters:
Redact and tokenize before the data ever reaches training. The fix has to sit outside the model. Strip or pseudonymize PII — names, account numbers, identifiers — before a single record enters the fine-tuning set. Encouragingly, this is now practical: a 2025 study fine-tuned mid-size open models (Mistral 7B) on 425,000 hospital notes purely to de-identify them, hitting F1 above 0.96 — meaning you can use a model to scrub the data before you use data to build a model.
Ask whether it belongs in the weights at all — or in a retrieval store instead. This is the strategic control, and it's the bridge to next week. A growing body of work shows you can get most of the benefit of fine-tuning by keeping your sensitive data in an external, access-controlled store the model reads at inference (retrieval-augmented generation) instead of baking it into weights. One 2026 clinical study found retrieval-based adaptation recovered 32–67% of fine-tuned performance (depending on the metric) while keeping patient data out of the weights entirely — which means it supports traceability, selective deletion, and GDPR-aligned erasure, because you can just delete the record from the store. Reversible by design.
Treat every output surface as a leak surface. Chat, API response, generated summary — assume PII can emerge from any of them and put deterministic filters and least-privilege access between the model and the data it can reach. This is the L2 wrapper doing its job: the model proposes, a deterministic layer decides what's allowed out.
Version and audit the corpus like code. Know exactly what went into each fine-tune, when, and under what consent basis — so that when a regulator, or an erasure request, or an incident arrives, you can answer "what did the model learn, from whom, and when?" If you can't answer that, you don't own the model; it owns you.
The takeaway
Fine-tuning feels like the fast, cheap way to make a third-party model "yours." But the moment your data enters the weights, three things become true at once: the model may reproduce it, an attacker may infer it, and you may be legally unable to delete it. The failure, as ever in this series, is almost never the model — it's the absence of a control at the data layer, upstream, before anything is irreversible.
So the question to ask before your team fine-tunes anything: does this data need to live in the model's brain — or would it be safer, cheaper and more compliant sitting in a store the model can read, and you can delete?
More on this next week. Same time, Saturday.
References — verified as of 28 August 2026
Memorization & extraction
Carlini et al., "Extracting Training Data from Large Language Models," USENIX Security 2021 — https://arxiv.org/abs/2012.07805
Nasr, Carlini et al., "Scalable Extraction of Training Data from (Production) Language Models," 2023 — https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html
Jagannatha, Rawat & Yu, "Membership Inference Attack Susceptibility of Clinical Language Models," 2021 — https://arxiv.org/abs/2104.08305
"LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models," 2025 — https://arxiv.org/abs/2507.18302
Bias introduced by fine-tuning
Zhang & Zhou, "Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency," 2024 (preprint) — https://arxiv.org/abs/2403.00625
Itzhak, Stanovsky, Rosenfeld & Belinkov, "Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias," TACL 2024 (peer-reviewed) — https://aclanthology.org/2024.tacl-1.43/
Wang et al. (Holistic AI / UCL), "Bias Amplification: Large Language Models as Increasingly Biased Media," 2024 (preprint) — https://arxiv.org/abs/2410.15234
Law & erasure
GDPR, Regulation (EU) 2016/679, Article 17 (right to erasure) — https://gdpr-info.eu/art-17-gdpr/
Digital Personal Data Protection Act, 2023 (India), Section 12 (right to correction and erasure) — Gazette of India / MeitY
EDPB Support Pool of Experts, "AI — Effective implementation of data subjects' rights," January 2025 — https://www.edpb.europa.eu/system/files/2025-01/d2-ai-effective-implementation-of-data-subjects-rights_en.pdf
SEBI (Intermediaries) (Amendment) Regulations, 2025, Regulation 16C — https://www.sebi.gov.in
Controls
De-identification: "Harnessing Moderate-Sized Language Models for Reliable Patient Data Deidentification in Emergency Department Records," 2025 (Mistral 7B, 425k notes, F1 0.9673) — https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12223680/
Retrieval vs. weights: "Privacy-Preserving Retrieval for Auditable Clinical Language Modeling on Real-World Radiology Data" (R-LM; recovers 32–67% of fine-tuning gains while keeping patient data out of the weights) — https://spiral.imperial.ac.uk
AI in Finance
Have thoughts on this?
Let's continue the conversation on LinkedIn.
Get new posts by email
Occasional notes on the BA → AI PM journey. No spam.