AI in Finance #007 — Data in AI #003: Fix the Data, Not the Model
Originally published in Weekend Reads on LinkedIn · Subscribe
There is a cheap solution to enhance the accuracy of AI-based systems, but it requires manual effort and is somewhat monotonous. Fix the underlying data — it can be training data, data vectorized and stored in RAG-based systems, etc.
For most of the last decade, the instinct when a model underperformed was to reach for a better model — or to put in more training data and more context to get better results. Another path people take is technical in nature: they try to enhance system architecture, code, etc. Then there is fine-tuning to make LLMs more accurate. But fixing the underlying data is still cheaper and more efficient than these methods. This article presents examples and studies that support this claim.
A fraud detection team spent a quarter chasing three points of precision. They swapped gradient boosting for a deep network, then swapped it back. They tuned hyperparameters until the training runs blurred together. The model got heavier, the GPU bill got larger, and the fraud analysts kept seeing the same false positives land on their desks every Monday.
Then someone did something unglamorous. They pulled two hundred transactions the model had flagged and the analysts had cleared, and they read the labels one by one. Roughly a fifth of them were labelled wrong. "Confirmed fraud" that had been reversed. "Legitimate" that was a chargeback three weeks later. The model was not failing to learn. It was learning exactly what the data told it, and the data was lying.
In a cost-conscious, heavily regulated industry, fixing the data happens to be the most underrated lever we have.
That instinct shows up as more layers, more parameters, a newer architecture off the shelf. Andrew Ng calls this the model-centric habit, and in 2021 he put a number on how far it gets you.
His team at Landing AI worked on a computer-vision system inspecting steel sheets for defects. The baseline sat at 76.2% accuracy; the target was 90%. One set of teams took the model-centric route — hold the data fixed, improve the code. After weeks of effort, their accuracy improved by essentially zero. Another approach held the model fixed and systematically improved the data instead: cleaner, more consistent labels, sharper definitions of what counted as a defect. That route reached 93.1% — a gain of nearly 17 points — and got there in a fraction of the time.
Once a model architecture is reasonable, the code is close to a solved problem — you can download something competent from any repository. The differentiator that is left, the part almost nobody else can copy, is the quality of your data. In finance, this matters even more, because our data is not a public benchmark. It is our proprietary, hard-won, regulated advantage.
The uncomfortable premise underneath all of this is that our "ground truth" is not as true as we treat it. A 2021 study from researchers at MIT and Cleanlab went looking for label errors in ten of the most widely used benchmark datasets in machine learning — the datasets thousands of papers are built on. They found an average of at least 3.3% of test labels were simply wrong, rising to around 6% in the famous ImageNet set.
These are the clean, curated, academic datasets. Now picture your own: default flags entered by three different collections teams under three different interpretations of "default." Complaint categories chosen from a dropdown by a rushed agent. Suspicious-activity labels shaped as much by which analyst was on shift as by the transaction itself. If the gold standard carries 3% to 6% noise, the working assumption for an internal financial dataset should be worse until proven otherwise.
This is why fixing the dataset so often beats fine-tuning. Fine-tuning teaches a model to fit your data more tightly. If the data is noisy, you are paying compute to memorize someone's mistakes with greater confidence.
Meta's LIMA study took a strong pretrained model and fine-tuned it on just 1,000 carefully curated examples — no reinforcement learning, no elaborate preference pipeline. In head-to-head human evaluations, its answers were rated equal to or better than GPT-4 in 43% of cases, and the gap widened against other leading models of the day. A thousand good examples outran the brute-force approach. The authors were honest about the limits — such a model is not as robust as a hardened production system — but the direction is unmistakable: curation is leverage.
For a BFSI leader, that reframes the budget conversation. The instinct is to fund better model. The evidence says fund more discipline around the data, and you will often do more with less.
Labelling is a governance decision, not a data-entry task
If labels are the problem, labelling is the fix, and here it stops being a machine-learning issue and becomes a governance one.
Ask five people at a bank to define a "high-risk customer" and you will get five answers. Every one of those answers becomes a label, and every inconsistent label becomes noise the model dutifully learns. Ng's own guidance for data-centric work reads like a control framework: make the labels consistent, use more than one reviewer to surface disagreement, chase down the ambiguous cases and clarify the instructions, and be willing to throw out noisy examples rather than assume more data is always better.
A written, version-controlled definition of every target variable — what "default," "fraud," "complaint," "suitable" actually mean — is a labelling standard. Measuring agreement between reviewers is a control. Sending contested cases to adjudication is an audit trail. Treating the label dictionary as a governed artefact, owned and signed off, is exactly the kind of pre-registered gate I keep arguing for: the standard is set before the data is used, not reverse-engineered after the model misbehaves. Labelling discipline is where model risk management should start, long before validation.
Feature stores: computing the truth once
Doing more with less also means not throwing away work you have already done. That is what a feature store is for.
The pattern originated at Uber, whose engineering team described it plainly in 2017: a feature store lets teams "share, discover, and use a highly curated set of features," because so many models rely on the same inputs and there is real value in computing them once and reusing them. Uber had around 10,000 shared features running the day they wrote it up. The idea has since become standard infrastructure, through open-source projects like Feast and commercial platforms like Tecton, built by the same people.
The banking version of the problem is painfully familiar. The risk team calculates a customer's 90-day average balance for a credit model. Marketing builds a near-identical metric for a churn model. Fraud computes its own variant for scoring. Three definitions, three pipelines, three subtly different numbers — and when a regulator asks which one drove a lending decision, three teams point at each other.
A feature store replaces that with a single governed definition, computed once and served everywhere. The benefits map almost one-to-one onto things we already have to prove:
- Consistency between training and production. Uber's team built the store specifically so the same process generates a feature at training time and at prediction time. In finance we call the failure mode training–serving skew, and it is a classic reason a model that dazzled in validation quietly degrades in production.
- Point-in-time correctness. A feature store worth the name reconstructs what was known as of a past date, so a model is never accidentally trained on information from the future. For anyone who has watched a backtest fall apart under look-ahead bias, this is not a convenience — it is the difference between a defensible model and a fictional one.
- Auditability and reuse. One owner, one definition, one lineage. Cheaper to build, and far cheaper to explain.
A feature store, in other words, is a deterministic data layer with governance baked in. It is the L1 that computes the numbers precisely and reproducibly, before any model — or any LLM narrating the output — ever touches them.
The through-line: prevention over hope
Step back and the three ideas are one idea. Clean labels, curated examples, and shared feature definitions are all ways of fixing the problem at the data layer, where it is cheapest to fix and easiest to audit, rather than hoping a bigger model will paper over it downstream.
That is the same principle I return to constantly: systems should behave by design, not by hope. A better dataset is architectural prevention. It is deterministic, inspectable, and it holds regardless of which model you plug in next quarter. Fine-tuning your way out of bad data is the opposite — expensive, opaque, and quietly fragile.
None of this is a claim that the model never matters; sometimes it genuinely does, and data-centric work is not a license to stop thinking. But for most financial institutions, the marginal rupee is far better spent on labelling standards and a governed feature layer than on the next architecture. It costs less, it explains better, and in a regulated industry, explaining better is the product.
References
- Andrew Ng, "MLOps: From Model-Centric to Data-Centric AI" (keynote, 24 March 2021). Steel-sheet inspection: 76.2% baseline, no gain from model-centric tuning, 93.1% (+16.9%) from improving the data. Figures reported in Insight Partners / AI Business: https://www.insightpartners.com/ideas/google-brains-andrew-ng-says-data-centric-approach-boosts-ai-success/
- Curtis G. Northcutt, Anish Athalye, Jonas Mueller, "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks," NeurIPS 2021 (Datasets & Benchmarks Track), arXiv:2103.14749: https://arxiv.org/abs/2103.14749 — interactive error browser: https://labelerrors.com
- Jeremy Hermann & Mike Del Balso, "Meet Michelangelo: Uber's Machine Learning Platform," Uber Engineering, 5 September 2017: https://www.uber.com/blog/michelangelo-machine-learning-platform/
- Feast — open-source feature store (Gojek + Google Cloud): https://feast.dev; Tecton — commercial feature store from Uber's Michelangelo team: https://www.tecton.ai
- Chunting Zhou et al., "LIMA: Less Is More for Alignment," NeurIPS 2023, arXiv:2305.11206: https://arxiv.org/abs/2305.11206
Data in AI
Have thoughts on this?
Let's continue the conversation on LinkedIn.
Get new posts by email
Occasional notes on the BA → AI PM journey. No spam.