AI in Finance #006 — Data in AI #002: Garbage In, Garbage Out, Revisited
Originally published in Weekend Reads on LinkedIn · Subscribe
Traditional systems — pre-AI-era systems — were built on a basic principle every IT professional knows: Garbage In, Garbage Out. You put in incorrect data — a wrong entry in the DB, a spelling mistake while filling a form, and so on — and the system gave you wrong outputs. Sometimes these situations were disastrous, sometimes not so much. But they were easy to find and fix. People knew their systems — where the data was, and how to find and fix it. Data governance is standardized for these systems; regulations and policies are in place. Though the nature of data sometimes makes governing it difficult, it is still manageable. And yet, even in the pre-AI era, data governance was not given the place it deserved by most organizations.
Then came systems equipped with AI, and with them came new challenges in data governance. The biggest one: if an AI system produces a wrong output, how do you find which data caused it? LLMs are like black boxes — you cannot find out why they produced a wrong answer. You retrain them, fine-tune them, or provide correct information at run time to fix the issue — but which particular set of weights caused that issue cannot be determined reliably. And it seems organizations have started to realize this as well. Data is becoming a major roadblock in establishing reliable AI systems.
Below are some stats on pre-AI (data governance alone) vs post-AI (data governance with AI) adoption. They show the readiness and willingness — including from a budget perspective — of these organizations to adopt a culture of data governance.
Data governance adoption: before AI vs after AI
Readiness — adoption & maturity
Willingness — investment & budget signal
Why the shift is happening: only 12% of organizations say their data is of sufficient quality and accessibility for AI (2024); data governance is the #1 data challenge blocking AI initiatives (62%, 2024); Gartner expects ≥30% of generative-AI projects to be abandoned after proof-of-concept — poor data quality a lead cause (2024); and poor data quality costs the average organization $12.9M a year (Gartner).
Numbers show intent; incidents show exposure. Here are four recent cases — each one a data failure, not a model failure.
A finance worker in Hong Kong joined a video call with his CFO and several colleagues. He had been sceptical of an earlier email asking for a set of confidential transfers — until he saw and heard the people he reported to, live, on screen, telling him to proceed. He made fifteen transfers totalling about US$25.6 million. Every other face on that call was a deepfake. The firm, the engineering group Arup, only discovered it when the employee later checked with head office.
Notice what did not fail. No system was breached. No model hallucinated. What failed was an input — the authenticity of a face and a voice — and there was no control standing between that input and an irreversible payment.
The input you trust isn't real. The Arup deepfake is not a cybersecurity story with an AI garnish. AI manufactured a trusted input and inserted it directly into a human decision. Arup's own CIO described it as technology-enhanced social engineering. The governance answer is unglamorous and old: channel separation. Any consequential instruction received on one channel must be verified through a second channel that the first cannot control — a call-back to a known number, not a number offered during the conversation. AI didn't defeat a control here. It exposed that there wasn't one.
A model that is "accurate" for the wrong job. In the United States, a proposed class action — Estate of Gene B. Lokken v. UnitedHealth Group, filed in Minnesota in November 2023 — alleges that UnitedHealth used an AI tool called nH Predict to cut off post-acute care for elderly Medicare Advantage patients. The tool was built to estimate how many days of care a patient might need. It was allegedly used to deny coverage, with staff pressed to keep actual stays within one percent of the algorithm's projection. The number that should stop any practitioner cold: roughly nine in ten of the denials that were appealed were reversed — and only about 0.2% of affected patients ever appealed. UnitedHealth disputes the claims and calls nH Predict a care-support tool that physicians remain free to override; the case is proceeding. The lesson holds regardless of the verdict. A model can be statistically fine and still be catastrophically unfit for the decision it's wired into. "Fit for purpose" is not a metrics question; it is a governance question — and, as the RBI's own framework puts it, accountability cannot be delegated to the model.
A dataset with no lawful provenance. In February 2025, a US federal court in Delaware ruled in Thomson Reuters v. Ross Intelligence that using Westlaw's copyrighted headnotes to train a competing legal-research AI was not fair use. What matters for governance is how Ross got the data: it bought training material from a third-party contractor, built from Westlaw's editorial content, and — the court found — public accessibility did not grant permission to reuse it. This is provenance failure in its purest form. If you cannot say where a training set came from, who collected it, and under what rights, you are not running a data risk; you are running a legal one, and it doesn't surface until a regulator or a plaintiff makes it surface.
A vendor's AI, your accountability. In 2025, two security researchers reached up to 64 million job-applicant records sitting behind McDonald's AI hiring platform, McHire, powered by a third-party chatbot from Paradox.ai. The way in was a test administrator account with the password "123456", no multi-factor authentication, left live for years, compounded by an API flaw that let them page through applicants by editing an ID in the address bar. Little of the data was actually exfiltrated, and it was patched within a day — but the exposure was real, and the accountability question is the one every bank should sit with. McDonald's chose, deployed, and relied on that vendor's AI. "The vendor built it" is not a governance position. You own the data your vendor's AI touches.
Four incidents — a fabricated input, a misused output, an unlawful dataset, an ungoverned vendor. All are issues related to data at different stages of the AI system lifecycle.
If you hadn't noticed yet, these were all issues caused by data — not the LLM using it.
Data governance in AI is not confined to input alone, and the examples above clearly show this. Output produced by an AI system can cause harm at a much larger and broader level than in traditional systems. The impact from AI systems can be societal or political, and may affect a huge population at rapid speed.
Here is a simple outline of how data governance should look in AI systems:
- Training data should be verified for copyright, correctness, and relevance to the end users, and so on. Data provenance and data lineage should be recorded.
- Make sure the outputs are correct, not biased, not fabricated, and relevant to the specifications as per the system design.
- There should be a strong policy for disaster control: if a system fails, how to stop it, fix the issue, and learn from the mishaps. Every incident must be triaged and recorded as per a defined policy.
This is a very high-level outline; actual governance is much more complicated and involves various departments at various stages of the AI system lifecycle to implement, monitor, and evolve the data governance policy.
Data governance is unremarkable right up until the moment it is the only thing standing between you and a headline.
References
Case, regulatory, and statistical references checked against primary or authoritative sources on 26 September 2026. Pending-litigation and EU-date items are time-sensitive.
Postmortems
- Arup deepfake (~US$25.6M). A finance worker at Arup's Hong Kong office was deceived by a video call in which the CFO and colleagues were AI-generated deepfakes, and made 15 transfers totalling HK$200 million (~US$25.6M). Hong Kong police reported it in February 2024; Arup confirmed it was the victim in May 2024. Sources: CNN Business ( cnn.com/2024/05/16/tech/arup-deepfake-scam-loss-hong-kong-intl-hnk ), Fortune, AI Incident Database ( incidentdatabase.ai/cite/634 ).
- UnitedHealth / nH Predict (allegations, pending). Estate of Gene B. Lokken v. UnitedHealth Group (D. Minn., filed 14 Nov 2023) alleges nH Predict was used to deny post-acute care, with staff pressed to keep stays within 1% of the algorithm's projection; ~90% of appealed denials were reversed while only ~0.2% of enrollees appealed (KFF analysis, cited in complaint). On 13 Feb 2025, Judge John Tunheim dismissed 5 of 7 counts on Medicare-preemption grounds but allowed breach-of-contract and breach-of-implied-covenant claims to proceed. Sources: Healthcare Finance News; KFF. Status: confirmed as allegations; case ongoing.
- Thomson Reuters v. Ross Intelligence. On 11 Feb 2025, Judge Bibas (D. Del.) granted partial summary judgment to Thomson Reuters and rejected Ross's fair-use defense over using Westlaw headnotes to train an AI legal-research tool; Ross had obtained the training data from a third party. On appeal to the Third Circuit (No. 25-2153); oral argument held 11 June 2026; decision pending (expected late 2026). Sources: Baker Botts; Sterne Kessler; LawSites. Status: confirmed; appeal pending.
- McHire /Paradox.aibreach. Researchers Ian Carroll and Sam Curry reached up to 64 million applicant records on McDonald's McHire platform ( Paradox.ai 's "Olivia" chatbot) via a test admin account using the password "123456" with no MFA, plus an insecure direct object reference (IDOR); patched within a day. Sources: Ian Carroll writeup ( ian.sh/mcdonalds ), BleepingComputer, SecurityWeek, AI Incident Database (cite/1179).
Regulatory anchors
- EU AI Act — Digital Omnibus. Regulation (EU) 2026/1744 (Digital Omnibus on AI), published in the Official Journal 24 July 2026, entered into force 27 July 2026, deferring standalone Annex III high-risk obligations — which include credit scoring and recruitment — from 2 Aug 2026 to 2 Dec 2027 (Annex I embedded systems to 2 Aug 2028); Article 50 transparency duties were not deferred. Sources: Gibson Dunn; DLA Piper.
- India DPDP Rules, 2025. Notified 13 November 2025 (Gazette 14 November) with a staggered ~18-month timeline; Significant Data Fiduciaries face annual DPIAs, independent audits, a DPO, and Rule 13(3) due diligence on algorithmic systems. Sources: MeitY/PIB; EY; Lexology.
- RBI FREE-AI report. Released 13 August 2025 with 7 "sutras" (including Accountability), 6 pillars, and 26 recommendations, drawing on an RBI survey in which ~20.8% of surveyed regulated entities were already deploying AI. Sources: RBI FREE-AI Committee report; Dvara Research; KPMG.
- Basel Committee (BCBS 239). In BCBS Newsletter No. 36 (6 January 2026) on BCBS 239 implementation, the Basel Committee stated that because the quality of AI-driven outputs depends on high-quality data, robust data management becomes more important, and risk-based compensating controls should apply more caution to output where the aggregation process has known shortcomings. (The newsletter notes it "does not constitute new supervisory guidance.") Source: BIS ( bis.org/publ/bcbs_nl36.htm ).
Data governance statistics
- NewVantage Partners / Wavestone / Data & AI Leadership Exchange — Big Data & AI / Data & AI Leadership Executive Survey (2021, 2024, 2025 editions): data-driven org, data culture, CDO/CDAO & CAIO appointment, investment and top-priority figures, Responsible-AI safeguards. Primary: businesswire.com (2021 release) and the 2025 survey report PDF.
- Precisely / Drexel LeBow — Trends in Data Governance and Data Quality (2021) for governance-program adoption (64%) and dedicated budget (57%); State of Data Integrity & AI Readiness (2024 "2025 Outlook" and 2026 editions) for 71% program adoption, 63% AI governance, 12% AI-ready, 62% governance-as-AI-blocker.
- Gartner — poor data quality cost ($12.9M/yr, 2020 Magic Quadrant for Data Quality Solutions); ≥30% of GenAI projects abandoned after PoC (press release, 29 July 2024).
Data in AI
Have thoughts on this?
Let's continue the conversation on LinkedIn.
Get new posts by email
Occasional notes on the BA → AI PM journey. No spam.