CompletedML + LLM engineering · Reliability

Determinism & Guardrails

Product Owner · Architecture & audit · 2026

Two parts, one thesis: the disciplined system you build around a model matters as much as the model itself. A classical-ML lifecycle and a small-LLM ‘rings of control’ study answer one sharp question with numbers — when a model gives a wrong or made-up answer, where do you fix it, and what does each fix actually buy you?

The problem

Most AI products hit the same wall: a model that looks brilliant in a demo and fails in production. The usual reflexes — a bigger model, cleverer prompting, ‘just be careful’ — are hope, not engineering. Architectural prevention you can point to beats hoping the model behaves.

Approach

Part A — a classical-ML lifecycle on real retail data (Online Retail II) built to trust its own measurements: dummy baselines, honest tuning, validated segments, and a leakage clinic that plants each leak, measures the damage, then fixes it architecturally.
Part B — three ‘rings of control’ around a frozen small language model, proving what each layer is worth in turn: prompting, then context (retrieval), then a code harness.
The seam — Part A’s discovered customer segments become the training signal that LoRA-adapts the model’s weights in Part B (fine-tune for form, not facts).
Built Product-Owner style: architecture, decisions and auditing on Claude Desktop; execution delegated to a coding agent (Claude Code). Every decision is an ADR with its rejected alternatives; every experiment is logged.

What it does

Predicts repeat customers (PR-AUC 0.78 vs a 0.44 no-skill floor) and discovers four RFM segments proven to beat shuffled noise before any strategy is built on them.
Runs a leakage clinic — inject → measure → fix → measure — showing a target leak inflate PR-AUC to a perfect 1.00 and honest removal restore 0.78.
Fine-tunes a small model on a synthetic corpus, then walks it up the ladder from bare → prompt → context → harness on a sealed test set.

Outcome & results

The ladder: accuracy 0.00 → 0.00 → 0.82 → 0.77; hallucination 1.00 → 0.52 → 0.28 → 0.22. Prompting moves honesty not accuracy; context is the big accuracy jump; the harness trades a little accuracy to kill the hallucination tail.
Retrieval beat context-stuffing on accuracy at ~7× fewer tokens — and plain keyword search (0.86) edged fancy vector search (0.82).
The harness gave 100% prompt-injection defense (8/8 blocked by a deterministic filter) vs 0% without — a guarantee, not a hope.
Fine-tuning won decisively at small scale: 0% → 79% format adherence, 22 → 0 fabrications, via LoRA on ~0.9% of params.

Python

scikit-learn

Small instruct LLM (local, CPU)

LoRA / PEFT

BM25 + vector RAG

Claude Desktop + Claude Code

Code Adjudicator (SEBI)

Approach

Part A — a classical-ML lifecycle on real retail data (Online Retail II) built to trust its own measurements: dummy baselines, honest tuning, validated segments, and a leakage clinic that plants each leak, measures the damage, then fixes it architecturally.
Part B — three ‘rings of control’ around a frozen small language model, proving what each layer is worth in turn: prompting, then context (retrieval), then a code harness.
The seam — Part A’s discovered customer segments become the training signal that LoRA-adapts the model’s weights in Part B (fine-tune for form, not facts).
Built Product-Owner style: architecture, decisions and auditing on Claude Desktop; execution delegated to a coding agent (Claude Code). Every decision is an ADR with its rejected alternatives; every experiment is logged.

What it does

Predicts repeat customers (PR-AUC 0.78 vs a 0.44 no-skill floor) and discovers four RFM segments proven to beat shuffled noise before any strategy is built on them.
Runs a leakage clinic — inject → measure → fix → measure — showing a target leak inflate PR-AUC to a perfect 1.00 and honest removal restore 0.78.
Fine-tunes a small model on a synthetic corpus, then walks it up the ladder from bare → prompt → context → harness on a sealed test set.

Outcome & results

The ladder: accuracy 0.00 → 0.00 → 0.82 → 0.77; hallucination 1.00 → 0.52 → 0.28 → 0.22. Prompting moves honesty not accuracy; context is the big accuracy jump; the harness trades a little accuracy to kill the hallucination tail.
Retrieval beat context-stuffing on accuracy at ~7× fewer tokens — and plain keyword search (0.86) edged fancy vector search (0.82).
The harness gave 100% prompt-injection defense (8/8 blocked by a deterministic filter) vs 0% without — a guarantee, not a hope.
Fine-tuning won decisively at small scale: 0% → 79% format adherence, 22 → 0 fabrications, via LoRA on ~0.9% of params.