Building a Small Language Model from scratch

I decided to learn about LLMs in a bit more detail. And the best way to learn is to build one. Every decision you make during the building process gives you clarity on what to use and when to use it. It helps you understand the dependencies on hardware, and how you can make the best use of what you have without compromising on your goal.

The goal was simple — learn about LLMs: how they're built, how they work, how they understand human language and respond so well.

With this in mind, the first thing I researched was what I could build on my laptop. My intent was to build a small model capable of conversation and a bit of reasoning.

What I understood from my research is that basic models with such capabilities start from billions of parameters. With my 4GB VRAM, that wasn't possible.

On digging further, I found I could build a small model trained on TinyStories — ~2.1 million simple children's stories — and the model would be able to write small stories of its own as output.

The scale I'm working at

To understand the scale of what I'm trying to build, it helps to put it next to the models everyone knows:

Same architecture as the models running the industry — a decoder-only transformer, the "GPT" design — just scaled down. A single-cylinder engine next to a V12: genuinely the same engineering, fewer cylinders. Everything I learn here scales up unchanged.

What it will (and won't) do

My model won't be able to know facts, answer trivia, or reason. It has only seen roughly 2 million stories, so it learns how English works — not what's true. But it will be developed using all the same steps that an actual large-scale model uses.

Project specifications are ready — a 14 million parameter model, with 6 attention heads and 6 layers. nLemon-14 is ready to be born.

Follow this space if you want to see how an LLM is really made, and learn the concepts involved.