Training a small language model on the KJV

# Training a small language model on the KJV

Canonical: https://fragments.ikon.io/pieces/training-a-small-llm

Type: folio

Published: 2026-08-26T12:00:00.000Z

Published version created: 2026-10-18T17:13:26.372Z

## Summary

i decided to train & use a small language model from scratch to learn how the process actually works. i worked with a slew of frontier & open models to both build it and explain it so i could grasp it. and then we docume

## Writing

i decided to train & use a small language model from scratch to learn how the process actually works. i worked with a slew of frontier & open models to both build it and explain it so i could grasp it. and then we documented it.

here's how it went.

## Contents

1. [the process in outline](https://fragments.ikon.io/pieces/the-process-in-outline)
   
   [Machine view](https://fragments.ikon.io/machine/the-process-in-outline) · [Markdown](https://fragments.ikon.io/machine/the-process-in-outline/index.md)

2. [the task and the text](https://fragments.ikon.io/pieces/the-task-and-the-text)
   
   [Machine view](https://fragments.ikon.io/machine/the-task-and-the-text) · [Markdown](https://fragments.ikon.io/machine/the-task-and-the-text/index.md)

3. [the transformer](https://fragments.ikon.io/pieces/the-transformer)
   
   [Machine view](https://fragments.ikon.io/machine/the-transformer) · [Markdown](https://fragments.ikon.io/machine/the-transformer/index.md)

4. [interpretation](https://fragments.ikon.io/pieces/interpretation)
   
   [Machine view](https://fragments.ikon.io/machine/interpretation) · [Markdown](https://fragments.ikon.io/machine/interpretation/index.md)
   
   Relation: chapter

5. [training](https://fragments.ikon.io/pieces/training)
   
   [Machine view](https://fragments.ikon.io/machine/training) · [Markdown](https://fragments.ikon.io/machine/training/index.md)

6. [data and scale](https://fragments.ikon.io/pieces/data-and-scale)
   
   [Machine view](https://fragments.ikon.io/machine/data-and-scale) · [Markdown](https://fragments.ikon.io/machine/data-and-scale/index.md)
   
   Relation: chapter

7. [inference](https://fragments.ikon.io/pieces/inference)
   
   [Machine view](https://fragments.ikon.io/machine/inference) · [Markdown](https://fragments.ikon.io/machine/inference/index.md)

8. [transformer variants](https://fragments.ikon.io/pieces/transformer-variants)
   
   [Machine view](https://fragments.ikon.io/machine/transformer-variants) · [Markdown](https://fragments.ikon.io/machine/transformer-variants/index.md)
   
   Relation: chapter

9. [evaluation](https://fragments.ikon.io/pieces/evaluation)
   
   [Machine view](https://fragments.ikon.io/machine/evaluation) · [Markdown](https://fragments.ikon.io/machine/evaluation/index.md)
   
   Relation: chapter

10. [post-training](https://fragments.ikon.io/pieces/post-training)
   
   [Machine view](https://fragments.ikon.io/machine/post-training) · [Markdown](https://fragments.ikon.io/machine/post-training/index.md)
   
   Relation: chapter

11. [the entire model](https://fragments.ikon.io/pieces/the-entire-model)
   
   [Machine view](https://fragments.ikon.io/machine/the-entire-model) · [Markdown](https://fragments.ikon.io/machine/the-entire-model/index.md)
   
   Relation: chapter

12. [from model to system](https://fragments.ikon.io/pieces/from-model-to-system)
   
   [Machine view](https://fragments.ikon.io/machine/from-model-to-system) · [Markdown](https://fragments.ikon.io/machine/from-model-to-system/index.md)
   
   Relation: chapter