Preparing a fine-tuning dataset

What you’ll learn

What a good fine-tuning dataset looks like, how to structure and size it, and the data mistakes that undermine a run before training even starts.

Why the data is almost everything

You can run the same LoRA training with the same settings on two datasets and get a useful model from one and a useless one from the other. The data is the largest single factor in how a fine-tune turns out, and far more time is well spent on the dataset than on tuning hyperparameters.

The reason follows from what fine-tuning does. It teaches a model a behaviour by example, so the examples are the lesson. Messy, inconsistent, or contradictory examples teach a messy, inconsistent model.

What a good dataset looks like

• Consistent format. Every example should follow the same structure, typically an input and the ideal output, or a chat exchange. Use the chat template your base model expects, and apply it uniformly. Inconsistent formatting is one of the most common causes of a disappointing fine-tune.
• High quality over high volume. A few hundred carefully written, correct examples usually beat thousands of mediocre ones. Every example is teaching the model something, so a wrong or sloppy example teaches the wrong thing.
• Representative and varied. The examples should cover the range of inputs the model will actually see, including the awkward cases. If the data is all easy, the model learns to handle only easy inputs.
• The behaviour you want, shown not told. If you want a particular tone or output shape, every example should demonstrate it. The model learns from what the outputs look like, not from instructions about them.

How much data

Less than people expect, for a LoRA. Teaching a style or a consistent output format can work with a few hundred good examples. More data helps up to a point, but only if it maintains quality. Doubling the size with weaker examples usually makes the model worse, not better. Start small and clean, evaluate, and grow the dataset deliberately.

Splitting for honest evaluation

Hold back a portion of your examples that the model never sees during training, and use it to judge the result. Without this, you cannot tell whether the model has learned the behaviour or simply memorised your examples. Make sure the held-out set does not overlap with the training data, or your evaluation will flatter the model.

What can go wrong

• Inconsistent formatting. Mixed templates or structures confuse the lesson. Normalise everything first.
• Too little, too repetitive data. A tiny, narrow dataset leads to overfitting: the model parrots your examples and gets worse at everything else.
• Contaminated evaluation. If your test examples leaked into training, your results are meaningless. Keep them strictly separate.
• Quantity over quality. Padding a dataset with weak examples drags the model down. Curate, do not just collect.

Next steps

With a dataset prepared, revisit LoRA and QLoRA for the training itself, and the fine-tuning tools in the catalogue, such as Unsloth and Axolotl, to run it.