Preparing data for LLM fine-tuning means turning your existing material into examples of the behavior you want to teach. You then check that the training setup presents those examples to the model correctly. There are five steps:
- Write down the behavior you want the model to learn, including what a correct output looks like.
- Collect source material that shows that behavior: existing records, documents, expert-written answers, or generated examples.
- Convert the material into input and output pairs, and correct any output you would not want the model to repeat.
- Clean the set by removing duplicates, resolving conflicting answers, and redacting personal data.
- Format the examples for the model you are training, hold out evaluation cases, and look at what the model will actually see before you start a training run.
Most guides spend their time on collection and cleaning. Step 3 gets the least attention, and it decides most of what the model learns. Your existing data shows how things were done, and fine-tuning teaches the model to do them that way again, mistakes included. Before an answer goes into the training set, edit it to reflect what the model should say.
This guide covers supervised fine-tuning, where each example pairs an input with the desired output. How many examples you need is a separate question, answered in How much data do you need to fine-tune an LLM?. This article is about what each example should look like and how to prepare it for training.
Start by deciding what a correct output looks like
Before you collect a single file, write down what the model should do and what a correct answer looks like. If two people on your team would answer the same input differently, the dataset will end up containing both answers, and the model will learn both.
A useful test is to write ten example inputs and the ideal output for each, by hand, before you touch any existing data. “Answer questions about our procedures” is too vague to test. “Given a question about purchasing approval, name the approver, the threshold, and the procedure version, in two sentences” is specific enough that a reviewer can mark an answer right or wrong. Those ten examples become the reference you compare every later example against.
This is also when to ask whether fine-tuning is the right tool at all. Fine-tuning teaches stable behavior well: a format, a tone, a classification scheme, a way of working through a certain kind of request. It is poorly suited to storing facts that change, like prices, stock levels, or this quarter’s policy. Those belong in a lookup or retrieval system that the model consults when it answers, as we described in How to Build an AI Chatbot With Your Own Data. If most of what you want the model to “know” changes every month, put your preparation effort into retrieval instead of a training file.
What format should fine-tuning data be in?
Most fine-tuning tools expect JSONL, a text file with one JSON object per line and one training example per object. For chat models, each example is usually a short conversation: a list of messages, each with a role (system, user, or assistant) and its content. One example looks like this:
{"messages": [{"role": "system", "content": "Extract the supplier, invoice date, and total. Reply in JSON."}, {"role": "user", "content": "Invoice from Nordline Logistics, dated 3 March 2026. Total due: 4,180.00 EUR."}, {"role": "assistant", "content": "{\"supplier\": \"Nordline Logistics\", \"date\": \"2026-03-03\", \"total\": 4180.00, \"currency\": \"EUR\"}"}]}Field names vary between platforms, so check the documentation for whatever service or library you train with. Hugging Face’s TRL library, for example, accepts this messages format as well as a prompt and completion format that keeps the input and output apart. TRL dataset formats
Getting the file right is usually quick. The chat template and the choice of tokens to train on need closer inspection: mistakes in either can pass unnoticed.
First, the chat template has to match the model. The model never sees your JSON. Before training, the messages are turned into one long string with special markers that show where each turn starts and ends, and each model family uses its own markers. Mistral’s instruction models wrap user turns in [INST] tags, while others use tags like <|user|> and <|assistant|>. Hugging Face’s documentation warns that a model given the wrong control tokens performs drastically worse. Hugging Face chat templates Let the model’s own tokenizer apply its template, and use the same template in training and in production.
Second, limit the training loss to the assistant’s answer. Training penalizes the model for every token it fails to predict. If those tokens include the user’s message and the system prompt, part of the training goes into reproducing inputs instead of producing good outputs. Most libraries can limit training to the assistant’s reply, but they don’t always do it by default. In recent versions of TRL, for example, a conversational dataset trains on the full sequence unless assistant-only training is switched on. TRL SFTTrainer Set it explicitly rather than assume.
Can you fine-tune an LLM on company documents?
Usually, you need to convert them first. Procedures, reports, emails, and system records were written for people. A procedure document describes a rule, but it may contain no examples of how to answer questions about that rule. Fine-tuning needs input and output pairs that resemble the requests the model will handle.
Most of the editing happens during that conversion. Existing data records what people wrote at the time, and some of those answers were outdated, inconsistent, or wrong. A model trained on those answers can repeat the same errors. Here is a line from an internal procedure, next to the training example it should become:
We’ve turned the procedure into a realistic question paired with an answer based on the current rule. The answer also follows the format set in the first step: approver, threshold, procedure version. Converting the source required correcting it as well as changing its format.
Every kind of existing data needs the same treatment. Old reports may follow a format the team dropped years ago. Emails bury the useful answer among greetings, signatures, and side topics. Records written by different people describe the same decision a dozen ways. For each source, someone who knows the current rules decides what to keep, what to rewrite, and what to drop. Attach a short note to each example saying where it came from, which rule or version it follows, and who reviewed it. When a policy changes next year, those notes tell you which examples to update, so you aren’t retraining on data nobody can trace.
Be careful about style when you rewrite, though. Experts sometimes produce answers the model would never write itself, full of unusual phrasing, dense jargon, or long chains of reasoning. Research on instruction-tuning data suggests that responses close to what the target model would naturally generate are easier for it to learn. A data selection method built on that idea beat training on 4.5 times as much data. GRAPE, 2025 Fix the content and keep the format consistent, but keep the wording close to what the model could naturally produce.
How do you clean and deduplicate fine-tuning data?
Cleaning fine-tuning data means dealing with exact duplicates, near-duplicates, and examples that contradict each other.
Exact duplicates are easy to spot. A record exported twice, or a template filled in with the same values, can be caught with a simple comparison once you normalize whitespace and capitalization.
Near-duplicates take more work and do more damage. Two examples can use different words and teach exactly the same thing: one request phrased two ways, the same report with a new date, ten variations generated from one seed. You usually need a similarity measure to find them, such as comparing text embeddings or overlapping word sequences, and then a person to decide which members of each cluster to keep. Every repeat gives its pattern extra weight in training, so the model drifts toward what was most common in your data rather than what matters most. A study of language model training data found that models trained on deduplicated data reproduced memorized text ten times less often and reached the same or better accuracy in fewer training steps. Lee et al., ACL 2022
Conflicting examples matter most and are the easiest to miss. If one example sends a request to finance and a nearly identical one sends it to procurement, the model learns that either answer is acceptable. Group similar inputs and compare their outputs. Where they disagree, decide which is right, write down why, and fix the rest. If your team can’t agree on the right answer, cleaning won’t solve it; someone has to make that decision before training starts.
The LIMA study makes a related point. Its authors fine-tuned a 65-billion-parameter model on just 1,000 carefully chosen and edited examples, and people preferred its answers to those of a model trained on 52,000 instructions. Their ablations found that output quality and input diversity mattered more than quantity. LIMA, NeurIPS 2023 Reviewing the examples you have is a better starting point than collecting more without checking them.
How do you remove personal data before fine-tuning?
Before training, find the personal and sensitive details in every example and replace them with consistent placeholders. That means names, email addresses, phone numbers, customer and employee IDs, account numbers, street addresses, and the passwords and access keys people paste into documents and messages. Run an automated detection tool first, then have a person review a sample, since automated tools miss unusual formats and details that are identifiable as personal information only in context.
Use placeholders that keep the example readable, like [CUSTOMER_NAME] or [ORDER_ID], instead of deleting the text. Deleting parts of a sentence can make it unreadable, and the model may learn to reproduce that broken wording. When the same person appears several times in one example, use the same placeholder each time.
Redaction removes personal details from your examples. You also need to establish whether you’re allowed to use the source data for training. Customer records, contracts, and licensed content can come with limits on how, where, and why they are processed. Settle that with whoever owns the data before you build the dataset, rather than after the model is trained.
Does training only on answers protect personal data?
No. Fine-tuned models can memorize rare details from their training data and repeat them later, which is why redaction can’t be skipped. You might assume details that appear only in the input are safe, since the model is never trained to produce them. A 2026 study found they aren’t. Personal information that appeared only in the inputs of a fine-tuning dataset could still be extracted from the trained model, and none of the privacy techniques the authors tested removed the risk without trade-offs. Szep et al., EACL 2026 So limiting training to the assistant’s reply, as described in the format section, helps the model learn, but you shouldn’t count on it for privacy.
Should you use synthetic data for fine-tuning?
Use synthetic data to fill specific gaps, and keep real examples as the base. Generated examples help with cases your existing data rarely covers, such as unusual inputs, edge cases, refusals, formats you plan to support but have little history for, or languages you have few examples in. Relying on them for the whole dataset is risky: they reflect the generating model’s interpretation of the task, which may differ from the inputs your model will receive.
The usual problem is sameness. Ask a model for 500 examples and you often get 500 variations on a handful of patterns, with similar sentence structures, lengths, and topics. Start from a varied set of real seed examples, ask for examples that fill the gaps you’ve identified, and run the results through the same near-duplicate check as everything else. Diversity has a limit too. A 2025 study found that moderately diverse generated data improved fine-tuning when labeled data was scarce, while highly diverse generated data hurt performance. Zhu et al., 2025 Generate examples for the situations your model will actually face.
Review generated examples like any others, because a fluent answer can still be wrong. And keep them out of the evaluation set, which should measure performance on real inputs. If your test cases were generated the same way as your training data, a good score only shows that the model picked up the generator’s habits.
How do you check fine-tuning data before training?
Run your checks on the formatted data, since that’s what the model trains on. Most data preparation mistakes don’t raise an error. Training finishes, the loss curve looks fine, and the model ends up worse than it should be. The only dependable way to catch these problems is to look before you train.
Start by reading a few examples exactly as the model will see them. Decode the formatted, tokenized text back into readable form. You should see the right chat template markers, one clear end-of-turn marker after each answer, and no duplicated tokens. It takes about ten minutes and catches template mismatches that would otherwise cost you a full training run.
Then check which tokens contribute to the training loss. Most libraries can show which tokens count toward training, so inspect a few examples to confirm that only the assistant’s reply is included.
Set aside examples for evaluation that the model will never train on. Split the data by group so related examples stay in the same set: variations from one seed, examples converted from the same document, or several cases from the same customer. Otherwise near-copies of your training examples end up in the evaluation set, and the score measures memory instead of skill.
Review a random sample by hand too, as described in the final section. Check example lengths as part of that review: truncation rarely shows up in the logs.
Why do fine-tuning examples get truncated silently?
Training tools cut off examples longer than a maximum length, often without warning, and in recent versions of TRL the default limit is 1,024 tokens. TRL SFTTrainer If a long example’s answer falls past the cutoff, the model trains on an input with a missing or partial reply. Measure the token length of every example, then raise the limit or shorten the examples that go over it.
Is your dataset ready for fine-tuning?
Seven quick questions about the data you plan to train on.
1 of 7
Have you written down what a correct output looks like?
Next question
Read twenty examples before you train
For a final check, set aside an afternoon to read through some examples. Pick twenty training examples at random, not ones you already know, and read each as if it were the model’s answer to a real request tomorrow. Would you send it? Would the person responsible for the rule approve it? Does it match the other nineteen?
If all twenty pass and you’ve completed the earlier checks, the dataset is probably ready for a first training run. Evaluation will show where it still falls short. If a few fail, you already know what needs editing. An hour spent on those corrections now can save another training run; finding the same mistakes in production usually costs more.
