Chapter 12 · Fine-Tuning Generative Models

Chapter goals: Master the two-step method for fine-tuning generative LLMs — first use QLoRA for instruction fine-tuning (SFT), then use DPO for preference alignment; complete the entire process on consumer-grade GPUs using the TRL library.

12.1 Environment Preparation (Optional)

If you are viewing this notebook on Google Colab (or any other cloud platform), you need to uncomment and run the code block below to install the dependencies for this chapter:

💡 NOTE: Running the examples in this chapter requires a GPU. In Google Colab, go to Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4.

python

# %%capture
# !pip install -q accelerate==0.31.0 peft==0.11.1 bitsandbytes==0.43.1 transformers==4.41.2 trl==0.9.4 sentencepiece==0.2.0 triton==3.1.0

12.2 Supervised Fine-Tuning (SFT)

12.2.1 Data Preprocessing

The first step is to format the conversational data into the chat template the model expects. We load TinyLlama's tokenizer specifically to call apply_chat_template:

python

from transformers import AutoTokenizer
from datasets import load_dataset


# Load a tokenizer to use its chat template
template_tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

def format_prompt(example):
    """Format the prompt to using the <|user|> template TinyLLama is using"""

    # Format answers
    chat = example["messages"]
    prompt = template_tokenizer.apply_chat_template(chat, tokenize=False)

    return {"text": prompt}

# Load and format the data using the template TinyLLama is using
dataset = (
    load_dataset("HuggingFaceH4/ultrachat_200k",  split="test_sft")
      .shuffle(seed=42)
      .select(range(3_000))
)
dataset = dataset.map(format_prompt)

Look at what a formatted prompt looks like:

python

# Example of formatted prompt
print(dataset["text"][2576])

12.2.2 Model Quantization

Load the base model TinyLlama-1.1B and apply 4-bit quantization (this is the "Q" in QLoRA). BitsAndBytesConfig configures NF4 quantization type, float16 compute dtype, and nested quantization:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"

# 4-bit quantization configuration - Q in QLoRA
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,  # Use 4-bit precision model loading
    bnb_4bit_quant_type="nf4",  # Quantization type
    bnb_4bit_compute_dtype="float16",  # Compute dtype
    bnb_4bit_use_double_quant=True,  # Apply nested quantization
)

# Load the model to train on the GPU
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",

    # Leave this out for regular SFT
    quantization_config=bnb_config,
)
model.config.use_cache = False
model.config.pretraining_tp = 1

# Load LLaMA tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
tokenizer.pad_token = "<PAD>"
tokenizer.padding_side = "left"

12.2.3 Configuration

LoRA Configuration

LoRA does not update the original model weights; instead, it injects low-rank adapters onto the attention and MLP projection layers. r=64 is the rank; lora_alpha=32 is the scaling factor:

python

from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model

# Prepare LoRA Configuration
peft_config = LoraConfig(
    lora_alpha=32,  # LoRA Scaling
    lora_dropout=0.1,  # Dropout for LoRA Layers
    r=64,  # Rank
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=  # Layers to target
     ['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
)

# prepare model for training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, peft_config)

Training Configuration

Training arguments use gradient accumulation (equivalent batch size of 8) and gradient checkpointing to save VRAM:

python

from transformers import TrainingArguments

output_dir = "./results"

# Training arguments
training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    optim="paged_adamw_32bit",
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    num_train_epochs=1,
    logging_steps=10,
    fp16=True,
    gradient_checkpointing=True
)

12.2.4 Start Training!

Use TRL's SFTTrainer to execute supervised fine-tuning; after training completes, save the QLoRA weights:

python

from trl import SFTTrainer

# Set supervised fine-tuning parameters
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    dataset_text_field="text",
    tokenizer=tokenizer,
    args=training_arguments,
    max_seq_length=512,

    # Leave this out for regular SFT
    peft_config=peft_config,
)

# Train model
trainer.train()

# Save QLoRA weights
trainer.model.save_pretrained("TinyLlama-1.1B-qlora")

12.2.5 Merging the Adapter

What training produces is the LoRA adapter weights; before inference, merge them back into the base model:

python

from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained(
    "TinyLlama-1.1B-qlora",
    low_cpu_mem_usage=True,
    device_map="auto",
)

# Merge LoRA and base model
merged_model = model.merge_and_unload()

12.2.6 Inference

Test the instruction-tuned model using TinyLlama's predefined prompt template:

python

from transformers import pipeline

# Use our predefined prompt template
prompt = """<|user|>
Tell me something about Large Language Models.</s>
<|assistant|>
"""

# Run our instruction-tuned model
pipe = pipeline(task="text-generation", model=merged_model, tokenizer=tokenizer)
print(pipe(prompt)[0]["generated_text"])

12.3 Preference Alignment Fine-Tuning (PPO/DPO)

12.3.1 Data Preprocessing

DPO requires data in the shape of "the same prompt paired with a chosen answer and a rejected answer." Here we use the argilla/distilabel-intel-orca-dpo-pairs dataset, filtering out ties, low-score samples, and samples belonging to the GSM8K training set:

python

from datasets import load_dataset

def format_prompt(example):
    """Format the prompt to using the <|user|> template TinyLLama is using"""

    # Format answers
    system = "<|system|>\n" + example['system'] + "</s>\n"
    prompt = "<|user|>\n" + example['input'] + "</s>\n<|assistant|>\n"
    chosen = example['chosen'] + "</s>\n"
    rejected = example['rejected'] + "</s>\n"

    return {
        "prompt": system + prompt,
        "chosen": chosen,
        "rejected": rejected,
    }

# Apply formatting to the dataset and select relatively short answers
dpo_dataset = load_dataset("argilla/distilabel-intel-orca-dpo-pairs", split="train")
dpo_dataset = dpo_dataset.filter(
    lambda r:
        r["status"] != "tie" and
        r["chosen_score"] >= 8 and
        not r["in_gsm8k_train"]
)
dpo_dataset = dpo_dataset.map(format_prompt, remove_columns=dpo_dataset.column_names)
dpo_dataset

12.3.2 Model Quantization

Reload the QLoRA weights saved during the SFT phase, merge them, and prepare the base model for the DPO phase:

python

from peft import AutoPeftModelForCausalLM
from transformers import BitsAndBytesConfig, AutoTokenizer

# 4-bit quantization configuration - Q in QLoRA
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,  # Use 4-bit precision model loading
    bnb_4bit_quant_type="nf4",  # Quantization type
    bnb_4bit_compute_dtype="float16",  # Compute dtype
    bnb_4bit_use_double_quant=True,  # Apply nested quantization
)

# Merge LoRA and base model
model = AutoPeftModelForCausalLM.from_pretrained(
    "TinyLlama-1.1B-qlora",
    low_cpu_mem_usage=True,
    device_map="auto",
    quantization_config=bnb_config,
)
merged_model = model.merge_and_unload()

# Load LLaMA tokenizer
model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
tokenizer.pad_token = "<PAD>"
tokenizer.padding_side = "left"

12.3.3 Configuration

The LoRA configuration is the same as the SFT phase:

python

from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model

# Prepare LoRA Configuration
peft_config = LoraConfig(
    lora_alpha=32,  # LoRA Scaling
    lora_dropout=0.1,  # Dropout for LoRA Layers
    r=64,  # Rank
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=  # Layers to target
     ['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
)

# prepare model for training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, peft_config)

DPO uses the specialized DPOConfig; the learning rate is smaller (1e-5), and max_steps controls training steps:

python

from trl import DPOConfig

output_dir = "./results"

# Training arguments
training_arguments = DPOConfig(
    output_dir=output_dir,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    optim="paged_adamw_32bit",
    learning_rate=1e-5,
    lr_scheduler_type="cosine",
    max_steps=200,
    logging_steps=10,
    fp16=True,
    gradient_checkpointing=True,
    warmup_ratio=0.1
)

12.3.4 DPO Training

DPOTrainer controls the deviation from the reference model via beta=0.1; after training, save the DPO adapter:

python

from trl import DPOTrainer

# Create DPO trainer
dpo_trainer = DPOTrainer(
    model,
    args=training_arguments,
    train_dataset=dpo_dataset,
    tokenizer=tokenizer,
    peft_config=peft_config,
    beta=0.1,
    max_prompt_length=512,
    max_length=512,
)

# Fine-tune model with DPO
dpo_trainer.train()

# Save adapter
dpo_trainer.model.save_pretrained("TinyLlama-1.1B-dpo-qlora")

Finally, merge the SFT model and DPO adapter layer by layer to obtain the final preference-aligned model:

python

from peft import PeftModel

# Merge LoRA and base model
model = AutoPeftModelForCausalLM.from_pretrained(
    "TinyLlama-1.1B-qlora",
    low_cpu_mem_usage=True,
    device_map="auto",
)
sft_model = model.merge_and_unload()

# Merge DPO LoRA and SFT model
dpo_model = PeftModel.from_pretrained(
    sft_model,
    "TinyLlama-1.1B-dpo-qlora",
    device_map="auto",
)
dpo_model = dpo_model.merge_and_unload()

Compare output quality before and after DPO using the same prompt template:

python

from transformers import pipeline

# Use our predefined prompt template
prompt = """<|user|>
Tell me something about Large Language Models.</s>
<|assistant|>
"""

# Run our instruction-tuned model
pipe = pipeline(task="text-generation", model=dpo_model, tokenizer=tokenizer)
print(pipe(prompt)[0]["generated_text"])

Chapter Summary

The two-step method for fine-tuning generative LLMs: first SFT instruction tuning teaches the model "how to answer," then DPO preference alignment teaches it "to answer better";
QLoRA = 4-bit NF4 quantization + LoRA: BitsAndBytesConfig handles quantization configuration, enabling fine-tuning of a 1.1B model on a single T4;
The chat template must match the base model: reuse TinyLlama tokenizer's apply_chat_template directly to avoid format mismatch;
SFT uses SFTTrainer + TrainingArguments; DPO uses DPOTrainer + DPOConfig, with the latter additionally requiring chosen/rejected paired data and a beta strength parameter;
Multi-stage adapters can be stacked and merged: QLoRA → merge → attach DPO LoRA → merge again, yielding a single complete model.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What does the "Q" in QLoRA stand for?

2. What is the core difference between DPO data and SFT data?

3. Regarding the training configurations for SFT and DPO, which statement is correct?

4. How are the LoRA adapters from the SFT and DPO phases combined into the final model?

🛠️ Hands-on Practice

Change the SFT data size from range(3_000) to range(1_000); keep all other hyperparameters unchanged; compare the coherence differences in inference outputs across the two data scales.
Change LoRA's r from 64 to 16 and lora_alpha from 32 to 16; rerun the full SFT流程; compare changes in training VRAM usage and generation quality.
In the DPO data filtering condition, relax chosen_score >= 8 to >= 5; observe the change in available sample count; and compare the model's response style before and after the relaxation.

Chapter 12 · Fine-Tuning Generative Models ​

12.1 Environment Preparation (Optional) ​

12.2 Supervised Fine-Tuning (SFT) ​

12.2.1 Data Preprocessing ​

12.2.2 Model Quantization ​

12.2.3 Configuration ​

LoRA Configuration ​

Training Configuration ​

12.2.4 Start Training! ​

12.2.5 Merging the Adapter ​

12.2.6 Inference ​

12.3 Preference Alignment Fine-Tuning (PPO/DPO) ​

12.3.1 Data Preprocessing ​

12.3.2 Model Quantization ​

12.3.3 Configuration ​

12.3.4 DPO Training ​

Chapter Summary ​

🧪 随堂测验

🛠️ Hands-on Practice ​

Chapter 12 · Fine-Tuning Generative Models

12.1 Environment Preparation (Optional)

12.2 Supervised Fine-Tuning (SFT)

12.2.1 Data Preprocessing

12.2.2 Model Quantization

12.2.3 Configuration

LoRA Configuration

Training Configuration

12.2.4 Start Training!

12.2.5 Merging the Adapter

12.2.6 Inference

12.3 Preference Alignment Fine-Tuning (PPO/DPO)

12.3.1 Data Preprocessing

12.3.2 Model Quantization

12.3.3 Configuration

12.3.4 DPO Training

Chapter Summary

🛠️ Hands-on Practice