Simpo — Reference-free preference alignment, simpler than DPO
Simpo — Reference-free preference alignment, simpler than DPO
Start with meaning, then move to detail.
This lesson explains Simpo — Reference-free preference alignment, simpler than DPO as part of extending Hermes and connecting external tools. You will learn what it does, when it matters, and the smallest safe test that proves it works.
If you are new, do not memorize names. Focus on three questions: what problem does this solve, what access does it need, and how can you verify the result?
For practice, inspect the first example, identify its effects, run it on test data, and compare the result with the source claim.
For advanced readers, inspect Skill metadata, Reference: full SKILL.md, Quick start, then verify failure modes and version compatibility.
Complete installation and one successful task before adding new capabilities.
A clear outcome before you read.
- Understand Simpo — Reference-free preference alignment, simpler than DPO without assumed prior knowledge.
- Separate the source description from what still needs testing in your environment.
- Read the first command and identify its inputs and outputs before copying it.
Short definitions before the details.
- Provider
- The service that runs or provides access and authentication to a model.
- Skill
- An instruction bundle that teaches Hermes a repeatable workflow without necessarily adding an external service.
Reference-free preference alignment, simpler than DPO
What does the source say, and in what order?
- 01Skill metadata
Start here to understand the core idea or structure.
- 02Reference: full SKILL.md
Read this after the foundation, then connect it to the previous step.
- 03Quick start
Read this after the foundation, then connect it to the previous step.
- 04Common workflows
Read this after the foundation, then connect it to the previous step.
- 05Workflow 1: Train from base model (Mistral 7B)
Read this after the foundation, then connect it to the previous step.
- 06Workflow 2: Fine-tune instruct model (Llama 3 8B)
Read this after the foundation, then connect it to the previous step.
- 07Workflow 3: Reasoning-intensive tasks (lower LR)
Read this after the foundation, then connect it to the previous step.
- 08When to use vs alternatives
Read this after the foundation, then connect it to the previous step.
- 09Common issues
Read this after the foundation, then connect it to the previous step.
- 10Advanced topics
Finish here to verify the result and special cases.
Copy only after you understand the effect.
# Create environment
conda create -n simpo python=3.10 && conda activate simpo
# Install PyTorch 2.2.2
# Visit: https://pytorch.org/get-started/locally/
# Install alignment-handbook
git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook
python -m pip install .
# Install Flash Attention 2
python -m pip install flash-attn --no-build-isolationACCELERATE_LOG_LEVEL=info accelerate launch \
--config_file accelerate_configs/deepspeed_zero3.yaml \
scripts/run_simpo.py \
training_configs/mistral-7b-base-simpo.yaml# Model
model_name_or_path: mistralai/Mistral-7B-v0.1
torch_dtype: bfloat16
# Dataset
dataset_mixer:
HuggingFaceH4/ultrafeedback_binarized: 1.0
dataset_splits:
- train_prefs
- test_prefs
# SimPO hyperparameters
beta: 2.0 # Reward scaling (2.0-10.0)
gamma_beta_ratio: 0.5 # Target margin (0-1)
loss_type: sigmoid # sigmoid or hinge
sft_weight: 0.0 # Optional SFT regularization
# Training
learning_rate: 5e-7 # Critical: 3e-7 to 1e-6
num_train_epochs: 1
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
# Output
output_dir: ./Read the first command and identify its inputs and outputs before copying it.
Match every command to your installed Hermes version, review the files and accounts it can reach, and use non-sensitive data for the first test. If this explanation differs from the source, the official source wins.