SkilloptionalHermes Optional Skills

Trl Fine Tuning

Name: Trl Fine Tuning
Author: Orchestra Research

Trl Fine Tuning: مهارة ينشرها Orchestra Research. يوفّر السجل أمر تثبيت جاهزًا يظهر في هذه الصفحة. الرخصة MIT. الإصدار الموثّق 1.0.1. تُفعَّل بأمر `hermes skills install trl-fine-tuning` بعد مراجعة ما تطلبه. تعتمد على: trl, transformers, datasets, peft, accelerate, torch. تدعم: linux، macos، windows.

Post-TrainingTRLReinforcement LearningFine-TuningSFTDPOGRPORLOO

آخر تحقق من السجل2026-08-18v1.0.1Orchestra Research

افتح المصدر الأصلي ↗Read in English

المعنى ببساطة

ماذا يضيف إلى Hermes؟

Trl Fine Tuning: مهارة ينشرها Orchestra Research. يوفّر السجل أمر تثبيت جاهزًا يظهر في هذه الصفحة. الرخصة MIT. الإصدار الموثّق 1.0.1. تُفعَّل بأمر hermes skills install trl-fine-tuning بعد مراجعة ما تطلبه. تعتمد على: trl, transformers, datasets, peft, accelerate, torch. تدعم: linux، macos، windows.

Trl Fine Tuning هي مهارة مرتبطة بمجال توسيع قدرات الوكيل. يضيف قدرة أو سير عمل إلى Hermes. الوصف الأصلي يحدد التفاصيل، بينما تحدد الصلاحيات ما يستطيع فعله فعليًا.

هذا تفسير مبسّط مبني على وصف الناشر. أبقينا الوصف الإنجليزي بجانبه حتى تستطيع مقارنة المعنى بالمصدر.

استخدمه عندما

استخدمه عندما يكون هدفك واضحًا في توسيع قدرات الوكيل وتستطيع تحديد البيانات والأفعال التي يحتاجها فقط.

لا تحتاجه عندما

لا تضفه لمجرد التجربة إذا كان لديك طريق أبسط داخل Hermes، أو إذا لم تستطع مراجعة المصدر والصلاحيات.

لمن يناسب؟

مناسب لمن يريد طريقة عمل قابلة للتكرار داخل Hermes.

أول اختبار آمن

ابدأ ببيانات غير حساسة ومهمة صغيرة يمكن التحقق من نتيجتها والتراجع عنها.

الوصف الأصلي من الناشر، من دون ترجمة تغيّر المعنى

TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF

✓

مصدر البيانات

فُهرس هذا الإدخال من Hermes Optional Skills. الشرح العربي يفسّر النوع والمجال ولا يضيف وظيفة غير مذكورة في المصدر.

مراجعة الأمان

المصدر رسمي أو خضع لمراجعة تحريرية، لكن ذلك لا يغني عن مراجعة الصلاحيات والإصدار.

مسار تثبيت آمن

افحص، ثبّت، ثم اختبر.

01
افتح المصدر
طابق اسم الناشر والرخصة والوصف مع حاجتك، وراجع آخر تحديث فعلي.
02
راجع الصلاحيات والأسرار
لا تلصق قيمة سر داخل الموقع. استخدم أسماء متغيرات البيئة وامنح أقل نطاق ممكن.
03
انسخ الإعداد فقط بعد المراجعة
الأزرار أدناه تنسخ نصًا إلى الحافظة ولا تشغّل أمرًا على جهازك.
04
اختبر بمهمة غير حساسة
تحقق من الأدوات الظاهرة، ثم استبعد أدوات الكتابة أو الحذف التي لا تحتاجها.

أمر التثبيت

راجع الأمر ثم انسخه.

hermes skills install trl-fine-tuning

لا ينفّذ Hermes بالعربي هذا الأمر. التثبيت يحدث داخل جهازك ويظل خاضعًا لفحص Hermes ومراجعتك.

تعريف المهارة كاملًا

ما الذي يحمّله Hermes بالضبط عند تشغيل هذه المهارة.

منقول من التوثيق الرسمي. اقرأه قبل تفعيل المهارة، فهذا النص يصبح تعليمات الوكيل نفسه.

TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.

Skill metadata

جدول مرجعي. لا تقرأه كله، ابحث عن السطر الذي يخصّك فقط.

Source	Optional — install with `hermes skills install official/mlops/trl-fine-tuning`
Path	`optional-skills/mlops/training/trl-fine-tuning`
Version	`1.0.1`
Author	Orchestra Research
License	MIT
Dependencies	`trl`, `transformers`, `datasets`, `peft`, `accelerate`, `torch`
Platforms	linux, macos, windows
Tags	`Post-Training`, `TRL`, `Reinforcement Learning`, `Fine-Tuning`, `SFT`, `DPO`, `GRPO`, `RLOO`, `RLHF`, `Preference Alignment`, `HuggingFace`

Reference: full SKILL.md

شرح للفكرة نفسها. اقرأه ببطء، فبقية الأقسام تبني عليه.

Quick start

خطوات عملية بالترتيب. نفّذ خطوة وتأكد أنها نجحت قبل الانتقال للتالية.

TRL provides post-training methods for aligning language models with human preferences.

Installation:

Shellسطر واحد

pip install trl transformers datasets peft accelerate

Supervised Fine-Tuning (instruction tuning):

Python7 أسطر

from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()

DPO (align with preferences):

Python10 أسطر

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()

Common workflows

إعدادات تضبطها مرة وتنساها. غيّر واحدًا في كل مرة حتى تعرف أثر كل تغيير.

Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO)

Complete pipeline from base model to human-aligned model.

Note (TRL 1.x): PPO has been removed from TRL — PPOTrainer, PPOConfig, and python -m trl.scripts.ppo no longer exist. Use an online-RL trainer TRL still ships: RLOO (RLOOTrainer / trl rloo) is the closest drop-in for a reward-model-driven RLHF pipeline, and GRPO (GRPOTrainer / trl grpo, see Workflow 3) is the memory-efficient alternative. The step below uses RLOO.

Copy this checklist:

Text5 أسطر

RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: RLOO reinforcement learning
- [ ] Step 4: Evaluate aligned model

Step 1: Supervised fine-tuning

Train base model on instruction-following data:

Python30 سطرًا

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer
)
trainer.train()
trainer.save_model()

Step 2: Train reward model

Train model to predict human preferences:

Python30 سطرًا

from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig

# Load SFT model as base
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-SFT",
    num_labels=1  # Single reward score
)
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")

# Load preference data (chosen/rejected pairs)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

# Configure training
training_args = RewardConfig(
    output_dir="Qwen2.5-0.5B-Reward",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=1e-5
)

# Train reward model
trainer = RewardTrainer(
    model=model,
    args=training_args,
    processing_class=tokenizer,
    train_dataset=dataset
)
trainer.train()
trainer.save_model()

Step 3: RLOO reinforcement learning

Optimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI (trl rloo) with the trained reward model passed via --reward_model_name_or_path:

Shell8 أسطر

trl rloo \
    --model_name_or_path Qwen2.5-0.5B-SFT \
    --reward_model_name_or_path Qwen2.5-0.5B-Reward \
    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
    --output_dir Qwen2.5-0.5B-RLOO \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 64 \
    --num_generations 4

Equivalent Python (RLOOTrainer / RLOOConfig):

Python22 سطرًا

from trl import RLOOTrainer, RLOOConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer

reward_model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-Reward", num_labels=1
)

config = RLOOConfig(
    output_dir="Qwen2.5-0.5B-RLOO",
    per_device_train_batch_size=64,
    learning_rate=3e-6,
    num_generations=4,
)

trainer = RLOOTrainer(
    model="Qwen2.5-0.5B-SFT",
    reward_funcs=reward_model,   # a reward model (or a callable reward function)
    args=config,
    train_dataset=dataset,       # prompt-only dataset
    processing_class=tokenizer,
)
trainer.train()

Step 4: Evaluate

Python9 أسطر

from transformers import pipeline

# Load aligned model
generator = pipeline("text-generation", model="Qwen2.5-0.5B-RLOO")

# Test
prompt = "Explain quantum computing to a 10-year-old"
output = generator(prompt, max_length=200)[0]["generated_text"]
print(output)

Workflow 2: Simple preference alignment with DPO

Align model with preferences without reward model.

Copy this checklist:

Text5 أسطر

DPO Training:
- [ ] Step 1: Prepare preference dataset
- [ ] Step 2: Configure DPO
- [ ] Step 3: Train with DPOTrainer
- [ ] Step 4: Evaluate alignment

Step 1: Prepare preference dataset

Dataset format:

JSON5 أسطر

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I don't know."
}

Load dataset:

Python5 أسطر

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# Or load your own
# dataset = load_dataset("json", data_files="preferences.json")

Step 2: Configure DPO

Python12 سطرًا

from trl import DPOConfig

config = DPOConfig(
    output_dir="Qwen2.5-0.5B-DPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=5e-7,
    beta=0.1,  # KL penalty strength
    max_prompt_length=512,
    max_length=1024,
    logging_steps=10
)

Step 3: Train with DPOTrainer

Python15 سطرًا

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer
)

trainer.train()
trainer.save_model()

CLI alternative:

Shell7 أسطر

trl dpo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-7 \
    --beta 0.1

Workflow 3: Memory-efficient online RL with GRPO

Train with reinforcement learning using minimal memory.

For in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see references/grpo-training.md ↗. A production-ready training script is in templates/basic_grpo_training.py ↗.

Copy this checklist:

Text4 أسطر

GRPO Training:
- [ ] Step 1: Define reward function
- [ ] Step 2: Configure GRPO
- [ ] Step 3: Train with GRPOTrainer

Step 1: Define reward function

Python17 سطرًا

def reward_function(completions, **kwargs):
    """
    Compute rewards for completions.

    Args:
        completions: List of generated texts

    Returns:
        List of reward scores (floats)
    """
    rewards = []
    for completion in completions:
        # Example: reward based on length and unique words
        score = len(completion.split())  # Favor longer responses
        score += len(set(completion.lower().split()))  # Reward unique words
        rewards.append(score)
    return rewards

Or use a reward model:

Python10 أسطر

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):
    # Combine prompt + completion
    full_texts = [p + c for p, c in zip(prompts, completions)]
    # Get reward scores
    results = reward_model(full_texts)
    return [r["score"] for r in results]

Step 2: Configure GRPO

Python10 أسطر

from trl import GRPOConfig

config = GRPOConfig(
    output_dir="Qwen2-GRPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=1e-5,
    num_generations=4,  # Generate 4 completions per prompt
    max_new_tokens=128
)

Step 3: Train with GRPOTrainer

Python14 سطرًا

from datasets import load_dataset
from trl import GRPOTrainer

# Load prompt-only dataset
dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_function,  # Your reward function
    args=config,
    train_dataset=dataset
)

trainer.train()

CLI:

Shell5 أسطر

trl grpo \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --dataset_name trl-lib/tldr \
    --output_dir Qwen2-GRPO \
    --num_generations 4

When to use vs alternatives

شرح للفكرة نفسها. اقرأه ببطء، فبقية الأقسام تبني عليه.

Use TRL when:

Need to align model with human preferences
Have preference data (chosen/rejected pairs)
Want to use reinforcement learning (RLOO, GRPO)
Need reward model training
Doing RLHF (full pipeline)

Method selection:

SFT: Have prompt-completion pairs, want basic instruction following
DPO: Have preferences, want simple alignment (no reward model needed)
RLOO: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)
GRPO: Memory-constrained, want online RL with reward functions
Reward Model: Building RLHF pipeline, need to score generations

Use alternatives instead:

HuggingFace Trainer: Basic fine-tuning without RL
Axolotl: YAML-based training configuration
LitGPT: Educational, minimal fine-tuning
Unsloth: Fast LoRA training

Common issues

شرح للفكرة نفسها. اقرأه ببطء، فبقية الأقسام تبني عليه.

Issue: OOM during DPO training

Reduce batch size and sequence length:

Python5 أسطر

config = DPOConfig(
    per_device_train_batch_size=1,  # Reduce from 4
    max_length=512,  # Reduce from 1024
    gradient_accumulation_steps=8  # Maintain effective batch
)

Or use gradient checkpointing:

Pythonسطر واحد

model.gradient_checkpointing_enable()

Issue: Poor alignment quality

Tune beta parameter:

Python5 أسطر

# Higher beta = more conservative (stays closer to reference)
config = DPOConfig(beta=0.5)  # Default 0.1

# Lower beta = more aggressive alignment
config = DPOConfig(beta=0.01)

Issue: Reward model not learning

Check loss type and learning rate:

Python4 أسطر

config = RewardConfig(
    learning_rate=1e-5,  # Try different LR
    num_train_epochs=3  # Train longer
)

Ensure preference dataset has clear winners:

Python3 أسطر

# Verify dataset
print(dataset[0])
# Should have clear chosen > rejected

Issue: Online RL (RLOO/GRPO) training unstable

Adjust the KL/beta regularization toward the reference policy:

Python6 أسطر

from trl import RLOOConfig

config = RLOOConfig(
    beta=0.05,          # KL coefficient toward the reference model (increase for stability)
    num_generations=4,  # more samples per prompt = lower-variance advantage estimates
)

Advanced topics

شرح للفكرة نفسها. اقرأه ببطء، فبقية الأقسام تبني عليه.

SFT training guide: See references/sft-training.md ↗ for dataset formats, chat templates, packing strategies, and multi-GPU training.

DPO variants: See references/dpo-variants.md ↗ for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.

Reward modeling: See references/reward-modeling.md ↗ for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.

Online RL methods: See references/online-rl.md ↗ for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.

GRPO deep dive: See references/grpo-training.md ↗ for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in templates/basic_grpo_training.py ↗.

Hardware requirements

شرح للفكرة نفسها. اقرأه ببطء، فبقية الأقسام تبني عليه.

GPU: NVIDIA (CUDA required)
VRAM: Depends on model and method
SFT 7B: 16GB (with LoRA)
DPO 7B: 24GB (stores reference model)
RLOO 7B: 40GB (policy + reward model)
GRPO 7B: 24GB (more memory efficient)
Multi-GPU: Supported via accelerate
Mixed precision: BF16 recommended (A100/H100)

Memory optimization:

Use LoRA/QLoRA for all methods
Enable gradient checkpointing
Use smaller batch sizes with gradient accumulation

Resources

شرح للفكرة نفسها. اقرأه ببطء، فبقية الأقسام تبني عليه.

Docs: https://huggingface.co/docs/trl/
GitHub: https://github.com/huggingface/trl
Papers:
"Training language models to follow instructions with human feedback" (InstructGPT, 2022)
"Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)
"Group Relative Policy Optimization" (GRPO, 2024)
Examples: https://github.com/huggingface/trl/tree/main/examples/scripts