SKILLoptionalHermes Optional Skills

Accelerate

Name: Accelerate
Author: Orchestra Research

Run PyTorch training across GPUs with minimal changes

Distributed TrainingHuggingFaceAccelerateDeepSpeedFSDPMixed PrecisionPyTorchDDP

Last registry verification2026-08-18v1.0.1Orchestra Research

Open original source ↗اقرأ بالعربية

Plain meaning

What does it add to Hermes?

Run PyTorch training across GPUs with minimal changes

Accelerate is a skill related to extending the agent. It adds a capability or workflow to Hermes. The publisher description explains the intent, while granted permissions determine what it can actually do.

This plain-language explanation is based on the publisher description. The original text remains visible for verification.

Use it when

Use it when your goal in extending the agent is clear and you can limit it to the data and actions it actually needs.

Skip it when

Do not add it merely to experiment when Hermes already has a simpler path, or when you cannot review its source and permissions.

Who is it for?

Best for users who want a repeatable way of working inside Hermes.

Safe first test

Start with non-sensitive data and a small task whose result can be verified and reversed.

Original publisher description

Run PyTorch training across GPUs with minimal changes

✓

Data source

This entry was indexed from Hermes Optional Skills. Our explanation interprets the type and domain without inventing a capability not present upstream.

Security review

The source is official or editorially reviewed, but you still need to review permissions and version compatibility.

Safe setup path

Inspect, install, then test.

01
Open the source
Match the publisher, license, and description to your need. Check the real update history.
02
Review permissions and secrets
Never paste a secret value into this site. Use environment-variable names and grant the smallest scope.
03
Copy setup only after review
The controls below copy text. They do not execute commands on your device.
04
Test with a non-sensitive task
Inspect the visible tools, then exclude write or delete tools you do not need.

Install command

Review the command, then copy it.

hermes skills install accelerate

Hermes Belarabi does not execute this command. Installation happens on your device and remains subject to Hermes scanning and your review.

The full skill definition

Exactly what Hermes loads when this skill runs.

Reproduced from the official documentation. Read it before enabling the skill: this text becomes the agent's instructions.

Run PyTorch training across GPUs with minimal changes.

Skill metadata

A lookup table. Do not read it all; find the row that applies to you.

Source	Optional — install with `hermes skills install official/mlops/accelerate`
Path	`optional-skills/mlops/accelerate`
Version	`1.0.1`
Author	Orchestra Research
License	MIT
Dependencies	`accelerate`, `torch`, `transformers`
Platforms	linux, macos, windows
Tags	`Distributed Training`, `HuggingFace`, `Accelerate`, `DeepSpeed`, `FSDP`, `Mixed Precision`, `PyTorch`, `DDP`, `Unified API`, `Simple`

Reference: full SKILL.md

Explains the idea itself. Read it slowly; the later sections build on it.

Quick start

Ordered, practical steps. Run one and confirm it worked before moving on.

Accelerate simplifies distributed training to 4 lines of code.

Installation:

Shell1 line

pip install accelerate

Convert PyTorch script (4 lines):

Python17 lines


+ from accelerate import Accelerator

+ accelerator = Accelerator()

  model = torch.nn.Transformer()
  optimizer = torch.optim.Adam(model.parameters())
  dataloader = torch.utils.data.DataLoader(dataset)

+ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

  for batch in dataloader:
      optimizer.zero_grad()
      loss = model(batch)
-     loss.backward()
+     accelerator.backward(loss)
      optimizer.step()

Run (single command):

Shell1 line

accelerate launch train.py

Common workflows

Settings you configure once. Change one at a time so you can see what each does. Set MASTER_ADDR, FULL_SHARD in your environment, not in the chat.

Workflow 1: From single GPU to multi-GPU

Original script:

Python14 lines

# train.py


model = torch.nn.Linear(10, 2).to('cuda')
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)

for epoch in range(10):
    for batch in dataloader:
        batch = batch.to('cuda')
        optimizer.zero_grad()
        loss = model(batch).mean()
        loss.backward()
        optimizer.step()

With Accelerate (4 lines added):

Python19 lines

# train.py

from accelerate import Accelerator  # +1

accelerator = Accelerator()  # +2

model = torch.nn.Linear(10, 2)
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)  # +3

for epoch in range(10):
    for batch in dataloader:
        # No .to('cuda') needed - automatic!
        optimizer.zero_grad()
        loss = model(batch).mean()
        accelerator.backward(loss)  # +4
        optimizer.step()

Configure (interactive):

Shell1 line

accelerate config

Questions:

Which machine? (single/multi GPU/TPU/CPU)
How many machines? (1)
Mixed precision? (no/fp16/bf16/fp8)
DeepSpeed? (no/yes)

Launch (works on any setup):

Shell11 lines

# Single GPU
accelerate launch train.py

# Multi-GPU (8 GPUs)
accelerate launch --multi_gpu --num_processes 8 train.py

# Multi-node
accelerate launch --multi_gpu --num_processes 16 \
  --num_machines 2 --machine_rank 0 \
  --main_process_ip $MASTER_ADDR \
  train.py

Workflow 2: Mixed precision training

Enable FP16/BF16:

Python18 lines

from accelerate import Accelerator

# FP16 (with gradient scaling)
accelerator = Accelerator(mixed_precision='fp16')

# BF16 (no scaling, more stable)
accelerator = Accelerator(mixed_precision='bf16')

# FP8 (H100+)
accelerator = Accelerator(mixed_precision='fp8')

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

# Everything else is automatic!
for batch in dataloader:
    with accelerator.autocast():  # Optional, done automatically
        loss = model(batch)
    accelerator.backward(loss)

Workflow 3: DeepSpeed ZeRO integration

Enable DeepSpeed ZeRO-2 (pass a DeepSpeedPlugin, not a raw dict):

Python15 lines

from accelerate import Accelerator, DeepSpeedPlugin

deepspeed_plugin = DeepSpeedPlugin(
    zero_stage=2,                     # ZeRO-2
    offload_optimizer_device="none",  # or "cpu" to offload
    gradient_accumulation_steps=4,
)

accelerator = Accelerator(
    mixed_precision='bf16',
    deepspeed_plugin=deepspeed_plugin,  # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin])
)

# Same code as before!
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

Or point at a full DeepSpeed JSON config via the plugin:

Python5 lines

from accelerate import Accelerator, DeepSpeedPlugin

# hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict)
deepspeed_plugin = DeepSpeedPlugin(hf_ds_config="ds_config.json")
accelerator = Accelerator(mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin)

ds_config.json (a raw DeepSpeed config — passed via the plugin, NOT via --config_file):

JSON10 lines

{
    "fp16": {"enabled": false},
    "bf16": {"enabled": true},
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {"device": "cpu"},
        "allgather_bucket_size": 5e8,
        "reduce_bucket_size": 5e8
    }
}

Or via interactive config:

Shell3 lines

accelerate config
# Select: DeepSpeed → ZeRO-2
# This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml)

Launch (--config_file expects an accelerate YAML, not a raw DeepSpeed JSON):

Shell5 lines

# Uses the default accelerate config written by `accelerate config`
accelerate launch train.py

# Or point at a specific accelerate YAML
accelerate launch --config_file accelerate_deepspeed.yaml train.py

Workflow 4: FSDP (Fully Sharded Data Parallel)

Enable FSDP:

Python14 lines

from accelerate import Accelerator, FullyShardedDataParallelPlugin

fsdp_plugin = FullyShardedDataParallelPlugin(
    sharding_strategy="FULL_SHARD",  # ZeRO-3 equivalent
    auto_wrap_policy="transformer_based_wrap",  # valid: transformer_based_wrap | size_based_wrap | no_wrap
    cpu_offload=False
)

accelerator = Accelerator(
    mixed_precision='bf16',
    fsdp_plugin=fsdp_plugin
)

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

Or via config:

Shell2 lines

accelerate config
# Select: FSDP → Full Shard → No CPU Offload

Workflow 5: Gradient accumulation

Accumulate gradients:

Python12 lines

from accelerate import Accelerator

accelerator = Accelerator(gradient_accumulation_steps=4)

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

for batch in dataloader:
    with accelerator.accumulate(model):  # Handles accumulation
        optimizer.zero_grad()
        loss = model(batch)
        accelerator.backward(loss)
        optimizer.step()

Effective batch size: batch_size * num_gpus * gradient_accumulation_steps

When to use vs alternatives

Explains the idea itself. Read it slowly; the later sections build on it.

Use Accelerate when:

Want simplest distributed training
Need single script for any hardware
Use HuggingFace ecosystem
Want flexibility (DDP/DeepSpeed/FSDP/Megatron)
Need quick prototyping

Key advantages:

4 lines: Minimal code changes
Unified API: Same code for DDP, DeepSpeed, FSDP, Megatron
Automatic: Device placement, mixed precision, sharding
Interactive config: No manual launcher setup
Single launch: Works everywhere

Use alternatives instead:

PyTorch Lightning: Need callbacks, high-level abstractions
Ray Train: Multi-node orchestration, hyperparameter tuning
DeepSpeed: Direct API control, advanced features
Raw DDP: Maximum control, minimal abstraction

Common issues

Explains the idea itself. Read it slowly; the later sections build on it.

Issue: Wrong device placement

Don't manually move to device:

Python5 lines

# WRONG
batch = batch.to('cuda')

# CORRECT
# Accelerate handles it automatically after prepare()

Issue: Gradient accumulation not working

Use context manager:

Python5 lines

# CORRECT
with accelerator.accumulate(model):
    optimizer.zero_grad()
    accelerator.backward(loss)
    optimizer.step()

Issue: Checkpointing in distributed

Use accelerator methods:

Python6 lines

# Save only on main process
if accelerator.is_main_process:
    accelerator.save_state('checkpoint/')

# Load on all processes
accelerator.load_state('checkpoint/')

Issue: Different results with FSDP

Ensure same random seed:

Python2 lines

from accelerate.utils import set_seed
set_seed(42)

Advanced topics

Explains the idea itself. Read it slowly; the later sections build on it.

Megatron integration: See references/megatron-integration.md ↗ for tensor parallelism, pipeline parallelism, and sequence parallelism setup.

Custom plugins: See references/custom-plugins.md ↗ for creating custom distributed plugins and advanced configuration.

Performance tuning: See references/performance.md ↗ for profiling, memory optimization, and best practices.

Hardware requirements

Explains the idea itself. Read it slowly; the later sections build on it.

CPU: Works (slow)
Single GPU: Works
Multi-GPU: DDP (default), DeepSpeed, or FSDP
Multi-node: DDP, DeepSpeed, FSDP, Megatron
TPU: Supported
Apple MPS: Supported

Launcher requirements:

DDP: torch.distributed.run (built-in)
DeepSpeed: deepspeed (pip install deepspeed)
FSDP: PyTorch 1.12+ (built-in)
Megatron: Custom setup

Resources

Explains the idea itself. Read it slowly; the later sections build on it.

Docs: https://huggingface.co/docs/accelerate
GitHub: https://github.com/huggingface/accelerate
Version: 1.11.0+
Tutorial: "Accelerate your scripts"
Examples: https://github.com/huggingface/accelerate/tree/main/examples
Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries