SKILLbundledHermes Bundled Skills

Evaluating Llms Harness

Name: Evaluating Llms Harness
Author: Orchestra Research

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.)

EvaluationLM Evaluation HarnessBenchmarkingMMLUHumanEvalGSM8KEleutherAIModel Quality

Last registry verification2026-08-18v1.0.1Orchestra Research

Open original source ↗اقرأ بالعربية

Plain meaning

What does it add to Hermes?

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.)

Evaluating Llms Harness is a skill related to extending the agent. It adds a capability or workflow to Hermes. The publisher description explains the intent, while granted permissions determine what it can actually do.

This plain-language explanation is based on the publisher description. The original text remains visible for verification.

Use it when

Use it when your goal in extending the agent is clear and you can limit it to the data and actions it actually needs.

Skip it when

Do not add it merely to experiment when Hermes already has a simpler path, or when you cannot review its source and permissions.

Who is it for?

Best for users who want a repeatable way of working inside Hermes.

Safe first test

Start with non-sensitive data and a small task whose result can be verified and reversed.

Original publisher description

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.)

✓

Data source

This entry was indexed from Hermes Bundled Skills. Our explanation interprets the type and domain without inventing a capability not present upstream.

Security review

The source is official or editorially reviewed, but you still need to review permissions and version compatibility.

Safe setup path

Inspect, install, then test.

01
Open the source
Match the publisher, license, and description to your need. Check the real update history.
02
Review permissions and secrets
Never paste a secret value into this site. Use environment-variable names and grant the smallest scope.
03
Copy setup only after review
The controls below copy text. They do not execute commands on your device.
04
Test with a non-sensitive task
Inspect the visible tools, then exclude write or delete tools you do not need.

Setup method

Already installed with Hermes.

This skill ships with Hermes and loads when the agent decides it is relevant. There is nothing to install; read the definition below so you know what it will do.

Open the official page ↗

The full skill definition

Exactly what Hermes loads when this skill runs.

Reproduced from the official documentation. Read it before enabling the skill: this text becomes the agent's instructions.

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).

Skill metadata

A lookup table. Do not read it all; find the row that applies to you.

Source	Bundled (installed by default)
Path	`skills/mlops/evaluation/evaluating-llms-harness`
Version	`1.0.1`
Author	Orchestra Research
License	MIT
Dependencies	`lm-eval`, `transformers`, `vllm`
Platforms	linux, macos
Tags	`Evaluation`, `LM Evaluation Harness`, `Benchmarking`, `MMLU`, `HumanEval`, `GSM8K`, `EleutherAI`, `Model Quality`, `Academic Benchmarks`, `Industry Standard`

Reference: full SKILL.md

Explains the idea itself. Read it slowly; the later sections build on it.

What's inside

Explains the idea itself. Read it slowly; the later sections build on it.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quick start

Ordered, practical steps. Run one and confirm it worked before moving on.

lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Installation:

Shell1 line

pip install lm-eval

Evaluate any HuggingFace model:

Shell5 lines

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag \
  --device cuda:0 \
  --batch_size 8

View available tasks:

Shell1 line

lm-eval ls tasks

Common workflows

Settings you configure once. Change one at a time so you can see what each does. Set CHECKPOINT_DIR in your environment, not in the chat.

Workflow 1: Standard benchmark evaluation

Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval).

Copy this checklist:

Text5 lines

Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model
- [ ] Step 3: Run evaluation
- [ ] Step 4: Analyze results

Step 1: Choose benchmark suite

Core reasoning benchmarks:

MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice
GSM8K - Grade school math word problems
HellaSwag - Common sense reasoning
TruthfulQA - Truthfulness and factuality
ARC (AI2 Reasoning Challenge) - Science questions

Code benchmarks:

HumanEval - Python code generation (164 problems)
MBPP (Mostly Basic Python Problems) - Python coding

Standard suite (recommended for model releases):

Shell1 line

--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge

Step 2: Configure model

HuggingFace model:

Shell5 lines

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16 \
  --tasks mmlu \
  --device cuda:0 \
  --batch_size auto  # Auto-detect optimal batch size

Quantized model (4-bit/8-bit):

Shell4 lines

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,load_in_4bit=True \
  --tasks mmlu \
  --device cuda:0

Custom checkpoint:

Shell4 lines

lm_eval --model hf \
  --model_args pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer \
  --tasks mmlu \
  --device cuda:0

Step 3: Run evaluation

Shell16 lines

# Full MMLU evaluation (57 subjects)
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu \
  --num_fewshot 5 \  # 5-shot evaluation (standard)
  --batch_size 8 \
  --output_path results/ \
  --log_samples  # Save individual predictions

# Multiple benchmarks at once
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \
  --num_fewshot 5 \
  --batch_size 8 \
  --output_path results/llama2-7b-eval.json

Step 4: Analyze results

Results saved to results/llama2-7b-eval.json:

JSON21 lines

{
  "results": {
    "mmlu": {
      "acc": 0.459,
      "acc_stderr": 0.004
    },
    "gsm8k": {
      "exact_match": 0.142,
      "exact_match_stderr": 0.006
    },
    "hellaswag": {
      "acc_norm": 0.765,
      "acc_norm_stderr": 0.004
    }
  },
  "config": {
    "model": "hf",
    "model_args": "pretrained=meta-llama/Llama-2-7b-hf",
    "num_fewshot": 5
  }
}

Workflow 2: Track training progress

Evaluate checkpoints during training.

Text5 lines

Training Progress Tracking:
- [ ] Step 1: Set up periodic evaluation
- [ ] Step 2: Choose quick benchmarks
- [ ] Step 3: Automate evaluation
- [ ] Step 4: Plot learning curves

Step 1: Set up periodic evaluation

Evaluate every N training steps:

Shell12 lines

#!/bin/bash
# eval_checkpoint.sh

CHECKPOINT_DIR=$1
STEP=$2

lm_eval --model hf \
  --model_args pretrained=$CHECKPOINT_DIR/checkpoint-$STEP \
  --tasks gsm8k,hellaswag \
  --num_fewshot 0 \  # 0-shot for speed
  --batch_size 16 \
  --output_path results/step-$STEP.json

Step 2: Choose quick benchmarks

Fast benchmarks for frequent evaluation:

HellaSwag: ~10 minutes on 1 GPU
GSM8K: ~5 minutes
PIQA: ~2 minutes

Avoid for frequent eval (too slow):

MMLU: ~2 hours (57 subjects)
HumanEval: Requires code execution

Step 3: Automate evaluation

Integrate with training script:

Python6 lines

# In training loop
if step % eval_interval == 0:
    model.save_pretrained(f"checkpoints/step-{step}")

    # Run evaluation
    os.system(f"./eval_checkpoint.sh checkpoints step-{step}")

Or use PyTorch Lightning callbacks:

Python12 lines

from pytorch_lightning import Callback

class EvalHarnessCallback(Callback):
    def on_validation_epoch_end(self, trainer, pl_module):
        step = trainer.global_step
        checkpoint_path = f"checkpoints/step-{step}"

        # Save checkpoint
        trainer.save_checkpoint(checkpoint_path)

        # Run lm-eval
        os.system(f"lm_eval --model hf --model_args pretrained={checkpoint_path} ...")

Step 4: Plot learning curves

Python20 lines




# Load all results
steps = []
mmlu_scores = []

for file in sorted(glob.glob("results/step-*.json")):
    with open(file) as f:
        data = json.load(f)
        step = int(file.split("-")[1].split(".")[0])
        steps.append(step)
        mmlu_scores.append(data["results"]["mmlu"]["acc"])

# Plot
plt.plot(steps, mmlu_scores)
plt.xlabel("Training Step")
plt.ylabel("MMLU Accuracy")
plt.title("Training Progress")
plt.savefig("training_curve.png")

Workflow 3: Compare multiple models

Benchmark suite for model comparison.

Text4 lines

Model Comparison:
- [ ] Step 1: Define model list
- [ ] Step 2: Run evaluations
- [ ] Step 3: Generate comparison table

Step 1: Define model list

Shell5 lines

# models.txt
meta-llama/Llama-2-7b-hf
meta-llama/Llama-2-13b-hf
mistralai/Mistral-7B-v0.1
microsoft/phi-2

Step 2: Run evaluations

Shell19 lines

#!/bin/bash
# eval_all_models.sh

TASKS="mmlu,gsm8k,hellaswag,truthfulqa"

while read model; do
    echo "Evaluating $model"

    # Extract model name for output file
    model_name=$(echo $model | sed 's/\//-/g')

    lm_eval --model hf \
      --model_args pretrained=$model,dtype=bfloat16 \
      --tasks $TASKS \
      --num_fewshot 5 \
      --batch_size auto \
      --output_path results/$model_name.json

done < models.txt

Step 3: Generate comparison table

Python28 lines




models = [
    "meta-llama-Llama-2-7b-hf",
    "meta-llama-Llama-2-13b-hf",
    "mistralai-Mistral-7B-v0.1",
    "microsoft-phi-2"
]

tasks = ["mmlu", "gsm8k", "hellaswag", "truthfulqa"]

results = []
for model in models:
    with open(f"results/{model}.json") as f:
        data = json.load(f)
        row = {"Model": model.replace("-", "/")}
        for task in tasks:
            # Get primary metric for each task
            metrics = data["results"][task]
            if "acc" in metrics:
                row[task.upper()] = f"{metrics['acc']:.3f}"
            elif "exact_match" in metrics:
                row[task.upper()] = f"{metrics['exact_match']:.3f}"
        results.append(row)

df = pd.DataFrame(results)
print(df.to_markdown(index=False))

Output:

Text6 lines

| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |
|------------------------|-------|-------|-----------|------------|
| meta-llama/Llama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |
| meta-llama/Llama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |
| mistralai/Mistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |
| microsoft/phi-2        | 0.560 | 0.613 | 0.682     | 0.447      |

Workflow 4: Evaluate with vLLM (faster inference)

Use vLLM backend for 5-10x faster evaluation.

Text4 lines

vLLM Evaluation:
- [ ] Step 1: Install vLLM
- [ ] Step 2: Configure vLLM backend
- [ ] Step 3: Run evaluation

Step 1: Install vLLM

Shell1 line

pip install vllm

Step 2: Configure vLLM backend

Shell4 lines

lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \
  --tasks mmlu \
  --batch_size auto

Step 3: Run evaluation

vLLM is 5-10× faster than standard HuggingFace:

Shell11 lines

# Standard HF: ~2 hours for MMLU on 7B model
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu \
  --batch_size 8

# vLLM: ~15-20 minutes for MMLU on 7B model
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=2 \
  --tasks mmlu \
  --batch_size auto

When to use vs alternatives

Explains the idea itself. Read it slowly; the later sections build on it.

Use lm-evaluation-harness when:

Benchmarking models for academic papers
Comparing model quality across standard tasks
Tracking training progress
Reporting standardized metrics (everyone uses same prompts)
Need reproducible evaluation

Use alternatives instead:

HELM (Stanford): Broader evaluation (fairness, efficiency, calibration)
AlpacaEval: Instruction-following evaluation with LLM judges
MT-Bench: Conversational multi-turn evaluation
Custom scripts: Domain-specific evaluation

Common issues

Explains the idea itself. Read it slowly; the later sections build on it.

Issue: Evaluation too slow

Use vLLM backend:

Shell2 lines

lm_eval --model vllm \
  --model_args pretrained=model-name,tensor_parallel_size=2

Or reduce fewshot examples:

Shell1 line

--num_fewshot 0  # Instead of 5

Or evaluate subset of MMLU:

Shell1 line

--tasks mmlu_stem  # Only STEM subjects

Issue: Out of memory

Reduce batch size:

Shell1 line

--batch_size 1  # Or --batch_size auto

Use quantization:

Shell1 line

--model_args pretrained=model-name,load_in_8bit=True

Enable CPU offloading:

Shell1 line

--model_args pretrained=model-name,device_map=auto,offload_folder=offload

Issue: Different results than reported

Check fewshot count:

Shell1 line

--num_fewshot 5  # Most papers use 5-shot

Check exact task name:

Shell1 line

--tasks mmlu  # Not mmlu_direct or mmlu_fewshot

Verify model and tokenizer match:

Shell1 line

--model_args pretrained=model-name,tokenizer=same-model-name

Issue: HumanEval not executing code

Code-executing tasks (HumanEval, MBPP, etc.) are gated behind an explicit confirmation flag — you must pass --confirm_run_unsafe_code to run them:

Shell4 lines

lm_eval --model hf \
  --model_args pretrained=model-name \
  --tasks humaneval \
  --confirm_run_unsafe_code  # Required to run tasks that execute generated code

Without this flag lm-eval refuses to run the task rather than silently skipping code execution.

Advanced topics

Explains the idea itself. Read it slowly; the later sections build on it.

Benchmark descriptions: See references/benchmark-guide.md ↗ for detailed description of all 60+ tasks, what they measure, and interpretation.

Custom tasks: See references/custom-tasks.md ↗ for creating domain-specific evaluation tasks.

API evaluation: See references/api-evaluation.md ↗ for evaluating OpenAI, Anthropic, and other API models.

Multi-GPU strategies: See references/distributed-eval.md ↗ for data parallel and tensor parallel evaluation.

Hardware requirements

Explains the idea itself. Read it slowly; the later sections build on it.

GPU: NVIDIA (CUDA 11.8+), works on CPU (very slow)
VRAM:
7B model: 16GB (bf16) or 8GB (8-bit)
13B model: 28GB (bf16) or 14GB (8-bit)
70B model: Requires multi-GPU or quantization
Time (7B model, single A100):
HellaSwag: 10 minutes
GSM8K: 5 minutes
MMLU (full): 2 hours
HumanEval: 20 minutes

Resources

Explains the idea itself. Read it slowly; the later sections build on it.

GitHub: https://github.com/EleutherAI/lm-evaluation-harness
Docs: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs
Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.
Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (uses this harness)