Credential Pools

Credential pools let you register multiple API keys or OAuth tokens for the same provider. When one key hits a rate limit or billing quota, Hermes automatically rotates to the next healthy key — keeping your session alive without switching providers.

This is different from fallback providers, which switch to a different provider entirely. Credential pools are same-provider rotation; fallback providers are cross-provider failover. Pools are tried first — if all pool keys are exhausted, then the fallback provider activates.

If you already have an API key set in .env, Hermes auto-discovers it as a 1-key pool. To benefit from pooling, add more keys:

# Add a second OpenRouter key
hermes auth add openrouter --api-key sk-or-v1-your-second-key

# Add a second Anthropic key
hermes auth add anthropic --type api-key --api-key sk-ant-api03-your-second-key

# Add an Anthropic OAuth credential (requires Claude Max plan + extra usage credits)
hermes auth add anthropic --type oauth
# Opens browser for OAuth login

Check your pools:

hermes auth list

Output:

openrouter (2 credentials):
  #1  OPENROUTER_API_KEY   api_key env:OPENROUTER_API_KEY ←
  #2  backup-key           api_key manual

anthropic (3 credentials):
  #1  hermes_pkce          oauth   hermes_pkce ←
  #2  claude_code          oauth   claude_code
  #3  ANTHROPIC_API_KEY    api_key env:ANTHROPIC_API_KEY

The ← marks the currently selected credential.

Run hermes auth with no subcommand for an interactive wizard:

hermes auth

This shows your full pool status and offers a menu:

What would you like to do?
  1. Add a credential
  2. Remove a credential
  3. Reset cooldowns for a provider
  4. Set rotation strategy for a provider
  5. Exit

For providers that support both API keys and OAuth (Anthropic, Nous, Codex), the add flow asks which type:

anthropic supports both API keys and OAuth login.
  1. API key (paste a key from the provider dashboard)
  2. OAuth login (authenticate via browser)
Type [1/2]:

Configure via hermes auth → "Set rotation strategy" or in config.yaml:

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used

The pool handles different errors differently:

Provider-supplied reset_at timestamps override these default cooldowns.

The has_retried_429 flag resets on every successful API call, so a single transient 429 doesn't trigger rotation.

Custom OpenAI-compatible endpoints (Together.ai, RunPod, local servers) get their own pools, keyed by the endpoint name from the providers: dict in config.yaml (or the legacy custom_providers list, which is auto-migrated).

When you set up a custom endpoint via hermes model, it auto-generates a name like "Together.ai" or "Local (localhost:8080)". This name becomes the pool key.

# After setting up a custom endpoint via hermes model:
hermes auth list
# Shows:
#   Together.ai (1 credential):
#     #1  config key    api_key config:Together.ai ←

# Add a second key for the same endpoint:
hermes auth add Together.ai --api-key sk-together-second-key

Custom endpoint pools are stored in auth.json under credential_pool with a custom: prefix:

{
  "credential_pool": {
    "openrouter": [...],
    "custom:together.ai": [...]
  }
}

Hermes automatically discovers credentials from multiple sources and seeds the pool on startup:

Auto-seeded entries are updated on each pool load — if you remove an env var, its pool entry is automatically pruned. Manual entries (added via hermes auth add) are never auto-pruned.

Borrowed runtime secrets (for example env vars, Bitwarden/Vault/keyring/systemd references, and custom config values) are reference-only at the auth.json boundary. Hermes can use the resolved value in memory for the current run, but it persists only metadata such as the source ref, label, status, request counters, and a non-reversible fingerprint. Manual entries and Hermes-owned OAuth/device-code state keep the durable tokens they need to refresh.

When the agent spawns subagents via delegate_task, the parent's credential pool is automatically shared with children:

• Same provider — the child receives the parent's full pool, enabling key rotation on rate limits
• Different provider — the child loads that provider's own pool (if configured)
• No pool configured — the child falls back to the inherited single API key

This means subagents benefit from the same rate-limit resilience as the parent, with no extra configuration needed. Per-task credential leasing ensures children don't conflict with each other when rotating keys concurrently.

The credential pool uses a threading lock for all state mutations (select(), mark_exhausted_and_rotate(), try_refresh_current(), mark_used()). This ensures safe concurrent access when the gateway handles multiple chat sessions simultaneously.

For the full data flow diagram, see docs/credential-pool-flow.excalidraw ↗ in the repository.

The credential pool integrates at the provider resolution layer:

1. agent/credential_pool.py — Pool manager: storage, selection, rotation, cooldowns
2. hermes_cli/auth_commands.py — CLI commands and interactive wizard
3. hermes_cli/runtime_provider.py — Pool-aware credential resolution
4. run_agent.py — Error recovery: 429/402/401 → pool rotation → fallback

Pool state is stored in ~/.hermes/auth.json under the credential_pool key:

{
  "version": 1,
  "credential_pool": {
    "openrouter": [
      {
        "id": "abc123",
        "label": "OPENROUTER_API_KEY",
        "auth_type": "api_key",
        "priority": 0,
        "source": "env:OPENROUTER_API_KEY",
        "secret_source": "bitwarden",
        "secret_fingerprint": "sha256:12ab34cd56ef7890",
        "last_status": "ok",
        "request_count": 142
      }
    ],
    "anthropic": [
      {
        "id": "manual1",
        "label": "personal-api-key",
        "auth_type": "api_key",
        "priority": 0,
        "source": "manual",
        "access_token": "sk-ant-api03-..."
      }
    ]
  }
}

The OpenRouter entry above was borrowed from an external source, so the raw key is not stored in auth.json. The manual Anthropic entry was intentionally added to Hermes' credential store, so its token remains persistable.

Strategies are stored in config.yaml (not auth.json):

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used