Python Debugpy

Debug Python: pdb REPL + debugpy remote (DAP).

Three tools, picked by situation:

Start with breakpoint(). It's the cheapest thing that works.

• A test fails and the traceback doesn't reveal why a value is wrong
• You need to step through a function and watch a collection mutate
• A long-running process (hermes gateway, tui_gateway) misbehaves and you can't restart it
• Post-mortem: an exception fired in prod-ish code and you want to inspect locals at the crash site
• A subprocess / child (Python _SlashWorker, PTY bridge worker) is the actual bug site

Don't use for: things print() / logging.debug solve in under a minute, or things pytest -vv --tb=long --showlocals already reveals.

Inside any pdb prompt ((Pdb)):

The interact command is the most powerful — you can import anything, inspect complex objects, even call methods that mutate state. Locals are read-only by default; use !x = 42 from the (Pdb) prompt to mutate.

Easiest. Edit the file:

def compute(x, y):
    result = some_helper(x)
    breakpoint()           # <-- drops into pdb here
    return result + y

Run the code normally. You land at the breakpoint() line with full access to locals.

Don't forget to remove breakpoint() before committing. Use git diff or a pre-commit grep:

rg -n 'breakpoint\(\)' --type py

The hermes test runner and pytest both support this:

# Drop to pdb on failure (or on any raised exception):
scripts/run_tests.sh tests/path/to/test_file.py::test_name --pdb

# Drop to pdb at the START of the test:
scripts/run_tests.sh tests/path/to/test_file.py::test_name --trace

# Show locals in tracebacks without pdb:
scripts/run_tests.sh tests/path/to/test_file.py --showlocals --tb=long

Note: scripts/run_tests.sh runs each test file in a captured subprocess via run_tests_parallel.py (no xdist), so interactive pdb does NOT work under the wrapper. Run pytest directly for --pdb:

source .venv/bin/activate
python -m pytest tests/foo_test.py::test_bar --pdb

This bypasses the hermetic-env guarantees — fine for debugging, but re-run under the wrapper to confirm before pushing.

try:
    run_the_thing()
except Exception:
    pdb.post_mortem(sys.exc_info()[2])

Or wrap a whole script:

python -m pdb -c continue script.py
# When it crashes, pdb catches it and you're in the frame of the exception

Or set a global hook in a repl/jupyter:

def excepthook(etype, value, tb):
    import pdb; pdb.post_mortem(tb)
sys.excepthook = excepthook

For long-lived processes: Hermes gateway, tui_gateway, a daemon, a process that's already misbehaving and can't be restarted clean.

Setup

source <hermes-agent-repo>/.venv/bin/activate
pip install debugpy

Pattern A: Source-edit — process waits for debugger at launch

Add near the top of the entry point (or inside the function you want to debug):

debugpy.listen(("127.0.0.1", 5678))
print("debugpy listening on 5678, waiting for client...", flush=True)
debugpy.wait_for_client()
debugpy.breakpoint()       # optional: pause immediately once attached

Start the process; it blocks on wait_for_client().

Pattern B: No source edit — launch with -m debugpy

python -m debugpy --listen 127.0.0.1:5678 --wait-for-client your_script.py arg1

Equivalent for module entry:

python -m debugpy --listen 127.0.0.1:5678 --wait-for-client -m your.module

Pattern C: Attach to an already-running process

Needs the PID and debugpy preinstalled in the target's environment:

python -m debugpy --listen 127.0.0.1:5678 --pid <pid>
# debugpy injects itself into the process. Then attach a client as below.

Some kernels/security configs block the ptrace-based injection (/proc/sys/kernel/yama/ptrace_scope). Fix with:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Connecting a client from the terminal

The easiest terminal-side DAP client is VS Code CLI or a small script. From inside Hermes you have two practical options:

Option 1: debugpy's own CLI REPL — not an official feature, but a tiny DAP client script:

# /tmp/dap_client.py


HOST, PORT = "127.0.0.1", 5678
s = socket.create_connection((HOST, PORT))
seq = itertools.count(1)

def send(msg):
    msg["seq"] = next(seq)
    body = json.dumps(msg).encode()
    s.sendall(f"Content-Length: {len(body)}\r\n\r\n".encode() + body)

def recv():
    header = b""
    while b"\r\n\r\n" not in header:
        header += s.recv(1)
    length = int(header.decode().split("Content-Length:")[1].split("\r\n")[0].strip())
    body = b""
    while len(body) < length:
        body += s.recv(length - len(body))
    return json.loads(body)

send({"type": "request", "command": "initialize", "arguments": {"adapterID": "python"}})
print(recv())
send({"type": "request", "command": "attach", "arguments": {}})
print(recv())
send({"type": "request", "command": "setBreakpoints",
      "arguments": {"source": {"path": sys.argv[1]},
                    "breakpoints": [{"line": int(sys.argv[2])}]}})
print(recv())
send({"type": "request", "command": "configurationDone"})
# ... loop reading events and sending continue/stepIn/etc.

This is fine for one-off automation but painful as an interactive UX.

Option 2: Attach from VS Code / Cursor / Zed — if the user has one open, they can add a launch.json:

{
  "name": "Attach to Hermes",
  "type": "debugpy",
  "request": "attach",
  "connect": { "host": "127.0.0.1", "port": 5678 },
  "justMyCode": false,
  "pathMappings": [
    { "localRoot": "${workspaceFolder}", "remoteRoot": "<hermes-agent-repo>" }
  ]
}

Option 3: Ditch DAP, use remote-pdb — usually what you actually want from a terminal agent:

pip install remote-pdb

In your code:

from remote_pdb import set_trace
set_trace(host="127.0.0.1", port=4444)   # blocks until connection

Then from the terminal:

nc 127.0.0.1 4444
# You get a (Pdb) prompt exactly as if debugging locally.

remote-pdb is the cleanest agent-friendly choice when debugpy's DAP protocol is overkill. Use debugpy only when you actually need IDE integration.

Tests

See Recipe 3. The wrapper captures subprocess output, so run pytest directly for interactive pdb.

runagent.py / CLI — one-shot

Easiest: add breakpoint() near the suspect line, then run hermes normally. Control returns to your terminal at the pause point.

tuigateway subprocess (spawned by hermes --tui)

The gateway runs as a child of the Node TUI. Options:

A. Source-edit the gateway:

# tui_gateway/server.py near the top of serve()

debugpy.listen(("127.0.0.1", 5678))
debugpy.wait_for_client()

Start hermes --tui. The TUI will appear frozen (its backend is waiting). Attach a client; execution resumes when you continue.

B. Use remote-pdb at a specific handler:

from remote_pdb import set_trace
set_trace(host="127.0.0.1", port=4444)   # in the RPC handler you want to trap

Trigger the matching slash command from the TUI, then nc 127.0.0.1 4444 in another terminal.

SlashWorker subprocess

Same pattern — remote-pdb with set_trace() inside the worker's exec path. The worker is persistent across slash commands, so the first trigger blocks until you connect; subsequent slash commands pass through normally unless you re-arm.

Gateway (gateway/run.py)

Long-lived. Use remote-pdb at a handler, or debugpy with --wait-for-client if you're restarting the gateway anyway.

1. pdb under a parallel/output-capturing runner silently does nothing. You won't see the prompt, the test just hangs (true of pytest-xdist and of scripts/run_tests.sh's captured per-file subprocesses). Run pytest directly on a single file for interactive debugging.

1. breakpoint() in CI / non-TTY contexts hangs the process. Safe locally; never commit it. Add a pre-commit grep as a safety net.

1. PYTHONBREAKPOINT=0 disables all breakpoint() calls. Check the env if your breakpoint isn't hitting:

   echo $PYTHONBREAKPOINT

1. debugpy.listen blocks only if you also call wait_for_client(). Without it, execution continues and your first breakpoint may fire before the client is attached.

1. Attach to PID fails on hardened kernels. ptrace_scope=1 (Ubuntu default) allows only same-user ptrace of child processes. Workaround: echo 0 > /proc/sys/kernel/yama/ptrace_scope (needs root) or launch under debugpy from the start.

1. Threads. pdb only debugs the current thread. For multithreaded code, use debugpy (thread-aware DAP) or set threading.settrace() per thread.

1. asyncio. pdb works in coroutines but await inside pdb requires Python 3.13+ or await from interact mode on older versions. For 3.11/3.12, use asyncio.run_coroutine_threadsafe tricks or !stmt-based awaits via asyncio.ensure_future.

1. scripts/run_tests.sh strips credentials and sets HOME=<tmpdir>. If your bug depends on user config or real API keys, it won't reproduce under the wrapper. Debug with raw pytest first to repro, then re-confirm under the wrapper.

1. Forking / multiprocessing. pdb does not follow forks. Each child needs its own breakpoint() or set_trace(). For Hermes subagents, debug one process at a time.

• [ ] After pip install debugpy, confirm: python -c "import debugpy; print(debugpy.__version__)"
• [ ] For remote debug, confirm the port is actually listening: ss -tlnp | grep 5678
• [ ] First breakpoint actually hits (if it doesn't, you likely have PYTHONBREAKPOINT=0, you're under a parallel/capturing runner, or execution finished before attach)
• [ ] where / w shows the expected call stack
• [ ] Post-debug cleanup: no stray breakpoint() / set_trace() in committed code

  rg -n 'breakpoint\(\)|set_trace\(|debugpy\.listen' --type py

"Why is this dict missing a key?"

# add above the KeyError site
breakpoint()
# then in pdb:
(Pdb) pp d
(Pdb) pp list(d.keys())
(Pdb) w                # how did we get here

"This test passes in isolation but fails in the suite."

scripts/run_tests.sh tests/the_test.py   # confirm it fails under the isolated runner first
# For interactive debugging, or if it only fails WITH other tests:
source .venv/bin/activate
python -m pytest tests/ -x --pdb
# Now it pdb-traps at the exact failing test after state accumulated.

"My async handler deadlocks."

# Add at handler entry

Trigger the handler. nc 127.0.0.1 4444, then w to see the suspended frame, !import asyncio; asyncio.all_tasks() to see what else is pending.

"Post-mortem on a crash in an Ink child process / subprocess."

PYTHONFAULTHANDLER=1 python -m pdb -c continue path/to/entrypoint.py
# On crash, pdb lands at the frame of the exception with full locals