Debugging Linux applications — local and remote over gdbserver¶
gdbforge is a Vim-inspired GDB terminal UI for user-space programs — on an embedded Linux board or on the host. This guide covers deploy + gdbserver over SSH (:lua remotegdb) and where program stdin/stdout goes. For the kernel or a loadable module, where a breakpoint stops the whole system, see KERNEL_KGDB.md.
Prerequisites¶
Local debugging needs almost nothing; the remote path needs a little setup on both sides.
| Need | Where | Why |
|---|---|---|
| gdbforge installed | host | install — verify with gdbforge -version |
gdb |
host | the backend. For a remote board of a different architecture, a cross GDB (e.g. aarch64-linux-gnu-gdb), selected with gdbforge -d <gdb> |
Your program built with -g |
host | source-level debugging needs DWARF |
gdbserver on the target |
target | :lua remotegdb runs it over SSH; most Yocto/Buildroot images ship it |
| Key-based SSH to the target | both | every ssh/scp in the script uses -o BatchMode=yes, so password prompts fail. Run ssh-copy-id root@<board> first |
| Matching unstripped binary on the host | host | the host GDB reads symbols locally while the target runs the stripped copy |
GDBFORGE_TERMINAL (optional) |
host | only for the external-terminal I/O mode below |
Quick sanity check before you start:
ssh -o BatchMode=yes root@<board> 'gdbserver --version'
If that prints a version without asking for a password, the remote path will work.
To your first breakpoint (local program)¶
gdbforge ./my_app
- The Code pane opens on your source. Move the cursor to a line and press Space to set a breakpoint.
- Type
:b gdbandrun(orstart, which breaks atmainimmediately). - On the stop,
━━▶marks the program counter and the Call Stack, Threads and Breakpoints panes refresh together. There is no variables pane yet — useprint/info localsin:b gdb. - c continues, n steps over, s steps into, f
finishes the frame. Program output lands in
:b iounless you redirect it below.
For the remote equivalent, jump to Remote board — :lua remotegdb.
Demo — Linux app I/O¶
Linux app — external terminal print vs the internal IO pane (:b io). Watch on YouTube.

Program I/O — internal vs external¶
Every debug session has an inferior (your app). Its stdin/stdout can go to three places:
1. Internal IO pane (default)¶
- Pane:
:b io(alias:b output) inside gdbforge - Best for: line-oriented output, simple
printf, stdin while stopped or running - Setup: none — works out of the box
- Limit: not a full VT100; heavy TUI/curses or flood printing may feel sluggish (Ctrl-C still works)
2. External terminal (host)¶
- Window: real emulator (
mate-terminal,kitty,xterm, … viaGDBFORGE_TERMINAL) - Best for: full-screen TUI, curses, high-rate stdout
- Commands:
:lua external_tty— open tty, thenfile/runyourself:lua terminal_debug [prog] [run]— tty + optional load/break/run:set inferior-tty— same idea from the GDB console (no Lua):set inferior-tty internal— switch back to:b io- When external,
:b ioshows a note only — type in the other window
export GDBFORGE_TERMINAL=mate-terminal
./bin/gdbforge ./my_tui
:lua terminal_debug ./my_tui run
3. gdbserver (inferior in gdbserver's tty)¶
The debugged program runs under gdbserver; its stdio is gdbserver's terminal (local or on the board).
| Script | Target | I/O location |
|---|---|---|
:lua gdbserver_tui [prog] [port] |
Same PC | External terminal running local gdbserver |
:lua remotegdb [app] [host] [port] |
Board | SSH opens terminal; gdbserver on board runs the app |
:lua remotedlv [app] [host] [port] |
Board (Go) | headless dlv on board via SSH; see below |
gdbforge attaches with target remote; breakpoints and stepping stay in gdbforge panes.
flowchart TB
subgraph internal [Internal IO default]
App1[your app] --> IO[":b io pane"]
end
subgraph external [External terminal]
App2[your app] --> Term[mate-terminal / kitty]
Lua1[":lua external_tty"]
Lua2[":lua terminal_debug"]
Set[":set inferior-tty"]
end
subgraph gdbserver [gdbserver]
App3[your app on board or host] --> GS[gdbserver process]
GS --> Term2[external terminal or SSH window]
GF[gdbforge GDB] -->|target remote| GS
end
Remote board — :lua remotegdb¶
Deploy binary, start gdbserver over SSH, attach with target remote.
mkdir -p .gdbforge/lua
cp -r lua/embedded/remotegdb .gdbforge/lua/
export GDBFORGE_REMOTE_HOST=192.168.20.50
export GDBFORGE_TERMINAL=mate-terminal
./bin/gdbforge ./hello
:lua remotegdb
:lua remotegdb ./hello 192.168.20.50 1234
Workflow¶
- Local app path — arg,
GDBFORGE_REMOTE_APP, or session program - MD5 compare; scp only if remote copy missing or stale
- SSH runs
gdbserver :PORT /tmp/app [args…]in an external terminal wait_port→file+target remote host:PORT
Environment variables¶
| Variable | Default | Meaning |
|---|---|---|
GDBFORGE_REMOTE_APP |
(empty) | Local binary |
GDBFORGE_REMOTE_APP_ARGS |
(empty) | Inferior argv |
GDBFORGE_REMOTE_HOST |
192.168.20.50 |
Board IP |
GDBFORGE_REMOTE_USER |
root |
SSH user |
GDBFORGE_REMOTE_PORT |
1234 |
gdbserver port |
GDBFORGE_REMOTE_DIR |
/tmp |
Remote install dir |
GDBFORGE_TERMINAL |
auto | Terminal for ssh/gdbserver window |
Remote board, Go — :lua remotedlv¶
Same flow as remotegdb, but for a Go program built for the board: deploy, start headless Delve over SSH, connect with dlv_connect. Needs dlv on the target and a session started with the Delve backend.
How it differs from C/C++ with gdbserver¶
The idea is the same: the program runs on the board and you drive it from the PC. Who reads the debug info is different:
C/C++ (gdbserver + host GDB) |
Go (headless dlv + dlv connect) |
|
|---|---|---|
| Board | gdbserver runs the program; it knows nothing about symbols |
dlv exec --headless runs the program and reads its DWARF (symbols, lines, variables) |
| PC | GDB loads the same ELF for symbols and source lines | gdbforge is only a client: it needs the source files, not the ELF |
| Debug info must be in | the host's copy of the ELF (the board copy can be stripped) | the board's copy (Delve reads it there) |
So for Go:
- The binary deployed to the board must keep its debug info. Do not strip it (
-ldflags="-s -w"), and build with-gcflags='all=-N -l'so stepping and variables work. - The PC needs the source tree at the paths recorded at build time. Building on the PC and running gdbforge from the project directory takes care of that. If you built somewhere else, map the paths in the Delve console with
config substitute-path <build-dir> <local-dir>. - The arm64 ELF never runs on the PC. Start
gdbforge -g dlvwith no program and let:lua remotedlvdeploy and connect.
Steps¶
cp -r lua/embedded/remotedlv .gdbforge/lua/
export GDBFORGE_REMOTE_HOST=192.168.20.50
export GDBFORGE_REMOTE_DLV=/usr/bin/dlv # where dlv lives on the board
./bin/gdbforge -g dlv # no program: nothing runs on the PC
:lua remotedlv ./hello-arm64
:lua remotedlv ./hello-arm64 192.168.20.50 2345
gdbforge -g dlv with no program starts an idle Delve session; :lua remotedlv fills it by connecting to the board. Passing the arm64 binary at startup instead (gdbforge -g dlv ./hello-arm64) runs it with a local dlv exec and fails on an x86 PC with exec format error.
It uses the same GDBFORGE_REMOTE_APP, _APP_ARGS, _HOST, _USER and _DIR variables as remotegdb, plus:
| Variable | Default | Meaning |
|---|---|---|
GDBFORGE_REMOTE_DLV |
dlv |
Path of dlv on the target |
GDBFORGE_REMOTE_DLV_PORT |
2345 |
Delve listen port |
The Delve log is /tmp/dlv.log on the board.
Connecting to a dlv that is already running — -connect¶
If dlv is already listening on the board (started by hand or by an init script), skip the local program entirely:
ssh root@192.168.20.50 'dlv exec --headless --listen=:2345 --api-version=2 --accept-multiclient /tmp/hello'
gdbforge -g dlv -connect 192.168.20.50:2345
No local program is started. The remote side owns the program's stdio, so :set inferior-tty is not available in this mode.
Getting dlv onto an arm64 board¶
Cross-compile Delve on the host as a static binary and copy it over:
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \
go install github.com/go-delve/delve/cmd/dlv@latest
# result: $(go env GOPATH)/bin/linux_arm64/dlv
scp $(go env GOPATH)/bin/linux_arm64/dlv root@192.168.20.50:/usr/local/bin/dlv
ssh root@192.168.20.50 'chmod +x /usr/local/bin/dlv && dlv version'
export GDBFORGE_REMOTE_DLV=/usr/local/bin/dlv
Use a writable location such as /tmp/dlv if the root filesystem is read-only. Build your own program for the board with optimizations off so stepping and variables work:
GOOS=linux GOARCH=arm64 go build -gcflags='all=-N -l' -o hello .
For 32-bit ARM use GOARCH=arm (plus GOARM=7). Other architectures follow the same pattern.
Local host — gdbserver or direct GDB¶
:lua |
Purpose |
|---|---|
gdbserver_tui |
gdbserver in external terminal + target remote localhost:PORT |
terminal_debug |
Direct GDB + external tty (+ optional file/break/run) |
external_tty |
External tty only |
remotegdb_log |
Tail /tmp/gdbserver.log on board via SSH |
Script install paths¶
All under lua/embedded/:
cp -r lua/embedded/remotegdb .gdbforge/lua/
cp -r lua/embedded/remotedlv .gdbforge/lua/
cp -r lua/embedded/terminal_debug .gdbforge/lua/
cp -r lua/embedded/external_tty .gdbforge/lua/
cp -r lua/embedded/gdbserver_tui .gdbforge/lua/
Kernel debug (not user-space): KERNEL_KGDB.md. PTY details: PTY_ARCHITECTURE.md.
See also: Lua catalog — embedded · User Guide — Lua