Testing How-to

Step-by-step tutorial for automated spec-based testing of Astrobe Oberon programs on hardware

This tutorial walks through running automated tests against an Astrobe Oberon program on hardware using the gdbtestengine. It uses the SignalSync example project on an RP2350 (Pico2) board. The same steps apply to other supported targets (STM32, RP2040) with the appropriate GDB server.

The gdbtestengine runs inside GDB and executes declarative test specifications. Each .spec file describes what to check or trace at specific procedure entry points. The engine sets breakpoints, collects values from registers and memory, and produces a timestamped report.

Prerequisites

Before starting, ensure you have:

completed the ELF build workflow (compile, gen-rdb, make-elf with --debug) – see Debugging steps 1–3
arm-none-eabi-gdb-py3 installed (the -py3 variant is required; the plain arm-none-eabi-gdb lacks Python support)
a GDB server running and connected to the target
the run-test alias configured in ~/.gdbinit (see next section)

GDB Set-up

The gdbtestengine is loaded via a GDB alias defined in ~/.gdbinit. Add the following lines:

define run-test
  source <path-to>/tools/pylib/gdbtestengine.py
end

set environment RTK_GDB_TESTS_SPECS_DIR rdb-gdb-tests/specs

Replace <path-to> with the actual path to the repository root. The RTK_GDB_TESTS_SPECS_DIR variable tells the engine where to find spec files. Reports are written to a reports/ directory next to the specs directory (ie. rdb-gdb-tests/reports/).

For RP2350 targets, also add the restart command for resetting the target without reloading the ELF (see Debbugging, Restarting the Target).

Step 1: Create the Test Directories

Create the spec and report directories in your project:

mkdir -p rdb-gdb-tests/specs
mkdir -p rdb-gdb-tests/reports

Step 2: Write a Spec File

Create a test specification in rdb-gdb-tests/specs/. Here is a simple spec that checks argument values during the program's initialisation phase:

-- SignalSync init-phase test spec
-- Tests known argument values at procedure entry points.

option timeout = 15
option max-hits = 20

-- Stop after seeing all init calls
stop after 10 hits

-- LED.Set is called with {LED.Pico} = 128 = 0x80
proc LED.Set
  at entry
    check arg leds = 128

-- Kernel.SetPeriod is called twice during init:
--   SetPeriod(50, 0)    -- T0period=50, startAfter=0
--   SetPeriod(100, 100) -- T3period=100, startAfter=100
-- We trace both args to see the values.
proc Kernel.SetPeriod
  at entry
    trace arg period
    trace arg startAfter

-- Cores.CoreId is a leaf proc with no args.
proc Cores.CoreId
  at entry
    trace local cid

Save this as rdb-gdb-tests/specs/init-test.spec.

Key Elements

option timeout = 15 – abort if the test runs longer than 15 seconds
option max-hits = 20 – abort after 20 total breakpoint hits
stop after 10 hits – end the test after 10 breakpoint hits
proc Module.Procedure – select a procedure to instrument
at entry – set the breakpoint at the procedure entry point
check arg name = value – verify an argument value (PASS or FAIL)
trace arg name – record an argument value (always PASS)
trace local name – record a local variable value

Comments start with --.

Step 3: Run a Test Interactively

Start GDB, connect to the target, load the ELF, then run the test:

arm-none-eabi-gdb-py3 SignalSync.elf
(gdb) target extended-remote localhost:3333
(gdb) load
(gdb) set environment SPEC init-test.spec
(gdb) run-test

The engine reads init-test.spec from the specs directory, sets breakpoints on the named procedures, and lets the program run. Each time a breakpoint is hit, the engine checks or traces the specified values and prints progress to the console.

When the test ends (by reaching the stop condition or timing out), the engine writes a report and returns to the GDB prompt.

To run another test in the same session, use restart (on RP2350) to reset the target, set the new spec, and run again:

(gdb) restart
(gdb) set environment SPEC global-test.spec
(gdb) run-test

Step 4: Read the Report

Reports are written to rdb-gdb-tests/reports/ with a timestamped file name, for example init-test-report-20260329-142025.txt:

================================================================
Code Test Report
================================================================
  Spec:  init-test.spec
  ELF:   SignalSync.elf
  Date:  2026-03-29 14:20:25
  Hits:  6
  Stop:  timeout: stopped at Kernel.loopc 0C004EBAH, module offset 1274
  PASS:  1
  FAIL:  0
  TRACE: 7
  ERROR: 0

TIMELINE
----------------------------------------
    1 [LED.Set at entry] PASS  arg leds = 128
    2 [Kernel.SetPeriod at entry] TRACE arg period = 50
    2 [Kernel.SetPeriod at entry] TRACE arg startAfter = 0
    3 [Cores.CoreId at entry] TRACE local cid = 1
    4 [Cores.CoreId at entry] TRACE local cid = 2
    5 [Kernel.SetPeriod at entry] TRACE arg period = 100
    5 [Kernel.SetPeriod at entry] TRACE arg startAfter = 100
    6 [Cores.CoreId at entry] TRACE local cid = 805570016

TRACES
----------------------------------------
    2 [Kernel.SetPeriod at entry] TRACE arg period = 50
    2 [Kernel.SetPeriod at entry] TRACE arg startAfter = 0
    3 [Cores.CoreId at entry] TRACE local cid = 1
    4 [Cores.CoreId at entry] TRACE local cid = 2
    5 [Kernel.SetPeriod at entry] TRACE arg period = 100
    5 [Kernel.SetPeriod at entry] TRACE arg startAfter = 100
    6 [Cores.CoreId at entry] TRACE local cid = 805570016

PASSED
----------------------------------------
    1 [LED.Set at entry] PASS  arg leds = 128

The report has four sections:

Header – spec name, ELF, timestamp, hit count, stop reason, and summary counts (PASS, FAIL, TRACE, ERROR)
TIMELINE – all events in execution order
TRACES – only trace results (useful for extracting runtime values)
PASSED – only check results that passed

If any checks fail, a FAILED section appears listing the failures with expected and actual values.

Step 5: Run in Batch Mode

For unattended or scripted execution, use GDB's -batch flag:

arm-none-eabi-gdb-py3 -batch \
    -ex "target extended-remote localhost:3333" \
    -ex "load" \
    -ex "set environment SPEC init-test.spec" \
    -ex "run-test" \
    SignalSync.elf

GDB runs the test, writes the report, and exits. The exit code is 0 regardless of test outcome; check the report file for PASS/FAIL results.

Writing More Specs

The spec language supports several features beyond simple argument checks:

Comparison operators:

check arg dev # NIL                        -- not equal
check mem [arg dev + 18H] > 050000000H     -- greater than
check mem [arg dev + 00H] <= 0 as unsigned -- signedness override

Memory reads with field and bit-range access:

check mem MCU.FLASH_ACR [3:0] = 4     -- read register, extract bits 3:0
trace mem [arg dev + 18H]             -- read memory at pointer + offset

Cross-module constants:

check arg pllId = CLK.PLL1            -- constant from another module

Sentinel conditions to control test windows:

start proc Kernel.SetPeriod at entry   -- open the test window here
stop after all checks                  -- stop when all checks evaluated

Halt on failure (interactive debugging of test failures):

check arg leds = 128 halt              -- halt GDB if this check fails

The spec grammar covers the spec language in detail.

Constant Resolution

Constants referenced in specs (Module.Constant) are resolved from the .alst files at start-up. Constants defined as literals or single-module expressions resolve correctly. Constants whose defining expression references another module's constant (eg. DEV0.X = BASE + offset where BASE = MCU.Y) may not resolve – use the constant from the module that owns the base definition instead.

Troubleshooting

"ModuleNotFoundError" or GDB Python error

Ensure you are using arm-none-eabi-gdb-py3 (with -py3), not plain arm-none-eabi-gdb. The test engine requires GDB's embedded Python.

"SPEC not set" error

Set the spec file name before running:

(gdb) set environment SPEC init-test.spec

Timeout with no hits

The program may not be reaching the instrumented procedures. Check that the ELF was loaded (load command) and that the procedure names in the spec match the module and procedure names exactly (case-sensitive).

Unexpected variable values

Ensure the ELF matches the binary on the target. If the program was recompiled, repeat the full workflow (gen-rdb, make-elf, load).