Initial framework: command language, flow layer, host build

A C framework for commanding a Pico WH robot over USB, WiFi or BLE with a
small text DSL. No hardware attached yet, so the whole stack runs on the
host against a simulated HAL and is covered by tests.

Two specifications drive the code:

  docs/grammar.md  the language — one namespace of callables (native,
                   firmware-resident, and defined over the wire), procedures
                   with named arguments and defaults, REPEAT, and the rules
                   that keep it safe on a microcontroller.
  docs/flow.md     transport, bracket-balance framing, the envelope,
                   admission, execution, abort and failsafe.

Three properties the design leans on:

  - Bodies may only reference callables that already exist, so the call
    graph is acyclic by construction, recursion is unrepresentable, and
    interpreter depth is checked at definition time rather than discovered
    at runtime.
  - Nothing blocks. The interpreter is a resumable state machine over an
    explicit frame stack, so ABORT is honoured within one tick even in the
    middle of a long move, and no program can overflow the MCU stack.
  - No queue. One outstanding statement at a time, which keeps abort to a
    single unambiguous victim and avoids inventing an answer to "the
    running program failed, does the queued one still go?".

Named arguments are resolved to the callee's parameter order at definition
time, so they cost nothing at execution.

rubo_core is target-independent. Porting to the Pico means providing a
non-blocking rubo_transport_t and a rubo_hal_t, and nothing else.

Not built yet: the Pico transport and HAL, the camera DAT/DEND bulk path,
and IF/ELIF, which is reserved in the grammar and lands with the sensors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hfu1EFVbpee92zvzQunsKb

docs/flow.md

@@ -0,0 +1,429 @@
+# Rubo Command Flow — Transport, Framing and Execution
+
+Status: **draft, v1**. Companion to `docs/grammar.md`, which specifies the
+language. This document specifies everything around it: how bytes become
+statements, how statements become motion, and how the robot is stopped.
+
+## 1. Principles
+
+1. **Nothing blocks.** The interpreter is a resumable state machine driven from
+   the main loop. A `MOVE` lasting ten seconds must not prevent `ABORT` from
+   being received in second three. This constraint shapes the entire design.
+2. **The envelope is not the language.** Correlation tags and control words are
+   handled by the session layer and never reach the parser. `docs/grammar.md`
+   stays a specification of a language, not of a wire protocol.
+3. **One framer for every transport.** USB, WiFi, BLE and the host test harness
+   all reduce to a byte stream. Only MTU and connection semantics differ.
+4. **A dropped link stops the robot.** Every path that engages a motor is
+   covered by a deadman timer. This is the one requirement that is not
+   negotiable for convenience.
+
+## 2. Layer stack
+
+```
+     ┌──────────────────────────────────────────────┐
+     │ transport   usb_cdc │ wifi_tcp │ ble_nus │ host_stdio
+     │             byte stream in/out, MTU-aware
+     ├──────────────────────────────────────────────┤
+     │ framer      bracket-balance statement extraction (§4)
+     ├──────────────────────────────────────────────┤
+     │ session     tags, control words, ACK/DONE/ERR, backpressure (§5–§7)
+     ├──────────────────────────────────────────────┤
+     │ parser      → validator → arena          (docs/grammar.md)
+     ├──────────────────────────────────────────────┤
+     │ scheduler   resumable interpreter, explicit stack, ticks (§8)
+     ├──────────────────────────────────────────────┤
+     │ HAL         motor │ camera │ battery      sim | pico
+     └──────────────────────────────────────────────┘
+```
+
+Layers below the parser are shared; only `transport` and `HAL` have per-target
+implementations. The host build swaps exactly those two and runs everything
+else unmodified, which is what makes the language testable with no hardware.
+
+## 3. Transport interface
+
+```c
+typedef struct {
+    int    (*open)(void *cfg);
+    int    (*read)(uint8_t *buf, size_t len);        /* non-blocking; 0 = no data */
+    int    (*write)(const uint8_t *buf, size_t len);
+    bool   (*connected)(void);
+    void   (*close)(void);
+    size_t mtu;                                      /* max bytes per write */
+} rubo_transport_t;
+```
+
+`read` must never block. Everything above this line is polled from the main
+loop; a blocking read anywhere forfeits §1.1.
+
+| Transport   | Implementation | MTU | Notes |
+|-------------|----------------|-----|-------|
+| `usb_cdc`   | TinyUSB CDC (pico-sdk) | 64 | the development default; enumerate as ACM |
+| `wifi_tcp`  | lwIP listener, one client | ~1460 | CYW43; port configurable, default 3141 |
+| `ble_nus`   | Nordic UART Service | 20–244 | notify for TX, write-without-response for RX; chunking mandatory |
+| `host_stdio`| stdin/stdout | — | host build; drives the whole stack under a test harness |
+| `host_tcp`  | POSIX socket | — | host build; lets a real client speak to a simulated robot |
+
+BLE's small MTU is not a special case: the framer is byte-oriented, so a
+statement split across five notifications reassembles exactly as one arriving
+in a single USB read.
+
+## 4. Framing
+
+`docs/grammar.md` §3.3 established that statements end at **bracket balance**,
+not at newlines. The framer is therefore a small state machine, not a line
+reader.
+
+### 4.1 Rule
+
+> A top-level statement is complete at the first newline encountered while
+> bracket depth is zero and the buffer is non-empty.
+
+Inside brackets, newlines are ordinary whitespace. A multi-line `DEF` pasted
+into a terminal arrives as one statement; a one-liner arrives as one statement.
+Interactive typing and bulk paste need no mode switch.
+
+### 4.2 State machine
+
+| State     | Input            | Action |
+|-----------|------------------|--------|
+| `STMT`    | `#`              | → `COMMENT` |
+| `STMT`    | `[`              | `depth++`; append |
+| `STMT`    | `]`              | `depth--`; append; if `depth < 0` → error `E_SYNTAX`, → `DISCARD` |
+| `STMT`    | newline          | if `depth == 0 && len > 0` → **emit**; else treat as space |
+| `STMT`    | other            | append |
+| `COMMENT` | newline          | → `STMT` (apply the newline rule above) |
+| `COMMENT` | other            | discard |
+| `DISCARD` | newline          | reset `depth`/`len`, → `STMT` |
+| `DISCARD` | other            | discard |
+
+The framer must be **comment-aware**: a `[` inside a comment must not affect
+depth, or a single `#` in a comment would desynchronise the stream permanently.
+There are no string literals in the language, so no quote handling is needed.
+
+### 4.3 Failure and resynchronisation
+
+| Condition | Response | Recovery |
+|-----------|----------|----------|
+| `len > RUBO_MAX_STMT` | `ERR 10 statement too long` | → `DISCARD` |
+| unbalanced `]` | `ERR 1 unexpected ]` | → `DISCARD` |
+| partial statement idle > `RUBO_FRAME_TIMEOUT_MS` | `ERR 1 incomplete statement` | reset |
+| transport disconnect mid-statement | — | reset buffer; §10 deadman applies |
+
+The idle timeout matters most on BLE, where a client can vanish mid-frame
+leaving an unterminated `[` that would otherwise wedge the framer forever.
+
+### 4.4 Top-level `;`
+
+A bare `;` at depth zero is `E_SYNTAX`. One frame carries one statement; to
+submit several as a unit, wrap them: `[ A; B; C ]`. This keeps "one statement,
+one tag, one ACK, one DONE" exact.
+
+## 5. Envelope
+
+Two things wrap a statement and are stripped by the session layer before
+parsing. Neither appears in `docs/grammar.md`, because neither is part of the
+language.
+
+### 5.1 Correlation tags
+
+An optional `@<uint16>` prefix. Responses echo it.
+
+```
+> @7 SQUARE side=1000
+< @7 ACK
+< @7 DONE
+```
+
+Untagged statements are legal and produce untagged responses — convenient when
+a human is typing. Tags are the client's to allocate; the robot never
+interprets them beyond echoing.
+
+### 5.2 Control words
+
+Recognised before parsing, executed immediately, never queued, never subject to
+`E_BUSY`. They bypass the interpreter entirely.
+
+| Word    | Effect |
+|---------|--------|
+| `ABORT` | unwind the interpreter stack, `hal_stop_all()`, end the running program (§10) |
+| `PING`  | `PONG` — liveness, and it feeds the deadman timer |
+| `STAT`  | one-line status: running callable, tag, depth, arena free, link age |
+
+Control words are deliberately few. `ABORT` in particular cannot be a native
+callable, because a callable can only run when the interpreter is free to run
+it — which is precisely when you do not need it.
+
+## 6. Response vocabulary
+
+```
+@7 ACK                              statement admitted, parsed, validated; now executing
+@7 DONE                             execution finished normally
+@7 ERR <code> <message>             rejected or failed; codes in grammar.md §10
+EVT <name> [args...]                asynchronous, untagged, broadcast
+DAT <id> <seq> <total> <payload>    bulk data chunk (§11)
+DEND <id>                           bulk data complete
+PONG                                reply to PING
+STAT <k>=<v> ...                    reply to STAT
+```
+
+**ACK and DONE are always both sent**, including for instantaneous statements
+like `DEF` and `DEL`. Collapsing them for non-executing statements would save a
+few bytes and cost every client a special case; a terminal client is free to
+hide `ACK` when `DONE` follows immediately.
+
+`ERR` may arrive instead of `ACK` (rejected at parse or validation) or after
+`ACK` (failed during execution). A client waits for exactly one of `DONE` or
+`ERR` per tag.
+
+## 7. Statement lifecycle
+
+```
+  bytes ─► RECEIVING ─► ADMISSION ─► PARSING ─► VALIDATING ─► RUNNING ─► DONE
+             │             │            │            │           │
+             │             │            │            │           ├─► ERR (runtime: E_RANGE, E_FAULT)
+             │             │            │            │           └─► ABORTED (E_ABORT)
+             │             │            │            └─► ERR (E_UNKNOWN, E_ARITY, E_ARGNAME,
+             │             │            │                     E_UNDEFINED, E_CYCLE, E_NOMEM…)
+             │             │            └─► ERR (E_SYNTAX, E_LIMIT)
+             │             └─► ERR 12 busy
+             └─► discarded (§4.3)
+```
+
+**Admission** comes before parsing: if a program is already running, the
+statement is rejected immediately without being parsed at all (§9). There is
+no point spending cycles validating something that cannot run.
+
+Validation is the last point at which rejection is free. Past `ACK`, a failure
+means the robot has already moved, so the majority of `docs/grammar.md` §10 is
+deliberately checkable before execution begins.
+
+## 8. Execution model
+
+### 8.1 Explicit stack, no C recursion
+
+The interpreter holds a statically allocated array of frames:
+
+```c
+typedef struct {
+    const rubo_node_t *node;      /* current statement in the body        */
+    uint16_t           pc;        /* index within the enclosing sequence  */
+    uint16_t           iter;      /* REPEAT counter                       */
+    rubo_value_t       args[RUBO_MAX_PARAMS];
+    uint32_t           deadline;  /* native scratch: ms, 0 = none         */
+} rubo_frame_t;
+
+static rubo_frame_t stack[RUBO_MAX_DEPTH];
+```
+
+`docs/grammar.md` §5.1 guarantees the call graph is acyclic, so `RUBO_MAX_DEPTH`
+frames are provably sufficient and depth is checked at definition time. Nothing
+here recurses in C, so the MCU stack is not at risk regardless of what the user
+sends.
+
+The explicit stack buys three things beyond safety: `ABORT` is a matter of
+resetting an index, `STAT` can report exactly what is executing, and the whole
+interpreter is trivially unit-testable on the host.
+
+### 8.2 Natives are resumable
+
+```c
+typedef enum { RUBO_STEP_DONE, RUBO_STEP_PENDING, RUBO_STEP_ERR } rubo_step_t;
+
+typedef rubo_step_t (*rubo_native_fn)(rubo_frame_t *f, uint32_t now_ms);
+```
+
+A native with a duration sets `f->deadline` on first entry and returns
+`RUBO_STEP_PENDING` until `now_ms >= f->deadline`. `MOVE dir=FWD speed=50
+dur=1000` engages the motors, returns immediately, and is re-entered each tick
+until its deadline passes — during which the transport is polled normally and
+`ABORT` is honoured within one tick.
+
+### 8.3 Main loop
+
+```c
+for (;;) {
+    uint32_t now = rubo_millis();
+    rubo_transport_poll(now);   /* read bytes, feed framers               */
+    rubo_session_poll(now);     /* frame → envelope → admit → parse → run */
+    rubo_exec_tick(now);        /* advance the interpreter, bounded work  */
+    rubo_hal_poll(now);         /* sensors, deadman, event generation     */
+}
+```
+
+Cooperative and single-core; no RTOS, and the second RP2040 core stays free for
+the camera pipeline later. `rubo_exec_tick` runs at most `RUBO_STEPS_PER_TICK`
+interpreter steps before yielding, so a deeply nested `REPEAT` of instantaneous
+statements cannot starve the transport.
+
+## 9. Concurrency and backpressure
+
+**One interpreter, one running program, no queue.**
+
+| Situation | Behaviour |
+|-----------|-----------|
+| statement arrives, nothing running | `ACK`, execute |
+| statement arrives, program running | `ERR 12 busy` — not parsed, not stored |
+| `DEF`/`DEL` arrives while a program is running | `ERR 12 busy` |
+| `ABORT` arrives | immediate, always (§5.2) |
+
+The client contract is therefore exactly one sentence: **one outstanding
+statement at a time — wait for `DONE` or `ERR` before sending the next.**
+
+### 9.1 Why no queue
+
+Queueing looks friendlier and is not, for three reasons.
+
+**The language already solves it.** Chaining work is what sequences are for:
+`[ SQUARE side=1000; ZIGZAG deg=30; STOP ]` is one statement, one tag, one
+`ACK`, one `DONE`. It is validated as a whole before anything moves, runs with
+no inter-statement gaps, and aborts as a whole. A queue is a worse duplicate of
+a feature that already exists, reachable only by giving up whole-program
+validation.
+
+**A queue invents a question with no good answer.** If `A` is running, `B` is
+queued, and `A` fails with `E_FAULT` — does `B` run? Yes means the robot keeps
+driving after a fault. No means an `ACK` was a promise the robot silently
+broke. With `[A; B]` the question never arises: the sequence stops. Every
+queued design has to pick a side and surprise somebody.
+
+**A queue is wrong for the case people actually want it for.** Live driving —
+arrow keys, a joystick app — wants *preemption*: press left while moving
+forward and left should happen now. A queue delivers the opposite, turning
+steering input into a laggy backlog. Batch work wants sequences; interactive
+work wants preemption; the queue serves neither.
+
+`DEF` and `DEL` are refused during execution for a separate reason: mutating
+the arena while a callable may be live on the interpreter stack is not worth
+making safe for a convenience nobody needs.
+
+### 9.2 If preemption is needed later
+
+Should live driving materialise, the addition is explicit and small: a `!`
+prefix on the envelope meaning *abort whatever is running and execute this
+instead* — `!DRIVE left=40 right=-40`. It reuses the `ABORT` path wholesale and
+requires no queue, no reordering, and no change to §9.1's guarantees. It is
+deliberately not in v1 because a preempting command is a loaded gun and should
+be designed against a real teleop client rather than an imagined one.
+
+## 10. Abort and failsafe
+
+`ABORT` unwinds the stack, calls `hal_stop_all()`, and emits `EVT ABORT`
+followed by `ERR 13` for the tag of the program it killed. With no queue there
+is nothing else to flush — which is the second benefit of §9: abort has exactly
+one victim, and its identity is never in doubt.
+
+Three independent guards cover the case where nobody is left to send `ABORT`:
+
+| Guard | Trigger | Action |
+|-------|---------|--------|
+| **Deadman** | no inbound bytes on the owning session for `RUBO_LINK_TIMEOUT_MS` **while motors are engaged** | abort + stop, `EVT LINKLOST` |
+| **Disconnect** | transport reports `!connected()` while motors are engaged | abort + stop |
+| **Motor cap** | any single motor-engaging native exceeds `RUBO_MOTOR_MAX_MS` | stop that native, `ERR 6` |
+
+The deadman is armed only while motors are engaged, so an idle robot on a quiet
+link is not perpetually resetting itself. `PING` is sufficient to feed it, which
+is why it is a control word rather than a native.
+
+### 10.1 A client must ping while it waits
+
+This falls out of §9 and is worth stating plainly, because it is the one place
+where two rules interact in a way that surprises people:
+
+> A program that engages the motors for longer than `RUBO_LINK_TIMEOUT_MS`
+> will be killed by the deadman unless the client sends something during it.
+
+`SQUARE side=1000` runs for about six seconds. A client that submits it and
+then waits quietly for `DONE` — the obvious reading of §9's "wait for `DONE`
+or `ERR`" — gets `EVT LINKLOST` instead. That is the deadman working
+correctly: it cannot distinguish an attentive operator from a crashed one, and
+guessing in the robot's favour is how robots drive into walls.
+
+So the client contract has a second sentence: **`PING` on an interval
+(≈500 ms) whenever a program is running.** The reference client in
+`host/main.c --script` does exactly this.
+
+## 11. Bulk data
+
+The protocol is text; camera frames are not. Frames leave as `DAT` records,
+chunked to the transport MTU, base64-encoded:
+
+```
+< DAT 3 0 8 <base64…>
+< DAT 3 1 8 <base64…>
+  …
+< DEND 3
+```
+
+`<id>` correlates chunks to the `CAM action=SNAP` that produced them; `<seq>`
+and `<total>` let a client detect loss and reassemble. Base64 costs 33%
+overhead, which for a B&W QQVGA snapshot (160×120×1bpp ≈ 2.4 KB → ≈ 3.2 KB) is
+acceptable for stills and firmly not acceptable for video. A binary side
+channel can be added later without disturbing the text protocol; streaming
+video was never in scope for the command path.
+
+## 12. Multiple transports
+
+All configured transports may be open simultaneously. Each holds its own framer
+and buffer. Statements from any transport compete for the same single execution
+slot, `E_BUSY` is global, and `EVT`/`DAT` are broadcast to every connected
+transport.
+
+This is honest rather than clever: two clients can interleave commands and
+confuse each other. Exclusive session ownership — first-come with explicit
+release or timeout takeover — is the obvious v2 refinement, but it needs a
+policy decision about who may seize control of a moving robot, and that
+decision is better made with the hardware in front of us.
+
+The deadman (§10) tracks the session that submitted the running program, not
+the aggregate of all links.
+
+## 13. Constants
+
+| Constant                   | Suggested | Meaning |
+|----------------------------|-----------|---------|
+| `RUBO_FRAME_TIMEOUT_MS`    | 2000      | partial statement idle timeout |
+| `RUBO_LINK_TIMEOUT_MS`     | 2000      | deadman, while motors engaged |
+| `RUBO_MOTOR_MAX_MS`        | 10000     | hard cap on one motor-engaging native |
+| `RUBO_STEPS_PER_TICK`      | 32        | interpreter steps before yielding |
+| `RUBO_RX_BUF`              | 1088      | ≥ `RUBO_MAX_STMT` + envelope |
+| `RUBO_TCP_PORT`            | 3141      | `wifi_tcp` default |
+
+## 14. Worked exchange
+
+```
+> @1 DEF ZIGZAG $deg=45 $reps=3 [
+>   REPEAT $reps [
+>     TURN dir=LEFT deg=$deg;  MOVE dir=FWD speed=40 dur=300;
+>     TURN dir=RIGHT deg=$deg; MOVE dir=FWD speed=40 dur=300
+>   ]
+> ]
+< @1 ACK
+< @1 DONE
+
+> @2 [ SQUARE side=1000; ZIGZAG deg=30 reps=5 ]
+< @2 ACK
+> @3 SQUARE side=200
+< @3 ERR 12 busy                  # one outstanding statement at a time
+> @4 DEF FOO [ STOP ]
+< @4 ERR 12 busy                  # definitions are not an exception
+> ABORT
+< EVT ABORT
+< @2 ERR 13 aborted               # exactly one victim
+> @3 SQUARE side=200
+< @3 ACK
+< @3 DONE
+```
+
+Note that the multi-line `DEF` at `@1` is a single statement: the framer emits
+it only at the newline following the closing bracket.
+
+## 15. Open questions
+
+1. **Should `EVT` be subscribable?** Broadcasting every event to every
+   transport is fine at this scale and wasteful over BLE if telemetry gets
+   chatty. An `EVT` mask per session is cheap to add later.
+2. **`STAT` output format.** Currently ad-hoc `k=v` pairs. If it grows, it
+   wants a schema.
+3. **Exclusive session ownership** (§12) — deferred pending hardware.