rubo/docs/flow.md

Rubo Command Flow — Transport, Framing and Execution

Status: draft, v1. Companion to docs/grammar.md, which specifies the language. This document specifies everything around it: how bytes become statements, how statements become motion, and how the robot is stopped.

1. Principles

1. Nothing blocks. The interpreter is a resumable state machine driven from the main loop. A MOVE lasting ten seconds must not prevent ABORT from being received in second three. This constraint shapes the entire design.
2. The envelope is not the language. Correlation tags and control words are handled by the session layer and never reach the parser. docs/grammar.md stays a specification of a language, not of a wire protocol.
3. One framer for every transport. USB, WiFi, BLE and the host test harness all reduce to a byte stream. Only MTU and connection semantics differ.
4. A dropped link stops the robot. Every path that engages a motor is covered by a deadman timer. This is the one requirement that is not negotiable for convenience.

2. Layer stack

     ┌──────────────────────────────────────────────┐
     │ transport   usb_cdc │ wifi_tcp │ ble_nus │ host_stdio
     │             byte stream in/out, MTU-aware
     ├──────────────────────────────────────────────┤
     │ framer      bracket-balance statement extraction (§4)
     ├──────────────────────────────────────────────┤
     │ session     tags, control words, ACK/DONE/ERR, backpressure (§5–§7)
     ├──────────────────────────────────────────────┤
     │ parser      → validator → arena          (docs/grammar.md)
     ├──────────────────────────────────────────────┤
     │ scheduler   resumable interpreter, explicit stack, ticks (§8)
     ├──────────────────────────────────────────────┤
     │ HAL         motor │ camera │ battery      sim | pico
     └──────────────────────────────────────────────┘

Layers below the parser are shared; only transport and HAL have per-target implementations. The host build swaps exactly those two and runs everything else unmodified, which is what makes the language testable with no hardware.

3. Transport interface

typedef struct {
    int    (*open)(void *cfg);
    int    (*read)(uint8_t *buf, size_t len);        /* non-blocking; 0 = no data */
    int    (*write)(const uint8_t *buf, size_t len);
    bool   (*connected)(void);
    void   (*close)(void);
    size_t mtu;                                      /* max bytes per write */
} rubo_transport_t;

read must never block. Everything above this line is polled from the main loop; a blocking read anywhere forfeits §1.1.

Transport	Implementation	MTU	Notes
usb_cdc	TinyUSB CDC (pico-sdk)	64	the development default; enumerate as ACM
wifi_tcp	lwIP listener, one client	~1460	CYW43; port configurable, default 3141
ble_nus	Nordic UART Service	20–244	notify for TX, write-without-response for RX; chunking mandatory
host_stdio	stdin/stdout	—	host build; drives the whole stack under a test harness
host_tcp	POSIX socket	—	host build; lets a real client speak to a simulated robot

BLE's small MTU is not a special case: the framer is byte-oriented, so a statement split across five notifications reassembles exactly as one arriving in a single USB read.

4. Framing

docs/grammar.md §3.3 established that statements end at bracket balance, not at newlines. The framer is therefore a small state machine, not a line reader.

4.1 Rule

A top-level statement is complete at the first newline encountered while bracket depth is zero and the buffer is non-empty.

Inside brackets, newlines are ordinary whitespace. A multi-line DEF pasted into a terminal arrives as one statement; a one-liner arrives as one statement. Interactive typing and bulk paste need no mode switch.

4.2 State machine

State	Input	Action
STMT	#	→ COMMENT
STMT	[	depth++; append
STMT	]	depth--; append; if depth < 0 → error E_SYNTAX, → DISCARD
STMT	newline	if depth == 0 && len > 0 → emit; else treat as space
STMT	other	append
COMMENT	newline	→ STMT (apply the newline rule above)
COMMENT	other	discard
DISCARD	newline	reset depth/len, → STMT
DISCARD	other	discard

The framer must be comment-aware: a [ inside a comment must not affect depth, or a single # in a comment would desynchronise the stream permanently. There are no string literals in the language, so no quote handling is needed.

4.3 Failure and resynchronisation

Condition	Response	Recovery
len > RUBO_MAX_STMT	ERR 10 statement too long	→ DISCARD
unbalanced ]	ERR 1 unexpected ]	→ DISCARD
partial statement idle > RUBO_FRAME_TIMEOUT_MS	ERR 1 incomplete statement	reset
transport disconnect mid-statement	—	reset buffer; §10 deadman applies

The idle timeout matters most on BLE, where a client can vanish mid-frame leaving an unterminated [ that would otherwise wedge the framer forever.

4.4 Top-level ;

A bare ; at depth zero is E_SYNTAX. One frame carries one statement; to submit several as a unit, wrap them: [ A; B; C ]. This keeps "one statement, one tag, one ACK, one DONE" exact.

5. Envelope

Two things wrap a statement and are stripped by the session layer before parsing. Neither appears in docs/grammar.md, because neither is part of the language.

5.1 Correlation tags

An optional @<uint16> prefix. Responses echo it.

> @7 SQUARE side=1000
< @7 ACK
< @7 DONE

Untagged statements are legal and produce untagged responses — convenient when a human is typing. Tags are the client's to allocate; the robot never interprets them beyond echoing.

5.2 Control words

Recognised before parsing, executed immediately, never queued, never subject to E_BUSY. They bypass the interpreter entirely.

Word	Effect
ABORT	unwind the interpreter stack, hal_stop_all(), end the running program (§10)
PING	PONG — liveness, and it feeds the deadman timer
STAT	one-line status: running callable, tag, depth, arena free, link age

Control words are deliberately few. ABORT in particular cannot be a native callable, because a callable can only run when the interpreter is free to run it — which is precisely when you do not need it.

6. Response vocabulary

@7 ACK                              statement admitted, parsed, validated; now executing
@7 DONE                             execution finished normally
@7 ERR <code> <message>             rejected or failed; codes in grammar.md §10
EVT <name> [args...]                asynchronous, untagged, broadcast
DAT <id> <seq> <total> <payload>    bulk data chunk (§11)
DEND <id>                           bulk data complete
PONG                                reply to PING
STAT <k>=<v> ...                    reply to STAT

ACK and DONE are always both sent, including for instantaneous statements like DEF and DEL. Collapsing them for non-executing statements would save a few bytes and cost every client a special case; a terminal client is free to hide ACK when DONE follows immediately.

ERR may arrive instead of ACK (rejected at parse or validation) or after ACK (failed during execution). A client waits for exactly one of DONE or ERR per tag.

7. Statement lifecycle

  bytes ─► RECEIVING ─► ADMISSION ─► PARSING ─► VALIDATING ─► RUNNING ─► DONE
             │             │            │            │           │
             │             │            │            │           ├─► ERR (runtime: E_RANGE, E_FAULT)
             │             │            │            │           └─► ABORTED (E_ABORT)
             │             │            │            └─► ERR (E_UNKNOWN, E_ARITY, E_ARGNAME,
             │             │            │                     E_UNDEFINED, E_CYCLE, E_NOMEM…)
             │             │            └─► ERR (E_SYNTAX, E_LIMIT)
             │             └─► ERR 12 busy
             └─► discarded (§4.3)

Admission comes before parsing: if a program is already running, the statement is rejected immediately without being parsed at all (§9). There is no point spending cycles validating something that cannot run.

Validation is the last point at which rejection is free. Past ACK, a failure means the robot has already moved, so the majority of docs/grammar.md §10 is deliberately checkable before execution begins.

8. Execution model

8.1 Explicit stack, no C recursion

The interpreter holds a statically allocated array of frames:

typedef struct {
    const rubo_node_t *node;      /* current statement in the body        */
    uint16_t           pc;        /* index within the enclosing sequence  */
    uint16_t           iter;      /* REPEAT counter                       */
    rubo_value_t       args[RUBO_MAX_PARAMS];
    uint32_t           deadline;  /* native scratch: ms, 0 = none         */
} rubo_frame_t;

static rubo_frame_t stack[RUBO_MAX_DEPTH];

docs/grammar.md §5.1 guarantees the call graph is acyclic, so RUBO_MAX_DEPTH frames are provably sufficient and depth is checked at definition time. Nothing here recurses in C, so the MCU stack is not at risk regardless of what the user sends.

The explicit stack buys three things beyond safety: ABORT is a matter of resetting an index, STAT can report exactly what is executing, and the whole interpreter is trivially unit-testable on the host.

8.2 Natives are resumable

typedef enum { RUBO_STEP_DONE, RUBO_STEP_PENDING, RUBO_STEP_ERR } rubo_step_t;

typedef rubo_step_t (*rubo_native_fn)(rubo_frame_t *f, uint32_t now_ms);

A native with a duration sets f->deadline on first entry and returns RUBO_STEP_PENDING until now_ms >= f->deadline. MOVE dir=FWD speed=50 dur=1000 engages the motors, returns immediately, and is re-entered each tick until its deadline passes — during which the transport is polled normally and ABORT is honoured within one tick.

8.3 Main loop

for (;;) {
    uint32_t now = rubo_millis();
    rubo_transport_poll(now);   /* read bytes, feed framers               */
    rubo_session_poll(now);     /* frame → envelope → admit → parse → run */
    rubo_exec_tick(now);        /* advance the interpreter, bounded work  */
    rubo_hal_poll(now);         /* sensors, deadman, event generation     */
}

Cooperative and single-core; no RTOS, and the second RP2040 core stays free for the camera pipeline later. rubo_exec_tick runs at most RUBO_STEPS_PER_TICK interpreter steps before yielding, so a deeply nested REPEAT of instantaneous statements cannot starve the transport.

9. Concurrency and backpressure

One interpreter, one running program, no queue.

Situation	Behaviour
statement arrives, nothing running	ACK, execute
statement arrives, program running	ERR 12 busy — not parsed, not stored
DEF/DEL arrives while a program is running	ERR 12 busy
ABORT arrives	immediate, always (§5.2)

The client contract is therefore exactly one sentence: one outstanding statement at a time — wait for DONE or ERR before sending the next.

9.1 Why no queue

Queueing looks friendlier and is not, for three reasons.

The language already solves it. Chaining work is what sequences are for: [ SQUARE side=1000; ZIGZAG deg=30; STOP ] is one statement, one tag, one ACK, one DONE. It is validated as a whole before anything moves, runs with no inter-statement gaps, and aborts as a whole. A queue is a worse duplicate of a feature that already exists, reachable only by giving up whole-program validation.

A queue invents a question with no good answer. If A is running, B is queued, and A fails with E_FAULT — does B run? Yes means the robot keeps driving after a fault. No means an ACK was a promise the robot silently broke. With [A; B] the question never arises: the sequence stops. Every queued design has to pick a side and surprise somebody.

A queue is wrong for the case people actually want it for. Live driving — arrow keys, a joystick app — wants preemption: press left while moving forward and left should happen now. A queue delivers the opposite, turning steering input into a laggy backlog. Batch work wants sequences; interactive work wants preemption; the queue serves neither.

DEF and DEL are refused during execution for a separate reason: mutating the arena while a callable may be live on the interpreter stack is not worth making safe for a convenience nobody needs.

9.2 If preemption is needed later

Should live driving materialise, the addition is explicit and small: a ! prefix on the envelope meaning abort whatever is running and execute this instead — !DRIVE left=40 right=-40. It reuses the ABORT path wholesale and requires no queue, no reordering, and no change to §9.1's guarantees. It is deliberately not in v1 because a preempting command is a loaded gun and should be designed against a real teleop client rather than an imagined one.

10. Abort and failsafe

ABORT unwinds the stack, calls hal_stop_all(), and emits EVT ABORT followed by ERR 13 for the tag of the program it killed. With no queue there is nothing else to flush — which is the second benefit of §9: abort has exactly one victim, and its identity is never in doubt.

Three independent guards cover the case where nobody is left to send ABORT:

Guard	Trigger	Action
Deadman	no inbound bytes on the owning session for RUBO_LINK_TIMEOUT_MS while motors are engaged	abort + stop, EVT LINKLOST
Disconnect	transport reports !connected() while motors are engaged	abort + stop
Motor cap	any single motor-engaging native exceeds RUBO_MOTOR_MAX_MS	stop that native, ERR 6

The deadman is armed only while motors are engaged, so an idle robot on a quiet link is not perpetually resetting itself. PING is sufficient to feed it, which is why it is a control word rather than a native.

10.1 A client must ping while it waits

This falls out of §9 and is worth stating plainly, because it is the one place where two rules interact in a way that surprises people:

A program that engages the motors for longer than RUBO_LINK_TIMEOUT_MS will be killed by the deadman unless the client sends something during it.

SQUARE side=1000 runs for about six seconds. A client that submits it and then waits quietly for DONE — the obvious reading of §9's "wait for DONE or ERR" — gets EVT LINKLOST instead. That is the deadman working correctly: it cannot distinguish an attentive operator from a crashed one, and guessing in the robot's favour is how robots drive into walls.

So the client contract has a second sentence: PING on an interval (≈500 ms) whenever a program is running. The reference client in host/main.c --script does exactly this.

11. Bulk data

The protocol is text; camera frames are not. Frames leave as DAT records, chunked to the transport MTU, base64-encoded:

< DAT 3 0 8 <base64…>
< DAT 3 1 8 <base64…>
  …
< DEND 3

<id> correlates chunks to the CAM action=SNAP that produced them; <seq> and <total> let a client detect loss and reassemble. Base64 costs 33% overhead, which for a B&W QQVGA snapshot (160×120×1bpp ≈ 2.4 KB → ≈ 3.2 KB) is acceptable for stills and firmly not acceptable for video. A binary side channel can be added later without disturbing the text protocol; streaming video was never in scope for the command path.

12. Multiple transports

All configured transports may be open simultaneously. Each holds its own framer and buffer. Statements from any transport compete for the same single execution slot, E_BUSY is global, and EVT/DAT are broadcast to every connected transport.

This is honest rather than clever: two clients can interleave commands and confuse each other. Exclusive session ownership — first-come with explicit release or timeout takeover — is the obvious v2 refinement, but it needs a policy decision about who may seize control of a moving robot, and that decision is better made with the hardware in front of us.

The deadman (§10) tracks the session that submitted the running program, not the aggregate of all links.

13. Constants

Constant	Suggested	Meaning
RUBO_FRAME_TIMEOUT_MS	2000	partial statement idle timeout
RUBO_LINK_TIMEOUT_MS	2000	deadman, while motors engaged
RUBO_MOTOR_MAX_MS	10000	hard cap on one motor-engaging native
RUBO_STEPS_PER_TICK	32	interpreter steps before yielding
RUBO_RX_BUF	1088	≥ RUBO_MAX_STMT + envelope
RUBO_TCP_PORT	3141	wifi_tcp default

14. Worked exchange

> @1 DEF ZIGZAG $deg=45 $reps=3 [
>   REPEAT $reps [
>     TURN dir=LEFT deg=$deg;  MOVE dir=FWD speed=40 dur=300;
>     TURN dir=RIGHT deg=$deg; MOVE dir=FWD speed=40 dur=300
>   ]
> ]
< @1 ACK
< @1 DONE

> @2 [ SQUARE side=1000; ZIGZAG deg=30 reps=5 ]
< @2 ACK
> @3 SQUARE side=200
< @3 ERR 12 busy                  # one outstanding statement at a time
> @4 DEF FOO [ STOP ]
< @4 ERR 12 busy                  # definitions are not an exception
> ABORT
< EVT ABORT
< @2 ERR 13 aborted               # exactly one victim
> @3 SQUARE side=200
< @3 ACK
< @3 DONE

Note that the multi-line DEF at @1 is a single statement: the framer emits it only at the newline following the closing bracket.

15. Open questions

1. Should EVT be subscribable? Broadcasting every event to every transport is fine at this scale and wasteful over BLE if telemetry gets chatty. An EVT mask per session is cheap to add later.
2. STAT output format. Currently ad-hoc k=v pairs. If it grows, it wants a schema.
3. Exclusive session ownership (§12) — deferred pending hardware.