MeshCore Willow KISS framing in Hew

Willow, the Hew host variant of Birch, talks to its Wi-Fi connected MeshCore modem over TCP, which leaves the host with a slightly awkward job: the modem has packets to deliver, but a socket read gives it whatever bytes are available. A perfectly good frame can arrive in several reads, and one of those reads can stop halfway through an escape sequence.

For a small synthetic example, I’m using five bytes that include both values the framing protocol has to escape:

00 01 c0 db 02

Those c0 and db values are where the trouble starts. The host/modem protocol here is KISS, whose SLIP-style framing uses c0 to mark a frame boundary and db to introduce an escape. To carry those values inside a record, a literal c0 becomes db dc, and a literal db becomes db dd. The other bytes pass through unchanged, so the encoded frame looks like this:

c0 00 01 db dc db dd 02 c0

Now suppose the first read returns c0 00 01 db, and the next returns dc db dd 02 c0. At the end of the first read we’ve collected 00 01, but that trailing db still needs another byte before we know what to append. If the read loop starts a fresh decoder for the next chunk, it loses both the unfinished record and the information that turns the next dc back into a literal c0.

That split read leaves a little more to remember than just “the bytes so far”. Willow’s framing module carries the unfinished record and its escape flag into the next feed, then returns the updated state together with any frames that feed completed:

pub type SlipState {
    pending: bytes;
    escaped: bool;
    discard: bool;
}

pub type SlipBatch {
    state: SlipState;
    frames: Vec<bytes>;
    discarded: i64;
}

In the split above, pending keeps 00 01, while escaped tells the next call what to do with dc. A feed can finish with no complete frame at all, so the caller saves the returned state and waits for more input. When the closing c0 finally arrives, the decoded bytes can go into frames. The discard flag gives the decoder a way to wait for its first delimiter or skip a damaged record; we’ll give it a damaged one in a moment.

To make the split unavoidable, the driver feeds the encoded frame one byte at a time, using Willow’s actual framing module and binary helpers. That puts a read boundary between every pair of bytes, including the two halves of each escape, and needs no modem or radio keys. The payload and encoded bytes have already been prepared before this loop, and 514 is the decoded-record limit chosen for this example:

var state = framing.slip_state();
var completed: i64 = 0;
for index in 0..encoded.len() {
    let chunk = binary.slice(encoded, index, index + 1).expect("one byte");
    let batch = framing.slip_feed(state, chunk, 514).expect("feed");
    state = batch.state;
    assert(batch.discarded == 0);
    for frame in batch.frames {
        assert(frame == payload);
        completed += 1;
        println("decoded=" + hex.encode(frame));
    }
}

Follow state = batch.state through those calls and you can see how the last byte of one feed affects the first byte of the next. There is no special treatment for a socket read that ends on db; the escape flag simply stays set until another byte arrives. In a real read loop, chunk would be whatever the socket returned, while the state would still travel around the same loop.

The completed frames have their own bytes, too, which matters when this goes back into a socket handler. The caller can process them while keeping the unfinished suffix for the next feed, without holding a slice into a read buffer that will be overwritten. The KISS command and logical port are inside the completed record, ready for the next layer to interpret:

So far, every byte has been valid, even when its arrival was awkward. Now put c0 db 00 c0 in front of the good frame. The db still introduces an escape, but 00 isn’t a valid second byte. At that point the decoder clears the pending record and uses discard to skip through the next c0, where it can start collecting again. That gives the following valid frame a chance to be decoded without mixing in bytes from the damaged one:

var damaged = hex.decode("c0db00c0");
damaged.append(encoded);
let recovered = framing.slip_feed(
    framing.slip_state(), damaged, 514
).expect("recover");
assert(recovered.discarded == 1 && recovered.frames.len() == 1);
assert(recovered.frames[0] == payload);

This feed returns one discarded record and one recovered frame.

There also has to be a limit on how much we keep while waiting for that delimiter. If a decoded record grows past the selected limit, Willow drops it and skips to the next boundary in the same way. The decoder’s size limit applies to the unescaped record. Here it is 514 bytes; escaping can make the wire representation longer. An input chunk longer than 4096 bytes or an invalid limit returns an error.

A read can just as easily contain two complete frames, so the returned batch has to handle that too. The driver joins two encoded records and checks that both payloads come back. To check that the escape rules preserve ordinary data as well as c0 and db, it also takes every byte from 00 through ff through the one-byte loop. The same recovery check with a valid record after an oversized one confirms that the next delimiter lets the decoder resume after a length failure.

You can run those cases from the example files (download ZIP), which include the driver and both helper modules. This copy is pinned to the Willow implementation preceding its move into MeshCore Aspen. On Linux x86-64, with Hew 0.6.0-rc7 installed, run:

cd examples/willow
hew --version
hew build framing_demo.hew -o framing-demo
./framing-demo

The output is:

payload=0001c0db02
encoded=c00001dbdcdbdd02c0
decoded=0001c0db02
one-byte reads: 1 frame
malformed escape: discarded=1 recovered=1
one read: 2 frames
byte alphabet, oversize recovery and invalid limits: ok

In the service, a complete frame goes on to KISS command and port handling, and a room operation must pass authentication before it can change room state. To connect a host to a modem, follow the Willow host setup guide for the host’s matching compiler, radio profile and startup commands.

To try a different read pattern, feed c0 00 01 db and then dc db dd 02 c0 in two calls instead of using the one-byte loop. Keep passing batch.state from the first call into the second and compare the returned frame with the original five bytes.