For Good Measure

How I debugged Terobo

by Colby Russell. 2019 May 14.

I encountered some issues during the development of TeroBuild/Terobo. This post discusses how I handled those issues.

Terobo is a replacement for a small runtime called Norebo written by Peter De Wachter. Norebo was originally implemented in C, and its purpose is to act as a bridge from a host system (i.e., your computer) into the world that Wirth's Oberon system expects to inhabit, by simulating it. That means simulating the instruction set of the bespoke RISC design that Wirth cooked up, plus a handful of system calls (or "sysreqs") to interact with the outside world on the host machine.

Under Terobo or Norebo, when an Oberon program uses the Files.Seek or Files.Write APIs within the system, the bridge on the inside of the system relies on the machine code executing no differently than the CPU embodied in real world hardware would, up to and including the branch instruction that jumps from the calling code into subroutine being called. The Norebo bridge at this point, however, attempts to write to a very large memory address: something like 0xFFFFFFFC, or -4 if the bit pattern is interpreted as a 32-bit two's complement signed integer. Since the hardware being emulated has nowhere near 2^32 (4GiB) of memory, this address space is reserved for memory-mapped IO, and it's known that this address in particular is used for making sysreq calls.

Both Terobo and Norebo, simulating the hardware in question, know to intercept reads and writes in this address space and handle them accordingly. For example, the runtimes associate a Files.Seek request with the constant 15. When a machine-level store instruction attempts to write the value 15 to 0xFFFFFFFC, the runtime delegates to an equivalent operation on the side of the bridge rooted in the host system. Otherwise, Terobo is very much concerned only with the fetch/decode/execute cycle present in anything dealing with machine-level instructions. This can pose some problems, especially when it happens on such an obscure platform.

Terobo is using a special build of the Oberon system crafted specifically to operate non-interactively, with no peripherals to provide input and output for the system, making diagnosis difficult when a problem rears its head, given how opaque this works out to be. Disregarding that, the debugging situation even within a full-fledged Oberon system is basically non-existent. Despite the language being an ostensibly "safe" one, Oberon programs—especially the system-level modules that get much of the CPU time within Terobo and Norebo—are perfectly capable of and prone to doing the sorts of things commonly associated with a language like C, such as branching into the void, otherwise dereferencing a bad pointer, or getting caught in an infinite loop, all because, say, my Files API implementation for Terobo is buggy.

So aside from writing perfect code on the first pass, how does one deal with this?

For one, Peter's implementation in C already existed, which was helpful as an overall guide, but not in fixing implementation-specific problems in the code I was writing. (Side note: I'm not a fan of the buffer and typed array design that TC-39 came up with for working with binary data. I find it really cumbersome, and it's fraught with gotchas. For all its problems, I think the way C exposes this kind of stuff to the programmer, if you consider language design as a sort of UI, to be better, even with the proliferation of pointers and the perils of using arithmetic for them.)

Secondly, I'd already begun working on my own interactive gdb-like debugger to operate at the machine-level, allowing you to pause execution by setting breakpoints, peek and poke at memory addresses, disassemble blocks of instructions, and step through execution. Since the debugger fully controlled the machine simulation, I'd even implemented reversible debugging so you could step backwards through code. It's on this basis that I began working on Terobo. Rather than starting from a blank slate, I just forked my debugger (called rewrd), and began patching it in the general direction of Norebo, implementing Peter's design for the special memory mapping scheme I explained earlier.

Here's a look at rewrd in action:

Occasionally, while developing Terobo, there would be problems that I had no idea how to solve, due to how difficult it was trying to peer into a machine operating so opaquely and where you had no bearings. I did already have support for importing source maps to follow an arbitrary machine instruction back to the original line of Oberon source code that was responsible for the position in the current stack frame, although I didn't have any such maps on hand for the binaries distributed in the Norebo directory. Generating them would have been and still is a fairly cumbersome process, and finding out which blocks of memory they mapped to would be something like an order of magnitude even harder than that. I'd also already added a way to produce something akin to core dumps, but this wasn't particularly useful given the absence of any other tools that could process these dumps and communicate anything meaningful about what they'd contain.

Two things that proved invaluable were adding tracing to both my implementation and the C implementation, and I added memory dumps as well. I patched the C implementation to take command-line flags to enable tracing, limit the number of steps the simulated machine would take before halting, or limit the number of memory-mapped sysreq invocations to handle, and upon reaching the limit, immediately dump the machine's memory to disk, along with the binary log containing the execution trace.

I implemented tracing very simply: every time the machine executed an instruction, it would log the memory address of the instruction in question, along with four bytes containing the instruction itself. Based on this, and the same mechanism implemented in Terobo, I could take these traces, convert them into human readable text files using a pipeline, and then diff them, all with tools from the standard UNIX toolbox.

Here's the magic pipeline I used:

od -v -A n -t x4 .../terobo_trace | sed \
  "s/\([^ ]\+\) \([^ ]\+\) /\1 \2\n /"

I was pretty bummed when I checked the man pages for od and didn't find any way to force it to output to two columns (for two four-byte fields) instead of four columns (for 16 bytes per row of output). The sed transformation fixes this, although it can be somewhat slow for very large trace logs—I didn't feel like stopping to write the code to do my own text conversion, though, even though I should have.

Given two textual trace logs, one from my implementation and one from Norebo, the line offset where diff reported a disparity may or may not be the offending instruction. It's possible that due to a bad sysreq implementation of some earlier call, the machine was ending up in a bad state but in such a way it wasn't apparent by looking solely at the execution trace. So having narrowed down a place where a problem was known to occur, I then turned to looking at memory.

If some difference was apparent between Norebo and Terobo's traces at step N, I'd take snapshots of the contents of memory there (only 8MB—up from the 1MB that Oberon ordinarily runs under), and look at how the two compared. I'd then establish the existence of some prior state where the two implementations agreed, and work my way towards the problem area using the familiar process of bisection. It was reasonable, though, to expect in most cases that any given problem that occurred was a result of the implementation for the previous sysreq.

And that's essentially it for half the bugs that I spent my time on.

Of the other half, they turned out being problems either in how I was dealing with asynchronous file reads and attempting to re-enter the fetch/decode/execute loop, or the poor way I'd hacked Terobo on top of rewrd's debugger repl. The debugger ended up being very useful, however.

Even without the class of bugs that wouldn't have occurred had I not decided to start off with rewrd forming the initial basis of the implementation, I'm not sure how much longer development would have taken if I'd not had the interactive debugging shell available. Additionally, I ended up fixing a number bugs and adding features to rewrd as a result of my needs developing Terobo, so there came some other good out of it.

Here, now, is a link (above) to a 2 minute video demonstrating the utility of this work—extremely portable build tooling with no dependencies other than the universally accessible application runtime.