The Second Executable in the Data Object Framework

J. Rogers, SE Ohio

Building a unit-test host on top of the framew
ork library, and what five simultaneous builds of it found

---

The framework has always carried its own tests. Every core module has a self-test compiled into it — DataTest() in data.c, NodeTest() in node.c, SchedTest() in sched.c, BuffTest() in dyn/, NameSpaceTest() in namespace.c — and ./framework -t runs them in sequence. They ship inside libframework.so alongside the code they exercise, which is exactly where they belong.

The problem isn’t the tests. It’s who runs them. main.c runs them, and main.c is the application. A program grading its own homework will always hand back an A.

unit_test.c

The fix started as a copy: cp main.c unit_test.c, delete the default app, then bring each module’s test across under a renamed symbol — UT_DataTest, UT_NodeTest, UT_BuffTest, and so on. Both sets now exist. ./unit_test -t calls the library’s copies through the same path framework -t uses; ./unit_test with no arguments calls the UT_ copies compiled into the executable. If those two ever disagree, one of them has drifted, and you find out immediately instead of six months later.

The build was restructured to match: sources moved to src/, every artifact lands in build/, and unit_test is built whenever framework is — same rules, same directory, no special case. A test binary that is a chore to build is a test binary that doesn’t get built.

Three of the copied tests are currently #if 0’d with the reason written where the code was. They reach into module-private state — a TaskPtr here, a static class list there — that a second host legitimately cannot see. That’s not a defeat; it’s the first thing the exercise told us.

What a second host sees that the first one can’t

Here is the part I didn’t expect to be the main event.

The library is supposed to be an embeddable core. The whole hosting contract is five calls: set the registry list, set the task list, install objects, then pump TimeUpdate() and ExecTasks() from whatever event loop you already have. Anything that can dlopen a shared object and call five functions gets the entire object system.

Except it didn’t, quite. Writing a second host that wanted the fabric and none of the furniture surfaced a list of things that had quietly become requirements of the library rather than of the app: building the palette, building the chrome, a known-classes list, the object scan path, the settings home, the reserved view names. GUI concerns, sitting in the core, invisible for as long as the only program linking the core was the GUI’s own host.

You cannot find that by reading main.c. main.c is the app, so everything in it looks like it belongs. You find it by writing a second program that links the same library and wants a different subset — the moment it needs a palette it will never draw, the boundary you thought you had is drawn in the wrong place. The unit-test host is a boundary detector that happens to also run tests.

That’s the microkernel argument made empirical instead of aspirational.

Five builds, one wall clock

Once tests run from a host that isn’t the app, the same source can be compiled several ways and tested identically. The harness now builds five variants — debug, release, ASAN, UBSAN, gcov — each into its own directory, each with its own ports above the production instance, each with its own hardlinked copy of the objects and the web client, each running the full suite list against its own engine.

They run in parallel. There are sixteen cores; five builds sitting on five of them cost the wall-clock time of the slowest one. Coverage instrumentation and sanitizer instrumentation are effectively free, because the thing you were waiting for anyway is the debug run.

Each variant writes its logs under its own log/, keeps its saved flows for forensics, and the only thing at the base of the tree is one report:

20:51:46  [release] framework quit when asked - exit-time leak check ran
20:51:46  [ubsan] framework quit when asked - exit-time leak check ran
20:51:47  all variants finished, writing the report
suite               debug    release  asan     ubsan    gcov     
---------------------------- -------- -------- -------- -------- 
unit_test           0        0        0        0        0        
rawtest             0        0        0        0        0        
connectiontest      0        0        0        0        0        
leaktest            0        0        0        0        0        
flowtest            2        2        2        2        2        
viewclonetest       0        0        0        0        0        
jstest              0        0        0        0        0        
scriptboxtest       1        1        CRASH    1        1        
scriptedwidgettest  0        0        CRASH    0        0        
widgettest          0        0        CRASH    0        0        
tcpporttest         6        10       CRASH    8        8        
guitest             6        6        CRASH    6        6        
---------------------------- -------- -------- -------- -------- 
failures            15       19       2        17       17       
crashed             0        0        5        0        0        
leaked              0        0        0        0        0        
sanitizer           0        0        1        0        0        

debug: build return code=0  web=8084 raw=8092 cdp=9224  logs=testharness/tests/20260805_203113/debug/log/
release: build return code=0  web=8085 raw=8093 cdp=9225  logs=testharness/tests/20260805_203113/release/log/
asan: build return code=0  web=8086 raw=8094 cdp=9226  logs=testharness/tests/20260805_203113/asan/log/
ubsan: build return code=0  web=8087 raw=8095 cdp=9227  logs=testharness/tests/20260805_203113/ubsan/log/
gcov: build return code=0  web=8088 raw=8096 cdp=9228  logs=testharness/tests/20260805_203113/gcov/log/
coverage: testharness/tests/20260805_203113/gcov/coverage.txt
logs: testharness/tests/20260805_203113/log/ and testharness/tests/20260805_203113/<variant>/log/   summary: testharness/tests/20260805_203113/report.txt
20:51:48  70 failures, 5 crashes and 0 leaks across all variants

Builds across the top, suites down the side. Getting the vocabulary right mattered more than it sounds: a dead engine and a single failed assertion both used to print 1, so one death read as one failure and eleven cascading connection errors read as eleven more. Now a suite that never measured anything says CRASH, a suite that ran but leaked says LEAK (or 2+LEAK, keeping its own count), and a crash ends that variant on the spot instead of running ten more suites against a corpse. Zero everywhere says success. The return code is failures plus crashes plus leaks.

The bugs

An overlapping memcpy, running for years. ASAN stopped on buffResize, on a line whose own comment predicted it would be fine:

/* we shouldn't need to use move mem here, there shouldn't be a memory overlap */
memcpy((char *)obj->buffer, (char *)obj->buffer+obj->tail, obj->head-obj->tail);

The ranges overlap whenever the live span is longer than the distance it slides down — in the observed case, dest and src 0x800 apart with a 0x9ad0 span. Undefined behaviour that happens to work, because glibc copies forward and dest < src. Two different callers reach it: the line reader, and the TCP drain. It is now a memmove.

Worth noting that the near-identical slide inside buffAdd is not the bug — it’s guarded by tail > size/2, so it genuinely cannot overlap. Two copies of almost the same code, one correct by accident and one incorrect by accident, is its own lesson.

A test that leaked 585 bytes. With the overlap fixed, the ASAN build stopped aborting, and for the first time LeakSanitizer’s exit-time check actually ran. It reported 14 allocations: five DataObjs created by the data test and never freed, plus the nine strings they allocated while walking the type-conversion matrix. The test, not the library — which is its own kind of finding, since it means every framework -t has leaked that much forever, and nobody could see it.

A use-after-free in a widget swap. The reward for keeping the sanitizer alive past its first find. Changing a scripted widget’s language deletes the old host and creates a new one, and the swap holds a node belonging to the deleted instance across its own DeleteInstance — freed at line 152, read at line 180 through Connect → ResolvePort → GetPropNode. Every build has this bug; only the instrumented one says so, because freed memory usually still reads plausibly. The fix is architectural rather than a patch: the widget should own its engine as a private opaque handle the way the network widgets do, so there is no internal node for it to hold in the first place.

Getting the leak check to run at all

LeakSanitizer only reports when a process exits normally, and the harness was killing every engine with a signal. The framework can be asked to leave — and it turned out no new mechanism was needed to ask. The main loop runs while Main’s State property is non-zero, /Main is an ordinary registered path, and set-property is an ordinary command. So:

{"cmd":"set-property","instance":"/Main","prop":"State","value":"0"}

The engine shuts down through the same cleanup path it uses when the flow drains, in about a second, and the leak check runs. A “quit verb” that already existed, spelled out of parts that were already there — which is the point of the design, and a nice reminder to look before adding.

What it cost, and what it’s worth

Most of a day was spent on the un-glamorous half: making the harness lie less. Hard linking the web client into each variant directory, because the HTTP object serves relative to the working directory and twenty GUI tests were cascading off a blank page. Separating “crashed” from “failed”. Not claiming a leak check ran on a process that died an hour earlier. Refusing to interpret a missing pid without asking what it means first.

That work isn’t decoration. A grid that reads 1 in twelve cells sends you looking for twelve bugs. The same grid reading CRASH in twelve cells sends you to one log, which is where the one bug is.

The core tests still don’t cover the objects — fifty modules with no self-tests reachable from the harness, though the registry already has the mechanism to register them the same way classes register. And a matrix makes flakiness visible without making it go away: one suite reported 7, 6, 1, 10 and 9 failures across five builds of identical source, which is a timing bug wearing five costumes.

But the instrument exists now, and it’s pointed at the right thing. The tests aren’t run by the program under test anymore. They’re run by a second program that wants the same library for different reasons — and that difference is what makes them tests.