Test Harness Plan for libframework.so

 J. Rogers, SE Ohio

1. Overview

We will extract all unit tests from the core source files (data.c, node.c, namespace.c, sched.c, object.c, widget.c, etc.) into a separate test harness that links against libframework.so. The harness will run every test against multiple builds of the library, each compiled with different instrumentation (debug, ASAN, UBSAN, coverage, release), and report only on failures. Each core module will have its own dedicated test function (e.g., test_data(), test_node(), …) that verifies correct behavior, checks memory integrity, and asserts expected outcomes.

The goal is to achieve 100% branch coverage and zero undefined behavior across all build variants, with the test suite serving as both regression protection and living documentation.

---

2. Architecture

2.1 Core Library (libframework.so)

• Production build – -O2 -DNDEBUG, stripped, no test code.

• Debug build – -O0 -g, full symbols, asserts enabled.

• ASAN build – -O1 -fsanitize=address for memory error detection.

• UBSAN build – -O1 -fsanitize=undefined for UB detection.

• GCOV build – -O0 -fprofile-arcs -ftest-coverage for coverage analysis.

All builds share the same source code; no #ifdef TESTBUILD remains in production files. All test logic lives outside the library.

2.2 Test Harness (testharness.c)

A standalone executable that:

• Links against libframework.so (or a variant via LD_LIBRARY_PATH).

• Calls each module’s test function in sequence.

• Each test function exercises the public API (DataNew, NSCreate, CreateObject, SetProp, Connect, DeliverMsg, etc.).

• Each test function returns 0 on success, 1 on failure.

• The harness aggregates results and exits with 0 only if all tests pass.

2.3 Per‑Module Test Functions

Each core module exposes a test function in its public header (or via a separate test_*.h):

c

// data.h
int test_data(void);

// node.h
int test_node(void);

// namespace.h
int test_namespace(void);

// sched.h
int test_sched(void);

// object.h
int test_object(void);

// widget.h
int test_widget(void);

These functions:

• Perform systematic checks on all API entry points.

• Use assert() or custom check macros that print specific error messages.

• Exercise edge cases (null pointers, out‑of‑range values, memory exhaustion simulation).

• (For memory testing) can be run under Valgrind or ASAN; the harness itself will be compiled with the same sanitizer flags.

---

3. Build System (Makefile)

3.1 Build Targets

make

# Shared library variants
libframework.so:         ... $(CC) -shared -O2 ...
libframework-debug.so:   ... $(CC) -shared -O0 -g ...
libframework-asan.so:    ... $(CC) -shared -O1 -fsanitize=address ...
libframework-ubsan.so:   ... $(CC) -shared -O1 -fsanitize=undefined ...
libframework-gcov.so:    ... $(CC) -shared -O0 -fprofile-arcs -ftest-coverage ...

# Test harness (links against the default production library)
testharness: testharness.c
	$(CC) -o $@ testharness.c -L. -lframework

# Variant‑specific harnesses (optional, or use LD_LIBRARY_PATH)
testharness-debug: testharness.c
	$(CC) -o $@ testharness.c -L. -lframework-debug

# ... similarly for asan, ubsan, gcov

Alternatively, use one harness binary and set LD_LIBRARY_PATH to point to the desired .so at runtime.

3.2 Test Execution

A top‑level make test target runs all variants:

make

test: all
	@echo "Running tests against all library variants..."
	@$(MAKE) -s run-tests VARIANT=release
	@$(MAKE) -s run-tests VARIANT=debug
	@$(MAKE) -s run-tests VARIANT=asan
	@$(MAKE) -s run-tests VARIANT=ubsan
	@$(MAKE) -s run-tests VARIANT=gcov
	@echo "All variants passed."

run-tests:
	@LD_LIBRARY_PATH=. ./testharness-$(VARIANT) --quiet

The --quiet flag (or -q) suppresses all output except errors. The harness will print a brief summary (e.g., PASS / FAIL) per module, but only details on failure.

---

4. Test Harness Implementation (testharness.c)

4.1 Structure

c

#include <stdio.h>
#include <stdlib.h>
#include "data.h"
#include "node.h"
#include "namespace.h"
#include "sched.h"
#include "object.h"
#include "widget.h"

/* Each test function returns 0 on success, 1 on failure */
extern int test_data(void);
extern int test_node(void);
extern int test_namespace(void);
extern int test_sched(void);
extern int test_object(void);
extern int test_widget(void);

typedef struct { const char *name; int (*fn)(void); } TestEntry;

static TestEntry all_tests[] = {
    {"data",       test_data},
    {"node",       test_node},
    {"namespace",  test_namespace},
    {"sched",      test_sched},
    {"object",     test_object},
    {"widget",     test_widget},
    {NULL, NULL}
};

int main(int argc, char **argv) {
    int quiet = 0;
    if (argc > 1 && strcmp(argv[1], "--quiet") == 0)
        quiet = 1;

    int failures = 0;
    for (TestEntry *t = all_tests; t->name; t++) {
        int ret = t->fn();
        if (ret != 0) {
            if (!quiet)
                fprintf(stderr, "FAIL: %s\n", t->name);
            failures++;
        } else {
            if (!quiet)
                printf("PASS: %s\n", t->name);
        }
    }
    if (failures == 0) {
        if (!quiet) printf("All tests passed.\n");
        return 0;
    } else {
        if (!quiet) fprintf(stderr, "Total failures: %d\n", failures);
        return 1;
    }
}

4.2 Per‑Module Test Skeleton (example for data.c)

c

// test_data.c (included in testharness, or #included from testharness.c)
#include "data.h"
#include <assert.h>
#include <string.h>

int test_data(void) {
    DataNode *d = DataNew();
    DataSetInt(d, 42);
    // Check basic retrieval
    if (DataGetInt(d) != 42) return 1;
    // Check lazy conversion to string
    const char *s = DataGetString(d);
    if (strcmp(s, "42") != 0) return 1;
    // Check caching – second call should return same pointer
    const char *s2 = DataGetString(d);
    if (s2 != s) return 1;
    // Check float conversion
    double f = DataGetFloat(d);
    if (f != 42.0) return 1;
    // Clean up
    DataRelease(d);
    return 0;
}

Each test function must be self‑contained, creating its own objects and cleaning up after itself. The harness can optionally run each test under a memory checker; ASAN will catch leaks automatically.

---

5. Brutality and Output Control

• Only output on error: The harness uses --quiet to suppress pass messages. CI logs only failures, keeping noise low.

• Memory checks: ASAN/UBSAN builds will abort on error; the harness exits with non‑zero, making failure obvious.

• Coverage: After running the GCOV variant, we run gcov on all source files to produce a coverage report. The harness itself does not output coverage; it’s a separate step.

• Repeated runs: The Makefile can loop the tests multiple times (e.g., for i in $(seq 1 1000); do ...) to catch intermittent issues.

---

6. Integration with CI

• Each commit triggers a build of all variants.

• The test harness runs against each variant; if any fails, the CI build fails.

• The GCOV build is used to generate a coverage report and optionally enforce a threshold (e.g., 100% branch coverage for the core).

• The entire suite (including all variants) completes in under a minute on a modern machine.

---

7. Benefits

• No test code in production – libframework.so is pure, minimal, and fast.

• Cross‑variant validation – catches compiler‑dependent bugs, sanitizer hits, and coverage gaps.

• Silent on success – developers only see failures, encouraging frequent test runs.

• Modular – each module’s tests live separately, making it easy to add new tests.

• Memory‑safe – ASAN and Valgrind run as part of the standard test regime.

---

8. Next Steps

1. Remove all #ifdef TESTBUILD blocks from core .c files.

2. Extract each module’s test code into standalone test functions (e.g., test_data()).

3. Create testharness.c that calls all test functions.

4. Update the Makefile to build all variants and the harness.

5. Integrate the new test regime into the daily workflow and CI.

With this plan, the 200 KB core will be mathematically verified across all builds, and the test harness will serve as the ultimate guardian of correctness