Known issues

Honest caveats, workarounds, and the platform-verification status.

---

RESOLVED: runtime-thread signal mask (process-directed signal on a scheduler thread)

Status: RESOLVED. The carrier reported a process-directed SIGCHLD delivered to a libxtc scheduler thread (where MyProcPid == 0) instead of the thread the embedder designated.

Three layers:

  1. Every runtime thread is created with all signals blocked (__os_pthread_create_masked).

  2. The real residual: the ucontext coroutine substrate restores uc_sigmask on every swapcontext, and getcontext captured the CREATING thread’s mask – so a fiber created from a thread with a signal unblocked (a proc spawned from the embedder’s main thread) unblocked that signal on whatever runtime loop/worker thread later ran the fiber. (The hand-written fcontext substrate does not touch the signal mask, so it was immune – which is why the forced-fcontext CI never caught this.) Fixed in src/evt/coro_uctx.c: the fiber’s uc_sigmask blocks process-directed signals but EXEMPTS SIGVTALRM (preemption), the synchronous fault signals SIGSEGV/SIGBUS/SIGFPE/ SIGILL (which the R1 fault guard must catch – blocking a hardware-generated fault is undefined behavior anyway), and SIGABRT (assert/panic).

  3. A proc fiber’s uc_sigmask is inherited across fork(); xtc_osproc’s child path now resets its own mask to empty immediately after fork so the child (and any exec’d image) starts clean, not with the runtime mask.

Evidence: test/m1/test_thread_sigmask.c (main is the only SIGUSR1-unblocked thread; multi-loop executor + blocking pool + 200 process-directed SIGUSR1s) reports 0 deliveries to any libxtc thread, 20/20 under CPU load and in the gated parallel make check; R1 fault containment (test_proc fault_contain/fault_early_contain/ recovery_registry) passes; fork/exec (test_osproc) passes; the three preemption tests pass; ASan make check is clean. test_thread_sigmask is now in the gated make check set.

RESOLVED: xtc_exec_fini cross-thread-spawn teardown leak (LSan)

Status: RESOLVED. Surfaced 2026-07 via test/concurrency/repro_idle_uring_wake.c.

Symptom: procs spawned CROSS-THREAD (xtc_proc_spawn from an OS thread that is not on the target loop) onto a service-mode executor that is stopped mid-flight were not fully reclaimed by xtc_exec_fini – LeakSanitizer reported the task+coro (xtc_async -> __os_calloc, coro_uctx.c) as leaked, run-to-run variable (~3-7%).

Fix, in two parts:

  1. __xtc_inbox_fini now, for each undrained XTC_INB_PUBLISH message, runs the carried task’s cleanup (releasing the fiber stack + coro struct) and frees the task – exactly as xtc_loop_fini’s all_tasks walk does for drained tasks. XTC_INB_WAKE references an already-tracked task and is left untouched. This reclaimed procs left sitting in a loop’s inbox at fini (the bulk).

  2. THE RESIDUAL (the run-queue case) root cause: xtc_async stores t->cleanup only AFTER __xtc_task_spawn_ex makes the task visible (it pushes an XTC_INB_PUBLISH for a foreign spawn). A short-lived cross-thread-spawned coro can be drained, run, and reach DONE on the target loop’s thread while the spawning thread has not yet stored t->cleanup – so the DONE path read cleanup==NULL, recycled the task as a PLAIN task (freeing the task struct without releasing its coro/fiber-stack -> the coro leaked), and left the spawner’s pending t->cleanup store to land on freed memory (a latent UAF). Fixed in __xtc_loop_step: the recycle-as-plain path now also requires t->fn != __xtc_coro_step. t->fn is set before the task is published and never changes, so it is a race-free discriminator: any coro-backed task is left on all_tasks for the fini walk (which runs its by-then-stored cleanup exactly once); only genuinely plain pinned tasks recycle.

Evidence: repro_idle_uring_wake under ASan leak-detection goes to 0 leaks across 70+ consecutive runs on both epoll and io_uring (was ~3-7%); the full proc/sup/svr/tnt suites pass under ASan with no use-after-free or double-free; a drain-to-idle shutdown was already leak-clean and stays so.

Why it was not caught earlier: the leak is only visible with a WORKING LeakSanitizer – GitHub’s containerized CI runner restricts the ptrace LSan needs, so LSan silently no-ops there. The lost-wakeup guarantee the reproducer checks is covered leak-clean under DST by test/sim/test_sim_wake_park.c.

Platform runtime-verification status (1.x readiness, B2)

For an honest 1.x stance, here is exactly which platforms run at RUNTIME (the full test suite executes) versus which only COMPILE:

  • Linux (epoll + io_uring), macOS (kqueue): runtime-verified on every commit in CI.
  • FreeBSD 15 (clang, kqueue): NOW in per-commit CI via a FreeBSD VM on the Linux runner (vmactions/freebsd-vm) – full gmake check. Also re-verified by hand on the FreeBSD host (nuc, 2026-07, gmake check passes).
  • RISC-V 64 (Linux, gcc): NOW in per-commit CI via QEMU user-mode emulation (riscv64-qemu job: cross-arch build + C unit + property suites) – catches width/endian/atomics bugs. Also re-verified on a native RISC-V host (rv, 2026-07, full make check passes).
  • illumos (SunOS 5.11, UltraSPARC v9 / sparcv9 big-endian, gcc, event ports): re-verified against the tree (2026-07 host was unreachable this round; last full gmake check passed 2026-07, including the property suites on big-endian SPARC, OpenSSL 3, and the native event-port file-AIO path (SIGEV_PORT -> PORT_SOURCE_AIO) incl. /aio/roundtrip). NOT in per-commit CI – deliberately: there is no GitHub illumos runner, and running OmniOS/OpenIndiana in a nested QEMU VM on the Linux runner is slow and flaky enough that it is not worth the per-commit cost (unlike FreeBSD’s mature vmactions VM or RISC-V’s fast qemu-user). illumos stays a periodic manual re-verification (an x86_64 illumos-in-CI experiment is a possible future item, but not “easy”).
  • Windows: the IOCP runtime (AFD socket poll, cross-thread wakeup, file AIO) was RUNTIME-verified on a Windows host with MinGW (2026-06 – see “IOCP backend status” below; three bugs found and fixed). Per-commit Windows CI remains an MSVC xtc.lib + smoke build.
  • AIX (pollset): compiles, code-reviewed, no test host.

RESOLVED (partial): native stack backtrace beyond execinfo

Status: real backends added; one (Windows) remains compiled-but-not- runtime-verified for lack of a Windows host.

Previous issue: src/os/os_backtrace.c had only the execinfo backend (glibc/macOS/BSD) and a no-op stub everywhere else, so musl and Windows got an empty C backtrace in xtc_dump / the panic+crash handler.

What changed this round:

  • libunwind backend (XTC_HAVE_LIBUNWIND) for libc’s lacking <execinfo.h> (notably musl): __os_backtrace walks the calling thread with unw_step; __os_backtrace_emit symbolizes best-effort with dladdr (XTC_HAVE_DLADDR) or prints raw addresses otherwise. Detected by dist/configure.ac (--with-libunwind=auto|yes|no|PATH, OPTIONAL); execinfo wins when both are present.
  • DbgHelp backend (_WIN32): CaptureStackBackTrace + SymFromAddr, linked with -ldbghelp.
  • New test test/m12/test_backtrace.c drives a 3-deep call chain, asserts >= 2 captured frames, and (on a symbolizing platform) requires at least one of its own frames to be named.

Verified on this round’s Linux glibc x86_64 host:

  • execinfo path: full make check for m1/m12 green, test_backtrace green, the existing test_dump panic/crash path green.
  • libunwind+dladdr path: built by forcing XTC_HAVE_LIBUNWIND and linking the system libunwind; produced a REAL symbolized trace and passed test_backtrace.
  • libunwind addresses-only path (dladdr disabled): test_backtrace passes on frame count alone.
  • All three Unix variants compile -Wall -Wextra -Wpedantic clean.

NOT runtime-verified (honest gaps):

  • The libunwind path has not run on an actual musl host yet, only by forcing the backend on glibc.
  • The Windows DbgHelp path is COMPILED-NOT-RUNTIME-VERIFIED: it cross-compiles clean with mingw-w64 and links with -ldbghelp, but no Windows machine in the CI matrix exercises it. Reviewed against the Win32 DbgHelp API docs; same status as src/io/io_aix.c.
  • Platforms with neither execinfo nor libunwind still get the honest no-op stub (__os_backtrace_supported() returns 0).

See docs/guide/debugging.md for the per-platform symbolization matrix.

RESOLVED: xtc_slab SHARED_MEMORY mode cross-process support

Status: FIXED in this round.

Previous issue: The SHARED_MEMORY mode tests used MAP_PRIVATE | MAP_ANONYMOUS, which is single-process memory, not actual shared memory. Additionally, the slab’s shm_cursor was stored in per-process private memory, so two processes with their own xtc_slab_t structs would each start carving chunks at offset 0, causing collisions.

Fix: The cursor now lives in a 64-byte header at the start of the shared region, using atomic CAS for coordination. First attacher initializes the header (magic=0x5854435F534C4142 “XTC_SLAB”, version=1, cursor=64); subsequent attachers verify magic and use the existing cursor.

Verification: New test file test/m11/test_slab_shm.c exercises real cross-process sharing via fork(2) + POSIX shm_open:

  • test_shm_basic_fork: parent allocs, child reads+modifies via offset
  • test_shm_alloc_in_child: child allocs, parent reads via offset
  • test_shm_concurrent_alloc: 50 concurrent allocs from each process, no overlap
  • test_shm_size_too_small: XTC_E_RESOURCE when region < header+chunk
  • test_shm_resolve_invalid_offset: NULL on junk offsets

The previous misleading tests have been renamed to clarify their scope:

  • test_slab.c: test_shm_offset_resolve -> test_shm_offset_resolve_single_process
  • pbt_slab.c: prop_shm_offset_roundtrip -> prop_shm_offset_roundtrip_single_process

RESOLVED: Windows fault containment (SEH) + fiber teardown double-free

Status: resolved. xtc_fault_guard_install was a no-op on Windows; it now installs a Vectored Exception Handler that contains a fiber-attributable hardware fault by restoring the CONTEXT captured at xtc_proc_recovery_arm() (via EXCEPTION_CONTINUE_EXECUTION – no stack unwinding, which is what makes it safe on a fiber stack; a longjmp driven from a VEH reliably corrupted the CRT heap). Runtime-verified on a Windows host (santorini, MinGW gcc) by a dedicated driver: a proc arms recovery, registers resources, and takes a real access violation – the VEH contains it (exit 0), the recovery resource registry releases the proc’s fd + callback automatically, and the monitor observes DOWN(reason). A companion driver confirms the ESCALATION half of the contract: a fault inside a critical section is NOT contained – the VEH returns EXCEPTION_CONTINUE_SEARCH and the process dies with 0xC0000005 (EXCEPTION_ACCESS_VIOLATION), preserving the PG critical-section PANIC semantics. While wiring this up, a pre-existing Windows double-free was found and fixed in coro_winfiber.c (the done branch destroyed the coro eagerly and via the task cleanup at loop_fini), which had made any loop+process tear down with heap corruption on Windows.

Windows: test_proc (incl. selective_receive) – RESOLVED, now GATED

Status: RESOLVED (2026-09-09). The full munit m8/test_proc now BUILDS AND RUNS under MSVC and is part of the dist/build_msvc.bat hard gate: 18 of 18 cases pass, 1 skips (/fault_escalate only, which needs fork()). /selective_receive – the historically suspect case – passes. Verified interactively on an EC2 Windows Server 2022 x86_64 host (c7i.4xlarge, MSVC 19.44.35228), not inferred from CI.

The earlier note said “the full munit test_proc cannot build under cl.exe (munit uses GCC-isms)” and treated the smoke test as the only possible Windows guard. That was WRONG on the cause: munit itself builds fine (the MUNIT_ARRAY_PARAM VLA fix landed long ago). What actually blocked it was FOUR test-side portability defects, all fixed:

  1. #include <sys/wait.h> at file scope, needed only by the one fork()-based case. Now _WIN32-guarded, and that single case returns MUNIT_SKIP on Windows (its escalation half is covered by the SEH driver – see “Windows fault containment (SEH)” above).
  2. THREE hand-rolled __attribute__((packed)) mirror structs for the DOWN / EXIT signal layouts. The tree already has the portable XTC_PACK_PUSH / XTC_PACKED / XTC_PACK_POP trio for exactly this (MSVC needs #pragma pack, which an attribute cannot express); the test now uses it.
  3. clock_gettime(CLOCK_PROCESS_CPUTIME_ID) in /recv_inf_parks (the “a parked proc burns no CPU” proof). Windows DOES have a clean equivalent – GetProcessTimes (kernel+user) – so the case now runs on Windows instead of being written off as unportable. It is worth more there than elsewhere: the IOCP backend’s 8 ms AFD repoll sweep is exactly the kind of thing a regression could turn into a spin, and this case would catch it.
  4. An LLP64 truncation: 8L * 1024 * 1024 * 1024 overflows a 32-bit MSVC long (C4307). Now 8LL. Same class as the documented test_blocking _Atomic long bug.

Two REAL bugs were found by getting the test to run – one in the library, one in the test harness. Both are documented next.

Windows: contained-fault DOWN reason was a raw EXCEPTION_* code (LIBRARY BUG, FIXED)

Status: FIXED (2026-09-09). Found by running m8/test_proc on a real Windows host for the first time: /fault_early_contain asserted s.reason == 11 against an actual value of -1073741819.

Root cause: the SEH vectored handler (__xtc_veh in src/ptc/proc.c) stored the raw Win32 exception code into p->fault_sig:

p->fault_sig = (int)code;       /* EXCEPTION_ACCESS_VIOLATION */

EXCEPTION_ACCESS_VIOLATION is 0xC0000005, i.e. -1073741819 as a signed int. That value became the proc’s exit_reason and was delivered to every monitor as the DOWN reason. It broke three documented promises at once (src/inc/xtc_proc.h): the reason is specified to be a POSITIVE signal number (“e.g. 11 for SIGSEGV”), to sit inside 1..255, and to be distinguishable from XTC_DOWN_NOPROC (-100000) precisely so a supervisor can tell a contained fault from “target already gone”. A negative, out-of-range, platform-specific value satisfies none of them – so a supervisor written to the documented contract misclassified every contained Windows fault.

Fix: a __xtc_veh_code_to_signo() mapping applied where the code is recorded, so Windows reports the same signal numbers POSIX does: access violation / misalignment / in-page error -> SIGSEGV, the two divide-by-zero codes -> SIGFPE, illegal / privileged instruction -> SIGILL. (SIGBUS does not exist on Windows, hence misalignment folding into SIGSEGV, which is what a POSIX kernel reports for the same fault on the architectures libxtc targets.)

Evidence: /m8/proc/fault_early_contain went from FAIL (-1073741819) to OK (reason 11) on the host; /fault_contain and /recovery_registry (whose reasons flow through the same field) pass; the POSIX path is untouched (19 of 19 on Linux; the change is inside #if defined(_WIN32)).

Windows: close(fd) == -1 as a “was it closed?” probe is FATAL (TEST-HARNESS BUG, FIXED)

Status: FIXED (2026-09-09). /m8/proc/recovery_registry did not merely fail on Windows – it killed the whole test binary with 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN), so no later case ran.

Root cause: the TEST, not the library. The library closed the tracked fd correctly; the test proved it had, with the POSIX idiom

munit_assert_int(close(g_rec.fd), ==, -1);   /* expect EBADF */

On Windows the MSVC CRT’s _close() on an unused descriptor invokes the invalid-parameter handler, whose default action is __fastfail(FAST_FAIL_INVALID_ARG): the process dies immediately and no assertion is ever evaluated. Isolated with a 15-line standalone program (no libxtc linked): a bare double close() aborts with 0xC0000409 every time. _get_osfhandle() on a closed fd was measured to be JUST AS FATAL, so it is not a safe alternative.

Fix: test/include/fd_probe_compat.h provides xtc_test_fd_is_closed(fd) – on POSIX a plain close(fd) == -1, on Windows the same _close() bracketed by a no-op thread-local invalid-parameter handler (_set_thread_local_invalid_parameter_handler with a REAL no-op function; passing NULL restores the fatal default), which was measured to make the CRT return -1/EBADF exactly as POSIX does. A general Windows test hazard, so it lives in test/include/ rather than inside one test.

Windows: tnt/test_tnt all-zero counters – NOT A BUG; the module is compiled out

Status: RESOLVED as a DOCUMENTATION ERROR (2026-09-09). The previous entry called this “a real cross-shard-wake bug still under investigation… likely the self-wake pipe’s interaction with the IOCP readiness model.” That diagnosis was wrong in the strongest possible way: there is no tnt code on Windows to have a bug in.

src/orc/tnt.c is wrapped in #if !defined(_WIN32); the Windows half of the file (from the #else) is nothing but XTC_E_NOSYS stubs, with the in-source comment “tnt is a POSIX feature (raw socket I/O in the couriers)”. There is no self-wake pipe, no shard scheduler, and no cross-shard sender in a Windows build.

Verified, not reasoned: a standalone probe on the Windows host prints xtc_tnt_start rc=-3 (XTC_E_NOSYS). The all-zero counters were the arithmetic consequence of a runtime that was never compiled in – exactly what a NOSYS stub is supposed to produce.

Two further corrections to the record:

  • test_tnt had ALREADY grown a rc == XTC_E_NOSYS -> return 77 (SKIP) branch in commit e41f128 (2026-08-03), so the “all-zero counters” symptom the entry described was already gone before this investigation; the entry had simply gone stale.
  • But that branch was UNREACHABLE on Windows, because a later commit (53e6ea1, 2026-09-05) added #include <sys/socket.h> at file scope for the socketpair-staged echo scenario – so test_tnt did not even COMPILE under MSVC (C1083: Cannot open include file: 'sys/socket.h') and the SKIP could never be reported.

Fix: the POSIX-only body of test_tnt.c is now _WIN32-guarded as a whole, with a small Windows main() that prints the SKIP and returns

  1. Deciding it at COMPILE time is what lets test_tnt join the MSVC gate at all: the Windows translation unit never references <sys/socket.h>, so it builds, and it reports its status honestly instead of failing to compile. tnt/test_tnt is now IN dist/build_msvc.bat (as a standalone, non-munit driver – it has its own main() and no munit.c) and reports SKIP. On Linux it is unchanged and still passes the full scenario.

IOCP backend status (Windows)

Status: round-2 native rewrite; the smoke test now runtime-verifies more of it on santorini (VS18, ARM64). As of 2026-07 the MSVC smoke gate passes: version, strerror, clocks, slab, lwlock, SEH fault containment, xtc_fs, selective_receive (IOCP wakeup ordering), single-op AND multi-op native file AIO (overlapped pwrite/pread reaped from the port), and cross-thread wakeup coalescing (256 foreign xtc_send via PostQueuedCompletionStatus) all pass on the real host.

ONE scenario was a SKIP through v1.20.1 and is now FIXED (with a documented workaround, not a root-cause fix in the AFD driver itself – see item 4 below): the loopback-socket connect/accept/echo over the AFD poll path (smoke_sock_server / smoke_sock_client) now passes on an EC2 Windows Server 2022 host, verified across 5+ consecutive smoke runs, a ASan build (7/8 clean; the 1/8 failure is a pre-existing, unrelated flake documented separately below), and a dedicated CPU-scale test at 1,000/2,000/5,000/10,000 idle pending sockets (0.00% measured CPU at every size).

Status note (2026-07): substantial progress on an MSVC Windows host (Windows Server 2022, EC2); SEVEN real bugs found and fixed across two work sessions (three in the IOCP round below, plus four more – poll-timeout, pthread retval, thread affinity, slab mmap zero-fill – in the full test-surface sweep; see the sweep update further down and docs/M_WINDOWS_MATRIX.md); the one then-remaining item (tnt/test_tnt) turned out not to be a bug at all – see the tnt entry above – and a EIGHTH real bug (the raw EXCEPTION_* DOWN reason) was found in 2026-09 by getting m8/test_proc to run on the host:

  1. FIXED – the AFD poll IOCTL code was wrong. It was built as CTL_CODE(0x12, 9, METHOD_BUFFERED, FILE_ANY_ACCESS) = 0x00120024; the AFD driver rejected every poll with STATUS_INVALID_DEVICE_REQUEST (0xC0000010). The value the kernel accepts is the wepoll/libuv literal 0x00012024. After the fix NtDeviceIoControlFile returns STATUS_SUCCESS/PENDING.
  2. FIXED – a stale-GetLastError loop-killer. On an empty 0 ms poll, GetQueuedCompletionStatusEx returns FALSE with n_done == 0 but GetLastError can carry a STALE code (e.g. ERROR_ALREADY_EXISTS 183) rather than WAIT_TIMEOUT, so xtc_io_poll wrongly returned XTC_E_INTERNAL and killed the loop. Now treats n_done == 0 as the benign timeout it is.
  3. FIXED – synchronous AFD completions. When the fd is already ready the poll IOCTL returns STATUS_SUCCESS and does NOT queue a port completion, so accept-ready / connect-complete were never reaped. The arm now posts a self-completion for the synchronous case; a reg->pending guard makes a duplicate a no-op. Accept and connect readiness now flow (verified: the server accepts, the client connects + sends).

  4. WORKED AROUND (2026-07-13) – the ASYNC (STATUS_PENDING) AFD poll does not complete when the socket LATER becomes ready. Root cause confirmed with a standalone reproducer that has NO libxtc code in it: a ~150-line program opens \Device\Afd directly, arms one async IOCTL_AFD_POLL for AFD_POLL_RECEIVE on a not-yet-readable accepted socket, sends from the client, and waits on the port – the poll returns STATUS_PENDING and NEVER completes, even though the socket is genuinely readable afterward. A follow-up experiment proved AFD CAN always answer with CURRENT readiness synchronously (arming a poll on a socket that already has data waiting returns STATUS_SUCCESS with the correct event bits, reliably, every time) but never notifies of a FUTURE readiness change through this driver/version’s \Device\Afd handle. Cancelling a stuck pending poll does not help either: NtCancelIoFileEx finalizes the IRP’s IOSB in-process (STATUS_CANCELLED) but, like every other synchronous AFD outcome on this driver, does not post a port completion. This rules out libxtc’s IOCP reap loop, registration lifetime, base-handle resolution (SIO_BSP_HANDLE_POLL resolves correctly and identically every time), and the IOSB/OVERLAPPED aliasing as causes – the defect is in the AFD driver’s future-edge tracking for this handle, upstream of anything libxtc controls.

    The workaround (__xtc_iocp_repoll_sweep in src/io/io_iocp.c): xtc_io_poll periodically re-checks every PENDING registration older than XTC_IOCP_REPOLL_NS (8 ms) with a single BATCHED, zero-timeout, throwaway AFD poll covering up to 64 overdue sockets per syscall (confirmed by direct experiment: a second, independent poll on a socket that already has its own long-pending poll outstanding does not disturb it, and Timeout=0 always resolves synchronously with the true current state). Only the sockets the probe actually flags ready are canceled and re-armed on their real registration OVERLAPPED, which self-posts through the existing synchronous-arm path. A first per-socket (not batched) version of this workaround measured 50%, 90%, and 98% CPU with 1,000, 2,000, and 5,000 idle pending sockets respectively (one cancel+rearm syscall pair per socket per interval, regardless of readiness); the batched redesign measured 0.00% CPU at 1,000/2,000/5,000/10,000 idle pending sockets (one cheap syscall per up-to-64 sockets when nothing is ready).

    Latency caveat (measured, not theoretical): GetQueuedCompletionStatusEx’s millisecond timeout is quantized to the process’s current Windows timer resolution, which defaults to ~15.6 ms (the classic 64 Hz system tick) unless something has raised it. On the EC2 host used for this work the observed worst-case echo latency was ~15.6 ms regardless of the 8 ms constant; calling timeBeginPeriod(1) (from ANY process on the system, not necessarily this one) measured ~3-5 ms instead. libxtc does not call timeBeginPeriod itself – it would raise the SYSTEM’s timer resolution for as long as this process runs, a global side effect this project does not impose on an embedder’s behalf (the same reasoning as not pinning threads to cores). An application that wants tighter Windows socket-readiness latency than the default tick can call timeBeginPeriod(1) itself.

    This is a workaround, not a fix to the AFD driver’s own behavior – if a future Windows version or driver starts posting the completion correctly, the sweep still works (it only ever refreshes sockets that are ALREADY overdue; a socket whose real completion arrives promptly never reaches the sweep). Full repro harness, the batching experiments, and the AWS/SSH recipe are recorded in .agent/AFD_ASYNC_COMPLETION_2026-07.md.

Update (2026-07, full MSVC test-surface sweep, EC2 Windows Server 2022): every TESTS_C test was built + run individually under cl.exe 2022 and the Windows gate grew from 16 to a 100-test HARD GATE (build_msvc.bat step 5, all-pass, 0 warnings under /WX; per-test matrix in docs/M_WINDOWS_MATRIX.md). Four more real bugs were found and fixed on the host, plus the poll-timeout bug below:

  1. FIXED – xtc_io_poll ignored its caller timeout across AFD repoll slices. It capped the wait to the 8 ms sweep interval and returned after a SINGLE GetQueuedCompletionStatusEx slice, so a poll armed while the socket was not yet readable (bug 4 above) returned XTC_OK with zero events even though the caller asked to block up to timeout_ns and data had since arrived. test/m2/test_io_events E1/E3/E5 (readable/HUP/many-ready) caught it: writable fired (socket immediately writable) but readable never did. Fix: a deadline-bounded loop that sweeps + re-arms between 8 ms slices until an event/wakeup is emitted or the caller’s real deadline elapses (src/io/io_iocp.c).
  2. FIXED – the pthread compat shim discarded the thread return value (m1/test_thread T1/T2); see docs/M_WINDOWS_MATRIX.md.
  3. FIXED – __os_thread_set_affinity was XTC_E_NOSYS on Windows (m1/test_cpu); now SetThreadAffinityMask.
  4. FIXED – test_slab’s Windows mmap shim used malloc, not calloc, so the shm slab version-check tripped on a reused heap block (m11/test_slab shm_reclaim); plus test_blocking stored a 64-bit clock in _Atomic long (LLP64 truncation).

MSVC ASan: pre-existing, rare thread-startup flake (not the AFD workaround)

Status: KNOWN FLAKE, pre-existing, unrelated to any libxtc code. While validating the AFD workaround above under an MSVC /fsanitize=address build (2026-07-13, EC2 Windows Server 2022), the smoke test’s overall pass rate was 7/8 clean runs; the 1/8 failure reproduces identically with EVERY libxtc change reverted (confirmed by building and running the unmodified pre-workaround baseline the same number of times, which flaked at the same ~1-in-6-to-8 rate with the identical signature):

AddressSanitizer: stack-buffer-overflow ... WRITE of size 360
    #0 _asan_wrap_memset (clang_rt.asan_dynamic-x86_64.dll)
    #1 RtlInitializeResource+0x759 (ntdll.dll)
    #2 BaseThreadInitThunk+0xf (KERNEL32.DLL)
    #3 RtlUserThreadStart+0x2a (ntdll.dll)

The fault is inside RtlInitializeResource, called from a brand-new OS thread’s startup (BaseThreadInitThunk -> RtlUserThreadStart), before any libxtc code on that thread runs – ASan’s own __asan_handle_no_return/stack-switch interposition on Windows fibers/threads is a documented rough edge (https://github.com/google/sanitizers/issues/189, which the smoke test’s own “ASan is ignoring requested __asan_handle_no_return” warning already references). Always occurs, when it occurs, AFTER the loopback socket echo check has already printed ok – it is a teardown/thread-creation-path artifact, not a data-path bug. Non-ASan MSVC builds (the /WX zero-warning gate) and the plain smoke run never exhibit it. Tracked as a documented flake, not a release blocker; re-running the smoke binary passes.

Round 2 (current source): native completion port + AFD poll

src/io/io_iocp.c was rewritten from the round-1 readiness emulation to a native completion-port design (details in docs/M_WINDOWS_MATRIX.md):

  • CreateIoCompletionPort + GetQueuedCompletionStatusEx is the only wait primitive; the round-1 64-handle WaitForMultipleObjects cap is GONE.
  • Socket readiness uses the AFD poll fast path (\Device\Afd + IOCTL_AFD_POLL via NtDeviceIoControlFile), re-armed per completion (level-triggered). This is the wepoll/libuv design.
  • Wakeup is PostQueuedCompletionStatus; file AIO is overlapped ReadFile/WriteFile reaped from the same port (no hEvent).
  • OVERLAPPED-ownership rule enforced: the kernel owns an OVERLAPPED from request-accept to completion-dequeue; deregister cancels with NtCancelIoFileEx and defers the free to the reap; registration nodes have stable heap addresses so the kernel-held back-pointer never dangles. dist/configure.ac adds -lntdll; dist/build_msvc.bat links ntdll.lib.

What is verified (Linux dev host this round): cross-compiles clean with mingw-w64 gcc 14.3.0 -std=c11 -Wall -Wextra, links into a PE32+ binary against -lntdll -lws2_32, and leaves the Linux build untouched (io_iocp.c is XTC_IO_BACKEND_IOCP-only – an empty TU on Linux; the full C munit suite stays green on epoll/uring).

What is NOT verified: the backend has NOT executed on Windows. The AFD poll correctness, the level-triggered re-arm (no busy-loop, no dropped edges), the cancel/lifetime under churn (no double-free, no freeing a kernel-owned OVERLAPPED), the wakeup coalescing/ordering, and the file-AIO port round-trip must all be validated on santorini before this is production quality. Same reviewed-but-untested status as src/io/io_aix.c. This rewrite is the suspected fix territory for the test_proc::selective_receive flake (the wakeup ordering changed), but that cannot be confirmed without the host.

RUNTIME-VERIFIED (2026-08, EC2 Windows Server 2022, c5.2xlarge, MSVC 2022 Build Tools): the four IOCP-runtime smoke scenarios below ALL PASS on real Windows – native IOCP file AIO (8 overlapped pwrite/pread at distinct offsets), cross-thread wakeup (256 foreign xtc_send via PostQueuedCompletionStatus coalescing), loopback socket echo via AFD poll (level-triggered re-arm), and the AFD repoll-sweep scale (256 idle pending sockets, batched probe, not O(n)/socket). Additionally the MSVC munit subset (16 tests across m0/m1/m10/m11/m14) builds and passes 16/16 (after fixing the per-milestone munit git-symlink checkout on Windows – see dist/build_msvc.bat), and the DbgHelp backtrace backend is runtime-verified (test/m12/test_backtrace_win.c: 5 frames captured, SymFromAddr resolved a known function name). The historical “awaiting a santorini run” text is kept below for provenance.

(historical) Smoke coverage WRITTEN, awaiting a santorini run (NOT yet verified): test/msvc/smoke.c now drives four IOCP-runtime scenarios beyond the already-verified strerror/clocks/slab/lwlock/fault-containment set. They COMPILE (cross-checked with mingw-w64 gcc 14.3.0 -std=c11 -Wall -Wextra -fsyntax-only against the real Windows headers, with the MSVC pthread shim forced into its _MSC_VER path) but have NOT been built or run under cl.exe on a Windows host:

  • smoke_aio_proc (pre-existing): a single overlapped pwrite+pread round-trip at offset 0 – the file-AIO port round-trip.
  • smoke_aio_multi_proc (new): several positioned overlapped pwrite/pread ops at distinct offsets in one loop run, so more than one AIO completion is reaped from the port per run (the reap loop, not just a single completion).
  • smoke_xt_worker + smoke_xt_sender (new): a foreign OS thread bursts xtc_send at a batch of parked worker procs, exercising the cross-thread PostQueuedCompletionStatus wakeup and its coalescing (__xtc_io_iocp_wakeup_post); the loop must reap every delivery and return once all workers exit.
  • smoke_sock_server + smoke_sock_client (new): a 127.0.0.1 connect/accept/echo driven entirely by xtc_proc_wait_fd on the raw Winsock socket fds – the AFD poll fast path and its level-triggered re-arm (WRITABLE for connect completion, READABLE for accept, then READABLE/WRITABLE for the echo). Send/recv is raw Winsock in the test so its error handling does not depend on any errno mapping in the library net helpers; if a listen port cannot be bound on the runner the scenario SKIPS rather than fails.

These are the runtime scenarios the “What is NOT verified” list above names; the smoke test is the intended Windows regression guard for them once santorini runs dist/build_msvc.bat and reports the ok lines. Until that run they remain COMPILED-NOT-RUNTIME-VERIFIED.

Round 1 (historical): WSAEventSelect emulation – SUPERSEDED

The round-1 backend (WSAEventSelect + WaitForMultipleObjects, hard-capped at 64 handles, ~60% of native IOCP throughput) passed the full suite on the reference Windows toolchain (MinGW64: 233/233). Those results describe the SUPERSEDED source, not the round-2 rewrite above; they are retained here only as the baseline the round-2 code must re-establish on santorini. Round-1 notes that still stand:

  • Clang64 POSIX-only test ports. test_net_udp used a bare nanosleep (absent in the Clang64/MinGW runtime) – now portable (a test_msleep shim). test_proc_wait_fd still uses pipe(2) + clock_gettime(CLOCK_PROCESS_CPUTIME_ID) + pthreads; porting it to the socket-pipe compat is straightforward but must be verified on the host before landing (an unverifiable Windows edit risks a silent Linux regression), so it is deferred to a santorini pass.
  • test_proc::selective_receive flake (above): the cooperative equivalent (test/otp/test_otp_proc_lib.c) passes on every platform, so selective receive itself is correct; the flake is in test_proc’s exact IOCP-wakeup timing and needs the host to chase.

Native file AIO (mechanism): xtc_io_aio_submit issues overlapped ReadFile/WriteFile; fsync/fdatasync have no async form on Windows (FlushFileBuffers is synchronous) and are offloaded. Round 1 joined the OVERLAPPED.hEvent to the WaitForMultipleObjects set and was verified on santorini under VS2022 (17) and VS2026 (18) by the smoke test (an overlapped pwrite+pread round-trip). Round 2 instead reaps the AIO completion from the port (no hEvent); that variant is COMPILED-NOT-RUNTIME-VERIFIED and the smoke test must be re-run.

Driving the santorini host non-interactively from CI/automation is not yet wired (it is configured for an interactive PowerShell session); the Windows matrix is run by hand via dist/santorini-matrix.sh.

Async file I/O: backend coverage

xtc_aio_pread/pwrite/fsync/fdatasync present one portable API; the native completion mechanism is per backend, with a blocking-pool offload everywhere else (always correct, just thread-backed). Native completion is implemented and validated on:

  • io_uring (Linux): IORING_OP_READ/WRITE/FSYNC, reaped from the CQE ring. CI build-and-test + ASan/UBSan.
  • IOCP (Windows): overlapped ReadFile/WriteFile reaped from the completion port; fsync offloaded. santorini smoke test.

  • kqueue (FreeBSD, macOS): POSIX AIO (aio_read/aio_write/ aio_fsync) with SIGEV_KEVENT -> EVFILT_AIO. CI macos job compiles and runs test_aio against it (green).

  • illumos / Solaris event ports (SunOS 5.11): POSIX AIO (aio_read/aio_write/aio_fsync) with SIGEV_PORT posting the completion as a PORT_SOURCE_AIO event on the loop’s existing event port, reaped in the port_getn drain (aio_return -> publish a->res -> wake a->tag). Implemented in src/io/io_solaris.c (xtc_io_aio_submit + struct sol_aio); RUNTIME-verified on the illumos/OpenIndiana host (sun) – full gmake check passes, including the native /aio/roundtrip case.

Still offloaded to the blocking pool (native AIO is a follow-up with platform-specific completion plumbing – they are DIFFERENT mechanisms, not one shared path):

  • epoll / poll / select (Linux without io_uring, generic POSIX): would need POSIX AIO with SIGEV_SIGNAL delivered to a self-pipe / signalfd the loop already watches (epoll has no AIO filter). A new shared io_posixaio.c submodule.
  • AIX: AIX aio_* (or the legacy LIO interface).

The offload fallback makes “write storage code once as if AIO is always available” hold on all of these today; the follow-ups only remove the thread hop.

Decision (updated 2026-07): illumos native AIO is now DONE and verified; epoll/poll/select and AIX stay offload-backed deliberately. Assessment of the remaining targets:

  • Linux without io_uring (epoll/poll/select): the only ZERO-THREAD clean form is libaio (io_submit + an eventfd via IOCB_FLAG_RESFD), which adds a dependency; POSIX AIO with SIGEV_SIGNAL is signal-safety-hazardous and SIGEV_THREAD reintroduces the very thread hop this would remove. On Linux the zero-thread answer already exists and is native: io_uring. So this target has near-zero value.
  • illumos event ports (SIGEV_PORT -> PORT_SOURCE_AIO on the existing port): DONE. It was implemented WITH the illumos host in the loop (the bring-up did surface an EINVAL from a stack-local port_notify_t; fixed by heap-allocating the notify+aiocb in struct sol_aio so they outlive the async call) and now passes the full suite on sun.
  • AIX aio_*: no test host at all; remains offload-backed. The offload path is CORRECT and COMPLETE on the still-offloaded targets (native AIO is a pure performance optimisation that removes a thread hop, not a correctness gap).

Portable block-device I/O layer – DONE (v1.4.0)

Implemented as xtc_bdev (src/io/io_bdev.c, xtc_bdev.h): open a raw device / partition (or a regular file), query logical+physical sector size and capacity via the native per-OS ioctl (BLKSSZGET/BLKGETSIZE64, DIOCGSECTORSIZE/DIOCGMEDIASIZE, DKIOCGMEDIAINFO, the Windows drive geometry IOCTL) with a fstat fallback, aligned pread/pwrite through the xtc_aio path, and flush via xtc_aio_fsync. See xtc_bdev(3).

test_alloc M7 – NOW ENABLED on Windows (the skip was stale)

Status: RESOLVED (2026-09-09); the documented reason was STALE. The previous entry read: “intentional – _aligned_malloc returns memory that requires _aligned_free, not plain free. The hook surface uses a single free path. Keeping the M7 case Windows-skipped is correct.”

That premise stopped being true when commit d10c257 (“os/alloc: fix UBSan misalignment on cache-line-aligned allocations”) gave the allocator vtable a MATCHED aligned() / aligned_free() PAIR. There is no longer a single free path: __os_aligned_free() exists, routes to _aligned_free on Windows and free elsewhere, and src/inc/os_alloc.h documents the two as a pair that MUST be used together. The AGENTS.md over-aligned-allocation rule depends on that pair existing.

What actually kept M7 unrunnable on Windows was that the TEST released with the WRONG half – __os_free(q) on memory from __os_aligned_alloc(), which is exactly the mismatched-free bug the pair exists to prevent and which would corrupt the heap on Windows. So the skip was masking a defect IN THE TEST, and the case as written could never have been simply un-skipped.

Fix: M7 releases with __os_aligned_free() (the documented pair) and runs on every platform. It also gained the case the AGENTS.md rule actually cares about – an XTC_CACHE_LINE alignment stricter than max_align_t – plus an aligned_free(NULL) no-op check.

Evidence: /m1/alloc/M7_aligned OK on the Windows host (test_alloc 8 of 8, 0 skipped – was 7 + 1 skip) and still OK on Linux (8 of 8).

Windows multi-core scalability: datapoint RE-MEASURED, curve reproduces

Re-measurement (2026-09-09): re-run on the same instance shape as the original datapoint (EC2 x86_64 Windows Server 2022, c7i.4xlarge, 16 vCPU, MSVC 2022, IOCP backend) with the same per_loop=30000, so it is a like-for-like comparison – on a newer source tree (v1.43.0):

loops 2026-07 (below) 2026-09 run 1 2026-09 run 2
1 77 K 55 K -
2 102 K 81 K -
4 127 K 121 K 124 K
8 91 K 97 K 100 K
16 82 K 80 K 90 K

Every run: spawn_ok == done, fail = 0. The documented shape holds – spawn throughput peaks at 4 loops and regresses past it – and the peak reproduces within ~5%. The 1- and 2-loop numbers came in lower than 2026-07 and the 8/16-loop numbers slightly higher, which narrows the peak-to-tail ratio without changing the finding; repeated runs at the top three points show a several-percent run-to-run spread, so the small deltas are not worth attributing to any source change. The 2026-07 conclusion (the ceiling is cross-loop work-steal cache thrash + OS thread placement, NOT the slab allocator) stands unmodified.

Status: MEASURED on EC2 x8664 Windows Server 2022 (c7i.4xlarge, 16 vCPU, MSVC 2022, IOCP backend) via bench/bench_win_scale.c – a portable spawn-throughput probe buildable under cl.exe (public xtc* API + xtc_clock_mono only, no POSIX headers). The Win32-fiber substrate corruption blocker is fixed (above), so these numbers are on a sound runtime.

Spawn throughput (spawn + immediate-exit child, per-loop drivers, per_loop=30000, all runs done==total, fail=0):

loops= 1   77 K spawns/s
loops= 2  102 K spawns/s
loops= 4  127 K spawns/s   <- peak
loops= 8   91 K spawns/s
loops=16   82 K spawns/s

Two honest findings:

  1. Spawn throughput peaks at 4 loops and regresses past it (as measured). This is NOT the slab allocator: the FlsAlloc fiber-local magazine (below) is validated engaging (a slab alloc/free microbench on the MSVC host shows fast=2,000,000 slow=256 – every op hits the fast path – at 26.3 M ops/s), and re-measuring the spawn curve WITH the magazine active gives the SAME shape (74K->129K peak at 4 loops ->83K at 16). So the spawn bottleneck past 4 loops is cross-loop work-steal cache thrash + OS thread placement (the same NUMA/steal placement characteristic documented for the Linux reuse benchmark), not allocation. All runs correct (spawn_ok==done, fail=0). The slab magazine is now FlsAlloc-backed on Windows and VALIDATED on an MSVC host (Windows Server 2022, c7i.4xlarge): FlsAlloc/ FlsGetValue/FlsSetValue give each fiber its own tls_mag array that correctly follows the fiber across SwitchToFiber, unlike the __declspec(thread) static TLS (the original corruption cause); the FlsAlloc callback frees it on fiber exit. The MSVC smoke (slab round-trip + the xtc_xproc DOWN-reason=42 path, which drives __mon_alloc through the slab on a fiber) passes with no crash/hang.

  2. A high-volume ceiling was found AND fixed. At per_loop=50000 x 16 loops (800 K short-lived procs) only ~422 K children ran (deterministic), because completed coros are not freed until loop_fini (see __xtc_coro_step / loop.c – true on all platforms) and each accumulated Win32 fiber COMMITTED its full 64 KiB working stack, exhausting the process commit charge (~27 GiB on a 30 GiB box). Fixed in coro_winfiber.c: CreateFiberEx now commits only a small initial 16 KiB and lets the OS grow the stack up to a >= 1 MiB reserve, so the commit charge is proportional to actual stack use; 800 K x 16 loops now completes done==total. The deeper item – eagerly freeing completed coros instead of accumulating them until loop_fini – remains a cross-platform improvement (Windows just hit the wall first because of the fiber commit).

A Seastar/Tokio comparison on Windows is now UNBLOCKED but not yet run: it wants (a) the FlsAlloc magazine so the fast path is measured, and (b) message-throughput + tasks/sec workloads added to bench_win_scale.c. POSIX scalability was measured at scale (EC2 Phase B, 192 vCPU); the Windows curve above is the first cross-platform datapoint.

xtc_xproc end-to-end spawn+monitor on Windows – RESOLVED

Status: RESOLVED (2026-07, EC2 x86_64 Windows Server 2022 + MSVC 2022). The full cross-process path now works end to end on Windows: CreateProcess re-exec + --xtc-xproc-child sentinel + loopback-TCP control connect (nonce) + dedicated reader thread + shadow-proc monitor

  • exit-latch loop wakeup deliver a normal DOWN with the child’s exit reason. The MSVC smoke test xtc_xproc: re-exec child spawn+send+monitor, DOWN reason=42 is now a HARD GATE (was a SKIP); 40/40 non-ASan runs pass and 3/3 MSVC-ASan runs are clean.

Two substrate bugs were the blocker (both fixed), found via MSVC AddressSanitizer on the box:

  1. coro_winfiber.c fiber stack was undersized. CreateFiberEx was called with dwStackCommitSize == dwStackReserveSize == 64 KiB, leaving zero growth headroom. A proc’s exit teardown (__proc_entry -> __notify_links_and_monitors -> xtc_send -> slab) plus a Win32 API callback (a ~360-byte ntdll memset seen under ASan) overflowed the fiber stack and corrupted adjacent memory (the near-NULL slab-pointer AV). Fix: reserve generously (>= 1 MiB, or 16x the committed working size) so the OS grows the stack normally, committing only the requested size.

  2. The slab per-thread magazine used __declspec(thread) (static TLS), which is unsafe under Win32 fibers. The compiler may cache the static-TLS base across a SwitchToFiber, so a magazine pointer handed to xtc_slab_alloc dereferenced a stale TLS block on a resumed fiber – traced to a hang/corruption at mag->slots[--mag->n] inside __mon_alloc (the monitor entry allocation), which is exactly the path xtc_xmonitor drives. Fix: __tls_mag_for returns NULL on Windows, so every alloc/free takes the locked slow path (touches no __declspec(thread) state, correct on a fiber). Performance-only tradeoff – a fiber-local-storage (FlsAlloc) magazine is a later optimization; Windows is not the throughput target yet.

POSIX xtc_xproc (fork/waitpid) and the fcontext/ucontext substrates are unchanged. The reader-thread control channel and the io_wakeup exit-callback (which fixed the earlier proc-table-SRWLock deadlock) are kept.

Fiber-switch sanitizer annotations enable detect_stack_use_after_return=1

Status: SHIPPED (v1.13.0 for ASan; TSan fiber-identity added v1.17.0). The fcontext and ucontext coro substrates annotate every user-space stack switch for the sanitizers, using the CORRECT, mutually-exclusive API for each:

  • ASan/LSan (XTC_FIBER_SWITCH_ANNOTATE, __SANITIZE_ADDRESS__ / __has_feature(address_sanitizer)): the stack-switch API __sanitizer_start_switch_fiber / __sanitizer_finish_switch_fiber around every switch, so ASan tracks the fiber stacks instead of mis-attributing stack memory. The full fiber runtime (test_fctx/proc/async/svr/fsm) passes ASan with detect_stack_use_after_return=1 on BOTH substrates.
  • TSan (XTC_TSAN_FIBERS, clang __has_feature(thread_sanitizer) only): the fiber-IDENTITY API – __tsan_create_fiber at coro create, __tsan_switch_to_fiber at every switch (into a coro, and back to the once-captured scheduler fiber), __tsan_destroy_fiber at teardown – so TSan carries per-fiber happens-before across cooperative switches instead of seeing one confused thread. TSan does NOT provide __sanitizer_*_switch_fiber, so the two guards are mutually exclusive (TSan decided first; a TSan build emits ONLY the identity calls, an ASan build ONLY the stack-switch calls). A full clang -fsanitize=thread libxtc build runs test_fctx/async/proc/ svr/chan with ZERO TSan warnings. clang-only by design: gcc’s libtsan has no fiber support, so a gcc TSan build emits no fiber annotations (documented; not a gcc-buildable TSan-with-fibers configuration). Requested by the PG-integration team and delivered.

Compiled to nothing in a non-sanitized build (0 sanitizer references, verified: nm libxtc.a | grep __tsan_ is empty).

CI: the shared ASan job runs with detect_stack_use_after_return=1 on every commit.

__notify_links_and_monitors DOWN-send vs proc-teardown race (RESOLVED)

Status: RESOLVED (v1.13.0) by a teardown refcount on the proc struct. Surfaced by the fiber-switch annotations under detect_stack_use_after_return=1 (which shifts scheduling enough to widen the window): when a proc exited, __notify_links_and_monitors sent a DOWN to each monitoring proc via xtc_send; a monitor proc concurrently torn down on another thread could be freed between __resolve returning its pointer and __mbox_deliver_locked touching its mailbox – a heap-use-after-free. This was the SAME root cause as the blocking-pool wake UAF and (suspected) the macOS sqlxtc MT-load flake: __resolve handed out a struct xtc_proc * after releasing the table lock, with no lifetime guarantee.

Fix: an atomic refs on struct xtc_proc. __table_lookup takes a reference WHILE HOLDING the owning table lock – atomic with the detach in the teardown path – so a resolver either pins a live proc or sees NULL, never a freed pointer. The owner (spawn) holds one ref; teardown detaches from the table then drops the owner ref, and the struct’s mailbox-drain / lock-destroy / free happen in __proc_free only when the last ref (owner + any in-flight resolver) is released. Every __resolve caller (xtc_send, xtc_exit_pid, xtc_proc_wake, the deferred-delivery callback, link/monitor push, xtc_proc_mailbox_stats) releases its ref. CI now runs the ASan job with detect_stack_use_after_return=1; validated by 5 clean full-make-check runs under ASan+SUAR (0 UAF, 0 leaks). A DST test (test_sim_proc_teardown) models the resolve-vs-exit race deterministically.

svr.c branch coverage

Status: improved this round with NULL/invalid-argument guard tests (test/coverage/test_fault_inject.c /svr/null_guards), which exercise the early-return XTC_E_INVAL edges across xtc_svr_start/stop/join/reply/ call/cast that the happy-path server tests skipped, PLUS the reply-path OOM edge (/svr/reply_oom): a new svr.reply.oom injection point at the top of xtc_svr_reply forces the non-empty-reply allocation to fail, covering the XTC_E_NOMEM return on BOTH reply branches (the synchronous slot path and the in-proc reply_pid tag+payload path) – ASan-clean, no leak on the failure path.

Targets remaining: the deeper call-after-stop timing edge (a call that races xtc_svr_stop) still needs a scheduler-ordering harness rather than a plain injection point; it is error handling, not mainline behavior.

io_common.c coverage

Status: improved this round. The four io.init.* fault-injection points (calloc/pipe/fcntl/backend fail) were already tested; added /io/null_guards (xtc_io_init/fini/wakeup reject NULL with XTC_E_INVAL) and /io/wakeup_roundtrip (the live xtc_io_wakeup post + coalesced second post) in test/coverage/test_fault_inject.c. The remaining uncovered branches are backend-specific cleanup edges in xtc_io_fini and the ENOMEM/EAGAIN corners only reachable on a particular backend; a per-backend fault-injection sweep is the way to close them further.

RESOLVED: macOS/arm64 xtc_dump() SIGBUS when called from a live fiber

Status: RESOLVED (fiber-stack-aware backtrace walker, 8034f93; fallback hardened so it can never reach the unbounded system backtrace() from a fiber).

xtc_dump() captures a C backtrace of the calling thread. On macOS/arm64, when called from inside a running fiber (a live xtc_proc body, not the top-of-thread panic/abort path), the system backtrace() walks the frame-pointer chain of the small guard-paged fiber stack and could run past the stack top into unmapped memory, raising SIGBUS – intermittently (it depended on what lay just past the fiber stack and whether that address was mapped), which is why it surfaced only occasionally in CI’s test_dump/basic.

Fix: src/os/os_backtrace.c now uses a fiber-stack-aware frame walker on Apple targets instead of the system backtrace(). It discovers the active stack’s VM region at call time from its own SP (mach_vm_region – a query of the task’s VM map, which cannot itself fault) and walks the FP chain manually, stopping the instant the next frame record would leave the mapped region, is misaligned, or does not ascend – so an over-walk past a guard-paged fiber stack top can never touch an unmapped page. No coroutine-layer coupling: the bound is derived from the running SP, so it works whether the caller is on a fiber mmap or an OS-thread stack. If the region cannot be resolved (a rare mach_vm_region failure), the walker returns NO frames rather than falling back to the unbounded system backtrace() – the dump then prints “backtrace unavailable”, which is strictly better than a SIGBUS. The symbolization step (backtrace_symbols_fd) only resolves the already-captured, in-bounds return addresses (a dladdr-style lookup, not a stack walk), so it does not fault.

Other platforms were never affected: Linux uses the EH-ABI unwinder (_Unwind_Backtrace), which terminates cleanly on a fiber stack. test_dump/basic exercises the from-a-fiber path (its dump_driver runs as an xtc_proc and calls xtc_dump), so the macOS CI runner covers the fixed path per commit.

AIX: not a supported target (off the roadmap)

Status: UNSUPPORTED / UNMAINTAINED. An AIX/ppc64 OS-layer port (os_thread/os_mutex/os_tls + an io_aix.c pollset backend + POSIX AIO offload) exists in-tree and compiles, but AIX is deliberately NOT a supported platform: it is never built or run in CI, has no runtime verification, and is off the roadmap. AIX is IBM-proprietary (ppc64-only; PostgreSQL itself dropped AIX in v16), the install media is licensed, and emulating it under QEMU/TCG is impractical for a reliable CI gate. The in-tree port is left in place (it is harmless and compiles cleanly) but carries no support commitment – treat any AIX behavior as unverified. Do not file AIX as a gap; it is a decision, not a TODO.

RESOLVED: macOS now in CI

The macos GitHub Actions job (macos-latest, Apple Silicon) builds and runs the full C munit suite every commit – kqueue backend, ucontext substrate, GCD dispatch semaphores. Standing it up fixed six real portability bugs (Darwin feature macro, rwlock storage size, unnamed semaphores, _SC_NPROCESSORS_ONLN, hardcoded -lrt, lrlock slot reclamation teardown order).

Apple Silicon (macOS arm64): now defaults to the fcontext substrate (macOS x86-64 still ucontext)

Status: RESOLVED for Apple Silicon as of 1.23.x – macOS/arm64 now defaults to the hand-written fcontext coroutine substrate, closing the per-switch performance gap. macOS x86-64 still uses ucontext (there is no Mach-O x86-64 fcontext assembly), and involuntary preemption remains Linux-only everywhere on macOS (see below).

src/os/asm/fctx_aarch64_aapcs_macho.S provides the AAPCS64 make/jump_fcontext in the Mach-O assembler dialect (leading-underscore symbols, .p2align, no ELF-only directives), byte-identical in register discipline to the ELF variant. src/evt/coro_fctx.c and src/evt/coro_uctx.c select it as the default on __APPLE__ && __aarch64__ (equivalent to -DXTC_CORO_FORCE_FCTX); build with -DXTC_CORO_FORCE_UCONTEXT to fall back to swapcontext. This removes the per-switch sigprocmask syscall (measured on Apple M-series: ~34 us/task-switch on ucontext under bench_million_tasks.c at 1M fibers -> ~23 us there, with a larger per-switch win masked by memory pressure at that scale; a 2-fiber micro-bench shows the true delta). The fcontext path on macOS declines involuntary preemption, so it falls back to Phase-1 cooperative preemption there.

Involuntary preemption (src/ptc/preempt.c, Phase 2) is STILL Linux-only: it needs a per-thread SIGVTALRM-class timer (timer_create/CLOCK_THREAD_CPUTIME_ID), which macOS does not offer (ITIMER_VIRTUAL is process-wide, not per-thread). On macOS, xtc_preempt degrades to cooperative-only: a fiber that never calls a yield point will not be involuntarily interrupted. See man/man3/xtc_preempt.3 for the per-platform statement.

Remaining: a macOS x86-64 Mach-O fcontext variant (low priority – Apple has moved to arm64), and a GCD dispatch-timer or kqueue EVFILT_TIMER per-worker tick to give Phase-2 preemption a real signal source on macOS (tracked in PLAN.md). Neither changes correctness.

sqlxtc multi-thread load test flakes on the macOS CI runner

Status: KNOWN FLAKE (transient, re-run passes). The sqlxtc multi-thread load CI step (test/sqlxtc/test_sqlxtc_mt.sh -> test_sqlxtc_concurrent.py, N clients * M queries against a shared server on a plain CREATE TABLE) intermittently fails on the macOS runner with all clients timing out and server CRASHED under multi-loop load (cores=3). Observed once on 5d78c00 and passed on re-run with no code change; the 5 preceding commits’ macOS jobs were green.

The failing path is the VDBE/connection handling under concurrent load (the test table is NOT xstore-backed, so it never reaches the vexec fast path) – so it is independent of the vexec/Track-B work. Likely the same multi-loop cross-thread timing sensitivity seen elsewhere on the macOS/kqueue runner. Treat a lone macOS MT-load failure as flaky: re-run the job (gh run rerun <id> --failed) and judge by the other 12 jobs. A persistent failure across re-runs would indicate a real concurrency regression and must be chased.

pbt_proc::send_recv_roundtrip and pbt_proc::fifo_order flake under make check

Status: RESOLVED in M11.5b. The proc registry’s __lt[] table was leaking entries on xtc_loop_fini; consecutive PBT loops were aliasing stale entries. Fix: added __xtc_proc_loop_unregister(loop) called from xtc_loop_fini. Both properties are re-enabled.

xtc_cfg: missing features

Status: Config-file parsing and reload DONE; per-session scoping is out of scope (it belongs to a downstream consumer, not the runtime).

  • Configuration-file parsing (postgresql.conf reader): DONE – xtc_cfg_load_file() reads name = value lines (comments, quotes, per-kind parsing, bounds/validators), skipping unknown/bad lines.
  • SIGHUP-driven reload: DONE as a mechanism – xtc_cfg_reload() re-reads the last loaded file. The app wires SIGHUP to it from the event loop (the function is not async-signal-safe, by documentation).
  • Per-session/per-database scoping: out of scope for xtc – it needs a session/override-stack model that is a downstream consumer’s concern, not the general-purpose runtime’s.

xtc_slab_pressure_stop API incomplete

Status: DONE.

Resolved: xtc_slab_pressure_listen_ex() returns an opaque xtc_slab_pressure_t handle and xtc_slab_pressure_stop(handle) joins the listener thread, closes its fds, and frees it. Plain xtc_slab_pressure_listen() delegates to _ex and discards the handle (unchanged fire-and-forget behaviour).

epoll backend: rare lost blocking-I/O-completion wakeup under heavy churn

Status: RESOLVED (pending continued monitoring). The primary causes were fixed earlier (see (A)/(A-residual)/(B) below); the last remaining epoll-only residual in the buffer-manager stress test appears CLOSED by the v1.8.0 cross-thread prepare/park wake fix.

The closing fix (v1.8.0): the residual was the same lost-wake class as the PostgreSQL carrier’s cross-thread fd/latch wake miss – a cross-thread wake (here, the blocking pool completing an offloaded fsync and waking the parked evictor) arriving while the target task was still RUNNING, between arming its waker and the loop transitioning it to PARKED on yield, was DROPPED by the XTC_INB_WAKE drain (which only enqueued PARKED tasks). The fix latches task->wake_pending when the task is not yet PARKED and the RUNNING->PARKED transition consumes it and re-schedules instead of parking (see src/evt/loop.c).

Evidence: test_bufmgr_mt, which historically hung ~3-7% of runs on epoll, now passes 130/130 consecutive runs on the epoll backend with zero hangs, and the reproducer test/concurrency/repro_blocking_epoll.c (which hung 8/8 at its worst) passes 10/10. test_bufmgr_mt is no longer gated off epoll: it runs on the Codeberg (epoll) CI as well as the io_uring CI. Kept in this document (rather than deleted) so the long investigation and the several rejected approaches below stay on record; if any epoll wake-miss recurs, this is the history to read first.

Historical investigation (the fixes that led here) ### (A) FIXED -- xtc_yield / xtc_await did not preserve __current_proc The primary lost-wakeup was not in the epoll backend at all. It was a fiber-context bug: the per-thread "current proc" pointer (`__current_proc`, an L3 process-layer TLS) was not preserved across a yield. When a proc calls `xtc_yield()` (or `xtc_await`) the scheduler runs OTHER procs in between -- each setting `__current_proc` to itself -- and on resume the public yield primitives did NOT restore it. Internal parks (`xtc_proc_wait_fd`, `xtc_proc_sleep`, `xtc_recv`, the amutex) each restored it by hand, but a plain `xtc_yield()` did not. A proc that yielded therefore resumed running as whatever proc ran last; its next `xtc_blocking_run` -> `wait_fd` then registered the completion fd under the WRONG task and parked the WRONG task, so the real proc never woke. Minimal reproducer: ONE loop, TWO procs, three `xtc_blocking_run` calls each -- hangs 10/10 on epoll (a trace showed proc P1, after its worker's bare `xtc_yield()`, registering its pipe fd under proc P2's task tag and "P2" parking twice without waking). io_uring masked it: its completion path re-samples readiness, so the misattributed park self-healed; epoll's `epoll_wait(-1)` blocked forever. Fix: preserve the process context across every coro yield via a hook (`__xtc_fiber_ctx_save` / `__xtc_fiber_ctx_restore`, installed by `xtc_proc_spawn`, no-op when no process layer is in use, so the L2 coro layer keeps no hard dependency on L3). Applied to `xtc_yield` and `xtc_await` in all three coro substrates (fctx, ucontext, Win32 fiber). Verified: the pure reproducer goes 0 -> 20/20 on epoll (and the N-loop/N-proc one 0 -> 20/20); io_uring `make check` + ASan + UBSan + epoll `make check` all clean; io_uring `bufmgr_mt` 50/50. ### (A-residual) FIXED -- epoll-only hang in test_bufmgr_mt was a pin underflow After the (A) fix, `test_bufmgr_mt` still hung about 3-7% on the epoll backend (io_uring far less, and confounded by host load). The earlier read of this as a benign "thrash livelock" was WRONG: it was a real pin-accounting bug in the example's from-scratch buffer manager (examples/06_sqlxtc/bufmgr.c), traced to ground with armed abort probes + post-mortem cores and now fixed at root. Root cause (swip mode only): a fixer can hold a STALE swip word `w` pointing at a frame that has since been evicted, freed, and recycled by the demand-load path for a DIFFERENT slot. The fixer does `try_pin(sw_frame(w))` and transiently bumps that recycled frame's pin (it fails its slot recheck a moment later and unpins -- net zero). But the demand-load path claimed the frame with an UNCONDITIONAL `store(pin, 1)`, which clobbered the fixer's transient increment; the fixer's matching `unpin` then drove pin to -1. A frame wedged at pin == -1 is indistinguishable from an eviction reservation, so every later fixer spins `try_pin` on it forever -- the hang. (epoll merely lost the timing lottery more often; io_uring re-samples readiness and mostly masked it.) Fixes, all in bufmgr.c, verified epoll 80/80 + io_uring 30/30 + ASan 12/12 + UBSan 8/8 clean, with the btree/xstore (pid-mode) tests green: - The swip demand-load and swip alloc paths claim a recycled frame with a CAS loop (`claim_frame`: spin CAS pin 0 -> 1, yielding), NOT a blind store, so a fixer's transient stale pin is never clobbered. Scoped to swip mode only: pid-mode (`bm_fix_pid`, the B-tree path) has no swip references, so no stale fixer exists; it keeps the plain store (a CAS-wait there would deadlock against a latch-coupling pin that cannot drain while the claimer spins). - Eviction releases a reservation with CAS(-1 -> 0), never a blind store, so it cannot clobber a concurrent loader's fresh pin; and it re-validates `state == BM_COOL` after reserving (a stale COOL read could otherwise reserve a frame already on the free list). - `free_push` sets `state = FREE` before clearing pin, closing the COOL+pin==0 window an eviction sweep could otherwise reserve in. Two general robustness improvements landed alongside (they reduce churn and were part of the original investigation, kept because they are correct on their own): `bm_fix` yields on a contended `try_pin` retry so the loop regains control, and a CLOCK second-chance reference bit on frames spares a recently touched COOL page one eviction sweep. test_bufmgr_mt is no longer gated: it runs on BOTH the io_uring CI (GitHub) and the epoll CI (Codeberg). ### (B) FIXED -- buffer-manager load/publish pin-ordering race While chasing the epoll hang, a genuine buffer-manager race was found and fixed (examples/06_sqlxtc/bufmgr.c). In bm_fix's demand-load path the frame was published as BM_COOL and only THEN pinned (`state = COOL; pin = 1`). In that window a concurrent evict_one -- which acts on COOL frames -- could win `try_reserve` (CAS pin 0 -> -1), and the load's unconditional `pin = 1` store then clobbered the reservation, leaving the frame both published (in use) and pushed back onto the free list: a double-owned frame that corrupts the free list (`free_n` seen at 762 for a 32-frame pool) and livelocks get_free_frame. Fix: pin the frame BEFORE it is ever visible as COOL/HOT (own it from get_free_frame on), so eviction's try_reserve always fails on it. Same reorder applied to bm_fix_pid. Verified: io_uring make check + ASan + UBSan clean, bufmgr_mt 12/12. This race is normally hidden when the offloaded load completes fast; it surfaces when completion is slow. The buffer-manager multi-threaded stress test (`examples/06_sqlxtc`, `test_bufmgr_mt`) hangs intermittently when libxtc is built with the epoll I/O backend (`--with-io-backend=epoll`); the same test passes reliably under io_uring (130+ runs). It is therefore run only on the io_uring CI (GitHub) and skipped on the epoll CI (Codeberg containers, whose seccomp profile blocks io_uring). Diagnosis (post-mortem core, non-instrumented timing): - All executor loop threads are idle in `epoll_wait`, the blocking thread-pool workers are idle on their condvar (so every submitted disk I/O has COMPLETED and its wakeup byte was written), free frames are available (`free_n == 8`), and there are zero data mismatches -- yet `xtc_exec_run` never returns because one worker proc (`g_workers_done == N_WORKERS - 1`) never finishes. - The stuck worker is the buffer-manager EVICTOR: it reserved a frame for eviction (`pin == -1`), parked in `xtc_proc_wait_fd` on the offloaded flush write, and its completion wakeup was lost, so the frame stays `EVICTING` forever. The loss is rare (~1 in many thousands of `xtc_blocking_run` calls) and timing-dependent, which is why the heavy-churn stress test triggers it while the lighter loop tests (test-vfs-loop, test-xstore, ...) do not. - io_uring masks it: its completion path samples current readiness rather than blocking indefinitely on a level/edge fd transition, so a missed edge self-heals; epoll's `epoll_wait(timeout = -1)` blocks forever. The lost wakeup is in `xtc_proc_wait_fd`'s epoll arm/park/dispatch path, hit through `xtc_blocking_run`. A later session built the minimal isolated reproducer this asked for -- N procs issuing concurrent `xtc_blocking_run` on an N-loop epoll executor, no buffer manager (test/concurrency/repro_blocking_epoll.c) -- and it hangs 8/8, so the bug is purely in the library primitive. Findings: - **It is NOT fd reuse.** A persistent per-proc wakeup pipe (never reused across procs while a registration is live) still hangs, and a bounded 50ms re-poll of every loop does not recover the parked proc. Ground truth from a core: stuck procs are PARKED on fds that are READABLE (the wake byte was written) yet are NOT in any epoll instance -- the fd was unregistered while the proc was still parked on it. The only `del_fd` that can do this is dispatch's `del_fd(t->park_fd)`. - **Approaches tried, and why each was rejected (none shipped):** 1. *Cross-thread waker* (pool wakes the proc via xtc_waker_wake): fixes the hang but ASan reports a heap-use-after-free -- the proc can return from xtc_blocking_run and exit the instant its result is read, racing a wake that holds its task pointer. NOTE (v1.13.0): this is the SAME root cause as the now-fixed resolve-then-deliver UAF (see "proc-teardown race RESOLVED" above). The proc-teardown refcount makes it safe for a peer to hold a proc reference across the wake; revisiting this approach with the refcount taken around the pool wake is the likely clean fix for the lost-wakeup, and should be tried in the dedicated session. 2. *Timer-poll* (proc polls a done flag via xtc_proc_sleep): memory-safe and fixes the pure reproducer, but its added latency makes test_bufmgr_mt deadlock on BOTH backends (it slows completion enough to provoke buffer-manager frame exhaustion), i.e. it REGRESSES io_uring, which passes today. 3. *Persistent per-proc wakeup pipe*: still hangs (ruling out fd reuse, as above). 4. *Remove del_fd from dispatch* (rely on the parker's cleanup + run-before-poll): made the reproducer hang 12/12 -- the still-readable fd is mishandled, so the dispatch del_fd is load-bearing in a way not yet understood. 5. *Event carries the fired fd* (epoll stores the fd in `data.u64`, a per-loop fd->tag map recovers the registrant, and dispatch dels exactly `ev->fd` -- the fd that actually fired -- instead of a possibly-stale `t->park_fd`): implemented across all backends, verified to NOT regress io_uring (make check + reproducer + bufmgr_mt all green), but the epoll reproducer still hangs 15/15. So the wrong-fd-del was not the root cause; reverted rather than ship inert complexity (a per-loop map). 6. *Non-blocking completion read + re-wait* (set the pipe read end O_NONBLOCK and loop wait_fd until the byte is actually read, so a wakeup with no byte yet cannot block the loop thread in read()): contained to blocking.c, but the epoll reproducer still hangs 15/15. Reverted. Sharper diagnosis (instrumented, this is where it stands): - A parked proc resumes from its wait_fd `xtc_yield()` with `wake_revents == 0` AND `park_fd` still set -- i.e. it was re-scheduled WITHOUT the fd dispatcher running (dispatch sets wake_revents and clears park_fd before waking). Such "spurious" resumes are not rare: tens of thousands per run. The committed blocking_run then does a blocking read() on the not-yet-readable pipe, wedging the loop thread so it stops polling -- a plausible cascade into the hang -- but making that read non-blocking (approach 6) did not fix it, so the spurious resume is not the whole story. - The spurious resumes are NOT cross-thread: an instrumented `__xtc_inbox_push` counted ZERO `XTC_INB_WAKE` pushes for the whole run, so no `xtc_waker_wake` cross-thread path fired. The only same-thread enqueue of a PARKED proc is the fd dispatcher, which clears park_fd and sets wake_revents -- so a resume with neither updated is self-contradictory under the current model and indicates the park/dispatch bookkeeping is desynchronised from the actual scheduler wakeup in a way the probes perturb rather than pin. Per-fd counters also show stuck fds with `del == reg` and `deliv == 0/1` (the fd is unregistered, by the parker's own cleanup on a spurious resume, before epoll ever delivers its event), consistent with the "readable but not in any epoll" post-mortem above. The bug needs a dedicated redesign rather than a point fix -- most likely a per-proc wakeup eventfd registered ONCE at spawn with EPOLLONESHOT re-arm (no per-wait registration churn, memory-safe, and no cross-thread proc reference), or moving blocking-completion off the fd-park mechanism entirely. Until then test_bufmgr_mt runs only on the io_uring CI and is skipped on the epoll CI, and xtc_blocking_run keeps its committed pipe + wait_fd path (fast on io_uring). The reproducer and this write-up set up that focused effort. ### FIXED -- test_server_storage flaky hang (lost cross-thread wakeup) `examples/06_sqlxtc/test_server_storage` hangs intermittently (~1 in 8 locally, io_uring backend) in the connection-per-proc + storage workload. A SIGABRT core of a hung run shows the scheduler loops blocked in `io_uring_wait_cqes` while two blocking-pool workers sit in `fdatasync` -- the WAL group-commit writer (wal.c `flush_io_fn`) and the double-write buffer (bufmgr.c `dw_io_fn`). The signature is a lost cross-thread completion wakeup: a fiber offloaded an fsync via `xtc_blocking_run`, the worker is (or just finished) running it, but the loop never observes the completion and idle-waits forever. Present at the shipped state (e36a68d), so it is NOT caused by the xtc_aio page-I/O wiring; that work (do_io / bm_sync on xtc_aio) is orthogonal. It was surfaced while attempting to convert the WAL writer and double-write buffer onto xtc_aio: that conversion must wait until this lost-wakeup is understood, since adding more concurrent completion traffic to the same hot path would only worsen it. Likely the same lost-wakeup class as the (A) investigation; needs a dedicated session with a clean core walk of the parked fiber's waker state and the blocking-pool -> loop wake path. Until then it is a known flake on the io_uring examples CI. FIXED (commit after b7a6558): the root cause was a cross-thread io_uring submission. xtc_proc_wait_fd -- which xtc_blocking_run parks on -- registered its wait fd on self->task->loop->io (the proc's HOME loop), but under the multi-loop executor a proc runs stolen on another loop, so the POLL_ADD went to a ring this thread does not own and was silently dropped. Fixed by registering/timing/cleaning up on __xtc_current_loop (the running loop). test_server_storage 30/30; the WAL/double-write xtc_aio conversion is now unblocked.

RESOLVED: macOS build break: sigev_notify_kqueue (io_kqueue.c)

Status: FIXED. A 2026-06 macOS SDK image dropped the BSD struct sigevent member sigev_notify_kqueue, so the native-file-AIO path added in ce0cacc failed to compile on macOS with “no member named ‘sigev_notify_kqueue’”. The #if guard (defined(EVFILT_AIO) && defined(SIGEV_KEVENT)) passed, but the member was absent under the current SDK.

Fix: the native kqueue file-AIO path is now restricted to the platforms that actually provide the member AND honor SIGEV_KEVENT completion for regular files – FreeBSD and DragonFly (#if ... && (defined(__FreeBSD__) || defined(__DragonFly__)) in src/io/io_kqueue.c). macOS is deliberately excluded and falls through to the blocking-pool offload (the XTC_E_NOSYS path), which is correct and never blocks the loop – Darwin’s POSIX AIO does not reliably support SIGEV_KEVENT completion on regular files anyway. The macOS CI job (macos-latest, Apple Silicon) builds and runs the full C suite green on every commit.

pbt-hegel CI job fails on ubuntu-24.04 with no output (OPEN, advisory)

Status: OPEN. The job is continue-on-error: true so it does not block merges; the property tier itself is verified and passing.

The property tier works. All 36 properties across 17 suites build and pass under --with-hegel, verified locally in several configurations:

  • --with-hegel via pkg-config (the nix develop path);
  • --with-hegel=PREFIX with a hand-assembled prefix (header + .so copied in), which is exactly what the CI job constructs;
  • with LD_LIBRARY_PATH unset, confirming configure’s -Wl,-rpath for the explicit-prefix case is sufficient;
  • under CI’s -Werror -Wall -Wextra -Wpedantic, in both the hegel-enabled and default (SKIP-stub) builds.

On the ubuntu-24.04 runner, make -C build_pbt tests-pbt exits 2 with no output whatsoever – not a compile error, not a property failure, not a SKIP. The job now always prints the captured output before deciding (an earlier version lost it to set -e around out=$(make ...)), and there is still nothing to print, which suggests make itself fails before running a recipe.

Unverified hypotheses, in the order worth testing:

  1. The Build the library step and this step disagree about the build directory or a generated file, so make has nothing to do and errors.
  2. Something in the runner image’s make/sh differs enough that the tests-pbt recipe’s $$((...)) arithmetic or case fails immediately.
  3. The libhegel .so fetched by curl is fine to link against but the runner’s loader rejects it at first use in a way that kills make’s subshell without output.

Anyone picking this up: reproduce on an actual ubuntu-24.04 container rather than a nix shell, which is where every local attempt succeeded and therefore where the difference is invisible.