- Generated by
1.9.8
|
libxtc 0.4.0
Async concurrency for C: Tokio + Seastar + BEAM, in one library
|
title: Known issues parent: Reference nav_order: 7 lede: >- Honest caveats, workarounds, and the platform-verification status.
Status: RESOLVED. The carrier reported a process-directed SIGCHLD delivered to a libxtc scheduler thread (where MyProcPid == 0) instead of the thread the embedder designated.
Three layers:
Evidence: test/m1/test_thread_sigmask.c (main is the only SIGUSR1-unblocked thread; multi-loop executor + blocking pool + 200 process-directed SIGUSR1s) reports 0 deliveries to any libxtc thread, 20/20 under CPU load and in the gated parallel make check; R1 fault containment (test_proc fault_contain/fault_early_contain/ recovery_registry) passes; fork/exec (test_osproc) passes; the three preemption tests pass; ASan make check is clean. test_thread_sigmask is now in the gated make check set.
Status: RESOLVED. Surfaced 2026-07 via test/concurrency/repro_idle_uring_wake.c.
Symptom: procs spawned CROSS-THREAD (xtc_proc_spawn from an OS thread that is not on the target loop) onto a service-mode executor that is stopped mid-flight were not fully reclaimed by xtc_exec_fini – LeakSanitizer reported the task+coro (xtc_async -> __os_calloc, coro_uctx.c) as leaked, run-to-run variable (~3-7%).
Fix, in two parts:
Evidence: repro_idle_uring_wake under ASan leak-detection goes to 0 leaks across 70+ consecutive runs on both epoll and io_uring (was ~3-7%); the full proc/sup/svr/tnt suites pass under ASan with no use-after-free or double-free; a drain-to-idle shutdown was already leak-clean and stays so.
Why it was not caught earlier: the leak is only visible with a WORKING LeakSanitizer – GitHub's containerized CI runner restricts the ptrace LSan needs, so LSan silently no-ops there. The lost-wakeup guarantee the reproducer checks is covered leak-clean under DST by test/sim/test_sim_wake_park.c.
For an honest 1.x stance, here is exactly which platforms run at RUNTIME (the full test suite executes) versus which only COMPILE:
Status: real backends added; one (Windows) remains compiled-but-not- runtime-verified for lack of a Windows host.
Previous issue: src/os/os_backtrace.c had only the execinfo backend (glibc/macOS/BSD) and a no-op stub everywhere else, so musl and Windows got an empty C backtrace in xtc_dump / the panic+crash handler.
What changed this round:
XTC_HAVE_LIBUNWIND) for libc's lacking <execinfo.h> (notably musl): __os_backtrace walks the calling thread with unw_step; __os_backtrace_emit symbolizes best-effort with dladdr (XTC_HAVE_DLADDR) or prints raw addresses otherwise. Detected by dist/configure.ac (--with-libunwind=auto|yes|no|PATH, OPTIONAL); execinfo wins when both are present._WIN32): CaptureStackBackTrace + SymFromAddr, linked with -ldbghelp.test/m12/test_backtrace.c drives a 3-deep call chain, asserts >= 2 captured frames, and (on a symbolizing platform) requires at least one of its own frames to be named.Verified on this round's Linux glibc x86_64 host:
make check for m1/m12 green, test_backtrace green, the existing test_dump panic/crash path green.XTC_HAVE_LIBUNWIND and linking the system libunwind; produced a REAL symbolized trace and passed test_backtrace.test_backtrace passes on frame count alone.-Wall -Wextra -Wpedantic clean.NOT runtime-verified (honest gaps):
-ldbghelp, but no Windows machine in the CI matrix exercises it. Reviewed against the Win32 DbgHelp API docs; same status as src/io/io_aix.c.__os_backtrace_supported() returns 0).See docs/guide/debugging.md for the per-platform symbolization matrix.
Status: FIXED in this round.
Previous issue: The SHARED_MEMORY mode tests used MAP_PRIVATE | MAP_ANONYMOUS, which is single-process memory, not actual shared memory. Additionally, the slab's shm_cursor was stored in per-process private memory, so two processes with their own xtc_slab_t structs would each start carving chunks at offset 0, causing collisions.
Fix: The cursor now lives in a 64-byte header at the start of the shared region, using atomic CAS for coordination. First attacher initializes the header (magic=0x5854435F534C4142 "XTC_SLAB", version=1, cursor=64); subsequent attachers verify magic and use the existing cursor.
Verification: New test file test/m11/test_slab_shm.c exercises real cross-process sharing via fork(2) + POSIX shm_open:
The previous misleading tests have been renamed to clarify their scope:
Status: resolved. xtc_fault_guard_install was a no-op on Windows; it now installs a Vectored Exception Handler that contains a fiber-attributable hardware fault by restoring the CONTEXT captured at xtc_proc_recovery_arm() (via EXCEPTION_CONTINUE_EXECUTION – no stack unwinding, which is what makes it safe on a fiber stack; a longjmp driven from a VEH reliably corrupted the CRT heap). Runtime-verified on a Windows host (santorini, MinGW gcc) by a dedicated driver: a proc arms recovery, registers resources, and takes a real access violation – the VEH contains it (exit 0), the recovery resource registry releases the proc's fd + callback automatically, and the monitor observes DOWN(reason). A companion driver confirms the ESCALATION half of the contract: a fault inside a critical section is NOT contained – the VEH returns EXCEPTION_CONTINUE_SEARCH and the process dies with 0xC0000005 (EXCEPTION_ACCESS_VIOLATION), preserving the PG critical-section PANIC semantics. While wiring this up, a pre-existing Windows double-free was found and fixed in coro_winfiber.c (the done branch destroyed the coro eagerly and via the task cleanup at loop_fini), which had made any loop+process tear down with heap corruption on Windows.
Status: RESOLVED (2026-09-09). The full munit m8/test_proc now BUILDS AND RUNS under MSVC and is part of the dist/build_msvc.bat hard gate: 18 of 18 cases pass, 1 skips (/fault_escalate only, which needs fork()). /selective_receive – the historically suspect case – passes. Verified interactively on an EC2 Windows Server 2022 x86_64 host (c7i.4xlarge, MSVC 19.44.35228), not inferred from CI.
The earlier note said "the full munit `test_proc` cannot build under
cl.exe (munit uses GCC-isms)" and treated the smoke test as the only possible Windows guard. That was WRONG on the cause: munit itself builds fine (the MUNIT_ARRAY_PARAM VLA fix landed long ago). What actually blocked it was FOUR test-side portability defects, all fixed:
#include <sys/wait.h> at file scope, needed only by the one fork()-based case. Now _WIN32-guarded, and that single case returns MUNIT_SKIP on Windows (its escalation half is covered by the SEH driver – see "Windows fault containment (SEH)" above).__attribute__((packed)) mirror structs for the DOWN / EXIT signal layouts. The tree already has the portable XTC_PACK_PUSH / XTC_PACKED / XTC_PACK_POP trio for exactly this (MSVC needs #pragma pack, which an attribute cannot express); the test now uses it.clock_gettime(CLOCK_PROCESS_CPUTIME_ID) in /recv_inf_parks (the "a parked proc burns no CPU" proof). Windows DOES have a clean equivalent – GetProcessTimes (kernel+user) – so the case now runs on Windows instead of being written off as unportable. It is worth more there than elsewhere: the IOCP backend's 8 ms AFD repoll sweep is exactly the kind of thing a regression could turn into a spin, and this case would catch it.8L * 1024 * 1024 * 1024 overflows a 32-bit MSVC long (C4307). Now 8LL. Same class as the documented test_blocking _Atomic long bug.Two REAL bugs were found by getting the test to run – one in the library, one in the test harness. Both are documented next.
Status: FIXED (2026-09-09). Found by running m8/test_proc on a real Windows host for the first time: /fault_early_contain asserted s.reason == 11 against an actual value of -1073741819.
Root cause: the SEH vectored handler (__xtc_veh in src/ptc/proc.c) stored the raw Win32 exception code into p->fault_sig:
EXCEPTION_ACCESS_VIOLATION is 0xC0000005, i.e. -1073741819 as a signed int. That value became the proc's exit_reason and was delivered to every monitor as the DOWN reason. It broke three documented promises at once (src/inc/xtc_proc.h): the reason is specified to be a POSITIVE signal number ("e.g. 11 for SIGSEGV"), to sit inside 1..255, and to be distinguishable from XTC_DOWN_NOPROC (-100000) precisely so a supervisor can tell a contained fault from "target already gone". A negative, out-of-range, platform-specific value satisfies none of them – so a supervisor written to the documented contract misclassified every contained Windows fault.
Fix: a __xtc_veh_code_to_signo() mapping applied where the code is recorded, so Windows reports the same signal numbers POSIX does: access violation / misalignment / in-page error -> SIGSEGV, the two divide-by-zero codes -> SIGFPE, illegal / privileged instruction -> SIGILL. (SIGBUS does not exist on Windows, hence misalignment folding into SIGSEGV, which is what a POSIX kernel reports for the same fault on the architectures libxtc targets.)
Evidence: /m8/proc/fault_early_contain went from FAIL (-1073741819) to OK (reason 11) on the host; /fault_contain and /recovery_registry (whose reasons flow through the same field) pass; the POSIX path is untouched (19 of 19 on Linux; the change is inside #if defined(_WIN32)).
Status: FIXED (2026-09-09). /m8/proc/recovery_registry did not merely fail on Windows – it killed the whole test binary with 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN), so no later case ran.
Root cause: the TEST, not the library. The library closed the tracked fd correctly; the test proved it had, with the POSIX idiom
On Windows the MSVC CRT's _close() on an unused descriptor invokes the invalid-parameter handler, whose default action is __fastfail(FAST_FAIL_INVALID_ARG): the process dies immediately and no assertion is ever evaluated. Isolated with a 15-line standalone program (no libxtc linked): a bare double close() aborts with 0xC0000409 every time. _get_osfhandle() on a closed fd was measured to be JUST AS FATAL, so it is not a safe alternative.
Fix: test/include/fd_probe_compat.h provides xtc_test_fd_is_closed(fd) – on POSIX a plain close(fd) == -1, on Windows the same _close() bracketed by a no-op thread-local invalid-parameter handler (_set_thread_local_invalid_parameter_handler with a REAL no-op function; passing NULL restores the fatal default), which was measured to make the CRT return -1/EBADF exactly as POSIX does. A general Windows test hazard, so it lives in test/include/ rather than inside one test.
Status: RESOLVED as a DOCUMENTATION ERROR (2026-09-09). The previous entry called this "a real cross-shard-wake bug still under investigation... likely the self-wake pipe's interaction with the IOCP readiness model." That diagnosis was wrong in the strongest possible way: there is no tnt code on Windows to have a bug in.
src/orc/tnt.c is wrapped in #if !defined(_WIN32); the Windows half of the file (from the #else) is nothing but XTC_E_NOSYS stubs, with the in-source comment "tnt is a POSIX feature (raw socket I/O in the
couriers)". There is no self-wake pipe, no shard scheduler, and no cross-shard sender in a Windows build.
Verified, not reasoned: a standalone probe on the Windows host prints xtc_tnt_start rc=-3 (XTC_E_NOSYS). The all-zero counters were the arithmetic consequence of a runtime that was never compiled in – exactly what a NOSYS stub is supposed to produce.
Two further corrections to the record:
test_tnt had ALREADY grown a rc == XTC_E_NOSYS -> return 77 (SKIP) branch in commit e41f128 (2026-08-03), so the "all-zero
counters" symptom the entry described was already gone before this investigation; the entry had simply gone stale.53e6ea1, 2026-09-05) added #include <sys/socket.h> at file scope for the socketpair-staged echo scenario – so test_tnt did not even COMPILE under MSVC (‘C1083: Cannot open include file: 'sys/socket.h’`) and the SKIP could never be reported.Fix: the POSIX-only body of test_tnt.c is now _WIN32-guarded as a whole, with a small Windows main() that prints the SKIP and returns
test_tnt join the MSVC gate at all: the Windows translation unit never references <sys/socket.h>, so it builds, and it reports its status honestly instead of failing to compile. tnt/test_tnt is now IN dist/build_msvc.bat (as a standalone, non-munit driver – it has its own main() and no munit.c) and reports SKIP. On Linux it is unchanged and still passes the full scenario.Status: round-2 native rewrite; the smoke test now runtime-verifies more of it on santorini (VS18, ARM64). As of 2026-07 the MSVC smoke gate passes: version, strerror, clocks, slab, lwlock, SEH fault containment, xtc_fs, selective_receive (IOCP wakeup ordering), single-op AND multi-op native file AIO (overlapped pwrite/pread reaped from the port), and cross-thread wakeup coalescing (256 foreign xtc_send via PostQueuedCompletionStatus) all pass on the real host.
ONE scenario was a SKIP through v1.20.1 and is now FIXED (with a documented workaround, not a root-cause fix in the AFD driver itself – see item 4 below): the loopback-socket connect/accept/echo over the AFD poll path (smoke_sock_server / smoke_sock_client) now passes on an EC2 Windows Server 2022 host, verified across 5+ consecutive smoke runs, a ASan build (7/8 clean; the 1/8 failure is a pre-existing, unrelated flake documented separately below), and a dedicated CPU-scale test at 1,000/2,000/5,000/10,000 idle pending sockets (0.00% measured CPU at every size).
Status note (2026-07): substantial progress on an MSVC Windows host (Windows Server 2022, EC2); SEVEN real bugs found and fixed across two work sessions (three in the IOCP round below, plus four more – poll-timeout, pthread retval, thread affinity, slab mmap zero-fill – in the full test-surface sweep; see the sweep update further down and docs/M_WINDOWS_MATRIX.md); the one then-remaining item (tnt/test_tnt) turned out not to be a bug at all – see the tnt entry above – and a EIGHTH real bug (the raw EXCEPTION_* DOWN reason) was found in 2026-09 by getting m8/test_proc to run on the host:
WORKED AROUND (2026-07-13) – the ASYNC (STATUS_PENDING) AFD poll does not complete when the socket LATER becomes ready. Root cause confirmed with a standalone reproducer that has NO libxtc code in it: a ~150-line program opens \Device\Afd directly, arms one async IOCTL_AFD_POLL for AFD_POLL_RECEIVE on a not-yet-readable accepted socket, sends from the client, and waits on the port – the poll returns STATUS_PENDING and NEVER completes, even though the socket is genuinely readable afterward. A follow-up experiment proved AFD CAN always answer with CURRENT readiness synchronously (arming a poll on a socket that already has data waiting returns STATUS_SUCCESS with the correct event bits, reliably, every time) but never notifies of a FUTURE readiness change through this driver/version's \Device\Afd handle. Cancelling a stuck pending poll does not help either: NtCancelIoFileEx finalizes the IRP's IOSB in-process (STATUS_CANCELLED) but, like every other synchronous AFD outcome on this driver, does not post a port completion. This rules out libxtc's IOCP reap loop, registration lifetime, base-handle resolution (SIO_BSP_HANDLE_POLL resolves correctly and identically every time), and the IOSB/OVERLAPPED aliasing as causes – the defect is in the AFD driver's future-edge tracking for this handle, upstream of anything libxtc controls.
The workaround (__xtc_iocp_repoll_sweep in src/io/io_iocp.c): xtc_io_poll periodically re-checks every PENDING registration older than XTC_IOCP_REPOLL_NS (8 ms) with a single BATCHED, zero-timeout, throwaway AFD poll covering up to 64 overdue sockets per syscall (confirmed by direct experiment: a second, independent poll on a socket that already has its own long-pending poll outstanding does not disturb it, and Timeout=0 always resolves synchronously with the true current state). Only the sockets the probe actually flags ready are canceled and re-armed on their real registration OVERLAPPED, which self-posts through the existing synchronous-arm path. A first per-socket (not batched) version of this workaround measured 50%, 90%, and 98% CPU with 1,000, 2,000, and 5,000 idle pending sockets respectively (one cancel+rearm syscall pair per socket per interval, regardless of readiness); the batched redesign measured 0.00% CPU at 1,000/2,000/5,000/10,000 idle pending sockets (one cheap syscall per up-to-64 sockets when nothing is ready).
Latency caveat (measured, not theoretical): GetQueuedCompletionStatusEx's millisecond timeout is quantized to the process's current Windows timer resolution, which defaults to ~15.6 ms (the classic 64 Hz system tick) unless something has raised it. On the EC2 host used for this work the observed worst-case echo latency was ~15.6 ms regardless of the 8 ms constant; calling timeBeginPeriod(1) (from ANY process on the system, not necessarily this one) measured ~3-5 ms instead. libxtc does not call timeBeginPeriod itself – it would raise the SYSTEM's timer resolution for as long as this process runs, a global side effect this project does not impose on an embedder's behalf (the same reasoning as not pinning threads to cores). An application that wants tighter Windows socket-readiness latency than the default tick can call timeBeginPeriod(1) itself.
This is a workaround, not a fix to the AFD driver's own behavior – if a future Windows version or driver starts posting the completion correctly, the sweep still works (it only ever refreshes sockets that are ALREADY overdue; a socket whose real completion arrives promptly never reaches the sweep). Full repro harness, the batching experiments, and the AWS/SSH recipe are recorded in .agent/AFD_ASYNC_COMPLETION_2026-07.md.
Update (2026-07, full MSVC test-surface sweep, EC2 Windows Server 2022): every TESTS_C test was built + run individually under cl.exe 2022 and the Windows gate grew from 16 to a 100-test HARD GATE (build_msvc.bat step 5, all-pass, 0 warnings under /WX; per-test matrix in docs/M_WINDOWS_MATRIX.md). Four more real bugs were found and fixed on the host, plus the poll-timeout bug below:
xtc_io_poll ignored its caller timeout across AFD repoll slices. It capped the wait to the 8 ms sweep interval and returned after a SINGLE GetQueuedCompletionStatusEx slice, so a poll armed while the socket was not yet readable (bug 4 above) returned XTC_OK with zero events even though the caller asked to block up to timeout_ns and data had since arrived. test/m2/test_io_events E1/E3/E5 (readable/HUP/many-ready) caught it: writable fired (socket immediately writable) but readable never did. Fix: a deadline-bounded loop that sweeps + re-arms between 8 ms slices until an event/wakeup is emitted or the caller's real deadline elapses (src/io/io_iocp.c).m1/test_thread T1/T2); see docs/M_WINDOWS_MATRIX.md.__os_thread_set_affinity was XTC_E_NOSYS on Windows (m1/test_cpu); now SetThreadAffinityMask.m11/test_slab shm_reclaim); plus test_blocking stored a 64-bit clock in _Atomic long (LLP64 truncation).Status: KNOWN FLAKE, pre-existing, unrelated to any libxtc code. While validating the AFD workaround above under an MSVC /fsanitize=address build (2026-07-13, EC2 Windows Server 2022), the smoke test's overall pass rate was 7/8 clean runs; the 1/8 failure reproduces identically with EVERY libxtc change reverted (confirmed by building and running the unmodified pre-workaround baseline the same number of times, which flaked at the same ~1-in-6-to-8 rate with the identical signature):
The fault is inside RtlInitializeResource, called from a brand-new OS thread's startup (BaseThreadInitThunk -> RtlUserThreadStart), before any libxtc code on that thread runs – ASan's own __asan_handle_no_return/stack-switch interposition on Windows fibers/threads is a documented rough edge (https://github.com/google/sanitizers/issues/189, which the smoke test's own "ASan is ignoring requested __asan_handle_no_return" warning already references). Always occurs, when it occurs, AFTER the loopback socket echo check has already printed ok – it is a teardown/thread-creation-path artifact, not a data-path bug. Non-ASan MSVC builds (the /WX zero-warning gate) and the plain smoke run never exhibit it. Tracked as a documented flake, not a release blocker; re-running the smoke binary passes.
src/io/io_iocp.c was rewritten from the round-1 readiness emulation to a native completion-port design (details in docs/M_WINDOWS_MATRIX.md):
CreateIoCompletionPort + GetQueuedCompletionStatusEx is the only wait primitive; the round-1 64-handle WaitForMultipleObjects cap is GONE.\Device\Afd + IOCTL_AFD_POLL via NtDeviceIoControlFile), re-armed per completion (level-triggered). This is the wepoll/libuv design.PostQueuedCompletionStatus; file AIO is overlapped ReadFile/WriteFile reaped from the same port (no hEvent).NtCancelIoFileEx and defers the free to the reap; registration nodes have stable heap addresses so the kernel-held back-pointer never dangles. dist/configure.ac adds -lntdll; dist/build_msvc.bat links ntdll.lib.What is verified (Linux dev host this round): cross-compiles clean with mingw-w64 gcc 14.3.0 -std=c11 -Wall -Wextra, links into a PE32+ binary against -lntdll -lws2_32, and leaves the Linux build untouched (io_iocp.c is XTC_IO_BACKEND_IOCP-only – an empty TU on Linux; the full C munit suite stays green on epoll/uring).
What is NOT verified: the backend has NOT executed on Windows. The AFD poll correctness, the level-triggered re-arm (no busy-loop, no dropped edges), the cancel/lifetime under churn (no double-free, no freeing a kernel-owned OVERLAPPED), the wakeup coalescing/ordering, and the file-AIO port round-trip must all be validated on santorini before this is production quality. Same reviewed-but-untested status as src/io/io_aix.c. This rewrite is the suspected fix territory for the test_proc::selective_receive flake (the wakeup ordering changed), but that cannot be confirmed without the host.
RUNTIME-VERIFIED (2026-08, EC2 Windows Server 2022, c5.2xlarge, MSVC 2022 Build Tools): the four IOCP-runtime smoke scenarios below ALL PASS on real Windows – native IOCP file AIO (8 overlapped pwrite/pread at distinct offsets), cross-thread wakeup (256 foreign xtc_send via PostQueuedCompletionStatus coalescing), loopback socket echo via AFD poll (level-triggered re-arm), and the AFD repoll-sweep scale (256 idle pending sockets, batched probe, not O(n)/socket). Additionally the MSVC munit subset (16 tests across m0/m1/m10/m11/m14) builds and passes 16/16 (after fixing the per-milestone munit git-symlink checkout on Windows – see dist/build_msvc.bat), and the DbgHelp backtrace backend is runtime-verified (test/m12/test_backtrace_win.c: 5 frames captured, SymFromAddr resolved a known function name). The historical "awaiting a santorini run" text is kept below for provenance.
**(historical) Smoke coverage WRITTEN, awaiting a santorini run (NOT yet verified):** test/msvc/smoke.c now drives four IOCP-runtime scenarios beyond the already-verified strerror/clocks/slab/lwlock/fault-containment set. They COMPILE (cross-checked with mingw-w64 gcc 14.3.0 -std=c11 -Wall -Wextra -fsyntax-only against the real Windows headers, with the MSVC pthread shim forced into its _MSC_VER path) but have NOT been built or run under cl.exe on a Windows host:
smoke_aio_proc (pre-existing): a single overlapped pwrite+pread round-trip at offset 0 – the file-AIO port round-trip.smoke_aio_multi_proc (new): several positioned overlapped pwrite/pread ops at distinct offsets in one loop run, so more than one AIO completion is reaped from the port per run (the reap loop, not just a single completion).smoke_xt_worker + smoke_xt_sender (new): a foreign OS thread bursts xtc_send at a batch of parked worker procs, exercising the cross-thread PostQueuedCompletionStatus wakeup and its coalescing (__xtc_io_iocp_wakeup_post); the loop must reap every delivery and return once all workers exit.smoke_sock_server + smoke_sock_client (new): a 127.0.0.1 connect/accept/echo driven entirely by xtc_proc_wait_fd on the raw Winsock socket fds – the AFD poll fast path and its level-triggered re-arm (WRITABLE for connect completion, READABLE for accept, then READABLE/WRITABLE for the echo). Send/recv is raw Winsock in the test so its error handling does not depend on any errno mapping in the library net helpers; if a listen port cannot be bound on the runner the scenario SKIPS rather than fails.These are the runtime scenarios the "What is NOT verified" list above names; the smoke test is the intended Windows regression guard for them once santorini runs dist/build_msvc.bat and reports the ok lines. Until that run they remain COMPILED-NOT-RUNTIME-VERIFIED.
The round-1 backend (WSAEventSelect + WaitForMultipleObjects, hard-capped at 64 handles, ~60% of native IOCP throughput) passed the full suite on the reference Windows toolchain (MinGW64: 233/233). Those results describe the SUPERSEDED source, not the round-2 rewrite above; they are retained here only as the baseline the round-2 code must re-establish on santorini. Round-1 notes that still stand:
test_net_udp used a bare nanosleep (absent in the Clang64/MinGW runtime) – now portable (a test_msleep shim). test_proc_wait_fd still uses pipe(2) + clock_gettime(CLOCK_PROCESS_CPUTIME_ID) + pthreads; porting it to the socket-pipe compat is straightforward but must be verified on the host before landing (an unverifiable Windows edit risks a silent Linux regression), so it is deferred to a santorini pass.test_proc::selective_receive flake** (above): the cooperative equivalent (test/otp/test_otp_proc_lib.c) passes on every platform, so selective receive itself is correct; the flake is in test_proc's exact IOCP-wakeup timing and needs the host to chase.Native file AIO (mechanism): xtc_io_aio_submit issues overlapped ReadFile/WriteFile; fsync/fdatasync have no async form on Windows (FlushFileBuffers is synchronous) and are offloaded. Round 1 joined the OVERLAPPED.hEvent to the WaitForMultipleObjects set and was verified on santorini under VS2022 (17) and VS2026 (18) by the smoke test (an overlapped pwrite+pread round-trip). Round 2 instead reaps the AIO completion from the port (no hEvent); that variant is COMPILED-NOT-RUNTIME-VERIFIED and the smoke test must be re-run.
Driving the santorini host non-interactively from CI/automation is not yet wired (it is configured for an interactive PowerShell session); the Windows matrix is run by hand via dist/santorini-matrix.sh.
xtc_aio_pread/pwrite/fsync/fdatasync present one portable API; the native completion mechanism is per backend, with a blocking-pool offload everywhere else (always correct, just thread-backed). Native completion is implemented and validated on:
IORING_OP_READ/WRITE/FSYNC, reaped from the CQE ring. CI build-and-test + ASan/UBSan.ReadFile/WriteFile reaped from the completion port; fsync offloaded. santorini smoke test.aio_read/aio_write/ aio_fsync) with SIGEV_KEVENT -> EVFILT_AIO. CI macos job compiles and runs test_aio against it (green).aio_read/aio_write/aio_fsync) with SIGEV_PORT posting the completion as a PORT_SOURCE_AIO event on the loop's existing event port, reaped in the port_getn drain (aio_return -> publish a->res -> wake a->tag). Implemented in src/io/io_solaris.c (xtc_io_aio_submit + struct sol_aio); RUNTIME-verified on the illumos/OpenIndiana host (sun) – full gmake check passes, including the native /aio/roundtrip case.Still offloaded to the blocking pool (native AIO is a follow-up with platform-specific completion plumbing – they are DIFFERENT mechanisms, not one shared path):
SIGEV_SIGNAL delivered to a self-pipe / signalfd the loop already watches (epoll has no AIO filter). A new shared io_posixaio.c submodule.aio_* (or the legacy LIO interface).The offload fallback makes "write storage code once as if AIO is always available" hold on all of these today; the follow-ups only remove the thread hop.
Decision (updated 2026-07): illumos native AIO is now DONE and verified; epoll/poll/select and AIX stay offload-backed deliberately. Assessment of the remaining targets:
Implemented as xtc_bdev (src/io/io_bdev.c, xtc_bdev.h): open a raw device / partition (or a regular file), query logical+physical sector size and capacity via the native per-OS ioctl (BLKSSZGET/BLKGETSIZE64, DIOCGSECTORSIZE/DIOCGMEDIASIZE, DKIOCGMEDIAINFO, the Windows drive geometry IOCTL) with a fstat fallback, aligned pread/pwrite through the xtc_aio path, and flush via xtc_aio_fsync. See xtc_bdev(3).
Status: RESOLVED (2026-09-09); the documented reason was STALE. The previous entry read: "intentional – <tt>_aligned_malloc</tt> returns memory that requires <tt>_aligned_free</tt>, not plain <tt>free</tt>. The hook surface uses a single free path. Keeping the M7 case Windows-skipped is correct."
That premise stopped being true when commit d10c257 ("os/alloc: fix
UBSan misalignment on cache-line-aligned allocations") gave the allocator vtable a MATCHED aligned() / aligned_free() PAIR. There is no longer a single free path: __os_aligned_free() exists, routes to _aligned_free on Windows and free elsewhere, and src/inc/os_alloc.h documents the two as a pair that MUST be used together. The AGENTS.md over-aligned-allocation rule depends on that pair existing.
What actually kept M7 unrunnable on Windows was that the TEST released with the WRONG half – __os_free(q) on memory from __os_aligned_alloc(), which is exactly the mismatched-free bug the pair exists to prevent and which would corrupt the heap on Windows. So the skip was masking a defect IN THE TEST, and the case as written could never have been simply un-skipped.
Fix: M7 releases with __os_aligned_free() (the documented pair) and runs on every platform. It also gained the case the AGENTS.md rule actually cares about – an XTC_CACHE_LINE alignment stricter than max_align_t – plus an aligned_free(NULL) no-op check.
Evidence: /m1/alloc/M7_aligned OK on the Windows host (test_alloc 8 of 8, 0 skipped – was 7 + 1 skip) and still OK on Linux (8 of 8).
Re-measurement (2026-09-09): re-run on the same instance shape as the original datapoint (EC2 x86_64 Windows Server 2022, c7i.4xlarge, 16 vCPU, MSVC 2022, IOCP backend) with the same per_loop=30000, so it is a like-for-like comparison – on a newer source tree (v1.43.0):
| loops | 2026-07 (below) | 2026-09 run 1 | 2026-09 run 2 |
|---|---|---|---|
| 1 | 77 K | 55 K | - |
| 2 | 102 K | 81 K | - |
| 4 | 127 K | 121 K | 124 K |
| 8 | 91 K | 97 K | 100 K |
| 16 | 82 K | 80 K | 90 K |
Every run: spawn_ok == done, fail = 0. The documented shape holds – spawn throughput peaks at 4 loops and regresses past it – and the peak reproduces within ~5%. The 1- and 2-loop numbers came in lower than 2026-07 and the 8/16-loop numbers slightly higher, which narrows the peak-to-tail ratio without changing the finding; repeated runs at the top three points show a several-percent run-to-run spread, so the small deltas are not worth attributing to any source change. The 2026-07 conclusion (the ceiling is cross-loop work-steal cache thrash + OS thread placement, NOT the slab allocator) stands unmodified.
Status: MEASURED on EC2 x86_64 Windows Server 2022 (c7i.4xlarge, 16 vCPU, MSVC 2022, IOCP backend) via bench/bench_win_scale.c – a portable spawn-throughput probe buildable under cl.exe (public xtc_* API + xtc_clock_mono only, no POSIX headers). The Win32-fiber substrate corruption blocker is fixed (above), so these numbers are on a sound runtime.
Spawn throughput (spawn + immediate-exit child, per-loop drivers, per_loop=30000, all runs done==total, fail=0):
loops= 1 77 K spawns/s loops= 2 102 K spawns/s loops= 4 127 K spawns/s <- peak loops= 8 91 K spawns/s loops=16 82 K spawns/s
Two honest findings:
A Seastar/Tokio comparison on Windows is now UNBLOCKED but not yet run: it wants (a) the FlsAlloc magazine so the fast path is measured, and (b) message-throughput + tasks/sec workloads added to bench_win_scale.c. POSIX scalability was measured at scale (EC2 Phase B, 192 vCPU); the Windows curve above is the first cross-platform datapoint.
Status: RESOLVED (2026-07, EC2 x86_64 Windows Server 2022 + MSVC 2022). The full cross-process path now works end to end on Windows: CreateProcess re-exec + --xtc-xproc-child sentinel + loopback-TCP control connect (nonce) + dedicated reader thread + shadow-proc monitor
xtc_xproc: re-exec child spawn+send+monitor, DOWN reason=42 is now a HARD GATE (was a SKIP); 40/40 non-ASan runs pass and 3/3 MSVC-ASan runs are clean.Two substrate bugs were the blocker (both fixed), found via MSVC AddressSanitizer on the box:
CreateFiberEx was called with dwStackCommitSize == dwStackReserveSize == 64 KiB, leaving zero growth headroom. A proc's exit teardown (__proc_entry -> __notify_links_and_monitors -> xtc_send -> slab) plus a Win32 API callback (a ~360-byte ntdll memset seen under ASan) overflowed the fiber stack and corrupted adjacent memory (the near-NULL slab-pointer AV). Fix: reserve generously (>= 1 MiB, or 16x the committed working size) so the OS grows the stack normally, committing only the requested size.__declspec(thread) (static TLS), which is unsafe under Win32 fibers. The compiler may cache the static-TLS base across a SwitchToFiber, so a magazine pointer handed to xtc_slab_alloc dereferenced a stale TLS block on a resumed fiber – traced to a hang/corruption at mag->slots[--mag->n] inside __mon_alloc (the monitor entry allocation), which is exactly the path xtc_xmonitor drives. Fix: __tls_mag_for returns NULL on Windows, so every alloc/free takes the locked slow path (touches no __declspec(thread) state, correct on a fiber). Performance-only tradeoff – a fiber-local-storage (FlsAlloc) magazine is a later optimization; Windows is not the throughput target yet.POSIX xtc_xproc (fork/waitpid) and the fcontext/ucontext substrates are unchanged. The reader-thread control channel and the io_wakeup exit-callback (which fixed the earlier proc-table-SRWLock deadlock) are kept.
Status: SHIPPED (v1.13.0 for ASan; TSan fiber-identity added v1.17.0). The fcontext and ucontext coro substrates annotate every user-space stack switch for the sanitizers, using the CORRECT, mutually-exclusive API for each:
XTC_FIBER_SWITCH_ANNOTATE, __SANITIZE_ADDRESS__ / __has_feature(address_sanitizer)): the stack-switch API __sanitizer_start_switch_fiber / __sanitizer_finish_switch_fiber around every switch, so ASan tracks the fiber stacks instead of mis-attributing stack memory. The full fiber runtime (test_fctx/proc/async/svr/fsm) passes ASan with detect_stack_use_after_return=1 on BOTH substrates.XTC_TSAN_FIBERS, clang __has_feature(thread_sanitizer) only): the fiber-IDENTITY API – __tsan_create_fiber at coro create, __tsan_switch_to_fiber at every switch (into a coro, and back to the once-captured scheduler fiber), __tsan_destroy_fiber at teardown – so TSan carries per-fiber happens-before across cooperative switches instead of seeing one confused thread. TSan does NOT provide __sanitizer_*_switch_fiber, so the two guards are mutually exclusive (TSan decided first; a TSan build emits ONLY the identity calls, an ASan build ONLY the stack-switch calls). A full clang -fsanitize=thread libxtc build runs test_fctx/async/proc/ svr/chan with ZERO TSan warnings. clang-only by design: gcc's libtsan has no fiber support, so a gcc TSan build emits no fiber annotations (documented; not a gcc-buildable TSan-with-fibers configuration). Requested by the PG-integration team and delivered.Compiled to nothing in a non-sanitized build (0 sanitizer references, verified: nm libxtc.a | grep __tsan_ is empty).
CI: the shared ASan job runs with detect_stack_use_after_return=1 on every commit.
Status: RESOLVED (v1.13.0) by a teardown refcount on the proc struct. Surfaced by the fiber-switch annotations under detect_stack_use_after_return=1 (which shifts scheduling enough to widen the window): when a proc exited, __notify_links_and_monitors sent a DOWN to each monitoring proc via xtc_send; a monitor proc concurrently torn down on another thread could be freed between __resolve returning its pointer and __mbox_deliver_locked touching its mailbox – a heap-use-after-free. This was the SAME root cause as the blocking-pool wake UAF and (suspected) the macOS sqlxtc MT-load flake: __resolve handed out a struct xtc_proc * after releasing the table lock, with no lifetime guarantee.
Fix: an atomic refs on struct xtc_proc. __table_lookup takes a reference WHILE HOLDING the owning table lock – atomic with the detach in the teardown path – so a resolver either pins a live proc or sees NULL, never a freed pointer. The owner (spawn) holds one ref; teardown detaches from the table then drops the owner ref, and the struct's mailbox-drain / lock-destroy / free happen in __proc_free only when the last ref (owner + any in-flight resolver) is released. Every __resolve caller (xtc_send, xtc_exit_pid, xtc_proc_wake, the deferred-delivery callback, link/monitor push, xtc_proc_mailbox_stats) releases its ref. CI now runs the ASan job with detect_stack_use_after_return=1; validated by 5 clean full-make-check runs under ASan+SUAR (0 UAF, 0 leaks). A DST test (test_sim_proc_teardown) models the resolve-vs-exit race deterministically.
Status: improved this round with NULL/invalid-argument guard tests (test/coverage/test_fault_inject.c /svr/null_guards), which exercise the early-return XTC_E_INVAL edges across xtc_svr_start/stop/join/reply/ call/cast that the happy-path server tests skipped, PLUS the reply-path OOM edge (/svr/reply_oom): a new svr.reply.oom injection point at the top of xtc_svr_reply forces the non-empty-reply allocation to fail, covering the XTC_E_NOMEM return on BOTH reply branches (the synchronous slot path and the in-proc reply_pid tag+payload path) – ASan-clean, no leak on the failure path.
Targets remaining: the deeper call-after-stop timing edge (a call that races xtc_svr_stop) still needs a scheduler-ordering harness rather than a plain injection point; it is error handling, not mainline behavior.
Status: improved this round. The four io.init.* fault-injection points (calloc/pipe/fcntl/backend fail) were already tested; added /io/null_guards (xtc_io_init/fini/wakeup reject NULL with XTC_E_INVAL) and /io/wakeup_roundtrip (the live xtc_io_wakeup post + coalesced second post) in test/coverage/test_fault_inject.c. The remaining uncovered branches are backend-specific cleanup edges in xtc_io_fini and the ENOMEM/EAGAIN corners only reachable on a particular backend; a per-backend fault-injection sweep is the way to close them further.
Status: RESOLVED (fiber-stack-aware backtrace walker, 8034f93; fallback hardened so it can never reach the unbounded system backtrace() from a fiber).
xtc_dump() captures a C backtrace of the calling thread. On macOS/arm64, when called from inside a running fiber (a live xtc_proc body, not the top-of-thread panic/abort path), the system backtrace() walks the frame-pointer chain of the small guard-paged fiber stack and could run past the stack top into unmapped memory, raising SIGBUS – intermittently (it depended on what lay just past the fiber stack and whether that address was mapped), which is why it surfaced only occasionally in CI's test_dump/basic.
Fix: src/os/os_backtrace.c now uses a fiber-stack-aware frame walker on Apple targets instead of the system backtrace(). It discovers the active stack's VM region at call time from its own SP (mach_vm_region – a query of the task's VM map, which cannot itself fault) and walks the FP chain manually, stopping the instant the next frame record would leave the mapped region, is misaligned, or does not ascend – so an over-walk past a guard-paged fiber stack top can never touch an unmapped page. No coroutine-layer coupling: the bound is derived from the running SP, so it works whether the caller is on a fiber mmap or an OS-thread stack. If the region cannot be resolved (a rare mach_vm_region failure), the walker returns NO frames rather than falling back to the unbounded system backtrace() – the dump then prints "backtrace unavailable", which is strictly better than a SIGBUS. The symbolization step (backtrace_symbols_fd) only resolves the already-captured, in-bounds return addresses (a dladdr-style lookup, not a stack walk), so it does not fault.
Other platforms were never affected: Linux uses the EH-ABI unwinder (_Unwind_Backtrace), which terminates cleanly on a fiber stack. test_dump/basic exercises the from-a-fiber path (its dump_driver runs as an xtc_proc and calls xtc_dump), so the macOS CI runner covers the fixed path per commit.
Status: UNSUPPORTED / UNMAINTAINED. An AIX/ppc64 OS-layer port (os_thread/os_mutex/os_tls + an io_aix.c pollset backend + POSIX AIO offload) exists in-tree and compiles, but AIX is deliberately NOT a supported platform: it is never built or run in CI, has no runtime verification, and is off the roadmap. AIX is IBM-proprietary (ppc64-only; PostgreSQL itself dropped AIX in v16), the install media is licensed, and emulating it under QEMU/TCG is impractical for a reliable CI gate. The in-tree port is left in place (it is harmless and compiles cleanly) but carries no support commitment – treat any AIX behavior as unverified. Do not file AIX as a gap; it is a decision, not a TODO.
The macos GitHub Actions job (macos-latest, Apple Silicon) builds and runs the full C munit suite every commit – kqueue backend, ucontext substrate, GCD dispatch semaphores. Standing it up fixed six real portability bugs (Darwin feature macro, rwlock storage size, unnamed semaphores, _SC_NPROCESSORS_ONLN, hardcoded -lrt, lrlock slot reclamation teardown order).
Status: RESOLVED for Apple Silicon as of 1.23.x – macOS/arm64 now defaults to the hand-written fcontext coroutine substrate, closing the per-switch performance gap. macOS x86-64 still uses ucontext (there is no Mach-O x86-64 fcontext assembly), and involuntary preemption remains Linux-only everywhere on macOS (see below).
src/os/asm/fctx_aarch64_aapcs_macho.S provides the AAPCS64 make/jump_fcontext in the Mach-O assembler dialect (leading-underscore symbols, .p2align, no ELF-only directives), byte-identical in register discipline to the ELF variant. src/evt/coro_fctx.c and src/evt/coro_uctx.c select it as the default on __APPLE__ && __aarch64__ (equivalent to -DXTC_CORO_FORCE_FCTX); build with -DXTC_CORO_FORCE_UCONTEXT to fall back to swapcontext. This removes the per-switch sigprocmask syscall (measured on Apple M-series: ~34 us/task-switch on ucontext under bench_million_tasks.c at 1M fibers -> ~23 us there, with a larger per-switch win masked by memory pressure at that scale; a 2-fiber micro-bench shows the true delta). The fcontext path on macOS declines involuntary preemption, so it falls back to Phase-1 cooperative preemption there.
Involuntary preemption (src/ptc/preempt.c, Phase 2) is STILL Linux-only: it needs a per-thread SIGVTALRM-class timer (timer_create/CLOCK_THREAD_CPUTIME_ID), which macOS does not offer (ITIMER_VIRTUAL is process-wide, not per-thread). On macOS, xtc_preempt degrades to cooperative-only: a fiber that never calls a yield point will not be involuntarily interrupted. See man/man3/xtc_preempt.3 for the per-platform statement.
Remaining: a macOS x86-64 Mach-O fcontext variant (low priority – Apple has moved to arm64), and a GCD dispatch-timer or kqueue EVFILT_TIMER per-worker tick to give Phase-2 preemption a real signal source on macOS (tracked in PLAN.md). Neither changes correctness.
Status: KNOWN FLAKE (transient, re-run passes). The sqlxtc multi-thread load CI step (test/sqlxtc/test_sqlxtc_mt.sh -> test_sqlxtc_concurrent.py, N clients * M queries against a shared server on a plain CREATE TABLE) intermittently fails on the macOS runner with all clients timing out and server CRASHED under multi-loop load (cores=3). Observed once on 5d78c00 and passed on re-run with no code change; the 5 preceding commits' macOS jobs were green.
The failing path is the VDBE/connection handling under concurrent load (the test table is NOT xstore-backed, so it never reaches the vexec fast path) – so it is independent of the vexec/Track-B work. Likely the same multi-loop cross-thread timing sensitivity seen elsewhere on the macOS/kqueue runner. Treat a lone macOS MT-load failure as flaky: re-run the job (gh run rerun <id> --failed) and judge by the other 12 jobs. A persistent failure across re-runs would indicate a real concurrency regression and must be chased.
Status: RESOLVED in M11.5b. The proc registry's __lt[] table was leaking entries on xtc_loop_fini; consecutive PBT loops were aliasing stale entries. Fix: added __xtc_proc_loop_unregister(loop) called from xtc_loop_fini. Both properties are re-enabled.
Status: Config-file parsing and reload DONE; per-session scoping is out of scope (it belongs to a downstream consumer, not the runtime).
xtc_cfg_load_file() reads name = value lines (comments, quotes, per-kind parsing, bounds/validators), skipping unknown/bad lines.xtc_cfg_reload() re-reads the last loaded file. The app wires SIGHUP to it from the event loop (the function is not async-signal-safe, by documentation).Status: DONE.
Resolved: xtc_slab_pressure_listen_ex() returns an opaque xtc_slab_pressure_t handle and xtc_slab_pressure_stop(handle) joins the listener thread, closes its fds, and frees it. Plain xtc_slab_pressure_listen() delegates to _ex and discards the handle (unchanged fire-and-forget behaviour).
Status: RESOLVED (pending continued monitoring). The primary causes were fixed earlier (see (A)/(A-residual)/(B) below); the last remaining epoll-only residual in the buffer-manager stress test appears CLOSED by the v1.8.0 cross-thread prepare/park wake fix.
The closing fix (v1.8.0): the residual was the same lost-wake class as the PostgreSQL carrier's cross-thread fd/latch wake miss – a cross-thread wake (here, the blocking pool completing an offloaded fsync and waking the parked evictor) arriving while the target task was still RUNNING, between arming its waker and the loop transitioning it to PARKED on yield, was DROPPED by the XTC_INB_WAKE drain (which only enqueued PARKED tasks). The fix latches task->wake_pending when the task is not yet PARKED and the RUNNING->PARKED transition consumes it and re-schedules instead of parking (see src/evt/loop.c).
Evidence: test_bufmgr_mt, which historically hung ~3-7% of runs on epoll, now passes 130/130 consecutive runs on the epoll backend with zero hangs, and the reproducer test/concurrency/repro_blocking_epoll.c (which hung 8/8 at its worst) passes 10/10. test_bufmgr_mt is no longer gated off epoll: it runs on the Codeberg (epoll) CI as well as the io_uring CI. Kept in this document (rather than deleted) so the long investigation and the several rejected approaches below stay on record; if any epoll wake-miss recurs, this is the history to read first.
The primary lost-wakeup was not in the epoll backend at all. It was a fiber-context bug: the per-thread "current proc" pointer (__current_proc, an L3 process-layer TLS) was not preserved across a yield. When a proc calls xtc_yield() (or xtc_await) the scheduler runs OTHER procs in between – each setting __current_proc to itself – and on resume the public yield primitives did NOT restore it. Internal parks (xtc_proc_wait_fd, xtc_proc_sleep, xtc_recv, the amutex) each restored it by hand, but a plain xtc_yield() did not. A proc that yielded therefore resumed running as whatever proc ran last; its next xtc_blocking_run -> wait_fd then registered the completion fd under the WRONG task and parked the WRONG task, so the real proc never woke.
Minimal reproducer: ONE loop, TWO procs, three xtc_blocking_run calls each – hangs 10/10 on epoll (a trace showed proc P1, after its worker's bare xtc_yield(), registering its pipe fd under proc P2's task tag and "P2" parking twice without waking). io_uring masked it: its completion path re-samples readiness, so the misattributed park self-healed; epoll's epoll_wait(-1) blocked forever.
Fix: preserve the process context across every coro yield via a hook (__xtc_fiber_ctx_save / __xtc_fiber_ctx_restore, installed by xtc_proc_spawn, no-op when no process layer is in use, so the L2 coro layer keeps no hard dependency on L3). Applied to xtc_yield and xtc_await in all three coro substrates (fctx, ucontext, Win32 fiber). Verified: the pure reproducer goes 0 -> 20/20 on epoll (and the N-loop/N-proc one 0 -> 20/20); io_uring make check + ASan + UBSan + epoll make check all clean; io_uring bufmgr_mt 50/50.
After the (A) fix, test_bufmgr_mt still hung about 3-7% on the epoll backend (io_uring far less, and confounded by host load). The earlier read of this as a benign "thrash livelock" was WRONG: it was a real pin-accounting bug in the example's from-scratch buffer manager (examples/06_sqlxtc/bufmgr.c), traced to ground with armed abort probes
Root cause (swip mode only): a fixer can hold a STALE swip word w pointing at a frame that has since been evicted, freed, and recycled by the demand-load path for a DIFFERENT slot. The fixer does try_pin(sw_frame(w)) and transiently bumps that recycled frame's pin (it fails its slot recheck a moment later and unpins – net zero). But the demand-load path claimed the frame with an UNCONDITIONAL store(pin, 1), which clobbered the fixer's transient increment; the fixer's matching unpin then drove pin to -1. A frame wedged at pin == -1 is indistinguishable from an eviction reservation, so every later fixer spins try_pin on it forever – the hang. (epoll merely lost the timing lottery more often; io_uring re-samples readiness and mostly masked it.)
Fixes, all in bufmgr.c, verified epoll 80/80 + io_uring 30/30 + ASan 12/12 + UBSan 8/8 clean, with the btree/xstore (pid-mode) tests green:
claim_frame: spin CAS pin 0 -> 1, yielding), NOT a blind store, so a fixer's transient stale pin is never clobbered. Scoped to swip mode only: pid-mode (bm_fix_pid, the B-tree path) has no swip references, so no stale fixer exists; it keeps the plain store (a CAS-wait there would deadlock against a latch-coupling pin that cannot drain while the claimer spins).state == BM_COOL after reserving (a stale COOL read could otherwise reserve a frame already on the free list).free_push sets state = FREE before clearing pin, closing the COOL+pin==0 window an eviction sweep could otherwise reserve in.Two general robustness improvements landed alongside (they reduce churn and were part of the original investigation, kept because they are correct on their own): bm_fix yields on a contended try_pin retry so the loop regains control, and a CLOCK second-chance reference bit on frames spares a recently touched COOL page one eviction sweep.
test_bufmgr_mt is no longer gated: it runs on BOTH the io_uring CI (GitHub) and the epoll CI (Codeberg).
While chasing the epoll hang, a genuine buffer-manager race was found and fixed (examples/06_sqlxtc/bufmgr.c). In bm_fix's demand-load path the frame was published as BM_COOL and only THEN pinned (state = COOL; pin = 1). In that window a concurrent evict_one – which acts on COOL frames – could win try_reserve (CAS pin 0 -> -1), and the load's unconditional pin = 1 store then clobbered the reservation, leaving the frame both published (in use) and pushed back onto the free list: a double-owned frame that corrupts the free list (free_n seen at 762 for a 32-frame pool) and livelocks get_free_frame. Fix: pin the frame BEFORE it is ever visible as COOL/HOT (own it from get_free_frame on), so eviction's try_reserve always fails on it. Same reorder applied to bm_fix_pid. Verified: io_uring make check + ASan + UBSan clean, bufmgr_mt 12/12. This race is normally hidden when the offloaded load completes fast; it surfaces when completion is slow.
The buffer-manager multi-threaded stress test (examples/06_sqlxtc, test_bufmgr_mt) hangs intermittently when libxtc is built with the epoll I/O backend (--with-io-backend=epoll); the same test passes reliably under io_uring (130+ runs). It is therefore run only on the io_uring CI (GitHub) and skipped on the epoll CI (Codeberg containers, whose seccomp profile blocks io_uring).
Diagnosis (post-mortem core, non-instrumented timing):
epoll_wait, the blocking thread-pool workers are idle on their condvar (so every submitted disk I/O has COMPLETED and its wakeup byte was written), free frames are available (free_n == 8), and there are zero data mismatches – yet xtc_exec_run never returns because one worker proc (g_workers_done == N_WORKERS - 1) never finishes.pin == -1), parked in xtc_proc_wait_fd on the offloaded flush write, and its completion wakeup was lost, so the frame stays EVICTING forever. The loss is rare (~1 in many thousands of xtc_blocking_run calls) and timing-dependent, which is why the heavy-churn stress test triggers it while the lighter loop tests (test-vfs-loop, test-xstore, ...) do not.epoll_wait(timeout = -1) blocks forever.The lost wakeup is in xtc_proc_wait_fd's epoll arm/park/dispatch path, hit through xtc_blocking_run. A later session built the minimal isolated reproducer this asked for – N procs issuing concurrent xtc_blocking_run on an N-loop epoll executor, no buffer manager (test/concurrency/repro_blocking_epoll.c) – and it hangs 8/8, so the bug is purely in the library primitive. Findings:
del_fd that can do this is dispatch's del_fd(t->park_fd).data.u64, a per-loop fd->tag map recovers the registrant, and dispatch dels exactly ev->fd – the fd that actually fired – instead of a possibly-stale t->park_fd): implemented across all backends, verified to NOT regress io_uring (make check + reproducer + bufmgr_mt all green), but the epoll reproducer still hangs 15/15. So the wrong-fd-del was not the root cause; reverted rather than ship inert complexity (a per-loop map).Sharper diagnosis (instrumented, this is where it stands):
xtc_yield() with wake_revents == 0 AND park_fd still set – i.e. it was re-scheduled WITHOUT the fd dispatcher running (dispatch sets wake_revents and clears park_fd before waking). Such "spurious" resumes are not rare: tens of thousands per run. The committed blocking_run then does a blocking read() on the not-yet-readable pipe, wedging the loop thread so it stops polling – a plausible cascade into the hang – but making that read non-blocking (approach 6) did not fix it, so the spurious resume is not the whole story.__xtc_inbox_push counted ZERO XTC_INB_WAKE pushes for the whole run, so no xtc_waker_wake cross-thread path fired. The only same-thread enqueue of a PARKED proc is the fd dispatcher, which clears park_fd and sets wake_revents – so a resume with neither updated is self-contradictory under the current model and indicates the park/dispatch bookkeeping is desynchronised from the actual scheduler wakeup in a way the probes perturb rather than pin. Per-fd counters also show stuck fds with del == reg and deliv == 0/1 (the fd is unregistered, by the parker's own cleanup on a spurious resume, before epoll ever delivers its event), consistent with the "readable but not in any epoll" post-mortem above.The bug needs a dedicated redesign rather than a point fix – most likely a per-proc wakeup eventfd registered ONCE at spawn with EPOLLONESHOT re-arm (no per-wait registration churn, memory-safe, and no cross-thread proc reference), or moving blocking-completion off the fd-park mechanism entirely. Until then test_bufmgr_mt runs only on the io_uring CI and is skipped on the epoll CI, and xtc_blocking_run keeps its committed pipe + wait_fd path (fast on io_uring). The reproducer and this write-up set up that focused effort.
examples/06_sqlxtc/test_server_storage hangs intermittently (~1 in 8 locally, io_uring backend) in the connection-per-proc + storage workload. A SIGABRT core of a hung run shows the scheduler loops blocked in io_uring_wait_cqes while two blocking-pool workers sit in fdatasync – the WAL group-commit writer (wal.c flush_io_fn) and the double-write buffer (bufmgr.c dw_io_fn). The signature is a lost cross-thread completion wakeup: a fiber offloaded an fsync via xtc_blocking_run, the worker is (or just finished) running it, but the loop never observes the completion and idle-waits forever.
Present at the shipped state (e36a68d), so it is NOT caused by the xtc_aio page-I/O wiring; that work (do_io / bm_sync on xtc_aio) is orthogonal. It was surfaced while attempting to convert the WAL writer and double-write buffer onto xtc_aio: that conversion must wait until this lost-wakeup is understood, since adding more concurrent completion traffic to the same hot path would only worsen it.
Likely the same lost-wakeup class as the (A) investigation; needs a dedicated session with a clean core walk of the parked fiber's waker state and the blocking-pool -> loop wake path. Until then it is a known flake on the io_uring examples CI.
FIXED (commit after b7a6558): the root cause was a cross-thread io_uring submission. xtc_proc_wait_fd – which xtc_blocking_run parks on – registered its wait fd on self->task->loop->io (the proc's HOME loop), but under the multi-loop executor a proc runs stolen on another loop, so the POLL_ADD went to a ring this thread does not own and was silently dropped. Fixed by registering/timing/cleaning up on __xtc_current_loop (the running loop). test_server_storage 30/30; the WAL/double-write xtc_aio conversion is now unblocked.
Status: FIXED. A 2026-06 macOS SDK image dropped the BSD struct sigevent member sigev_notify_kqueue, so the native-file-AIO path added in ce0cacc failed to compile on macOS with "no member named
'sigev_notify_kqueue'". The #if guard (defined(EVFILT_AIO) && defined(SIGEV_KEVENT)) passed, but the member was absent under the current SDK.
Fix: the native kqueue file-AIO path is now restricted to the platforms that actually provide the member AND honor SIGEV_KEVENT completion for regular files – FreeBSD and DragonFly (#if ... && (defined(__FreeBSD__) || defined(__DragonFly__)) in src/io/io_kqueue.c). macOS is deliberately excluded and falls through to the blocking-pool offload (the XTC_E_NOSYS path), which is correct and never blocks the loop – Darwin's POSIX AIO does not reliably support SIGEV_KEVENT completion on regular files anyway. The macOS CI job (macos-latest, Apple Silicon) builds and runs the full C suite green on every commit.
Status: OPEN. The job is continue-on-error: true so it does not block merges; the property tier itself is verified and passing.
The property tier works. All 36 properties across 17 suites build and pass under --with-hegel, verified locally in several configurations:
--with-hegel via pkg-config (the nix develop path);--with-hegel=PREFIX with a hand-assembled prefix (header + .so copied in), which is exactly what the CI job constructs;LD_LIBRARY_PATH unset, confirming configure's -Wl,-rpath for the explicit-prefix case is sufficient;-Werror -Wall -Wextra -Wpedantic, in both the hegel-enabled and default (SKIP-stub) builds.On the ubuntu-24.04 runner, make -C build_pbt tests-pbt exits 2 with no output whatsoever – not a compile error, not a property failure, not a SKIP. The job now always prints the captured output before deciding (an earlier version lost it to set -e around out=$(make ...)), and there is still nothing to print, which suggests make itself fails before running a recipe.
Unverified hypotheses, in the order worth testing:
Build the library step and this step disagree about the build directory or a generated file, so make has nothing to do and errors.make/sh differs enough that the tests-pbt recipe's $$((...)) arithmetic or case fails immediately..so fetched by curl is fine to link against but the runner's loader rejects it at first use in a way that kills make's subshell without output.Anyone picking this up: reproduce on an actual ubuntu-24.04 container rather than a nix shell, which is where every local attempt succeeded and therefore where the difference is invisible.
1.9.8