net.
moves
at light.

A security release — one critical auth fix, and the nRPC wire path keeps shrinking

Where v0.27.1 was pure performance and nothing on the wire moved, v0.27.2 leads with a four-pass security audit of the net crate and the fixes it surfaced — headlined by a critical authorization-bypass in the capability fold — then continues the hot-path work on the nRPC dispatch layer the hot-path audit opened. The full security review is recorded in docs/misc/SECURITY_AUDIT_2026_06_09_NET_CRATE.md; this log is the operator-facing summary.

The reassuring part first: the audit found the crate unusually well-hardened. Untrusted-wire parsing, token/chain auth, nonce/randomness handling, handshake identity binding, secret hygiene, and the filesystem/path surfaces all came back clean and verified (not assumed) — most classic hazards already had named, tested mitigations. One finding stood apart and is fixed below; the rest are medium/low hardening and defense-in-depth.

Interop: honest v0.27.1 peers are unaffected. The critical fix only rejects forged input that the old code silently trusted — a legitimate node always announced its own node_id, so nothing on the honest path changes. No wire-format change.

🔴 The critical fix — the capability fold now binds wire node_id to the verified signer

SignedAnnouncement::verify checked that an announcement carried a valid Ed25519 signature over a transcript that included node_id — but never that the claimed node_id was actually the signer's. The dispatch and apply layers then keyed all capability/reservation state on that attacker-supplied node_id.

The exploit chain (all four links confirmed against the code):

1. Peer A — legitimately authenticated via PSK + Noise — signs a CapabilityMembership envelope with its own entity key but sets the internal node_id to victim C's.
2. verify passes: it is a valid signature by A over those bytes; nothing required the node id to be A's.
3. apply installs the entry under key (class_hash, C) — a forged capability now lives in C's state (e.g. tags:[nrpc:<service>], allowed_nodes:[A]).
4. A calls the gated service; the callee gate reads by_node[C], finds the forged entry, and returns true.

Impact: complete bypass of the per-node nRPC capability allow-list — any authenticated participant could invoke any capability-gated service on any node — plus global forge/overwrite/strip of other nodes' advertised capabilities (cap-stripping DoS, scheduler-placement poisoning). The same unbound-node_id primitive hit ReservationFold, enabling reservation/lock hijacking on behalf of arbitrary node ids.

The fix is exactly the one the audit prescribed — surgical, outsized impact: verify / decode_and_verify (and the reservation path) now reject any envelope where ann.node_id != publisher.node_id(), returning WireError::NodeIdMismatch. This closes capability injection and reservation hijack simultaneously. The check is effectively free — Ed25519 verification (~50 µs) already dominates every inbound envelope. Pinned by a full-dispatch multi-publisher regression test, and the Fold::restore trust invariant ("only restore from local snapshots") is now documented alongside it.

FFI hardening — the aggregator handles join the crate's UAF protection

Two aggregator FFI handles (RegistryClientHandle, FoldQueryClientHandle) did an unconditional drop(Box::from_raw(handle)) on free, lacking the HandleGuard every other opaque handle in the crate carries — so a caller racing free against an in-flight op (a pattern the handles' own docs invite) could deallocate the client out from under a live read. Closed:

• Both handles adopt the standard HandleGuard + leak-on-free + try_enter()-gated ops treatment, with quiesce-on-free, and no longer hold the guard across the blocking RPC.
• net_registry_last_error_detail / net_fold_query_last_error_detail now return a caller-owned char* (freed with net_free_string) instead of a pointer into a Mutex-owned CString a concurrent erroring op could free out from under the reader; net.h documents the ownership, and the Go bindings free the returned strings.
• Free now warns only on a genuine drain timeout (via begin_free_detailed), and a new-handle adoption checklist was added to handle_guard so the next FFI handle gets this by construction.

Filesystem — symlink-escape closed in directory reconstruction, including the subtle FS bypasses

fetch_dir sanitized a symlink's link path via safe_join but wrote its target verbatim from the attacker-controlled manifest — the classic "symlink in an archive" exposure. v0.27.2 rejects absolute / escaping symlink targets, and — crucially — closes the bypasses a naïve check misses:

• Composed-link and symlinked-parent escapes (a link whose escape only materializes through an earlier-reconstructed link).
• Case- and normalization-insensitive FS bypasses — default macOS APFS compares filenames both case- and normalization-insensitively, so the lexical traversal check now folds component case and applies NFC before comparing. (Reconstruction was already strictly ordered — dirs, then files, then symlinks last — so this was never a traversal write; the fix removes the residual risk to whatever later reads the tree.)

Hardening grab-bag (medium/low, from the audit's backlog)

• Constant-time secret compares. GroupId (32-byte) / SubnetId (16-byte) bearer secrets now compare via subtle::ConstantTimeEq instead of derived PartialEq / Vec::contains (early-exit, data-dependent timing). Remote timing recovery of a 128/256-bit secret was already impractical; closed for completeness.
• PSK config permissions. The aggregator daemon now warns when its TOML config (which holds the mesh PSK) is group/world-readable — mirroring the 0600 discipline the CLI identity seed already enforces. The check runs before parse and warns on non-Unix too.
• Cap-filters documented as advisory. subscribe_caps / publish_caps are self-asserted matchmaking, not an access boundary — the real boundary is require_token + token_roots (root-anchored TokenChain). This is now prominently documented so no one mistakes a cap-filter for access control.
• Fuzz coverage widened. New fuzz targets for the nRPC request decode, channel-membership decode, migration bindings decode, and blob-transfer header decode — attacker-reachable, manually-hardened decoders that previously lacked a continuous regression guard. The fuzz crate gained the bytes / postcard deps and the cortex / dataforts features to reach them.

Correctness — a node no longer expires its own capability entry (self-inflicted outage)

Surfaced while baselining the nRPC QPS bench (audit §15): capability fold entries carry a TTL (default 300 s) and the sweeper reaps them on expiry, but nothing periodically re-announced the node's own entry — serve_rpc's announce is one-time. So any node serving RPC continuously past one TTL (≈5 min) without re-announcing would start rejecting all inbound calls (the callee-side cap gate finds no self-entry → CapabilityDenied) and drop off peer discovery — masked until now because every test/bench runs far under 300 s.

Fixed with a periodic re-announce loop (spawn_capability_reannounce_loop) that re-broadcasts the node's capabilities every MeshNodeConfig::capability_reannounce_interval (default 150 s), refreshing both the local self-index (callee gate) and peers' folds (discovery). Re-broadcasting needs an owned Arc, so MeshNode::start_arc(self: &Arc<Self>) stores a Weak the loop upgrades each tick; the SDK (Mesh::start) and FFI (net_mesh_start) call it. A bare start(&self) keeps its signature — no test-caller churn — and simply omits the loop. A/B tests pin both directions.

A follow-up review hardened the TTL math: because announce_capabilities_with rate-limits the network broadcast to min_announce_interval, a re-announce interval set below it would have let peer entries expire before the throttle released the next broadcast. The stamped TTL is now 2 × max(reannounce_interval, min_announce_interval) — sized to the cadence peers actually see a refresh at, not the bare tick. (The local self-index is refreshed every call regardless, so only peers were ever at risk.)

A companion bench-harness fix bounds call_*_retrying backpressure retries with a 20 s RETRY_DEADLINE, so a saturated benchmark bar fails fast (transport saturated … not measurable here) instead of livelocking past a TTL.

nRPC hot path — fewer wakeups, fewer allocations per round-trip

The audit's headline conclusion holds — the system is syscall- and wakeup-bound, not compute-bound (51% wake/scheduling, 22% transport syscalls, ~5% AEAD). v0.27.2 lands the contained, no-wire-change items that attack the wakeup count on the unary response leg:

• §8a — one fewer tokio::spawn per response. The response emit closure used to spawn a task for every response publish (~1–2 µs of scheduling on a wake-bound path). It now builds the wire payload synchronously and hands an RpcResponseJob to a single per-service drain task (the same drainer pattern as grant-coalescing) — bounded channel, drop-on-overflow, FIFO. Streaming/duplex variants keep their per-emit spawn; unary is the QPS hot path.
• §8b — reply-channel name cached per caller. format!("{service}.replies.{caller_origin:016x}") + ChannelName::new() was two heap allocs on every response, deterministic from (service, caller_origin). Now cached alongside the response path's origin cache — a cache hit is an Arc bump.
• T2.2 — RpcPayload::encode_into. The encode path allocated a Vec then extend_from_slice'd it into the caller's buffer — a double copy. encode_into(&mut buf) writes once (≈200–400 ns saved at 1 KiB).

Measurement honesty: these are µs-and-below wins per round-trip — below this dev box's Windows-loopback variance floor. A before/after on nrpc_qps moved c1/32B ~−3.6% with the other bars inside noise; the authoritative measurement remains the Linux flamegraph environment, consistent with the rest of the wire-path audit. They are landed because they are correct, contained, and directly reduce the 4–5 wakeups/RT the flamegraph attributes 51% of CPU to — not because the dev-box microbench can prove the delta.

Memory-amplification guard — and an unexpected speedup

A review of the §8b work flagged that the per-serve_rpc caller-keyed caches (the reply-channel cache and the response-path origin→node cache) were unbounded maps keyed by the wire-claimed caller origin — and the origin cache is populated before the capability gate. A single authenticated peer could spray distinct origins and amplify server memory without limit; "bounded by peer count" held only for well-behaved callers.

Both are now a single bounded OriginKeyedLru (parking_lot::Mutex<lru::LruCache>) capped at the legitimate active-caller working set per service (4096). Eviction is always safe — a miss just rebuilds the channel name or falls back to the roster lookup, never correctness.

The bound turned out to be faster, not a trade-off: DashMap's hash-to-shard + per-shard RwLock buys nothing when one bridge/fold task owns the cache, and the LRU under an uncontended mutex skips it. Measured on the single-accessor workload serve_rpc actually produces:

So the hardening caps memory and shaves ~6 ns off the per-response hot path. A before/after micro-bench (benches/origin_cache_bench.rs) ships alongside it.

Benchmark hygiene — corrections and a first capture

• FailureDetector rows annotated (audit §14). failure_detector/check_all (~342 ms) and stats (~80–100 ms) in BENCHMARKS.md are benchmark-fixture artifacts of an O(nodes) scan reported per-element, not hot-path costs — check_all runs once per heartbeat_interval (default 5 s) and costs ~204 µs at 5,000 nodes (~40 µs/s amortized); stats is observability-only. The genuine per-heartbeat costs are 14–242 ns. The rows now carry a warning so nobody chases them.
• bench_append_batch_disk run for the first time (audit §10). 64-event batches: 64 B → 20.2 µs/batch (~3.17 M ev/s), 1 KiB → 62.5 µs/batch (~1.02 M ev/s) ≈ ~315 ns/event. The policy companion confirms the RedEX Phase 3/4 background-fsync design directly: never/default ~23 µs/batch vs every_n_1 (synchronous fsync per batch) ~853 µs/batch — coalescing is ~40× faster than per-batch fsync.

What's deliberately parked (and why)

The audit's three highest-leverage levers are identified, scoped, and intentionally not in this release — each is gated on something this release isn't the place to spend:

1. Recv-loop batching (§1) — recvmmsg batched ingress is built through Stage 5 but stays default-off behind a Cargo feature + runtime flag, pending the c128 latency measurement (and a ~40-LoC channel-hop gap-fix that must land first, or the measurement understates the design).
2. Ack-piggyback (§2) — the one lever on the unary QPS ceiling (~70K → 150–200K target), but it's a wire-format change with cross-binding compat work; scheduled as its own effort.
3. Crypto SIMD (§3) — RUSTFLAGS="-C target-feature=+avx2" recovers 5–10× on the per-packet AEAD cost, but a baked-in floor would SIGILL on pre-AVX2 x86-64. The committed-config decision stands; the open question is per-artifact build flags for the published wheels/prebuilds.

Breaking changes

None on the wire, and none for honest peers. v0.27.2 interoperates with honest v0.27.1 / v0.27.0 peers freely — the capability node_id binding only rejects forged envelopes (a node_id that doesn't match the signer) that no legitimate node ever produced; they now return WireError::NodeIdMismatch instead of being silently accepted.

Two source/ABI-level notes for FFI consumers (no in-tree non-test callers affected):

• Aggregator error strings are now caller-owned. net_registry_last_error_detail / net_fold_query_last_error_detail return a heap char* the caller must release with net_free_string (the bundled Go bindings already do). Previously they returned a borrowed pointer.
• The aggregator handle free path now leaks-on-free + quiesces (matching every other handle); double-free / free-during-op transitions to a logged drain warning rather than UB.

The SDK surface (Mesh::start, etc.) is unchanged — start_arc is wired internally.

How to upgrade

Upgrade promptly — this release closes a critical authorization bypass. Bump the dependency to 0.27.2; for the common case (Rust core + SDK) it is drop-in, with no atomic peer roll and no config changes. Notes:

1. C/Go FFI consumers of the aggregator error-detail accessors must free the returned string with net_free_string (or use the updated Go bindings, which do).
2. Capability re-announce is automatic — long-lived RPC servers now refresh their own entry every 150 s by default; tune via MeshNodeConfig::with_capability_reannounce_interval (set to Duration::MAX to disable). If you set it below min_announce_interval, the TTL is sized to the throttle, so peers stay live regardless.
3. Optional, unchanged from v0.27.1: rebuild the x86-64 target class with RUSTFLAGS="-C target-feature=+avx2" to unlock the AEAD fast path. Default builds are unchanged.

Named after Prince's 1984 closer to the album and the film — the eight-minute power ballad cut live one August night at First Avenue in Minneapolis in 1983 and never re-recorded, the Wendy Melvoin guitar lead and the Lisa Coleman piano answering each other under a vocal Prince once said started life as a Bob Seger country song before he heard it differently. "Dearly beloved, we are gathered here today to get through this thing called life." The film's last shot is the Kid walking offstage after the band has finally come together; the album's last note is the long-decaying piano chord that follows. v0.27 is the substrate's same shape: the long-running reliability, security, and concurrency threads that have been running through the codebase since v0.21 close out together. Stream retransmit wires every piece of the reliable-stream machinery the substrate had implemented but never connected. The reliable-stream hardening pass closes the cluster of deficiencies that wiring surfaced. The channel-auth audit replaces bare-token credentials with root-anchored token chains. The capability fold's bulk-query path turns 100 ms scans into 100-µs lookups. The polling-to-event-driven SDK migration ends a class of CPU waste across every language tier. And a new fair-scheduler transport primitive ships datafort blob transfer in the same release as the SDK that exposes it. Same nRPC, same fold, same fold-driven discovery — but the substrate finally plays the chord that resolves the act.

A long act closes; a new transport primitive opens the next one

The v0.27 release converges a stack of work that has been threading through the codebase since v0.21. None of it introduces a new system — every piece either finishes wiring a substrate machine the codebase had built but never connected, hardens a path that has been carrying production traffic, or strips waste from a layer that has been overpaying. The public type surface bumps in a handful of places where the shape change earns its keep many times over (root-anchored token chains, the new scheduled flag on StreamConfig); everything else lands under the hood.

The organizing observation: the substrate already had every primitive it needed, just not always connected to itself. The reliability layer's retransmit code had shipped in isolation and lived in a separate code path the MeshNode receive loop didn't use; v0.27 wires it. The fair scheduler arbitrated relayed traffic but not originating sends; v0.27 adds an opt-in scheduled flag and a new blob-transfer subprotocol that rides it. The capability fold had a query surface but cloned the whole CapabilityMembership payload to extract a NodeId from each match; v0.27 makes the bulk-query path index-driven and the payload clone go away. The MeshOS snapshot publisher fired its change signal every tick regardless of whether anything structural had changed; v0.27 gates the signal on a structural-view diff while keeping the snapshot itself live. The substrate stops paying for what it isn't using, finishes what it's using halfway, and ships a new SDK surface for what's been sitting under it.

Below: the wins, grouped by where they fire.

Reliable streams — retransmit wired end-to-end, full hardening pass

MeshNode reliable streams provided dedup + in-order accounting + flow control, but did not retransmit lost packets — a dropped packet was a permanent gap, and the receiver stalled to the 30-second transfer timeout. The machinery existed (ReliableStream::{on_send, on_nack, get_timed_out, build_nack} were all implemented), but send_on_stream never called on_send, there was no retransmit loop, and the receive path never emitted a NACK. v0.27 connects every piece and then closes the cluster of deficiencies the connection surfaced.

Retransmit wired end-to-end. MeshNode::send_on_stream now registers a RetransmitDescriptor on every reliable send. The receive path emits a NackPayload-carrying SUBPROTOCOL_STREAM_NACK packet whenever an out-of-order arrival opens a gap, coalesced per (session, stream) through a per-mesh drainer. The sender consumes NACKs and resends from its descriptor window. A timeout backstop walks active reliable streams every RTO interval, resending tail packets a lost-final-packet case can't NACK. Verified by a test that drives a multi-MiB transfer under 1-in-10 drop and asserts byte-for-byte completion.

Retransmit window auto-sized to the tx-window. Pre-v0.27 the retransmit window was fixed at 32 entries; a tx-credit window admitting more than 32 in-flight packets silently evicted unacked descriptors and lost them permanently. v0.27 derives max_pending from the tx-window so the invariant tx-window ≤ retransmit-window holds for any window. Eviction-as-silent-loss is now a misconfiguration the runtime won't reach by default.

untracked_evictions surfaced. The eviction counter that should have been a metric for years gets a rate-limited warn! (first occurrence + every 64th) and an untracked_evictions() accessor so production loss is visible in dashboards.

Hard-failure signal on retransmit give-up. A descriptor past max_retries now flags the stream failed and emits a SUBPROTOCOL_STREAM_RESET to the peer; the receiver's blob-transfer engine maps the reset to BlobError and fails its pending read promptly instead of stalling to the caller's 30-second timeout.

Ack-driven pruning of the retransmit window. The retransmit window was never pruned on the happy path — packets lingered until the RTO and spuriously resent. The receiver's next_expected is now piggybacked on StreamWindow grants (now 24 bytes, +ack_seq); the sender prunes via ReliableStream::on_ack. Without this, the new give-up signal turned the spurious resend into a spurious give-up.

Proactive gap NACKs. A receiver whose consumption stalls on a gap can't drive a grant-piggybacked NACK. The retransmit loop now calls collect_gap_nacks per tick so recovery happens within an RTO instead of waiting on the sender's timeout backstop.

Adaptive RTO. RFC 6298 SRTT/RTTVAR with Karn's algorithm, clamped to [10 ms, 2 s]. Replaces the fixed 50 ms RTO that spurious-resent on slow WANs and was sluggish on fast links.

Reno-style congestion window. Slow-start and congestion-avoidance growth, multiplicative decrease on NACK loss, reset-to-floor on timeout. Gates send_on_stream via can_send; no-op on loss-free paths.

Graceful close. New MeshNode::close_stream_graceful waits for the reliable layer to drain (every send acked) or a timeout before closing — serve_chunk's hand-rolled ack-wait close becomes a substrate primitive.

In-order contract clarified. The substrate delivers events in arrival order plus seq; the blob-transfer engine reorders by seq itself; nRPC frames its own order and is fire-and-forget. Reliability::Reliable's docstring previously claimed in-order delivery; v0.27 corrects it and pins the contract at the delivery site. A general in-order buffer is deferred (no consumer needs it).

Channel auth — root-anchored token chains, locally-held publish chains

Root-anchored credentials. Bare-token credentials are replaced with TokenChain everywhere; a presented credential is honored only if it roots at one of the channel's token_roots. The subscribe path carries the chain over the wire end to end (subscribe_channel_with_chain).

Locally-held publish chains. The above broke delegated publishers — a node holding a publish grant via owner → org → this_node could only wrap its leaf token from the local cache, whose issuer is the immediate delegator, not the channel owner; the root-anchor check then failed. v0.27 adds MeshNode::set_publish_chain(channel, chain) so a delegated publisher can install the full chain locally; publish_many consults published_chains first and falls back to the cache-derived single-link form for direct-issued grants. Direct-issued publishers (the common case) need no change.

The publish self-check gates a node against itself, so this is correctness for honest delegated publishers rather than a closed attack surface — a deployment that grants publish rights by delegation silently lost the ability to publish post-audit until v0.27.

Capability fold — bulk-query path goes index-driven

v0.25 moved CapabilitySet's typed fields into a canonical HashSet<Tag> source of truth. The fold's bulk-query path didn't get the corresponding rework — composite_query was still cloning the whole CapabilityMembership payload for every candidate so find_nodes_matching could read the NodeId and throw the rest away. v0.27 closes the gap.

Whole-candidate-set clone removed. The bulk-query path returns NodeIds directly; the payload clone is gone.

Index-driven complex queries. query_model / query_tool were full-scan + clone + re-parse-every-tag operations against ~10k-node folds. v0.27 makes the index seed the candidate set; the post-filter walks the index, not the payload.

Benchmarks (M1 Max, 10k-node fold):

The locking surface is unchanged — concurrent queries already parallelize through the dual-RwLock-read structure that v0.22 shipped. The fix is to make each individual query cheaper, not to touch the locks.

MeshOS — snapshot change-gating, structural-view diff

The MeshOS loop runs publish_snapshot() at the end of every reconcile pass (default tick_interval 500 ms). Pre-v0.27 the call unconditionally stored a fresh MeshOsSnapshot and fired the change signal, waking every Deck consumer twice a second on a totally quiet cluster.

Structural-view signal gate. MeshOsSnapshot derives PartialEq, which makes "publish only when new != last" look obvious — except the snapshot is pervaded by server-projected relative-time fields (age_ms, freeze_remaining_ms, restart-backoff until_ms, migration elapsed_ms, peer since_ms, avoid-list TTLs, recently_emitted[].age_ms) that advance every tick. Equality gating would never suppress; gating the store would freeze the counters. v0.27 keeps the store live (counters tick, the swap is cheap) and gates only the change signal on whether the structural content changed. The alternative — explicit "what changed" telemetry on the producer side — is documented in the design record as deliberately not taken.

Polling → event-driven SDK migration

Several SDK "watch" surfaces ran interval poll loops that re-walked the capability fold every second and emitted a delta. The audit traced four candidates from binding down to substrate source; three turned out to already be push-based (memories watch, tasks watch, redex tail). The fourth — the Deck cohort (watch, watch_timeout, SnapshotStream, StatusSummaryStream) — was the real candidate and now consumes the substrate's existing change signal directly. Latency floor drops from 100 ms to single-digit ms; idle CPU drops to zero.

The substrate-side MeshNode::watch_tools is the load-bearing one — every binding's watchTools / watch_tools / WatchTools forwards its ToolListChange stream, so fixing it once at the substrate fixes all four SDKs.

Datafort blob transfer — fair-scheduler transport, SDK in five languages

The substrate's router has had every primitive bulk byte movement needs — streams, fair scheduling, per-packet priority, per-packet reliability flags, the FIN-driven lifecycle, ChaCha20-Poly1305 encryption — but no convention layer that said "blob transfer uses streams this way." v0.27 ships that layer plus the SDK that exposes it.

Scheduled streams. New StreamConfig.scheduled: bool (default false). When set, MeshNode::send_on_stream enqueues each built packet to the router's scheduler instead of calling socket.send_to directly — so the per-stream weights from set_stream_weight actually apply on originating sends, not just relayed ones. Default-false keeps every existing caller (nRPC streaming, replication) on the direct path.

SUBPROTOCOL_BLOB_TRANSFER. A new subprotocol carries content-addressed fetches over scheduled streams. Discovery rides the capability fold's causal:<hex> advertisement; the requester picks a peer, sends a control packet on a freshly-allocated transfer stream, the server validates possession-of-hash as the capability and chunks the blob into ≤8108-byte reliable events terminated by FIN; the receiver concatenates by arrival order and verifies BLAKE3. The hash is an unguessable 256-bit bearer token — sensitive-content callers must treat it as a secret or layer channel / capability auth above this transport.

Atomic fetch_dir. dataforts::dir::fetch_dir used to write directly under the caller's dest and leave it in a partial state on any mid-fetch failure. v0.27 replaces the direct write with a sibling-temp-path + atomic-rename pattern — the destination either becomes the complete new tree or remains exactly as it was before the call. Failure is a complete rollback with no SDK-side wrapping required.

Streaming send and receive. Pre-v0.27, single-file transfers were bounded by available process memory, not by disk: the receive path assembled the entire blob in one BytesMut before writing, and the send path slurped the whole source file before chunking. v0.27 streams chunk-at-a-time on both sides. The receive path appends each verified chunk to an <out>.partial and renames on commit; the send path reads through a new store_blob_reader substrate helper that hashes and stores each chunk as it's consumed. Large directory leaves (above one chunk) get the same treatment inside fetch_dir. Peak memory drops to roughly one chunk (4 MiB) everywhere; the only remaining cap is the per-chunk TRANSFER_MAX_CHUNK_BYTES ceiling (16 MiB), and total transfer size is now disk-bound. The CLI's recv-blob also gains a determinate byte-progress bar driven from the per-chunk loop.

Transport SDK in five tiers. Rust (net_sdk::transport), C (net.h extensions), Python (pyo3), TypeScript (napi-rs), Go (CGO over C) all gain fetch_blob(blob_ref) -> Bytes, store_dir(adapter, root) -> blob_ref, fetch_dir(adapter, root_blob_ref, dest), plus the DirManifest / DirEntry introspection types. The SDK stays thin — no retry policy, no rollback machinery, no directory-sync primitives. Substrate primitives exposed; applications compose policy above.

Operator CLI — net-mesh transfer and net-mesh typegen

The operator surface grows two new subcommands, both layered over primitives that already shipped.

net-mesh transfer. A first-class CLI for blob and directory transport. Six verbs against a live MeshNode resolved through the standard CliContext:

Progress renders as a determinate byte bar for sized fetches and a spinner for unknown sizes; piping into send-blob from stdin or redirecting recv-blob to stdout lets the verbs compose with the rest of the shell. The commands ship behind the existing cli feature flag — library consumers don't pay the clap build cost.

net-mesh typegen. Code generation from discovered AI tool descriptors. Walks the capability fold for ai-tool:* tags, fetches each matching descriptor's metadata (via tool.metadata.fetch), and emits typed bindings:

net-mesh typegen generate --language ts     --tags weather              --out ./generated
net-mesh typegen generate --language python --tools acme.web-search     --out ./generated

Output is one module per tool. The tool's JSON Schema lowers to TypeScript interfaces (for ts) or Pydantic v2 models (for python); each module also exports a typed call helper (callAcmeWebSearch(mesh, request) for TS, call_acme_web_search(mesh, request) for Python) plus a …Meta constant carrying the descriptor's metadata (tool id, version, streaming flag, tags, description). The bindings work cross-language by construction — every typed call lands on the same wire RPC, so a Python agent calling a TypeScript tool calling a Go server is the same shape as a Rust client calling a Rust server.

The companion verbs make the workflow reproducible:

The substrate-side surface (list_tools, the capability fold, tool.metadata.fetch) all shipped earlier; v0.27 is the operator-facing assembly.

nRPC — recv batching, send batching, QPS scaling diagnosis

The v0.27 nRPC pass started as a nrpc_qps audit and turned into three threads. The diagnosis itself is in tree; the implementation is opt-in.

recvmmsg ingress batching (opt-in). New build feature batched-ingress + runtime MeshNodeConfig::batched_ingress enables the Linux recvmmsg path through BatchedPacketReceiver for the mesh receive loop. The BatchedPacketReceiver itself shipped in v0.21 and was already used by NetAdapter; v0.27 wires it into the MeshNode side. The shared receiver also now hands a whole recvmmsg batch over the channel per syscall instead of one packet per blocking_send. Default off until the c128 measurement justifies the cross-thread channel-hop tax.

sendmmsg batching for relayed traffic. A per-mesh group-by-dest drain on the scheduler send loop coalesces packets to the same destination into a single sendmmsg syscall. Disabled on the originating fast path because nrpc_qps is latency-bound with send-queue depth ≈ 1 — the diagnosis explains why the send-batching premise that worked for saturated one-way blasts doesn't apply to request/response. Where it does apply (concurrent relayed traffic between two peers), the win is real.

QPS scaling diagnosis. nrpc_qps scales c1 → c16 at ~4×, not 16×. The wall is the shared recv loop's single-consumer pipeline (recv → AEAD decrypt → bridge task → fold mutex), not the send path or the handler. The fix is the ack-piggyback protocol, in flight for a future release; v0.27 lands the diagnosis and the bench infrastructure that pins the ceiling.

Breaking changes

Wire format for channel auth. The token-chain change is breaking on the auth path; v0.26 and v0.27 peers don't interoperate on auth-gated channels. Roll the substrate across peers atomically for any deployment that uses channel auth tokens. Direct-issued publishers continue to work via the single-link fallback in publish_many; delegated publishers must install their chain locally (see below).

StreamConfig.scheduled (new field). Default false; the existing call surface is unchanged. Callers that constructed StreamConfig exhaustively without ..Default::default() need to add the field.

SUBPROTOCOL_STREAM_RESET (new subprotocol). Allocated for the hard-failure signal. Receivers must register a handler if they want the prompt-fail behavior; without the handler the legacy 30-second-timeout path still applies.

StreamWindow payload grew to 24 bytes (+ack_seq). Required by ack-driven pruning. The grant codec is versioned by SUBPROTOCOL_STREAM_WINDOW; old peers reading the new grant get a BadRange NACK and fall back to the heartbeat-cycle recovery path — backwards-readable, not backwards-compatible.

MeshNode::set_publish_chain (new method). Required only by deployments using delegated publish grants. Direct-issued publishers need no change.

MeshNodeConfig::batched_ingress (new field, feature-gated). Present only when the crate is built with --features batched-ingress; default false. Linux-only effect.

How to upgrade

1. Roll the substrate across peers atomically if the deployment uses channel auth tokens. v0.27 doesn't handshake cleanly with v0.26 on auth-gated channels.
2. Delegated publishers install their chain via MeshNode::set_publish_chain(channel, chain) at node startup. Direct-issued publishers need no change.
3. Reliable-stream consumers see the new SUBPROTOCOL_STREAM_RESET if they want the prompt-fail behavior on retransmit give-up; otherwise the legacy 30-second timeout is the fallback.
4. Deck consumers see fewer wakeups on quiet clusters; no source change required.
5. Capability fold users get the new query latencies automatically; no API change.
6. Linux deployments wanting recvmmsg rebuild with --features batched-ingress and set MeshNodeConfig::batched_ingress = true. Default behavior is unchanged.

Dependency updates

One major-version bump and a clutch of routine patches.

shlex 1.3.0 → 2.0.1. The only major-version bump in the set. The crate's quoting surface at the substrate's call sites is unchanged in behavior, but downstream consumers that pin shlex = "1" in their own Cargo.toml will need to widen the requirement to resolve cleanly against v0.27.

Routine patch bumps. Alphabetical: bitflags (2.11.1 → 2.12.1), cc (1.2.62 → 1.2.63), ctor (1.0.6 → 1.0.7), generator (0.8.8 → 0.8.9), hyper (1.10.0 → 1.10.1), igd-next (0.17.0 → 0.17.1), log (0.4.30 → 0.4.31), mio (1.2.0 → 1.2.1), redis (1.2.1 → 1.2.2), rustls-native-certs (0.8.3 → 0.8.4), socket2 (0.6.3 → 0.6.4), typenum (1.20.0 → 1.20.1), unicode-segmentation (1.13.2 → 1.13.3), uuid (1.23.1 → 1.23.2), zerocopy and zerocopy-derive (both 0.8.49 → 0.8.50). Cargo.lock carries the exact pinned versions.

Named after the lead single from Billy Idol's 1993 album Cyberpunk — the one he cut as a concept record about networks reshaping how people would work, recorded with a Mac LC III in the booth and a Macromedia Director CD-ROM tucked into the jewel case, panned at release for being too-soon and now read as a marker of the moment the network stopped being a thing other people did. Same wire, same nRPC, same capability fold — but every typed service is now an LLM-callable tool, and the capability subsystem stopped paying for what every other discovery layer is paying for.

One surface every agent can call, and a capability hot path that got back to single-digit nanoseconds

The v0.25 release is the result of two pushes against the same mesh-discovery surface from opposite ends. The agent-facing push exposes every typed nRPC service as an LLM tool — serve_tool / list_tools / watch_tools / call_tool in Rust, Node, Python, and Go, plus format translators for OpenAI / Anthropic / Gemini / MCP so the descriptor lowers directly into whichever provider the agent already runs. The substrate-facing push is a perf audit against the capability subsystem after Phase A.5.N moved CapabilitySet's typed-struct fields into a canonical HashSet<Tag>: a per-tag String::clone in Tag::axis_key() plus a Tag::to_string()-keyed sort in the wire serializer had quietly turned a 3.7 ns match_min_memory filter into a 46 µs one. Four targeted fixes recovered the regression; the perf audit doc lands in tree alongside the release.

The release's organizing observation: discovery should be free in the hot path and cheap to author at the edges. The capability fold already aggregates every node's capabilities — agent discovery just walks it. The tag-set source-of-truth pattern is the right architecture, but allocating a String per tag per predicate match isn't its tax to pay.

Where v0.25 lands against the rest of the service-discovery field

In-process capability-filter evaluation in v0.25 sits 3–7 orders of magnitude below the published latencies of the network-coordinated discovery systems the field treats as fast:

Caveat — apples-vs-oranges: the v0.25 numbers measure in-process predicate evaluation against capability announcements already gossiped into the local fold. Consul / etcd / Kubernetes DNS are answering "where is service X across the cluster" with a network round-trip and (usually) a consensus quorum read. They aren't doing the same job. The fair comparison is the in-mesh agent scheduling loop: once announcements are in your fold (Net does that propagation via the same gossip path every other capability rides), filtering and dispatching against them is genuinely four to seven orders of magnitude faster than the registries an agent author would otherwise reach for.

External sources for the published latencies in the table: Consul DNS perf thread, Consul DNS perf issue #1535, Consul server resource requirements, etcd recommended practices (OKD), Kubernetes DNS ndots:5 latency, mDNS / DNS-SD discovery.

Below: the wins, grouped by where they fire.

AI tool calling — every typed nRPC service is an LLM-callable tool

NRPC_AI_TOOL_CALLING_AND_AGENT_DX.md (the plan shipping alongside this release) makes the bet that tool calling is what nRPC already does — "send a JSON object to a named handler, await a JSON response, optionally stream chunks" — with three gaps: metadata so a model can decide when/how to call, a server-streaming primitive matching the unary call_service, and a structured event envelope for streaming output. v0.25 closes all three and ships the agent-author surface across every binding.

One identifier, one source of truth. A tool registered as web_search IS the nRPC service at channel nrpc:web_search.requests IS the announcement carrying the ai-tool:web_search capability tag. No separate registry, no mapping table. Plain rpc.serve("x", handler) continues to register a service without the ai-tool:* tag — invisible to list_tools(). The serve_tool / tool({...}) / @tool opt-in is what makes a service agent-discoverable; operators retain control.

Discovery is capability-fold-native, not RPC-fanout. The capability fold already aggregates ToolCapability instances across every node. list_tools(matcher) walks the fold in-memory and returns ToolDescriptors carrying id + version + node_count + small metadata. Heavy fields (oversized JSON Schemas) fall back to an on-demand tool.metadata.fetch RPC, which serve_tool auto-installs on the host the first time it's called. Subnet visibility, capability auth, region filtering — all inherited from the existing fold + TagMatcher plumbing.

Streaming tools share one event envelope. ToolEvent is a tagged JSON enum every streaming handler emits per chunk:

• start { tool_id, call_id, metadata? } — fires once on open.
• progress { pct?, message? } — coarse progress for spinners.
• delta { data } — partial output (model tokens, file bytes, log lines).
• result { data } — terminal full result; client sees one on success.
• error { code, message, details? } — terminal failure with structured detail.

Unary tools synthesize a single result envelope under the hood. The convention lets every adapter (OpenAI / Anthropic / Gemini / MCP / Hermes / custom) lower envelopes into the framework's native streaming protocol without per-pair negotiation. Two synthesized error shapes round out the contract: missing_terminal on the streaming caller when the server closed without a result/error chunk, and handler_error on the streaming server when the handler raised mid-stream. Both are part of the T-2 JSON byte-equality fixture so adapters can match on the code reliably.

serve_tool is atomic w.r.t. observable mesh state. Either all of (handler registration, capability-fold publish, nrpc:<tool_id> tag, ai-tool:<tool_id> tag, auto-installed tool.metadata.fetch if first) succeed, or none do. Drop on the returned handle reverses all four.

Cross-language by construction. The wire is unchanged: call_tool is call_service with the typed wrapper, call_tool_streaming rides the new call_service_streaming substrate primitive (mirror of call_service returning an RpcStream). A Python Hermes agent calling a Go-hosted database tool calling a TypeScript browser tool is transparent over the existing nRPC wire. The T-1 cross-language test pins byte-equality of every format translator output (to_openai_tool / to_anthropic_tool / to_gemini_tool / to_mcp_tool) across Rust / Node / Python / Go for every fixture descriptor.

Surface by language:

Format translators ship in one package per language. net-mesh-tools (pip) carries formats/{openai,anthropic,gemini,mcp} submodules; @net-mesh/tools (npm) carries formats/{openai,anthropic,gemini,mcp} submodules. Each translator is a small pure function from ToolDescriptor → provider tool-array entry, plus a reverse lower_tool_call(call) -> CallSpec for going from a provider's tool_use block back into a typed nRPC call. No transitive dep on any provider SDK — users wire the translator output into their OpenAI / Anthropic / Hermes / framework-of-choice client themselves.

No wire ABI bump for unary tool calls. Streaming tools use the new call_service_streaming substrate primitive; the wire shape of an individual stream is unchanged from call_streaming today. ToolEvent envelopes are JSON-encoded chunks on existing streams. NET_RPC_ABI_VERSION stays at 0x0004.

Capability perf — closing the Phase A.5.N regression cliff

PERF_AUDIT_2026_05_28_CAPABILITY.md (the audit doc shipping alongside this release) compared two M1 Max criterion runs and found that the Phase A.5.N migration — which moved CapabilitySet's typed HardwareCapabilities / Vec<ModelCapability> / etc. fields into a canonical HashSet<Tag> source of truth — had silently regressed eight capability microbenchmarks by 100× to 1,200,000×. The headline cases:

The migration was the right architectural call — tag-set as source of truth makes the diff / aggregation / federated-predicate stories cohere — but four hot-path costs piggybacked on the change. v0.25 closes all four:

Fix 1 — cheaper decoder sort (capability.rs). CapabilityViews::sorted_tags() and the three From<&CapabilitySet> projection impls were calling sort_by_key(|t| t.to_string()) — a fresh String allocation per comparison, ~150 allocations per views() call for a 35-tag set. v0.25 adds a separate decoder_sorted_tag_vec using Tag's derived Ord via sort_unstable(). The original sorted_tag_vec stays in place for the wire serializer (signed-announcement bytes need the Tag::to_string() canonical order for cross-version signature verification) — only the decoder paths switch.

Fix 2 — tag-direct fast paths in CapabilityFilter::matches. Single-field hardware predicates were forcing a full HardwareCapabilities decode (sort + per-tag axis_key parse + per-field value.parse()) just to read one tag. v0.25 adds CapabilitySet::axis_value(axis, key) -> Option<&str> (pub(crate)) and rewrites matches() so min_memory_gb / gpu_vendor / min_vram_gb probe the tag set directly the way has_gpu() already did. The views() call is now lazily guarded behind min_context_length and require_modalities — predicates that don't set those fields never decode.

Fix 3 — drop axis_key()'s per-tag String::clone (has_model / has_tool and 14 hot-path callers). Tag::axis_key() returns an owned TagKey containing a cloned key string. Every caller that iterated a tag set through it was paying ~35 String allocations per call. v0.25 adds Tag::axis_key_ref() -> Option<(TaxonomyAxis, &str)> and migrates the five view decoders (hardware_from_tags, software_from_tags, resource_limits_from_tags, models_from_tags, tools_from_tags), the five is_*_owned_tag predicates, Predicate::Exists, match_axis_tag, RequiredCapability::AxisKey, and MatchKey::{Axis, AxisKey} in capability aggregation. axis_key() is kept for callers that genuinely need an owned TagKey (diff.rs collects into HashSet<TagKey>).

Fix 4 — postcard compact codec for CapabilitySet. to_bytes is serde_json::to_vec and isn't going anywhere on the wire (signed-announcement byte stability + cross-version peer compat). v0.25 adds CapabilitySet::to_bytes_compact that emits 0x01 <postcard payload>, and from_bytes sniffs the first byte (b'{' → JSON, 0x01 → postcard, anything else → None) so receivers on this code accept both formats. The actual win came from serialize_tags_sorted branching on serializer.is_human_readable(): JSON keeps the canonical sort, postcard skips it (no signing on this path; the only consumer is a from_bytes that reconstructs the same HashSet regardless of element order).

Benchmarks (Windows host, same-run before/after per fix):

All 4137 lib tests pass (3 new tests pin the compact codec round-trip and the unknown-format rejection). Wire format is unchanged for any current peer: to_bytes is still JSON, the wire serializer keeps Tag::to_string() sorting, signed announcements stay byte-stable across versions. The compact codec is opt-in via the new to_bytes_compact — flipping the default writer to compact is a separate, deliberate rollout commit (every receiver must be on v0.25 first).

What's not in this release. CapabilityAnnouncement::to_bytes_compact is deferred. The struct has six #[serde(skip_serializing_if = ...)] fields (signature, hop_count, reflex_addr, the three allowed_* lists) whose omission is load-bearing for pre-M-1 / pre-v0.4 signed-byte compat, and postcard's positional encoding can't reconstruct an omitted field. A separate canonicalized wire struct is the right fix; tracked in the audit doc as a follow-up.

Test hygiene

• Two new audit docs shipped in tree. docs/plans/NRPC_AI_TOOL_CALLING_AND_AGENT_DX.md covers the agent surface (eight locked decisions, phasing, per-binding status); docs/misc/PERF_AUDIT_2026_05_28_CAPABILITY.md covers the capability perf pass (headline regressions, root causes with file:line pointers, ranked fixes with risk/touch columns, before/after numbers per fix).
• T-1 cross-language tool-format byte-equality, ratcheted across all four bindings. The tests/cross_lang_tool_formats/golden_vectors.json fixture is consumed by Rust / Node / Python / Go verifiers in lockstep — adding a new descriptor / lower case / error case means updating all four. Drift surfaces as CI failure, not a runtime surprise.
• T-2 ToolEvent envelope round-trip, same posture across all four bindings. JSON tag-form ({"type": "start", ...}) deserializes + re-serializes byte-equal for every variant + every optional-field combination listed in tests/cross_lang_tool_formats/tool_event_vectors.json. The synthesized Error { code: "missing_terminal", ... } shape is part of the fixture so adapters can match on the code reliably.
• Capability perf — all 195 capability lib tests pass at every commit in the perf series (bd58b90b, 20dba467, 00aa6f75, 2cb28f7d). Three new tests pin the compact codec: compact_wire_format_round_trips_and_interops_with_json, from_bytes_rejects_unknown_format_tag, announcement_* (JSON-only, since the announcement compact path is deferred).
• cargo clippy --features meshos,deck,aggregator --all-features --all-targets -- -D warnings clean. The strict floor from v0.20.2 stays armed; the clippy::useless-vec lint that landed in Rust 1.95 caught one pre-existing vec![] in the capability test suite — fixed in deebf93e.
• cargo doc --features meshos,deck,aggregator --no-deps clean under RUSTDOCFLAGS="-D warnings". All new ToolDescriptor / ToolEvent / tool::* intra-doc links resolve; the compact-codec docstrings inline 0x01 instead of linking to the private COMPACT_FORMAT_TAG constant (94c87537).

Breaking changes

tool cargo feature on net-mesh

New optional tool = [] feature gates the tool.rs module + ToolEvent wire type. The Node / Python / Go binding default feature sets include tool — most users see no change. Direct net-mesh consumers who want serve_tool / call_tool need cargo add net-mesh --features tool.

The wire-level pieces this composes against (ToolCapability in behavior::capability, the capability fold, call_service_streaming) compile unconditionally so peers without the feature still exchange ToolCapability announcements.

call_service_streaming is a new substrate primitive

Mesh::call_service_streaming mirrors Mesh::call_service returning an RpcStream instead of a single response. Capability-routed + auth-gated through the same path as the unary variant. Every streaming tool client (Rust / Node / Python / Go) depends on it; downstream consumers who built their own streaming client on top of capability fold lookups can switch to this primitive.

tool.metadata.fetch is a new reserved RPC service name

Auto-installed by serve_tool on the first tool registration per node. Downstream consumers MUST NOT register an unrelated handler under this name — the auto-install asserts the slot is theirs and panics on collision. The reserved-name boundary is documented in docs/AGENT_TOOLS.md.

CapabilitySet::from_bytes accepts both JSON and the compact (0x01-prefixed postcard) format

Behavior-preserving for every JSON caller. A byte stream whose first byte is neither b'{' nor 0x01 now returns None instead of attempting a JSON parse — previously the JSON parser would have returned its own None, so the observable contract is unchanged. The first-byte sniff is documented on from_bytes.

CapabilitySet::to_bytes_compact is a new opt-in serializer

Default to_bytes is still JSON; flipping the default writer to compact is a separate rollout decision (every receiver must be on v0.25 first or it can't decode the new bytes). The compact codec is for local-only callers and a future deliberate wire-format flip.

Tag::axis_key_ref is a new method on Tag

Additive. axis_key() is unchanged (returns owned TagKey); axis_key_ref() returns Option<(TaxonomyAxis, &str)> without cloning. Hot-path iteration callers SHOULD prefer the borrowing variant — the cloning variant is only worth it when the caller actually needs an owned TagKey (e.g. collecting into HashSet<TagKey> for diff).

serialize_tags_sorted now branches on serializer.is_human_readable()

Internal-only break. JSON callers continue to get the sorted canonical form (signed-announcement byte stability); postcard callers skip the sort. No observable change unless a downstream consumer was relying on the Tag::to_string() order in a non-human-readable serializer output — that wasn't a supported contract.

CapabilityAnnouncement does NOT have a to_bytes_compact

Deferred. The struct's six #[serde(skip_serializing_if)] fields are required for pre-M-1 / pre-v0.4 signed-byte cross-version compat, and postcard's positional encoding can't tolerate omitted fields. A separate canonicalized wire struct is the right path; not in this release.

gpu_vendor_str is now pub(crate) in tag_codec.rs

Internal-only. Required by CapabilityFilter::matches's tag-direct vendor probe (constructs the expected Tag::AxisValue from the matcher's GpuVendor for O(1) HashSet::contains). No public surface.

ai-tool:<tool_id> capability tag is reserved

Substrate emits this automatically on every serve_tool registration. Downstream code SHOULD NOT emit ai-tool:* tags by hand — list_tools() filters on this prefix and a hand-emitted tag without the matching nrpc:<tool_id> service registration would surface as a phantom tool with no handler.

How to upgrade

1. Rust consumers — update the dependency to 0.25. No source changes required unless you (a) want to author or call tools (serve_tool / call_tool — enable the tool feature), or (b) iterate a tag set through Tag::axis_key() in a hot path (switch to axis_key_ref() for the per-call allocation saving).

2. Agent authors — pick your binding and follow docs/AGENT_TOOLS.md. Rust: Mesh::serve_tool<Req, Resp>(...) (the #[tool] proc macro is the follow-up; runtime APIs are usable as-is). Node: tool({ name, description, schema, handle }) with Zod schemas. Python: @tool decorator on a Pydantic-typed handler (sync or async). Go: net.RegisterTool[Req, Resp](rpc, descriptor, handler). Discovery is the same shape in every binding: list_tools(matcher?) returns descriptors, watch_tools(matcher?) streams ToolListChange::{Added, Removed, NodeCountChanged}.

3. Agent authors using OpenAI / Anthropic / Gemini / MCP — install the format package. Python: pip install net-mesh-tools; import from net_mesh.tools.formats.{openai,anthropic,gemini,mcp}. Node: npm install @net-mesh/tools; import from @net-mesh/tools/formats/{openai,anthropic,gemini,mcp}. Each translator is a pure function from ToolDescriptor → provider tool-array entry; the reverse lower_<provider>_tool_call(call) returns a CallSpec you pass into call_tool / call_tool_streaming. No transitive provider-SDK dep — wire the translator output into your existing OpenAI / Anthropic client.

4. Operators with capability-filter throughput pressure — expect the µs→ns recovery to land out of the box. No config knobs to flip. The four perf fixes are unconditional on the substrate path. Re-run cargo bench --bench net -- "capability_(filter|set)" to confirm against your hardware; the audit doc has the same-host before/after numbers for cross-checking.

5. Operators with binary-size budgets — tool is opt-in. Direct net-mesh consumers who don't want the agent surface keep their default feature list. Binding artifacts: the binding's tool feature flag is on by default in Node / Python / Go; downstream consumers who don't want it pass --no-default-features and enumerate the features they do want.

6. Downstream consumers caching capability bytes — opt into to_bytes_compact when you control both sides. Local persistence, intra-process caches, and any storage path where the byte format is yours to choose can switch to the compact codec for the ~27× serialize win and ~10× roundtrip win. Wire callers (mesh.rs, swarm.rs, proximity.rs, the CLI announce path) should NOT switch until the entire fleet is on v0.25 — receivers on this release accept both formats, but receivers on v0.24 can't decode 0x01-prefixed bytes.

7. Operators on mixed-version fleets — wire format is unchanged. CapabilitySet::to_bytes is still JSON, CapabilityAnnouncement::to_bytes is still JSON, serialize_tags_sorted still produces the Tag::to_string() canonical order for JSON serializers, signed-announcement bytes are byte-stable across versions. v0.24 and v0.25 peers handshake cleanly.

8. Downstream Go binding consumers — ABI version unchanged. NET_RPC_ABI_VERSION stays at 0x0004. The Go tool surface (net.RegisterTool, net.RegisterStreamingTool, net.CallToolStreaming, net.ListTools, net.WatchTools) is additive.

9. CI — no config change required. Strict clippy floor stays armed (the new clippy::useless-vec in Rust 1.95 caught one pre-existing test-fixture site, fixed in this release); rustdoc warnings stay denied; the cross-language tool-format byte-equality fixture is the new CI gate. Adding a new descriptor / lower case / error case in tests/cross_lang_tool_formats/golden_vectors.json must be done in lockstep across all four binding verifiers.

10. Operators — bump the binary. Pre-built net-mesh, net-deck, net-aggregator-daemon archives land for every supported target (Linux x86_64 / aarch64, macOS x86_64 / aarch64, Windows x86_64). Wire format is unchanged from v0.24.

Named after the Rolling Stones' 1969 opener on Let It Bleed — the one Keith Richards wrote during a thunderstorm at his Robert Fraser flat, the one Merry Clayton recorded in a single overnight session. Same wire, same semantics, same surface — gimme shelter, or I'm gonna fade away.

Three waves, one substrate primitive, every binding finally idiomatic

The v0.23 release is the result of three planning passes against the user-facing nRPC + Python surfaces. The first wave — Slice 2 + Slice 1 from NRPC_STREAMING_PARITY_AND_GO_BINDING.md — closes the streaming-typed gap on Node and Python (client-streaming + duplex typed wrappers + observer + metrics) and ships a Go typed binding from day one with the same shapes. The second wave — NRPC_V3_OBSERVER_MPSC_AND_CANCELLATION.md — promotes cancellation to a substrate primitive (Mesh::reserve_cancel_token / Mesh::cancel(token)) and routes every binding through it instead of letting three parallel binding-local cancel registries diverge further; in the same wave, every observer hook gets a bounded mpsc + drop counter so a slow callback can no longer pin the substrate's dispatch thread. The third wave — PYTHON_ASYNC_SDK_SIDE_BY_SIDE.md — adds Async-prefixed siblings to every Python class that today calls runtime.block_on(...), so asyncio-native consumers (FastAPI, LangGraph, an aiohttp sidecar, an asyncio.gather fan-out) don't have to fall back to a ThreadPoolExecutor.

The release's organizing observation: every binding had been growing its own shim layer to compensate for substrate gaps. The napi binding owned a cancel_registry: HashMap<u64, AbortHandle> with its own CR-13 race fix. The pyo3 binding owned a Cancellable pyclass with its own close_notify + tokio::select! pattern. The Go FFI owned a cancel_registry with its own Q18 orphan-TTL GC. Three implementations of the same idea, each with its own subtle bug fixes. v0.23 promotes the idea once, at the substrate, and the bindings stop holding their own state. Same shape for the observer machinery: three "callbacks must be cheap" footguns collapse into one bounded queue at the substrate boundary with a single drop counter that surfaces on every language's metricsSnapshot / metrics_snapshot / MetricsSnapshot. And the Python async surface lifts every block_on site once instead of asking users to wrap each binding call in a thread pool.

Below: the wins, grouped by where they fire.

Streaming parity across Node, Python, Go (one typed shape, four call shapes)

Before v0.23, the typed-nRPC matrix had holes. Node + Python had unary + server-streaming typed wrappers but no client-streaming or duplex typed surface. Go had no typed wrapper at all. v0.23 fills the matrix.

Node TypedMeshRpc.serveClientStream + callClientStream + TypedClientStreamCall. Mirror of the Rust SDK's serve_rpc_client_stream_typed + call_client_stream_typed. JSON encode on send, JSON decode on finish; encode failures throw nrpc:codec_encode, decode failures throw nrpc:codec_decode, all through the existing classifyError mapping. The server-side handler shim decodes each chunk and surfaces a malformed-request as RpcAppError(NRPC_TYPED_BAD_REQUEST, ...) so callers observe typed Application status instead of generic Internal.

Node TypedMeshRpc.serveDuplex + callDuplex + duplex typed wrappers. TypedDuplexCall<Req, Resp> with send / finishSending / next / intoSplit / close; TypedDuplexSink<Req> + TypedDuplexStream<Resp> for the split halves; TypedResponseSink<Resp> for the server side. Handler signature is the JS-idiomatic (stream, sink) => form, not the napi-binding's destructured [stream, sink] tuple — the typed wrapper destructures before invoking the user handler.

Python TypedMeshRpc.serve_client_stream + call_client_stream + TypedClientStreamCall. Same shape as Node. Sync iterator + context-manager (__enter__ / __exit__); decode failure on a chunk raises RpcCodecError and closes the underlying stream. Handler signature is (stream: TypedRequestStream) -> Resp; decode failure on the first request chunk surfaces as RpcAppError(NRPC_TYPED_BAD_REQUEST, ...).

Python TypedMeshRpc.serve_duplex + call_duplex + duplex typed wrappers. TypedDuplexCall / TypedDuplexSink / TypedDuplexStream / TypedResponseSink; __next__ raises StopIteration on EOF, decode failure closes the call and raises RpcCodecError.

TypedMeshRpc.setObserver + metricsSnapshot on Node and Python. The raw napi + pyo3 MeshRpc classes gain setObserver(handler) + metricsSnapshot() (and set_observer(callable) + metrics_snapshot() for Python); the typed wrappers expose the same surface. RpcCallEvent is a JS interface / Python dataclass with tagged-union status (Ok / Error(message) / Timeout / Canceled), per-call latency, request/response byte counts, and direction. The mid-call swap is atomic via the substrate's ArcSwapOption<RpcObserverHandle>.

Go typed binding — full surface in one file. New bindings/go/net/mesh_rpc_typed.go ships every shape from day one:

• TypedCall[Req, Resp] + TypedCallService[Req, Resp] + TypedServe[Req, Resp] — unary, mirror of rpc.call<Req, Resp>(...) ergonomics with one extra positional argument (the *TypedMeshRpc itself), free-function shape because Go forbids type parameters on methods.
• TypedCallStreaming[Req, Resp] + *TypedRpcStream[Resp] — server-streaming with Recv() returning (Resp, error) + ErrStreamDone sentinel on EOF.
• TypedCallClientStream[Req, Resp] + *TypedClientStreamCall[Req, Resp] + TypedServeClientStream[Req, Resp] — client-streaming.
• TypedCallDuplex[Req, Resp] + *TypedDuplexCall[Req, Resp] + Split() halves + TypedServeDuplex[Req, Resp] — duplex.
• (*TypedMeshRpc).SetObserver(handler) + MetricsSnapshot() — observer + metrics through the new net_rpc_set_observer + net_rpc_metrics_snapshot FFI symbols.

RpcAppError(code, detail) minted through NewRpcAppError(NrpcTypedBadRequest, ...) / NewRpcAppError(NrpcTypedHandlerError, ...) matches the canonical nrpc:app_error:0x<code>:<body> shape; the Rust binding's parse_js_app_error reuses the same parser for Go consumers. The existing *RpcError Go type classifies codec failures as RpcKindCodecEncode / RpcKindCodecDecode — no RpcError changes needed.

Cross-language streaming round-trip test. tests/cross_lang_nrpc/ grows golden vectors for client-stream + duplex Application-status round-trips; Rust-side reference asserts the typed-handler-raising-RpcAppError shape lands at the caller as the expected wire-level error.

Observer dispatch as bounded mpsc + drop counter

The v1 typed-nRPC observer contract was "callbacks must be cheap; the substrate dispatch thread blocks until your callback returns." That contract lasts about a week in production before a user wires a Prometheus exporter or a disk-flushing log sink into setObserver and mesh-wide RPC latency spikes. v0.23 fixes the contract.

Bounded-mpsc per mesh, drop counter as a monotonic u64. Each binding now wires a 1024-event bounded mpsc between the substrate's dispatch path and the observer worker. Substrate on_call does try_send; full → atomic-counter increment, never blocks. The dispatch thread's per-event cost drops from "TSFN Mutex acquire" (napi) / "fresh spawn_blocking per event" (pyo3) / "synchronous C function pointer call" (Go FFI) to "atomic counter inc on a single AtomicU64."

One worker per binding installs the observer. The worker drains the receiver and pumps each event to the registered consumer: napi → TSFN; pyo3 → GIL-acquired Python callable; Go FFI → C function pointer. One worker = serialized callback invocation, matching each language's natural threading model. The worker dies when the sender drops (i.e. when setObserver(None) is called and the observer Arc is released).

observerDroppedTotal / observer_dropped_total / ObserverDroppedTotal on every snapshot. Process-global AtomicU64; reads-and-leaves (monotonic) for Prometheus exporter ergonomics. Surfaces as a top-level field on every binding's RpcMetricsSnapshot. Go consumers additionally get a net_rpc_observer_dropped_total() -> u64 FFI symbol so they can read the counter without paying the JSON-decode cost on the snapshot path.

Cortex-side consolidation. The mpsc plumbing lives in the substrate's cortex module (ObserverChannel<E> + OBSERVER_BUFFER_CAPACITY constant) — one centralized implementation, three thin per-binding wrappers. A future tunable on the buffer capacity is a one-place change instead of three.

Arc<RpcCallEvent> through the channel. Observer events now flow as Arc<RpcCallEvent> from the substrate's emit site, deferring the per-binding POD-conversion work to the drain worker. The dispatch thread allocates the event once, increments the Arc, and moves on; the worker pays the per-binding conversion cost on a non-hot-path thread.

Cancellation as a substrate primitive

Three bindings, three cancel registries. Promoted to one substrate primitive in v0.23.

CallOptions::cancel_token: Option<u64> + Mesh::reserve_cancel_token + Mesh::cancel(token). Reserve a token from the mesh; pair with cancel(token) from any thread to abort the in-flight call. Honored uniformly by call / call_service / call_streaming / call_client_stream / call_duplex — the substrate registers the token's abort handle at construction and removes it on resolution. Drop-on-cancel emits CANCEL on the wire via the existing per-call-shape Drop impls (UnaryCallGuard::Drop, ClientStreamCallRaw::Drop, DuplexCallRaw::Drop).

Per-mesh cancel_registry. parking_lot::Mutex<HashMap<u64, CancelEntry>> keyed by token. CancelEntry carries cancelled: bool (CR-13: cancel before register), handle: Option<AbortHandle> (unary + streaming construction), close_notify: Option<Weak<Notify>> (streaming post-construction), marked_at: Option<Instant> (Q18: orphan TTL). Lifted from the napi binding's existing pattern with the Go FFI's orphan-TTL GC (default 120s) merged in.

Race-safe across the reserve-then-call gap. A cancel that arrives BEFORE the call's abort handle is registered (the gap between reserve and call construction) latches a cancelled = true flag on the orphan entry; when the call later registers, it observes the flag and aborts immediately. Mirrors the napi binding's CR-13 fix at the SDK layer once instead of three times.

Binding migration: thin pass-through over the substrate primitive.

• napi. lock_cancel_registry() and NEXT_CANCEL_TOKEN: AtomicU64 are deleted. reserveCancelToken / cancelCall napi methods now delegate to Mesh::reserve_cancel_token / Mesh::cancel. callClientStream and callDuplex populate opts.cancel_token from the incoming CallOptions. The typed wrapper drops stripSignal for streaming entries and wires wireAbortSignal end-to-end.
• pyo3. Cancellable.__init__ reserves a token from the mesh; Cancellable.cancel() calls mesh.cancel(token). call_client_stream / call_duplex extract opts['cancel'] and populate CallOptions::cancel_token. The Notify-based close_notify path on PyClientStreamCall / PyDuplexCall becomes an internal implementation detail; the substrate registers a Weak<Notify> against it.
• Go FFI. The file-local cancel_registry is deleted. net_rpc_reserve_cancel_token / net_rpc_cancel_call become pass-throughs. New cancellable FFI variants — net_rpc_call_client_stream_cancellable + net_rpc_call_duplex_cancellable — populate opts.cancel_token and forward to the SDK. The Go typed wrapper's TypedCallClientStream / TypedCallDuplex now propagate ctx.Context through unchanged; the raw layer honors it.

Typed-wrapper pass-throughs. Node AbortSignal for streaming, Python Cancellable for streaming, Go ctx.Context for streaming — all wired end-to-end. The v1-era "v1: close()-only" caveat is gone from every streaming-entry docstring.

SDK-level cancel-contract integration tests. tests/integration_mesh_cancel.rs pins the contract before any binding depends on it: cancel_unary_mid_flight_emits_cancel_on_wire, cancel_streaming_mid_drain_emits_cancel, cancel_client_stream_mid_send_emits_cancel, cancel_duplex_mid_send_emits_cancel, cancel_before_construction_aborts_cleanly, cancel_after_resolution_is_noop, cancel_zero_token_is_noop, orphan_ttl_gc_evicts_unused_reservations. The bindings then test only their pass-through layer.

Cancellation cookbook (cross-binding). Documented in the v3 plan and surfaced in the per-binding README: Node AbortSignal, Python Cancellable, Go context.Context — three idiomatic surfaces, one substrate primitive, the same wire-level outcome. Power users in every language can also reserve tokens directly via the raw FFI surface for cross-call cancel sharing.

Python async SDK — side-by-side Async* surface for every I/O class

The pyo3 binding spans 16 modules and ~141 runtime.block_on(...) call sites. Every one of those took a Python thread + a tokio worker hostage for the call's duration; in an asyncio-native consumer that pattern serializes what should be concurrent and burns the application's thread budget. v0.23 adds Async-prefixed siblings for every class with async-worthy I/O. Existing sync API is unchanged.

AsyncNetMesh + AsyncNetStream. Shared MeshNode with the sync NetMesh — AsyncNetMesh(mesh) constructs against the existing peer-connection state without re-handshaking. connect / accept / stream enumeration return awaitables; peer_count / node_id / public_key are sync (in-memory reads).

AsyncMeshRpc — full raw client + server.

• call / call_service / find_service_nodes (unary + service-discovery unary + local lookup).
• call_streaming → AsyncRpcStream with __aiter__ + __anext__ (server-streaming).
• call_client_stream → AsyncClientStreamCall with awaitable send / finish (client-streaming).
• call_duplex → AsyncDuplexCall / AsyncDuplexSink / AsyncDuplexStream (duplex + split halves).
• serve / serve_client_stream / serve_duplex — accept EITHER sync def or async def handlers, detected via inspect.iscoroutinefunction at register time. Sync handlers run on the substrate's spawn_blocking path; async handlers run as coroutines on a dedicated dispatcher event loop so the tokio worker can drive them without a Python loop on its own thread.

AsyncTypedMeshRpc + every typed streaming companion. AsyncTypedMeshRpc, AsyncTypedRpcStream, AsyncTypedClientStreamCall, AsyncTypedDuplexCall, AsyncTypedDuplexSink, AsyncTypedDuplexStream, AsyncTypedRequestStream, AsyncTypedResponseSink — JSON-encode/decode the same way the sync wrappers do; only difference is await self._raw.foo(...) vs self._raw.foo(...).

Wave T2 production essentials. AsyncNetDb (get/put/delete/list/batch_put), AsyncRedex + AsyncRedexFile + AsyncRedexTailIter (async for evt in file.tail(from_seq)), AsyncMemoriesAdapter + AsyncMemoryWatchIter, AsyncTasksAdapter + AsyncTaskWatchIter (every push-stream becomes a PEP-525 async iterator). AsyncDaemonRuntime + AsyncDaemonHandle + AsyncMigrationHandle (daemon spawn / migrate / wait). AsyncMeshBlobAdapter + module-level async_blob_publish / async_blob_resolve.

Wave T3 operator / specialty. AsyncDeckClient + AsyncSnapshotStream + AsyncStatusSummaryStream + AsyncAdminCommands + AsyncIceCommands. AsyncMeshOsDaemonSdk + AsyncMeshOsDaemonHandle (async receive / publish_log / graceful_shutdown). AsyncMeshQueryRunner.execute (async-RPC backed; the query AST classes stay sync — they're pure data builders). AsyncFoldQueryClient + AsyncRegistryClient.

Shared tokio runtime + shared MeshNode. Sync and async classes share one runtime per process and one Arc<MeshNode> per NetMesh. A server registered via MeshRpc.serve(...) is callable from AsyncMeshRpc.call(...); an async def handler registered via AsyncMeshRpc.serve(...) is callable from MeshRpc.call(...). Same wire, same identity, same cap-index entries.

Async substrate-cancel propagation via the v3 primitive. Every async I/O method mints a substrate cancel token, attaches an asyncio-cancel listener that calls Mesh::cancel(token), detaches on call resolution. asyncio.wait_for(async_rpc.call(...), timeout=0.1) on a long-running call surfaces asyncio.TimeoutError on the caller and Cancelled on the server's observer — same shape as Node's AbortSignal and Go's ctx.Context, just driven by the Python task lifecycle.

Server-side dispatcher event loop. async def server handlers dispatched from a tokio worker need a Python asyncio loop to drive the coroutine. The bridge lazily spawns a single daemon Python thread running asyncio.run_forever() on a fresh loop and routes every handler coroutine through pyo3_async_runtimes::into_future_with_locals(&dispatcher_locals, coro) — one loop per process, serialized GIL acquisition on the drain worker, no per-handler thread costs.

pyo3-async-runtimes (tokio backend) as a default dep. Bridges init-once via init_with_runtime(&runtime) in the pymodule init. Wheel grows ~50 KB; no feature flag bifurcation.

Test surface — TX cross-cutting matrix. tests/test_async_interop.py pins the (sync | async) caller × (sync | async) server matrix on the unary shape (4 tests; full-shape coverage lands incrementally as a follow-up). AsyncNetMesh(mesh) shared-handshake invariant pinned (no re-handshake when constructed against an already-connected mesh). Per-module smoke tests cover the T2 + T3 surfaces.

Test hygiene

• Lib suite continues to expand. New tests across the v3 cancel contract (integration_mesh_cancel.rs — every call shape, the orphan-TTL GC, the CR-13 race), the bounded-mpsc drop counter (per-binding under-load tests in napi + pyo3 + rpc-ffi), the cross-language streaming round-trip (tests/integration_nrpc_cross_lang_streaming.rs — client-stream + duplex + observer firing), and the Python async surface (tests/test_async_interop.py + per-module smoke tests for every Async* class).
• Cross-language wire contract test. Every binding now also pins the canonical observer event shape and metrics-snapshot envelope (observer_dropped_total field present, abi_version_expected = 4). A binding that drifts fails its compatibility test.
• ABI version bump to 0x0004 (rpc-ffi). Additive — existing 0x0003 symbols stay unchanged; new symbols are net_rpc_call_client_stream_cancellable, net_rpc_call_duplex_cancellable, net_rpc_set_observer, net_rpc_metrics_snapshot, net_rpc_observer_dropped_total.
• cargo clippy --features meshos,deck,aggregator --all-features --all-targets -- -D warnings clean. Strict floor from v0.20.2 stays armed.
• cargo doc --features meshos,deck,aggregator --no-deps clean under RUSTDOCFLAGS="-D warnings". Intra-doc links across the new substrate cancel API + the per-binding cancel cookbook + the Python async surface all resolved.
• Codecov coverage unchanged in posture — ~90% substrate, informational on CI status.

Breaking changes

NET_RPC_ABI_VERSION bumps from 0x0003 to 0x0004

Additive symbol additions only; existing 0x0003 functions are unchanged. Downstream Go binding consumers compiled against the pre-bump version panic at process init via the ExpectedABIVersion check at bindings/go/net/mesh_rpc.go:586-595. Override with NET_RPC_SKIP_ABI_CHECK=1 for in-development consumers; the next downstream Go binding cut should pin 0x0004.

Observer dispatch is no longer synchronous on the substrate dispatch thread

The v1 contract "callbacks must be cheap; the dispatch thread blocks until your callback returns" no longer holds. Observer events flow through a 1024-event bounded mpsc + worker task per binding. Behavior change: slow callbacks no longer pin the substrate; on overflow, events are dropped and the observer_dropped_total counter increments. Consumers that relied on the sync ordering (none in tree) get the new shape; callbacks should still be cheap, but the substrate is no longer on fire when they aren't.

CallOptions::cancel_token lands on the substrate CallOptions struct

Additive Rust-side field with Default::default() == None. Existing ..Default::default() callers continue to compile. Direct struct-literal callers that named every field need to add cancel_token: None.

Per-binding cancel registries are gone (internal-only break)

The napi binding's lock_cancel_registry + NEXT_CANCEL_TOKEN, the pyo3 binding's Cancellable internal state, and the Go FFI's cancel_registry + CancelEntry are deleted. Public APIs on each binding (reserveCancelToken / cancelCall napi methods, Cancellable pyclass, net_rpc_reserve_cancel_token / net_rpc_cancel_call FFI symbols) are preserved as pass-throughs; consumers that reached into the binding's private state directly need to switch to the substrate primitive.

Node TypedMeshRpc.callClientStream / callDuplex honor signal (was previously ignored)

The streaming entries' opts.signal was documented v1-and-only as close()-only; the wrapper called stripSignal to drop it. v0.23 wires wireAbortSignal end-to-end, so signal.aborted now fires raw.cancelCall(token) and the call rejects with RpcCancelledError. Consumers passing opts.signal to streaming entries were no-ops before; they'll start firing on cancel now.

Python TypedMeshRpc.call_client_stream / call_duplex honor opts['cancel'] (was previously ignored)

Same shape as Node. Passing a Cancellable to a streaming entry was a no-op in v1; it now propagates to substrate-level cancel. Consumers that were relying on "Cancellable is ignored for streaming" need to invoke cancel() only when they actually want cancel — the previous behavior of silent ignoring is no longer the default.

Go TypedCallClientStream / TypedCallDuplex honor ctx.Done() (was previously deadline-only)

The streaming entries previously honored ctx.Deadline() for the wire deadline but did not wire ctx.Done() to a cancel propagation path. v0.23 wires both. Cancelling the context now fires CANCEL on the wire.

RpcMetricsSnapshot grows observer_dropped_total (envelope-level u64)

Wire shape additive — postcard appends; existing readers tolerate the additional field. SDK consumers that built the struct by hand grow one field; consumers that rendered the snapshot via the per-binding MetricsSnapshot POD see the new field populated.

New Python Async* classes are exported from net

from net import AsyncMeshRpc, AsyncTypedMeshRpc, AsyncNetMesh, AsyncNetStream, AsyncNetDb, AsyncRedex, AsyncRedexFile, AsyncRedexTailIter, AsyncMemoriesAdapter, AsyncMemoryWatchIter, AsyncTasksAdapter, AsyncTaskWatchIter, AsyncDaemonRuntime, AsyncMigrationHandle, AsyncMeshBlobAdapter, AsyncDeckClient, AsyncAdminCommands, AsyncIceCommands, AsyncMeshOsDaemonSdk, AsyncMeshOsDaemonHandle, AsyncMeshQueryRunner, AsyncRegistryClient, AsyncFoldQueryClient all succeed. Existing sync imports are unchanged. Consumers that introspect dir(net) may see the new names.

Go typed binding lives in a new file

bindings/go/net/mesh_rpc_typed.go is new. Existing consumers of the raw *MeshRpc are unaffected; consumers who want the typed surface import the new symbols from the same net package.

How to upgrade

1. Rust consumers — update the dependency to 0.23. Direct struct-literal callers of CallOptions add cancel_token: None; everyone else recompiles unchanged.

2. Operators with custom observer callbacks — re-read the contract. The "callbacks must be cheap" guidance still applies (don't intentionally make the worker the bottleneck), but the substrate no longer pins the dispatch thread when a callback misbehaves. Monitor observerDroppedTotal / observer_dropped_total / ObserverDroppedTotal in metricsSnapshot to see if your callback is too slow for production load; size the upstream queue or push events into an asyncio.Queue / Go channel / Node async iterator if drops are appearing.

3. Cancellation users — migrate from binding-specific cancel surfaces to idiomatic per-language ones.

   • Node: new AbortController() + pass signal in CallOptions. Streaming entries now honor it end-to-end.
   • Python: Cancellable() + pass opts={'cancel': cancellable}. Streaming entries now honor it. Async callers use asyncio.wait_for(...) or task.cancel() for free — the bridge converts asyncio cancellation into a substrate Mesh::cancel.
   • Go: context.WithCancel(parent) + pass ctx to every call. Streaming entries now honor ctx.Done().
   • Power users in any language: raw reserveCancelToken / reserve_cancel_token / net_rpc_reserve_cancel_token + pass the token across multiple calls for shared-cancel scenarios.

4. Node + Python typed users — adopt client-streaming + duplex. TypedMeshRpc.serveClientStream / serveDuplex (Node) and serve_client_stream / serve_duplex (Python) are the new entry points. Handler signatures match the Rust SDK's typed surface. JSON codec is unchanged.

5. Go users — adopt TypedMeshRpc from day one. import "net/bindings/go/net"; t := NewTypedMeshRpc(rawMesh); result, err := TypedCall[Req, Resp](ctx, t, target, "svc.echo", req). Streaming + observer / metrics work the same way as the Rust SDK.

6. Python asyncio consumers — flip every blocking call to its async sibling. MeshRpc(mesh) → AsyncMeshRpc(mesh); rpc.call(...) → await arpc.call(...); for chunk in stream → async for chunk in astream. Sync API unchanged; both APIs coexist on the same NetMesh. The migration cookbook in bindings/python/README.md walks through a service-by-service move. asyncio cancellation works transparently via asyncio.wait_for / task.cancel().

7. async def server handlers — Python only. Register an async def handler with AsyncMeshRpc.serve(...) or AsyncMeshRpc.serve_client_stream(...) or AsyncMeshRpc.serve_duplex(...). The bridge detects the coroutine function at register time and dispatches every invocation through a server-side dispatcher event loop. Sync handlers continue to run on spawn_blocking; the choice is per-handler, not per-class.

8. No CI config change required. Strict clippy floor stays armed; rustdoc warnings stay denied; the test-side allow-list is unchanged. CI adds the cross-language streaming round-trip job and bumps the Go binding's pinned ABI version to 0x0004.

9. Operators — bump the binary. Pre-built net-mesh, net-deck, net-aggregator-daemon archives land for every supported target (Linux x86_64 / aarch64, macOS x86_64 / aarch64, Windows x86_64). Wire format is additive from v0.22; mixed-version fleets handshake cleanly. The new RpcMetricsSnapshot.observer_dropped_total field and the CallOptions::cancel_token field are postcard-appended; pre-v0.23 readers ignore them.

10. Downstream Go binding consumers — update or override the ABI pin. bindings/go/net/mesh_rpc.go::ExpectedABIVersion is now 0x0004. Consumers compiled against 0x0003 panic at init. NET_RPC_SKIP_ABI_CHECK=1 is the development override.

Named after Golden Earring's 1973 cut — the one with Cesar Zuiderwijk's two-bar drum intro that every garage band on three continents has tried to copy, and George Kooymans' lyric about a driver getting a wordless lover's-distress signal at half past four in the morning and burning it down the highway to answer it. "I been driving all night, my hand's wet on the wheel" — the song's whole urgency is in the gap between the call landing and the driver arriving, and shrinking that gap to as close to zero as the road will allow. v0.19 pushed the substrate past its prior throughput ceilings; v0.20 added a signed authorization gate on top of every nRPC invoke. v0.21 turns the dial toward latency — eliminating dead time on the hot path. Per-packet RX no longer pre-zeroes a 1500-byte buffer just for the kernel to overwrite it. Manifest fetches no longer wait for the previous chunk before requesting the next one. The capability index no longer clones a HashSet to compute an intersection. The replay-window check no longer takes the same lock twice. Across ~100 fixed items the substrate gets a faster reflex arc on every hot path that runs more than once per request.

A faster reflex arc on every hot path

The v0.21 release is the result of five back-to-back performance audits across the substrate (net-perf-analysis.md), the dataforts compositional layer (net-dataforts-analysis.md), the crypto + session + reliability wire-path (net-crypto-session-reliability-analysis.md), the discovery + routing surface (net-discovery-routing-analysis.md), the compute runtime (net-compute-analysis.md), and the dormant federated-query layer (net-meshdb-analysis.md). Each audit produced a ranked-by-impact list of "this allocates per event when it doesn't need to" / "this takes a lock per call when it could take none" / "this scans linearly when an index would be O(1)" / "this re-encodes when it could measure" items. v0.21 lands roughly 100 of them — the high-impact ones on every audit, plus the cleanest of the mediums where the fix was small enough to bundle.

The pattern across all five audits is the same: the substrate had a steady-state shape that worked but paid for it in allocator pressure, lock contention, and memcpy. Pre-v0.21 a 1M-packet-per-second receive workload spent more time zero-filling a buffer before each recvmmsg slot than it spent verifying the AEAD tag on the resulting packet. A 1024-chunk manifest fetch waited for each chunk's HTTP-equivalent round-trip before asking for the next one — a 1 s wall-clock fetch where 64 ms was the actual chunk-service-time minimum. A LeastLatency endpoint selection at 100 endpoints did 100 RwLock acquires and 100 LoadMetrics deep-clones per event just to read the values used to pick one. None of it was visible at a small scale; all of it bit at the throughput ceiling the v0.19 streaming surface exposed.

The fixes are localized — no architectural rework, no protocol changes, no fold rewrites. The wire format is unchanged; the public substrate API moves on a handful of types where the shape change pays for itself many times over (Bytes vs Vec<u8> on RPC and blob bodies, Arc<Batch> instead of Batch on the bus retry path, Vec<Arc<Memory>> instead of Vec<Memory> on CortEX query returns). Everything else lands under the hood — operators get the wins by bumping the dependency.

Below: the wins, grouped by where they fire.

Transport + crypto: zero-copy RX, half the locks per packet

The per-packet receive path was the single biggest pool of waste in the substrate. v0.21 closes most of it.

Kill the recv-buffer zero-fill. PacketReceiver::recv used to call resize(MAX_PACKET_SIZE, 0) per packet — a ~1500-byte memset whose only purpose was to give the kernel a slice to overwrite milliseconds later. The fix is tokio's recv_buf_from/recv_buf/try_recv_buf_from, which write directly into BufMut spare capacity without the pre-zero. The same pattern showed up on three sibling NetSocket entry points and on Linux's BatchedTransport::recv_batch (which zero-filled all 64 batch slots, ~512 KiB of memset per batch). At 1 M pps that's around 9 GB/sec of memory bandwidth gone — entirely.

Zero-copy RX decrypt. The receive path used to call an allocating decrypt that produced a fresh Vec per packet. v0.21 adds PacketCipher::decrypt_to_bytes, which tries Bytes::try_into_mut first (the common UDP refcount-1 case decrypts in place into the inbound buffer's allocation) and falls back to allocation only when the buffer is shared. At 1 M pps with 1 KB packets this saves roughly 1 GB/sec of allocator churn.

Cached nonce template. nonce_from_counter rebuilt the 12-byte AEAD nonce from scratch (prefix memcpy + zero-init + counter write) on every encrypt and decrypt. v0.21 caches the template on the cipher; only the counter bytes get written per call.

Single-lock RX admit. The replay-window check used to take two separate parking_lot::Mutex acquisitions per inbound packet: one before decrypt (is_valid_rx_counter) and one after (update_rx_counter). v0.21 collapses them into a single try_admit_rx_counter. Replays now pay an AEAD verify before rejection (priced in — replays are rare), and the steady-state path drops 10–20 M lock ops/sec at 1 M pps.

Verify-only heartbeat path. Heartbeats used the allocating decrypt purely to drop the result (the call was decrypt(...).is_err()). v0.21 adds PacketCipher::verify, which runs the AEAD tag check via in-place decrypt over a single scratch BytesMut. No alloc per heartbeat.

Arc-shared retransmit descriptors. RetransmitDescriptor carries a Vec<Bytes> per outbound chunk-group; the reliability layer used to deep-clone it on every NACK or timeout emission. v0.21 switches the reliability-mode trait to exchange Arc<RetransmitDescriptor> — retransmits are one refcount bump regardless of inner Vec length.

Dataforts: parallel chunk I/O, Bytes through the trait, lookup-table hex

The blob fabric grew a 16-wide concurrent fetch and an end-to-end Bytes flow.

Parallel manifest fetch + store. MeshBlobAdapter::fetch(Manifest) used to walk chunks sequentially — for a 1024-chunk replicated blob at 1 ms per chunk, that was a 1-second wall-clock fetch where the actual chunk-service-time minimum was 64 ms. v0.21 wraps the chunk iteration in stream::iter(...).buffered(16) (ordered, to preserve assembly correctness). The store path gets the symmetric treatment via buffer_unordered(16) — content-addressed writes are order-independent and idempotent — with a hoisted verification prepass so the "no chunks stored on a caller-poisoned manifest" contract still holds. Roughly a 15× latency reduction on bulk manifest operations.

Bytes through the BlobAdapter trait. BlobAdapter::fetch / fetch_range / MeshBlobAdapter::fetch_chunk returned Vec<u8>, which forced a .to_vec() memcpy of the chunk's Bytes payload at every layer boundary. v0.21 switches the trait surface across every implementor (Rust, FFI, Python, Node) to Bytes. The blob-tree node cache also moves to Bytes — cache hits become Arc clones rather than Vec::clone. On bulk-fetch workloads this saves gigabytes per second of memcpy.

Lookup-table hex encoding. hex32 and chunk_channel used to call write!("{:02x}", b) 32 times per blob op. v0.21 adds a HEX_LOWER lookup table + zero-alloc hex32_into(&[u8; 32], &mut [u8; 64]) — roughly 10× faster, and zero allocation. parse_blob_heat_tag got the symmetric treatment on the decode side via a nibble lookup table.

Single-pass manifest verification. verify_manifest_chunks used to walk the chunk list twice (a sum-check pass for total-size validation, then a hash-check pass for content validation). v0.21 fuses them into a single pass with checked_add for overflow and an end > fetched.len() bounds-check before each slice.

Measure, don't re-encode. BlobRef::encoded_len used to do a full re-encode for Manifest and Tree variants — allocate a Vec, postcard-serialize, read .len(), drop. The common pairing of encoded_len + encode was paying the encode cost twice. v0.21 switches encoded_len to postcard::experimental::serialized_size, which walks the type tallying bytes without allocating.

Greedy + heat micro-wins. GreedyCacheRegistry::contains_origin was O(N), called per admission carrying a colocation hint; v0.21 adds an origin_counts: HashMap<u64, usize> reverse index for O(1) lookups. GreedyRuntime::local_caps was Mutex<Arc<CapabilitySet>> cloned per dispatch_event; v0.21 switches to ArcSwap — reads become one lock-free Acquire load. Post-fetch heat-bump used to build a Vec<[u8; 32]> of hashes per call; bump_heat now takes impl IntoIterator<Item = [u8; 32]> and streams directly. Manifest-assembly Vec is pre-allocated to total_size.min(MAX_BULK_FETCH_BYTES) (256 MiB cap protects against hostile manifests).

Discovery + routing: single-pass filters, cached resolution, smaller dedup

The capability surface and the publish path picked up a clutch of localized wins where the per-call work was steady-state O(N) on values that didn't need to be.

Cached session NodeId. dispatch_packet used to resolve session → NodeId per inbound packet, which meant two DashMap lookups and a possible O(N) peer scan when the source address had drifted. v0.21 caches the resolved id on NetSession as cached_node_id: AtomicU64 (sentinel 0 = unresolved); the fast path is one Relaxed load.

Arc-shared publish events. publish_many used to clone the events Vec<Bytes> per spawned task. For 100 subscribers × 1000 events: 100 Vec allocs + 100 K Bytes refcount bumps. v0.21 hoists into Arc<[Bytes]> — 1 Vec alloc + 1 K Bytes bumps + 100 Arc bumps.

Single-pass subscriber filter. publish used to do two sequential retain passes over the subscriber Vec (subnet visibility, then auth/token). v0.21 fuses them into one retain closure with the cheapest check first — single walk over the Vec.

Vec-based capability intersection. CapabilityIndex::build_candidate_set used to clone full HashSets to intersect them — one HashSet alloc per indexed filter clause. v0.21 switches the working set to Vec<u64>: the first match materializes one Vec, subsequent clauses use in-place retain(|n| index.contains(n)) — no new container per clause.

Threshold-based dedup HashSet. dispatch_recipients did O(N) out.contains(&picked) for dedup. v0.21 keeps the linear scan as the fast path and promotes to HashSet<u64> only once out crosses a 32-entry threshold — the common small-recipient-set case stays branch-predictor-friendly, the large case stops being O(N²).

Single-pass scoped find. find_nodes_scoped used to run the filter then re-iterate doing get(node_id) per survivor (full CapabilitySet clone per node) just to read caps.tags. v0.21 folds scope resolution into the same shard-lock guard as filter re-validation — zero CapabilitySet clones.

Compute + load balance: lock-free metrics, O(1) reverse indexes

The compute runtime gets its hot paths the same treatment as the wire path.

Lock-free LoadMetrics. EndpointState::metrics() used to do a RwLock read plus a 9-field clone per call — per endpoint per select. At 100 endpoints with LeastLatency, that was 100 RwLock acquires + 100 deep clones per event. v0.21 switches to ArcSwap<LoadMetrics> and adds a load_score() helper that reads via guard; the metrics fields stay private but the score is one indirection.

max_by, not sort. Scheduler::pick_best_candidate was doing an O(N log N) sort_by only to take the first element. v0.21 switches to O(N) max_by with inverted tie-break direction.

Reverse index in the group coordinator. origin_hash_for_entity_id was a linear scan comparing 32-byte NodeIds. For a 100-member group at 100 K ev/s that meant 10 M × 32-byte comparisons/sec. v0.21 adds origin_hash_by_entity_id: HashMap<NodeId, u64> — lookup is O(1).

Skip horizon encode on empty outputs. DaemonHost::deliver was calling horizon.encode() (walks the horizon map + xxh3 per entry) per event even when the daemon produced no outputs. v0.21 early-returns when outputs.is_empty() after observation accounting.

Streaming parent-hash. compute_parent_hash used to allocate a Vec, memcpy the 32-byte link + payload in, hash, drop. At 100 K ev/s with 1 KB payloads: 100 K allocs/sec + 100 MB/sec memcpy. v0.21 switches to streaming Xxh3::update — zero alloc.

AtomicU64 last_selected. EndpointState::last_selected was Mutex<Instant> used purely as cell storage. v0.21 switches to AtomicU64 of nanos since a OnceLock<Instant> baseline. For 100 K successful selections/sec: 100 K lock+unlock pairs gone.

O(1) member-index lookup. mark_healthy / mark_unhealthy / update_member_placement used to do a linear iter_mut().find(|m| m.index == index). v0.21 switches to members.get_mut(index as usize) with a defensive re-check — O(1) with the same correctness invariant.

Core bus + RedEX + RPC: zero-copy reads, Bytes payloads, Arc-shared batches

The substrate's per-event spine — the bus, the append-only log, the RPC payloads — gets the same treatment.

Arc-shared batch on retry. dispatch_batch used to clone the entire Batch on every retry attempt, including attempt 0. v0.21 switches Adapter::on_batch to take Arc<Batch> — retries are now a refcount bump. For a 1000-event batch with retries, this saves 1000+ Bytes refcount bumps and one Vec alloc per dispatch.

ArcSwap shard selection. Mapper::select_shard used to do two Vec allocs + a RwLock read per event in dynamic mode. v0.21 pre-computes the selection into ArcSwap<SelectionTable> and reads via guard. At 10 M ev/s this removes 20 M allocs/sec.

Amortized TLS pool reaping. ThreadLocalPool::acquire/release used to run a HashMap retain + Weak::strong_count walk on every call. v0.21 amortizes to every 4096th call via a per-thread counter. The published "thread-local 2× slower than shared" benchmark anomaly should erase.

Zero-copy HeapSegment::read. Reads used to do Bytes::copy_from_slice per call. v0.21 switches the internal buffer to Bytes; reads are refcount slices, appends use Bytes::try_into_mut. At 4 KB payloads × 100 K ev/s watcher load, this is 400 MB/sec of pure memcpy gone.

Binary search read_one / read_range. RedexFile::read_one and read_range were linear scans over a sorted index. v0.21 switches to partition_point — O(N) → O(log N).

Compiled consumer filter. Filtered poll used to re-split the path string and re-parse indices per event. v0.21 adds CompiledFilter, which runs the path-split + integer-parse once per poll.

Bytes-based RPC payloads. RpcRequestPayload / RpcRequestChunkPayload / RpcResponsePayload::body was Vec<u8>, which forced a .to_vec() memcpy per frame decode. v0.21 switches the body field to Bytes end-to-end across substrate + Node / Python / Go FFI boundaries. At 100 K RPCs/sec with 1 KB bodies: 100+ MB/sec of memcpy gone.

In-place router forward. Router::route_packet used to allocate a fresh buffer and full-copy the body just to flip an 18-byte header. v0.21 adds RoutingHeader::write_at and uses the Bytes::try_into_mut fast path for sole-owned UDP packets, with allocate-and-copy as the fallback.

Lemire shard hash. select_shard_by_hash in static mode used hash % n (~20–25 cycles per event). v0.21 switches to Lemire multiply-shift reduction (~3 cycles) — ~7× cycle reduction on a per-event hot path.

Inline hot byte codecs. RedexEntry::to_bytes / from_bytes and EventMeta::to_bytes / from_bytes weren't marked #[inline]; v0.21 marks them #[inline(always)] — the codec is small enough that inlining lets the compiler erase the wrapper-call overhead across every event-handling site.

Redis adapter micro-wins. redis::serialize_event used to_string().as_bytes() for u64/u16; v0.21 uses write! into the existing Vec — two String allocs per event gone. parse_xrange_response was setting last_seen_id (String alloc) on every iteration even though only the last mattered; v0.21 tracks last_seen_idx: Option<usize> and materializes once — 9999 wasted allocs per 10 K-entry response gone. is_transient_error used to_string().to_uppercase() per classified error; v0.21 uses zero-alloc RedisError::detail() + starts_with.

CortEX content + title search: ASCII fast path

CortEX's MemoriesQuery::matches and TasksFilterSpec::matches used to_lowercase().contains() per row — a full Unicode case-folding pass on the entire content body, then a contains walk, per search predicate per row. v0.21 adds ContentNeedle and TitleNeedle wrappers with an ASCII fast path: when the needle is pure ASCII (the overwhelming common case), the match runs as eq_ignore_ascii_case byte scan with zero allocation. Non-ASCII needles fall back to the existing Unicode path. For 100 K memories at 4 KiB each, this eliminates roughly 400 MB of allocation and 400 MB of case-folding per content search.

Arc-shared memory state. MemoriesState used to store Memory by value, which meant query/watcher returns deep-cloned every Memory the caller observed. v0.21 switches the store to Arc<Memory>; query and watcher APIs return Vec<Arc<Memory>>; writers use Arc::make_mut for copy-on-write semantics; the FFI surface uses Arc::try_unwrap to avoid double-clone on the unwrap path.

MeshDB: pre-activation hardening for the dormant federated layer

The federated-query layer is gated behind the meshdb Cargo feature and currently dormant in production. v0.21 lands the pre-activation hardening pass so the layer is ready when the feature flips on.

• Parallel hash-join sub-fetches. tokio::try_join! around the two halves of a federated hash-join: pure 2× on every remote/remote join (50 ms RTT each: 100 ms → 50 ms wall).
• DashMap caller-side inflight. Caller-side inflight map used to be Arc<RwLock<HashMap>>; every send took a write lock and concurrent sends to distinct call_ids serialized. v0.21 switches to Arc<DashMap> — sharded per call_id.
• Cached approx_bytes on CachedResult. Walked every row per LRU bookkeeping call; v0.21 caches at construction in a private u64 field — single field load.
• Pre-sized drain_rows. Switched from Vec::new() + grow-by-doubling to Vec::with_capacity(DRAIN_INITIAL_CAPACITY).
• Lookup-table planner chain_hex. format!("{:016x}", origin_hash) replaced with a HEX_NIBBLES 16-shift unroll into a 16-byte stack buffer.

Architecture work: subnet spec + multifold plan land as design docs

Two architectural design documents land alongside the perf work, both targeting future releases:

• SCALING_SUBNET_SPEC.md — formal specification of how the substrate carves an arbitrarily-large mesh into operator-defined subnets, the membership protocol, the cross-subnet routing primitives, and the auth model that interacts with the v0.20 capability-auth SubnetId allow-list axis. Design doc only in v0.21; the implementation tracks separately.
• SCALING_MULTIFOLD_PLAN.md — plan for parallelizing the substrate's fold (consume-events-into-state) layer across multiple shards per stream rather than the current single-fold-per-stream model. Same — design only in v0.21.

Operators tracking the substrate's scaling roadmap should read both; neither is wired into the runtime yet.

Test hygiene

• Lib suite at 3950+ tests (was 3850+ at v0.20.2 release). 100+ net new tests across the perf-pinning regressions — zero-copy reads pinned against accidental Bytes::copy_from_slice reintroduction, ArcSwap selection table pinned against ABA on concurrent rebuilds, ASCII fast-path pinned to fall back correctly on non-ASCII needles, single-lock RX admit pinned against replay-after-AEAD-verify ordering, parallel manifest fetch pinned against ordering-vs-correctness, and the surface tests on the new public API (Bytes-returning blob fetches, Arc<Memory>-returning CortEX queries, Arc<Batch>-accepting bus adapters).
• cargo clippy --features meshos,deck --all-features --all-targets -- -D warnings clean. The strict floor from v0.20.2 (unwrap_used, expect_used, undocumented_unsafe_blocks, multiple_unsafe_ops_per_block) stays armed.
• cargo doc --features meshos,deck --no-deps clean under RUSTDOCFLAGS="-D warnings". Doc-comment hygiene includes a sweep of bracketed perf-doc refs that rustdoc was misinterpreting as broken intra-doc links.
• Codecov coverage sits at ~90% on the substrate feature set, informational on the CI status — same posture as v0.20.2.

Breaking changes

Adapter::on_batch signature

Adapter::on_batch(&self, batch: Batch) → Adapter::on_batch(&self, batch: Arc<Batch>). Retries are now a refcount bump on the same batch. Adapters that need ownership of the batch can use Arc::try_unwrap(batch).unwrap_or_else(|b| (*b).clone()) — falls into the clone branch only when a retry is actually in flight against the same batch.

Mesh::send_to_peer / Mesh::send_routed take &Batch

Both used to take Batch by value, which forced callers to clone-or-move. v0.21 takes &Batch — callers retain ownership and can re-send without re-cloning.

Rpc{Request,Response}{,Chunk}Payload.body is bytes::Bytes

The body field on every nRPC payload moves from Vec<u8> to bytes::Bytes. Constructors: Bytes::from(vec), Bytes::from_static(b"..."), Bytes::copy_from_slice(&buf). Accessors: as_ref() for &[u8]. The Node, Python, and Go FFI bindings wrap at the boundary so binding consumers are unaffected; Rust consumers update construction sites.

BlobAdapter::fetch / fetch_range / resolve_payload return bytes::Bytes

The blob trait surface returns Bytes instead of Vec<u8> across every implementor (Rust, FFI, Python, Node). Use as_ref() for &[u8]; call to_vec() only when you genuinely need an owned Vec. Small-range reads also use Bytes::slice internally — repeated range fetches over the same blob share a backing allocation.

CortEX MemoriesState query / watch returns Vec<Arc<Memory>>

Memory queries and watcher streams used to return Vec<Memory> (deep-cloning every observation). v0.21 returns Vec<Arc<Memory>>. Treat as read-only by default; call Arc::try_unwrap (or clone the inner Memory) when you need an owned mutable copy. Writers use Arc::make_mut for copy-on-write semantics, so observed handles remain stable.

bump_heat signature

bump_heat(hashes: &[[u8; 32]]) → bump_heat<I: IntoIterator<Item = [u8; 32]>>(hashes: I). Callers that previously built a Vec to pass in can now stream directly (iter::once, chunks.iter().map(|c| c.hash)).

Removed static shard hash via %

select_shard_by_hash static mode no longer does hash % n internally — it uses Lemire multiply-shift. The function signature is unchanged; the change is observable only via cycle count.

How to upgrade

1. Rust consumers — update the dependency to 0.21. Most of the wins land transparently. The breaking changes above are mechanical and the compiler points at every site.

2. Adapter implementations — switch to Arc<Batch>. If your Adapter::on_batch mutates the batch, replace batch.clone() with Arc::try_unwrap(batch).unwrap_or_else(|b| (*b).clone()). If your on_batch is read-only, drop the leading clone entirely.

3. nRPC callers / handlers — construct payloads with Bytes. Anywhere you built a RpcRequestPayload with a Vec<u8> body, swap to Bytes::from(vec) (no allocation, takes ownership) or Bytes::copy_from_slice(&slice) (one allocation for the new buffer). Handlers reading the body: payload.body.as_ref() for &[u8].

4. Blob consumers — handle Bytes. BlobAdapter::fetch and friends now return Bytes. Most call sites change from fetch(...).await? (returning Vec<u8>) to fetch(...).await? (returning Bytes); downstream &[u8] access uses .as_ref(). Sites that genuinely need an owned Vec<u8> call .to_vec(), but this is rarely the right answer — Bytes is cheaper to pass around.

5. CortEX consumers — Arc<Memory> handles. Query and watcher returns now carry Arc<Memory>. Reads work unchanged via deref coercion (memory.title, memory.content); mutation requires Arc::try_unwrap (succeeds when the Arc is sole-owned) or an explicit clone of the inner Memory.

6. No CI config change required. The strict clippy floor from v0.20.2 is still the floor; the test-side allow-list is unchanged.

7. Operators — bump the binary. Pre-built net-mesh and net-deck binaries land in the release archive for every supported target (Linux x86_64 / aarch64, macOS x86_64 / aarch64, Windows x86_64). Drop in /usr/local/bin (or your platform's equivalent) and restart the daemon. Wire format is unchanged from v0.20.x; mixed-version fleets handshake cleanly.

8. Operators tracking the scaling roadmap — read the new design docs. SCALING_SUBNET_SPEC.md and SCALING_MULTIFOLD_PLAN.md live under docs/. Neither is implemented in v0.21; both target future minor releases. Comments and pushback welcome before the implementation lands.

Named after Deep Purple's 1972 track — the one written from a hotel window across Lake Geneva on the night the Montreux Casino burned down. December 4th, 1971: Deep Purple had booked the casino's empty gambling theater to record what would become Machine Head, but the day before the session started they sat in on a Frank Zappa & the Mothers of Invention show in the same hall. Somebody in the audience fired a flare gun into the rattan-covered ceiling, the whole building went up, and the band watched the smoke drift out over the lake while the casino — and most of the gear they'd been planning to record on — collapsed into Funky Claude's frantic evacuation. The riff every guitar shop in the world has heard ten million times is what came out of that view. v0.19 pushed the substrate past its prior throughput ceilings: bidirectional streaming on nRPC, hierarchical manifests + erasure coding + durable staging on Dataforts. v0.20 turns the lens onto what every peer is allowed to actually do with that throughput. The mesh-wide capability surface gains a signed allow-list model — every announcer decides who can invoke its capabilities, the gate fires on both sides of the call, and the operator drives it from one CLI verb. Underneath, a deep-read audit of the cryptographic token surface closes a clutch of cross-channel collision, revocation, dedup-race, canonicalization, and clock-skew hazards that would otherwise have been load-bearing for the new gate.

When the smoke clears

For three releases the mesh-wide capability surface was discovery-only. A node announce_capabilities()'d, peers indexed the announcement, find_service_nodes(...) resolved targets, and that was it. Anyone who could reach a peer over the wire could call_service it; the only "auth" was channel-auth tokens scoped to channels (pub/sub), not capabilities. Anyone who could reach an nRPC endpoint could invoke any service the endpoint published.

v0.20 closes that gap end-to-end. Every CapabilityAnnouncement is now a signed policy unit — the announcer carries three explicit allow-lists (allowed_nodes, allowed_subnets, allowed_groups) directly on the wire, and the substrate honors them at every invoke. Empty allow-lists are the permissive default (any caller admitted, byte-identical to the pre-v0.20 wire form); non-empty lists union into a "node OR subnet OR group" admit rule. Revocation is just a new announcement at a higher version — no separate verb, no separate channel, no operator-key subsystem distinct from the entity key that already signs the announcement. The model is deliberately not an ACL engine: it's the smallest correct gate around announce and execute.

Underneath the new gate, the v0.20 hardening pass closes eight long-standing hazards on the cryptographic token surface and the multi-hop capability dispatch path — items that the v0.19 channel-hash widening had narrowed but not eliminated. Token revocation, forwarded-announcement dedup races, exact-match reserved-key gating on inbound peers, channel-name canonicalization, defense-in-depth subject cross-checks, clock-skew tolerance on delegation validity, and a wildcard-slot DoS shape. None of them were exploitable end-to-end on the v0.19 stack without operator misconfiguration, but several were one refactor away from becoming load-bearing — v0.20 closes them before the new authorization gate gets stacked on top.

Capability execution authorization

The mesh now answers a question it couldn't answer before: given that B can reach A's nRPC endpoint, is B authorized to invoke A's capabilities? Pre-v0.20 the answer was always yes; v0.20 lets the answer be no, decided by A's own signed policy, enforced on both sides of the wire.

The shape

Two new identity types land under net::adapter::net::behavior::{subnet, group}:

• SubnetId([u8; 16]) — 16-byte opaque identifier for a topology partition. Operators pick the value (random 16 bytes, a blake2s-of-name truncated to 16, or any operator-stable convention); the substrate doesn't interpret the bytes. A peer self-declares subnet membership via a subnet:<hex32> tag on its own announcement; the capability index parses the tag at fold time and stores NodeId → SubnetId on the peer view.
• GroupId([u8; 32]) — 32-byte opaque identifier for an operator-defined named collection of peers. Same self-declared pattern via group:<hex64> tags; a peer can emit multiple group tags to claim membership in multiple groups. The wider value-as-secret space lets operators use a random GroupId that's effectively unguessable, matching the substrate's existing channel-auth-token idiom.

Self-declaration is safe because the announcement is signed and TOFU-bound to the entity's ed25519 key — a peer can only claim membership for itself. Operators who want stricter membership use a random ID that's hard to guess; operators who want advisory routing use a public blake2s-of-name.

Three additive fields land on CapabilityAnnouncement:

#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub allowed_nodes: Vec<u64>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub allowed_subnets: Vec<SubnetId>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub allowed_groups: Vec<GroupId>,

All three default empty + skip-when-empty, so the signed byte form of an unrestricted announcement is byte-identical to the v0.19 wire shape — pre-v0.20 peers round-trip cleanly, and a v0.19 signature verifies on a v0.20 reader. Length caps at 64 entries per axis enforced both on the announce side (the CLI rejects oversized lists at build time) and the wire side (CapabilityAnnouncement::from_bytes rejects oversized payloads at deserialize, so a malicious or buggy peer can't ship a million-entry list and force linear scans on every may_execute call).

The gate

CapabilityIndex::may_execute(target_node, capability_tag, caller_node) -> bool is the canonical entry point. Permissive by default — an announcement with all three allow-lists empty admits any caller. Once any list is non-empty, the union of all three is enforced (node OR subnet OR group); the scan short-circuits on the first match. Returns false when the target has no indexed announcement, when the target's announcement doesn't list the requested capability tag, or when the target restricts and the caller matches no axis.

Two call sites consult it:

• Caller-side, inside Mesh::call_service. The candidate set returned by find_service_nodes is filtered through may_execute before the routing policy picks a target — so the policy never selects a peer the caller can't actually call, and "no peer advertises X" stays distinguishable from "every peer that does advertise X refused me." When the filter empties the set, the call returns RpcError::CapabilityDenied { target, capability } referencing one of the originally-advertised peers as a representative.

• Callee-side, inside serve_rpc's bridge — defense in depth for the well-behaved client path. A caller that bypasses the caller-side gate (direct call() instead of call_service, an out-of-date local index, or a buggy client) gets rejected at the receiver. The bridge emits an RpcStatus::CapabilityDenied (0x0008) response that the caller's MeshNode::call surfaces as the typed RpcError::CapabilityDenied — same variant regardless of which side of the gate fired.

serve_rpc lazily emits a default-permissive self-announcement at registration time that merges every currently-registered nrpc:<service> tag, so the callee-side gate always observes a real policy from the very first inbound event — no cold-start window, no order dependency between serve_rpc and announce_capabilities. The same call also schedules a peer-side broadcast so other nodes learn about the new service without the operator having to re-announce manually.

Membership parse determinism

Subnet membership is single-valued by design. An announcement carrying multiple distinct subnet:<hex> tags is out-of-model malformed input — the v0.20 parser collapses it to None (no membership) rather than picking one tag based on HashSet<Tag> iteration order, which is unspecified and would otherwise produce hash-order-dependent gate verdicts that diverge across receivers folding the same signed bytes. Single subnet tag works as expected; duplicate tags pointing at the same SubnetId also work (the underlying set dedups them). Groups sort by byte value so the iteration sequence is stable across receivers.

The CLI

One new operator verb on the existing CLI:

net-mesh cap announce \
    --tag nrpc:my-service \
    --tag dataforts.blob.overflow \
    --allow-node 42 \
    --allow-node 0xDEADBEEF \
    --allow-subnet 112233445566778899aabbccddeeff00 \
    --allow-group deadbeefcafef00d... \
    --key /etc/net-mesh/operator.toml \
    --version 7 \
    --ttl-secs 300

Builds a signed CapabilityAnnouncement with the supplied allow-lists and emits the JSON bytes to stdout (or --out <PATH>). The operator ships those bytes through any pub/sub path that calls CapabilityIndex::index on receipt. There's no separate revoke verb — revocation is a new announcement with a tighter allow-list (or [only_me] to deny everyone else), so the audit trail stays uniform. The --node-id override is supported only as an explicit confirmation that the supplied id matches the signing key's derived value; a mismatch is rejected at the CLI rather than producing bytes a receiver would refuse.

Wire status + caller-facing error

RpcStatus::CapabilityDenied = 0x0008 slots into the reserved canonical-status band; the reserved range pushes to 0x0009..=0x7FFF. RpcError::CapabilityDenied { target: u64, capability: String } is the typed caller-side variant. default_retryable(RpcError::CapabilityDenied) returns false — a deny verdict won't change on retry, so the retry budget isn't burned on a deterministic deny.

Conformance

tests/capability_auth_conformance.rs pins the six-scenario contract end-to-end against real MeshNode instances: permissive baseline admits any caller; allow-by-node admits the listed peer and denies others; allow-by-subnet admits subnet members; allow-by-group admits group claimants; revocation via a new announcement supersedes the old policy; callee-side defense in depth rejects when the caller bypasses the local gate. Plus standalone regression tests for the multi-subnet collapse, the wire-side allow-list cap, the caller-side candidate filter, and the call-path filtering when only a subset of advertising peers authorize a given caller.

Token + identity hardening

The v0.19 release widened ChannelHash from u32 to u64 (raising the targeted-collision cost from ~2^32 to ~2^64) and shipped the new 169-byte PermissionToken wire format. v0.20 closes the remaining items from the cryptographic-token surface audit — none of which were exploitable end-to-end on the v0.19 stack without operator misconfiguration, but several of which would have become load-bearing the moment the new capability-auth gate stacked on top.

Token revocation

Pre-v0.20 the substrate had no way to invalidate a token short of natural expiry. A parent that delegated a 1-year token to a child carried the child's signature past any "revoke" intent — even after rotating the parent's key, every cache holding the old EntityId continued to honour the child. v0.20 adds a per-issuer generation epoch on PermissionToken that the cache cross-checks on every check(). Bumping an issuer's generation drops every descendant in O(chain_depth) at lookup time without per-token state. The new field rides inside the signed payload so it can't be tampered post-issue. Operators rotate via EntityKeypair::bump_generation() followed by a re-issue of the still-current tokens at the new generation; the rotation step is one operator call, and the propagation is the same gossip path the existing identity broadcast already uses.

PermissionToken::delegate also now caps the child's not_after at min(parent.not_after, now + DELEGATION_MAX_TTL) — the operator-recovery window for a compromised delegate is bounded by the constant rather than the parent's full remaining lifetime.

Forwarded-announcement dedup race

The pre-v0.20 capability-announcement handler keyed the dedup table on (node_id, version) only, and the dedup insert ran BEFORE the TOFU bind that records from_node → entity_id for channel-auth lookups. A forwarded copy of a victim's signed announcement could land first, prime the dedup slot, and silently drop the victim's subsequent direct announcement on arrival — the victim's binding was never written, and any require_token channel keyed on that binding failed closed until the victim's next version increment.

v0.20 widens the dedup key to (node_id, version, hop_count == 0) so a direct announcement is never short-circuited by a prior forwarded copy. The TOFU bind runs unconditionally on every direct arrival. The forwarded-poisoning-then-direct-arrives race is now covered by a regression test alongside the existing forwarded_announcement_does_not_tofu_pin_forwarder_to_victim_entity invariant.

Reserved-key gate on inbound metadata

CapabilitySet::with_metadata enforced the prefix reserved-key list but not the exact-match reserved-key list (intent, colocate-with, priority, owner). These four keys are intentionally writable by user code on the local node, but the same field is populated by deserializing inbound peer announcements — and that path ran no gate at all. A peer could stamp intent = "high-priority-tenant-X" on its own announcement and steer the receiving node's greedy-admission to itself for tenant X's workloads.

CapabilityAnnouncement::strip_reserved_metadata is the new boundary: receivers strip the exact-match reserved keys (and any reserved-prefix matches) from inbound peer announcements before metadata is consulted by greedy admission, placement scoring, or anything else that lets a metadata value steer substrate decisions. The local node's own announcements still carry the keys — the strip runs only on the receive path. Greedy admission's chain_caps.metadata.get("intent") lookup now reads the local node's view, never an attacker-stamped peer value.

Channel-name canonicalization

ChannelName::new rejected explicit path-traversal (/./ and /../) but admitted trailing dots, repeated dots within a segment (foo..bar), and case-folded duplicates — foo.bar and FOO.BAR hashed to different ChannelHash outputs and registered as parallel namespaces. Combined with a registry miss falling into the permissive "no ACL" branch of authorize_subscribe, this opened a quiet bypass path for operators who registered one casing of a name but not the other.

v0.20 canonicalizes channel names on construction. Names lowercase to a single form before hashing; trailing dots, leading dots, and empty/dot-only segments are rejected with ChannelNameError::Malformed. Existing registered names keep working — the lowercase normalization is idempotent on already-lowercase names and the rejection rules only fire on names that previously hashed to a separate namespace from a sibling. The authorize_subscribe fall-through is also tightened: a registry miss with no matching prefix entry now returns Unauthorized rather than the prior permissive default.

Defense-in-depth subject cross-check on token cache

TokenCache::check walked the slot keyed on (subject_bytes, channel_hash) and authorized any token in the slot whose authorizes(action, channel_hash) returned true. Today the inserts always key by token.subject.as_bytes() so the invariant holds, but the predicate didn't re-confirm that the stored token's subject field matched the lookup key — a future refactor that indexed by hash-of-subject (for memory savings) or added a replace_unchecked constructor would silently authorize the wrong entity.

v0.20 adds the cross-check directly to the predicate: a stored token authorizes a lookup only if token.subject.as_bytes() == lookup_subject.as_bytes() in addition to the existing scope / channel / validity checks. Cost is one memcmp per check; benefit is a durable invariant the cache enforces on every lookup regardless of how future indexers are built.

Clock-skew tolerance on token validity

PermissionToken::is_valid did raw now < not_before / now >= not_after comparisons with no skew window. A node whose system clock drifted forward refused to delegate a freshly-issued parent token (it appeared expired locally); a node whose clock drifted backward refused not-yet-valid tokens. Worse, a node with a clock that ran 30 seconds slow accepted tokens that the rest of the mesh treated as expired — the channel-auth fast path used the same is_valid call.

v0.20 adds MeshNodeConfig::clock_skew_tolerance_secs (default 60 seconds) that applies symmetrically to both bounds: now + skew < not_before for the not-yet-valid check, now - skew >= not_after for the expired check. The constant is operator-tunable; the documentation calls out the source-of-truth assumption (the mesh trusts its own clock source within the configured window). Token issuers compensate by pulling not_after back by the skew window on mint, so a clock that ran skew-forward sees the same expiry boundary as a clock that ran on-time.

Wildcard-slot fast path

Every WILDCARD token landed in the slot (subject_bytes, 0), and the fallback path on check() walked the wildcard slot every time the exact slot missed. An attacker with a valid signing key and DELEGATE scope could mint up to MAX_TOKENS_PER_SLOT = 32 distinct WILDCARD tokens under the same subject and force every check for that subject to walk all 32 entries. Mostly a latency issue rather than a privilege gain, but a measurable CPU drag on hot lookup paths.

v0.20 caches a bool on each slot — slot.has_wildcard — that's set on insert and cleared on the last eviction. The check's fallback to the wildcard slot skips entirely when the bool is false. The exact-slot path is unaffected.

Test hygiene

• Lib suite at 3850+ tests (was 3700+ at v0.19 release). 150+ net new tests across the capability-auth allow-list wire format (signed byte-identity vs the v0.19 shape, round-trips with each axis populated, tamper detection on each new field), the gate semantics (permissive / allow-by-node / allow-by-subnet / allow-by-group / no-tag / no-announcement / multiple allow-lists overlap / revocation supersedes / cap enforcement / subnet-parse determinism), the call-path integration (caller-side candidate filter + callee-side defense in depth + auto-self-index from serve_rpc), the six-scenario conformance file, the CLI cap announce regression suite (signed-bytes round-trip, stdout-vs-file equivalence, --node-id mismatch rejection, duplicate-tag acceptance, malformed-arg exit codes), and the token-surface hardening regressions (revocation via generation bump, forwarded-then-direct dedup race, reserved-key strip on inbound, channel-name canonicalization, subject cross-check, clock-skew bounds, wildcard fast path).
• cargo clippy --features meshos,deck --all-features --all-targets -- -D warnings clean across substrate + every binding crate + the deck demo + the deck TUI + the net-mesh CLI.
• cargo doc --features meshos,deck --no-deps clean under RUSTDOCFLAGS="-D warnings" — every public item in the v0.20 surface carries a doc comment; intra-doc links resolve through the public re-exports.
• CI matrix carries the capability-auth feature gates alongside the v0.19 nrpc-streaming and dataforts-tree feature builds. Python / Node / Go / C bindings pick up the CapabilityDenied status code in their status enums and the typed error variant in their error mapping; cross-binding round-trip tests run on every CI build.

Breaking changes

CapabilityAnnouncement fields

Three new fields on CapabilityAnnouncement (allowed_nodes, allowed_subnets, allowed_groups). Wire-compatible with v0.19 when empty — the signed byte form of an unrestricted announcement is byte-identical to the v0.19 shape via #[serde(default, skip_serializing_if = "Vec::is_empty")]. Direct struct construction in v0.19 code (uncommon — most callers go through CapabilityAnnouncement::new) needs the three new fields explicitly. CapabilityAnnouncement::from_bytes rejects (returns None) on any announcement whose allow-list axis exceeds 64 entries.

RpcStatus::CapabilityDenied + RpcError::CapabilityDenied

New canonical status code 0x0008 and matching typed error variant. The reserved canonical-status range shifts to 0x0009..=0x7FFF. Application-layer status codes (0x8000..=0xFFFF) are unaffected. Callers matching on RpcError exhaustively need a new arm for CapabilityDenied { target, capability }; default_retryable returns false for the variant.

serve_rpc auto-self-indexes

MeshNode::serve_rpc now synchronously self-indexes a fresh CapabilityAnnouncement carrying every currently-registered nrpc:<service> tag before installing the dispatcher, and schedules a peer-side broadcast in the background. Callers that previously relied on serve_rpc being a no-op against the local capability index will see a self-announcement appear there immediately; callers that registered services before calling announce_capabilities no longer need to remember to re-announce.

Channel-name canonicalization

ChannelName::new lowercases names on construction and rejects trailing / leading dots, repeated dots within a segment, and empty/dot-only segments. Existing all-lowercase, dot-segmented names are unaffected. Operators with mixed-case registered names need to lowercase the registration; subscribers automatically address the lowercased form via the new normalization. authorize_subscribe registry-miss-with-no-matching-prefix now returns Unauthorized rather than the prior permissive default — channels with no ACL configured at all are deny-by-default.

PermissionToken generation epoch

PermissionToken wire size grows from 169 bytes to 173 bytes — a generation: u32 field rides inside the signed payload so the cache can drop every descendant when an issuer rotates. Pre-v0.20 tokens (169 bytes) are rejected on decode; reissue tokens at v0.20. Short-TTL tokens roll naturally; long-TTL tokens require an explicit reissue pass.

delegate not_after cap

PermissionToken::delegate caps the child's not_after at min(parent.not_after, now + DELEGATION_MAX_TTL) where DELEGATION_MAX_TTL is a substrate constant (default 30 days, operator-tunable on MeshNodeConfig::delegation_max_ttl). Existing long-lived delegations from v0.19 continue to verify until natural expiry; new delegations cap at the constant.

MeshNodeConfig::clock_skew_tolerance_secs

New field on MeshNodeConfig (default 60 seconds). Symmetric tolerance window applied to both ends of PermissionToken::is_valid. Pre-v0.20 callers using struct-literal construction of MeshNodeConfig need to add the field; MeshNodeConfig::new(addr, psk) is unaffected (uses the default).

CapabilityAnnouncement::strip_reserved_metadata

New method on CapabilityAnnouncement. The receive path (handle_capability_announcement) now calls it on every inbound peer announcement before metadata is consulted by greedy admission, placement scoring, or any other substrate decision-maker. Custom dispatchers that route around handle_capability_announcement should call ann.strip_reserved_metadata() after from_bytes and before consuming ann.capabilities.metadata.

How to upgrade

1. Operators issuing restrictive policies. The new net-mesh cap announce subcommand builds and signs a CapabilityAnnouncement carrying allow-lists for the supplied identity. Pipe stdout into your gossip path or use --out <PATH> to write to disk. Use the same --key <PATH> your other operator subcommands already accept. Revocation is a new cap announce at a bumped --version with a tighter allow-list — there's no separate revoke verb.

2. Callers handling RpcError. Add a RpcError::CapabilityDenied { target, capability } arm to exhaustive match expressions. The variant is non-retryable; surface it to the application layer rather than looping. Caller-side call_service filters the candidate set before target selection, so a CapabilityDenied return from call_service means every peer advertising the service refused this caller — distinct from NoRoute, which means no peer advertises the service at all.

3. Servers using serve_rpc. No code change required. serve_rpc self-indexes a permissive baseline announcement immediately on registration and schedules a peer-side broadcast. Operators who previously called announce_capabilities after serve_rpc can remove the call (it's now redundant); operators who called it before serve_rpc no longer need to remember the order.

4. Servers wanting to restrict access. Publish a policy announcement via net-mesh cap announce (offline build + ship) or by constructing a CapabilityAnnouncement with non-empty allow-lists and folding via your existing capability-broadcast path. The version bump rule still applies — restrictive policies use a version strictly greater than any prior announcement from the same node_id.

5. Subnet and group membership. Peers self-declare via subnet:<hex32> and group:<hex64> tags on their own announcement. Use the CLI's --tag subnet:<value> and --tag group:<value> to add them, or CapabilitySet::add_tag(...) programmatically. Operators picking subnet/group identifiers can use random bytes (value-as-secret) or a blake2s-of-name (advisory routing).

6. Reissue tokens. v0.19 tokens (169 bytes) fail decode on v0.20 (which expects 173 with the generation epoch). Run your token-mint pipeline against the v0.20 SDK and propagate the new tokens to every client. Short-TTL tokens roll naturally; long-TTL tokens require an explicit reissue pass.

7. Rotate compromised issuers. Call EntityKeypair::bump_generation() on the rotated issuer, then re-issue any still-current tokens against the new generation. Every cache holding a descendant of the old generation drops the descendant on the next check() — no per-token revocation entry, no CRL gossip, just the generation cross-check the cache already runs.

8. Channel-name casing. Lowercase any registered channel names that contain uppercase characters. The new canonicalization treats foo.bar and FOO.BAR as the same channel rather than parallel namespaces. Subscribers calling subscribe(name) against a previously-uppercase name continue to work — the lowercase normalization is applied symmetrically on both sides.

9. Clock-skew tuning. The default 60-second clock_skew_tolerance_secs covers typical NTP-synced nodes. Tighten via MeshNodeConfig::with_clock_skew_tolerance_secs(secs) on networks with strict clock discipline; widen for satellite or air-gapped deployments where the clock source drifts more.

10. Operator dashboards. New per-node metrics: capability_denied_caller_side and capability_denied_callee_side (counts of denials by each gate), subnet_parse_collapsed_multi_tag (counts announcements with multiple subnet tags that collapsed to no membership), token_dropped_by_generation (counts tokens dropped by a generation-epoch bump). All start at zero in steady state and only fire under attack or misconfiguration; alarm on sustained non-zero.

Named after Guns N' Roses's 1987 Appetite for Destruction opener. v0.15 stood up the Dataforts data plane. v0.16 stacked the MeshDB query plane on top. v0.17 stacked the MeshOS behavior plane on both. v0.18 is the operator plane — the TUI cyberdeck, the command-line surface, and the daemon-author / operator SDKs across five languages — that turns the substrate the prior releases built into something a human can see, command, and break-glass.

The operator plane

For three releases the substrate has been growing in capabilities that nothing outside the cluster could observe. v0.15 made replicas place themselves and blobs move under operator-defined policies; v0.16 made every chain federally queryable; v0.17 turned every node into a single reconciling event loop with admission control and an admin-event ledger. By the end of v0.17 the cluster was a living distributed operating system — and the only way to interact with it was to write Rust against the substrate crate. v0.18 closes that gap.

Three surfaces land in this release. Deck is the operator TUI — a real-time terminal cyberdeck rendering everything MeshOS, MeshDB, RedEX, and Dataforts are doing, with signed admin actions, ICE break-glass overrides, and a blast-radius preview that simulates every dangerous action before it commits. Net CLI is the command-line operator surface — the same admin verbs Deck exposes (drain / cordon / maintenance / drop-replicas / restart-all / invalidate-placement / clear-avoid-list), the same ICE break-glass actions, the same audit-chain reads, the same NetDB local-store + MeshDB query surfaces, every command JSON-output-able for scripting, every operation gated by the same operator identity. MeshOS SDK and Deck SDK ship daemon-author and operator-tooling surfaces in Rust (canonical), Python (pyo3), TypeScript (napi-rs), Go (cgo), and C (raw FFI) — five-language parity behind one common substrate-side wire contract.

There is no separate observability service to provision. There is no admin RPC to harden. The TUI, the CLI, and every binding compose against the same MeshOsRuntime + DeckClient + admin-chain primitives the substrate already ships. Operators see the cluster move, and they can move with it, in whatever language they already write.

v0.18 lands the full Deck TUI (cluster topology map, replica + placement inspector, daemon supervision panel, maintenance node control, behavior timeline, admin surface with signed ops, MeshDB console, log matrix, operator identity + audit trail, node inventory, ICE break-glass with blast-radius preview — see DECK_FEATURES.md for the operator-facing tour), the Net CLI Phase 1 + 2 + 3 (read-only inspection, mesh + nRPC client + capability surface, admin verbs + ICE preview + audit + identity store + NetDB + MeshDB query — see NET_CLI_PLAN.md), the MeshOS daemon-author SDK in all five languages (Rust canonical, Python pyo3 wrapper with Protocol class + async control-event iterator, TypeScript napi-rs with full TSFN-bridged daemon trait + AsyncIterable control events, Go cgo with MeshOsDaemon interface + cgo trampolines for snapshot/restore/onControl/health/saturation, C raw FFI with vtable + last-error pair — see MESHOS_SDK_PLAN.md), and the Deck operator SDK in all five languages covering DeckClient lifecycle, all 9 AdminCommands verbs, ICE break-glass simulate → commit typestate, snapshot + status-summary + log + failure + audit streams (see DECK_SDK_PLAN.md).

Every binding ships behind one common wire contract. The Python wheel, the npm package, the Go binding's bindings/go/net tree, and the C libnet_deck / libnet_meshos cdylibs all serialize the same MeshOsSnapshot + ChainCommit + StatusSummary shapes the substrate emits, with one operator-id + signature envelope across every admin commit. A Python operator's blast-radius dry-run produces the same affected_nodes / affected_replicas / affected_daemons count a Rust operator would see; a TypeScript admin verb commits the same ActionChainRecord a CLI invocation would. Cross-language SDK consumers see one cluster.

The hardening posture from the Black Diamond → Rebel Yell → Eye of the Tiger → Atomic Playboys line continues. Five coordinated code-review passes landed before the v0.18 branch cut.

The Rust crate now ships the same default feature stack as the Python wheel and the Node npm package (net, nat-traversal, cortex, meshdb, meshos, dataforts) — cargo add ai2070-net previously gave you nothing; v0.18 gives Rust consumers the full operator stack out of the box. The redis external-service dep stays opt-in. The Python and Node defaults are unchanged.

No new dependencies. No protocol changes. The crate version moves from 0.17.x to 0.18.0.

Deck

The operator cyberdeck. Lives in the workspace member deck/ (binary-only — [[bin]] with no [lib]).

deck/src/
├── app.rs              — main event loop + tab routing
├── tabs/
│   ├── net_map.rs      — cluster topology map (RTT edges, avoid-list, maintenance flags)
│   ├── replicas.rs     — replica + placement inspector
│   ├── daemons.rs      — daemon supervision panel
│   ├── daemon_page.rs  — per-daemon log tail + control surface
│   ├── nodes.rs        — node inventory + saturation trends
│   ├── node_page.rs    — per-node deep dive
│   ├── behavior.rs     — MeshOsSnapshot timeline
│   ├── blobs.rs        — blob explorer (replica locations, heat, ancestry)
│   ├── dataforts.rs    — local + remote adapter telemetry
│   ├── admin.rs        — signed admin surface
│   ├── ice.rs          — break-glass overrides with blast-radius
│   ├── meshdb.rs       — MeshDB query console
│   ├── logs.rs         — RED/HEAT/INFO log matrix
│   ├── audit.rs        — operator audit trail
│   ├── failures.rs     — recent-failure ring
│   └── groups.rs       — ReplicaGroup / ForkGroup / StandbyGroup roster
├── widgets/
│   ├── confirm.rs      — blast-radius confirm modal
│   ├── footer.rs       — toast + status footer
│   └── cursor.rs       — cursor + `/` filter primitives
├── streams.rs          — RedEX + Deck subscription routing
├── bookmarks.rs        — persistent cluster bookmarks
├── lineage.rs          — chain ancestry walks
└── demo/               — 9-node spawn harness for local development

The TUI composes against DeckClient (the operator SDK that ships in the same release — see below). Every tab is a ratatui widget that reads from a MeshOsSnapshotReader + a set of stream readers, renders at 60 fps when there's activity, idles at 1 Hz otherwise. The cursor + / filter primitives are tab-uniform; g / G jump to top / bottom on every cursor tab; Enter opens a detail page; ? shows context-aware help. A blast-radius confirm modal pre-flights every admin commit — the modal prints "This action affects N nodes, M replicas, K daemons. Type YES to confirm" and refuses to dispatch if the operator types anything else. Warnings beyond the modal's 3-row cap surface as … +N more (see AUDIT).

The ICE break-glass surface is the cyberpunk SRE panel — seven force-level operators (force-drain, force-evict-replica, force-restart-daemon, force-cutover, kill-migration, flush-avoid-lists, freeze-cluster / thaw-cluster) each gated through a simulate() → commit(signatures) typestate. The simulation runs the same reconcile arms the production loop runs and surfaces the projected blast radius (which replicas move, which daemons restart, which nodes become hot, expected drain delay, placement stability impact). The commit threshold defaults to 1-of-1 in development; production deployments raise it to 2-of-N via DeckClientConfig::ice_signature_threshold and the M-of-N gate enforces operator-id deduplication before counting signatures.

Behavior visibility folds back through the MeshDB query plane the prior releases shipped. The behavior timeline tab reads MeshOsSnapshot from a MeshQuery::Latest against the per-node snapshot chain — no Deck-specific RPC, no separate observability stack. The MeshDB console tab is a fully-interactive query editor: write a QueryBuilder chain in the operator's preferred shape, hit Enter, watch the federated executor route to a node holding the relevant fold, scroll the streaming result rows.

The render pipeline carries the polish details a TUI needs to feel alive: net_map's unreachable peers render as the hollow diamond the legend advertises (not a red filled diamond — the second-pass review caught this); tabs::logs reuses an ascii_icontains helper instead of lowercasing the haystack per record per render (the previous form reintroduced a per-frame String alloc); the BLOBS poll unions every wired adapter (not just blob_adapters[0]) and surfaces per-adapter errors as footer toasts; the cursor_to_bottom for DATAFORTS uses the visible row count (not blob_adapters.len()) so G lands on the last visible row when remote dataforts exist. tabs::short_id is canonical across daemon_page and groups (0xXXXXXX, 6-padded). The fmt_ts_hms_ms and unix_now_ms helpers are hoisted once so the three prior copies don't drift.

Net CLI

The command-line operator surface. Lives in cli/ (a [[bin]]-only workspace member; binary name net).

cli/src/
├── main.rs        — clap dispatch + global flags
├── context.rs     — identity / config / output-format plumbing
├── identity.rs    — operator key generate / load / rotate
├── node.rs        — node start / status / health
├── chain.rs       — RedEX chain inspect + tail
├── netdb.rs       — local NetDB CRUD (tasks / memories)
├── meshdb.rs      — federated MeshDB query
├── admin.rs       — 9 signed admin verbs
├── ice.rs         — 7 ICE break-glass verbs with simulate → commit
├── audit.rs       — admin + ICE audit reads
├── logs.rs        — log stream subscription + filter
├── daemon.rs      — daemon roster + supervision
├── rpc.rs         — typed nRPC client
└── version.rs     — semver + feature surface report

The CLI is the same admin surface Deck exposes, mapped onto clap subcommands. net admin drain --node N --drain-for 30s commits the same ChainCommit Deck's admin tab would; net ice freeze-cluster --reason "incident X" --preview runs the same simulate-before-commit pre-flight Deck's ICE tab runs and prints the blast-radius JSON to stdout for scripting; net netdb tasks ls --json reads the local NetDB store and emits JSON for piping into jq. Every command honors --output {pretty,json} for human vs. script output and exits with stable error codes — 0 success, 1 generic failure, 2 invalid arguments, 3 not found, 4 already exists, 5 permission denied (no operator key for an admin commit), 6 not connected, 7 timeout, 8 confirmation refused on a non-TTY ICE without --yes.

The ICE preview workflow is locked in. net ice <verb> --preview runs IceProposal::simulate() and prints the BlastRadius JSON without committing — the same dry-run shape that ships in every SDK. net ice <verb> --yes commits without the TTY confirmation gate; net ice <verb> alone (TTY) prints the simulation and prompts for Type YES to confirm: before reaching the substrate. The TTY-only --yes path is the same gate net admin uses for cordon / drain / drop-replicas — the simulation isn't the gate, the confirmation is.

The identity store at ~/.config/net/operator/<name>.key is the canonical authentication source for every admin / ICE commit. net identity generate <name> creates a fresh ed25519 keypair, prints the public key for registry installation, and writes the private key to disk with 0600 mode. net identity ls enumerates installed identities; net identity rotate <name> rotates with audit-trail commitment. The CLI refuses to commit an admin or ICE event under an ephemeral keypair — admin writes require an explicit operator identity (the second-pass review caught the silent ephemeral-keypair fallback).

JSON output is stable. OperatorIdentity's operator_id is a u64 decimal; ChainCommit's event_kind is a string enum (drain / enter-maintenance / cordon / etc., not the Rust Debug form — the first-pass review caught TaskRow and MemoryRow shoving debug-printed structs into named fields and corrected them); timestamps are ISO 8601 with Z suffix; durations honor humantime input but always emit milliseconds-as-integer.

86 help-text snapshot tests pin every subcommand's --help output so wording can't drift accidentally. 11 exit-code tests cover the documented exit-code contract end-to-end through a real substrate boot. The CLI is the operator surface; "scriptable + stable + signed" is the contract.

MeshOS SDK

The daemon-author SDK. Mirrors the canonical Rust surface from v0.17 (MeshOsDaemonSdk + MeshOsDaemonHandle) in four more languages with one shared wire contract. Lives in:

• Rust (canonical): sdk/src/meshos/ — MeshOsDaemonSdk::start(config) / register_daemon(daemon, identity) returning MeshOsDaemonHandle with next_control() / try_next_control() / publish_log() / publish_capabilities() / graceful_shutdown().
• Python (pyo3): bindings/python/python/net/ + sdk-py/src/net_sdk/meshos.py — MeshOsDaemon Protocol class for type-checker verification; MeshOsDaemonSdk.start() returns a handle with sync next_control + async anext_control + async for ev in handle iterator; context-manager dunders drive graceful shutdown on scope exit.
• TypeScript (napi-rs): bindings/node/ + sdk-ts/src/meshos.ts — registerDaemon(daemon: DaemonObjectTsfns, identity) accepts a full daemon object with TSFN-bridged process / snapshot / restore / onControl / health / saturation; the napi MeshOsDaemonHandle exposes async control-event iteration through an AsyncIterable<DaemonControl>.
• Go (cgo): bindings/go/net/meshos.go — MeshOsDaemon interface with cgo trampolines (goMeshOsProcessTrampoline, goMeshOsSnapshotTrampoline, goMeshOsRestoreTrampoline, goMeshOsOnControlTrampoline, goMeshOsHealthTrampoline, goMeshOsSaturationTrampoline); MeshOsDaemonSdk.RegisterDaemon returns a handle with ControlEvents() <-chan DaemonControl (context.Context-cancellable) + TryNextControl().
• C (raw FFI): include/net_meshos.h + libnet_meshos.{so,dylib,dll} — NetMeshOsDaemonVtable carrying name / process / health / saturation / on_control / snapshot / restore function pointers; net_meshos_register_daemon(sdk, &vtable, ctx, &identity) returning an opaque handle; net_meshos_next_control(handle, timeout_ms, out) for blocking control reads; per-thread net_meshos_last_error_message / net_meshos_last_error_kind discriminator after every non-OK return.

Daemon-side only by lock. No placement APIs in any binding. No admin-event issuance. No MeshOS-control surfaces. The SDK is the daemon contract in five languages; operator tooling, federated interactions, and MeshDB queries belong to separate SDKs (the Deck SDK below, plus the existing MeshDB SDK that v0.16 shipped).

The cross-language hardening list is real work. The Go MeshOsDaemonHandle.Free() now blocks on the pump goroutine's in-flight NextControl before the C-free, so a concurrent shutdown can't race the cgo destructor (the second-pass review caught this); the Python PyDeckClient.close() path no longer swallows shutdown errors asymmetrically vs. Node; the Python standalone constructor gains __enter__ / __exit__ + __del__ so dropping a client without an explicit close still tears down the supervisor; the TypeScript handle classes gain explicit dispose() / [Symbol.dispose]() for TC39 explicit resource management; the Go Deck streams pick up the same Close-vs-Next race fix the MeshOS streams shipped; operator seed bytes are zeroized at the binding boundary across Node / Python / Go standalone constructors.

Cargo features gate the feature surface uniformly: meshos activates the runtime symbols, cortex activates the snapshot fold layer the runtime composes against. Wheels / npm packages / Go cdylibs ship the full default set; bindings/python/python/net/__init__.py and the npm @ai2070/net package's lazy feature checks raise ImportError (Python) or surface a typed missing-feature error (Node) rather than AttributeError if the wheel was built without a feature.

The Python _net.pyi stub now carries full method signatures for every feature-gated class — MeshOS (MeshOsDaemonSdk, MeshOsDaemonHandle), MeshDB (MeshQuery, MeshQueryRunner, QueryBuilder, Predicate, the result-row family), and Deck (every class below). A test_stub_drift.py regression test parametrizes over every stub class and asserts the runtime symbol matches, so future PyO3 method renames trip CI rather than silently break IDE autocomplete.

Deck SDK

The operator-tooling SDK. Mirrors the Rust DeckClient surface in four more languages with one shared substrate-side wire contract:

• Rust (canonical): sdk/src/deck/ — DeckClient::new(operator_identity, config) / from_runtime(runtime, ...) returning a client with .admin() / .ice() / .audit() / .snapshots() / .status_summary_stream() / .subscribe_logs(filter) / .subscribe_failures(since_seq).
• Python (pyo3): bindings/python/python/net/ + sdk-py/src/net_sdk/deck.py — DeckClient(operator_seed, config) + DeckClient.from_seed(seed, **config_kwargs) wrapper; admin verbs return typed ChainCommit dataclasses; ICE break-glass typestate enforced through IceProposal → SimulatedIceProposal → commit(signatures); context-manager dunders drive shutdown on scope exit.
• TypeScript (napi-rs): bindings/node/ + sdk-ts/src/deck.ts — DeckClient with readonly admin / readonly ice properties holding typed verb dispatchers; async-iterable SnapshotStream / StatusSummaryStream / LogStream / FailureStream; DeckSdkError carries the structured kind discriminator the substrate emits.
• Go (cgo): bindings/go/net/deck.go + top-level go/deck.go companion — DeckClient.Admin().Drain(node, drainForMs) / DeckClient.ICE().FreezeCluster(reason, ttl).Simulate() / .Commit(signatures); the stream surfaces use buffered <-chan with context-aware shutdown; DeckError wraps the substrate's <<deck-sdk-kind:KIND>>MSG envelope. The binding-tier bindings/go/net/deck.go covers all three slices (admin + ICE + logs + audit + failures); the top-level go/deck.go ships slice 1 (admin + status streams) — slices 2/3 callers use the binding tier directly.
• C (raw FFI): include/net_deck.h + libnet_deck.{so,dylib,dll} — net_deck_client_new(this_node, …, operator_seed, &out) constructor; 9 net_deck_admin_* verbs; 7 net_deck_ice_* factories returning NetIceProposal* that consumes itself on _simulate (yielding NetSimulatedIceProposal* that consumes itself on _commit); snapshot / status-summary / log / failure / audit streams; per-thread net_deck_last_error_kind / net_deck_last_error_message.

The break-glass typestate is enforced across every language. IceProposal carries an issued_at_ms-tagged signing payload with a substrate-side nonce, domain-separation prefix, and one-minute commit-window expiry; simulate() builds a SimulatedIceProposal and freezes the substrate's reconcile arms against the proposal's effect projection; commit(signatures) verifies signatures against the operator registry's M-of-N threshold (deduplicating by operator id before counting — same operator can't sign twice), then commits to the admin chain and clears the per-node ICE cooldown. The substrate enforces simulate-before-commit; commit without a fresh simulation returns consumed rather than re-running simulate from scratch. The simulate path consumes itself on success — a second simulate() on the same proposal returns consumed, not a fresh blast-radius (the first-pass review caught the consumed-state sentinel reading back as a valid timestamp; the typestate flip closes both regressions).

impl DeckClient {
    pub fn new(identity: OperatorIdentity, config: DeckClientConfig) -> Self;
    pub fn from_runtime(runtime: &MeshOsRuntime, identity: OperatorIdentity, ...) -> Self;
    pub fn with_operator_registry(self, registry: OperatorRegistry) -> Self;

    pub fn snapshots(&self) -> SnapshotStream;
    pub fn status(&self) -> Result<MeshOsSnapshot, DeckError>;
    pub fn status_summary(&self) -> Result<StatusSummary, DeckError>;
    pub fn status_summary_stream(&self) -> StatusSummaryStream;
    pub fn subscribe_logs(&self, filter: LogFilter) -> LogStream;
    pub fn subscribe_failures(&self, since_seq: Option<u64>) -> FailureStream;
    pub fn audit(&self) -> AuditQuery;

    pub fn admin(&self) -> AdminCommands<'_>;
    pub fn ice(&self) -> IceCommands<'_>;
}

pub struct DeckClientConfig {
    pub snapshot_poll_interval: Duration,
    pub ice_signature_threshold: usize,  // M-of-N for ICE commits
}

AdminCommands exposes the 9 substrate admin verbs (drain / enter-maintenance / exit-maintenance / cordon / uncordon / drop-replicas / invalidate-placement / restart-all-daemons / clear-avoid-list). IceCommands exposes the 7 break-glass operators (freeze-cluster / thaw-cluster / flush-avoid-lists / force-evict-replica / force-restart-daemon / force-cutover / kill-migration). The AuditQuery builder is fluent — .recent(n) / .by_operator(id) / .between(start, end) / .force_only() / .since(seq) / .collect() / .stream() — and surfaces the same admin-event ledger across every binding.

The operator registry is the M-of-N gate. OperatorRegistry::insert(id, public_key) registers operators; verify(payload, signatures) returns Ok(()) when the signature set meets the threshold and rejects duplicates by id. The registry survives RedEX chain replay so registrations made on one node propagate to every other node; bundle-verify (verify_bundle) covers the multi-signature ICE-commit path with the same dedup-by-id rule.

Default feature parity

net/crates/net/Cargo.toml now ships defaults matching the Python wheel and Node npm package, minus redis:

[features]
default = [
    "net",
    "nat-traversal",
    "cortex",      # → redex, redex-disk transitively
    "meshdb",
    "meshos",
    "dataforts",
]
redis = ["dep:redis"]   # opt-in: external service dep

cargo add ai2070-net now gives Rust consumers the same operator stack Python and Node consumers already get. Existing Rust consumers who want the prior minimal surface can opt out with --no-default-features.

The Cargo features section at the bottom of every binding README (bindings/python/README.md, bindings/node/README.md, bindings/go/net/README.md, sdk-py/README.md, sdk-ts/README.md) documents the five relevant feature flags (cortex, redex-disk, netdb, meshdb, meshos), what each enables, and the build invocation for each binding so consumers building from source know what to pass.

C ABI consolidation

The C SDK header story is cleaned up. net.h is the bus + shared error enum; net_cortex.h is the RedEX + CortEX + NetDb surface (new in this release — split out of the prior net.go.h catch-all); net_rpc.h is the RPC surface; net_meshdb.h is the MeshDB query layer; net_meshos.h is the MeshOS daemon-author surface; net_deck.h is the Deck operator surface. The convenience net.go.h #includes net_cortex.h and inlines RPC + Deck declarations for callers who want everything in one place.

The NetDb FFI lands in this release as well — net_netdb_open / net_netdb_open_from_snapshot / net_netdb_snapshot / net_netdb_tasks / net_netdb_memories / net_netdb_close / net_netdb_free plus net_netdb_free_bundle for the snapshot bytes. Adapter accessors hand out independent Arc-cloned net_tasks_adapter_t* / net_memories_adapter_t* handles — freeing them does NOT close the underlying adapter, and the NetDb itself can be freed before the adapter clones. The Go binding consumes this through bindings/go/net/netdb.go; Python and Node already consumed the Rust core directly.

The MeshDB and MeshOS cdylibs are unchanged (still separate libraries — libnet_meshdb.{so,dylib,dll}, libnet_meshos.{so,dylib,dll}). The new libnet_deck.{so,dylib,dll} cdylib ships alongside, built from the net-deck-ffi workspace member at bindings/go/deck-ffi. C consumers link the libs they want.

Substrate hardening — pre-watcher pass

Alongside the SDK / TUI / CLI work, a three-pass bug audit closed 42 substrate + CLI items across 42 commits before the v0.18 branch cut (BUG_AUDIT_2026_05_17_NET_CLI.md). Pass 1 covered the Net CLI command surface (17 items). Pass 2 extended outward into adapter/net/**, sdk/src/**, and the Python / Node / Go bindings (11 items). Pass 3 was specifically scoped to the layers a future netdb-watcher subscriber will sit on top of — the cortex adapter's fold loop, RedexFile::tail / append, and the NetDb façade (9 items). These are not the kind of bug that surfaces in unit tests at low load — they manifest under burst-write contention, lagged subscribers, or after the first restart. The Criticals would have made a watcher look broken; closing them ahead of any consumer means the watcher writes against substrate that already behaves.

Three Criticals closed in the cortex fold loop. The fold task used to be tokio::spawn-ed after file.tail(start_seq) already registered the live watcher — between registration and the spawned task being polled, concurrent appends could call notify_watchers, evict the watcher with try_send(Err(Lagged)), and the fold task's first stream.next() would yield Some(Err(Lagged)) and break out of the loop before processing a single event. Moving the tail call to be the first statement inside the spawned task gives registration and consumption a deterministic ordering. The second Critical was the live Lagged match arm permanently killing the adapter — any subscriber falling behind tail_buffer_size once now re-subscribes at folded_through_seq + 1 instead of silently halting the fold task forever. The third was wait_for_seq returning Ok(()) when running == false, which couldn't distinguish "your seq is folded" from "the fold task crashed before reaching your seq"; it now returns Err(folded_through_seq()) mirroring the sibling wait_for_applied_seq's contract, so a watcher polling for a seq the substrate will never reach gets a typed signal instead of false success.

Three Highs closed across the RedEX and watermark layers. notify_watchers previously fired before fsync, so a crash after the broadcast but before the kernel flushed the page cache could lose an event from disk that subscribers had already acted on; the durability contract is now explicit and watchers reconcile from last_persisted_seq + 1 on restart by (channel, seq) key. The next_seq.fetch_add(1) allocator could be visible to a concurrent LiveOnly opener across a failed-write rollback — a LiveOnly adapter opening during the rollback window would start its tail at the inflated seq, then silently filter out the real append at the lower seq. The applied_through_seq strict-prefix advance used wrapping_add(1), which collided with the u64::MAX "nothing applied yet" sentinel when seq reached the boundary — the snapshot would persist None, restore would re-replay everything, and every pending wait_for_applied_seq would block forever. All three close before the watcher tier writes.

Three Mediums closed in the NetDb façade and the cortex changes_tx ordering. NetDbBuilder::build() with both want_tasks=false and want_memories=false used to silently return a no-op NetDb whose accessors panicked on first use — combined with the CLI's --with-tasks / --with-memories flag work, a misconfigured profile or test fixture would have turned a config error into a process panic. The builder now returns NetDbError::NoModelsEnabled explicitly. NetDb::snapshot()'s sequential per-adapter capture documents its lack of a cross-model barrier so watchers snapshotting between event deliveries know to coordinate ordering. The changes_tx.send(seq) edge-trigger broadcast moved inside the state write-lock block so subscribers treating the seq value as authoritative ordering can't observe out-of-order seqs under contention.

The CLI-side fixes touch every operator-facing path. The restore safety gate no longer treats I/O errors as "store is empty" and lets restore overwrite a populated dir without --force. The --origin flag now requires an explicit value (or --allow-origin-zero) so a stray missing flag can't silently fold against the wrong chain. Snapshot writes are atomic (tmp.<pid> → fsync → rename → fsync parent) so a crash mid-write doesn't truncate the operator's previous snapshot. Operator seed files are created with OpenOptions::mode(0o600).create_new(true) so the seed never hits disk world-readable. The Windows strict-permissions path warns unconditionally on reads without --insecure-permissions instead of silently no-oping the gate the module-header doc promises. --force restore is now scoped to the replace semantic the verb name implies, not the silent-merge it was doing before. The ICE confirm prompt uses tokio::io::stdin so a blocking read doesn't park a Tokio worker thread for as long as the operator stares at the gate. The netdb subcommand finally honors --config / --profile (it was the only top-level that ignored both, so an operator with netdb = "/srv/netdb" in their profile would silently land in the default XDG path and write mutations into the wrong store).

The pass-2 substrate items hit the SDK and bindings: a thundering-herd retry jitter source that was contributing ~0 ns of entropy now seeds from a process-epoch Instant; three u32 → u8 truncation bugs in the Go compute-ffi spawn paths (replica count 300 was silently becoming 44) now mirror the scale_to validation that already existed alongside; the tasks/memories adapter fetch-add-then-ingest path either re-instates the rollback or rewrites the contract docs to acknowledge the gap; the set_local_capabilities lost-update race between fetch_add and the subsequent store(version) is corrected so the capability version monotonically advances; the loadbalance connections.fetch_sub(1) underflow that silently removed endpoints from rotation forever now saturates.

A handful of items reached "obsolete" rather than "fixed" — re-reading the code showed the audit had misread the contract (the has_more cursor advances correctly via last_seen_seq, PyNetDb::open's make_runtime() is already a process-wide OnceLock<Arc<Runtime>> singleton so multiple adapters share one runtime, next_seq() already takes the state lock per the existing docstring, changes_tx.send runs sequentially from the spawn task's loop so the ordering already matches the watermark ordering). The audit doc records them under Obsolete so a future reader knows the agents looked at those call sites and the contract was already correct.

The watcher work itself follows this release. The substrate beneath it is now clean.

Test hygiene

• Lib suite at 3115+ tests (was 2715+ at v0.17 release). 400+ net new tests across the Deck TUI snapshot suite + the cross-language SDK surfaces + the Net CLI exit-code / help-text regression suite + the substrate-side ICE / operator-registry / blast-radius simulation tests + the action-chain MeshOsSnapshot postcard / JSON forward-compat regression layer.
• cargo clippy --features meshos,deck --all-targets -- -D warnings clean across substrate + every binding crate + the deck demo + the deck TUI + the net CLI.
• cargo doc --features meshos,deck --no-deps clean under RUSTDOCFLAGS="-D warnings" — every public item in the v0.18 surface carries a doc comment; intra-doc links resolve through the public re-exports.
• CI matrix expanded. The Go CI step builds net-deck-ffi alongside the existing net-compute-ffi, net-meshdb-ffi, net-meshos-ffi cdylibs so the new go/deck.go cgo block links. The Python CI step builds with meshdb enabled so test_meshdb.py and the new test_stub_drift.py MeshDB class coverage exercise on every run.
• 86 help-text snapshot tests pin every Net CLI subcommand's --help output. 14 deck-pipeline integration tests cover the substrate-end-to-end behavior the TUI relies on (MeshOsSnapshot publish → MeshOsSnapshotFold consume → MeshDB Latest query → TUI render). 11 exit-code tests cover the documented Net CLI exit-code contract through a real substrate boot.

Breaking changes

Crate-level default features

ai2070-net's default = [...] moves from [] to ["net", "nat-traversal", "cortex", "meshdb", "meshos", "dataforts"]. Existing Rust consumers who add the crate without a --no-default-features flag will compile a larger feature surface and pull additional transitive deps (chacha20poly1305, snow, ed25519-dalek, x25519-dalek, blake3, postcard, tokio-stream, async-trait). No code breakage — every default-activated feature is stable; consumers paying for the previous minimal surface should --no-default-features and re-add what they need.

Workspace — new members

cli/ (Net CLI binary) and bindings/go/deck-ffi/ (Deck cdylib) are new workspace members. Existing build invocations are unaffected; consumers who cargo build without -p <name> will see the new members compiled by default. cargo build --workspace --release now produces five cdylibs (libnet, libnet_compute, libnet_meshdb, libnet_meshos, libnet_deck) and one new bin (net).

MeshOsSnapshot wire format

The snapshot's postcard wire format gains a wire_version: u8 prefix that the MeshOsSnapshotFold's decoder checks before postcard dispatch. Existing consumers reading raw postcard bytes need to strip the version byte before decode; consumers using MeshOsSnapshotReader::read() are unaffected (the reader returns the decoded struct). Regression tests pin a captured legacy byte string so accidental field reorders trip CI.

Deck SDK signing payload

ICE-commit signing payloads now carry a domain-separation prefix (b"net-deck-ice:"), a substrate-side nonce, and a one-minute commit-window expiry. Operators who had cached a signed payload from a prior version must re-sign — the substrate verifies the prefix + expiry before accepting the signature. The substrate-side bump is invisible to operators using the SDK's simulate() → commit(signatures) typestate; only consumers who hand-rolled the signing-payload bytes need to update.

Python MeshOsDaemonSdk.start signature

The MeshOS SDK plan documented a callback_timeout_ms parameter on MeshOsDaemonSdk.start; the runtime never accepted it (the original stub was wrong — see the cross-language review). The stub now matches the runtime: start(config: Optional[dict] = None, *, control_capacity: Optional[int] = None). Python consumers passing callback_timeout_ms to start() will see a TypeError; remove the argument.

How to upgrade

1. Bump your Cargo.toml / package.json / requirements.txt / go.mod to the v0.18 line. Rust consumers who want the prior minimal default-feature set add default-features = false to their ai2070-net dependency and re-list the features they need; everyone else gets the full operator stack out of the box.
2. Operators. Install the Net CLI via cargo install --path cli (workspace-relative) or your distro's release artifact. Generate an operator identity with net identity generate <name>, install the public key into the cluster's operator registry, and start running admin / ICE / audit commands. Run net --help for the full subcommand map.
3. Deck. Build with cargo build --release -p deck (binary at target/release/deck). Configure the deck connection target via ~/.config/net/deck.toml or the --cluster flag. Run deck from the operator workstation — the TUI auto-discovers reachable maintenance nodes and renders the cluster live. Press ? in any tab for context-aware help.
4. Daemon authors. Pick your language:
   • Rust: MeshOsDaemonSdk::start(...) returning MeshOsDaemonHandle — see sdk/src/meshos/.
   • Python: pip install ai2070-net-sdk (or build from source with --features meshos); implement the MeshOsDaemon Protocol and register via MeshOsDaemonSdk.start().register_daemon(daemon, identity).
   • TypeScript: npm install @ai2070/net-sdk; implement the daemon object shape (name, process, optional snapshot/restore/onControl/health/saturation) and pass it to registerDaemon.
   • Go: import "github.com/ai-2070/net/bindings/go/net"; implement the MeshOsDaemon interface and call meshos.RegisterDaemon(daemon, identity).
   • C: #include <net_meshos.h>; populate a NetMeshOsDaemonVtable and call net_meshos_register_daemon(...). Link against libnet_meshos.
5. Operator-tooling authors. Same per-language path with the Deck SDK: DeckClient::new(operator_identity, config) (Rust) / DeckClient.from_seed(seed) (Python) / new DeckClient({operatorSeed, ...}) (Node) / net.NewDeckClient(seed, config) (Go) / net_deck_client_new(...) (C). Drive .admin() for signed commits, .ice() for break-glass, .audit() for the admin-event ledger.
6. ICE workflow. Every break-glass operator runs through simulate() → commit(signatures). Build a proposal with client.ice().freeze_cluster(reason, ttl); call proposal.simulate().await to get a SimulatedIceProposal carrying the blast-radius projection; collect operator signatures over simulated.signing_payload(); call simulated.commit(signatures).await. The substrate enforces simulate-before-commit; commit without a fresh simulation returns consumed.
7. MeshOS daemon trait additions. If you implement MeshDaemon and want supervision participation, override health() / saturation() / on_control(DaemonControl). Defaults preserve compatibility from v0.17.
8. NetDB from Go. Go consumers who previously opened OpenTasksAdapter + OpenMemoriesAdapter separately can now use OpenNetDb(redex, NetDbConfig{...}) for the cross-adapter façade + snapshot bundle that round-trips across every binding. See bindings/go/net/netdb.go.
9. C ABI consumers. Migrate #include "net.go.h" callsites to the per-surface headers (net_cortex.h for RedEX/CortEX/NetDb, net_rpc.h for RPC, net_meshdb.h for MeshDB, net_meshos.h for MeshOS, net_deck.h for Deck). The convenience net.go.h #includes net_cortex.h automatically; existing consumers compile unchanged.

Named after Survivor's 1982 Rocky III anthem — a release that asks the substrate to see, after Rebel Yell asked it to hold. v0.15 made the Dataforts data plane stand up — content-addressed blobs, heat-driven gravity, read-your-writes. v0.16 stacks the MeshDB query plane on top: a federated AST + planner + executor that composes against the existing capability index, proximity graph, and causal: / fork-of: tag layer the Warriors substrate ships. No new substrate primitive — every operator rides what was already there. The meshdb Cargo feature gates whether the surface compiles at all; the substrate path is unchanged on non-meshdb builds.

MeshDB

MeshDB in Net is the query layer that grows on top of the substrate, and v0.16 is where it lands. Every prior approach to "query the cluster" presupposes a homogeneous shape — a SQL warehouse holds rows in tables, a graph database holds nodes in indexes, a search engine holds documents in shards. There is a query language, and there is data, and the language is shaped to the data. MeshDB inverts the relation. The data is causal chains of events across nodes; the query language composes operators against those chains; the capability index is the planner, the proximity graph is the cost model, the local RedEX file is the storage engine. There is no central catalog. There is no schema service. There is no shuffle plan.

A query in MeshDB is a tree of operators that traverse three axes the substrate already exposes — time (a chain's history at a specific seq, or across a seq range), lineage (the fork-of: graph back to a common ancestor, sibling chains, descendant cohorts), and chains (joins across causally-related but distinct chains, aggregates folded across them). The planner reads the capability index to discover which nodes hold which chains, walks the proximity graph to pick the cheapest holder, and emits an execution plan whose root operator is the data and whose leaves are remote sub-queries. Atomic operators (At / Between / Latest) read events from the substrate; composite operators (Join / Filter / Aggregate / Window / LineageEmit) compose against atomic results without owning state of their own. The runtime is per-node; the plan is per-query; the substrate is unchanged.

The same primitives that let The Warriors find a chain's holders let MeshDB find a chain's history. The same fork-of: propagation that lets Distributed RedEX replicate a chain forward lets MeshDB walk a chain's parents backward. The same PredicateWire that the Capability System uses to filter peer capabilities lets MeshDB filter rows. Hash-joins and sort-merge joins, exact-min / exact-max / exact-distinct-count / nearest-rank percentile aggregates, tumbling windows on seq, and a single-node LRU result cache all compose without a new wire protocol — every operator either rides the existing capability index, the existing RedEX read path, or a new SUBPROTOCOL_MESHDB envelope between federated executors. Plans are byte-deterministic; cache keys are content-addressed off the plan; cache invalidation is pull-based against a global CapabilityIndex mutation counter that bumps on every announcement / removal / GC sweep.

Federated execution arrives in code with the substrate. The FederatedMeshQueryExecutor fans atomic operators out to remote target_nodes via a pluggable MeshDbTransport; the LoopbackTransport drives three-node integration tests in-process. The wire-side hookup that registers the new SUBPROTOCOL_MESHDB = 0x0F00 on MeshNode's subprotocol dispatcher is the one piece that stays parked for a consumer to drive — the envelope shapes, the cancellation model, and the cross-node call multiplexing all ship in v0.16. The same model lifts MeshDB out of test-only loopback the moment a real subprotocol consumer (Hermes telemetry replay; Deck cross-rack metrics; AI fine-tuning across forked experiments) wires the dispatch.

The bindings ship in lockstep. Python, Node, Go, and C SDKs all expose the full operator surface — MeshQuery.at(...) through MeshQuery.join(...), the typed Predicate builder for filters, the fluent QueryBuilder for chained pipelines, the CachePolicy { Permanent | TimeBound { ttl } } knobs — plus a sentinel-envelope decoder that turns aggregate / joined / window result rows into host-language objects. Errors carry a structured kind discriminator (planner_error, executor_error, join_memory_exceeded, ambiguous_discovery, query_cancelled, runtime_panic, …) so callers can branch without parsing message strings. The substrate's MeshError is the single source of truth; every binding reflects it.

There is no separate query service to provision. There is no catalog to maintain. The query plan is on the mesh because the substrate is the database.

v0.16 lands the full MeshDB substrate behind the meshdb Cargo feature — AST + planner, local + federated executors, lineage walks via the fork-of: graph, hash + sort-merge joins (row-intrinsic + payload-keyed, all four JoinKinds), Count / Sum / Avg / Min / Max / DistinctCountExact / PercentileExact aggregates, Filter via synthetic-tag PredicateWire evaluation, tumbling-on-seq windowing, and the single-node LRU result cache with pull-based capability-version invalidation are all in code. The wire-subprotocol hookup that registers SUBPROTOCOL_MESHDB on MeshNode's dispatcher waits for a consumer to drive — the envelope shapes ship and the FederatedMeshQueryExecutor already speaks the protocol against a LoopbackTransport in three-node in-process integration tests today. The full surface ships across Rust core and Python / Node / Go / C SDKs.

The hardening posture from the Black Diamond / Rebel Yell line continues. Two coordinated code-review passes landed before the v0.16 branch cut, surfacing 52 items total — 9 Blockers, 19 Majors, 20 Minors, 4 Nits. Every Blocker and Major closed in-tree with regression tests; two Majors deferred with rationale (the deferred items need SDK surfaces — FederatedMeshQueryExecutor exposure, configurable budgets, Discovered resolution — that ship with their respective future slices). The four Minor deferrals all closed post-pass: substrate-side join-watermark clamp helper with f64-input tests pins the contract the Python test_join_accepts_watermark_secs_kwarg couldn't observe; substrate Unicode / singleton-aggregate / long-lineage test-gap fillers land; the Arc<LocalMeshQueryExecutor> indirection is dropped from all three runners; LineageEntry.depth is BigInt in the Node SDK for shape parity.

Alongside MeshDB, v0.16 carries a substrate-level routed-handshake replay-guard fix that was masking as a flaky NAT-traversal test. The guard previously refused any legitimate re-handshake from a peer with the same Noise static, indistinguishable from a passive attacker replaying captured msg1 bytes. The fix tracks the initiator's Noise ephemeral (in the clear at the front of NKpsk0 msg1) and only refuses replays that match BOTH static and ephemeral — a fresh ephemeral can only be produced by the static + PSK holder, per the Noise threat model. Plus a Duration::MAX-sentinel handling fix in the periodic sweep loops (spawn_token_sweep_loop, spawn_capability_gc_loop) that previously panicked on Instant-overflow when the documented "disable the sweep" sentinel was used.

The toolchain moves forward: Go 1.26, CI reads the Go version from go/go.mod (no more divergence between the local toolchain and the CI matrix), and the cross-binding cgo integration test creates responder / initiator nodes in parallel — eliminating the pre-fix handshake deadlock that randomly flaked the suite. Dependency bumps land cleanly: ctor 0.11.1 → 1.0.5, napi 3.8.6 → 3.9.0, napi-build 2.3.1 → 2.3.2, napi-derive 3.5.5 → 3.5.6.

MeshQuery AST + planner

The composable query language and the planner that translates queries into typed ExecutionPlans. Lives in src/adapter/net/behavior/meshdb/{query,planner,error}.rs.

MeshQuery versioned outer enum

pub enum MeshQuery {
    V1(QueryV1),
}

pub enum QueryV1 {
    At      { origin: ChainRef, seq: SeqNum },
    Between { origin: ChainRef, start: SeqNum, end: SeqNum },
    Latest  { origin: ChainRef },
    LineageBack    { origin: ChainRef, max_depth: u32 },
    LineageForward { origin: ChainRef, max_depth: u32 },
    Join { left: Box<MeshQuery>, right: Box<MeshQuery>,
           on: JoinKey, kind: JoinKind,
           strategy: JoinStrategy, watermark_secs: f64 },
    Filter { inner: Box<MeshQuery>, predicate: PredicateWire },
    Aggregate { inner: Box<MeshQuery>, group_by: Vec<Expr>,
                agg_fn: AggregateFn },
    Window  { inner: Box<MeshQuery>, spec: WindowSpec },
    Project { inner: Box<MeshQuery>, columns: Vec<Expr> },
    OrderBy { inner: Box<MeshQuery>, by: Vec<Expr>, limit: Option<u32> },
}

The MeshQuery::V1(...) wrapper is the stability hatch — postcard + JSON round-trip carries the version tag at the front of every wire encoding, so a v2 AST can land alongside without breaking on-disk plans. ChainRef separates direct origin-hash references (OriginHash(u64)) from capability-predicate references (Discovered(PredicateWire)); the planner resolves Discovered against the capability index at plan time and surfaces a typed MeshError::AmbiguousDiscovery { matches } when multiple origins match (deferring multi-origin fan-out until a future slice ships it explicitly, rather than silently truncating to the first match).

MeshQueryPlanner

impl<'a, F: Fn(NodeId) -> Option<Duration>> MeshQueryPlanner<'a, F> {
    pub fn new(index: &'a CapabilityIndex, rtt_lookup: F) -> Self { ... }
    pub fn plan(&self, q: &MeshQuery) -> Result<ExecutionPlan, MeshError> { ... }
}

Translates atomic operators to typed ExecutionPlans with proximity-ordered target_nodes (RTT-asc, lex-NodeId tiebreak). Composite operators wrap their planned children in NotYetImplemented placeholders so the tree still type-checks for variants outside this release's executor coverage (Project, OrderBy).

Plans are byte-deterministic. Two non-determinism leaks the review closed in this release: (1) caps.tags is a HashSet whose iteration order is RNG-stable across a single process but not across runs, so parent_of / children_of / collect_coverage collect every candidate, sort numerically, and pick the smallest; (2) CapabilityIndex::all_nodes iterates a DashMap whose order is unstable, so cross-replica fork-of selection now collects across all hosting nodes before picking. The cache key is content-addressed off the plan, so byte determinism is load-bearing for cache hit rate.

Time-travel + federated execution

🚧 Wire-side subprotocol dispatch hookup outstanding. Substrate complete; the envelope shapes, the cancellation model, and a LoopbackTransport-driven three-node integration test all ship — the one piece that waits for a consumer is MeshNode::register_subprotocol_handler(SUBPROTOCOL_MESHDB, ...).

MeshQueryExecutor async trait + LocalMeshQueryExecutor

#[async_trait]
pub trait MeshQueryExecutor: Send + Sync {
    async fn execute(&self, plan: ExecutionPlan)
        -> Result<RunningQuery, MeshError>;
    async fn execute_with(&self, plan: ExecutionPlan, options: ExecuteOptions)
        -> Result<RunningQuery, MeshError>;
}

pub struct RunningQuery {
    pub handle: QueryHandle,        // cooperative cancellation
    pub rows: ResultStream,         // Box::pin(Stream<Item = Result<ResultRow>>)
}

LocalMeshQueryExecutor<R: ChainReader> walks atomic plans against a pluggable ChainReader (in-memory store for tests; the integration layer wires it to RedEX). Cancellation flows via QueryHandle::cancel which flips an Arc<AtomicBool> checked at every row boundary.

Replica-aware routing — CausalClaim parsing

Three causal: tag forms get parsed into typed coverage claims: causal:<hex> (Presence — no range, permissive fallback), causal:<hex>:<tip_seq> (Tip — covers [0, tip_seq + 1)), causal:<hex>[start..end] (Range — covers [start, end)). The planner picks the most-specific-claim winner per holder (Range > Tip > Presence) with a deterministic tie-break key, then filters holders by covers_seq / covers_range. HistoricalRangeUnavailable carries per-replica available-range hints so callers can negotiate.

Wire protocol envelopes

pub const SUBPROTOCOL_MESHDB: u16 = 0x0F00;

pub enum MeshDbRequest {
    Execute { call_id: u64, plan: ExecutionPlan },
    Resume  { call_id: u64, token: ContinuationToken },
    Cancel  { call_id: u64 },
}

pub enum MeshDbResponse {
    Batch { call_id: u64, batch: ResultBatch },
    End   { call_id: u64 },
    Error { call_id: u64, error: MeshError },
}

Envelopes are defined and round-trip cleanly; MeshNode::register_subprotocol_handler(SUBPROTOCOL_MESHDB, ...) is the one piece that ships unwired until a consumer drives it. Substrate-side FederatedMeshQueryExecutor<T: MeshDbTransport> already speaks this protocol against LoopbackTransport in three-node in-process integration tests.

FederatedMeshQueryExecutor + LoopbackTransport

Fans atomic operators out to their proximity-ordered target_nodes over MeshDbTransport. On TransportError::NoRoute(target) the executor falls through to the next target; any other transport error bubbles up inside MeshError::ExecutorError. Composite operators (HashJoin / Aggregate* / Window / Filter) recurse on the federated executor so atomic leaves still dispatch via the transport.

Cancellation correctness. Pre-fix, each recursive execute_uncached allocated a fresh QueryHandle; the outer running.handle.cancel() was a no-op against the materialized futures::stream::iter(out) output of composite operators. Post-fix, one outer handle is allocated in execute_with and threaded through execute_uncached_with_handle into every recursive sub-fetch, and a stream_results_cancellable adapter re-checks the cancel flag per emitted row.

Call-ID uniqueness. The wire contract says call_id is "unique per (caller, executor) pair while in-flight". Pre-fix, each FederatedMeshQueryExecutor drew IDs from its own AtomicU64, so two federated executors on the same caller could collide at a shared remote demultiplexer. Post-fix, a process-global FEDERATED_CALL_ID_COUNTER trivially satisfies the contract.

Replay-guard fix in the mesh's routed-handshake path

Hardening surfaced a routed-handshake replay guard that flagged any legitimate re-handshake from a peer with the same Noise static as a passive replay attack — connect_direct(peer, via = X) against an existing session via R would time out at B's side because B refused the new handshake. The fix tracks the initiator's Noise ephemeral (in the clear at the front of NKpsk0 msg1 by Noise pattern) and only DropReplays when BOTH the static AND the ephemeral match. A fresh ephemeral can only be produced by the static + PSK holder (the legitimate peer); a captured-and-replayed msg1 has the original ephemeral verbatim.

struct PeerInfo {
    node_id: u64,
    addr: SocketAddr,
    session: Arc<NetSession>,
    remote_static_pub: [u8; 32],
    last_initiator_ephemeral: Option<[u8; 32]>, // new
}

fn routed_rotation_outcome(
    existing: &PeerInfo,
    new_static: &[u8; 32],
    new_ephemeral: &[u8; 32],
    session_timeout: Duration,
) -> RoutedRotationOutcome {
    if existing.remote_static_pub == *new_static {
        if existing.last_initiator_ephemeral.as_ref() == Some(new_ephemeral) {
            return RoutedRotationOutcome::DropReplay;
        }
        return RoutedRotationOutcome::AcceptRotation;
    }
    if existing.session.is_timed_out(session_timeout) {
        RoutedRotationOutcome::AcceptRotation
    } else {
        RoutedRotationOutcome::RefuseFresh
    }
}

Lineage walks via fork-of: graph

OperatorPlan::LineageEmit { origin, direction, entries } carries a materialized walk result. The planner walks the local capability-index snapshot at plan time — parent_of for back, BFS children_of lex-sorted for forward, both deterministic across runs. Cycle detection ships as explicit visited-set guards (MeshError::LineageCycleDetected { origin, cycle } with the path through the cycle for debugging). Depth bounds surface as MeshError::LineageMaxDepthExceeded { origin, depth }.

The executor emits one ResultRow per entry — payload empty, origin = entry.origin, seq = entry.tip_seq.unwrap_or(SeqNum(0)). Callers compose with At / Between to fetch event content for each ancestor / descendant. The federated executor handles LineageEmit locally (no remote dispatch needed; the walk already happened at plan time).

max_depth = 0 is correctly handled as "just-the-origin", not as a bound violation. Both walks previously surfaced LineageMaxDepthExceeded whenever the start origin had any unvisited neighbour, even when the caller explicitly asked for zero steps.

Cross-chain joins

Inner hash-join on row-intrinsic keys

OperatorPlan::HashJoin { left, right, key_mode, kind, strategy, watermark } with JoinKeyMode::{Origin, Seq, OriginSeq} for the row-intrinsic join-key extraction modes. Both local and federated executors implement build-on-left / probe-on-right; the federated path recurses through itself so atomic leaves still dispatch via the transport. Joined rows are sentinel ResultRows (origin = 0, seq = 0) whose payload is a postcard-encoded JoinedRowPayload { left, right }. MeshError::JoinMemoryExceeded surfaces at the 256-MiB build-side bound.

Outer joins + sort-merge + payload-keyed

All four JoinKinds ship: Inner / LeftOuter / RightOuter / FullOuter. JoinKeyMode::Field(String) extends the join-key surface to JSON payload paths via row::extract_string_projection; try_encode_join_key returns Option<Vec<u8>> so rows whose key field can't be resolved are silently dropped from both sides. JoinStrategy::{HashBroadcast, SortMerge} lets the planner pick between in-memory hashing (default; trips JoinMemoryExceeded past the bound) and sort-merge (sort both sides + two-pointer walk; memory-bounded by the inputs).

The three-way duplicated hash-join body (local one-sided + local full-outer + federated mirror) factored into a shared build_hash_join_table(rows, key_mode, strategy_label) -> Result<HashJoinTable, MeshError> helper. try_encode_join_key canonicalizes JoinKeyMode::Field("origin"|"seq"|"origin,seq") to the matching row-intrinsic encoding so probe tables built under Origin and Field("origin") cross-correlate.

Watermark is informational under snapshot semantics; streaming activation needs a future windowed-join slice. The default is 5 s.

Filter, aggregates, and tumbling windows

Count

OperatorPlan::AggregateCount { input, group_by } over row-intrinsic group keys (Origin, Seq, OriginSeq). Sentinel ResultRow per group with a postcard-encoded AggregateRowPayload { group, value: Count(u64) }.

Filter

Reuses the Capability System's PredicateWire. Every ResultRow projects to a synthetic (Vec<Tag>, BTreeMap) view via row::synthetic_row_view — dataforts.origin, dataforts.seq, plus flat JSON-object payload fields. Non-JSON payloads are opaque; predicates against missing fields simply don't match.

The FFI's JSON predicate parser bounds caller-supplied recursion at 64 deep (PREDICATE_PARSE_MAX_DEPTH); the substrate's Predicate::to_wire converts from recursion to a heap-allocated work stack so 10k+-deep typed predicates from Python / Node factories don't overflow the Rust thread stack on every execute.

Sum / Avg

OperatorPlan::AggregateNumeric { input, group_by, field_path, kind: Sum | Avg } over row::extract_numeric (JSON path → f64). Rows whose field fails to resolve are skipped; Avg(None) covers the empty-group case.

Min / Max / DistinctCountExact / PercentileExact

OperatorPlan::AggregateReduction { kind: Min | Max | Percentile { p } } over f64::total_cmp (so NaN ordering is well-defined) + OperatorPlan::AggregateDistinct { field_path } (canonical-string projection into a per-group BTreeSet). Nearest-rank percentile. The HLL p=14 / T-Digest c=100 sketch variants (DistinctCountHll, PercentileTDigest) remain PlannerError until a consumer drives the algorithmic complexity; the exact variants are the recommended path today.

Tumbling-on-seq windows

QueryV1::Window { inner, spec: WindowSpec::TumblingSeq { size } } buckets rows into fixed-size half-open intervals on seq; the executor emits one sentinel ResultRow per non-empty bucket with a postcard-encoded WindowBoundary { start, end, rows }. Sliding + session windows extend cleanly via additional WindowSpec variants when a consumer drives the shape.

Result cache

CachePolicy + ExecuteOptions

pub enum CachePolicy {
    Permanent,                   // hold until LRU eviction
    TimeBound { ttl: Duration }, // TTL-bounded; default 5 s
}

pub struct ExecuteOptions {
    pub bypass_cache: bool,             // skip both lookup AND writeback
    pub cache_policy: CachePolicy,
}

TimeBound { ttl: 5s } is the default policy (mirroring the join watermark). Permanent is the explicit-opt-in for queries over closed substrate ranges (At, bounded Between with end ≤ current_tip). bypass_cache skips both lookup and writeback (Deck operator-view authoritative reads; Hermes skill-routing under churn; diagnostics).

Global cache version, pull-based invalidation

CapabilityIndex carries an AtomicU64 mutation_version that bumps on every index / remove / gc mutation. The MeshDB cache key encodes the live version into CacheKey { plan_hash: u64, capability_version: u64 }; any divergence misses. Aggressive invalidation by design — softening it is not the answer to churn, the bypass_cache flag and the Permanent policy together cover the cases where staleness is preferable.

CacheKey::for_plan is encode-failure-safe

impl CacheKey {
    pub fn for_plan(plan: &ExecutionPlan, capability_version: u64) -> Option<Self>;
}

Returns None when the plan can't be postcard-encoded (currently: any plan variant carrying a PredicateWire, because PredicateNodeWire uses #[serde(tag = "kind")] which postcard rejects on decode). Cache call sites treat None as a transparent bypass rather than a panic — defence-in-depth against future plan variants that become un-encodable.

Hand-rolled LRU

HashMap<CacheKey, Node> + intrusive doubly-linked list over a Vec<Node>. Defaults: LRU_MAX_ENTRIES = 1024, LRU_MAX_BYTES = 256 MiB; either bound trips eviction of the LRU end. DefaultHasher over postcard-encoded plan bytes; no new external dependency.

insert of an oversized result (approx_bytes() > max_bytes) refuses up-front instead of inserting at head and immediately evicting itself from the tail. Pre-fix, a Permanent-policy cache call for an oversized result silently re-ran the plan on every subsequent execute; post-fix the no-op insert leaves the cache entry-count + byte-count untouched and the prior entry at the same key (if any) survives.

Top-level only — sub-plan executes inside the federated path bypass the cache. Recursive caching at HashJoin sides / Aggregate inner is a follow-up if profiling justifies the bookkeeping.

SDK shims — Python / Node / Go / C

Every binding ships the full operator surface in lockstep: atomic factories (at / between / latest), composite factories (window / count / numeric_agg / percentile / join / filter / lineage_emit), the typed Predicate builder, the fluent QueryBuilder, the cache options, and a sentinel-envelope decoder that turns aggregate / joined / window result rows into host-language objects. The substrate's MeshError reflects through every shim with a structured kind discriminator.

Python — pyo3 + maturin

MeshQuery / MeshQueryRunner / ResultRow / Predicate / QueryBuilder ship as #[pyclass] types in the _net extension module, re-exported from the net Python package behind the dataforts / meshdb extras. The sync MeshQueryRunner.execute(query, options) returns list[ResultRow]; aggregate / joined / window payloads decode via ResultRow.decode_aggregate() / decode_joined() / decode_window().

MeshDbError carries a structured kind attribute set via PyO3 setattr on the raised instance — callers branch on except MeshDbError as e: if e.kind == "join_memory_exceeded": ....

Node — napi-rs

MeshQuery / MeshQueryRunner / MeshQueryStream / ResultRow / Predicate ship through napi-rs 3.9. runner.execute(query, options) returns a Promise<MeshQueryStream>; the TS shim at bindings/node/meshdb.ts attaches Symbol.asyncIterator so for await (const row of stream) works.

The AsyncIterable shim defines return() and throw() hooks that call MeshQueryStream::release() on a break / exception unwind, freeing the backing Vec<ResultRow> immediately rather than pinning it on the AsyncMutex until JS GC fires.

Node errors embed the kind discriminator in the reason string via a <<meshdb-kind:KIND>>MSG prefix; the SDK ships parseMeshDbErrorKind(err) -> { kind, message } | null to decode it.

Go — cgo + reference SDK contract

net-meshdb-ffi is a cdylib exporting the C ABI (net_meshdb_* symbols); the Go-side reference contract at bindings/go/net/meshdb.go wraps it in a cgo-importing package with MeshDBReader / MeshDBQuery / MeshDBRunner / MeshDBQueryStream / MeshDBPredicate types. Execute returns a <-chan MeshDBResult; the fluent MeshDBQueryBuilder chains source / filter / aggregate / window / join steps.

Hardening closed for the Go SDK and the underlying FFI cdylib:

• Safe size_t → int payload conversion via unsafe.Slice + bytes.Clone — refuses payloads above math.MaxInt with ErrMeshDBRuntime rather than letting C.GoBytes's C.int cast silently truncate.
• ExecuteContext / ExecuteWithContext run the FFI execute call inside the spawned goroutine; the caller is never blocked on cgo, and ctx.Done() races the executor concurrently with row pumping.
• An ffi_guard! macro wraps every FFI entry point in catch_unwind; panics across the C ABI become null_mut() returns with kind runtime_panic populated on the thread-local last-error pair.
• Every factory validation null-return populates net_meshdb_last_error_message / _kind with a descriptive invalid_arg message; Go-side wrapMeshDBError(sentinel) reads both into a MeshDBError that wraps ErrMeshDBInvalidArg / ErrMeshDBRuntime for errors.Is routing.
• MeshDBQueryBuilder source-resets (.At / .Between / .Latest) preserve the accumulated b.err so Build still surfaces the first error in the chain; deterministically free the prior *MeshDBQuery handle in place; aliasing semantics documented explicitly.

C — libnet_meshdb cdylib + net_meshdb.h

The C header at include/net_meshdb.h documents every entry point: opaque handles (MeshDbReader / MeshDbQuery / MeshDbRunner / MeshDbIter), atomic + composite factories, runner + execute, the sentinel-envelope decoder, and the per-thread last-error trio (net_meshdb_last_error_message / _kind / _clear_last_error). A runnable example at examples/meshdb.c walks the canonical lifecycle — reader populate → atomic / composite / lineage query → execute → drain — plus a fourth section exercising the cached runner under NET_MESHDB_CACHE_PERMANENT.

runner_new / runner_new_cached / runner_execute / runner_execute_with take their borrowed handles by const T* for C++ const-correctness; Rust FFI signatures match (*const T).

Hardening — MeshDB code review

Two coordinated review passes landed before the v0.16 branch cut. The first pass surfaced 32 items (6 Blockers, 10 Majors, 12 Minors, 4 Nits); the second pass verified those closures and surfaced 20 new items (3 Blockers, 9 Majors, 8 Minors). Every Blocker and Major closed in-tree with regression tests; two Majors and four Minors deferred with rationale (the deferred items need SDK surfaces — FederatedMeshQueryExecutor exposure, configurable budgets, Discovered resolution — that ship with future slices).

Blockers (9, all closed)

• CacheKey::for_plan now returns Option<CacheKey>. Defence-in-depth against future un-encodable plans; pinned with a regression test verifying current Filter plans still encode.
• Federated handle.cancel() no longer no-ops on composite-operator output streams. The outer handle is threaded through every recursive sub-fetch and the materialized output wraps in a cancel-aware adapter.
• Go FFI reader / runner lifetime contract documented. Snapshot-then-free vs keep-alive, never free-then-append.
• Every Go FFI execute path traps panics via catch_unwind. The structured MeshError (display + kind) flows through a thread-local LAST_ERROR_* and three getters.
• Go SDK ExecuteContext / ExecuteWithContext take context.Context. Pumping goroutine selects on ctx.Done() per send. Drop-the-channel-to-cancel was a documented lie.
• MeshDBQueryBuilder source-resets free the prior *MeshDBQuery handle deterministically.
• Go SDK pumpIterRowsContext no longer truncates size_t payloads to C.int. unsafe.Slice + bytes.Clone + a math.MaxInt guard surfaces ErrMeshDBRuntime on oversized payloads rather than letting C.GoBytes silently sign-flip.
• ExecuteContext runs the FFI execute inside the spawned goroutine. Pre-fix it ran on the caller's stack before the pump goroutine spawned, so ctx.Done() was ignored until the executor returned.
• Every FFI entry point (not just the two runner_execute* paths) wraps in catch_unwind via a new ffi_guard!($default, { ... }) macro. Panics become null_mut() / NET_MESHDB_RUNTIME_ERR with kind runtime_panic populated.

Majors (19 — 13 closed in code, 6 deferred with rationale)

Closed:

• Planner non-determinism via HashSet<Tag> iteration. parent_of / children_of / collect_coverage collect every candidate, sort, and pick the smallest with a deterministic tie-break key.
• Discovered resolution surfaces MeshError::AmbiguousDiscovery { matches } when multiple origins match, rather than silently truncating to the first.
• call_id uniqueness — process-global FEDERATED_CALL_ID_COUNTER replaces the per-executor counter.
• AST drift across FFI shims — "origin,seq" canonicalized as the single accepted join-key separator across Python / Node / Go.
• Structured error kind discriminator on MeshError; surfaced through every binding.
• Node cache-policy factory validation brought to parity with Python / Go (reject non-finite / negative ttlSeconds at construction).
• Watermark API parity on Python's MeshQuery.join(...) (already shipped; pinned with a regression test).
• BFS in lineage walks uses VecDeque::pop_front and caches children_of.
• Go SDK wraps every non-OK FFI return with MeshDBError { Sentinel, Kind, Message } that reads the thread-local last-error pair.
• Lineage walks accept max_depth = 0 as "just-the-origin"; previously a present parent / child tripped LineageMaxDepthExceeded.
• parent_of collects across all replica hosts before picking the lex-smallest parent. Pre-fix the outer DashMap iteration short-circuited on the first hosting node, drifting the plan + cache key across runs.
• LruResultCache::insert of an oversized result refuses up-front instead of silently evicting itself.
• JSON predicate parsing bounds depth at 64; Predicate::to_wire converts to an iterative heap-allocated work stack.
• Every Go FFI factory's validation null-return populates last_error_* with a descriptive invalid_arg message.
• Node AsyncIterable shim defines return() / throw() that release the backing Vec<ResultRow> via a new MeshQueryStream::release() napi method.
• include/README.md error-reporting paragraph rewritten to match the actual net_meshdb_last_error_* contract; operator-families table gains the last-error row; quickstart migrated to <inttypes.h> PRIx64 / PRIu64.
• MeshDBQueryBuilder source-resets preserve b.err; aliasing across source-resets documented explicitly.

Deferred with rationale:

• Federated SDK tests. Need FederatedMeshQueryExecutor + LoopbackTransport exposed through the SDK shims; ships with a future federated-surface slice. Substrate-side coverage is solid in the meantime.
• Runner-side error-path coverage in SDKs. The runtime MeshError variants the review listed (JoinMemoryExceeded, QueryBudgetExceeded, AmbiguousDiscovery, HistoricalRangeUnavailable) aren't currently triggerable from the SDK surfaces — they need configurable per-query budgets, ChainRef::Discovered exposure, and capability-index gating, none of which ship in v0.16. The kind discriminator plumbing is pinned with a Node-side parseMeshDbErrorKind test against synthetic errors.

Minors (20) and Nits (4)

Closed:

• group_key_for defensive fallback for JoinKeyMode::Field replaced with unreachable!() and a descriptive message.
• row_overhead: u64 = 64 magic constant replaced with std::mem::size_of::<ResultRow>() as u64.
• translate_responses emits MeshError::ExecutorError on premature transport stream termination instead of treating it as clean EOS.
• The three-way duplicated hash-join body factored into the shared build_hash_join_table helper.
• C header threading section documents move-safe / not-Sync semantics for MeshDbRunner and MeshDbIter.
• meshdb.ts drops the typed-class re-export (the shim's job is just the AsyncIterable side-effect).
• Shared OnceLock<Runtime> per FFI shim instead of Runtime::new() per runner.
• MESHDB_PLAN.md and CORTEX_ADAPTER_PLAN.md reconciled with shipped reality.
• JoinKeyMode::Field("origin"|"seq"|"origin,seq") canonicalizes to the matching row-intrinsic encoding.
• parseMeshDbErrorKind regex accepts [a-z0-9_]+ for future numeric-suffixed kinds.
• C header const-correctness on runner_new / runner_execute / runner_execute_with.
• C example exercises the cached runner.
• examples/meshdb.c uses <inttypes.h> PRIx64 / PRIu64.
• Python lineage_emit doc-comment attached to the correct factory.
• Go FFI ffi_cached_runner_round_trips actually asserts a cache hit (mutates the underlying store between calls and verifies the Permanent-policy fetch returns pre-mutation bytes).
• translate_responses last-err rebuild uses the original error rather than re-constructing.
• Node LineageEntry.depth is bigint (shape parity with originHash / tipSeq). The factory rejects values exceeding u32::MAX with a typed error. Breaking for any Node SDK caller that previously constructed entries with plain number literals: pass 0n, 1n, … instead of 0, 1, ….

Closed (post-pass):

• MeshDbRunner.executor: Arc<LocalMeshQueryExecutor> indirection dropped across all three shims — the runner owns the executor directly, the FFI / NAPI / pyo3 entry points borrow it for the lifetime of the call.
• Substrate-side join-watermark clamp helper lands as clamp_join_watermark_secs(secs: Option<f64>) -> Duration in behavior::meshdb::query, alongside DEFAULT_JOIN_WATERMARK_SECS = 5. All three SDK shims now route their f64 watermark input through the helper, and four substrate-level unit tests pin the contract (None / NaN / +/-inf / negative → 5 s; finite non-negative → passes through). Closes the deferred concern that the Python test_join_accepts_watermark_secs_kwarg could only assert row count, not the clamp choice.
• Substrate test-gap fillers for the items the SDK suites couldn't reach cleanly: Unicode payload values (CJK / combining marks / emoji-ZWJ) under Filter; singleton-input percentile + avg aggregates across the full p ∈ [0, 1] range; empty-input group_by = origin aggregates that must not fabricate buckets; long-linear lineage walks (N = 500) backward and wide-fanout lineage walks (N = 1000) forward without stack overflow.

Substrate-side hardening (alongside the MeshDB passes)

• Routed-handshake replay guard now tracks the initiator's Noise ephemeral. Pre-fix, the guard refused any same-static re-handshake — indistinguishable from a passive attacker replaying captured msg1 bytes. The connect_direct(peer, via = X) retarget path (connect_direct_retargets_coordinator_does_not_short_circuit_on_stale_session) failed with a handshake-timeout against an existing session. Post-fix, routed_rotation_outcome only DropReplays when BOTH the static AND the initiator's ephemeral match.
• Duration::MAX sentinel handled in periodic sweep loops. spawn_token_sweep_loop and spawn_capability_gc_loop both documented Duration::MAX as "disable the loop". The implementations forwarded that value to tokio::time::interval(MAX), which panics on Instant + MAX overflow. Both loops now short-circuit to shutdown_notify.notified().await when the interval is MAX.