Architecture · August 18, 2026

Relay Mesh Experiment

The Relay Mesh Experiment is a policy on the path SolanaCDN already runs: Turbine into a Point of Presence, a QUIC mesh between POPs, Path A onto a validator’s UDP TVU, Path B onto a QUIC subscriber. It is not a second mesh. It is not a new shred format. It does not fork Agave. It is a hop budget and an assignment rule.

The paid mainnet fleet already runs a protocol-13 binary with that policy compiled in and switched off. [relay_mesh] enabled = false is the live default. Destinations, packets, and spanning-tree fanout match the previous binary. Enabling is a TOML flip on those units together, not a new process.

What RME is

A paying destination does not run a capture agent. Each POP joins mainnet-beta like an unstaked node — gossip, Turbine, repair — and writes raw_solana_udp at the registered TVU socket. Stock shred_fetch_stage treats those datagrams like Turbine: signatures checked, duplicates ignored. Native Turbine, gossip, and repair keep running on the validator. Extra datagrams on the same socket. Whichever source fills the gap first wins on the host; SolanaCDN does not tap the kernel to score that race.

Between POPs the fleet already has a mesh: QUIC sessions, a spanning tree, shred-batch replication so a destination served from Frankfurt can still see a shred that Amsterdam ingested. RME does not replace that mesh. When the flag is off, the tree is the path. When the flag is on, ingest stamps hop = 0 and unicasts unique merkle shreds to live peers in the configured relay set. A receiver does not re-relay. Path A and Path B sockets do not change. Votes stay on the vote path.

Today’s path

Flag off is not a stub. It is the production feed. Ingest lands in one cache. The mesh floods the tree — parent and children, flood if the tree is empty. Each POP writes Path A for the public UDP destinations the control plane reconciled onto it, and Path B for QUIC subscribers that dialed in. POP does not echo a published batch back to the publishing session.

That tree is how completeness travels. It is also how a shred that only needed to reach one serving POP still crosses every hop the spanning tree has. RME’s bet is that most destinations want one well-chosen POP to write TVU, and that replicating every unique shred across the whole tree is the wrong cost function once ingest is healthy in more than one facility group.

Why the tree is the wrong default for some dests

August 12 was the assignment lesson, not a mesh-protocol lesson. Terraswitch’s interconnect dropped a default across twelve EU and APAC sites. Solana kept producing. Validators behind that IGP did not. A feed that collects Turbine off a partitioned interconnect and writes it into local TVU is how a node stays on the cluster — we wrote that up as The interconnect failed. Solana didn’t.

The follow-on is assignment. If the serving POP shares fate with the destination’s hosting ASN, the overlay is not a second path. The paid fleet does not originate AS20326. fra-1, ams-1, and sgp-1 sit in an operator overlay we call terraswitch_eu_apac because dests on that interconnect share fate with those metros. lax-2 is in latitude_lax. A destination in Amsterdam on AS20326 should not get fra-1 as primary just because the metro string matches. It should get a POP that does not share that interconnect.

Flag on: unicast, then stop

When enabled = true, ingest still stamps hop = 0. Unique shreds unicast to live mesh sessions whose peer_id is in the configured relay set. Phase 1 is that set intersected with live sessions — not yet a per-destination unicast to only the serving POP. A receiver with the flag on does not re-forward, whatever hop it saw. Local cache and Path A/B still run. Empty peer set fail-softs back to the tree so a lonely ingest POP does not black-hole the feed.

The live knob is small on purpose:

[relay_mesh]
enabled = false
# relay_set = paid fleet pop ids
max_relay_set_size = 8
same_asn_as_dest_forbidden_for_primary = true

Absent [relay_mesh] is disabled. The kill switch is that boolean, or a revert to the previous binary. Stopping solanacdn-pop is not the off path. That unit is the paid mainnet feed.

Hop, TTL, protocol 13

PopToPop::ShredBatch gained hop: u8, serde default 0. Protocol version is 13; previous is 12. A v12 peer that still speaks the old struct maps the field to 0. That is decode compatibility, not a rolling-upgrade path.

We learned that the hard way. Staging protocol 13 on one POP against remaining v12 peers took mesh_sessions to zero with connection-lost on the rest of the mesh. Reverting that one binary restored the mesh in seconds. Cutting the paid fleet together recovered immediately. Mixed-version hello is not safe in this fleet. Enable, when we enable, is the same constraint: every paid unit, same flag, watch mesh sessions before declaring victory.

• Histogram hop 0: ingest POP is also egress POP. The shred did not need the mesh.
• Histogram hop 1: one mesh unicast, then Path A/B. TTL is 1, so that is the only legal hop under the flag.
• Flag off still puts hop = 0 on the wire and keeps tree re-forward.

Who serves a destination

Assignment is table-driven. No BGP, no live RTT in phase 1. Inventory is metro, origin ASN, facility group, IPv4. Primary prefers dest metro unless that POP is forbidden: same origin ASN as the dest when same_asn_as_dest_forbidden_for_primary is set, or facility group terraswitch_eu_apac when dest ASN is 20326. Backup must differ in both ASN and facility group when possible. Existing customers stay sticky unless primary is down; backup is reassignment, not dual-send.

Origin ASNs on the paid fleet, confirmed from prefix data, not from the hosting brochure: fra-1, sgp-1, and lax-2 originate AS396356; ams-1 originates AS262287. None of them originate AS20326. The facility group is an operator overlay for the dest-ASN lesson, not a claim that those POPs sit on Terraswitch’s ASN.

The control plane copy of assign_relays is on a sibling branch. Live assign() is still region and capacity until dest ASN is filled in — the lookup returns none today, on purpose, rather than guess from whois. Path A keeps working from assigned_pop_id. The table is what we will use for new dests once that column exists. Sticky customers do not move for a blog post.

Dedup, votes, fail-soft

The shred id on this path is (slot, fec_set_index, index, is_code). Slot-window eviction is 512. A duplicate including FEC set sends once. Vote and gossip bytes are not merkle shreds; the RME shred path drops them. VoteDatagramForward is untouched. Gossip ingest, FEC, TPU, trade, and billing are untouched.

Ingest health is a gate on who may be in the active relay set: latest slot nonzero, slot age under two seconds, injector not in a death spiral. Configured set ∩ registered mesh peers ∩ ingest-healthy. If that intersection is empty at unicast time, fanout falls back to the tree.

What we can measure

There is a delay clock. There is not a race winner. We did not ship a kernel tap on the validator that can say Turbine beat the overlay or the reverse. solanacdn_rme_race_wins_total labels winner=unknown. That is the fact, not a scoreboard. Flag on does not dual-send the same shred down two overlay paths to the dest. It unicasts, then the serving POP writes Path A/B.

Compare flag-off vs flag-on on egress_latency first. After enable, add the RME-labeled delay and the hop histogram. Do not page up on the RME series. Doctor prints rme enabled=false until the control plane echoes the optional status blob; the dest-port UDP probe still timestamps first shred on the validator host as winner=unknown.

Rolling the fleet

The paid unit is solanacdn-pop.service / /opt/solanacdn/bin/solanacdn-pop. Features match live: xdp,gossip,observation,repair,snapshot-mirror. Built without target-cpu=native so one binary runs on the mixed EPYC SKUs. Protocol 13 landed with the flag false. Gate after cutover: up=1, mesh sessions intact, solanacdn_rme_enabled 0 on the paid fleet.

A Rotor-shaped overlay

Alpenglow is two phases. Votor replaces TowerBFT. Rotor, later, is meant to replace Turbine. The Foundation’s write-up is the one to use: Rotor changes the block propagation plan from a tree of nodes to a single relay layer. Same erasure coding idea, same stake-adjusted bandwidth. The tree goes away. The leader sends shreds to a small relay set; those relays cover the rest of the cluster in one hop. That SIMD is not this binary. RME does not vote, does not sample stake, and does not become Rotor by renaming the flag.

It is the same hop budget, on a different plane. Turbine is a deep stake-weighted tree. Rotor collapses that tree to one relay hop inside the protocol. RME collapses the CDN mesh the same way: ingest unicasts to a small live relay set, receivers do not re-forward, Path A writes TVU. If you need a Rotor-shaped path onto a stock Agave socket while the cluster still speaks Turbine — and while Rotor is still the later phase — this is how the overlay does that job without waiting on the SIMD.

Where it can stand in for Rotor is the operators Rotor’s sample will still underserve. Rotor relays are stake-weighted. Stake sits in facilities. August 12 was a hosting IGP, not a missing relay SIMD: a stake-sampled layer can still share fate with the dest. RME’s assignment rule is the part Rotor does not have — do not pick a primary in the same ASN or in terraswitch_eu_apac for an AS20326 dest. The overlay is how a validator behind that interconnect gets a single-hop shred path that is not the interconnect, not the Turbine tree, and not whoever Rotor sampled this epoch.

The other fit is the migration window. Votor ships first. Rotor waits. Paid mainnet-beta still ingests Turbine. The hop-1 policy does not care whether those bytes came from a Turbine leaf or, later, from a Rotor relay the POP can hear. Change the ingest source; keep unicast, TTL=1, and Path A. Leader extra-dest into a POP is the next cut that makes the overlay a Rotor relay in the cluster sense: shreds leave the shredder once, hit the relay set, stop.

The flag is the first hop budget, not the ceiling. Keep engineering on this path and the overlay is a Rotor layer for the Solana cluster: grow the relay set with the fleet, take leader extra-dest so shreds enter at hop 0, weight fanout the way Rotor weights bandwidth, ingest Rotor when that phase lands. Paying dests already get Path A. The rest of stake is the same unicast once the fleet is the relay layer, not only a customer overlay. The Alpenglow community cluster on these hosts is a sibling process. This work is the paid mainnet unit, pointed at that job.

Engineering it into an active Rotor layer

Rotor is a single relay hop with stake-adjusted bandwidth and the same erasure coding Turbine already uses. The flag already has the hop budget: hop = 0 at ingest, TTL=1, no re-forward, shred id (slot, fec_set_index, index, is_code). Becoming an active Rotor layer for the cluster is filling the rest of that shape — origin, who receives the unicast, how bytes are weighted, what the dest map is — without inventing a second shred format.

Close origin first. Today hop=0 is whatever Turbine and repair the POP heard. Rotor’s first hop is the shredder to its relays. Leader extra-dest — the origin path already documented for Jito / Frankendancer / Harmonic — puts a second copy of those shreds on a POP socket at the shredder. Dedup drops the Turbine twin. The overlay then looks like Rotor from the leader’s side: one send, relay set, stop. Stamp source=leader_publish in telemetry; bytes stay merkle shreds.

Then stop treating the live relay set as the dest. Phase 1 unicasts to relay_set ∩ live mesh sessions. Rotor gives each relay a subset of validators. The next fanout mode is Unicast to the serving pop for dests that need that shred, not to every healthy peer. That needs the control-plane dest ASN column live so assign_relays is not region-and-capacity with dest_asn = None. Sticky customers stay; new dests get metro plus the facility-group veto.

Rotor’s other number is bandwidth ∝ stake. Equal unicast to every peer is not that. Weight solanacdn_rme_fanout_bytes_total and Path A send by dest stake, and by served stake on the POP. Cap a relay the same way Rotor caps a validator: more stake, more outbound. Fail-soft stays: empty serving set falls back to the tree so a lonely ingest POP does not black-hole a slot.

Erasure stays Reed-Solomon. Rotor’s simplification is one erasure-coded packet per shred instead of Turbine’s data-versus-coding forward games. The shred path already drops non-merkle bytes. Serving pop FEC-completes if the dest needs a reconstructed set; mesh does not re-forward either half. When the Rotor SIMD lands, change ingest — gossip TVU today, Rotor relay or extra-dest tomorrow — and leave hop, TTL, and Path A raw_solana_udp on stock shred_fetch_stage through the mixed epoch.

An active Rotor layer covers stake, not only invoices. Path A reconcile stays the paid, verified TVU sockets. Cluster fanout uses the same unicast onto ContactInfo UDP TVU from gossip: leader schedule plus TVU, not QUIC, not TPU. Paid dests keep heartbeat origin-IP. The rest of the set is best-effort, the way Turbine already is. Next-leader-first on that map is the Rotor ordering; sent_at_ms is already on the batch. Push t_rme_first back toward ingest so a miss is a targeted retry, not a Turbine retransmit wait.

Admission is already almost a relay ticket. ingest_healthy is latest slot nonzero, slot age under two seconds, injector not in a death spiral. active_relay_set is configured ∩ mesh ∩ that gate. Tighten it: require ingest healthy in more than one facility group before unicast is legal, and treat solanacdn_rme_ingest_pops_up as the live Rotor set. A POP that fails the gate is not a relay that epoch. Same as Rotor not sampling a mute validator.

What this is not

• This ship is not the Rotor SIMD. It is the hop budget you keep engineering toward that relay layer.
• Not a second shred protocol and not a consensus change.
• Not dual-send to primary and backup. Backup is control-plane reassignment when primary is down.
• Leader extra-dest is the next origin cut, not in the flag-off binary.
• Not live BGP. Dest ASN is a column we will fill; it is not a looking glass.
• Not a kernel race on the validator NIC. First-shred on TVU vs native Turbine remains the operator’s own measurement.

The product for a validator is still the same: keep stock Agave, register UDP TVU, heartbeat from that host, take extra shreds on the socket you already have. RME is how the fleet moves those shreds between POPs once we turn the flag on — one hop, then Path A.