Multipath streams

Status: implemented for reliable ordered Stream paths. The version-1 wire format remains experimental. Acknowledgements, retransmission, and UDP adapters are not implemented.

TeeForge.Networking.MultipathSenderStream distributes a logical byte sequence across application-supplied paths. MultipathReceiverStream reorders, deduplicates, or reconstructs groups and exposes sequential reads. Both derive from Stream, are non-seekable, and support only their named direction. Compose two sender/receiver pairs for full duplex.

Paths can be NamedQuicStream instances or other reliable ordered streams. The application opens and authenticates transports. The core does not open sockets, resolve endpoints, authenticate peers, or implement Ethernet PRP or IEC 62439-3. Paths sharing an underlying connection may share a failure domain; path count alone does not establish independence.

Working example

This .NET 10 example uses two in-process pipes as connected transports. Sending and receiving run concurrently so bounded queues and transport backpressure can make progress. In a network application, these operations run at their respective endpoints.

using System.IO.Pipelines;
using TeeForge.Networking;

using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(30));
CancellationToken token = timeout.Token;
var options = new MultipathStreamOptions(
    mode: MultipathStreamMode.Raid1,
    framePayloadSize: 16 * 1024,
    pathAvailabilityTimeout: TimeSpan.FromSeconds(5));

await using var sender = new MultipathSenderStream(options);
await using var receiver = new MultipathReceiverStream(sender.SessionId, options);
for (int index = 0; index < 2; index++)
{
    var pipe = new Pipe();
    Task<Guid> joinSender = sender.AddPathAsync(pipe.Writer.AsStream(), token).AsTask();
    Task<Guid> joinReceiver = receiver.AddPathAsync(pipe.Reader.AsStream(), token).AsTask();
    await Task.WhenAll(joinSender, joinReceiver);
    // The receiver reads the same path ID that the sender generated.
}

byte[] payload = new byte[1024 * 1024];
Random.Shared.NextBytes(payload);
using var source = new MemoryStream(payload);
using var destination = new MemoryStream();
Task receive = receiver.CopyToAsync(destination, token);
Task send = SendAsync();
await Task.WhenAll(send, receive);
Console.WriteLine(payload.AsSpan().SequenceEqual(destination.ToArray()));

async Task SendAsync()
{
    await source.CopyToAsync(sender, token);
    await sender.CompleteAsync(token);
}

Run initializers concurrently at the two endpoints if their greetings require bidirectional interaction. An initializer runs before the multipath hello. The sender writes and flushes the hello; the receiver reads and validates it.

Modes and local status

Write-call boundaries are not preserved. A group holds up to FramePayloadSize bytes in RAID 1 or RAID 0, or ErasureDataShardCount * FramePayloadSize bytes in erasure mode. Flush publishes a partial group. Erasure shards retain their configured size and pad unused space; logical length removes padding on receipt.

Erasure mode requires at least k + r locally active paths, where k and r are the data and parity counts. Each shard uses a distinct path. With fewer paths, subsequent groups use RAID 1 while desired mode remains erasure coding. Adding enough paths restores erasure coding. Existing groups retain their original metadata; new paths do not replay old groups.

sender.Status captures desired/effective mode, path count, shard counts, membership epoch, lifecycle state, and protection under one lock. Individual properties remain available, but separate reads may observe different moments.

Lifecycle states are Open, Completing, Completed, Faulted, and Disposed. Protection describes capacity for further publication, including pending data during completion. It does not confirm redundant delivery of a particular group. Underlying write success, path count, and path observations are not remote delivery acknowledgements.

Operation contract

Sender writes, flushes, mode changes, completion, and graceful removal serialize with publication. Receiver reads serialize with each other. Paths can be added while an ordinary data operation waits for connectivity. Synchronous calls block on the asynchronous implementation; prefer async calls for network paths.

Cancellation, outages, and errors

Cancelled or timed-out receiver reads leave frames available for the next read. The receiver availability timeout runs only while no paths exist: joining an idle path ends it, and losing the last path starts it. It is not a deadline for a stalled transport or an incomplete group. Use a cancellation token for an overall deadline. Synchronous calls use no cancellation token.

Sender operations without paths wait subject to cancellation and PathAvailabilityTimeout. An outage timeout is an IOException with a TimeoutException inner exception. The default is infinite. An unrecoverable gap can therefore wait indefinitely unless the application sets a timeout or cancels. New paths do not replay missing groups.

Cancelling a sender operation can leave a prefix buffered or published. There is no rollback or accepted-byte count. Cancellation during group publication faults the sender and removes interrupted write paths because a transport may contain a partial frame. Start a new session after that fault. Cancellation while merely waiting for an operation gate publishes no data. Do not blindly resend an entire buffer after any interrupted write. Failure after completion enters its publication phase faults the sender because EOF may be in flight.

Malformed frames fail their path; other valid copies or shards can still supply the group. Inconsistent group metadata, conflicting duplicate payloads, invalid completion sequences, and exceeded reorder limits fault the receiver. The current RAID-0 receiver policy is conservative: after observing any RAID-0 data, an unexpected path failure processed before logical EOF faults the session, even after a later mode change. In-band retirement avoids that policy. Transport EOF alone is not logical EOF.

Ownership

Ownership transfers only after AddPathAsync succeeds. After a failed or cancelled initializer/hello, the caller retains responsibility for a stream whose framing position may have changed. Do not retry a partial hello without an application-defined reset. A receiver can bind to its first path's session or require an expected session ID. Session IDs are not authentication.

After addition, the endpoint exclusively uses the supplied stream's bytes. With LeaveOpen = false, removal, detected failure, and disposal close owned paths; receiver pumps also close them at completion or retirement. With LeaveOpen = true, streams stay open, but cancellation cannot immediately stop a transport that ignores tokens. Coordinate transport shutdown before reusing those bytes. Receiver-local detach is not graceful sender-side retirement.

Options and resource bounds

MultipathStreamOptions remains shared for convenient paired construction. Sender settings do not constrain received metadata. Receiver limits are explicit and need not match the sender's initial mode.

The bounded receiver queue waits when full and never drops frames or failure events to make room. Each independent pump can stage one additional frame. Application reads resume pumps and propagate backpressure through transports.

The reorder byte budget reserves one payload per RAID-1/RAID-0 group. An erasure group reserves (k + r) * shardSize + logicalLength, including missing-shard reconstruction and decoded output, before retaining the group. Consumption releases the reservation. Exceeding the budget faults the receiver instead of waiting for space that an earlier missing group might itself need.

This is not a process-wide heap limit. Queue payloads occupy at most ReceiveQueueCapacity * MaximumReceiveFramePayloadSize. Each attached path additionally needs staging/parsing storage, temporarily up to two payload-sized arrays plus a header. Group metadata, codec matrices, and transport buffers are additional. The application controls the number of attached paths.

The sender evicts a path when its send queue has no free slot. It does not silently drop the only RAID-0 copy. A slow redundant path can finish after an operation returns or be evicted as later groups fill its queue.

Optional control channel

MultipathControlChannel frames messages over a separate reliable stream, readable, writable, or both. One send and one receive can run concurrently; same-direction calls serialize. Clean EOF returns null. Its leaveOpen argument controls ownership independently of data options. Finish or cancel control operations before disposal. Cancellation midway through a control frame does not reset framing; recreate the channel/transport.

Only these messages are implemented:

Accessors throw InvalidOperationException for the wrong kind. Payload data is exposed only through these checked accessors. CreatePathReceivingValidFrames constructs the path observation message, whose wire value is zero.

The channel does not automatically change modes, connect endpoints, or acknowledge delivery. The application authorizes a request and calls ChangeModeAsync; no acceptance handshake is required by the data plane. Non-erasure request shard counts are zero: omit those arguments when calling ChangeModeAsync to preserve its configured erasure geometry. Authenticate and authorize endpoint hints before connecting and calling AddPathAsync.

Version-1 wire format

A four-byte big-endian body length excludes its own prefix. Every body begins with four-byte magic, one-byte version, one-byte kind, and two zero reserved bytes. Integers and GUIDs use big-endian order. Data magic is 0x54464D50, control magic is 0x54464D43, and version is 1.

Data body offsets exclude the length prefix:

Modes encode as RAID 1 = 0, RAID 0 = 1, erasure = 2. Sequences start at zero. Membership and mode changes advance the epoch. Old groups remain decodable from their original metadata. XxHash64 covers payload bytes only and provides no authentication. Data frames carry no write-call boundary.

Control kinds are observation = 0, mode request = 1, endpoint advertisement = 2. After the common header, payloads are respectively: a 16-byte path GUID; four bytes containing mode, data count, parity count, and zero; or a one-byte UTF-8 scheme length, scheme bytes, two-byte big-endian data length, and opaque data. Scheme length is at most 255 bytes and endpoint data at most 65,535 bytes.

Deferred work and verification

Acknowledgements, selective missing-group reports, retransmission, automatic remote health tracking, mode acceptance/rejection messages, endpoint withdrawal, and UDP adapters are proposals. Reliable UDP additionally requires replay buffers, timers, flow/congestion control, and an explicit reliability contract. Framing and erasure coding alone cannot provide reliable datagram delivery.

Tests cover the three modes, path churn, reconstruction, fallback/restoration, mode boundaries, initializers, control round trips and checked payloads, timeout/cancellation recovery, bounded queue backpressure, queued failures, receive limits, memory reservation/release, disposal of pending reads, status transitions, concurrent completion, and interrupted publication. Randomized transport faults, partial-frame cancellation, and long-running multipath stress remain necessary before declaring the wire format stable.