Messaging

Max Size

A sensible upper-bound for message payloads is 1400 bytes:

1. Max Datagram (1472) - Aeron header (32) = 1440
2. Round-down to cache-line boundary: (1440 & ~63) = 1408
3. Subtract MPMC queue header (8): 1408 - 8 = 1400

UDP

The User Datagram Protocol (UDP) is a message-oriented transport layer protocol. UDP is the most common transport layer protocol used with multicast addressing.

UDP is a simple, stateless protocol that is best suited to applications that can tolerate:

• congested buffers and networks;
• duplicate messages;
• lost messages;
• messages received out-of-order.

For these reasons, UDP is often referred to as an unreliable protocol.

MTU

Although UDP supports datagrams of up to 64K in length, packing them into a single Maximum Transmission Unit (MTU) is highly desirable to avoid fragmentation and subsequent reassembly. By default, Linux will return EMSGSIZE if a user attempts to send a datagram that exceeds the target link's MTU.

UDP adds 8 bytes to the 20 byte header used by the IP protocol:

 0      7 8     15 16    23 24    31
+--------+--------+--------+--------+
|     Source      |   Destination   |
|      Port       |      Port       |
+--------+--------+--------+--------+
|                 |                 |
|     Length      |    Checksum     |
+--------+--------+--------+--------+
|
|          Data Octets ...
+---------------- ...

The maximum size of a datagram that avoids fragmentation is, therefore, the MTU size minus 28 bytes. This can be verified on a link with an MTU of 1500 bytes as follows:

$ ping -M do -c 1 -s 1472 www.reactivemarkets.com

Reliable UDP

UDP can be made more reliable by adding sequence numbers to the application protocol; receivers can implement a receive window to detect duplicates and restore datagram order.

Re-sending the last datagram at regular intervals may be used as a form of keep-alive and for detecting tail-loss on low volume channels.

When a sequence gap is detected, the receiver must use a suitable method for recovering lost transmissions. Recovery can be achieved by sending a Negative Acknowledgements (NAKs) back upstream to the sender. Care must be taken, however, to avoid re-request storms that further burden the network and risk compounding any capacity-related issues.

A solution often deployed by market-data feeds is to provide a separate recovery channel that periodically broadcasts snapshots for late-join or recovery purposes. Snapshot and delta channels use the same sequence numbers, so that they can be synchronised by downstream consumers. The main advantage of this approach is that events continue to flow in a single direction at a predictable rate.

TCP

Several characteristics of the Transmission Control Protocol (TCP) make it a more suitable for reliable messaging than UDP:

• Ordering TCP uses sequence numbers to ensure that data is received by the consuming application in the same order that it was sent. Receive windows are maintained by receivers for reassembling data in the correct order.
• Flow-control the receiver's remaining window-size is communicated back to the sender, so that the sender can limit the transmission rate to avoid overwhelming the consumer.
• Recovery sending applications will automatically retransmit data that has not been acknowledged by the consumer within a certain time period.

In addition, the connected nature of TCP streams and their APIs allow applications to perform specific actions when streams are connected and disconnected.

Disclaimer: these points are intended to highlight key reliability features only. (Consult your favourite TCP reference for more advanced features, including TCP's network congestion avoidance algorithms.)

In summary, UDP is ideal for messages that:

• fit within a single MTU;
• do not depend on their predecessors;
• may be received out-of-order;
• may occasionally be lost;
• use multicast addressing.

Good candidates for UDP are application heartbeats and snapshot-based market-data feeds.

TCP should generally be preferred otherwise.

Late-join

When a client subscribes to a stream, depending on the type of data, it may need to synchronise with the current "state of the world". This is normally achieved by replaying historical messages or by sending a snapshot.

A typical example would be a delta-based market-data feed, where clients must acquire the current state of the order-book, on which subsequent deltas can be applied.

Such problems are easily solved using a reliable, connection-oriented protocol like TCP:

• the client establishes a connection;
• the server sends an initial snapshot of the order-book;
• the server sends a sequence of deltas to apply.

Packing

Packed data-structures are important for message encodings where smaller sizes require less bandwidth, decrease transmit times, and help to avoid fragmentation when message-oriented protocols such as UDP are used.

This is often in contrast with design considerations for internal data-structures. Concurrent data-structures such as Events in the Disruptor pattern, for example, are often padded to avoid false sharing and to adhere with the single-writer principle.