This page is the second in a series of pages introducing the Multi-Gigabit Transceiver (MGT).
Introduction
Multi-Gigabit transceivers (MGTs) are the basic building block for many well-known protocols: PCIe, SATA, Gigabit Ethernet, SuperSpeed USB, Thunderbolt and Displayport. All of these protocols have one thing in common: There is a computer involved. There are also several protocols for telecommunications, intended for use to transport phone calls over a optic fiber link.
MGTs are also useful for exchanging data between two FPGAs. A few things to be aware of when using an MGT for this purpose are listed on a previous page. Clearly, there is a need for some kind of protocol to ensure that the data is transmitted properly and with an acceptable reliability over the physical channel. The implementation of such protocol is quite complicated, so the question is if there are ready-made protocols or other building blocks that can help.
This page is attempt to summarize the main alternatives. They are listed below according to the complexity level of the application logic, least complex listed first.
All protocols below can work as a bidirectional link (full duplex) as well as a one-way link (half duplex), unless stated otherwise.
Xillyp2p
Xillyp2p is a proprietary protocol that provides a reliable transport of multiple data streams between two FPGAs. The protocol manages the transmission, scheduling, retransmission and the flow control of the application data similar to the way that the TCP/IP protocol transports data over a network: All data is guaranteed to arrive correctly to the other side.
The application logic interacts with the protocol's implementation through standard FIFOs. The protocol creates an illusion of a standard FIFO than spans across the two FPGAs: This virtual FIFO's side for writing is on one FPGA, and the side for reading is on the other FPGA.
In each of the two FPGAs, one side of the FIFO is connected to the application logic and the other side interacts with the protocol's logic. Hence, the application logic on the transmitting side writes data into a FIFO and the application logic on the receiving side reads data from another FIFO. The protocol is responsible for moving the data from the FIFO on the transmitting FPGA to the FIFO on the receiving FPGA. The protocol's flow control ensures that the FIFO on the destination FPGA never gets full.
The protocol can serve multiple FIFOs in both directions. A fair data transmission scheduler ensures efficient utilization of the MGT's bandwidth. The data in the transmitter side's FIFO is consumed soon enough, so this FIFO won't become full (as long as the bandwidth allows this and the data is consumed from the FIFO on the other side).
The protocol also has a different interface for transmitting packets (with the help of an EOP port).
A half-duplex option is also supported. In this case, the protocol ensures that all data that arrives is correct. If a bit error occurs on the physical channel, the data flow is halted before faulty data reaches the application logic.
Gigabit Ethernet
Even though Gigabit Ethernet is intended for communication between computers, it's possible to use this simple protocol for sending packets between two FPGAs. A suitable IP is usually provided by the FPGA manufacturer. The application logic is hence responsible for creating and receiving Ethernet packets through one of the standard interfaces (GMII, RGMII, XGMII etc.).
As with any Ethernet link, it's the application logic's role to organize the data into packets, as well as handling errors on the link (e.g. with a retransmission).
Interlaken
Interlaken is an open protocol for communication of packets between chips. It is based upon 64b/67b encoding: The low-level data flow consists of repeatedly transmitting segments of 64 bits. Before each such segments, 3 bits are added in order to distinguish between application data and control words. The protocol is defined for a simplex link (unidirectional physical channel). If a full duplex link (bidirectional physical channel) is used, it works the same way as a simplex link, except that flow control messages may be sent in the opposite direction.
The protocol's basic transmission unit is a burst having a variable length. A control word is transmitted immediately before and after each such burst. These two control words contain information that allows an abstraction of packets and channels. This includes, among others:
- Channel number: A number between 0 and 255, associating the data burst that follows with a channel. It's possible to extend the number of channels up to 65536.
- SOP: A flag indicating that the data that follows is the beginning of a packet.
- EOP: A flag indicating that the packet before the control word was the last data of a packet. The EOP also indicates the number of valid bytes in the last word of the burst, and if the packet contains errors.
The length of each burst must not be longer than BurstMax and not shorter than BurstShort. Both BurstMax and BurstShort are parameters that are chosen for a specific application. BurstShort is at least 32 bytes (and must be a multiple of 8) and BurstMax must be a multiple of 64 bytes.
The content of the control word and the data burst that (possibly) came before it is checked for bit errors with a CRC24.
If a burst fails the CRC24 test (i.e. a bit error is detected), a retransmission can be requested as defined in the Interlaken Retransmit Extension Protocol Definition. According to the suggested mechanism, this request is sent through a set of three physical wires from the receiver to the transmitter. These wires have the names FC_CLK, FC_DATA and FC_SYNC, and are originally intended for out-of-band flow control (OOBFC). A simple serial data transmission protocol is defined for sending flow control requests. If retransmission is enabled, one of these bits is called "RT" and is used to request a retransmission. In other words, the protocol doesn't define a way to send the retransmission request on the MGT link itself, but rather on the three separate physical out-of-band wires.
The request for retransmission doesn't include information on which burst the retransmission should begin from. Instead, the transmitting side is required to save a defined amount of burst data in a buffer. When a retransmission request arrives, all bursts in the buffer are retransmitted. Hence the size of the buffer needs to be large enough to contain the lost burst. However if the buffer is too large, the retransmissions will be longer than necessary. In order to define the buffer's size, a careful calculation needs to be made regarding the round-trip time from delivering a burst to the logic that implements the Interlaken protocol until a retransmission request arrives.
The receiver detects retransmitted burst with the help of a counter that is incremented for each burst that is transmitted for the first time (it's also possible to increment the counter for each 2, 4, 8 etc. bursts). The value of this counter is transmitted in the Multiple-Use part of the control words that are transmitted before and after each burst. The receiver uses this counter to distinguish between a new burst and a retransmitted burst.
The Interlaken protocol also has a CRC32 check for diagnostic purposes (once for each Meta Frame). However, if an error is detected with the help of this test, it is related to large segment of data and not to a specific burst or packet. In other words, if data passed the CRC24 test despite a bit error, this will be detected only later, and without being able to point at the burst that contains faulty data.
It's possible to design and implement a retransmission mechanism that is based upon the MGT's data flow in the other direction. The Interlaken protocol doesn't suggest a method for such retransmission mechanism, but it's nevertheless possible to assign it a dedicated channel for packets requesting retransmissions. Such solution can have a significant advantage, as it can allow a more specific request of what to retransmit. It's also possible to apply a better CRC, possibly on packets rather than bursts. However, if this approach is taken, the entire mechanism must be implemented in application logic.
Interlaken's flow control mechanisms are discussed on a separate page.
Aurora
Aurora is a protocol developed by Xilinx (now AMD) for its own FPGAs. This protocol's basic transmission unit is a single word (having a fixed width depending on the number of MGTs involved). However, the protocol also supports transmission of packets (called frames). The application logic divides the data stream into packets with the help of a "last" input port.
There are two variants for Aurora: With 8b/10b encoding and with 64b/66b encoding. 64b/66b encoding is more efficient, so this variant should be preferred when possible.
Aurora is defined for a simplex link (unidirectional physical channel) as well as a full duplex link (bidirectional physical channel). There is no significant difference between the two options from the application logic's perspective, except that flow control is meaningless on a simplex link.
Note that unlike Interlaken, the Aurora protocol doesn't mention any channels. In other words, all data transmitted across the physical channel belongs to one data stream. When the word "channel" is used in the protocol, this refers to the physical channel, and not any division of the data into independent data streams. If such division is desired, it's the application logic's role to implement this, possibly by adding a header to each packet.
Bit errors on the physical channel are not corrected by the protocol. However, when the protocol is used to transmit packets, the transmitter optionally appends a CRC to the end of each packet (in Xilinx' implementation of the protocol). The protocol's implementation checks this CRC in the receiving side, and informs the application logic if an error was detected in the packet. If Aurora is used without packets (without a "last" signal), error detection is not supported by the protocol.
The protocol's own control information is transmitted without any protection against bit errors. There is no CRC on any such traffic. If there are bit errors on the physical link, the protocol may malfunction in a wide range of ways.
Any retransmission mechanism, multiple channel multiplexing and transmission scheduling must be implemented by the application logic. A separate page discusses the possibilities for implementing flow control with Aurora.
Serial Lite
Altera has a series of protocols and IP cores sharing the name Serial Lite:
- SerialLite II: Packet-based or non-packet interface for receiving and transmitting data (named "Atlantic Interface"). Supports retransmission in response to bit errors. Applicable to early FPGAs only (ranging from Arria II to series-V FPGAs).
- Serial Lite III: Supports Continuous and Burst mode. In Continuous mode, the data stream can flow uninterrupted and without gaps between the transmitter and receiver. This requires that both sides rely on the same reference clock. This protocol is internally based upon Interlaken, but without supporting channels or SOP/EOP. Accordingly, the interface's data width is 64 bits multiplied with the number of MGTs used. Bit errors on the physical link are reported as a diagnostic event with respect to the MGT that caused them, but not in relation to a specific data segment. Applicable to series-V FPGAs and series-10 FPGAs.
- Serial Lite IV: Based upon Avalon streaming interface with signals for start and end of packets (MAC). A CRC is optionally inserted at the end of packets by the transmitter and checked by the receiver. There is also a Basic mode, without division into packets. Applicable to Stratix 10 and Agilex E-tile.
The IP cores are not compatible across different members of this series of protocols. Only SerialLite II may initiate retransmissions.
RapidIO
RapidIO is a packet-based protocol that is similar to PCIe in the sense that the packet types it supports correspond to operations required by a CPU, among others:
- Write: Writes data to an address. There are several variants to this operation, one of which requires a response indication that the operation has been completed.
- Read: A request to read data from an address. A response packet is sent by the target to this request.
- Atomic read-modify-write: Request to write data to an address after reading the previous value, as an atomic operation. Among the supported operations: Atomic increment and atomic decrement, atomic swap, compare and swap etc.
- Several maintenance requests for discovery, control and status. Similar to the PCI protocol's configuration register space, these maintenance requests access registers capability registers (CAR) and command and status registers (CSR).
The main functional difference between RapidIO and PCIe is that the PCIe protocol requires that a central unit (a Root Complex, usually a CPU) configures all endpoints in the system. A RapioIO system doesn't require a central unit of this sort.
Another difference with PCIe is that retransmission of packets is optional. A retransmission occurs only for packets sent as "reliable traffic" (RT). Packets can also be sent as "continuous traffic" (CT). Such packets are neither acknowledged nor retransmitted.
The RapidIO protocol defines all aspects of the communication, from the electrical specification to packet formats, retransmission and flow control.
RapidIO may not be an attractive candidate for a simple point-to-point connection between two FPGAs, due to the complexity of this protocol. RapidIO may be more suitable as an interconnect between several FPGAs through a switch, in particular if PCIe is unsuitable because of its need for a CPU in the system.
Summary
Several protocols for transmitting application data were presented. Each protocol presents a different methods for implementing the transport of data, controlling its flow and responding to bit errors (if at all). The correct protocol for an application is a balance between the features that the protocol offers and the efforts needed for implementing the missing parts in the application logic.
This wraps up the second page in this series about MGTs. The next page introduces some encoding methods often used with MGTs.