Gen5 vs. Gen6 PCIe: What do you need to know?

Andy Norrie

The evolution of PCI Express (PCIe) has been instrumental in driving the performance of modern computing. When comparing Gen5 vs. Gen6 PCIe, there are some significant changes that you should be aware of.

 

What is driving PCIe development?

For the last few generations, PCIe performance has been driven by a wide variety of devices, including:

• • • • GPUs
      • NVMe storage devices (SSDs)
      • High-speed Network Interface Cards (NICs)
      • Other high-speed interconnects peripherals, such as USB-C and Thunderbolt
      • With the advent of Gen5 PCIe, however, we saw a smaller range of devices benefiting from the speed boost.

Most SSDs do not require Gen5 speeds due to the limitations of their controllers and NAND flash chips. Gen5 can be a benefit in allowing the same speed while using half the number of PCIe lanes.

Similarly, most NICs and other peripherals can gain the bandwidth they need from a small number of Gen5 lanes.

The main driver for Gen5 and higher speeds is now coming from gaming GPUs and (more recently) AI accelerator requirements. In both cases, it is common to use the full 16 lanes of a PCIe slot, so improving performance further requires a generational step.

Towards the second half of 2023, we have seen a big drive to bring in the roadmaps for Gen6 systems, with several companies aiming for significant development releases in 2024. The demand for GPU compute power is almost entirely driven by AI developments. Doubling the bus speed will be a major benefit to AI companies.

Speed changes

Gen5 PCIe was introduced with a maximum theoretical bandwidth of 32 GT/s (giga transfers per second) per lane. This equates to approximately 4 GB/s per lane (in each direction) or 64 GB/s for a x16 connection.

PCIe Gen6 takes this a step further, doubling the bandwidth again to 64 GT/s per lane. This translates to around 8 GB/s per lane in each direction, resulting in a staggering 128 GB/s for a x16 device.

Note that this is the ‘theoretical’ bandwidth, and real-world performance will be lower due to overhead in the protocol.

 

Signalling changes

• • PAM4
    Gen5 PCIe uses NRZ signalling with 2 voltage levels, which denote binary 0 and 1 . Gen6 uses a more complex PAM4 system that has 4 voltage levels. Both Gen5 and Gen6 have the same 16 GHz fundamental frequency, but due to the PAM4 encoding, Gen6 has one-third of the noise immunity of Gen5, requiring tighter design tolerances. Understanding NRZ and PAM4 Signalling
  • Forward Error Correction or FEC
    This is a new addition to the protocol and is used to correct smaller errors that occur in transmission without the need to re-transmit data. This additional error recovery data adds additional overhead compared to Gen 5.
  • Flow control unit (FLIT)
    Gen 6 PCIe introduces new data transmission structures along with FEC to reduce overhead and allow faster data transmission. This will require new decoding systems and is a big change from Gen 5.

 

Power efficiency

New power-saving states allow some lanes to shut down while others continue running. This allows for scalable performance as loads change while minimising power use. The new power state is called L0p

Connector changes

Gen6 connectors are already available on the market for AIC (slot) devices.

The modern EDSFF connector for E1, E3, and CXL are also expected to be available in Gen6 versions.

It is quite likely that SFF-8639 (Used for U.2 and U.3 drives) will not go to Gen6, given the older nature of this connector and the lower benefit for Gen6 on SSDs

 

Routing changes

Routing for Gen6 signalling will be a major challenge. FEC will help with the recovery of smaller errors, but the move to PAM4 will significantly reduce the SI (signal integrity) overheads in the system. This will make data liable to error due to loss and crosstalk

• • • Loss
      The total insertion loss budget for Gen6 is 32dB, down from 36dB in the Gen5 spec. This is a small but significant change and will limit the length of traces and the number of transitions (connectors and similar)
    • Cross-talk
      This is interference from one lane to another (Crosstalk). With PAM4, the probability of interference changing a bit of data increases significantly. This makes cross-talk a much higher risk for Gen6 systems and will require a more sophisticated design to mitigate.

 

Gen5 Vs Gen6 PCIe Compatibility

As with previous PCIe generations, back and forward compatibility had been maintained. Old devices should function in new Gen6 slots, and Gen6 devices should step down to lower speeds when placed in an older slot. The requirement for backwards compatibility adds significant complexity to Gen6 hardware, and in particular the SerDes, which must support both NRZ and PAM4 encoding and switch between them in flight.

Testing Gen6 devices

Quarch is working hard on Gen6 products, so when you have prototype devices, we should be ready with test solutions for power analysis and hot-plug / fault injection.

If you have  a schedule for when you need testing, please get in touch via support@quarch.com