Cignal AI has spoken with systems and component vendors over the last two months, including at September’s ECOC (see ECOC 2022 Show Report), to get their thoughts on the next wave of 100Gbps and 200Gbps PAM4-based client optics.
Complete quantitative forecasts of 800Gbs and 1.6Tbs optical modules are available in the Optical Components report. We wanted to share our observations and thoughts behind these numbers to help clients understand the next wave of speeds inside the data center:
- The optics follow the chip capabilities in both schedule and interface rate, so look to chip schedules to determine optics schedules.
- 800GbE will start large deployments next year with 1.6TbE and 3.2TbE coming over the next 2-3 years.
- The arguments over QSFP vs OSPF are far from settled, and thermal management will be one of the harder problems to address at the higher rates.
- There is some effort (but not a lot) being dedicated to upgrading lower-speed client optics with the new higher-speed PAM4 rates.
Current generation 400GbE client optics are built on 50Gbps per lane PAM4 electrical connections to the switch fabrics, GPUs, and other silicon chipsets. Next-generation 800GbE modules are being built on 100Gbps per lane PAM4, and 200Gbps PAM4 components have already been demoed. As electrical rates increase, there are more options for high-speed client optics as well as upgrades to lower-speed client optics.
100Gbps and 200Gbps PAM4 Overview
Current 400GbE modules use 8 lanes of electrical 56Gbps PAM4 (often simply referred to as 50Gbps) connections to provide a full 400Gbps of data. In most 400GbE modules, these 56Gbps electrical lanes connect to four 112Gbps optical lanes on separate fibers (e.g., DR4) or wavelengths (e.g., FR4).
| Module | Data Rate | Electrical Lanes (PAM4) |
|---|---|---|
| QSFP28 | 100Gbps | 4x28Gbps |
| QSFP56 | 200Gbps | 4x56Gbps |
| QSFP-DD (aka QSFP56-DD) | 400Gbps | 8x56Gbps |
| SFP-DD112 | 100Gbps | 2x56Gbps |
- A standard QSFP-DD (occasionally called QSFP56-DD) is designed to have eight lanes of 56Gbps signals. DD, or double density, means that the module has twice as many signal paths (8) as a standard QSFP (4).
- A QSFP56 module with four lanes of 56Gbps signals also exists for 200GbE modules and is widely available. An SFP-DD112 supporting 2x56Gbps signals for 100GbE is also defined but is much less popular.
- The widely deployed QSFP28 standard has four lanes of 28Gbps signals for 100GbE modules.
The move to 112Gbps PAM4 (often simply referred to as 100Gbps) doubles the speeds available on each module type. New modules supporting 112Gbps PAM4 are designed not only to support the higher speed, but also the higher power requirement associated with higher speeds. For example, the original specification for QSFP-DD was 14W, although modules are now approaching 20W in practice. The 112Gbps version specification starts at 25W with some vendors already targeting 30W capability.
224Gbps PAM4 (often simply referred to as 200Gbps) doubles the speeds again to ultimately support up to 1.6Tbps or 3.2Tbps per module. New module types with improved heat dissipation (e.g., OSFP-XD) will be required to support these much higher rates.
112Gbps components to support 800GbE modules are mature and early versions of modules based on the technology were demonstrated at OFC22 in March and again at ECOC22 in September.
224Gbps PAM4 is in the early stages of development. At OFC22 last March, COHERENT demonstrated its 224Gbps EMLs, Semtech showed 200Gbps TIAs and drivers, and many vendors hinted at their plans for 224Gbps components and modules.
100Gbps PAM4 Modules
The most popular 112Gbps module will certainly be QSFP-DD800, designed for 8x112Gbps operation to provide 800GbE. However, 112Gbps per lane PAM4 can also be applied to lower-speed modules. For example, an SFP112 is defined which supports a single 100GbE wavelength using 112Gbps PAM4 inputs and outputs, as compared to the current SFP maximum rate of 10-25Gbps. Gearboxing – or using additional logic to convert from lower electrical to higher optical rates – is generally avoided, as it adds cost and power to modules.
| Module | Data Rate | Electrical Lanes (PAM4) |
|---|---|---|
| QSFP-DD800 | 800Gbps | 8x112Gbps |
| QSFP112 | 400Gbps | 4x112Gbps |
| SFP112 | 100Gbps | 1x112Gbps |
| OSFP-XD | 1.6Tbps | 16x112Gbps |
The newest switch silicon supporting 112Gbps PAM-4 interfaces is designed with 800GbE in mind, and the initial hardware built using those chipsets will be designed for the highest density 800GbE support possible. Therefore, the primary focus of optical module makers is the QSFP-DD800 module. Initial versions will be 2x400GbE (2xFR4) and some 8x100GbE, both of which allow greater breakout density and avoid current delays in the definition of an 800GbE MAC by the IEEE. Single interface 800GbE devices will follow closely behind, with lower reach (500m / 2km) DR8 spans for use inside the datacenter being the highest priority.
At least one hardware vendor is also planning to build interfaces for the lower-speed modules (QSFP112 and SFP112). That vendor’s focus is on 100GbE speeds at service providers (as opposed to data center operators) who still see large demand for 100GbE and 400GbE in the metro. A 100GbE module with a single 112Gbps PAM4 input and output will be much lower power than a 4-lane version and can interconnect with the latest switch chips without the need for gearboxing. Putting a 100GbE interface into a smaller SFP format also reduces front panel space and the size of the equipment required to support the interface.
Hardware vendors that focus on data center operators will not build lower-speed (100GbE, 400GbE) modules based on 112Gbps interfaces. The large data center operators are migrating away from 100GbE to 400GbE and see no benefit to updating older equipment that is already capped in the network. The 400GbE modules being deployed now (8x56Gbps) are designed to interface with current switch chips, both of which are still increasing volume deployments. Rather than restart that process with new 4x112Gbps modules and new switch silicon, those operators will wait for the next generation of 1.6TbE modules (8x224Gbps or 16x112Gbps), which will be coming in a few years.
112Gbps interfaces will also enable the first 1.6TbE modules, based on the OSFP-XD specification. OSFP-XD is designed to support 16 signal lanes with the goal of supporting 1.6TbE and, eventually, 3.2TbE.
200Gbps PAM4 Modules
224Gbps PAM module definitions are in the early stages, but the ultimate goal of 224Gbps PAM is to support 3.2TbE (16x224Gbps). A QSFP224 module has been defined to support 800GbE, but it will likely fall into the same category as QSFP112 support for 400GbE with limited actual deployment. Supporters of the QSFP format are optimistic that a version of the QSFP-DD800 module can be designed to support 1.6TbE (8x224Gbps), but that effort has not formally been started.
| Module | Data Rate | Electrical Lanes (PAM4) |
|---|---|---|
| QSFP224 | 800Gbps | 4x224Gbps |
| OSFP-XD | 3.2Tbps | 16x224Gbps |
| QSFP-DD800 (?) | 1.6Tbps | 8x224Gbps |
QSFP vs OSFP
The QSFP and OSFP formats each have champions among module vendors, hardware vendors, and operators. QSFP has been the volume winner for rates up to 400GbE, with large OSFP deployments only at Google, but that may change in future generations.
Perhaps the biggest difference between the formats is the heat sink location. On OSFP, the heat sink is embedded into the module. On QSFP, the heat sink is on board the hardware – in the cage or equipment.
- An embedded heat sink is technically a better solution since designers don’t have to worry about thermal connections to the external sink, the design is more integrated, etc. However, embedded solutions are fixed to the point where they were designed. For example, original OSFP designs assumed 12W, but now solutions are being proposed that will require 30W, which necessitates higher airflow and more fan power.
- On-board heatsinks allow greater flexibility. For example, the onboard heat sink can be designed with different fin profiles from different materials, etc. to better maximize heat dissipation. As a result, QSFP-based high-power designs typically require less airflow and less fan power. One vendor claimed a system power savings of 20% over OSFP designs in their systems.
- However, as power dissipation becomes a bigger problem, the smaller size of QSFP will likely eliminate the ability of an external heatsink to remove all the heat. The OSFP-XD design, which is larger, incorporates both an internal and an external heat sink for both initial simplicity and future-proofing.
QSFP has additional advantages over OSFP that have contributed to its success to date. It is backward compatible, so current solutions can work with previous (and future) generations. An external heat sink also allows this to happen, as higher power generations can be accommodated with innovative heat sink designs. QSFP is also smaller than OSFP, so the number of units that can fit on a faceplate is higher. Generally, router designers prefer QSFP even if they support both module types, but most admit that QSFP will likely run out of gas at 1.6Tbps.
Release Schedules
The release schedule for switch chips determines the demand for the next wave of client optics. Most client optics are deployed in routers today, and client optics require router hardware. (Client optics are also deployed in servers, AI/ML nodes, and other hardware, but that hardware typically lags router deployment.) The general development schedule requires client optics to be available just before the switch chipsets first arrive to allow testing and integration into complete systems. But this is still up to a year before the systems enter volume production.
With 112Gbps-based optics (800GbE), the chipsets are in labs today, which means that small numbers of 800GbE optics have been shipped to systems vendors for integration testing. System deployments are planned for 1H23, but no one claims volume deployments until 2H23. Some whitebox developments at hyperscalers will likely be a little earlier, but no one can escape the switch chipset timing.
Some 800GbE interfaces have been shipped to AI/ML proprietary systems, but these are using custom gearboxes to upscale the 56Gbps PAM4 signals from the AI/ML chipsets to the 112Gbps. As these are not true 800GbE modules but instead are custom-built interfaces, they are not counted in Cignal AI’s 800GbE module shipments.
224Gbps-based optics are expected to be 18 to 24 months behind 112Gbps-based optics, which implies deployment into 2025.
Conclusions
Understanding the technology and roadmap behind the next wave of client optics is critical to router hardware vendors as well as optical hardware vendors who build transponders and muxponders supporting the latest rate. Cignal AI will use the information in this report not only to improve our Optical Components forecasts for 800GbE, 1.6TbE, and 3.2TbE but also to inform our analyses of the routing and optical hardware markets.
Cignal AI would like to thank all of the vendors and operators who provided comprehensive and insightful information for this report.