OFC 2017 – Inside the Datacenter

100G and the Road to 400G

The transition to 100G network speeds inside the data center is underway at every major hyperscale operator simultaneously, creating major industry bottlenecks. Despite QSFP28 being supply constrained, component and equipment suppliers are also trying to align on the next generation format for 400G operation. Cignal AI’s key takeaways from OFC 2017 with regard to data center optics include:

  • Updates on QSFP28 supply and demand
  • Alternative facts on both sides of the Octal SFP (OSFP) vs. QSFP DD debate
  • Impact of QSFP28 on CFP/CFP2 client demand
  • 400G CFP8 and 200G QSFP observations and outlook

QSFP28 Not Ramping Fast Enough

Two components are enabling this transition to 100G: cutting-edge Tomahawk Ethernet switch silicon from Broadcom and 100G QSFP28 pluggable optics. The Tomahawk switch silicon is ready and available in volume. But QSFP28 optics remain scarce.

The challenge with optics, a challenge that has been present for decades, is that the needed manufacturing and testing to support high production volumes lacks the ability to scale at the speed of silicon. Much discussion at OFC took place about both the short-term outlook for QSFP28 production, what technology will follow, and what can be done in the long term to improve manufacturability. Each component vendor is working to solve unique problems – from the availability of EML lasers to the test equipment needed to add manufacturing capacity.

Initially, demand from hyperscale data center operators centered around the PSM-4 version of the QSFP28 module supplied by Applied Optoelectronics, Luxtera, and others. These modules are widely available. But hyperscale tastes have shifted to the CWDM4 version, which uses a single fiber rather than the four-fiber parallel ribbon of PSM-4.

But what is compounding the problem more than anything else is that demand is arriving from all sources concurrently. Unlike the transition to 40G several years ago, in which the hyperscale operators each transitioned at various times, now all of the hyperscale providers want to move lockstep to 100G as soon as possible. Demand for QSFP28 is skyrocketing as a result.

OFC speakers from Finisar, Lumentum, Applied Optoelectronics, Oclaro, and others all agreed that demand for QSFP28 optics would exceed supply before the end of 2017. These companies are vigorously attacking various manufacturing bottlenecks as they spend capital to expand capacity. Despite this, the demand for 100G QSFP28 optics coming from the large cloud and colo operators is surpassing the rate at which capacity is coming online.

Simple laws of supply and demand dictate that scarcity creates a healthy pricing environment for these 100G client components—they are not cheap to come by. But demand for 100G QSFP28 continues nonetheless because it is such a compelling alternative to 40G, the current speed in most hyperscale data centers. The consensus among vendors is that even at current elevated pricing levels, 100G is still cheaper than 40G from a total cost of ownership perspective. This gives component makers hope that even as supply increases, QSFP28 pricing will remain steady.

Key Takeaways:

  • While there are many vendors advertising QSFP28s, there are not many suppliers who can deliver enough to justify qualifying a new vendor.
  • This market is sure to tip into oversupply at some point in the future but it doesn’t appear to be anytime in the next six months.
  • There does not appear to be any single bottleneck to growing supply as each vendor appears to be struggling with unique manufacturing issues.

400G Client Format Debate Rages

In a year of heated political debates, OFC had it’s very own in the discussion surrounding the format of choice for the next optical client speed – 400G. The two dueling optical module formats are QSFP DD (Double Density) and OSFP (Octal Small Form Factor Pluggable). Both formats may transmit 4x100Gbs optical signals using PAM-4 modulation at 53GBaud, but the OSFP format is slightly larger (32 modules vs. 36 modules in a rack for DD). But that’s where the facts end and the opinions start. What follows will be considered facts by some, opinions by others, and maybe even alternative facts! For example…

One of the advantages of OSFP – as a result of its larger size and depth – is a higher power specification, enabling it to accommodate longer reach formats, perhaps even coherent 100G+ at some point. This is the format championed by Google. Cisco takes a different tack, coming out strongly for DD. Cisco went so far as to present a mechanical and thermal mock-up at OFC illustrating the viability of DD’s recently expanded power specifications.

Cisco VP and General Manager Bill Gartner forcefully denounced OSFP when asked his opinion on the show floor. He refuted the proposition that the next generation format should be encumbered by much lower volume coherent applications, or now unknowable, far out in the future requirements. Katharine Schmidtke, who is responsible for Optical Technology Strategy at Facebook, echoed these views.

Andy Bechtolsheim, CTO of Arista (one of the original supporters of OSFP) argued that Cisco failed to consider thermal performance at high elevations and that there was no design margin in its mock-up. He claimed that in Arista’s opinion, coherent ports in data center equipment were poised to rise as a percentage of ports shipped, and a rearward-looking view that they were a very small percentage of the market was dangerous. Finally – he pointed out the electrical specifications of DD will not work for faster speeds beyond 400G – but OSFP can accommodate this transition.

Conversations and presentations by component vendors were refreshingly free from “alternative facts.” These companies are on the hook for designing and delivering products that are reliable, can be made in volume, and most importantly – in demand by customers. There was a clear consensus here – component vendors have QSFP DD as the design target for high volume data center client optical formats. If OSFP catches on, it will be easy to transfer the QSFP DD design to the bigger OSFP format with more relaxed power specifications. But given a choice between the two, component vendors all preferred OSFP because it is an easier product to make. Bill Gartner, however, had this to say: “This isn’t supposed to be easy.”

While the module format debate has yet to be settled, the component eco-system is rapidly aligning to deliver production grade 400G QSFP DD optics by early 2019. Optics vendors are working closely with the critical PAM-4 semiconductor components. InPhi was an early innovator in this regard, but now many other suppliers are emerging with MACOM and Semtech making announcements at OFC. Further, both public and private demonstrations were made at the show by several vendors including Source Photonics.

Key Takeaways:

  • QSFP DD will exist, the question is whether OSFP does concurrently. We think Google swings this argument.
  • We tend to agree with Cisco that the tail (coherent 100G/400G DCO) should not be wagging the dog (massive volumes of 2km optics).
  • The silver bullet is QSFP DD’s ability to be backward compatible with legacy 100G and 200G speeds.

100G Clients in Telecom Applications

Prior year OFC announcements and technical demos foreshadowed the universal adoption of QSFP28 format even in longer reach, non-hyperscale applications such as the 10km LR format. The recent arrival of QSFP28 LR modules impacts the much lower volume market for telecom client optics formats such as CFP and CFP2.

Oclaro, Lumentum, and Sumitomo Electric all demonstrated ER4 capability in the QSFP28 format, allowing distances as far as 40km to be spanned with QSFP28 optics. This application is currently the domain of the much larger CFP formats, and Oclaro highlighted its interoperability between the new QSFP28 and old CFP formats.

Growth in the larger CFP and CFP2 formats is stalling as OEMs target the smaller QSFP28 format, happy to leverage the massive volume the format will have due to hyperscale demand. Investor concerns were voiced at the show about excess CFP and CFP2 LR4 inventory, specifically at Chinese telecom OEMs such as Huawei and ZTE. Our investigation found these concerns to be valid. But demand for these formats will not disappear, given that telecom hardware has much longer life cycles than hyperscale.

Key Takeaways:

  • There are some short term inventory issues in China but the CFP/CFP2 market isn’t collapsing, it just isn’t growing the way it used to, and it never will again.

Early Adopters – 200G and 400G

There are those who need immediate improvements in speed and density, and they simply cannot wait for 400G QSFP DD in 2019. For these users, two options loom: CFP8 and 200G QSFP28.

Finisar, Oclaro, and Neophotonics have all demonstrated 400G solutions using a CFP8 format that they claim will be production grade by the end of the year. The CFP8 format uses an 8x25Gbaud optical interface using 50G PAM-4 modulation. The lower baud rate of this approach makes its implementation more feasible than QSFP DD in the near term, and the needed components are available now.

And yet, the big question at OFC 2017 was who the customers for CFP8 were, and – more importantly – whether their volume was significant enough to justify the R&D required to bring this product to market. IP Switch and Router companies will be the first to introduce this format, as well as some of the WDM transport vendors who are fielding 400G transport systems in 2018. But will these applications alone create enough demand for CFP8 before the QSFP DD (and yes, the OSFP too) reaches the market in 2019?

Component vendors are hedging their bets. The CFP8 is a great test vehicle for 400G, and if there is a delay in bringing QSFP DD to market, the CFP8 will become much more relevant. There is no expectation that CFP8 would be a high-volume format but rather that CFP8 would be more of an interim luxury good helping smooth the path to much higher volume formats.

But not all end users are waiting for 400G or interested in using the larger CFP8 format. Another alternative to waiting for QSFP28 DD is 200G, with the existing QSFP28 module being sped up with PAM-4 capabilities to double its capacity.

As with 100G, the switch silicon for 200G and 400G will beat the required 400G optics to market. 200G QSFP provides a way to tap this capacity sooner and de-risk the transition to challenging 400G QSFP formats. Broadcom’s upcoming switches with the faster I/O to enable 200G are joined by startups such as Barefoot Networks, which had a compelling demonstration of its new switch that can capitalize on 200G. Like CFP8, the technology is there – the question is whether the demand warrants yet another stopover on the migratory road to 400G.

Google thinks the answer to this question is YES.  Just as 40G was a stopover from 10G to 100G, 200G could serve the same purpose on the path to 400G. Google, with its considerable ability to buy huge amounts of product now, has the attention of component makers.  Most of these folks are keeping their developments quiet, except Luxtera, which used OFC to announce a 200G-specific part.

Key Takeaways:

  • CFP8 will never be a high volume format, but continues the tradition of lower volume telecom client interfaces blazing the trail for the higher volume more compact versions.200G looks like a real market for those who have the credibility to supply Google.
  • 200G looks like a real market for those who have the credibility to supply Google.

Leave a Comment