Open Ethernet, outside the NVIDIA box.

Disclosure

I’m a sales director at Edgecore Networks, which sells open networking switches, including some of the hardware discussed below. This is my personal blog. Views are my own, not Edgecore’s.

I use large language models (LLMs) to help draft and edit these posts. The thoughts, ideas and opinions are my own.

This is my personal blog. I work in sales at Edgecore Networks, but these are my own opinions, not Edgecore’s, and [EDIT: nobody at Edgecore reviewed or approved this / EDIT if Edgecore did review it]. Nothing here is a price, a roadmap, a certification, or a promise. Specs are linked to public pages; check them yourself before you buy anything, from anyone.

Contents

• TL;DR
• Why I’m writing this
• First, the boundary, so nobody is misled
• Who is building outside NCP
• The four networks, in one paragraph
• The pattern I keep seeing
• Where the work actually is
• Where I think Edgecore fits
• What’s known, what’s inferred, what nobody has published
• What I’d ask any vendor, including us
• FAQ
  • Can a Broadcom Tomahawk switch be used in the compute fabric of an NVIDIA NCP build?
  • What does an open Ethernet AI back-end fabric usually look like?
  • Do RoCEv2 AI fabrics need PFC?
  • Where does Edgecore not fit in an AI cluster?
• The upshot

TL;DR

• NCP names the switch. NVIDIA’s published NCP design puts the east-west GPU fabric on InfiniBand (Quantum) or Spectrum-X, so a Broadcom Tomahawk switch is not part of an NCP reference architecture’s compute fabric unless a deployment-specific agreement identifies the affected requirement and approved exception.
• A common open Ethernet pattern. The pattern I keep seeing is a rail-optimized, non-blocking (1:1) leaf-spine on 51.2T switches running SONiC: a 64-port 800G leaf split 1:1 gives 64 NIC ports at 400G down and 32x800G up to the spines.
• Congestion control hasn’t converged. The classic recipe is PFC plus ECN/DCQCN; Meta ran its 400G RoCE fabric on PFC alone with DCQCN off; OpenAI and Microsoft describe production fabrics running lossy, with PFC disabled, using MRC.
• The work is everything around the switch. Optics, NIC to switch link recipes, PFC/ECN tuning, ECMP collisions on elephant flows, host settings and full-scale NCCL/RCCL validation are where FarmGPU and SAKURAONE spent their effort, in public.

Why I’m writing this

I recently read FarmGPU’s post, Building an AI Cluster: Our 17-Day Crash Course in Open Networking, and kept nodding along. Optics that wouldn’t initialize. A ConnectX-7 that wouldn’t link to a SONiC switch until they forced the speed and turned off auto-negotiation. A BIOS setting that cost half the performance. Their line, “the network is about a lot more than just a switch,” is the most honest sentence I’ve read about AI infrastructure this year.

I sell switches for a living, so you can discount my opinion accordingly. But I spend most of my week talking to people building GPU clusters [EDIT: confirm this line is accurate], and the same questions come up every time. So I’m going to answer them here, once a month, as plainly as I can. First up: how are people building open Ethernet AI fabrics today if they’re not building to NVIDIA’s Cloud Partner (NCP) designs, and where does a company like mine actually fit?

First, the boundary, so nobody is misled

I’ll start with what I can’t do, because I see this muddled constantly.

NCP is a partner program for GPU clouds. It isn’t a switch certification. But if you are building a cluster that has to follow NCP’s published design, the east-west GPU fabric is spelled out. NVIDIA’s own NCP software reference guide lists the Cluster Interconnect Network as “InfiniBand (Quantum) or Ethernet (SpectrumX)“. NVIDIA’s GB300 NVL72 inference provider requirements make an “East-west back-end RDMA fabric (InfiniBand or Spectrum-X)” a MUST. NVIDIA’s AI cloud requirements say a deployment-specific agreement doesn’t waive a baseline requirement unless it identifies the requirement and approved exception.

So no, a Broadcom Tomahawk switch is not part of an NCP reference architecture’s compute fabric. Anyone who tells you otherwise should show you the exception in writing.

That’s fine. A lot of clusters don’t need to be NCP builds. This post is for them.

Who is building outside NCP

From the outside, three groups:

• Operators who own their GPUs and their customers. Their tenants care about throughput per dollar and uptime, not which badge is on the network.
• Research, sovereign, and enterprise clusters that want to run their own network stack the way they run their own Linux.
• Non-NVIDIA GPU clusters, where NVIDIA’s network designs don’t apply at all.

Two public examples are worth reading in full. FarmGPU built a B200 cluster on Celestica switches with SONiC and Hedgehog, and wrote down every lesson. And Sakura Internet’s SAKURAONE is 100 nodes of eight H100s each on a rail-optimized, 800GbE leaf-spine running SONiC. It placed 49th on the TOP500 at ISC 2025 [EDIT: confirm the TOP500 #49 cite against the Macnica release]. (Full disclosure: SAKURAONE runs on Edgecore AIS800-64O switches with Enterprise SONiC, per Macnica’s release. I’m citing it because it’s public, not because it’s mine.)

The four networks, in one paragraph

NVIDIA’s NCP guide describes four networks, and it’s a helpful map even if you’re not building to it:

1. Scale-up (NVLink). GPUs sharing memory inside a server or rack. This is not Ethernet, and on NVIDIA GPUs nobody like me plays here.
2. The back-end, east-west fabric. One NIC per GPU, RDMA between servers. This is where the money and the pain are.
3. The front-end. Storage, in-band management, tenant traffic.
4. Out-of-band management. Boring 1G Ethernet to every BMC and PDU, which you will be very grateful for at 2am.

Open Ethernet can live in 2, 3 and 4 when you’re not bound by a program that names the switch.

The pattern I keep seeing

This isn’t an official reference architecture. It’s the rough shape that keeps showing up on whiteboards.

flowchart TB
  subgraph Spines["Spine layer (51.2T, 800G ports)"]
    S1[Spine 1]
    S2[Spine 2]
  end
  subgraph Leaves["Rail leaves (one leaf per GPU rail, 1:1)"]
    L1[Rail 1 leaf]
    L2[Rail 2 leaf]
    L3[Rail 3 leaf]
    L4[Rail 4 leaf]
  end
  subgraph SA["Server A"]
    A1[GPU1 + NIC]
    A2[GPU2 + NIC]
    A3[GPU3 + NIC]
    A4[GPU4 + NIC]
  end
  subgraph SB["Server B"]
    B1[GPU1 + NIC]
    B2[GPU2 + NIC]
    B3[GPU3 + NIC]
    B4[GPU4 + NIC]
  end
  A1 --- L1
  B1 --- L1
  A2 --- L2
  B2 --- L2
  A3 --- L3
  B3 --- L3
  A4 --- L4
  B4 --- L4
  L1 --- S1
  L1 --- S2
  L2 --- S1
  L2 --- S2
  L3 --- S1
  L3 --- S2
  L4 --- S1
  L4 --- S2

A rail-optimized back end, simplified to four rails and two servers. Real HGX servers have eight GPUs and eight rails; front-end and out-of-band networks are separate and not shown.

Rail-optimized leaf-spine, non-blocking. GPU 1 in every server goes to leaf 1, GPU 2 to leaf 2, and so on. Same-rail traffic within a shared leaf group is one hop, which is why SAKURAONE describes its fabric as “Rail-Optimized” with full bisection. Back ends run 1:1. Don’t oversubscribe the thing you bought GPUs to feed.

51.2T switches at 800G, broken out to 400G NICs. A 64-port 800G Tomahawk 5 box can run 64x800G or 128x400G. The napkin math: split a 64-port leaf 1:1 and you get 32 ports down, broken out 2x400G. That’s 64 NIC ports per rail leaf, with 32x800G up to the spines. FarmGPU’s B200 nodes have eight 400G ConnectX-7 NICs, so you can see how quickly the leaf count follows the GPU count.

RoCEv2, usually made lossless on purpose. The classic recipe is PFC to keep RDMA packets from dropping, plus ECN marking so the NICs back off (DCQCN) before the PFC pauses start. The biggest operators have since diverged: Meta ran its 400G RoCE fabric on PFC alone, with DCQCN turned off, and OpenAI and Microsoft now run their newest fabrics lossy, with PFC disabled, using a multipath transport called MRC. These are different transport and workload designs; their settings are not interchangeable. There’s a good plain-English explainer, with the PFC-storm warning, on Edgecore’s blog. Yes, it’s my employer’s. It’s still a good explainer.

Load balancing that understands AI traffic. AI collectives are a few elephant flows, and classic ECMP hashing piles them onto the same link. Tomahawk-class switches offer dynamic load balancing, and newer approaches spray packets based on congestion. FarmGPU’s own table says to turn on RoCE QPN hashing from day one.

SONiC on whitebox. The switch ships with ONIE and you load the NOS. That’s what lets you change vendors without changing how you operate.

Separate front-end and 1G OOB. Keep storage and tenant traffic off the GPU fabric unless you have a really good reason.

Meta’s example also depends on workload and receiver coordination. MRC is a distinct multipath transport with specific implementation requirements, not a generic SONiC setting. Neither is a portable instruction to disable congestion control.

Where the work actually is

Here’s my version of FarmGPU’s table. It’s what I’d plan time for before the GPUs arrive.

Area	What bites people	What I’d do
Optics	Lead times, interop, dust, MPO polarity	Validate every optic SKU on the exact switch; inspect and clean everything
NIC ↔ switch link	Speed, auto-neg, breakout order	Write the link recipe down once and automate it
Lossless config	PFC/ECN thresholds that are “almost right”	Tune with the NIC’s congestion control, not separately
Load balancing	ECMP collisions on elephant flows	Turn on DLB/spraying or QPN-aware hashing early
Host	BIOS, kernel, driver, peermem	Treat the host as part of the network
Validation	”It links” ≠ “it performs”	Run NCCL/RCCL all-reduce at full scale before you hand it over

None of this is exotic. On SAKURAONE, the integrator proposed PFC and ECN parameter settings and worked chip-level troubleshooting with Edgecore. FarmGPU had their switch, NOS, silicon and optics vendors on calls together for days. Open networking isn’t “no help.” It’s choosing which people help you, and getting them to talk to each other.

Where I think Edgecore fits

My opinion, with links so you can check me [EDIT: re-check every product page below right before publishing]:

• Back-end leaf and spine at 51.2T: the AIS800-64O (64x OSFP800) and AIS800-64D (64x QSFP-DD800). Both are Tomahawk 5, 2RU, with RoCEv2, DLB and in-band telemetry listed on the product pages.
• Smaller pods or 1RU leaves: the AIS800-32O and AIS800-32D, with 32x800G on Tomahawk 5.
• 102.4T designs: the AIS1600-64O (64x1.6T) and AIS800-128O (128x800G), both Tomahawk 6, announced in February. The 128-port radix is the interesting part. More ports per box means fewer tiers and fewer hops for a given GPU count.
• NOS: SONiC. Edgecore announced it would collaborate with Broadcom to deliver Enterprise SONiC.
• Edgecore’s own published data: an AIS800-64O / Enterprise SONiC test on AMD MI300X with Broadcom Thor 2 NICs. It showed up to 35.6% better RCCL collective performance with congestion-aware packet spraying than with traditional load balancing. Read the caveats: “up to,” best case at 1:1, AMD, not ConnectX.

And where we don’t fit: NVLink scale-up, and the compute fabric of a build that has to be NCP-compliant. I’d rather lose a deal than have you find that out after the fact.

What’s known, what’s inferred, what nobody has published

• Known (public docs): NCP’s east-west fabric is Quantum InfiniBand or Spectrum-X. Tomahawk 5 + SONiC + ConnectX-7 backends run in production (FarmGPU, SAKURAONE).
• Inferred: a ConnectX NIC on a non-Spectrum switch gives you standard RoCEv2 plus the switch’s own load balancing, not Spectrum-X’s end-to-end adaptive routing.
• Not established by the sources reviewed here: a production-validated lossless RoCEv2 config for Tomahawk 5 with ConnectX-8. If you have one, I’d love to compare notes.

What I’d ask any vendor, including us

1. Which exact optics have you validated on this switch, and at which firmware?
2. Which NIC, NOS version, and PFC/ECN/load-balancing config did you test, and can I have it?
3. Show me NCCL (or RCCL) all-reduce numbers at a scale near mine.
4. Who picks up the phone when the switch vendor, the NIC vendor and the optics vendor all say it’s the other guy?

If a vendor can’t answer those in writing, that’s your answer.

FAQ

Can a Broadcom Tomahawk switch be used in the compute fabric of an NVIDIA NCP build?

Not under NCP’s published design. NVIDIA’s NCP software reference guide lists the Cluster Interconnect Network as InfiniBand (Quantum) or Ethernet (Spectrum-X), and its inference provider requirements make an InfiniBand or Spectrum-X east-west fabric a MUST. A baseline requirement is only waived by an agreement that identifies the affected requirement and approved exception.

What does an open Ethernet AI back-end fabric usually look like?

A rail-optimized, non-blocking (1:1) leaf-spine built on 51.2T switches at 800G, broken out to 400G NICs, running RoCEv2 with tuned congestion control, AI-aware load balancing, and SONiC loaded on whitebox switches via ONIE, with a separate front-end and 1G out-of-band network.

Do RoCEv2 AI fabrics need PFC?

The classic recipe uses PFC plus ECN and DCQCN, but large operators have diverged: Meta ran its 400G RoCE fabric on PFC alone with DCQCN off, and OpenAI and Microsoft describe production fabrics running lossy with PFC disabled, using a multipath transport called MRC.

Where does Edgecore not fit in an AI cluster?

NVLink scale-up, and the compute fabric of a build that has to follow NCP’s published design.

The upshot

FarmGPU said it best: the pain is real, but it’s a one-time cost once somebody writes the recipe down. That’s the point of this blog. Open Ethernet AI fabrics are being built today, in public, at real scale. They just aren’t NCP builds, and they shouldn’t pretend to be.

Next month I want to get into the front-end and storage network, which is the part everyone under-designs. If you’re building one of these and want to compare notes, my inbox is open [EDIT: add contact email or link]. No slide deck, I promise.

Elliot