Non-blocking

Before the switches, name the networks.

Tue, 03 Nov 2026 19:00:00 GMT

Disclosure

I work in sales at Edgecore Networks. This is my personal blog, and these are my views. [EDIT: confirm author wording and review status before publication]. No company endorsement, certification or deployment acceptance is claimed.

I use large language models (LLMs) to help draft and edit these posts. The thoughts, ideas and opinions are my own.

TL;DR

Before choosing switches, name the job each network does. NVIDIA’s terminology is a useful map: NVLink for scale-up, CIN for external compute communication, TAN for tenant access and SMN for infrastructure management. Storage and inference traffic need to be placed deliberately on that map.

That gives us two different conversations about Edgecore: building an independent Ethernet network around NVIDIA GPUs, and proposing specific Edgecore placements in an NCP deployment.

Start with the job, then the hardware

A diagram can make every line look equivalent. They aren’t. A model loading from storage, GPUs exchanging collectives and a management system reaching a BMC all need different things from the network.

NVIDIA’s NCP network table gives those roles names:

Term	The practical meaning
NVLink scale-up	The local GPU communication domain. Its physical boundary depends on the platform.
CIN: Cluster Interconnect Network	External compute communication between GPU systems.
TAN: Tenant Access Network	Tenant access to storage and external services. Shared services can be logical overlays here.
SMN: Secure Management Network	Restricted access to infrastructure management interfaces.

Those are responsibilities, not a universal instruction to buy four physically separate switch fabrics. A storage network can be converged or dedicated. Tenant isolation involves the switches, endpoints and control plane.

NVL72 is a platform, not an Ethernet topology

This distinction matters when somebody asks whether an open Ethernet switch can replace “the NVL72 network.”

The rack includes an internal NVLink/NVSwitch system. Edgecore Ethernet doesn’t replace that domain. An external scale-out fabric is a different design question. Frontend and infrastructure management connections are different again. NVIDIA’s component description is the place to establish the rack boundary.

An HGX B300 server and a GB300 NVL72 rack also need different attachment plans. We shouldn’t carry a server count, DPU link rate or management-port budget from one into the other without checking the selected profile.

Inference has more than one network conversation

A user request entering the service is one path. Model weights moving from storage are another. Disaggregated serving can move KV cache between prefill and decode workers, while model-parallel GPUs exchange their own communication.

The NVIDIA inference RA separates the serving architecture from the capabilities delivered by the provider. That’s a useful discipline: identify the flow, the endpoints and the required transport before drawing the fabric.

For a customer, I’d put each flow on the diagram and ask what happens when it competes with the others. Then measure the serving result. Time to first token and latency under concurrency tell us something a switch’s port count cannot.

Two Edgecore conversations

An independent open Ethernet deployment. Here, we can propose Edgecore for external compute communication as well as frontend, storage and management. The proposal still needs a selected NOS release, optics, NIC configuration and workload tests. Port speed alone doesn’t establish the implementation.

A deployment following NCP requirements. Here, the selected profile governs the design. The GB300 NVL72 provider requirements prescribe BlueField-3 north-south under NET-1 and InfiniBand or Spectrum-X east-west under NET-2. Proposed Edgecore placements elsewhere require their own design and acceptance review. Calling traffic “inference interconnect” doesn’t remove the requirement that applies to its actual role.

This is why I prefer a placement map to a blanket compatibility statement. It tells the customer which equipment we’re proposing, what stays in place and what needs to be tested.

What I’d want the diagram to tell me

For each connection, show the source and destination, physical interface, logical network and failure boundary. For each number, say whether it is a port-rate sum, an allocation or a measured result.

For each conclusion, distinguish a documented specification, a design assumption, a tested configuration and an accepted deployment. Those are useful answers to different questions.

[EDIT: add one customer-approved example once the exact platform and test profile are available]

FAQ

Does an Ethernet link establish NCP acceptance?

No. The complete applicable requirements and deployment conditions govern acceptance. NVIDIA’s v2.4 baseline extends well beyond the network hardware.

Is the tenant network another physical fabric?

It can be, but it can also be an overlay on TAN. State the isolation model and actual connections rather than assuming a separate tier of switches.

Where should I go next?

Read the open Ethernet fabric discussion for the independent design, or the frontend, storage and OOB discussion for the surrounding networks.

Sources

Open Ethernet, outside the NVIDIA box.

Tue, 03 Nov 2026 19:00:00 GMT

Disclosure

I’m a sales director at Edgecore Networks, which sells open networking switches, including some of the hardware discussed below. This is my personal blog. Views are my own, not Edgecore’s.

I use large language models (LLMs) to help draft and edit these posts. The thoughts, ideas and opinions are my own.

This is my personal blog. I work in sales at Edgecore Networks, but these are my own opinions, not Edgecore’s, and [EDIT: nobody at Edgecore reviewed or approved this / EDIT if Edgecore did review it]. Nothing here is a price, a roadmap, a certification, or a promise. Specs are linked to public pages; check them yourself before you buy anything, from anyone.

TL;DR

NCP names the switch. NVIDIA’s published NCP design puts the east-west GPU fabric on InfiniBand (Quantum) or Spectrum-X, so a Broadcom Tomahawk switch is not part of an NCP reference architecture’s compute fabric unless a deployment-specific agreement identifies the affected requirement and approved exception.
A common open Ethernet pattern. The pattern I keep seeing is a rail-optimized, non-blocking (1:1) leaf-spine on 51.2T switches running SONiC: a 64-port 800G leaf split 1:1 gives 64 NIC ports at 400G down and 32x800G up to the spines.
Congestion control hasn’t converged. The classic recipe is PFC plus ECN/DCQCN; Meta ran its 400G RoCE fabric on PFC alone with DCQCN off; OpenAI and Microsoft describe production fabrics running lossy, with PFC disabled, using MRC.
The work is everything around the switch. Optics, NIC to switch link recipes, PFC/ECN tuning, ECMP collisions on elephant flows, host settings and full-scale NCCL/RCCL validation are where FarmGPU and SAKURAONE spent their effort, in public.

Why I’m writing this

I recently read FarmGPU’s post, Building an AI Cluster: Our 17-Day Crash Course in Open Networking, and kept nodding along. Optics that wouldn’t initialize. A ConnectX-7 that wouldn’t link to a SONiC switch until they forced the speed and turned off auto-negotiation. A BIOS setting that cost half the performance. Their line, “the network is about a lot more than just a switch,” is the most honest sentence I’ve read about AI infrastructure this year.

I sell switches for a living, so you can discount my opinion accordingly. But I spend most of my week talking to people building GPU clusters [EDIT: confirm this line is accurate], and the same questions come up every time. So I’m going to answer them here, once a month, as plainly as I can. First up: how are people building open Ethernet AI fabrics today if they’re not building to NVIDIA’s Cloud Partner (NCP) designs, and where does a company like mine actually fit?

First, the boundary, so nobody is misled

I’ll start with what I can’t do, because I see this muddled constantly.

NCP is a partner program for GPU clouds. It isn’t a switch certification. But if you are building a cluster that has to follow NCP’s published design, the east-west GPU fabric is spelled out. NVIDIA’s own NCP software reference guide lists the Cluster Interconnect Network as “InfiniBand (Quantum) or Ethernet (SpectrumX)“. NVIDIA’s GB300 NVL72 inference provider requirements make an “East-west back-end RDMA fabric (InfiniBand or Spectrum-X)” a MUST. NVIDIA’s AI cloud requirements say a deployment-specific agreement doesn’t waive a baseline requirement unless it identifies the requirement and approved exception.

So no, a Broadcom Tomahawk switch is not part of an NCP reference architecture’s compute fabric. Anyone who tells you otherwise should show you the exception in writing.

That’s fine. A lot of clusters don’t need to be NCP builds. This post is for them.

Who is building outside NCP

From the outside, three groups:

Operators who own their GPUs and their customers. Their tenants care about throughput per dollar and uptime, not which badge is on the network.
Research, sovereign, and enterprise clusters that want to run their own network stack the way they run their own Linux.
Non-NVIDIA GPU clusters, where NVIDIA’s network designs don’t apply at all.

Two public examples are worth reading in full. FarmGPU built a B200 cluster on Celestica switches with SONiC and Hedgehog, and wrote down every lesson. And Sakura Internet’s SAKURAONE is 100 nodes of eight H100s each on a rail-optimized, 800GbE leaf-spine running SONiC. It placed 49th on the TOP500 at ISC 2025 [EDIT: confirm the TOP500 #49 cite against the Macnica release]. (Full disclosure: SAKURAONE runs on Edgecore AIS800-64O switches with Enterprise SONiC, per Macnica’s release. I’m citing it because it’s public, not because it’s mine.)

The four networks, in one paragraph

NVIDIA’s NCP guide describes four networks, and it’s a helpful map even if you’re not building to it:

Scale-up (NVLink). GPUs sharing memory inside a server or rack. This is not Ethernet, and on NVIDIA GPUs nobody like me plays here.
The back-end, east-west fabric. One NIC per GPU, RDMA between servers. This is where the money and the pain are.
The front-end. Storage, in-band management, tenant traffic.
Out-of-band management. Boring 1G Ethernet to every BMC and PDU, which you will be very grateful for at 2am.

Open Ethernet can live in 2, 3 and 4 when you’re not bound by a program that names the switch.

The pattern I keep seeing

This isn’t an official reference architecture. It’s the rough shape that keeps showing up on whiteboards.

flowchart TB
  subgraph Spines["Spine layer (51.2T, 800G ports)"]
    S1[Spine 1]
    S2[Spine 2]
  end
  subgraph Leaves["Rail leaves (one leaf per GPU rail, 1:1)"]
    L1[Rail 1 leaf]
    L2[Rail 2 leaf]
    L3[Rail 3 leaf]
    L4[Rail 4 leaf]
  end
  subgraph SA["Server A"]
    A1[GPU1 + NIC]
    A2[GPU2 + NIC]
    A3[GPU3 + NIC]
    A4[GPU4 + NIC]
  end
  subgraph SB["Server B"]
    B1[GPU1 + NIC]
    B2[GPU2 + NIC]
    B3[GPU3 + NIC]
    B4[GPU4 + NIC]
  end
  A1 --- L1
  B1 --- L1
  A2 --- L2
  B2 --- L2
  A3 --- L3
  B3 --- L3
  A4 --- L4
  B4 --- L4
  L1 --- S1
  L1 --- S2
  L2 --- S1
  L2 --- S2
  L3 --- S1
  L3 --- S2
  L4 --- S1
  L4 --- S2

A rail-optimized back end, simplified to four rails and two servers. Real HGX servers have eight GPUs and eight rails; front-end and out-of-band networks are separate and not shown.

Rail-optimized leaf-spine, non-blocking. GPU 1 in every server goes to leaf 1, GPU 2 to leaf 2, and so on. Same-rail traffic within a shared leaf group is one hop, which is why SAKURAONE describes its fabric as “Rail-Optimized” with full bisection. Back ends run 1:1. Don’t oversubscribe the thing you bought GPUs to feed.

51.2T switches at 800G, broken out to 400G NICs. A 64-port 800G Tomahawk 5 box can run 64x800G or 128x400G. The napkin math: split a 64-port leaf 1:1 and you get 32 ports down, broken out 2x400G. That’s 64 NIC ports per rail leaf, with 32x800G up to the spines. FarmGPU’s B200 nodes have eight 400G ConnectX-7 NICs, so you can see how quickly the leaf count follows the GPU count.

RoCEv2, usually made lossless on purpose. The classic recipe is PFC to keep RDMA packets from dropping, plus ECN marking so the NICs back off (DCQCN) before the PFC pauses start. The biggest operators have since diverged: Meta ran its 400G RoCE fabric on PFC alone, with DCQCN turned off, and OpenAI and Microsoft now run their newest fabrics lossy, with PFC disabled, using a multipath transport called MRC. These are different transport and workload designs; their settings are not interchangeable. There’s a good plain-English explainer, with the PFC-storm warning, on Edgecore’s blog. Yes, it’s my employer’s. It’s still a good explainer.

Load balancing that understands AI traffic. AI collectives are a few elephant flows, and classic ECMP hashing piles them onto the same link. Tomahawk-class switches offer dynamic load balancing, and newer approaches spray packets based on congestion. FarmGPU’s own table says to turn on RoCE QPN hashing from day one.

SONiC on whitebox. The switch ships with ONIE and you load the NOS. That’s what lets you change vendors without changing how you operate.

Separate front-end and 1G OOB. Keep storage and tenant traffic off the GPU fabric unless you have a really good reason.

Meta’s example also depends on workload and receiver coordination. MRC is a distinct multipath transport with specific implementation requirements, not a generic SONiC setting. Neither is a portable instruction to disable congestion control.

Where the work actually is

Here’s my version of FarmGPU’s table. It’s what I’d plan time for before the GPUs arrive.

Area	What bites people	What I’d do
Optics	Lead times, interop, dust, MPO polarity	Validate every optic SKU on the exact switch; inspect and clean everything
NIC ↔ switch link	Speed, auto-neg, breakout order	Write the link recipe down once and automate it
Lossless config	PFC/ECN thresholds that are “almost right”	Tune with the NIC’s congestion control, not separately
Load balancing	ECMP collisions on elephant flows	Turn on DLB/spraying or QPN-aware hashing early
Host	BIOS, kernel, driver, peermem	Treat the host as part of the network
Validation	”It links” ≠ “it performs”	Run NCCL/RCCL all-reduce at full scale before you hand it over

None of this is exotic. On SAKURAONE, the integrator proposed PFC and ECN parameter settings and worked chip-level troubleshooting with Edgecore. FarmGPU had their switch, NOS, silicon and optics vendors on calls together for days. Open networking isn’t “no help.” It’s choosing which people help you, and getting them to talk to each other.

Where I think Edgecore fits

My opinion, with links so you can check me [EDIT: re-check every product page below right before publishing]:

Back-end leaf and spine at 51.2T: the AIS800-64O (64x OSFP800) and AIS800-64D (64x QSFP-DD800). Both are Tomahawk 5, 2RU, with RoCEv2, DLB and in-band telemetry listed on the product pages.
Smaller pods or 1RU leaves: the AIS800-32O and AIS800-32D, with 32x800G on Tomahawk 5.
102.4T designs: the AIS1600-64O (64x1.6T) and AIS800-128O (128x800G), both Tomahawk 6, announced in February. The 128-port radix is the interesting part. More ports per box means fewer tiers and fewer hops for a given GPU count.
NOS: SONiC. Edgecore announced it would collaborate with Broadcom to deliver Enterprise SONiC.
Edgecore’s own published data: an AIS800-64O / Enterprise SONiC test on AMD MI300X with Broadcom Thor 2 NICs. It showed up to 35.6% better RCCL collective performance with congestion-aware packet spraying than with traditional load balancing. Read the caveats: “up to,” best case at 1:1, AMD, not ConnectX.

And where we don’t fit: NVLink scale-up, and the compute fabric of a build that has to be NCP-compliant. I’d rather lose a deal than have you find that out after the fact.

What’s known, what’s inferred, what nobody has published

Known (public docs): NCP’s east-west fabric is Quantum InfiniBand or Spectrum-X. Tomahawk 5 + SONiC + ConnectX-7 backends run in production (FarmGPU, SAKURAONE).
Inferred: a ConnectX NIC on a non-Spectrum switch gives you standard RoCEv2 plus the switch’s own load balancing, not Spectrum-X’s end-to-end adaptive routing.
Not established by the sources reviewed here: a production-validated lossless RoCEv2 config for Tomahawk 5 with ConnectX-8. If you have one, I’d love to compare notes.

What I’d ask any vendor, including us

Which exact optics have you validated on this switch, and at which firmware?
Which NIC, NOS version, and PFC/ECN/load-balancing config did you test, and can I have it?
Show me NCCL (or RCCL) all-reduce numbers at a scale near mine.
Who picks up the phone when the switch vendor, the NIC vendor and the optics vendor all say it’s the other guy?

If a vendor can’t answer those in writing, that’s your answer.

FAQ

Can a Broadcom Tomahawk switch be used in the compute fabric of an NVIDIA NCP build?

Not under NCP’s published design. NVIDIA’s NCP software reference guide lists the Cluster Interconnect Network as InfiniBand (Quantum) or Ethernet (Spectrum-X), and its inference provider requirements make an InfiniBand or Spectrum-X east-west fabric a MUST. A baseline requirement is only waived by an agreement that identifies the affected requirement and approved exception.

What does an open Ethernet AI back-end fabric usually look like?

A rail-optimized, non-blocking (1:1) leaf-spine built on 51.2T switches at 800G, broken out to 400G NICs, running RoCEv2 with tuned congestion control, AI-aware load balancing, and SONiC loaded on whitebox switches via ONIE, with a separate front-end and 1G out-of-band network.

Do RoCEv2 AI fabrics need PFC?

The classic recipe uses PFC plus ECN and DCQCN, but large operators have diverged: Meta ran its 400G RoCE fabric on PFC alone with DCQCN off, and OpenAI and Microsoft describe production fabrics running lossy with PFC disabled, using a multipath transport called MRC.

Where does Edgecore not fit in an AI cluster?

NVLink scale-up, and the compute fabric of a build that has to follow NCP’s published design.

The upshot

FarmGPU said it best: the pain is real, but it’s a one-time cost once somebody writes the recipe down. That’s the point of this blog. Open Ethernet AI fabrics are being built today, in public, at real scale. They just aren’t NCP builds, and they shouldn’t pretend to be.

Next month I want to get into the front-end and storage network, which is the part everyone under-designs. If you’re building one of these and want to compare notes, my inbox is open [EDIT: add contact email or link]. No slide deck, I promise.

Elliot

Everything around the GPU fabric.

Tue, 03 Nov 2026 19:00:00 GMT

Disclosure

I’m a sales director at Edgecore Networks, which sells open networking switches, including some of the hardware discussed below. This is my personal blog. Views are my own, not Edgecore’s.

I use large language models (LLMs) to help draft and edit these posts. The thoughts, ideas and opinions are my own.

TL;DR

Treat each NCP placement as its own design decision. NVIDIA’s NCP guide lists the Cluster Interconnect Network as InfiniBand (Quantum) or Spectrum-X, and the Secure Management Network as “Ethernet (1GbE)” with no vendor. The Tenant Access Network is listed as “Ethernet (Spectrum)”, and public documentation does not establish acceptance of a substitute switch.
Count frontend links separately from usable throughput. Using NVIDIA’s HGX B300 interfaces, a 32-node HGX B300 cluster puts 64 x 400G frontend links (25.6 Tb/s) next to 204.8 Tb/s of ConnectX-8 bandwidth. A pair of 64-port 800G switches covers it with ports to spare. The hard part is BGP/EVPN and the BlueField-3 DPU, which NET-1 makes a MUST for the cited GB300 NVL72 provider profile.
Storage forces a choice before cabling: RoCE or TCP. If storage rides the converged frontend with RDMA, you are running a lossless class next to tenant TCP on the same switches, and that needs testing.
OOB is bigger than everyone draws it. NVIDIA counts six 1G management connections per HGX B300 server: 192 ports for 32 servers before a single switch, PDU or console port.

Why I’m writing this

In my last post I promised the frontend and storage network next. The most common reply was fair: “Our build has to follow NCP, so the GPU fabric is Spectrum-X. Is anything left for open networking?”

Yes, and it’s not small. A GPU cluster has four networks, with distinct requirements for NVLink scale-up and the CIN. Frontend and management substitutions still need deployment acceptance. This post examines those candidate placements and storage, using a 32-node B300 design I worked on: 32 HGX B300 servers, 256 GPUs, GPU fabric held constant as Spectrum-X.

First, the boundary

NVIDIA’s NCP Part 1 guide has a “Data Centre Networks” table. It lists the CIN as “InfiniBand (Quantum) or Ethernet (SpectrumX)”, the TAN as “Ethernet (Spectrum)”, and the Secure Management Network as “Ethernet (1GbE)” with no vendor. The data center architecture page says “The TAN and SMN are always Ethernet”.

So I won’t tell you NCP “defines nothing” for the frontend. It names Spectrum in a reference table. The OOB row names no vendor, but this does not establish acceptance of an Edgecore substitution. The NCP Hardware Reference Design is behind a partner login, and I haven’t read it. If your build has to claim compliance, get the frontend and OOB switch choice approved in writing.

One more honest note. NVIDIA’s HGX B300 Enterprise RA uses SN5600 switches for the converged fabric and SN2201 for OOB. Swap those and you are following its topology, not the RA.

flowchart TB
  subgraph Locked["Named by NVIDIA for NCP builds"]
    BE["GPU fabric (CIN)<br/>Spectrum-X or Quantum"]
  end
  subgraph Open["Where open Ethernet can fit"]
    FE["Frontend / converged pair<br/>2 x 64-port 800G switches"]
    OOB["OOB / SMN<br/>48-port 1G switches"]
  end
  SRV["HGX B300 server<br/>8 GPUs"]
  ST["Storage servers"]
  BR["Border / internet"]
  CON["Console server"]
  SRV -- "8 x ConnectX-8" --> BE
  SRV -- "BlueField-3 DPU, 2 x 400G" --> FE
  ST --- FE
  FE --- BR
  SRV -- "BMCs, 1G" --> OOB
  FE -. "mgmt port" .-> OOB
  BE -. "mgmt port" .-> OOB
  CON -. "serial" .-> FE
  CON -. "serial" .-> OOB

One server, three networks you can touch. NVLink scale-up and the storage system’s internal network are not shown.

The frontend: small pipes, hard rules

What it carries

NVIDIA’s HGX B300 RA calls it the CPU Converged (North/South) network and says it “is used for node communications with compute, storage, in-band management, and end-user connections.” On NCP, in-band management, storage, and tenant traffic all ride it.

The sizing math for 32 nodes

Each B300 server in that RA has “One NVIDIA BlueField-3 B3240 dual port 400 GbE DPU”, connected “using two 400 GbE ports to two separate switches”. These are port line rates, not a guarantee that one BlueField-3 delivers 800 Gb/s aggregate host throughput. NVIDIA’s card-limit descriptions vary by page and attachment profile; confirm the selected card and configuration. See the NVL72 networking hardware description.

So:

32 servers x 2 = 64 x 400G links, 32 to each switch, 25.6 Tb/s total.
The GPU fabric: 32 servers x 8 ConnectX-8 at 800G (configured as 2x400G) = 204.8 Tb/s. That is a 1:8 ratio of nominal link rates, not a measure of usable BlueField throughput. NVIDIA’s own 4-server block shows the same ratio: 3.2 Tb/s converged next to 25.6 Tb/s compute.

The design I worked on used two AIS800-64O switches for the frontend. Each has 64 OSFP800 ports, each runnable as 2 x 400G. The 32 server links on each switch take 16 ports. That leaves 48 ports per switch for storage, the link between the pair, and uplinks to the border [EDIT: border uplink count and speed for the example]. NVIDIA’s own 32-node page calls for a “cost-efficient converged two-switch fabric”, so the shape matches. At 64 nodes the design adds a frontend spine pair and a separate storage leaf pair.

The hard parts

NET-1 makes the DPU mandatory in the cited GB300 NVL72 profile. NVIDIA’s inference provider requirements, written for NCPs building GB300 NVL72 clusters, make NET-1 a MUST: “North-south through the BlueField-3 DPU enforcing zero trust and host multi-tenancy; carries control plane, storage, external connectivity”. NET-5 makes BGP with “IPv4 and EVPN/VPNv4 address families” a MUST, and NET-6 says multitenancy SHOULD use BGP underlay, EVPN, VXLAN overlay and DPU ACLs.

My inference: if the DPU terminates the overlay, the leaf is mostly a BGP underlay router. If the leaf is the VTEP, its EVPN implementation is the product. Know which one you have before you pick a NOS.

Dual homing. Two links to two switches means MLAG, EVPN multihoming, or routing to the host. Each fails differently. Pick one and pull cables during testing.

Storage: converged until it isn’t

What it carries

Checkpoints, datasets, model weights. The NCP guide calls storage the TAN’s “primary consumer”. In the 32-node design, four storage servers dual home to the same frontend pair [EDIT: storage vendor, NIC speed and ports per storage server]. At 64 nodes storage gets its own storage fabric leaf pair, uplinked to the frontend spines.

RoCE or TCP

This is the decision that drives switch config. GPUDirect Storage enables “a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU.” The RDMA flavors of that path want a tuned RoCEv2 class with PFC and ECN on the frontend. TCP-based storage needs none of that.

On a converged frontend, RoCE storage means a lossless class sharing buffers with lossy tenant traffic. That works, but it’s a design, not a checkbox. Ask the storage vendor which transport they tested at your scale, then configure exactly that. If it’s TCP, keep the frontend lossy and simple.

OOB: the network everyone undercounts

What it carries

NVIDIA’s RA connects “BMC ports of the server nodes”, “BMC ports of the Bluefield-3 DPU and SuperNICs” and “OOB management ports of switches”, and says to scale it with a 25 or 100 Gbps spine layer. Add PDUs, CDUs and the console server.

The sizing math, and my own miss

The first pass of the 32-node design counted about 54 OOB ports (one BMC per server, 18 switch management ports, 4 storage servers) and two 48-port 1G switches. Then I read NVIDIA’s HGX B300 page again: “4 servers, each with 6x 1Gb/s connections”.

32 servers x 6 = 192 ports
18 data switches in that design + 4 storage servers = 22 more
Total 214, before PDUs, CDUs or the console server

A 48-port 1G switch like the EPS201 (AS4630-54TE), with 48 x 1G RJ45, 4 x 25G and 2 x 100G ports, means five switches, not two [EDIT: confirm the actual OEM management-port schedule, then add infrastructure and spare capacity]. The 25G and 100G uplinks give you the spine layer NVIDIA describes.

Console

Every SONiC switch should have a serial console reachable when its management port isn’t. That’s how you recover a box stuck in ONIE or a bad upgrade. Count consoles like you count BMCs: 18 data switches plus the OOB switches themselves [EDIT: console server model and port count used].

What to validate, by network

Network	What it carries	32-node sizing	What bites people	What I’d validate
Frontend (TAN)	In-band mgmt, tenant traffic, internet	64 x 400G from BlueField-3, 2 switches	BGP/EVPN behavior, dual homing failover	EVPN with your DPU overlay design; pull one link and one switch under load
Storage	Checkpoints, datasets, weights	4 storage servers dual homed (converged)	RoCE lossless class next to TCP	Your storage vendor’s exact transport; checkpoint write at full scale
OOB (SMN)	BMCs, switch mgmt, PDUs, CDUs	214+ x 1G, 5 x 48-port switches	Six ports per server, not one	Every BMC reachable; serial console to every switch
GPU fabric (CIN)	NCCL between GPUs	256 x 800G ConnectX-8	Not this post	Spectrum-X on NCP builds

What’s known, what’s inferred, what nobody has published

Known (public docs): NCP lists the CIN as Quantum or Spectrum-X, the TAN as “Ethernet (Spectrum)”, and the SMN as 1GbE with no vendor. NET-1 makes BlueField-3 north-south a MUST for NCPs hosting inference providers on GB300 NVL72. NVIDIA’s HGX B300 RA uses one dual port 400 GbE BlueField-3 per server and six 1G OOB connections per server.
Inferred: frontend and OOB switch substitutions remain proposed; public documentation does not establish their acceptance. When the DPU terminates the overlay, the frontend leaf’s job gets simpler.
Not established by the sources reviewed here: the gated NCP Hardware Reference Design’s frontend and OOB switch rules. The tables in NVIDIA’s HGX guide also disagree on converged switch counts at larger sizes, so I haven’t quoted them. If you’ve seen either settled, I’d like to compare notes.

FAQ

Can the frontend network of an NVIDIA NCP build use non-NVIDIA switches?

Public NCP text doesn’t exclude a non-NVIDIA frontend switch, but its reference table lists the Tenant Access Network as Ethernet (Spectrum), and the gated NCP Hardware Reference Design wasn’t public for me to check. Get it in writing for your specific build.

Is a BlueField-3 DPU required on the frontend?

For NCPs hosting inference providers on GB300 NVL72, yes: NVIDIA’s NET-1 requirement makes north-south through the BlueField-3 DPU a MUST, carrying control plane, storage and external connectivity. NVIDIA’s HGX B300 Enterprise RA also uses one BlueField-3 B3240 dual port 400 GbE DPU per server.

How much frontend bandwidth does a 32-node HGX B300 cluster need?

In NVIDIA’s HGX B300 design each server has two 400 GbE frontend ports to two separate switches, so 32 servers bring 64 x 400G links, or 25.6 Tb/s. As a sum of port line rates, that is one eighth of the 204.8 Tb/s the eight ConnectX-8 SuperNICs per server put on the GPU fabric.

How many out-of-band ports does a 32-node B300 cluster need?

NVIDIA’s HGX B300 Enterprise RA counts six 1 Gb/s management connections per server, so 32 servers need 192 ports before you add switches, storage, PDUs and console servers. That is at least five 48-port switches once the additional equipment in this example is included.

The upshot

Even on a build where NVIDIA names the GPU fabric, a 32-node B300 cluster still has a frontend pair, a storage decision and a few hundred OOB ports that public NCP text doesn’t lock to one vendor. Those networks are smaller than the GPU fabric, but they are where clusters break at 2am.

Next up: drawing a 32-server rail group honestly, every cable. If you’re sizing one of these and want to compare notes, my inbox is open [EDIT: add contact email or link]. No slide deck, I promise.

Elliot

Sources

NCP Software Reference Guide, Data Center Architecture: “Networking” and compute tray table, NVIDIA, accessed 2026-11-02
NCP Software Reference Guide Part 1, Infrastructure as a Service: “Data Centre Networks” table, NVIDIA, accessed 2026-11-02
Inference Provider Platform Requirements on GB300 NVL72 for NCPs: “Networking and Multi-tenancy”, NET-1 to NET-8, NVIDIA, accessed 2026-11-02
HGX AI Factory Enterprise RA, Networking Logical Architecture: “CPU Converged (North/South) Network”, “Enterprise RA Scalable Unit (SU)”, “Out-of-Band (OOB) Management Network”, Table 5, “32 Nodes with 256 NVIDIA HGX B300 SXM GPUs”, NVIDIA, accessed 2026-11-02
GPUDirect Storage Overview Guide: 1.1 Introduction, NVIDIA, accessed 2026-11-02
AIS800-64O product page, Edgecore Networks, accessed 2026-11-02
EPS201 (AS4630-54TE) product page, Edgecore Networks, accessed 2026-11-02

Profile boundary: General storage designs can use TCP or RDMA. The cited GB300 NVL72 provider profile specifically requires RDMA and a certified storage partner under STO-2/3. An Edgecore switch placement remains proposed until the hardware/storage design and deployment acceptance are checked. NVIDIA provider requirements.

About Elliot Lombardo

Sun, 01 Nov 2026 06:00:00 GMT

Elliot Lombardo is a sales director at Edgecore Networks who writes this personal, unofficial blog about AI cluster networking and open networking topologies. It is not an Edgecore publication, and the views here are his own.

Disclaimer

Disclosure

I’m a sales director at Edgecore Networks, which sells open networking switches, including some of the hardware discussed below. This is my personal blog. Views are my own, not Edgecore’s.

The same disclosure appears at the top of every post. A few more things worth being clear about:

Nothing here is a price, a roadmap, an availability date, a certification, or a promise from Edgecore or anyone else.
I don’t publish confidential material from my employer, its partners, or customers, and I don’t name customers or prospects without their written permission.
Product names are trademarks of their owners and are used here only to describe products. This site is personal and isn’t sponsored or endorsed by the companies I discuss. My employment at Edgecore is disclosed above.
Specs are linked to public pages. Check them yourself before you buy anything, from anyone.

Writing with LLM assistance

I use large language models (LLMs) to help draft and edit these posts. The thoughts, ideas and opinions are my own. Sources and open questions remain visible so readers can assess the claims.

About me

I spend my working week with people building and running GPU clusters: neoclouds, research and sovereign clusters, and enterprises. I don’t build the clusters myself. These posts are field notes from the person in the room, with links so you can check every claim.

About this blog

What: how AI clusters are networked with open Ethernet, written plainly.
How often: about one post a month. See upcoming topics.
Method: every post separates what’s known (public sources), what’s inferred, and what nobody has published.
Terms: I use the definitions in the glossary.
Corrections: if I get something wrong, I fix it and update the “last updated” date on the page.

AI cluster networking glossary

Sun, 01 Nov 2026 06:00:00 GMT

This glossary defines the terms I use on this blog for GPU cluster networks, in plain language and the way I use them in every post. Each term has its own heading, so you can link straight to it (for example, rail-optimized).

A note on naming: NVIDIA program and product names (NCP, Reference Architecture, SuperPOD, Spectrum-X) refer to specific NVIDIA things. When I describe a design that hasn’t been validated by NVIDIA, I call it a “pattern” or a “rail group,” never an RA.

Scale-up

GPUs sharing memory directly over NVLink and NVSwitch as “one big GPU,” inside a server (HGX = 8 GPUs) or a rack (NVL72 = 72 GPUs). It is not Ethernet.

NVLink domain

The set of GPUs on one scale-up fabric. I say “the NVLink domain,” not “the rack network.”

Scale-out / backend / east-west compute fabric

The RDMA network joining GPUs across servers, usually one NIC per GPU into rail leaves. NVIDIA’s NCP documents call it the CIN.

CIN (Cluster Interconnect Network)

The NCP name for the back-end fabric. NCP documents list InfiniBand (Quantum) or Ethernet (Spectrum-X) for it.

Frontend / north-south (converged)

The network on the DPU or front-end NIC ports: storage, in-band management, tenant and internet traffic. It is “converged” when storage rides on it. The NCP name is TAN.

TAN (Tenant Access Network)

NCP’s name for the front-end network. Always Ethernet; NCP’s IaaS table lists “Ethernet (Spectrum).”

Tenant network

A customer’s isolated overlay (a VPC) riding on the front-end, usually EVPN-VXLAN.

OOB / management (SMN)

A separate 1G Ethernet network to BMCs, switch management ports, PDUs and CDUs. NCP calls it the SMN and doesn’t name a vendor.

In-band management

Provisioning, Kubernetes or Slurm control, and monitoring traffic carried over the front-end. Not the same thing as out-of-band management.

Storage fabric

The network to file and object storage. It may be separate or converged onto the front-end. NVIDIA’s storage reference designs default to Spectrum SN5600 switches in the bill of materials.

Border / edge / DCI

The uplink out of the cluster to the internet, the WAN or other sites (sometimes called “scale-across”).

ToR (top-of-rack)

A switch in the rack. On NVL72 the only ToRs are the two management switches; back-end leaves sit in separate network racks.

Leaf / spine / super-spine

The tiers of a Clos fabric. A super-spine joins multiple spine groups.

Rail-optimized

GPU n of every server connects to leaf n, so same-rail traffic is one hop. Rails cut across servers, not racks.

Plane (dual-plane)

Independent, parallel back-end fabrics. Each GPU’s NIC ports are split across the planes for bandwidth and failure isolation.

Oversubscription

Downlink bandwidth divided by uplink bandwidth on a switch. Back ends are usually 1:1 (non-blocking); front ends can be oversubscribed.

SU (Scalable Unit)

A repeatable scale-out building block. NVIDIA defines it differently across its reference architectures; the term is not exclusive to NVIDIA. For non-NVIDIA designs I say “pod” or “rail group” instead.

POD / BasePOD / SuperPOD

“POD” is generic in NCP documents. DGX BasePOD and DGX SuperPOD are NVIDIA products built from DGX systems with NVIDIA networking only; the names don’t apply to other designs.

RA (Reference Architecture)

A documented design for a particular use case and set of assumptions. NVIDIA publishes specific RAs with their own prescribed components. An Edgecore architecture proposal does not establish conformance to one of those NVIDIA designs.

Enterprise RA

NVIDIA’s on-premises designs (for example HGX AI Factory and NVL72 AI Factory) with partner endorsement, built on Spectrum-X.

NCP (NVIDIA Cloud Partner)

An NVIDIA partner program for GPU clouds. It is not a switch certification; there is no such thing as an “NCP-certified switch.”

NCP Hardware Reference Design

The gated (partner login) design that NCP builds are expected to follow closely.

Deployment-specific agreement / exception

The only documented way a baseline NCP requirement is waived. The agreement must identify the affected requirement and approved exception.

Spectrum-X

NVIDIA’s Ethernet AI platform: Spectrum switches and SuperNICs working together for adaptive routing and congestion control. It is more than “NVIDIA Ethernet switches.”

Spectrum (SN5600/SN5610/SN6600/SN2201)

NVIDIA switch families. The SN2201 is the 1G management switch.

InfiniBand (Quantum)

NVIDIA’s non-Ethernet back-end option.

NIC / SuperNIC (ConnectX-7/8/9)

The data mover, one per GPU on the back end, using GPUDirect RDMA straight into GPU memory. ConnectX-8 on HGX B300 runs 2x400GbE.

DPU (BlueField-3/4)

A NIC plus Arm cores running its own OS. Used as the front-end and storage card for overlays, isolation and security. It offloads the host CPU, not the GPU.

BlueField-3 SuperNIC

BlueField silicon sold as a back-end NIC for early Spectrum-X deployments; not the DPU role.

RoCEv2

RDMA over Converged Ethernet, version 2. The classic recipe runs it on a lossless, tuned fabric (PFC plus ECN/DCQCN), Operator-specific alternatives include PFC-only designs with workload coordination and distinct multipath transports such as MRC. Those are not interchangeable configuration recipes.

PFC / ECN / DCQCN

Priority Flow Control, Explicit Congestion Notification and the DCQCN congestion control algorithm: the flow control and congestion signaling that keep RoCE traffic from dropping.

Adaptive routing / DLB / packet spraying

Per-packet or per-flow path balancing. Spectrum-X does it across switch and NIC; Broadcom Tomahawk switches offer dynamic load balancing (DLB).

NCCL

NVIDIA’s collective communications library (all-reduce, all-to-all) that rides the back-end fabric. AMD’s equivalent is RCCL.

Tomahawk (TH5 51.2T, TH6 102.4T, TH Ultra)

Broadcom Ethernet switch ASICs. Tomahawk 5 is 51.2 Tb/s; Tomahawk 6 is 102.4 Tb/s.

AIS800-64O / -64D / -32O / -32D

Edgecore Tomahawk 5 AI switches: 64x800G OSFP or QSFP-DD in 2RU, and 32-port 1RU versions. (Disclosure: I work at Edgecore.)

Jericho3-AI

A Broadcom scheduled-fabric ASIC (cells and credits), an alternative to a Tomahawk Clos fabric.

SONiC / Enterprise SONiC

SONiC is an open source network operating system built on SAI. Enterprise SONiC is Broadcom’s commercially supported distribution.

SAI

The Switch Abstraction Interface: an open API between the NOS and the switch ASIC. It is why SONiC is portable across hardware.

ONIE

The Open Network Install Environment: a whitebox installer that lets you load any compatible NOS onto a switch.

EVPN-VXLAN

The standard multi-tenant overlay on a BGP underlay. NCP inference provider builds require BGP with EVPN.

UALink / UALoE / ESUN / SUE

Open scale-up efforts. UALoE is UALink over Ethernet (used in AMD Helios). None of them can be used as NVIDIA GPU scale-up.

Operator vs tenant

The operator owns the cluster and buys the switches; the tenant (for example an inference provider) rents managed capacity such as Kubernetes. I say “inference provider,” not “ISP.”

AI Cloud Ready / Exemplar Cloud

NVIDIA validation and performance badges for clouds. They are not NCP tiers.

Deployment pattern

What I call a design that hasn’t been tested or validated as an NVIDIA reference architecture. Placing open Ethernet next to NVIDIA systems is a pattern, not an RA.

Non-blocking

Sun, 01 Nov 2026 06:00:00 GMT

Architecture Field Notes

Sun, 01 Nov 2026 06:00:00 GMT

Every post on this blog, newest first. Each one opens with a short answer, shows its last updated date, and links its sources.

Latest: Before the switches, name the networks.

Upcoming topics

Sun, 01 Nov 2026 06:00:00 GMT

These are the posts planned for the next six months, about one a month, from November 2026 to April 2027. Plans change; this page shows the latest version and its last updated date.

Month	Planned post
Nov 2026	Open Ethernet AI fabrics outside the NVIDIA box (published)
Nov 2026	Everything around the GPU fabric: frontend, storage and OOB networks of a B300 cluster (published)
Nov 2026	NVIDIA network terminology: now drafted in Field note 03
Jan 2027	Drawing a rail group honestly: 32 servers, 8 rails, every cable
Feb 2027	The 90% problem: optics, breakouts and dust
Mar 2027	RoCEv2 on Tomahawk 5: what you tune, and what nobody has published (PFC only vs DCQCN vs lossy MRC/UEC)
Apr 2027	Mailbag and corrections: six months of reader questions

In reserve: 102.4T switching and Tomahawk 6, from public specs only.

Have a question you want answered in the April mailbag? See the contact note on the About page.