Disclosure
I’m a sales director at Edgecore Networks, which sells open networking switches, including some of the hardware discussed below. This is my personal blog. Views are my own, not Edgecore’s.
I use large language models (LLMs) to help draft and edit these posts. The thoughts, ideas and opinions are my own.
Contents
TL;DR
- Treat each NCP placement as its own design decision. NVIDIA’s NCP guide lists the Cluster Interconnect Network as InfiniBand (Quantum) or Spectrum-X, and the Secure Management Network as “Ethernet (1GbE)” with no vendor. The Tenant Access Network is listed as “Ethernet (Spectrum)”, and public documentation does not establish acceptance of a substitute switch.
- Count frontend links separately from usable throughput. Using NVIDIA’s HGX B300 interfaces, a 32-node HGX B300 cluster puts 64 x 400G frontend links (25.6 Tb/s) next to 204.8 Tb/s of ConnectX-8 bandwidth. A pair of 64-port 800G switches covers it with ports to spare. The hard part is BGP/EVPN and the BlueField-3 DPU, which NET-1 makes a MUST for the cited GB300 NVL72 provider profile.
- Storage forces a choice before cabling: RoCE or TCP. If storage rides the converged frontend with RDMA, you are running a lossless class next to tenant TCP on the same switches, and that needs testing.
- OOB is bigger than everyone draws it. NVIDIA counts six 1G management connections per HGX B300 server: 192 ports for 32 servers before a single switch, PDU or console port.
Why I’m writing this
In my last post I promised the frontend and storage network next. The most common reply was fair: “Our build has to follow NCP, so the GPU fabric is Spectrum-X. Is anything left for open networking?”
Yes, and it’s not small. A GPU cluster has four networks, with distinct requirements for NVLink scale-up and the CIN. Frontend and management substitutions still need deployment acceptance. This post examines those candidate placements and storage, using a 32-node B300 design I worked on: 32 HGX B300 servers, 256 GPUs, GPU fabric held constant as Spectrum-X.
First, the boundary
NVIDIA’s NCP Part 1 guide has a “Data Centre Networks” table. It lists the CIN as “InfiniBand (Quantum) or Ethernet (SpectrumX)”, the TAN as “Ethernet (Spectrum)”, and the Secure Management Network as “Ethernet (1GbE)” with no vendor. The data center architecture page says “The TAN and SMN are always Ethernet”.
So I won’t tell you NCP “defines nothing” for the frontend. It names Spectrum in a reference table. The OOB row names no vendor, but this does not establish acceptance of an Edgecore substitution. The NCP Hardware Reference Design is behind a partner login, and I haven’t read it. If your build has to claim compliance, get the frontend and OOB switch choice approved in writing.
One more honest note. NVIDIA’s HGX B300 Enterprise RA uses SN5600 switches for the converged fabric and SN2201 for OOB. Swap those and you are following its topology, not the RA.
flowchart TB subgraph Locked["Named by NVIDIA for NCP builds"] BE["GPU fabric (CIN)<br/>Spectrum-X or Quantum"] end subgraph Open["Where open Ethernet can fit"] FE["Frontend / converged pair<br/>2 x 64-port 800G switches"] OOB["OOB / SMN<br/>48-port 1G switches"] end SRV["HGX B300 server<br/>8 GPUs"] ST["Storage servers"] BR["Border / internet"] CON["Console server"] SRV -- "8 x ConnectX-8" --> BE SRV -- "BlueField-3 DPU, 2 x 400G" --> FE ST --- FE FE --- BR SRV -- "BMCs, 1G" --> OOB FE -. "mgmt port" .-> OOB BE -. "mgmt port" .-> OOB CON -. "serial" .-> FE CON -. "serial" .-> OOB
One server, three networks you can touch. NVLink scale-up and the storage system’s internal network are not shown.
The frontend: small pipes, hard rules
What it carries
NVIDIA’s HGX B300 RA calls it the CPU Converged (North/South) network and says it “is used for node communications with compute, storage, in-band management, and end-user connections.” On NCP, in-band management, storage, and tenant traffic all ride it.
The sizing math for 32 nodes
Each B300 server in that RA has “One NVIDIA BlueField-3 B3240 dual port 400 GbE DPU”, connected “using two 400 GbE ports to two separate switches”. These are port line rates, not a guarantee that one BlueField-3 delivers 800 Gb/s aggregate host throughput. NVIDIA’s card-limit descriptions vary by page and attachment profile; confirm the selected card and configuration. See the NVL72 networking hardware description.
So:
- 32 servers x 2 = 64 x 400G links, 32 to each switch, 25.6 Tb/s total.
- The GPU fabric: 32 servers x 8 ConnectX-8 at 800G (configured as 2x400G) = 204.8 Tb/s. That is a 1:8 ratio of nominal link rates, not a measure of usable BlueField throughput. NVIDIA’s own 4-server block shows the same ratio: 3.2 Tb/s converged next to 25.6 Tb/s compute.
The design I worked on used two AIS800-64O switches for the frontend. Each has 64 OSFP800 ports, each runnable as 2 x 400G. The 32 server links on each switch take 16 ports. That leaves 48 ports per switch for storage, the link between the pair, and uplinks to the border [EDIT: border uplink count and speed for the example]. NVIDIA’s own 32-node page calls for a “cost-efficient converged two-switch fabric”, so the shape matches. At 64 nodes the design adds a frontend spine pair and a separate storage leaf pair.
The hard parts
NET-1 makes the DPU mandatory in the cited GB300 NVL72 profile. NVIDIA’s inference provider requirements, written for NCPs building GB300 NVL72 clusters, make NET-1 a MUST: “North-south through the BlueField-3 DPU enforcing zero trust and host multi-tenancy; carries control plane, storage, external connectivity”. NET-5 makes BGP with “IPv4 and EVPN/VPNv4 address families” a MUST, and NET-6 says multitenancy SHOULD use BGP underlay, EVPN, VXLAN overlay and DPU ACLs.
My inference: if the DPU terminates the overlay, the leaf is mostly a BGP underlay router. If the leaf is the VTEP, its EVPN implementation is the product. Know which one you have before you pick a NOS.
Dual homing. Two links to two switches means MLAG, EVPN multihoming, or routing to the host. Each fails differently. Pick one and pull cables during testing.
Storage: converged until it isn’t
What it carries
Checkpoints, datasets, model weights. The NCP guide calls storage the TAN’s “primary consumer”. In the 32-node design, four storage servers dual home to the same frontend pair [EDIT: storage vendor, NIC speed and ports per storage server]. At 64 nodes storage gets its own storage fabric leaf pair, uplinked to the frontend spines.
RoCE or TCP
This is the decision that drives switch config. GPUDirect Storage enables “a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU.” The RDMA flavors of that path want a tuned RoCEv2 class with PFC and ECN on the frontend. TCP-based storage needs none of that.
On a converged frontend, RoCE storage means a lossless class sharing buffers with lossy tenant traffic. That works, but it’s a design, not a checkbox. Ask the storage vendor which transport they tested at your scale, then configure exactly that. If it’s TCP, keep the frontend lossy and simple.
OOB: the network everyone undercounts
What it carries
NVIDIA’s RA connects “BMC ports of the server nodes”, “BMC ports of the Bluefield-3 DPU and SuperNICs” and “OOB management ports of switches”, and says to scale it with a 25 or 100 Gbps spine layer. Add PDUs, CDUs and the console server.
The sizing math, and my own miss
The first pass of the 32-node design counted about 54 OOB ports (one BMC per server, 18 switch management ports, 4 storage servers) and two 48-port 1G switches. Then I read NVIDIA’s HGX B300 page again: “4 servers, each with 6x 1Gb/s connections”.
- 32 servers x 6 = 192 ports
- 18 data switches in that design + 4 storage servers = 22 more
- Total 214, before PDUs, CDUs or the console server
A 48-port 1G switch like the EPS201 (AS4630-54TE), with 48 x 1G RJ45, 4 x 25G and 2 x 100G ports, means five switches, not two [EDIT: confirm the actual OEM management-port schedule, then add infrastructure and spare capacity]. The 25G and 100G uplinks give you the spine layer NVIDIA describes.
Console
Every SONiC switch should have a serial console reachable when its management port isn’t. That’s how you recover a box stuck in ONIE or a bad upgrade. Count consoles like you count BMCs: 18 data switches plus the OOB switches themselves [EDIT: console server model and port count used].
What to validate, by network
| Network | What it carries | 32-node sizing | What bites people | What I’d validate |
|---|---|---|---|---|
| Frontend (TAN) | In-band mgmt, tenant traffic, internet | 64 x 400G from BlueField-3, 2 switches | BGP/EVPN behavior, dual homing failover | EVPN with your DPU overlay design; pull one link and one switch under load |
| Storage | Checkpoints, datasets, weights | 4 storage servers dual homed (converged) | RoCE lossless class next to TCP | Your storage vendor’s exact transport; checkpoint write at full scale |
| OOB (SMN) | BMCs, switch mgmt, PDUs, CDUs | 214+ x 1G, 5 x 48-port switches | Six ports per server, not one | Every BMC reachable; serial console to every switch |
| GPU fabric (CIN) | NCCL between GPUs | 256 x 800G ConnectX-8 | Not this post | Spectrum-X on NCP builds |
What’s known, what’s inferred, what nobody has published
- Known (public docs): NCP lists the CIN as Quantum or Spectrum-X, the TAN as “Ethernet (Spectrum)”, and the SMN as 1GbE with no vendor. NET-1 makes BlueField-3 north-south a MUST for NCPs hosting inference providers on GB300 NVL72. NVIDIA’s HGX B300 RA uses one dual port 400 GbE BlueField-3 per server and six 1G OOB connections per server.
- Inferred: frontend and OOB switch substitutions remain proposed; public documentation does not establish their acceptance. When the DPU terminates the overlay, the frontend leaf’s job gets simpler.
- Not established by the sources reviewed here: the gated NCP Hardware Reference Design’s frontend and OOB switch rules. The tables in NVIDIA’s HGX guide also disagree on converged switch counts at larger sizes, so I haven’t quoted them. If you’ve seen either settled, I’d like to compare notes.
FAQ
Can the frontend network of an NVIDIA NCP build use non-NVIDIA switches?
Public NCP text doesn’t exclude a non-NVIDIA frontend switch, but its reference table lists the Tenant Access Network as Ethernet (Spectrum), and the gated NCP Hardware Reference Design wasn’t public for me to check. Get it in writing for your specific build.
Is a BlueField-3 DPU required on the frontend?
For NCPs hosting inference providers on GB300 NVL72, yes: NVIDIA’s NET-1 requirement makes north-south through the BlueField-3 DPU a MUST, carrying control plane, storage and external connectivity. NVIDIA’s HGX B300 Enterprise RA also uses one BlueField-3 B3240 dual port 400 GbE DPU per server.
How much frontend bandwidth does a 32-node HGX B300 cluster need?
In NVIDIA’s HGX B300 design each server has two 400 GbE frontend ports to two separate switches, so 32 servers bring 64 x 400G links, or 25.6 Tb/s. As a sum of port line rates, that is one eighth of the 204.8 Tb/s the eight ConnectX-8 SuperNICs per server put on the GPU fabric.
How many out-of-band ports does a 32-node B300 cluster need?
NVIDIA’s HGX B300 Enterprise RA counts six 1 Gb/s management connections per server, so 32 servers need 192 ports before you add switches, storage, PDUs and console servers. That is at least five 48-port switches once the additional equipment in this example is included.
The upshot
Even on a build where NVIDIA names the GPU fabric, a 32-node B300 cluster still has a frontend pair, a storage decision and a few hundred OOB ports that public NCP text doesn’t lock to one vendor. Those networks are smaller than the GPU fabric, but they are where clusters break at 2am.
Next up: drawing a 32-server rail group honestly, every cable. If you’re sizing one of these and want to compare notes, my inbox is open [EDIT: add contact email or link]. No slide deck, I promise.
Elliot
Sources
- NCP Software Reference Guide, Data Center Architecture: “Networking” and compute tray table, NVIDIA, accessed 2026-11-02
- NCP Software Reference Guide Part 1, Infrastructure as a Service: “Data Centre Networks” table, NVIDIA, accessed 2026-11-02
- Inference Provider Platform Requirements on GB300 NVL72 for NCPs: “Networking and Multi-tenancy”, NET-1 to NET-8, NVIDIA, accessed 2026-11-02
- HGX AI Factory Enterprise RA, Networking Logical Architecture: “CPU Converged (North/South) Network”, “Enterprise RA Scalable Unit (SU)”, “Out-of-Band (OOB) Management Network”, Table 5, “32 Nodes with 256 NVIDIA HGX B300 SXM GPUs”, NVIDIA, accessed 2026-11-02
- GPUDirect Storage Overview Guide: 1.1 Introduction, NVIDIA, accessed 2026-11-02
- AIS800-64O product page, Edgecore Networks, accessed 2026-11-02
- EPS201 (AS4630-54TE) product page, Edgecore Networks, accessed 2026-11-02
Profile boundary: General storage designs can use TCP or RDMA. The cited GB300 NVL72 provider profile specifically requires RDMA and a certified storage partner under STO-2/3. An Edgecore switch placement remains proposed until the hardware/storage design and deployment acceptance are checked. NVIDIA provider requirements.