Open Ethernet
Design Guide

Every NVIDIA term above is taken from the page in brackets, with NVIDIA's own spelling. VERIFIED [S1 "Networking"; S2 "Data Centre Networks"; S6 "Networking Logical Architecture"; IRA "Data Center Architecture" and "Networking"]. The inference provider requirements use a fourth pair of words for the same split: a "north-south plane on the BlueField-3 DPU" and an "east-west plane on the back-end RDMA fabric" VERIFIED [IPR "Networking and Multi-tenancy"]. The two ownership columns are this document's design, explained in section 2.

S1 adds one rule that holds in every NCP build: "The TAN and SMN are always Ethernet; the CIN can be configured as either Ethernet or InfiniBand, and NVLink is a proprietary NVIDIA standard." VERIFIED [S1 "Networking"].

NVIDIA's POD is also not a Kubernetes pod, as S1 itself notes. VERIFIED [S1 "Data Center View" note].

• Levels. The inference provider requirements follow RFC 2119: MUST and MUST NOT are mandatory, SHOULD is a strong recommendation, MAY is optional, and INFO rows "place no obligation on the NCP". VERIFIED [IPR "Purpose" tip]. The AI Clouds requirements use shall and must for mandatory and should for recommended. VERIFIED [R24 "Normative Language"].
• Tables bind, prose explains. In the inference provider requirements, "The tables are the normative part of this document. The prose explains why a requirement exists ... it does not itself impose requirements." VERIFIED [IPR "Purpose" tip]. This matters in section 10, where one prose sentence names Spectrum switches and no table row does.
• IDs collide between documents. The inference provider requirements number their network rows NET-1 to NET-8. The AI Clouds requirements v2.4 number theirs NET01 to NET06, and they mean different things: IPR NET-2 is the east west fabric, R24 NET02 is the NVLink Domain API. VERIFIED [IPR; R24 "Network Transport and Fabric Visibility"]. This document always names the document with the ID.
• Scope. R24 is the bar for "NVIDIA Cloud Partners (NCP) delivering GPU capacity to NVIDIA", covering what is "needed to run NVIDIA DGX Cloud, expanding on the NVIDIA hardware reference design". VERIFIED [R24 "Purpose and Intent"]. The inference provider requirements apply to "an NVIDIA Cloud Partner (NCP) building a GB300 NVL72 compute cluster" for inference service providers. VERIFIED [IPR title and "Purpose"].
• Gated documents decide some questions. The GB300 NVL72 Reference Design (IPR RD-2), the NCP storage reference design (IPR RD-3, STO-1) and the hardware reference for each inference profile are on NVIDIA's partner portal. VERIFIED [IPR "Reference Designs"; IRA "Physical Architecture"]. We haven't read them, and this document marks every question they decide as UNKNOWN.

• VERIFIED: a primary source in section 19 says it (document plus section, table or page).
• INFERRED: it follows from verified facts, and the arithmetic or reasoning is shown so anyone can check it. INFERRED means the math is right. It doesn't mean anyone has built it or NVIDIA has approved it.
• UNKNOWN: no source we hold settles it. Section 18 says what would.

Why Mode A can exist at all. NVIDIA's Inference RA hands the fabric choice to the hardware profile: "Treat RDMA, GPUDirect RDMA, dual-plane networking, BlueField use, and separate compute fabric as profile decisions instead of universal dependencies." VERIFIED [IRA "Architecture Decision Guardrails"]. For every profile it then says to "Confirm implementation details with the NVIDIA-approved, access-controlled hardware reference for the selected profile." VERIFIED [IRA "Physical Architecture"]. So the Inference RA doesn't call the fabric open, it defers. The only public requirement that names InfiniBand or Spectrum-X is IPR NET-2, which belongs to the GB300 NVL72 NCP profile. INFERRED. Mode A is for clusters that aren't built to an NVIDIA profile at all. Those clusters can still run NVIDIA's inference software, which IRA lists with public GitHub sources (Dynamo, NIXL, ModelExpress, KAI Scheduler, Grove, AIPerf). VERIFIED [IRA "Reference Component Version And Source Snapshot"]. How that software performs on an Edgecore fabric is ours to prove, not NVIDIA's to vouch for (sections 8 and 15).

• R24 never names InfiniBand, Spectrum, Quantum, ConnectX or a switch vendor for any network. The two NVIDIA networking names it does use are BlueField, in STG06 for BlueField JBOF storage, and "UFM event logs" in its list of required logs, which presumes an InfiniBand fabric manager where one exists. VERIFIED [R24 STG06, "Logs"].
• R24 says a deployment agreement "does not waive or weaken a baseline requirement in this guide unless it expressly identifies the affected requirement and the approved exception." VERIFIED [R24 "Relationship to Deployment-Specific Requirements"]. That clause governs R24's own baseline, and R24 has no fabric vendor baseline to waive. INFERRED. The v1 draft of this document cited it as the gate on the east west fabric; that was wrong.
• The vendor names come from the S2 reference table (CIN "InfiniBand (Quantum) or Ethernet (SpectrumX)", TAN "Ethernet (Spectrum)"), from IPR NET-2, and from the gated GB300 NVL72 Reference Design that IPR RD-2 makes a MUST. VERIFIED [S2 "Data Centre Networks"; IPR NET-2, RD-2].

The practical difference: NVL72 moves the hardest GPU traffic onto NVLink inside the rack, so its east west fabric works more like an inference interconnect for KV cache and model movement. IRA describes the CIN as carrying "east-west model serving traffic, distributed inference coordination, model data movement, and high-volume cache or tensor movement". VERIFIED [IRA "Networking"]. HGX leans on the fabric harder, which is where Edgecore's 32, 64 and 256 node design lives. INFERRED.

1. One switch model for every 800G role. AIS800-64O for backend leaf and spine, frontend leaf and spine, and storage leaf. Fewer spares, one NOS image, one optic family. VERIFIED as design [E4 slide 3]. Cisco does the same with the N9364E-SG2-O. VERIFIED [C1 "Cisco N9364E-SG2-O"].
2. Rail optimized backend. NIC n of every node lands on leaf n of its rail block, so same rail traffic is one hop. VERIFIED as design [E4 slide 4; N1 section 4.1].
3. Nonblocking 1:1 backend. Leaf uplink bandwidth equals downlink bandwidth. VERIFIED as design [E4 slide 3].
4. Separate networks for separate jobs. Backend, frontend, storage and OOB are separate fabrics, matching NVIDIA's four networks [S1] and Cisco's three fabric split plus storage leaves [C1 "Overview"].
5. Cable the full block. If fewer nodes are bought, cable the full block's leaves and leaf to spine links, as NVIDIA advises: "the fabric should be designed to support the full SU, including leaf switches and leaf-spine cables". VERIFIED [N1 Chapter 4 intro, p.14].
6. Plan the top tier on day one. If the cluster may pass the two tier ceiling, choose dual plane or three tier at the start. Cisco gives the same rule: customers who will scale beyond 8K GPUs "should deploy the compute network with a three-tier topology from the beginning". VERIFIED [C2 "East-West compute network"].
7. 800G fabric links where possible. Prefer 1 x 800G ECMP links over 2 x 400G breakouts: more bandwidth per link tolerates hash collisions better. INFERRED; confirm on the customer's traffic in section 15 test 6.
8. Optical in the 800G fabrics. No copper, for link integrity. VERIFIED as design assumption [E4 slide 2].
9. Open and swappable software. ONIE on every switch, a SONiC based NOS, standard BGP and EVPN, no proprietary controller. VERIFIED [E1 "Software"].
10. Don't touch NVLink. Scale up stays exactly as NVIDIA builds it, in both modes and on both platforms.

The model doesn't tell you the network, the role does. ConnectX-8 is the backend card on B300 and BlueField-3 is the frontend card. VERIFIED [S6 Table 5]. In Mode A a ConnectX NIC is acceptable on the frontend; the generic Edgecore design draws "CX7/BF3". VERIFIED as design [E4 slide 4].

One rail block is 32 HGX B300 nodes and the 8 AIS800-64O backend leaves that serve them, leaf n carrying GPU rail n of all 32 nodes. Each leaf uses 32 ports down (one 800G port per node) and 32 ports up. The rail block is the unit of growth, of tenant allocation and of failure isolation for same rail traffic. Spines are shared across rail blocks. VERIFIED as design [E4 slides 3 to 6, which call it "Scalable Units of 32 servers"].

Why 32: a 64 port 800G leaf split 1:1 gives exactly 32 node ports, so one leaf per rail fills to 100 percent at 32 nodes. INFERRED [E1 port count; E4 slide 3].

Why a new name: NVIDIA's HGX Enterprise RA uses SU for 4 nodes [S6], DGX SuperPOD for 64 systems [N1], and Cisco for 32 nodes [C2]. "POD" is NVIDIA's 1,152 GPU building block [S1]. Rail block collides with none of them. The Edgecore design deck still says "Scalable Units"; read that as rail blocks.

Assumptions: 8 GPUs and 8 ConnectX-8 per node at 800G per GPU; 1 frontend card per node at 2 x 400G, dual homed; single plane backend; every 800G switch an AIS800-64O. VERIFIED as design [E4 slides 2, 3; S6 Table 5]. Counts are from the Edgecore design deck and the arithmetic is shown. VERIFIED [E4 slides 3 to 9].

At 32 nodes the minimum is 4 spines (256 uplinks / 64 ports). The design keeps 8 spines half populated so a second rail block adds leaves only, and because "an underutilized fabric will help reduce congestion risks". VERIFIED as design choice [E4 slide 3, companion proposal deck notes].

Per GPU backend bandwidth is the same 800G in Cisco's dual plane design, NVIDIA's dual plane HGX design and Edgecore's single plane design. INFERRED from the cited tables and [S6].

NVIDIA's own HGX B300 design is dual plane at every size it documents: "Each GPU has individual 400Gb connection to each plane via 800Gb port broken out to 2x400Gb". VERIFIED [S6 "32 Nodes", "64 Nodes", "128 Nodes"]. Cisco also recommends it "to avoid a single point of failure". VERIFIED [C1 "Compute (Node East-West) Network"]. On AIS800-64O the switch counts at 64 and 256 nodes are unchanged (each plane has half the leaves and half the spines, and leaf ports run 2 x 400G toward nodes); at 32 nodes the minimum becomes 8 leaves and 4 spines. INFERRED arithmetic. NVIDIA's dual plane failover relies on the ConnectX-8 "onboard plane load balancer unit" and "NVIDIA Route software". VERIFIED [N1 section 4.1]. Whether that works behind a non NVIDIA switch is UNKNOWN (gap G3).

VERIFIED as design counts [E4 slides 7 to 9]. The optic specifications are UNKNOWN here because no Edgecore transceiver datasheet is in the library. Two specific unknowns: whether the NIC side optic has the riding heat sink form the ConnectX-8 cage needs (Cisco uses a distinct OSFPR part on ConnectX-8 [C1 Table 1, VERIFIED]), and ConnectX-8 interoperability of the Edgecore optic. NVIDIA recommends a "Dual-ported optic ... to simplify breakout at NVIDIA ConnectX-8 OSFP port". VERIFIED [S6 "64 Nodes"]. See gap G4.

This section applies only to GB300 NVL72 bought outside an NCP engagement. Inside an NCP inference build, the CIN is InfiniBand or Spectrum-X, a MUST. VERIFIED [IPR NET-2]. No public NVIDIA document forbids an Ethernet CIN on a non NCP NVL72 build, and none blesses it either. INFERRED.

Assumptions: every ConnectX-8 port runs 1 x 800G; single plane; each leaf runs 1:1 with 32 ports down and 32 up; a 64 port spine. All of it INFERRED arithmetic from [S1] and [E1].

Why 16 racks is one POD: S1 sizes a GPU POD at "up to 1152 GPUs", and 1,152 / 72 = 16. VERIFIED [S1]; INFERRED arithmetic. Why 28 racks is the ceiling: a 64 port switch in two tiers serves 64 x 64 / 2 = 2,048 endpoints, and 28 racks (2,016) is the most whole racks under that. INFERRED. Beyond that, the options are the same as on HGX (section 7). A dual plane build with each ConnectX-8 at 2 x 400G halves the per plane radix use and doubles the ceiling in endpoints. INFERRED, and the ConnectX-8 plane failover behind a non NVIDIA switch is UNKNOWN (gap G3).

• Rail mapping. How NVIDIA maps the 72 ConnectX-8 ports per rack to rails is in the gated GB300 NVL72 Reference Design. UNKNOWN (gap G16). The leaf and spine counts above hold for any mapping that keeps 32 ports down per leaf. INFERRED.
• NVL72 traffic mix on Tomahawk 5. NIXL KV transfer and Dynamo disaggregated serving on a RoCEv2 fabric with Tomahawk 5 and ConnectX-8: no published result. UNKNOWN (gap G1, G2).
• Management port count per rack. Beyond the 18 compute tray ports, UNKNOWN (gap G19).

For inference, IPR gives one east west number any fabric can be held to: pull weights "fast enough to hydrate a cold node (~400 GB 405B FP8) in under 60 s". VERIFIED [IPR KUB-38, a SHOULD]. That's about 6.7 GB/s, or about 53 Gbps per node, against 800G per GPU on this fabric. INFERRED arithmetic. The hard part is the transfer software and the lossless profile, not the bandwidth.

256 HGX nodes is the largest two tier, single plane, 800G fabric with AIS800-64O: 64 leaves x 32 uplinks = 2,048 = 32 spines x 64 ports, every spine port used. INFERRED arithmetic [E1; E4 slide 6]. The same ceiling is 28 NVL72 racks (section 6.2). Three ways to grow:

Worked example for option B at 512 HGX nodes (4,096 GPUs): 4 pods, 128 leaves, 128 pod spines, 64 super spines (4 pods x 32 spines x 32 uplinks = 4,096 / 64), 320 switches. INFERRED arithmetic. Cisco builds the same shape: "an SU-group of four Scalable Units" with super spine groups. VERIFIED [C2].

In Mode B, the super spine is part of the CIN and stays NVIDIA's. A Tomahawk super spine over NVIDIA leaves forfeits Spectrum-X features on paths that cross it and isn't addressed by any NCP page we read. INFERRED [L Q13].

The public result is Edgecore's AIS800-64O and Enterprise SONiC test on AMD MI300X with Broadcom Thor 2 NICs: "Nonblocking (1:1): Up to 35.6% improvement" in collective performance with congestion aware packet spraying, with up to 26.7% at 2:1. VERIFIED [E9]. It's AMD GPUs and Broadcom NICs, not ConnectX-8, so it shows what the switch and NOS can do and doesn't stand in for the B300 validation in section 15.

• Tuning method: run PFC only for a baseline, enable ECN and watch queue depth, set thresholds from observed buffer use so the mark rate doesn't hit 100 percent too early, then adjust mark probability. INFERRED (standard RoCEv2 practice).
• PFC alone reacts to congestion faster than ECN but can deadlock, which is why the watchdog stays on and ECN is added. INFERRED.
• Thresholds must be tuned on the customer's real traffic. INFERRED.

A validated Tomahawk 5 plus ConnectX-8 RoCEv2 lossless profile (PFC, ECN thresholds, buffer profile, DCQCN parameters) is UNKNOWN [L Q24]. It's the deciding item for Open Ethernet with Edgecore on both platforms and the first step of the validation plan (section 15).

The true sentence for Mode B: NCP's published normative requirements say what the frontend, storage, management and edge networks must do, but the public tables don't name who builds them. The reference tables and one line of IPR prose describe Spectrum for the frontend, and the gated GB300 NVL72 Reference Design decides. So every Mode B deal needs the switch choice confirmed in writing for that build. INFERRED from the sources below.

• NVIDIA owned: the NVLink domain and NVLink switch trays [S1], the CIN (InfiniBand or Spectrum-X, IPR NET-2 MUST), BlueField-3 on every tray (IPR NET-1 MUST), ConnectX-8. VERIFIED.
• Tenant overlay: owned by the BlueField DPU and NVIDIA Infra Controller. The DPU runs "DOCA/HBN to terminate tenant overlays at the edge", and Infra Controller isolates tenants on Ethernet "via HBN". VERIFIED [S2]. So the Edgecore switch is the BGP underlay, not the VTEP. INFERRED.
• Edgecore candidates: TAN underlay leaf and spine, storage leaves, SMN, and Core POD switching. Border and NVIDIA POP links only where the gated edge appliance rules allow (section 10.4).

The HGX Enterprise RA, which isn't an NCP document, does name SN5600 for the converged fabric and SN2201 for OOB [S6 Table 5], and the S2 reference table names Spectrum for the TAN.

All VERIFIED as requirements in [R24]. How each is met on Edgecore is INFERRED unless stated.

• NET01 Backend Switch Fabric API. Per node, the switch path to the core with "a unique, stable identifier" per switch. It "applies to north-south, east-west, and NVLink networks, but not to the management network", so Edgecore TAN switches must appear in it. A "switch" there "may represent a physical switch or a logical connectivity domain".
• CNP08 Stable Identifiers for "nodes and switches".
• SDN09 Observability. Tenant visible telemetry on "degraded states, errors, drops, utilization, and latency".
• Network telemetry across every network domain R24 lists (north south, east west, management, NVSwitch fabric and host network), so the Edgecore switches feed the north south and management parts, delivered within 120 seconds, with "Native OpenTelemetry Protocol delivery" preferred. Switch syslogs and kernel logs are on the required log list.
• SEC13 "Encryption and mutual authentication for all east-west and north-south network traffic." Where this is enforced (host, DPU or switch) isn't stated. UNKNOWN for the Edgecore role. Don't claim switch encryption.
• SEC16. The NCP must name its edge enforcement technology "and its specific placement in the packet path".

• IPR prose names NVIDIA switches. "Host networking on the BlueField DPUs and the network appliances, the Spectrum Ethernet switches and Quantum InfiniBand switches, keep tenants isolated end-to-end." The preceding paragraph puts the east west plane on "InfiniBand or Spectrum-X". VERIFIED [IPR "Networking and Multi-tenancy"]. IPR's own tip says prose "does not itself impose requirements". VERIFIED. Acknowledge the sentence; don't hide it.
• The S2 table names Spectrum for the TAN. VERIFIED [S2]. Our position is "not excluded by public normative text", never "NCP leaves the frontend open".
• RD-2 is a MUST and it's gated. "Adhere to the GB300 NVL72 Reference Design ... server spec, network cabling." VERIFIED [IPR RD-2]. Its switch rules are UNKNOWN.
• The edge is gated too. VERIFIED [IPR "Reference Designs"].
• Management stack. S2 describes NVIDIA's network management as NMX for NVLink, UFM for InfiniBand and "NetQ provides visibility for Spectrum switches running Cumulus". VERIFIED [S2]. A SONiC switch sits outside NetQ, so the NCP's own tooling has to cover it. UNKNOWN how NCPs handle this today (gap G17).
• Not every back end row is vendor locked, and it's fair to say so. IPR NET-3 (SHOULD, NIXL KV transfer and weight P2P) asks only for an "East-west back-end RDMA fabric", and NET-8 (MUST) says "Label InfiniBand and RoCE accurately". VERIFIED [IPR]. That doesn't open the NCP CIN, because NET-2 still binds it. INFERRED.

Public NCP pages describe NVL72 [S1, IPR]. For HGX B300, the HGX inference profile defers to a gated hardware reference [IRA "Physical Architecture"], so the honest yardstick is the HGX Enterprise RA [S6].

• The ERA builds every fabric on Spectrum-X switches (SN5600) and OOB on SN2201. VERIFIED [S6 "Spine-Leaf Networking", Table 5]. Replace the converged or OOB switches with Edgecore and the build follows the ERA's topology, not the ERA. INFERRED.
• Edgecore's frontend counts match NVIDIA's converged fabric counts: 2 switches at 32 nodes, 6 at 64 nodes. NVIDIA's Table 7 shows 2 leaves at 32 nodes, 4 leaves plus 2 spines at 64, and 8 leaves plus 4 spines at 128. VERIFIED [S6 Table 7; E4 slide 3]. NVIDIA's own 64 node page conflicts with Table 7: it says "Cost-efficient converged two-switch fabric" [S6 "64 Nodes"]. We follow Table 7, and the match holds only against Table 7.
• NVIDIA's bandwidth floors: "minimum 25Gb bandwidth per GPU" to the customer network and "minimum 12.5Gb bandwidth per GPU" for storage. VERIFIED [S6 "32 Nodes"].

All INFERRED arithmetic from [S1] and [E1].

• TAN demand. 36 x 200G BlueField-3 ports per rack, 576 per 16 rack POD. AIS800-64O ports break out to "4 x 200 GbE" [E1, VERIFIED], so a POD needs 144 switch ports of 4 x 200G, before Core POD and storage. INFERRED (576 / 4). Example at 1:1: 6 frontend leaves as 3 dual homed pairs and 3 to 4 spines per POD. INFERRED.
• Lane rate. The AIS800-64O supports 4 x 200G on 100G PAM4 lanes and 2 x 200G on 50G PAM4 lanes [E1, VERIFIED]. Which one the BlueField-3 port and optic need is UNKNOWN (gap G18).
• SMN. At least 18 x 1G per rack from compute trays [S1], so at least 288 per 16 rack POD, at least 6 x 48 port switches before NVLink switch trays, power shelves, cooling and switch management ports. INFERRED.

VERIFIED as design [E4 slides 3, 5 to 9]; per GPU figures INFERRED (25.6 Tbps / GPUs). The 256 node design lands on NVIDIA's "minimum 12.5Gb bandwidth per GPU" for storage [S6], and the 64 node design exceeds it. The deck labels frontend and storage counts as "estimates, before final storage sizing" [E4 slide 3].

Multitenancy on the frontend runs a BGP underlay with BGP EVPN and VXLAN, matching Cisco's VRF model [C2 "Multitenancy"], NVIDIA's SuperPOD Ethernet fabric "segregated with VXLAN and EVPN" [N1 section 4.4], and IPR NET-5 and NET-6. VERIFIED. Where a BlueField DPU terminates the overlay (always in Mode B), the leaf is mostly a BGP underlay router; where the leaf is the VTEP, its EVPN implementation is the product. INFERRED.

• Attachment. At 32 HGX nodes storage servers dual home to the two frontend switches; at 64 and 256 nodes to a dedicated storage leaf pair uplinked 16 x 800G each to the frontend spines. VERIFIED as design [E4 slides 4 to 6]. NVIDIA attaches storage at "a 1:1 port to uplink ratio". VERIFIED [N1 section 4.3].
• RDMA or TCP. For a GB300 NVL72 NCP inference build, storage MUST use RDMA [IPR STO-2], and GPUDirect Storage is a SHOULD [IPR STO-5]. VERIFIED. That means a lossless class on the storage path. Outside an NCP, the storage vendor's tested transport decides.
• Protocol, not product. Configure the storage path to the RoCEv2 QoS profile Cisco's storage RA requires of any switch: RDMA on DSCP 26, CNP on DSCP 48, PFC on queue 3, CNP on queue 6, no global link level flow control. VERIFIED [C4 p.5].
• Performance targets. NVIDIA's guideline per 64 node SuperPOD SU: "Good" 80 GBps aggregate read and 40 GBps write, "Better" 250 and 124 GBps. VERIFIED [N1 Table 5.2]. Scale to the node count; the storage system sets the achieved number.
• Partners. Any storage system with standard Ethernet and RoCEv2 clients. NVIDIA-Certified Storage status for a storage system on an Edgecore fabric: not documented for Edgecore. In Mode B, IPR STO-3 requires certified storage platforms. VERIFIED [IPR STO-3].
• Storage internal network. Where the storage system needs its own back end, put those ports in a provider only VRF. VERIFIED for Cisco [C2 "Storage Internal network"].

The v1 draft counted one BMC port per server and two OOB switches at 32 nodes. NVIDIA counts "6x 1Gb/s connections" per HGX B300 server and uses "SN2201 switch for every 2 SUs (4 switches total)" at 32 nodes. VERIFIED [S6 "Enterprise RA Scalable Unit", "32 Nodes"]. Corrected:

The per server rate is VERIFIED [S6]; totals and switch counts are INFERRED (ceil(ports / 48)), before PDUs, CDUs and console servers. The Edgecore design deck's 2, 3 and 8 OOB switches [E4 slide 3] are superseded.

• Switch. EPS201 (AS4630-54TE): 48 x 1G RJ45, 4 x 25G and 2 x 100G uplinks, Trident 3. VERIFIED [E3; E7].
• Uplinks. NVIDIA scales OOB "using a 25 Gbps or 100 Gbps spine layer". VERIFIED [S6 "Out-of-Band (OOB) Management Network"].
• BMC protocol. In an NCP build, Redfish over TLS only: "IPMI must be disabled." VERIFIED [R24 CNP10]. Outside an NCP it's still the right default. INFERRED.
• Isolation. OOB on a dedicated, restricted network reached only through a hardened bastion. VERIFIED [R24 SEC12]. Never tenant reachable [C2 "Multitenancy"; N1 section 4.4.2].
• Console. Every SONiC switch needs a serial console reachable when its management port isn't, to recover a box stuck in ONIE or a bad upgrade. INFERRED.

Host side software (NVIDIA AI Enterprise, drivers, NCCL, Dynamo) is unchanged by the switch choice. INFERRED.

• Controller. An Edgecore fabric controller for AI fabrics isn't documented. Operate with open tooling: ONIE plus zero touch provisioning, configuration as code, and the NOS's own APIs. INFERRED.
• Day 0: ONIE install, ZTP, BGP unnumbered underlay per rail leaf, cabling checked against the rail map with LLDP. INFERRED.
• Day 1: QoS and lossless profile (section 8), EVPN tenants on the frontend (section 14), OOB VRF.
• Day 2 telemetry: per queue ECN marks, PFC pause counters, buffer occupancy, link errors, optics DOM, and DLB or ECMP member load. Silicon support: "Programmable in-band telemetry" [E1, VERIFIED]; export by streaming telemetry from the NOS INFERRED.
• For Mode B, map every counter above into R24's network telemetry delivery (within 120 seconds, OpenTelemetry preferred) and NET01 fabric API (section 10.3). VERIFIED as requirement [R24].
• Resilience: "Hardware-based link failover for network resiliency and reduced job completion time"; hot swap redundant 3000 W PSUs and 7+1 fans. VERIFIED [E1].

VRF plan, mirroring Cisco's [C2 "Multitenancy"]:

• Backend tenancy in Mode A. Allocate tenants by whole rail blocks where possible: Cisco notes rail grouping "avoids one tenant's excessive resource usage from degrading another". VERIFIED [C2]. EVPN VXLAN on the RoCE backend with Tomahawk 5 and SONiC: performance and lossless behavior UNKNOWN (gap G6).
• Tenant isolation in Mode B. R24 SDN04: "One tenant shall not be accessible to or observable by another tenant." SEC11 requires hard physical or logical isolation. VERIFIED [R24]. The BlueField DPU enforces it on the host [IPR NET-1]; the Edgecore leaf carries the underlay and EVPN.
• Platform security. AST1060 root of trust securing OpenBMC; TPM 2.0. VERIFIED [E1]. Edgecore supplies no firewall or host security product; those are customer choices.
• Encryption. R24 SEC13 and SEC18 (section 10.3). MACsec on AIS800 UNKNOWN (gap G15).

Cisco reports end to end testing with "HPC Benchmark, IB PerfTest, NCCL Test, MLCommons Training, and Inference benchmarks". VERIFIED [C1, C2]. Edgecore has no equivalent end to end result for this design. The plan below borrows NVIDIA's own acceptance list so the results speak NVIDIA's language. Every IPR row cited here is a SHOULD, not a MUST. VERIFIED [IPR HW-1 to HW-5].

Existing public Edgecore evidence: the AIS800-64O collective communication benchmark on AMD MI300X with Broadcom Thor 2 NICs, up to 35.6% improvement at 1:1 and 26.7% at 2:1 with congestion aware packet spraying [E9]. It used AMD GPUs and Broadcom NICs, not ConnectX-8 or NCCL on B300, so it informs this plan and doesn't replace any test in it. VERIFIED [E9].

Model numbers from datasheets [E1, E3]; optics part numbers from the design deck [E4]; HGX quantities from [E4] slides 7 to 9; OOB from section 11.3.

Edgecore alongside NCP on HGX removes the backend leaf, spine and backend optics lines. INFERRED. A GB300 NVL72 BOM follows the section 6.2 counts (for example 54 AIS800-64O for one 16 rack POD backend in Mode A) and is INFERRED until a customer rack plan exists. The AIS800-32O (32 x OSFP800, Tomahawk 5, 25.6 Tbps, 1RU) [E2, VERIFIED] fits roles needing 32 ports or fewer, such as a storage leaf. INFERRED.

• [IPR] Inference Provider Platform Requirements on GB300 NVL72 for NCPs. Requirement IDs used: RD-1 to RD-3, NET-1 to NET-8, STO-1 to STO-7, HW-1 to HW-8, KUB-38, AS-5. https://docs.nvidia.com/dsx/ncp/inference-provider-requirements/home
• [IRA] NVIDIA Inference Reference Architecture. Sections used: Architecture Decision Guardrails, Data Center Architecture, Networking, Physical Architecture, Sizing Guidance, Design Alternatives And Tradeoffs, Reference Component Version And Source Snapshot. https://docs.nvidia.com/dsx/ncp/inference-ra/home
• [R24] NVIDIA Requirements for AI Clouds v2.4, September 1, 2026. Requirement IDs used: CNP08, CNP10, SDN04, SDN09, SEC11 to SEC13, SEC16, SEC18, STG06, NET01 to NET06, and the Telemetry and Logs sections. https://docs.nvidia.com/dsx/ncp/nvidia-requirements-for-ai-clouds/v2.4
• [S1] NCP Software Reference Guide, Data Center Architecture: GPU Compute Node table, Networking, Data Center View. https://docs.nvidia.com/dsx/ncp/software-reference-guide/data-center-architecture
• [S2] NCP Software Reference Guide Part 1, Infrastructure as a Service: "Data Centre Networks" table, Software Defined Networking. https://docs.nvidia.com/dsx/ncp/part-1-software-reference-guide/infrastructure-as-a-service
• [S6] NVIDIA HGX AI Factory Enterprise RA, Networking Logical Architecture: Enterprise RA Scalable Unit, Spine-Leaf Networking, OOB, Table 5, Table 7, and the 32, 64 and 128 node configurations. https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/network-logical-architecture.html
• [N1] NVIDIA SuperPOD DGX B300 Systems, Spectrum-4 Ethernet and DC Busbar Power Reference Architecture, RA-11337-001 V01. Available from Edgecore on request. Request this document ↗
• [N2] Edgecore review of NVIDIA Spectrum-X reference architectures and the public vendor claims they cite, 2026-08-26. Available from Edgecore on request. Request this document ↗

• [C1] Cisco N9000 Enterprise RA, "Full Stack AI Infrastructure compliant with NVIDIA HGX B300 Enterprise Reference Architecture", read 2026-08-26. Available from Edgecore on request. Request this document ↗
• [C2] Cisco N9000 Cloud RA with HGX H200 (Cisco NCP compliant Cloud Reference Architecture), read 2026-08-26. Available from Edgecore on request. Request this document ↗
• [C4] Cisco EBox NCP HGX High Performance Storage RA, 2026, p.5. Available from Edgecore on request. Request this document ↗

• [E1] AIS800-64O / AIS800-64D datasheet, EC-DS-0726-R04. https://www.edge-core.com/product/ais800-64o/
• [E2] AIS800-32O / AIS800-32D datasheet, EC-DS-0726-R03. https://www.edge-core.com/product/ais800-32o/
• [E3] EPS201 datasheet, EC-DS-0925-R06. https://www.edge-core.com/product/eps201/
• [E4] Edgecore Reference Design: 32x, 64x and 256x B300 GPU Clusters (generic), slides 2 to 9 and 11. Available from Edgecore on request. Request this document ↗
• [E7] EPS201 product page, "EPS201 AS4630-54TE". https://www.edge-core.com/product/eps201/
• [E8] Edgecore press release, 102.4T AIS1600-64O and AIS800-128O on Tomahawk 6, 2026-02-23. https://www.edge-core.com/press-release/edgecore-networks-sets-new-benchmark-for-ai-infrastructure-with-worlds-first-102-4t-open-networking-switches/
• [E9] Edgecore press release, up to 35.6% collective performance improvement, 2026-01-06. https://www.edge-core.com/press-release/edgecore-networks-demonstrates-up-to-35-6-performance-improvement-in-collective-communication-benchmark/