NVIDIA Mellanox MQM8790-HS2F InfiniBand Switch in Practice

NVIDIA Mellanox MQM8790-HS2F InfiniBand Switch in Practice | Low-Latency Interconnect Optimization for RDMA/HPC/AI Clusters

Background & Challenge: The Latency Bottleneck in Large-Scale AI and HPC Clusters

As AI training clusters scale to thousands of GPUs and HPC systems push toward exascale performance, the network fabric connecting compute nodes has become a critical performance determinant. In these environments, latency is not merely a metric — it directly impacts application performance, time-to-solution, and overall cluster efficiency. For workloads that rely heavily on MPI (Message Passing Interface) collective operations and all-to-all communication patterns, such as large language model training and computational fluid dynamics, even microsecond-level latency increases can translate into hours of additional runtime. Traditional Ethernet networks, even with RDMA over Converged Ethernet (RoCE), often struggle to deliver the deterministic low latency required by these demanding applications.

This challenge was recently confronted by a national research laboratory deploying a 2,000‑node HPC cluster for climate modeling and AI research. The cluster required 200Gb/s connectivity with sub‑100 nanosecond latency to support both MPI‑based HPC workloads and distributed AI training. The engineering team needed a switch that could deliver consistent, low‑latency performance at scale, while supporting advanced features such as adaptive routing and congestion control to maintain fabric efficiency under varying load conditions. The NVIDIA Mellanox MQM8790-HS2F emerged as the ideal solution, offering 40 ports of 200Gb/s HDR InfiniBand with sub‑100 nanosecond latency and advanced in‑network computing capabilities.

Solution & Deployment: Building a Low‑Latency InfiniBand Fabric

To address the latency and scalability challenges, the laboratory deployed the NVIDIA Mellanox MQM8790-HS2F as the core switch in a spine‑leaf fabric architecture. This MQM8790-HS2F InfiniBand switch provides 40 QSFP56 ports, each operating at 200Gb/s HDR, delivering a total switching capacity of 8Tb/s with sub‑100 nanosecond port‑to‑port latency. The fabric was designed with a 4‑spine, 16‑leaf topology, connecting 2,000 compute nodes each equipped with ConnectX‑6 HDR adapters. The MQM8790-HS2F 200Gb/s HDR 40-port QSFP56 configuration enabled the team to build a non‑blocking fabric with full bisection bandwidth, ensuring that every node could communicate with every other node at wire speed.

The deployment was executed in three key phases:

• Fabric Design: Using the MQM8790-HS2F InfiniBand switch solution, the team designed a spine‑leaf topology where each of the 16 leaf switches connected to 50 compute nodes (using a combination of 200Gb/s direct connections and 100Gb/s HDR100 breakout connections), while 4 spine switches provided inter‑leaf connectivity. The MQM8790-HS2F switches were configured with adaptive routing enabled, allowing the fabric to dynamically distribute traffic across available paths and avoid congestion points.
• Advanced Feature Configuration: The team enabled SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) on the MQM8790-HS2F switches to offload MPI collective operations from the compute nodes. This in‑network compute capability allowed the switches to perform all‑reduce and broadcast operations directly, reducing the number of network traversals and decreasing overall latency for collective communication.
• Performance Tuning: Subnet management was configured using the NVIDIA Unified Fabric Manager (UFM) platform, which provided real‑time visibility into fabric health, latency metrics, and congestion patterns. The team tuned congestion control parameters to optimize performance for the mixed HPC and AI workload profile.

Because the NVIDIA Mellanox MQM8790-HS2F is MQM8790-HS2F compatible with the broader NVIDIA InfiniBand ecosystem, including ConnectX‑6 and BlueField‑2 adapters, the deployment was seamless, requiring no custom drivers or firmware patches. The switch's integration with the UFM platform enabled the team to monitor fabric performance at scale, identifying and resolving potential bottlenecks before they impacted application runtime.

Results & Benefits: Measurable Improvements in Latency and Application Performance

Post‑deployment benchmarking across the 2,000‑node cluster revealed significant performance improvements. First, average port‑to‑port latency across the fabric was measured at 85 nanoseconds — consistent with the sub‑100 nanosecond specification documented in the MQM8790-HS2F datasheet. This low latency translated directly into application performance gains: MPI all‑reduce operations completed up to 35% faster compared to the lab's previous 100Gb/s InfiniBand fabric, while distributed AI training jobs (using NCCL‑based communication) saw end‑to‑end epoch times reduced by approximately 28%.

Second, the SHARP in‑network compute capability delivered substantial performance benefits. By offloading collective operations to the MQM8790-HS2F switches, the cluster reduced CPU and GPU utilization for communication tasks by up to 20%, freeing compute resources for actual computation. This was particularly beneficial for large‑scale AI training, where collective communication can account for 30‑40% of total runtime.

Third, the adaptive routing feature proved critical in maintaining consistent performance under varying load conditions. During peak usage periods, when the fabric handled a mix of MPI and AI traffic, adaptive routing dynamically distributed traffic across available paths, maintaining average latency within 10% of baseline and preventing congestion‑induced performance degradation. The team monitored fabric health using the UFM platform, which provided real‑time dashboards tracking latency, throughput, and link utilization across all 20 switches.

Fourth, the density of the MQM8790-HS2F 200Gb/s HDR 40-port QSFP56 switch enabled a compact fabric footprint. The lab reduced the number of switches required by 50% compared to the previous 100Gb/s InfiniBand infrastructure, reducing rack space consumption and power requirements. Each MQM8790-HS2F consumed less than 230W typical power, contributing to a 20% reduction in cooling costs for the networking infrastructure.

From an operational perspective, the switch's management capabilities simplified ongoing maintenance. The lab's network team used the CLI and Web UI interfaces to perform firmware upgrades and configuration changes without disrupting fabric operations, leveraging the switch's support for hitless upgrades. The MQM8790-HS2F specifications include comprehensive management features, including SNMP monitoring and syslog integration, enabling the team to integrate the fabric into their existing network operations center (NOC) monitoring framework.

Summary & Outlook: A Blueprint for Low‑Latency InfiniBand Fabrics

The deployment experience with the NVIDIA Mellanox MQM8790-HS2F across a 2,000‑node HPC and AI cluster clearly demonstrates that a 40‑port 200Gb/s HDR InfiniBand switch can deliver the low latency, scalability, and advanced features required for demanding research and enterprise workloads. By leveraging the switch's sub‑100 nanosecond latency, adaptive routing, and SHARP in‑network compute capabilities, organizations can build fabrics that accelerate MPI and AI communication, reduce time‑to‑solution, and improve overall cluster efficiency.

Looking ahead, as AI training clusters continue to grow toward 10,000+ GPUs and HPC systems scale to exascale, the demand for high‑density, low‑latency InfiniBand switches will only increase. The MQM8790-HS2F is well positioned for this trajectory, because its 40‑port density, 8Tb/s switching capacity, and support for HDR200 and HDR100 speeds ensure compatibility with both current and next‑generation compute nodes. For organizations planning similar HPC or AI cluster deployments, the tiered spine‑leaf approach validated in this deployment offers a practical roadmap: deploy MQM8790-HS2F leaf switches for access connectivity, use higher‑port‑density spine switches (such as the 64‑port QM9700 series) for larger fabrics, and maintain a unified management framework leveraging UFM for proactive fabric optimization.

For detailed fabric design templates, performance tuning guides, and deployment checklists, refer to the MQM8790-HS2F datasheet and the NVIDIA Mellanox InfiniBand architecture documentation.