GPU Clusters: Beyond Individual Servers

A single GPU server — even an 8-GPU DGX H100 — has fundamental limits. Training a 70B parameter model requires more GPU memory than a single DGX H100 provides (640GB). Training a 405B parameter model requires more than 10 DGX H100 systems. Achieving the training throughput needed to train frontier models in weeks rather than years requires thousands of GPUs working in parallel. This is why GPU clusters exist.

A GPU cluster is not simply a collection of GPU servers. It is a tightly integrated system where the interconnect fabric, storage architecture, and cluster management software are as important as the GPU hardware itself. A poorly designed cluster with excellent GPUs will underperform a well-designed cluster with the same GPUs.

8 GPUs

Smallest Cluster

IB NDR

Interconnect

100K+ GPUs

Hyperscale

80–90%

Scaling Efficiency

GPU Cluster Types

Training Cluster
Optimized for distributed model training. Requires high-bandwidth, low-latency interconnect (InfiniBand or RoCEv2) for all-reduce gradient synchronization. Requires parallel file system for high-throughput training data I/O. Job scheduling (Slurm or Kubernetes) manages multi-user access. The dominant GPU cluster type for AI labs and enterprises building foundation models.
Inference Cluster
Optimized for serving AI model predictions at scale. Lower interconnect requirements than training — inference requests are typically independent. Requires high-throughput networking for serving requests and loading model weights. Auto-scaling capabilities to match demand. Kubernetes is the dominant orchestration platform for inference clusters.
Mixed Training + Inference
Common in enterprise deployments where the same GPU infrastructure serves both training and inference workloads. Requires careful resource partitioning to prevent training jobs from starving inference capacity. NVIDIA Base Command and Slurm support priority queues and resource reservations for this use case.
Development Cluster
Smaller cluster (8–64 GPUs) for experimentation, fine-tuning, and development. Lower interconnect requirements — developers typically work on single-node or small multi-node jobs. Emphasis on ease of use and fast iteration. Often deployed with Jupyter notebooks and MLflow for experiment tracking.

GPU Cluster Architecture

Compute Layer
GPU servers (DGX H100, HGX-based OEM, or PCIe GPU servers). Each node contributes GPU compute, CPU compute, and local NVMe storage. Nodes are connected to the interconnect fabric via InfiniBand or Ethernet NICs. The compute layer is the most visible component but not the only performance determinant.
Interconnect Layer
InfiniBand or high-speed Ethernet fabric connecting all GPU nodes. Fat-tree or rail-optimized topology for full bisection bandwidth. InfiniBand switches (NVIDIA Quantum-2 for HDR, Quantum-3 for NDR) or Ethernet switches (Arista, Cisco, NVIDIA Spectrum). The interconnect layer determines multi-node scaling efficiency.
Storage Layer
Parallel file system (Lustre, GPFS/Spectrum Scale, WEKA) providing high-throughput shared storage for training datasets and checkpoints. Connected to GPU nodes via InfiniBand or high-speed Ethernet. Storage bandwidth requirements: 100–500 GB/s for large training clusters. NFS is insufficient for high-throughput training I/O.
Management Layer
Cluster management software (Slurm, Kubernetes, NVIDIA Base Command) for job scheduling, resource allocation, and monitoring. Out-of-band management network (IPMI/BMC) for hardware management. Monitoring stack (Prometheus, Grafana, NVIDIA DCGM) for GPU health and utilization visibility.

GPU Cluster Scale Tiers

GPU Cluster Scale Tiers

Cluster TypeGPU CountPrimary UseInterconnectStorageTypical Cost
Development cluster8–32 GPUsExperimentation, fine-tuning100GbE or IB HDRNFS or Lustre$500K–$3M
Mid-scale training cluster64–256 GPUsModel training (7B–70B)IB HDR or NDRLustre or GPFS$5M–$25M
Large-scale training cluster512–4,096 GPUsLarge model training (70B–405B)IB NDR fat-treeParallel FS (Lustre/GPFS)$50M–$300M
Hyperscale AI cluster4,096–100,000+ GPUsFrontier model trainingIB NDR/XDR or customDistributed object + parallel FS$500M+
Inference cluster8–512 GPUsProduction inference serving100GbE–400GbENFS or object storage$1M–$50M

GPU Cluster Software Stack

Job Scheduler (Slurm)
Slurm (Simple Linux Utility for Resource Management) is the dominant job scheduler for HPC and AI training clusters. Manages job queues, resource allocation (GPU, CPU, memory), and multi-user access. Supports priority queues, preemption, and reservation. Required for multi-user training clusters. Alternative: PBS Pro.
Container Orchestration (Kubernetes)
Kubernetes with GPU device plugins (NVIDIA GPU Operator) is the dominant platform for inference clusters and increasingly for training. Supports auto-scaling, rolling deployments, and service mesh. NVIDIA GPU Operator automates GPU driver and container runtime management. Alternative: Docker Swarm (less common).
NVIDIA Base Command Platform
NVIDIA's purpose-built cluster management platform for DGX systems. Combines job scheduling, container management, storage management, and monitoring. Simplifies cluster operations for organizations without dedicated HPC operations teams. Included with DGX systems; available separately for HGX-based clusters.
Distributed Training Frameworks
PyTorch Distributed (NCCL backend), DeepSpeed (Microsoft), Megatron-LM (NVIDIA), and JAX (Google) are the primary distributed training frameworks. NCCL (NVIDIA Collective Communications Library) handles all-reduce operations over InfiniBand or Ethernet. Framework choice impacts scaling efficiency and model parallelism capabilities.

Frequently Asked Questions

How many GPUs do I need for my AI workload?

It depends on model size, training time requirements, and budget. Rules of thumb: Fine-tuning a 7B model: 1–2 GPUs (H100 80GB). Fine-tuning a 70B model: 4–8 GPUs. Training a 7B model from scratch: 8–32 GPUs (days to weeks). Training a 70B model from scratch: 64–256 GPUs (weeks). Training a 405B model: 512–2,048 GPUs (weeks to months). For inference: 1 GPU per 70B model instance (FP8), scale horizontally for throughput. Start with the minimum required for your model size and scale based on time-to-result requirements.

What is the difference between a GPU cluster and a GPU cloud?

A GPU cluster is dedicated infrastructure — you own or lease the hardware and have exclusive access. A GPU cloud (AWS, Azure, GCP, CoreWeave, Lambda Labs) provides on-demand GPU access without hardware ownership. GPU clusters offer: lower cost per GPU-hour at sustained utilization (>60%), data sovereignty, customizable networking and storage, and predictable performance. GPU clouds offer: no upfront capital, instant scaling, no operations burden, and lower cost for bursty workloads. Most enterprises use a hybrid approach: owned clusters for baseline workloads, cloud for bursting.

What is the minimum viable GPU cluster for enterprise AI?

For a production enterprise AI cluster, the minimum viable configuration is: 2× DGX H100 (16 GPUs total) connected via InfiniBand HDR, with a shared Lustre or WEKA file system (100TB+), Slurm or NVIDIA Base Command for job scheduling, and DCGM for monitoring. This configuration supports: fine-tuning models up to 70B, running multiple concurrent inference instances, and supporting 5–20 data scientists. Total cost: approximately $1.5M–$2.5M for hardware. Smaller configurations (single DGX H100) are viable for development but insufficient for production multi-user workloads.