Introduction
Training large language models requires infrastructure at a scale that challenges even well-resourced organizations. Pre-training a 70B parameter model requires 1,000+ H100 GPUs running continuously for weeks, consuming megawatts of power and generating petabytes of checkpoint data. Getting this infrastructure right is the difference between a successful training run and weeks of wasted compute.
The three pillars of LLM training infrastructure are compute (GPU clusters), networking (InfiniBand fabric), and storage (parallel file systems). Each must be sized and configured correctly, and each can become the bottleneck that limits training throughput. A 1% degradation in any pillar translates directly to 1% longer training time — and at $10M+ training costs, that matters.
This guide covers the full infrastructure stack for LLM training: GPU cluster sizing, InfiniBand fabric design, checkpoint storage architecture, and the operational patterns for running multi-week training jobs reliably. It is written for infrastructure engineers who need to build or evaluate training infrastructure, not just use it.
H100s for 70B pre-training
Target MFU for H100 clusters
InfiniBand NDR per port
70B pre-training time (1K H100s)
Cluster sizing
GPU cluster sizing starts with the training compute budget (FLOPs) required for the target model. The Chinchilla scaling laws provide guidance: optimal training requires approximately 20 tokens per parameter. A 70B model trained optimally requires 1.4 trillion tokens, which at 1,000 H100s running at 50% MFU (Model FLOP Utilization) takes approximately 3-4 weeks.
MFU is the key efficiency metric for training clusters. It measures the fraction of theoretical peak FLOPS actually used for model computation. Well-optimized training achieves 40-60% MFU on H100 clusters. Poor network fabric, suboptimal parallelism strategies, or data loading bottlenecks can reduce MFU to 20-30%.
Model FLOP Utilization (MFU)
Cluster sizing must account for fault tolerance. At 1,000 GPUs, expect 1-2 GPU failures per week. The cluster must be sized to absorb failures without stopping training. Typically, provision 5-10% spare capacity and design the training job to continue with reduced parallelism when nodes fail.
Network fabric
The network fabric is the most critical and most often underestimated component of training infrastructure. All-reduce operations (gradient synchronization) require every GPU to communicate with every other GPU in the cluster. Network bandwidth directly determines how efficiently this communication can be overlapped with computation.
InfiniBand NDR (400 Gb/s per port) is the standard for H100 training clusters. A full-fat-tree topology provides non-blocking bandwidth between all nodes. For a 1,000-GPU cluster, this requires approximately 125 InfiniBand switches and 2,000+ cables — a significant infrastructure investment.
NCCL (NVIDIA Collective Communications Library) is the software layer that implements all-reduce, all-gather, and reduce-scatter operations over InfiniBand. NCCL tuning — ring vs tree topology, chunk size, protocol selection — can improve communication efficiency by 20-30% on the same hardware.
Network bottleneck detection
Checkpoint storage
Checkpoint storage is a critical and often overlooked component of training infrastructure. A 70B model checkpoint in FP16 requires 140GB of storage. With checkpointing every 1,000 steps and keeping the last 5 checkpoints, a training run requires 700GB of checkpoint storage — and this must be written at full speed without blocking training.
Parallel file systems (Lustre, GPFS, WekaFS) are required for checkpoint storage at scale. A single NFS server cannot sustain the write bandwidth required for large model checkpointing. Lustre with 10+ OSTs can sustain 100+ GB/s write throughput, enabling checkpoint writes in under 2 minutes for a 70B model.
Checkpoint storage must be separate from training data storage. Training data is read sequentially at high throughput; checkpoints are written in large bursts. Mixing these workloads on the same storage system degrades both. Use separate storage tiers with different performance characteristics.
Training infrastructure architecture
LLM Training Infrastructure Stack
Experiment Tracking
W&B, MLflow for training run management
Monitoring
GPU health, training metrics, alerting
Job Scheduler
Slurm or Kubernetes for job management
Checkpoint Storage
High-throughput checkpoint writes, S3 backup
Parallel File System
Lustre/GPFS for training data and checkpoints
InfiniBand NDR Fabric
400 Gb/s per port, full fat-tree topology
GPU Cluster
H100 SXM5 nodes, NVLink within node
Cluster requirements by model size
Training cluster requirements
| Model Size | GPU Count (H100) | Training Time | Storage Required | Network BW | Checkpoint Size | Estimated Cost |
|---|---|---|---|---|---|---|
| 7B params | 64-128 | 1-2 weeks | 5-10 TB | 400 Gb/s | 14 GB | $200K-500K |
| 70B params | 512-2,048 | 3-6 weeks | 50-100 TB | 400 Gb/s | 140 GB | $2M-10M |
| 405B params | 2,048-8,192 | 8-16 weeks | 200-500 TB | 400 Gb/s | 810 GB | $15M-60M |
| 1T params | 8,192-32,768 | 16-52 weeks | 1-5 PB | 400 Gb/s | 2 TB | $100M+ |
Training cost calculator
LLM Training Cost Calculator
Estimate total training cost based on model size, cluster configuration, and training duration.
Estimated results
Total GPU-hours
Total training cost
Cost per GPU-hour
Effective TFLOPS
Est. tokens trained
Cost per 1B tokens
Fault tolerance
At 1,000 GPUs, statistical failure rates guarantee hardware failures during any multi-week training run. Fault tolerance is not optional — it is a core infrastructure requirement. The goal is to minimize the amount of training work lost when failures occur.
GPU failure rates at scale
Elastic training frameworks (PyTorch Elastic, Megatron-LM with fault tolerance) can continue training with fewer GPUs after a node failure, rather than stopping the entire job. This reduces the impact of individual node failures from a full job restart to a brief pause.
Operational practices
Running multi-week training jobs reliably requires operational discipline: daily health checks of all GPUs, monitoring of training loss curves for anomalies, regular checkpoint validation, and clear escalation procedures for infrastructure failures.
Training loss monitoring
Maintain a training runbook that documents the expected loss curve, checkpoint schedule, escalation procedures, and recovery steps for common failure modes. This is essential for teams running 24/7 training jobs where different engineers may be on-call at different times.
Frequently asked questions
How many GPUs do you need to train a 70B model?
Training a 70B model from scratch requires 512-2,048 H100 GPUs depending on target training time. With 1,024 H100s at 50% MFU, pre-training on 1.4T tokens takes approximately 3-4 weeks. Fine-tuning a 70B model requires 32-128 H100s for 1-7 days depending on dataset size and technique (full fine-tuning vs LoRA).
How long does LLM pre-training take?
Pre-training time depends on model size, cluster size, and token count. LLaMA-70B on 1,024 H100s: ~3-4 weeks for 1.4T tokens. LLaMA-7B on 128 H100s: ~1-2 weeks for 1T tokens. GPT-4-class models (estimated 1T+ parameters): months on thousands of GPUs. Training time scales linearly with token count and inversely with GPU count × MFU.
How much does it cost to train an LLM?
Training costs depend heavily on model size and cluster efficiency. LLaMA-7B: $100K-500K. LLaMA-70B: $2M-10M. GPT-4-class: estimated $50M-100M+. These costs assume H100 hardware at $2-4/GPU-hour. On-premises hardware reduces costs by 3-5x for organizations with sufficient scale to justify the capital investment.
How do you handle training job failures?
Training job failures are inevitable at scale. Design for fault tolerance: checkpoint frequently (every 500-1,000 steps), use elastic training that can continue with fewer GPUs, implement automatic restart from the last checkpoint, and monitor GPU health metrics to detect failing hardware before it causes job failures. At 1,000 GPUs, expect 1-2 failures per week.