Introduction

Cost per token is the fundamental economic metric for LLM inference. It determines whether a product is economically viable, how pricing should be structured, and whether on-premises infrastructure makes sense compared to cloud APIs. Yet most organizations lack rigorous cost benchmarks for their specific workloads.

This guide presents empirical cost benchmarks across GPU platforms and cloud providers, with methodology for applying these benchmarks to your specific workload. The benchmarks cover LLaMA-7B through LLaMA-405B on H100, A100, L40S, and major cloud providers.

The key finding: on-premises H100 inference at scale costs $0.20-0.80 per million tokens, compared to $1-15 for cloud GPU instances and $10-60 for managed API services. The economics strongly favor on-premises at scale, but require sufficient volume to justify capital investment.

Benchmark methodology

Benchmarks were conducted using vLLM with continuous batching enabled, INT8 quantization (W8A8), and batch sizes optimized for maximum throughput. All measurements represent steady-state throughput, not burst performance.

Cost calculations include hardware amortization (3-year straight-line), power costs ($0.10/kWh), cooling overhead (PUE 1.3), and 10% for maintenance and operations. Cloud costs use on-demand pricing; reserved instance pricing reduces cloud costs by 30-40%.

Token counts use the standard definition: one token approximately equals 0.75 words in English. Benchmark prompts use a mix of short (200 token) and long (2000 token) inputs with 500-token average outputs.

On-premises benchmarks

On-premises H100 SXM5 achieves 2,000-8,000 tokens per second for LLaMA-7B and 200-800 tokens per second for LLaMA-70B, depending on batch size and quantization. At 80% utilization, this translates to $0.20-0.40 per million tokens for 7B models and $0.40-0.80 for 70B models.

L40S provides better cost efficiency for 7B-13B models. At $1.80/hour vs $3.50/hour for H100, and 80% of the throughput for small models, L40S achieves $0.15-0.30 per million tokens for 7B models — the lowest cost option for this model size.

A100 80GB sits between H100 and L40S in both performance and cost. For organizations with existing A100 infrastructure, it remains cost-competitive for 13B-70B models at $0.35-0.70 per million tokens.

Cloud benchmarks

Cloud GPU instances (AWS p4d, Azure NDv4, GCP A3) cost 3-8x more per token than on-premises for equivalent hardware, reflecting cloud provider margins and flexibility premium. AWS p4d (A100) costs approximately $2-4 per million tokens for LLaMA-70B.

Managed inference APIs (OpenAI, Anthropic, Google) cost $10-60 per million tokens — 25-150x on-premises cost at scale. The premium reflects model quality, reliability, and the absence of infrastructure management. For low-volume use cases, the convenience premium is justified.

The break-even point between cloud APIs and on-premises inference is typically 500M-2B tokens per month. Below this volume, cloud APIs are more economical. Above this volume, on-premises infrastructure pays back within 12-18 months.

Cost comparison table

Cost benchmark framework

On-premises ROI calculator

Break-even analysis

The break-even point for on-premises inference depends on three variables: API cost per token, on-premises infrastructure cost, and monthly token volume. Higher API costs and higher token volumes accelerate break-even.

Organizations should also factor in non-economic benefits of on-premises inference: data privacy (no tokens sent to third-party APIs), latency (no internet round-trip), customization (fine-tuned models), and independence from API pricing changes.

Optimization impact on cost

Frequently asked questions