NVIDIA

NVIDIA H200's 141GB HBM3e memory requires updated CUDA drivers and framework versions. Teams should verify compatibility before migration from H100.

AMD MI355X delivers lower TCO per million tokens than NVIDIA B200 for GPT-OSS 120B FP4 summarization at interactivity below 225 tok/s/user. MI300X also beats H100 on GPT-OSS 120B MX4 across all interactivity levels. B200 leads on LLaMA 70B FP4 and high-interactivity workloads.

NVIDIA's B200 Blackwell GPUs are shipping to hyperscalers, promising 2.5x performance gains over H100 for AI training workloads.

Installed AI compute from NVIDIA chips has more than doubled annually since 2020, with new flagship chips accounting for most compute within 3 years of release.

SemiAnalysis launched InferenceMAX, an open-source nightly benchmark comparing GPU inference performance across NVIDIA (H100, H200, B200, GB200 NVL72) and AMD (MI300X, MI325X, MI355X). Endorsed by Jensen Huang, Lisa Su, OpenAI, and Microsoft. First multi-vendor benchmark with TCO and power efficiency metrics.

vLLM has become the de facto standard for LLM inference, with major cloud providers and inference platforms adopting it for production deployments.

The best open models runnable on consumer GPUs lag frontier AI by only ~1 year across GPQA, MMLU, and LMArena benchmarks, suggesting rapid capability democratization and regulatory implications.

AMD's MI300X is seeing increased adoption as enterprises seek alternatives to NVIDIA's supply-constrained GPUs, with 192GB memory enabling larger model deployments.

NVIDIA GB200 NVL72 with TRT-LLM Dynamo achieves 4x better TCO per million tokens than single-node servers for DeepSeek R1 at 30 tok/s/user. Rack-scale inference with disaggregated prefill, wide expert parallelism, and multi-token prediction (MTP) delivers 2-3x throughput gains.

NVIDIA B200 delivers ~3x power efficiency vs H100 for GPT-OSS 120B FP4 (2.8M vs 900K tok/s/MW). Similar gains seen in AMD CDNA3 to CDNA4 (MI355X 3x better than MI300X). Blackwell 20% more energy efficient than MI355X due to lower TDP (1kW vs 1.4kW).

AMD MI300X achieves highest per-GPU LLM inference throughput in MLPerf Inference v5.1, delivering 21,150 tokens/s per GPU on llama2-70b, outperforming NVIDIA H100 (15,610 tok/s), B200 (13,015 tok/s), and H200 (10,917 tok/s). Industry-standard benchmark validates AMD's competitiveness in AI inference.

AWS has launched P5 instances featuring NVIDIA H200 GPUs, now generally available in US East and West regions with EU availability expected in Q1 2026.

PyTorch has surpassed 100 million weekly downloads on PyPI, cementing its position as the dominant deep learning framework for research and production deployments.