H100 vs A100: Separating Marketing Claims from Real-World Performance

At first glance, comparing NVIDIA's H100 and A100 GPUs seems straightforward. The H100 is newer, has a newer architecture, faster memory, newer Tensor Cores, FP8 support, and higher theoretical compute performance. So it is tempting to summarize the comparison with a single number: the H100 is 3× faster than the A100.

The problem is that there is no single H100-vs-A100 performance number.
Depending on what is being measured, published comparisons can produce dramatically different results. NVIDIA's Hopper architecture documentation lists roughly 3.2× the theoretical Tensor Core throughput of the A100 for FP16, BF16 and TF32.

NVIDIA has also published H100 results showing up to 2.6× the performance of the latest A100 submission in MLPerf Training, while some optimized transformer workloads can produce substantially larger gains. On the other hand, independent research has found much smaller improvements for some workloads.

Then there are headline figures such as NVIDIA's claim of up to 30× higher inference performance for certain large language model configurations. That number is real in the context NVIDIA specifies - but it is not a reasonable assumption that any LLM, or any AI workload, will run 30× faster simply because it moves from an A100 to an H100. NVIDIA's own comparison uses a specific 530-billion-parameter model, system configuration and interconnect setup.

Even the more modest-looking numbers can hide important differences. A benchmark comparing H100 FP8 with A100 FP16, for example, is measuring the benefit of both newer hardware and a different numerical precision. NVIDIA's Hopper documentation reports a 6.4× theoretical Tensor Core throughput advantage for this FP8-versus-FP16 comparison. That is useful information, but it should not be interpreted as “H100 applications are 6.4× faster.”

This is why many H100-vs-A100 comparisons are difficult to use for an actual purchasing decision.
A GPU buyer doesn't really need to know whether the H100 is “3× faster.” They need to know:

• How much faster will it be for my workload?

• Does my workload benefit from FP8 and the Transformer Engine?

• Does memory bandwidth matter more than raw compute?

• Will I be using PCIe or SXM?

• How does performance change when I scale from one GPU to eight or more?

• How much of the benchmark advantage comes from software optimization?

• How much more will the H100 cost?

• Does the higher-performance GPU actually reduce the cost of completing my workload?

• Does a higher-power GPU necessarily consume more energy per completed job?

Those questions produce very different answers from a simple specification-sheet comparison.

The distinction matters because the H100's advantages are genuine - but they are highly workload - and configuration-dependent. Hopper introduced fourth-generation Tensor Cores, FP8 and the Transformer Engine, faster HBM3 memory, and faster GPU-to-GPU interconnects.

Those capabilities can produce enormous gains for some transformer and large-scale AI workloads. But a GPU that is 3× faster in theoretical Tensor Core throughput does not automatically make every application 3× faster.

This article takes a different approach. Instead of searching for one definitive H100-vs-A100 speedup number, we examine what is actually being measured behind the numbers.
We will look at theoretical specifications, standardized benchmarks, independent research, software optimizations, GPU configurations, multi-GPU scaling, power consumption and - most importantly - the economics of completing a real workload.

The goal isn't to prove that the H100 is better than the A100, or vice versa.
It is to answer a more useful question: When does the H100's additional performance justify its additional cost - and when is an A100 still the smarter GPU to buy?

“H100 Is 3× Faster Than A100”: The Number Isn't Universal

One of the most common claims in H100-vs-A100 comparisons is that the H100 is roughly 3× faster than the A100.

There is a technical basis for that number. NVIDIA's Hopper architecture documentation lists the H100 at roughly 3.2× the A100's theoretical Tensor Core throughput for FP16, BF16 and TF32. For example, the published figures are 1,000 TFLOPS for H100 versus 312 TFLOPS for A100 in FP16 Tensor Core performance, or 2,000 versus 624 TFLOPS when NVIDIA's sparsity figures are included.

But there is an important distinction: 3.2× higher theoretical Tensor Core throughput does not mean every application runs 3.2× faster.
That difference is easy to miss when specifications are converted into a headline.

The numbers change depending on what you measure

Consider how widely published H100-vs-A100 performance figures can vary:

Measurement	Reported H100 advantage	What is actually being compared?
FP16/BF16 Tensor Core peak throughput	3.2×	Theoretical hardware throughput
MLPerf Training 2.1	Up to 2.6×	End-to-end training benchmark vs latest A100 result
MLPerf Training 2.1 vs original A100 submission	Up to 6.7×	Different benchmark generations and software states
MLPerf Training 3.0	Up to 3.1× per accelerator	H100 vs prior A100 submission
MLPerf Inference 2.1	Up to 4.5×	End-to-end inference performance
Some optimized transformer workloads	~3× or more	Specific model, precision and software stack
Some conventional/irregular workloads	Much lower	Workload-dependent application performance

NVIDIA's own MLPerf results illustrate the problem particularly well. In MLPerf Training 2.1, NVIDIA reported that H100 delivered up to 2.6× the performance of its latest A100 results. But when compared with the first A100 submission, the figure reached 6.7×.

Those two numbers describe the same generation of H100 hardware. The difference comes partly from what A100 result you choose as the baseline.

And there is another complication: the A100 itself became substantially faster over time without any change to the GPU hardware. NVIDIA reported that software improvements increased A100 performance by as much as 2.5× between its initial MLPerf Training submission and the later v2.1 submission.

This is an important lesson for anyone comparing GPUs: A benchmark number describes a particular hardware-and-software state, not an immutable property of the GPU.

Why theoretical performance and application performance diverge

A real AI workload rarely spends all of its time doing the exact matrix operations represented by a GPU's peak Tensor Core specification.

An application may instead be constrained by:

• memory bandwidth;

• memory capacity;

• CPU preprocessing;

• data movement;

• GPU-to-GPU communication;

• kernel efficiency;

• synchronization;

• batch size;

• numerical precision;

• framework overhead;

• storage or networking;

• or operations that do not benefit equally from the H100's newer hardware.

NVIDIA's own MLPerf work provides a good example. For one Mask R-CNN configuration, eliminating CPU-GPU synchronization and related bottlenecks improved H100 performance by 25–30% at 32 GPUs. NVIDIA also reported that optimizations to CPU-side evaluation and preprocessing were important because the faster H100 exposed bottlenecks that were less significant on A100.

In other words, once the GPU becomes faster, something else can become the bottleneck.

This is why a GPU's theoretical compute advantage should be treated as an upper-level hardware indicator rather than a promise about application speed.

Even standardized benchmarks produce different multipliers

MLPerf is particularly useful here because it provides standardized workload definitions, but even MLPerf results should not be reduced to one universal “H100 speedup.”

In MLPerf Training 3.0, NVIDIA reported that H100 delivered up to 3.1× more performance per accelerator than its previous A100 submission. At the same time, H100 performance on the same hardware improved by up to 17% in only six months, demonstrating how quickly software optimization can change measured performance.

MLPerf Inference 2.1 produced yet another number: NVIDIA reported up to 4.5× higher H100 performance than A100 across the data-center inference workloads in that submission.

There is therefore no contradiction between saying:

• H100 has roughly 3.2× the theoretical FP16 Tensor Core throughput;

• H100 can achieve around 2.6× the performance of A100 in some MLPerf Training comparisons;

• H100 can achieve up to 4.5× in particular MLPerf Inference comparisons.

They are simply different measurements under different conditions. The baseline matters, too

This is one of the easiest ways for an apparently impressive benchmark to become misleading.

Suppose GPU A produces a score of 100 and GPU B produces 300. The obvious conclusion is:
GPU B is 3× faster.

But what if GPU A's result came from an older software stack while GPU B's result uses newer libraries, optimized kernels and a newer framework?

The comparison is still useful if your goal is to understand the performance available to a buyer today. But it tells you less about the raw architectural difference between the two GPUs.

That distinction is particularly important with A100 because NVIDIA spent several years optimizing its software stack after the GPU launched. NVIDIA's own MLPerf history shows substantial performance gains on the same A100 hardware from software and system-level improvements alone.

So, how much faster is the H100?

The most defensible answer is: It depends.

For heavily optimized transformer workloads that can exploit Hopper's Tensor Cores, FP8 and Transformer Engine, H100 can deliver very large gains. Standardized benchmarks also demonstrate substantial generational improvements.

But a buyer should not take a headline such as “H100 is 3× faster” and multiply the expected performance of every workload by three.

Instead, ask:

1. What workload was benchmarked?

2. What precision was used?

3. Was sparsity enabled?

4. Which H100 and A100 configurations were tested?

5. How many GPUs were used?

6. Were the systems otherwise comparable?

7. Which CUDA, framework and inference/training libraries were used?

8. Was the result measuring throughput, latency or time-to-train?

9. How old was the A100 software stack?

10. Does the benchmark resemble the workload I actually need to run?

Only then does the performance number become useful for a purchasing decision.

Buyer takeaway: Don't budget for “3× H100 performance” simply because a comparison article says H100 is 3× faster.

Use the headline multiplier as a starting point, then find a benchmark that matches your model, precision, batch size, GPU configuration and software stack.

For an AI infrastructure buyer, workload-specific performance is more valuable than a generic generation-to-generation speedup.

“H100 Is 6.4× Faster”: The FP8 Comparison Trap

If the “H100 is 3× faster” claim needs qualification, the 6.4× figure needs even more careful handling.

NVIDIA's Hopper architecture documentation reports that the H100 can deliver 6.4× the theoretical Tensor Core throughput of the A100 when comparing H100's FP8 capability with the A100's FP16 capability. That number comes from a comparison of 4,000 TFLOPS of sparse FP8 Tensor Core performance on H100 SXM against 624 TFLOPS of sparse FP16 Tensor Core performance on A100. (developer.nvidia.com)

The number is technically meaningful. But it is misleading to simply describe it as: “H100 is 6.4× faster than A100.”

Why? Because this is not an apples-to-apples comparison of the same precision.
It compares FP8 on H100 with FP16 on A100.

That distinction matters because FP8 is itself one of Hopper's major new capabilities.

What is FP8?

FP8 is an 8-bit floating-point format introduced to H100's Tensor Cores for AI workloads. NVIDIA supports two FP8 formats:

• E4M3, which provides more precision but less numerical range.

• E5M2, which provides greater numerical range but less precision.

Compared with FP16 or BF16, FP8 uses half as many bits to represent each value. That can reduce the amount of data that needs to be stored or moved and can increase the throughput of supported matrix operations. (docs.nvidia.com)

But lower precision creates a fundamental trade-off:

You get greater computational efficiency by representing numbers with fewer bits, but you have less numerical precision and, depending on the format, a different representable range.

That is why simply replacing FP16 with FP8 is not equivalent to pressing a “faster” button.

The H100's advantage is really FP8 + Transformer Engine

The important innovation is not FP8 in isolation.

Hopper introduced the Transformer Engine, which dynamically uses FP8 and higher-precision formats during transformer workloads. NVIDIA designed it to take advantage of FP8 where appropriate while retaining higher precision where it is needed.

This matters because modern AI models do not necessarily need every operation to use exactly the same numerical precision.

A useful way to think about the Transformer Engine is: Use lower precision where the workload can tolerate it → use higher precision where necessary → reduce memory movement and computation → increase overall throughput.

That is fundamentally different from saying:

“Run the entire model in FP8.”

Why the 6.4× number exists

NVIDIA's published theoretical figures make the distinction clear:

Tensor Core operation	A100	H100	H100 vs A100
FP16	312 TFLOPS	1,000 TFLOPS	3.2×
BF16	312 TFLOPS	1,000 TFLOPS	3.2×
TF32	156 TFLOPS	500 TFLOPS	3.2×
FP8	Not supported	2,000 TFLOPS	New capability
FP8 with sparsity	Not supported	4,000 TFLOPS	6.4× vs A100 FP16 sparse

The comparison therefore contains two different sources of H100's advantage.

First, Hopper has substantially faster Tensor Cores even when using the same FP16/BF16/TF32 data types.

Second, Hopper adds FP8, which creates another potential performance increase for workloads that can exploit it.

This is an important distinction for buyers.

If your workload remains in FP16 or BF16, you can still benefit from H100's newer architecture.

If your workload can effectively exploit FP8 through an appropriate software stack, the potential advantage becomes considerably larger.

FP8 also reduces data size

There is another reason FP8 can matter beyond raw arithmetic throughput.

An FP16 value uses 16 bits.

An FP8 value uses 8 bits.

So, at the representation level, FP8 requires half the storage per value compared with FP16 or BF16. NVIDIA notes that this can reduce memory requirements and data movement while increasing throughput.

For AI workloads, this can have several consequences:

• more values can fit into a given amount of memory;

• less data needs to move through the memory hierarchy;

• matrix operations can be processed at higher throughput;

• bandwidth pressure can decrease for suitable operations;

• larger effective batch sizes may become possible in some workloads.

But this does not mean that a model requiring 100GB of FP16 storage will automatically require only 50GB of H100 memory in every practical situation.

Model weights are only one part of the memory footprint. Training also involves activations, gradients, optimizer states and temporary buffers, while inference can be heavily affected by the KV cache and batch size.

The accuracy question

The obvious concern with lower precision is whether the model still produces acceptable results.

This is why FP8 should not be treated as a generic “compress everything to 8-bit” technique.

The numerical behavior of different operations matters. Some operations are more sensitive to precision than others, and training and inference have different requirements.

H100's Transformer Engine was designed specifically around this problem: NVIDIA describes it as using FP8 and FP16 together for transformer workloads rather than requiring the entire computation to operate in FP8.

For a buyer, this creates an important question: Can my framework and model actually take advantage of FP8 without unacceptable accuracy or engineering overhead?

If the answer is no, the theoretical FP8 advantage may have little practical value.

Real-world results are smaller - and more informative

The difference between theoretical FP8 throughput and real application performance becomes obvious in actual benchmarks.

NVIDIA's TensorRT-LLM comparison, for example, tested GPT-J 6B on 80GB SXM GPUs using H100 FP8 and A100 FP16. At batch size 64, H100 achieved 10,907 output tokens/sec versus 3,679 tokens/sec on A100, or about 3× the throughput. At batch size 1, the numbers were 185 versus 111 tokens/sec, reducing the advantage to about 1.7× for first-token latency and about 2.4× for output throughput. (nvidia.github.io)

That is a much more useful way to understand the 6.4× theoretical figure.

The H100 can deliver an enormous FP8 hardware advantage, but the application-level improvement depends on what the application is actually doing.

And even within the same model, the measured advantage can change with batch size and the performance metric.

Precision can therefore change the economics

Consider two hypothetical workloads.

Workload A

• A100: FP16

• H100: FP16

• H100 is 3× faster.

Workload B

• A100: FP16

• H100: optimized FP8

• H100 is 4× faster.

If the H100 costs twice as much per hour, it is already economically attractive in both scenarios - but considerably more so in Workload B.

Now reverse the situation.

If a workload cannot effectively use FP8 and the H100 is only 1.3× faster while costing twice as much, the A100 may be the better economic choice.

This is why precision support belongs in a GPU buying decision, not just in an architecture specification table.

Another trap: sparsity

There is one more detail that can make H100/A100 comparisons look larger than they really are: sparsity.

NVIDIA's published Tensor Core figures often provide both dense and sparse performance.

Sparse figures assume the workload can exploit NVIDIA's supported structured sparsity pattern. NVIDIA's specifications explicitly mark the headline H100 Tensor Core numbers with sparsity. (nvidia.com)

For example, NVIDIA lists:

• H100 FP16 Tensor Core: 1,979 TFLOPS with sparsity

• H100 FP8 Tensor Core: 3,958 TFLOPS with sparsity

The corresponding non-sparse figures are approximately half those values.

Therefore, when comparing peak Tensor Core specifications, the buyer should always ask: Is this dense performance or sparse performance?

A headline number based on sparsity is not directly comparable with a dense benchmark.

What buyers should take away from FP8

The correct conclusion isn't: “H100 is 6.4× faster because it has FP8.”

A better conclusion is: “H100 introduces FP8 Tensor Cores and the Transformer Engine, giving it a major potential advantage on workloads that can effectively use lower-precision transformer computation. NVIDIA's theoretical FP8 throughput is 6.4× the A100's sparse FP16 figure, but real application speedups are workload-, precision-, software- and configuration-dependent.”

That distinction is critical.

For an LLM deployment built around an optimized FP8 stack, H100's FP8 capability can be one of the strongest reasons to choose it over A100.

For a workload that cannot use FP8 effectively, the buyer should focus instead on H100's other advantages - such as its faster FP16/BF16 Tensor Cores, higher memory bandwidth and faster interconnect.

Buyer takeaway

Don't put “6.4× faster” into your GPU budget model.

First determine:

1. Does your model support FP8?

2. Does your framework support optimized FP8 execution?

3. Can you maintain acceptable accuracy?

4. Are your bottlenecks compute, memory or communication?

5. Is the benchmark comparing dense or sparse performance?

6. Is it comparing the same precision on both GPUs?

7. Does the measured benchmark resemble your production workload?

If FP8 is highly effective for your workload, H100's premium can be much easier to justify. If it isn't, the relevant H100-vs-A100 comparison may be closer to the same-precision performance difference, rather than the much larger FP8 headline.

“80GB Is 80GB”: Memory Capacity Isn't the Whole Story

Another common shortcut in H100-vs-A100 comparisons is to look at GPU memory capacity and conclude that an 80GB H100 and an 80GB A100 are effectively equivalent.

They are not.

Both GPUs can be configured with 80GB of memory, but the underlying memory technology and bandwidth are different. An A100 80GB SXM uses HBM2e and provides up to 2.039 TB/s of memory bandwidth. An H100 SXM uses HBM3 and provides 3.35 TB/s - about 64% more bandwidth. (nvidia.com)

	A100 80GB SXM	H100 80GB SXM
Memory	80GB HBM2e	80GB HBM3
Memory bandwidth	2.039 TB/s	3.35 TB/s
NVLink	600 GB/s	900 GB/s
TDP	400W	Up to 700W

The important distinction is: Memory capacity determines how much data can fit; memory bandwidth determines how quickly the GPU can move that data.

Those are different constraints.

Why bandwidth matters

Consider a workload that repeatedly reads large tensors from GPU memory. If computation is waiting for data, adding more compute capability won't necessarily solve the bottleneck. Higher memory bandwidth can allow the GPU to feed its compute units more quickly.

This is why even NVIDIA's own comparison of A100 40GB and A100 80GB found that the additional memory bandwidth of the 80GB version could produce meaningful application-level gains despite the two GPUs having the same basic compute capability. NVIDIA reported up to 25% application speedup on selected deep-learning and HPC workloads from A100 80GB's higher bandwidth. (nvidia.com)

The same principle applies when moving from A100 to H100: 80GB tells you the capacity, not the complete memory-performance profile.

Capacity still matters for AI models

For LLMs, memory capacity determines whether a model and its associated data can fit on a GPU - or how aggressively the workload must be distributed across multiple GPUs.

But the memory footprint is larger than model weights alone.

Depending on the workload, GPU memory may also need to hold:

• activations;

• gradients;

• optimizer states;

• temporary tensors;

• attention-related data;

• KV cache during inference;

• framework/runtime buffers.

This is why two GPUs with the same VRAM capacity can still produce different practical results.

And 80GB isn't even the whole H100 story

The phrase “H100 80GB” itself hides configuration differences.

NVIDIA's current H100 specifications list the H100 SXM with 80GB HBM3 and 3.35 TB/s bandwidth, while the H100 NVL has 94GB and 3.9 TB/s. Their power and interconnect characteristics also differ.

The A100 similarly exists in 40GB and 80GB versions, with PCIe and SXM configurations offering different bandwidth and system characteristics. NVIDIA lists A100 80GB PCIe at 1.935 TB/s and A100 80GB SXM at 2.039 TB/s.

So a serious GPU comparison should never simply say:

A100: 80GB
H100: 80GB

It should specify the exact memory configuration and form factor.

Buyer takeaway

When evaluating memory, ask three separate questions:

1. How much memory do I need?
Determines whether the model/workload fits.

2. How much bandwidth do I need?
Determines whether memory movement becomes a performance bottleneck.

3. How quickly must GPUs communicate with each other?
Determines how effectively the workload scales across multiple GPUs.

The H100's advantage isn't that it has “more memory” than an 80GB A100. In the standard SXM comparison, both have 80GB. Its advantage comes from the combination of newer HBM3 memory, substantially higher bandwidth and a faster GPU interconnect.

For GPU buyers, “80GB” should therefore be treated as a capacity specification - not a complete statement of memory performance.

“The Benchmark Measures the GPU”: Hardware vs. Software vs. System

A benchmark result is often presented as if it were a direct measurement of GPU hardware:

H100: 1,000 images/sec
A100: 500 images/sec
Therefore, H100 is 2× faster.

In reality, an application benchmark measures a much larger stack.

The result can depend on: GPU hardware + memory + CPU + networking + drivers + CUDA + libraries + framework + optimized kernels + model + configuration.

This matters enormously when comparing H100 and A100 because NVIDIA has spent years optimizing the software stack around both generations.

The GPU is only one part of the benchmark

Consider an AI training workload.

Data may first move from storage to system memory, then through the CPU and PCIe subsystem to the GPU. During distributed training, GPUs may exchange gradients or other tensors through NVLink or the network. Frameworks then launch kernels that rely on CUDA libraries and highly optimized implementations.

A faster GPU can therefore expose bottlenecks elsewhere.

NVIDIA has repeatedly demonstrated this in MLPerf. Optimizations to CUDA kernels, data pipelines, synchronization and communication can produce substantial gains without changing the GPU itself. For example, NVIDIA has reported improvements from techniques such as asynchronous data loading, optimized all-reduce operations and eliminating unnecessary CPU-GPU synchronization. (developer.nvidia.com).

This means a benchmark can legitimately tell you: “This complete H100 system and software stack is faster than this complete A100 system and software stack.”
But it doesn't necessarily tell you: “The H100 silicon is exactly X times faster than the A100 silicon.”

For a buyer, the first statement is actually more useful - but the distinction is important when interpreting benchmark claims.

Software can change the performance of the same GPU

The A100 provides a useful example.

Its hardware did not change between MLPerf submissions, but NVIDIA continued optimizing the software stack and reported substantial improvements in measured performance over time. That means comparing a newly optimized H100 benchmark against an old A100 result can exaggerate the apparent generational advantage.

The reverse is also possible: a mature A100 software stack may perform extremely well on an established workload, while a newer workload or software stack may initially be less optimized.

This is why benchmark date and software version belong alongside the GPU model in any serious comparison.

Framework optimization can favor one GPU

Modern AI frameworks can detect the underlying GPU architecture and select different kernels or execution paths.

For example, optimized libraries can take advantage of:

• newer Tensor Cores;

• FP8;

• fused operations;

• CUDA Graphs;

• improved attention implementations;

• asynchronous execution;

• specialized communication primitives.

H100 therefore isn't simply a faster A100. It also exposes capabilities that software can explicitly target.

This is particularly important for transformer workloads, where H100's Transformer Engine and FP8 support can materially change the execution path.

System configuration matters too

A GPU benchmark may depend on:

• CPU model;

• CPU-to-GPU topology;

• PCIe generation;

• GPU interconnect;

• NVLink;

• NVSwitch;

• InfiniBand;

• network bandwidth;

• number of GPUs;

• storage subsystem.

NVIDIA's H100 platform, for example, can use 900 GB/s of NVLink bandwidth per GPU in an HGX/DGX configuration, while A100 SXM provides 600 GB/s.

For a single-GPU benchmark, that difference may matter little. For an eight-GPU training workload communicating continuously between GPUs, it can matter considerably.

This creates a useful hierarchy of benchmark questions

When you see an H100-vs-A100 benchmark, ask what level it represents:

Level 1  -  Theoretical hardware
Peak TFLOPS, memory bandwidth, Tensor Core throughput.

Level 2 - Kernel benchmark
Measures specific operations under controlled conditions.

Level 3 - Model benchmark
Runs a specific model such as BERT, Llama or GPT-J.

Level 4 - End-to-end application
Includes framework, preprocessing and other system components.

Level 5 - Distributed system
Measures multiple GPUs, communication, networking and scaling.

The higher you move up this hierarchy, the more relevant the result becomes to a production buyer - but also the more difficult it becomes to attribute the result to the GPU alone.

The “faster GPU exposes bottlenecks” problem

This is one of the most important ideas for infrastructure buyers.

Imagine an A100 spends:

• 70% of its time computing;

• 20% waiting for data;

• 10% on synchronization.

Now replace it with an H100 that makes computation three times faster.

You might expect the entire application to become three times faster.

But if data loading and synchronization don't accelerate, those components become a larger fraction of total runtime.

The result could be much less than 3×.

This is essentially the same principle behind Amdahl's Law: accelerating one part of a workload cannot produce proportional end-to-end gains if other parts remain bottlenecks.

NVIDIA's MLPerf work provides real-world examples of this phenomenon. In large-scale training, NVIDIA has described optimizations targeting CPU-GPU synchronization, data readers and communication because simply making GPU computation faster can leave other portions of the training pipeline exposed.

What this means for GPU buyers

If you are replacing an A100 with an H100, don't benchmark only: “How many TFLOPS does the H100 have?”

Benchmark: “How long does my complete workload take from input to output?”
For training, measure time to reach the target model quality.

For inference, measure:

• requests/sec;

• tokens/sec;

• time to first token;

• inter-token latency;

• concurrent users;

• cost per token.

For HPC, measure:

• time to solution;

• scaling efficiency;

• energy per simulation.

And for all of them, record the complete configuration.

Buyer takeaway

A benchmark is most useful when it resembles your deployment.
When comparing H100 and A100 results, record at least:

GPU model → memory → form factor → GPU count → CPU → interconnect → precision → framework → software versions → batch size → workload.

If those details are missing, treat a large performance multiplier as a data point, not a purchasing guarantee.

The most useful benchmark isn't the one with the biggest H100/A100 number.
It's the one that most closely reproduces the workload you actually intend to run.

“TDP Tells You Energy Efficiency” - Power vs. Energy per Workload

A common comparison is A100 at 400W vs H100 at 700W, followed by the conclusion that the H100 is less energy efficient. That conclusion is incomplete.

TDP tells you the GPU's power envelope, not how much energy it takes to complete a workload. The relevant metric is:

Energy per completed workload = average power × time to completion

An H100 can consume more watts while consuming less total energy if it finishes the same job substantially faster.

For example, an independent 2026 study of FourCastNet training measured:

• 4 GPUs: H100 - 105.32 hours and 115.63 kWh; A100 - 139.92 hours and 120.70 kWh.

• 16 GPUs: H100 - 32.26 hours and 145.31 kWh; A100 - 43.28 hours and 144.87 kWh.

So the H100 completed the workload faster in both cases, but the energy advantage was workload- and scale-dependent, rather than simply proportional to its higher performance.

The better question is therefore not “Which GPU has the lower TDP?” but “Which GPU uses fewer kWh to produce the required result?”

For data-center buyers, this also makes cooling, rack density and utilization important parts of the energy equation.

“GPU-Hour Price Tells You Which Is Cheaper” - Cost per Completed Workload

Comparing rental prices such as $X/hour for A100 vs $Y/hour for H100 can produce the wrong buying decision.

The meaningful metric is: Cost to complete workload = GPU hourly price × GPUs used × hours required

Suppose an H100 costs twice as much per hour but completes the workload three times faster. Its effective compute cost is actually lower:

• A100: 1 GPU × $2/hr × 3 hours = $6

• H100: 1 GPU × $4/hr × 1 hour = $4

The same principle applies to multi-GPU jobs, where communication overhead, scaling efficiency and idle time can materially change the result.

Current India AI pricing illustrates why raw hourly comparisons can be misleading: listed configurations vary significantly by GPU type and deployment mode, including A100 40GB, A100 80GB, H100 PCIe and H100 SXM, with different per-GPU economics.

For a real purchasing analysis, calculate at least four numbers:

1. Cost per GPU-hour

2. Cost per completed job

3. Cost per unit of useful output - such as tokens, images or training samples

4. Cost at required throughput/latency

The buyer's takeaway

Cheap GPU-hours do not necessarily mean cheap compute. Measure the cost of producing the result you actually need.

“Inference and Training Behave the Same” - Different Performance Metrics

The mistake here isn't simply assuming training and inference require identical hardware. The more subtle mistake is using the same performance metric to compare them.

Training benchmarks typically care about time to reach a target quality. Inference can care about tokens/second, requests/second, time-to-first-token (TTFT), inter-token latency (ITL), or cost per million tokens.

Read our article: AI Training Needs Much More GPU Memory

The H100 can therefore show very different advantages depending on what is being measured.
For example, NVIDIA's TensorRT-LLM GPT-J comparison reported:

• Batch 64: H100 delivered about 3× A100 output throughput, while first-token latency was about 4.7× better.

• Batch 1: throughput advantage fell to about 2.4×, while first-token latency improved by about 1.7×.

The hardware did not change. The workload condition and metric changed.

This is particularly important when reading claims such as “H100 is 3× faster.” Always ask: Faster at what - training completion, throughput, latency, or cost per useful output?

The detailed training-versus-inference hardware requirements are covered separately in our dedicated workload articles. Here, the key benchmark lesson is simply: never compare two GPUs without first defining the performance metric that matters to the application.

“H100 Makes A100 Obsolete” - Where A100 Still Makes Sense

The H100 is the more capable accelerator, but that does not make the A100 automatically obsolete.
A100 can remain attractive when:

• The workload does not benefit significantly from FP8 or Hopper-specific optimizations.

• Memory capacity is sufficient and bandwidth is not the primary bottleneck.

• Existing infrastructure is already built around A100 systems.

• Lower hourly cost matters more than maximum throughput.

• The workload is lightly utilized, making H100's higher absolute performance difficult to monetize.

• Older CUDA/software environments make migration costly or risky.

Independent testing reinforces this point: H100 gains can vary substantially by workload, with some applications showing modest improvements rather than headline-level speedups.

The A100 is therefore not “the faster GPU.” It can still be the better economic choice when its performance is sufficient for the workload.

The correct comparison is not old GPU vs new GPU; it is required performance vs total cost.

Benchmark Claim Audit

Not every H100/A100 performance claim should be treated equally.

Claim	What it actually means	Verdict
H100 is 3.2× faster	Theoretical Tensor Core comparison under specified precision/configuration	Conditional
H100 is 6.4× faster	H100 FP8 vs A100 FP16 theoretical Tensor Core throughput, using NVIDIA's comparison methodology	Highly conditional
H100 is 4.5× faster for inference	Specific MLPerf inference workloads/configurations	Valid benchmark result, not universal
H100 is 2.6× faster in training	NVIDIA analysis against later A100 MLPerf results	Benchmark-specific
H100 is 30× faster	Specific large-model inference/system configuration	Do not generalize
H100 always has better performance/$	Depends on workload, utilization and actual GPU pricing	Unsupported as a universal claim
Higher TDP means worse efficiency	Confuses power with energy required to complete work	Misleading
80GB H100 = 80GB A100	Ignores memory bandwidth and interconnect differences	Incomplete

The pattern is consistent: most headline numbers aren't necessarily false - they are missing their conditions.

A credible benchmark should disclose GPU variant, precision, model, batch size, software stack, number of GPUs, networking and measurement methodology.

How to Evaluate an H100 vs A100 Benchmark

Before trusting a benchmark, run it through this checklist:

1. Same workload?
Same model, dataset, sequence length and target quality.

2. Same GPU class?
Compare PCIe with PCIe or SXM with SXM whenever possible.

3. Same precision?
FP8 vs FP16 can dramatically change the apparent multiplier.

4. Same software conditions?
Check CUDA, drivers, framework, libraries and optimized kernels.

5. Same scale?
Single-GPU results do not predict 8-GPU or multi-node performance.

6. Same metric?
Compare throughput with throughput, latency with latency, and cost with cost.

7. Reproducible?
Look for configuration details and a methodology that another researcher could replicate.
If several of these conditions are missing, treat the headline performance number as directional rather than definitive.

The Five Layers of GPU Performance

A useful way to understand H100 vs A100 results is to view performance as five interconnected layers:

1. Silicon
Tensor Cores, CUDA cores, memory architecture and raw compute capability.

2. Memory & Interconnect
HBM bandwidth, cache, NVLink, NVSwitch and networking.

3. Software
CUDA, drivers, cuDNN, cuBLAS, TensorRT and optimized kernels.

4. Workload
Model architecture, precision, batch size, sequence length and workload characteristics.

5. System & Economics
CPU, storage, networking, scaling efficiency, electricity, GPU pricing and utilization.

A benchmark that isolates Layer 1 tells you what the chip can do. A production benchmark across all five layers tells you what the system actually delivers and what it costs.

That is the fundamental mistake behind many simplistic H100 vs A100 comparisons: they reduce a five-layer performance problem to a single GPU specification.

What This Means for GPU Buyers

The H100 vs A100 decision should start with the workload - not the GPU specification sheet. Use this decision framework:

If your priority is…	Lean toward…
Maximum AI training performance	H100
FP8/Transformer workloads	H100
High-throughput LLM inference	H100
Large multi-GPU training clusters	H100
Lowest upfront/hourly compute cost	A100
Existing A100 infrastructure	A100
Workloads with modest H100 acceleration	A100 may offer better value
High utilization where faster completion matters	H100

But these are starting points, not universal rules. The final decision should come from three numbers:

1. Time to complete the workload
2. Total cost to complete it
3. Energy consumed to complete it

If the H100 costs 2× more per hour but finishes the workload 3× faster, it can be the cheaper choice. If it is only 30% faster while costing 2× more, the A100 may win economically.

Buy the performance you can actually use - not the performance the specification sheet promises.

Conclusion

The biggest problem with H100 vs A100 comparisons isn't that the numbers are necessarily wrong. It is that the conditions behind those numbers are often hidden.

The H100 can deliver enormous gains in the right workloads, particularly when FP8, Transformer Engine, optimized kernels, high-bandwidth memory and faster GPU interconnects are fully utilized.

But those advantages are not universal. Independent benchmarks show meaningful variation across workloads, while software optimization, system configuration, scaling and pricing can significantly change the final result.

That leads to a better way of comparing GPUs [[link to article #2 of this document]]: Don't ask how many times faster the H100 is. Ask how much faster - and how much cheaper - it can complete your actual workload.

For researchers, that means demanding benchmark methodology. For engineers, it means testing the complete software and hardware stack. For buyers, it means calculating cost per completed workload rather than simply comparing GPU-hour prices.

• The H100 is generally the stronger accelerator.

• The A100 can still be the smarter purchase.

The winner is whichever delivers the required workload at the best combination of performance, cost, energy and operational complexity.

Network Outlet is the preferred source for high quality, tested refurbished NVIDIA products your organization needs.

Summarized FAQs

Is the H100 really 3× faster than the A100?
Not universally. The performance difference depends on workload, precision, GPU configuration, software stack, batch size, and system architecture. Some benchmarks show around 3× or higher gains, while other workloads show much smaller improvements.

Why is the H100 sometimes 6.4× faster?
The 6.4× figure comes from a specific theoretical Tensor Core comparison involving H100 FP8 and A100 FP16 performance. It should not be interpreted as a universal application-level performance improvement

Is H100 more energy efficient?
It depends on the workload. H100 has a higher power envelope in some configurations, but it can finish workloads substantially faster. Energy efficiency should therefore be evaluated using energy consumed to complete the workload rather than TDP alone

Is H100 worth the extra cost?
It depends on workload utilization and performance requirements. The H100 is generally more attractive for highly optimized AI workloads, FP8-capable models, high-throughput inference, and large multi-GPU systems. The A100 can remain more economical when its performance is sufficient and its acquisition or rental cost is substantially lower

What is the best way to compare H100 and A100?
Compare the GPUs using the same workload, precision, model, software stack, GPU configuration, scale, and performance metric. For purchasing decisions, calculate time to completion, cost per completed workload, and energy consumed