Cost-per-token breakdown by GPU: L4 vs A100, L40S and H100

Cost-per-token breakdown by GPU: L4 vs A100, L40S and H100

Which GPU is "cheapest" for running an LLM? If you are merely going to look at the hourly rate on the pricing page, then it's a pretty straightforward thing, but that is not actually useful. A $3 per hour GPU generating tokens fast can easily end up cheaper than a $0.50 per hour GPU generating them slowly. What actually matters is cost per token, not cost per hour.

Let me walk through how L4, A100, L40S and H100 actually compare once you look at the number that matters, and why the answer is not as simple as "buy the fastest one."

Why does looking at the hourly rate alone give you the wrong answer?

Because the real cost formula depends on throughput.

Cost per million tokens = (price per hour ÷ tokens per second ÷ 3600) × 1,000,000

A GPU with a higher hourly rate but significantly faster token generation can easily come out cheaper per million tokens than a cheaper card that takes much longer to produce the same output.

So the most important factor to consider here is memory bandwidth. It is what decides the token generation speed. During the decode phase, where a model generates output tokens one at a time, the GPU has to read the entire model weight matrix from VRAM for every single token. This makes decode almost entirely memory-bandwidth-bound rather than compute-bound, which is exactly why bandwidth, not raw TFLOPS, tends to predict real-world token throughput so closely.

How do these four GPUs actually compare on memory bandwidth?

Since bandwidth drives decode speed so directly, this is the number worth paying attention to first.

GPU

Memory bandwidth

Memory type

L4

300 GB/s

GDDR6

L40S

Higher than L4, still GDDR6

GDDR6

A100 (80GB)

Roughly 2,039 GB/s

HBM2e

H100

Roughly 3,350 GB/s

HBM3

That bandwidth gap between A100 and H100 maps almost directly onto real throughput. Published benchmarks show the H100 generating roughly 2.15 times more tokens per second than the A100 on a 70B model at batch size 1, tracking closely to their roughly 1.65x bandwidth ratio.

Does that mean H100 always wins on cost per token?

Not always, and this is where it gets genuinely interesting.

• On deep spot-market pricing, A100 instances can sometimes undercut H100 so significantly on hourly rate that the A100 ends up cheaper per token for large models, despite being slower, simply because the price gap outweighs the throughput gap

• Batching changes the economics dramatically on any GPU. Moving from single-stream requests to a batch of 8 concurrent requests can cut H100 cost per million tokens by roughly 4x, since GPU utilization affects real cost as much as the hardware itself does

What does the actual math look like for a 70B model?

Here is a rough, illustrative comparison based on published benchmark patterns. Real numbers vary by provider, quantization, and framework, so treat this as directional rather than exact.

Scenario

Approximate cost per million output tokens

H100, single-stream request

Roughly $0.40 to $0.45

H100, batch of 8 concurrent requests

Roughly $0.10 to $0.15

A100, on-demand pricing, similar batching

Noticeably higher than H100 at equal batch size, due to slower throughput

A100, deep spot pricing

Can undercut H100 despite slower speed, depending on the size of the discount

The pattern that holds up consistently is that batching and utilization move the needle on cost per token more dramatically than most people expect, often more than the choice of GPU generation itself.

Where does the L4 actually win, and where does it clearly lose?

The L4 was never built to compete with A100 or H100 on large model throughput. Its advantage lives entirely in a different lane.

But when we talk about small models, the Nvidia L4 vs A100 or H100, results will always be in the favor of L4. 

For 7B to 13B class models, the L4's much lower hourly rate more than compensates for its lower raw throughput, since these smaller models do not need the bandwidth headroom that makes the A100 worthwhile in the first place. Running a small model on an A100 is usually paying for capacity that workload will never use, which shows up directly as a worse cost-per-token outcome despite the A100's objectively faster hardware.

• L4 wins clearly on cost per token for small, latency-tolerant inference workloads

• A100 only starts making sense once model size or concurrency genuinely demands its extra bandwidth and memory

Where does the L40S fit between L4 and A100?

Right in the middle, which is exactly the gap it was built to fill. For mid-size models in the 7B to 30B range, the L40S frequently delivers the best cost-per-token of the group, offering meaningfully more throughput than the L4 without the cost premium that comes with A100 or H100 class hardware.

So which GPU should you actually pick to minimize cost per token?

The right answer depends entirely on model size and how much concurrency you actually need.

• Small models (7B to 13B), low to moderate concurrency: L4, almost every time

• Mid-size models (7B to 30B) needing more headroom: L40S is usually the sweet spot

• Large models (70B and up), production volume: H100, provided you are running with real batching rather than single-stream requests

• Large models on a tight budget: A100, particularly worth checking spot pricing before defaulting to H100 by assumption

Always check batching and utilization before assuming a newer GPU is automatically cheaper. A poorly utilized H100 can lose to a well-utilized A100 on cost per token, even though the H100 wins comfortably on every other spec.

Where this leaves you

The hourly price on a GPU listing answers the wrong question. Cost per token depends on throughput, memory bandwidth, batching, and how well you actually utilize the hardware, not just what the card costs to rent for sixty minutes. Match the GPU to your actual model size and concurrency pattern, check batching before assuming single-stream numbers apply, and the cheapest-looking hourly rate is not always the cheapest way to actually generate your tokens.

Frequently asked questions

Is it ever worth paying more per hour for a faster GPU just to lower cost per token? 

Often yes, specifically for large models running at real production volume with proper batching. The higher hourly rate is frequently offset, and sometimes more than offset, by the dramatically higher throughput once utilization is genuinely high.

Does batch size matter more than GPU choice for cost per token? 

For many workloads, yes. Moving from single-stream to a reasonable batch size can cut cost per token by several times on the same hardware, often a bigger swing than switching to a faster GPU while staying at low utilization.

Should I default to the newest GPU generation to minimize cost per token?

Not automatically. The newest generation usually wins for large models at high utilization, but for small models, an older or smaller card like the L4 frequently wins on cost per token simply because it is not being paid for capacity the workload never uses.