Serving AI to Millions: Inference, GPUs, Speed and Pricing

Training happens once. Inference, running the trained model to answer questions, happens billions of times a day. For a company like OpenAI, Anthropic, Google or xAI, making inference fast and cheap matters as much as making the model smart.

Prefill vs decode: why output tokens cost more#

There are two phases:

1. Prefill: the model processes your *input* tokens. They are all known in advance, so the GPU handles them in parallel.
2. Decode: the model generates *output* tokens one after another, because each new token depends on the previous one.

Decoding uses the GPU less efficiently, so most providers charge several times more per output token than per input token. That’s the reason behind a common API pricing shape: “₹X per million input tokens, ₹Y (higher) per million output tokens”.

def api_cost(input_tokens, output_tokens, in_price, out_price):
    """Prices are per million tokens, in any currency."""
    return input_tokens / 1e6 * in_price + output_tokens / 1e6 * out_price

# A chatbot answer: 1,500 tokens in (instructions + chat history), 400 out.
# Example prices only; always check the provider's pricing page.
cost = api_cost(1_500, 400, in_price=2.00, out_price=10.00)
print(f"one answer: ${cost:.4f}")
print(f"10,000 answers a day: ${cost * 10_000:,.2f}")

Output:

one answer: $0.0070
10,000 answers a day: $70.00

Notice that the 400 output tokens cost more than the 1,500 input tokens. Try different numbers in our free AI API cost calculator (in the Tools menu).

The KV cache: don’t re-read what you already read#

When writing token number 300, the model needs information from the previous 299 tokens. Recomputing all of them every step would be wasteful, so servers keep a KV cache (keys and values from the attention layers) in GPU memory. It makes decoding far faster, but uses a lot of memory, which is one reason very long conversations are slower and pricier.

Prompt caching is the same idea across requests: if many requests start with the same long instructions, the provider can reuse that work and charge less for the cached part. Several providers, including Anthropic, OpenAI and Google, offer it.

Batching: one GPU, many users#

A GPU is most efficient when doing many multiplications at once. Servers group requests from many users into a batch and run them together.

gpu_ms_per_step = 30
users = 8
alone = users * gpu_ms_per_step
batched = gpu_ms_per_step + users * 2
print("one at a time:", alone, "ms")
print("batched:", batched, "ms")

Making models smaller and faster#

params = 8e9                     # an 8-billion-parameter model
for bits in (16, 8, 4):
    gb = params * bits / 8 / 1e9
    print(f"{bits:>2}-bit weights: {gb:.0f} GB of memory")

Output:

16-bit weights: 16 GB of memory
 8-bit weights: 8 GB of memory
 4-bit weights: 4 GB of memory

That’s how open models like Llama or Mistral can run on a gaming laptop: a 4-bit 8B model fits in about 4 GB (plus some working memory).

Summary#

• Inference = running the model for users; it happens billions of times, so efficiency matters.
• Prefill reads your prompt in parallel; decode writes one token at a time, so output tokens cost more.
• The KV cache, prompt caching and batching make serving faster and cheaper.
• Quantization, distillation, speculative decoding and MoE make models smaller or faster.

Next: the major model families compared, closed and open, and how to choose one.