max_tokens isn't free: how a token budget became a 10-minute timeout

It started with a pipeline that wouldn't fail honestly.

A run that normally finished in well under a minute would, every so often, just… sit. No error, no output, the process alive and waiting. Two minutes. Five. Sometimes it came back, sometimes it timed out somewhere downstream and took the whole job with it. Intermittent, so of course it only showed up when I wasn't watching.

The change that introduced it was boring: I'd raised max_tokens on one of the LLM calls so a longer generation had room to finish. That's it. More headroom for output. It shouldn't slow anything down — you only pay for the tokens you actually generate, right?

Right on cost. Wrong on latency, in a way I didn't see coming.

What max_tokens actually does to your request

max_tokens is a ceiling, not a target. The model stops when it's done or when it hits the ceiling, whichever comes first. So a bigger ceiling doesn't make a short answer slower to produce.

What it does change is how long the client is willing to wait. The Anthropic SDK sizes the request timeout off max_tokens. Roughly:

timeout_seconds = (3600 * max_tokens) / 128000     # capped at 600s (10 min)

Plug in some numbers:

• max_tokens: 1024 → ~29 second timeout
• max_tokens: 4096 → ~115 seconds
• max_tokens: 16000 → ~450 seconds (7.5 minutes)
• max_tokens: 64000 → capped at 600 seconds (10 minutes)

The logic is fair enough: a request that's allowed to generate 16k tokens might genuinely need minutes, so don't kill it early. But it means the moment I set max_tokens: 16000, I also quietly told the client: wait up to seven and a half minutes before giving up.

So when a call stalled — a slow upstream, a model that got stuck, a network hiccup — it no longer failed fast. It sat in that 7.5-minute window, holding up everything behind it. The bug wasn't a crash. It was patience I'd configured by accident.

The part that bites: non-streaming

There's a second edge here. For large generations, a single non-streaming request can run past the SDK's allowed timeout entirely, and you're expected to stream instead. The rough line I hit: non-streaming is fine for smaller budgets, but once you're generating large outputs you have to switch to streaming.

That's not arbitrary. Streaming changes the failure mode. Instead of waiting for the entire response inside one timeout and hoping it arrives, you get tokens as they're produced. You see progress, you can set your own idle timeout ("no token in N seconds → bail"), and a long-but-healthy generation stops looking identical to a dead one.

The fix

Three changes, in order of how much they helped:

1. Set max_tokens to what the call actually needs, not a "safe" round number. A step that returns a verdict and a sentence does not need 16k. Most of my calls dropped back to 1–2k, and their timeout windows shrank with them. The stalls mostly vanished, because failing calls now failed in seconds instead of minutes.

2. Stream anything that can genuinely be large. Long generations go through the streaming path with an idle timeout I control, so "slow" and "dead" no longer look the same.

3. Treat the timeout as something you set on purpose. Don't inherit a 10-minute wait. If a call should never take more than 30 seconds, say so explicitly — independent of max_tokens.

The actual lesson

max_tokens looks like a cost knob. It's also a latency knob, because the client uses it to guess how long to wait. Set it high "just in case" and you're not buying safety — you're buying a long, quiet hang the next time something upstream misbehaves.

Set it to the size of the answer you expect. Stream when the answer can be big. And don't let a number you picked for headroom decide how long your system waits before it admits something's broken.

---

Neville James Achieng builds LLM and voice systems in Nairobi. Notes and code: github.com/Neville777.