Overview: Token Economics

Asteria I am weirdly relieved we’re doing this one. My tabs are all little token calculators right now, which is maybe the least glamorous mood a Wednesday can have.

Draco That is very Exploring Next. Not glamorous, somehow load-bearing, and guaranteed to make you say procurement logic before the episode ends.

Asteria I deserve that. But Draco, we have been saying “token economics” like it’s obvious for months. GPT-5.6 pricing, context windows, the open-source model thing, even prompt engineering. We keep pointing at the same invisible meter.

Draco Yeah. And it’s one of those concepts where if you already know it, you compress the whole idea into one sentence. If you don’t, every AI product decision looks mystical. Why did the bill jump? Why did the model forget the beginning? Why does a shorter prompt sometimes make the answer worse?

Asteria So this is episode six seventy-five, apparently, and we’re finally paying our own tiny debt. Not a grand theory episode. More like, what are we actually buying when we buy model usage?

Draco The cleanest picture for me is a mailroom with tiny slips of paper. The paragraph gets chopped into slips. Every slip that comes in has to be handled. Every slip the model writes back also has to be handled. And the desk only has room for a fixed number of slips at once.

Asteria Okay, I like that. The model is not reading “a document” in the way I read a document. It is running a metered mailroom where the currency is these little language slips.

Draco Exactly. Those slips are tokens. A token can be a whole word, part of a word, punctuation, whitespace, sometimes a piece of an emoji. Token economics is just the fact that tokens are the unit of computation, cost, and memory. If you understand the slips, pricing pages stop looking like dark magic.

Asteria Right.

Draco And before we get too far, tokenization is the chopping step. We did the full rabbit hole in episode six oh one. Quick version: text is converted into token IDs before the model sees it, and those IDs are what the model actually processes.

Asteria You did make me respect the fussy label-maker, annoyingly. But the part that always trips people up is that a token is not just “a word.” Like, “cat” might be one slip, but a rare technical term might be three slips, and an emoji might be more than anyone emotionally wanted.

Draco Right, and different models can use different chopping rules. A common method is byte-pair encoding, usually said as B P E. It learns frequent combinations, so common chunks become single tokens. That keeps sequences shorter than pure character-by-character encoding, without needing every possible word as its own token.

Draco But because the tokenizer is model-specific, the same text can have different token counts depending on the system. That’s why word count is a bad billing estimate. It’s like estimating postage by vibes instead of weighing the package.

Asteria There’s our off-topic riff: “postage by vibes” is absolutely a doomed startup. You upload a manuscript, it says, “feels expensive,” and charges your card.

Draco The enterprise tier adds a dashboard with a big red aura around the invoice.

Asteria And somehow I would still click it if the onboarding was good. Okay, back to slips before you accuse me of launch-day optimism again.

Draco Too late. So, after tokenization, the model processes the input tokens and then generates output tokens. Autoregressive generation is the key prerequisite there. We did the full Overview in episode six thirty, but the short version is: the model writes one token at a time, each new token based on the tokens already present.

Asteria So in the mailroom picture, the incoming pile is the prompt, and the outgoing pile is the answer. But the answer gets written slip by slip, not as one finished essay teleported onto the desk.

Draco Yes. And that matters for cost and latency. Input tokens can be processed relatively efficiently, especially in batches. Output tokens are produced step by step, so long answers usually take longer and often cost more per token. Public API prices commonly separate input and output for exactly that reason.

Draco As of mid twenty twenty-six, you see a huge spread. Some hosted or smaller models show prices down around cents per million tokens. On the other end, something like Claude Opus 4 has been listed around fifteen dollars per million input tokens and seventy-five dollars per million output tokens. That gap is the economics showing its teeth.

Asteria And this is where product teams stop caring about benchmark poetry and start asking boring questions. How many slips do we send? How many slips come back? How many attempts before the work is accepted? Because the “best” model can lose if the workflow needs thousands of cheap retries.

Draco That connects directly to the GPT-5.6 family we looked at. Sol was the halo model, Terra was the boring workhorse, Luna was the abundance tier. The important part wasn’t just capability. It was price shape: Sol at five dollars input and thirty dollars output per million tokens, Terra at two-fifty and fifteen, Luna at one and six.

Asteria I am accepting your Terra respect with grace. By grace I mean I’m smiling in a very restrained way.

Draco You earned that one. Terra made the token math legible. If the work is repetitive, tool-heavy, or good enough at that tier, the default-buy model is the one that clears the task for less money and less waiting. Not the one with the fanciest press line.

Asteria This is why “tokens per dollar” is useful but incomplete, right? Because if Luna is cheap but needs four tries, and Terra is pricier but lands it once, Terra may be cheaper per accepted result.

Draco Yes. The better unit is useful work per dollar. Tokens are the meter, but the job still has to succeed. Count model calls, tool calls, retries, latency, and human review. Otherwise you just optimize the prettiest number in the spreadsheet.

Draco Now the other big piece is the desk size: the context window. We did a full Overview on context windows in episode six hundred. Quick gloss: the context window is the maximum number of tokens the model can consider at once, including instructions, documents, chat history, tool outputs, and the answer it is producing.

Asteria So the finite tray. If the tray holds one hundred slips, I cannot hand you one hundred and fifty slips and ask you to “just be cool about it.” The extra slips either don’t fit, or something has to be left out.

Draco Exactly. And in real models the numbers are much bigger, but the limit is still real. Gemini 3 models have been described with one million token input windows and up to sixty-four thousand output tokens. Gemini 3.1 Pro keeps that one million token class. GPT-5.4 xhigh has been listed around one point one million tokens. Llama 4 Scout has advertised ten million tokens.

Asteria Those numbers sound absurdly roomy. Like, ten million slips is no longer a desk. That is a warehouse with opinions.

Draco It is. But bigger windows are not free warehouses. More tokens can mean more compute, more memory pressure, and more latency. Even when the architecture supports it, the product question is whether stuffing everything in helps enough to justify the bill and the delay.

Asteria Right, because I can imagine a team saying, “Great, give the model every support ticket ever,” and then discovering they bought a very expensive fog machine.

Draco Expensive fog machine is cruel, but fair. Large context helps when the relevant information is genuinely spread across a long document or conversation. It is wasteful when the model only needed three paragraphs and got a filing cabinet.

Asteria This is where retrieval-augmented generation shows up, yeah? Define that before I do my annoying product-person translation.

Draco Retrieval-augmented generation, or R A G, means the system searches a knowledge base first, pulls back relevant snippets, and gives only those snippets to the model. In the mailroom analogy, you don’t dump the whole archive onto the tray. You send a clerk to fetch the likely useful slips.

Asteria Product translation: cheaper, faster, and usually easier to debug, as long as the retrieval step doesn’t miss the important bit. It’s not magic. It’s selectivity.

Draco Yes, and selectivity is the heart of token economics. Every token you include is a bet that the information helps more than it costs. That applies to chat history, legal documents, code files, examples, tool logs, everything.

Draco It also applies to formatting. JSON, code, tables, indentation, repeated labels, non-English text, whitespace, and special characters can all change the slip count.

Asteria I want to slow that down because it feels like a gotcha, not a principle. If I send a neat-looking block of structured data, I might assume it’s compact because humans scan it easily. But the tokenizer may see punctuation and separators that all count.

Draco That’s the practical trap. Human readability and token efficiency overlap sometimes, but they are not the same thing. A tidy table may be easy for you and costly for the model. A compact sentence may be cheaper but ambiguous. You’re balancing cost against clarity.

Asteria And this connects back to prompt engineering from episode six sixty-one, where we landed on prompts as interface design. The prompt is not just “words I typed.” It’s the shape of the work, exposed to the model, inside a token budget.

Draco Exactly. Few-shot prompting means showing the model a few examples of the pattern you want. That adds tokens, so it costs more. But if those examples prevent bad outputs or retries, the total economics can improve.

Asteria So shorter is not automatically better. I can cut instructions down to four cryptic words and celebrate my token savings, then spend ten minutes repairing the weird answer.

Draco Yes. Token efficiency is not token starvation. The goal is not minimum tokens. The goal is minimum wasted tokens for the quality bar you need.

Draco That’s the part I’d keep. A shorter prompt that causes a bad answer is fake savings. A longer prompt that prevents retries might be cheap. Token economics is boring only until the invoice, the latency, or the context limit tells you it was the architecture all along.

Asteria Beautiful. We finally did the tiny-slip episode, Draco. I’m going to go close ninety tabs and pretend that counts as optimization.