TechWayFit
● Demystifying Generative AI for Developers

LLMs Explained Simply

Tech Buddy May 27, 2026 6 min read
LLMs Explained Simply

The chat feature demo went well. Then traffic doubled, latency doubled with it, and finance started asking about the GPU bill. Product wants "the smarter model". You're looking at a dashboard and wondering whether smarter means slower, pricier, or both.

This is the moment the science-fiction framing stops helping. An LLM isn't a mind. It is a very large pile of numbers, arranged in layers, that learned to guess the next token. Once you hold on to that, questions about size, speed and cost get much easier to answer.

💡
The short version

An LLM predicts one token at a time, over and over. Its parameters hold the patterns it learned. Everything you care about operationally (memory, latency, cost) comes from how many parameters it has and how long the conversation is. Model size is a systems decision before it is a quality decision.

Just a very good next-word guesser

Given the tokens so far, an LLM scores every token in its vocabulary and picks one. It doesn't look facts up. It continues the text the way its training data suggests. To see how little machinery that needs, here is a toy that "learns" by counting which word follows which:

import random
                      from collections import defaultdict, Counter
                      
                      corpus = ("a 404 means not found . a 500 means server error . "
                                "a 404 means the page is missing . a 200 means ok .").split()
                      
                      follows = defaultdict(Counter)
                      for prev, nxt in zip(corpus, corpus[1:]):
                          follows[prev][nxt] += 1               # "training": count what follows what
                      
                      def next_token(prev):
                          options = follows[prev]
                          return random.choices(list(options), weights=options.values())[0]
                      
                      random.seed(1)
                      text = ["a"]
                      while text[-1] != "." and len(text) < 12:
                          text.append(next_token(text[-1]))
                      print(" ".join(text))
                      print(dict(follows["means"]))
                      

It writes a fluent sentence, and the sentence is wrong: "a 404 means ok". The counting table has no idea what a 404 is. It only knows what tends to follow "means". A real LLM is vastly better at the same game, with attention over the whole context instead of a single previous word, but the game is the same. That is also where hallucinations come from, which the hallucination post picks up.

One trip through the loop

Each token goes through the same stages, many times per answer.

  1. Tokenise: split the text into subword pieces and map them to integer ids.
  2. Embed: turn each id into a vector.
  3. Transformer layers: attention decides which earlier tokens matter; a feed-forward network transforms the result. A KV cache stores the work already done for earlier tokens, so each new token doesn't redo it.
  4. Scores to probabilities: project to one score per vocabulary token, apply softmax, and sample.
  5. Append and repeat, streaming each token to the user as it appears.

The heavy compute is the attention and feed-forward work across every layer, for every generated token. Everything else is cheap by comparison. The transformers post opens that box.

Where the memory goes

Two things have to fit on the GPU: the weights, which are fixed, and the KV cache, which grows with every token in every active conversation. Teams routinely budget for the first and get surprised by the second.

GIB = 1024 ** 3
                      
                      def weights_gib(params_billion, bytes_per_param):
                          return params_billion * 1e9 * bytes_per_param / GIB
                      
                      def kv_cache_gib(layers, kv_heads, head_dim, tokens, bytes_per_value=2):
                          per_token = 2 * layers * kv_heads * head_dim * bytes_per_value   # keys + values
                          return per_token * tokens / GIB
                      
                      # Llama-3.1-style shapes (approximate; check your model's config)
                      for name, params, layers in [("8B", 8, 32), ("70B", 70, 80)]:
                          fp16 = weights_gib(params, 2)
                          int4 = weights_gib(params, 0.5)
                          kv = kv_cache_gib(layers, kv_heads=8, head_dim=128, tokens=8192)
                          print(f"{name}: weights {fp16:.0f} GiB (fp16) / {int4:.0f} GiB (4-bit), "
                                f"KV cache {kv:.1f} GiB per 8k-token conversation")
                      

The numbers are approximate and assume Llama-3.1-style shapes. The pattern is what matters. Weights dominate at the start, and quantising (storing them in fewer bits) shrinks them a lot. The KV cache then decides how many conversations you can serve at once, and it scales with context length.

Small, medium or large?

ClassOperationsLatency and throughputUsually good for
Small (about 7-8B)One modest GPU, even CPU for developmentFast first token, high concurrency, cheap to scaleChat UX, classification, extraction, answers grounded in retrieved text
Medium (about 13-34B)One large GPU, or severalSlower, lower concurrency per nodeBetter summarisation, multilingual work, coding hints
Large (about 65-70B and up)Multi-GPU, sharding, tuned batchingHigher latency and cost per requestHard reasoning, complex code, long-context synthesis

Parameter count is not the whole story. Training data, recipe, context length and alignment all matter, and a well-tuned small model with good retrieval beats a lazily prompted large one on plenty of tasks.

Base or aligned?

A base model is the raw next-token predictor: hand it text and it continues the text, which may mean a quote, a list, or a tangent. An aligned (instruction-tuned) model has been trained further to follow instructions, use roles like system and user, refuse unsafe requests, and produce tidier output. For anything user-facing you almost always want the aligned one. Neither guarantees a strict schema or correct facts, so you still validate and ground.

The solution architect's view

Choosing a model is a capacity-planning and cost decision, and I'd treat it like one rather than a leaderboard contest.

Start from the service levels, not the model

Write down what the experience needs: time to first token, tokens per second, the 95th-percentile latency under realistic concurrency, and a cost ceiling per request. Then see which model classes can meet them. A model that wins every benchmark and misses the latency target is the wrong model.

Design for tiers, not one model

A common and economical pattern is a fast small model as the default and a larger one for hard cases, selected by a simple rule, a classifier or a confidence signal. Put that routing behind an abstraction so models can change without touching callers. The decision to escalate should come from your own evaluations, not from hunches.

Plan capacity on weights and cache

Size the GPU for the weights plus the KV cache at the concurrency and context length you expect. Cap output tokens, trim prompts and watch cache usage per GPU. When the cluster saturates, degrade gracefully (shorter context, smaller model) instead of failing.

Keep an exit

Many providers and self-hosting stacks expose an OpenAI-compatible interface, which makes switching cheaper. Prompts, though, are tuned to specific models, so budget for re-evaluation whenever you change one. Decide early whether data residency, privacy or cost points you toward a hosted API or your own infrastructure, and what the migration path looks like either way.

Questions for the design review

  1. What latency and cost targets must the model meet, and at what concurrency?
  2. Would a small model plus retrieval do the job? What evidence says it wouldn't?
  3. How is escalation to a larger model decided, and who owns that rule?
  4. Have we sized for the KV cache as well as the weights?
  5. What would it take to swap this model for another one?

Key Takeaways

  • An LLM scores the next token, appends it, and repeats. That is the whole loop.
  • A fluent sentence tells you nothing about whether it is true.
  • Weights set the entry cost; the KV cache sets your concurrency.
  • Small plus retrieval covers more than people expect. Escalate on evidence.
  • Use aligned models for user-facing work, and still validate their output.
  • Pick the model from your latency and cost targets, and keep the ability to change it.

Comments

Loading comments…

Leave a comment