TechWayFit
● Demystifying Generative AI for Developers

How Transformers Actually Work

Tech Buddy May 27, 2026 5 min read
How Transformers Actually Work

A teammate plugged an LLM into our support chatbot. The demo looked great. Then a real user pasted a 3,000-word log, and the bot latched onto one line containing "timeout" and ignored everything else. It gave a generic retry tip. Not wrong, just useless.

That failure wasn't about model size. It was about how the model decides which words matter. That mechanism is the transformer, and once you see it, a lot of odd behaviour (prompt order mattering, long inputs getting expensive, the model fixating on the wrong detail) stops being odd.

💡
The short version

A transformer lets every token look at the other tokens and decide how much each one matters right now. That is attention. Stack many layers of it, add a small network per token, and repeat once per generated token. The price is that attention work grows with the square of the context length, which is why long prompts cost real money and time.

Why attention?

Older sequence models (RNNs, LSTMs) read text one token at a time and carried a running memory forward. They were slow to train and forgot things far back. A transformer instead lets every token consult every other token directly, and it does that for the whole sequence in parallel. That made training on huge datasets practical, and it is why long-range connections ("the server that we restarted yesterday ... it crashed") work so much better.

The parts, in order

  1. Tokens to vectors. Text is split into tokens, and each becomes an embedding vector. A position signal is added so the model knows order.
  2. Self-attention. Each token produces a query ("what am I looking for?"), a key ("what do I offer?") and a value ("what do I carry?"). Comparing one token's query with every key gives attention weights, and the token's new representation is a weighted mix of the values. Several heads do this in parallel, each free to pick up a different kind of relationship.
  3. Feed-forward network. A small network transforms each token's representation on its own.
  4. Residual connections and normalisation keep deep stacks trainable.
  5. Repeat the attention-plus-feed-forward block many times, then turn the final vectors into scores for every vocabulary token and sample the next one.
flowchart LR
                          A[Text] --> B[Token ids]
                          B --> C[Embeddings plus position]
                          C --> D[Block: attention, then feed-forward]
                          D --> E[... repeated N times ...]
                          E --> F[Scores over vocabulary]
                          F --> G[Sample next token]
                          G -->|append| B
                      

Attention in twenty lines

This shows the core idea on three tokens with tiny made-up vectors. Real models use hundreds or thousands of dimensions and learn separate projections for queries, keys and values. Chat models also mask future tokens, so each position only looks backwards. The arithmetic is the same.

import math
                      
                      # Three tokens, each already turned into a tiny 2-D vector (made-up numbers)
                      tokens = ["the", "server", "crashed"]
                      x = [[1.0, 0.0], [0.2, 1.0], [0.1, 0.9]]
                      
                      def softmax(xs):
                          m = max(xs)
                          e = [math.exp(v - m) for v in xs]
                          return [v / sum(e) for v in e]
                      
                      def dot(a, b):
                          return sum(p * q for p, q in zip(a, b))
                      
                      # Real models learn separate projections for queries, keys and values; here x plays all three roles.
                      for i, q in enumerate(x):
                          scores = [dot(q, k) / math.sqrt(len(q)) for k in x]                  # how well does token i match each token?
                          weights = softmax(scores)                                            # scores -> attention weights
                          mixed = [sum(w * v[d] for w, v in zip(weights, x)) for d in range(2)]  # weighted blend of the values
                          print(f"{tokens[i]:>8}", [f"{w:.2f}" for w in weights], "->", [f"{m:.2f}" for m in mixed])
                      

Each row says how much that token attends to "the", "server" and "crashed", and what it ends up as after the blend. "server" and "crashed" attend to each other more than to "the". Nothing here is programmed in; in a real model the weights come out of training.

The roundtable

Picture a code review. Every developer (token) has a sticky note (embedding). In each round (layer), each developer decides whom to listen to: the security engineer on auth code, the ops engineer on deployment. Several conversations run at once (heads). After the round, everyone folds what they heard into their note (the feed-forward step and residual). After enough rounds, there is a rich enough picture to write the next line of code.

Two costs that matter

  • Prefill versus decode. Reading your prompt can be done in parallel, so it is fast per token. Generating the answer happens one token at a time and tends to be limited by memory bandwidth. That is why the first token can arrive quickly after a long wait or slowly after a short one, depending on where the time goes.
  • The KV cache. During generation the model stores the keys and values of tokens it has already processed so it doesn't recompute them. That makes streaming snappy and consumes GPU memory in proportion to context length.

Plain attention compares every token with every other, so the work grows roughly with the square of the sequence length. Optimised variants soften this, but the practical message holds: every extra token you send costs something.

The solution architect's view

You'll never tune an attention head, but the mechanism leaks into design decisions in several places.

Context is a budget, not a bucket

A bigger context window doesn't mean you should fill it. Each token adds to cost, latency and cache memory, and long, crowded prompts make it more likely the model overlooks something. Decide what earns a place in the prompt and trim the rest. Retrieval exists partly for this reason.

Order and structure are design choices

Models don't weigh every part of a long prompt equally. Material at the start and end tends to be used more reliably than material buried in the middle. Put instructions and the critical facts where they will be seen, label retrieved chunks clearly, and test with realistic long inputs rather than short demos. The 3,000-word log in the opening story was exactly that test, and the demo skipped it.

Know where your latency comes from

Separate time spent reading the prompt from time spent generating. Long prompts hurt the first; long answers hurt the second. They are fixed by different levers (trimming and caching on one side, output limits and smaller models on the other), so measure both before deciding what to optimise.

Test at the edges

Include very long inputs, noisy logs and cases where the key fact sits in the middle. Make behaviour at context limits an explicit requirement, such as truncate, summarise or reject, so it isn't decided by accident in production.

Questions for the design review

  1. What goes into the prompt, and what is the justification for each part?
  2. Where are the critical instructions and facts placed, and have we tested long inputs?
  3. What happens when an input exceeds the context limit?
  4. Are we measuring prompt-processing time and generation time separately?
  5. What does the KV cache cost at our expected concurrency?

Key Takeaways

  • Attention lets each token weigh every other token; stacking layers builds richer meaning.
  • Queries, keys and values are the mechanism; the weights are learned, not hand-written.
  • Generation is token by token, with a KV cache to avoid recomputation.
  • Attention cost grows with context length, so every token you send has a price.
  • Prompt order and structure affect what the model uses. Test with long, messy input.
  • Treat context as a budget and measure prefill and decode separately.

Comments

Loading comments…

Leave a comment