TechWayFit
● Demystifying Generative AI for Developers

Foundations: What Exactly Is Generative AI?

Tech Buddy May 27, 2026 5 min read
Foundations: What Exactly Is Generative AI?

Your team adds a "smart" summarisation feature to a boring, dependable microservice. Yesterday the JSON tests were green. Today the same request returns a perfectly reasonable summary, worded differently, and a downstream parser falls over because one field came back as headline instead of title. Nothing in your code or infrastructure changed. The pager goes off anyway.

You haven't hit a bug. You've crossed a border: from software that computes the answer to software that samples an answer.

💡
The short version

Traditional code maps the same input to the same output. Generative AI picks the next token from a probability distribution, so the same input can produce different, equally valid outputs. That gives you flexibility and takes away exact repeatability. The engineering answer is a deterministic shell around a probabilistic core.

Two kinds of software

Classic software is a function: f(x) = y. Same branches fire, same output comes out. That is why billing and cryptography are built this way.

A generative model is closer to P(y | x): given this input, here is how likely each possible continuation is. It then samples from that distribution. During training it adjusted billions of numbers so that its predicted next token matched what people actually wrote next, over an enormous amount of text. What it learned is how text tends to continue, which is a different thing from knowing what is correct.

Traditional codeGenerative AI
Same input twiceSame outputDifferent, equally valid outputs
How you testExact assertionsSchema checks, semantic checks, golden sets
What failure looks likeExceptions, wrong branch, null referencePlausible but wrong text, format drift
Cost modelCPU and I/O, predictableTokens in and out, GPU time, variable
Your control knobsConfig, feature flagsPrompt, temperature, sampling settings, structured output

What happens when you hit send

The loop is short, and everything in the rest of this series hangs off it.

flowchart TD
                          A[Prompt] --> B[Tokenizer: text to token ids]
                          B --> C[Transformer layers: mix context]
                          C --> D[Scores for every token in the vocabulary]
                          D --> E[Softmax with temperature: scores to probabilities]
                          E --> F[Sample one token]
                          F --> G{Done?}
                          G -->|no| H[Append token to the context]
                          H --> C
                          G -->|yes| I[Return the text]
                      

The model never writes an answer in one go. It produces one token, appends it to the context, and goes again, which is why long answers are slow and expensive.

Turn the dial

Temperature rescales the scores before they become probabilities. Low values sharpen the peak, so the top token wins almost every time. High values flatten it, so unlikely tokens get a chance. Here it is with a four-token vocabulary:

import math, random
                      
                      tokens = ["alpha", "beta", "gamma", "delta"]
                      logits = [3.2, 2.1, 1.5, 0.1]            # raw scores a model might produce
                      
                      def probs(temperature):
                          scaled = [x / temperature for x in logits]
                          top = max(scaled)                     # subtract the max for numerical stability
                          exps = [math.exp(x - top) for x in scaled]
                          total = sum(exps)
                          return [e / total for e in exps]
                      
                      random.seed(7)
                      for t in (0.2, 1.0, 2.0):
                          p = probs(t)
                          print(f"T={t}: " + " ".join(f"{w}={x:.2f}" for w, x in zip(tokens, p)))
                          print("    ", random.choices(tokens, weights=p, k=8))
                      

At T=0.2 it picks alpha every time. At T=2.0 it wanders. Temperature isn't "how creative the model feels". It is the shape of the distribution you sample from.

A deterministic shell around a probabilistic core

You can't make the core deterministic, so you build the edges to be. Pin what you can, constrain the output, validate what comes back, and fail loudly instead of passing junk downstream.

import json
                      
                      def call_model(prompt: str, attempt: int) -> str:
                          """Stand-in for a real LLM call: the first attempt drifts, the second behaves."""
                          if attempt == 0:
                              return '{"headline": "Revenue up 12%"}'          # wrong key name
                          return '{"title": "Revenue up 12%", "summary": "Q3 revenue grew 12% year on year."}'
                      
                      REQUIRED = {"title", "summary"}
                      
                      def summarise(text: str, max_attempts: int = 3) -> dict:
                          for attempt in range(max_attempts):
                              raw = call_model(f"Summarise as JSON with keys title, summary:\n{text}", attempt)
                              try:
                                  data = json.loads(raw)
                              except json.JSONDecodeError:
                                  continue
                              if set(data) == REQUIRED and all(isinstance(v, str) and v for v in data.values()):
                                  return data
                          raise ValueError("model never produced valid output")   # fail loudly, don't pass junk downstream
                      
                      print(summarise("Quarterly report text..."))
                      

The stand-in model got the key name wrong on its first attempt. The validator noticed, the loop retried, and the caller never saw the bad version. Swap call_model for a real client and the structure stays the same.

The solution architect's view

The question I'd ask first isn't "which model?" but "does this part of the system want a function or a distribution?" Most of the design follows from that.

Where generative AI earns its place

It is strong where language is the interface and "good, but not identical every time" is acceptable: summarising, drafting, classifying messy text, extracting fields from documents, translating intent into a query. It is a poor fit for arithmetic, authorisation decisions, and anything that needs the same auditable answer every time. Keep those in ordinary code and let the model sit at the edges, turning language into structure and back again.

Treat the model, prompt and parameters as configuration

Behaviour is a function of the model version, the prompt, the sampling settings and any retrieved content. Pin all four, put the prompt in version control, and review changes to it like code. Be careful with model aliases that move: a provider can update what "latest" points to and change behaviour while your code stays identical.

Write the variability into your requirements

"The summary should be good" can't be tested. These can:

  • At least 99% of responses must pass schema validation after at most two retries.
  • No required fact from the source may be missing in the golden set.
  • Cost per request stays under a stated ceiling at the 95th percentile of input length.
  • Time to first token and total latency fit the user experience you promised.

Change how you test and operate

Exact-match assertions give way to schema checks, semantic checks and a golden set you re-run on every change to model, prompt or retrieval. Observability grows to include token counts, validation failure rate, retry rate and cost, alongside latency and errors. Make cache keys include the model version and sampling settings, not just the input.

Questions for the design review

  1. Which parts of this flow need an exact, repeatable answer, and are they kept out of the model?
  2. What validates the model's output before anything downstream trusts it?
  3. How do we pin and change the model, prompt and parameters?
  4. What is the acceptable failure rate, and how will we measure it?
  5. What does one request cost at realistic volume?

Key Takeaways

  • Generative AI samples from a distribution, so identical inputs can give different outputs.
  • Generation is one token at a time; cost and latency follow token counts.
  • Temperature reshapes the distribution. It doesn't add or remove knowledge.
  • Wrap the model in code that validates, retries and fails loudly.
  • Pin model, prompt and parameters, and test with schemas and golden sets instead of exact strings.
  • Give the model the language work; keep exact, auditable decisions in ordinary code.

Comments

Loading comments…

Leave a comment