Production Architecture for GenAI Systems

The prototype was one function: take the question, call the model, return the text. It worked. Then it needed authentication, per-customer limits, access rules on the data, answers you could trust, a way to find out why one went wrong, a fallback for when the provider had an outage, and a bill finance could forecast. The function became a system.

This final post pulls the series together. The model is one box in the diagram. Everything around it is what makes it production-ready.

A reference architecture

flowchart TD
                          U[Client] --> G[Gateway: auth, quotas, rate limits]
                          G --> IG[Input guard: limits, PII, injection checks]
                          IG --> O[Orchestrator: workflow or agent]
                          O --> R[Retrieval: filters, hybrid search, rerank]
                          O --> T[Tools with least privilege]
                          O --> MR[Model router and gateway]
                          MR --> LLM[LLM providers]
                          O --> V[Verifier: schema, citations, policy]
                          V --> OG[Output guard]
                          OG --> U
                          O -.-> OBS[Observability: traces, tokens, cost]
                          O -.-> CFG[Config: prompts, models, flags, versions]
                          V -.-> HITL[Human review queue]
                      

Each box maps to something earlier in the series: retrieval to RAG and vector search, the verifier to the hallucination controls, tools to tool use, and the model router to the cost work.

The request path in code

This skeleton shows the habits that matter: a trace id and version tags on every request, verification before anything reaches the user, a graceful fallback, and one structured log line per request.

import time, uuid, json
                      
                      def guard_input(q):   return q if len(q) < 2000 else None           # basic input limits
                      def retrieve(q):      return [("S1", "Refunds are available within 30 days.")]
                      def generate(q, ctx): return {"text": "Refunds within 30 days.", "source": "S1", "quote": "Refunds are available within 30 days."}
                      def verify(ans, ctx): return any(ans["quote"] in text for _, text in ctx)
                      FALLBACK = "I can't answer that reliably. A human agent will follow up."
                      
                      def handle(question):
                          trace = {"trace_id": str(uuid.uuid4())[:8], "prompt_version": "v12", "model": "model-x"}
                          t0 = time.perf_counter()
                          try:
                              q = guard_input(question)
                              if q is None:
                                  raise ValueError("input rejected")
                              ctx = retrieve(q)
                              ans = generate(q, ctx)
                              ok = verify(ans, ctx)
                              trace.update(sources=[s for s, _ in ctx], verified=ok)
                              reply = ans["text"] if ok else FALLBACK                  # degrade gracefully, never pass unverified output
                          except Exception as exc:
                              trace["error"] = str(exc)
                              reply = FALLBACK
                          trace["latency_ms"] = round((time.perf_counter() - t0) * 1000, 1)
                          print(json.dumps(trace))                                      # one structured log line per request
                          return reply
                      
                      print(handle("How long do I have to request a refund?"))
                      

Each stage here is a stub. The pattern is what to keep: no unverified output reaches the user, any failure degrades to a safe answer, and everything needed to debug the request is recorded.

What "production-ready" adds

• Resilience: timeouts, retries with backoff, circuit breakers, and a second model or provider to fall back on.
• Graceful degradation: if retrieval or the model is down, serve a cached answer, a simpler path or a handover instead of an error page.
• Cost control: token budgets, caching, model routing and per-tenant quotas.
• Safe change: versioned prompts and models, canary or shadow releases, and an evaluation gate.
• Data protection: redaction, retention limits and clear rules on what leaves your boundary.

The solution architect's view

This is where an architect earns the fee: the model gets the attention, and the surrounding system decides whether it can be run, trusted and afforded.

Start from risk, then pick the controls

Classify each use case by the cost of a wrong answer, then set the control level to match: from "a human reads it first" up to "verification, approval and audit for every action". Don't give every feature the heaviest process, and don't give a high-stakes feature the lightest one.

Put the model behind an interface

A gateway or router lets you change models, add fallbacks, enforce quotas, redact data and record usage in one place. It is also your exit strategy: models, prices and terms will change, and you'll want to move without rewriting every caller.

Write down the non-functional requirements

Targets for answer quality (from your evaluation set), invented-answer rate, time to first token, 95th-percentile latency, availability, cost per request and recovery behaviour. Review them like any other SLO, and include provider limits and outages in the failure analysis.

Observe more than latency

Trace every request with the prompt version, model, retrieved source ids, token counts, verification result and cost. Dashboards for abstention rate, verification failures and spend per tenant often warn you of problems before users do.

Govern the data and the decisions

Know what personal or confidential data enters prompts, indexes and logs, where it is stored and for how long. Name owners for content, prompts, evaluation sets and the model choice itself. Align with your organisation's AI policy and any regulation that applies to your sector.

Questions for the design review

1. What is the cost of a wrong answer here, and do the controls match it?
2. What happens when the model, retrieval or a tool is down?
3. How do we change a model or prompt safely, and roll it back?
4. Can we explain any answer from the logs, and what do we cost per request?
5. Who owns content, prompts, evaluation and the model decision?

That completes the core series. For implementation detail, see AI for .NET.