AI System Design

• Starting from requirements & constraints

  Start from the use case, success criteria, constraints, traffic, latency requirements, quality requirements, privacy, and cost budget.

• Prompting, RAG, agents or fine-tuning

  Decide deliberately between prompting, RAG, agents, fine-tuning, or combinations of them.

• Quality ↔ latency ↔ cost

  Reason about the central trade-offs: quality ↔ latency ↔ cost, plus complexity and reliability.

• Baseline before sophistication

  Establish a simple baseline before introducing sophisticated architecture.

Outcome · Turn a product requirement into a defensible AI architecture rather than starting with a favorite technology.

• Where knowledge should come from

  Decide what knowledge should come from model knowledge, prompt/context, RAG, memory, or external tools.

• The full context pipeline

  Design the full context pipeline: ingestion → retrieval → reranking → context assembly → generation.

• Combining retrieval approaches

  Combine vector, keyword, graph, and other retrieval approaches where justified.

• Freshness, access control & provenance

  Handle freshness, updates, access control, scale, and source provenance.

• Retrieved dynamically or held in context

  Decide what information should be retrieved dynamically versus kept permanently in the model's context or external state.

Outcome · Design the knowledge layer that supplies the model with the right information at the right time.

• What a model gateway does

  Understand the role of a model gateway as the abstraction between application code and model providers.

• Routing on task, cost & risk

  Route requests based on task type, capability, latency, context length, cost, modality, or risk.

• Model cascades

  Build model cascades where cheaper/faster models handle easy requests and stronger models handle difficult ones.

• Provider fallbacks & quality risk

  Add model/provider fallbacks and understand the quality risks introduced by switching models.

• Centralising auth, limits & tracking

  Centralize authentication, usage tracking, retries, rate limits, and provider abstraction where appropriate.

Outcome · Avoid designing applications around a single hard-coded model.

• Breaking latency into components

  Break latency into components such as retrieval, TTFT, model generation, tool calls, and network overhead.

• Streaming for perceived latency

  Use streaming to improve perceived latency.

• Parallelising independent work

  Parallelize independent operations and avoid unnecessary sequential LLM calls.

• When a smaller model will do

  Understand when smaller/faster models can replace larger models.

• Context reduction & batching

  Apply prompt/context reduction, batching, and inference-service optimizations where appropriate.

• Throughput & utilization at scale

  Reason about throughput, latency, and utilization at scale.

Outcome · Identify where an AI application is slow and choose the appropriate optimization instead of simply upgrading infrastructure.

• The four caches in an AI system

  Understand the different caches available in an AI system: exact-response caching, prompt/context caching, retrieval caching, and semantic caching.

• What is safe to cache

  Decide what is safe to cache and define appropriate expiration/invalidation strategies.

• Semantic caching

  Use semantic caching when different queries can reasonably share an answer.

• Freshness vs latency & cost

  Understand the trade-off between freshness, quality, latency, and cost.

• Measuring the hit rate

  Measure actual cache hit rate and savings rather than assuming caching helps.

Outcome · Use caching as an architectural optimization rather than an afterthought.

• Designing for every failure

  Design for model failures, provider outages, tool failures, rate limits, malformed outputs, and retrieval failures.

• Timeouts, circuit breakers & queues

  Add timeouts, retries, fallbacks, circuit breakers, queues, and rate limits where appropriate.

• Guardrails at system boundaries

  Place guardrails at important system boundaries rather than relying entirely on the model.

• Graceful degradation

  Design graceful degradation: simpler model, reduced functionality, cached answer, human escalation, or explicit inability to complete the request.

• Human approval for high-risk actions

  Use human approval for high-risk or irreversible actions.

• Service-level expectations

  Define service-level expectations for availability, latency, and quality.

Outcome · Design a system that remains useful when individual AI components fail.

• The end-to-end flow

  Design the end-to-end flow between retrieval, models, tools, agents, guardrails, caches, and external systems.

• Deterministic vs model-delegated decisions

  Decide which decisions should be deterministic and which should be delegated to the model.

• Sync, async, queued or long-running

  Choose between synchronous execution, asynchronous jobs, queues, and long-running agent workflows.

• Avoiding unnecessary agentic complexity

  Avoid unnecessary agentic complexity when a deterministic pipeline is sufficient.

• Component boundaries

  Define clear component boundaries so individual parts can be changed, evaluated, and scaled independently.

Outcome · Turn individual AI components into a coherent production system.

• Wiring evals & feedback into production

  Connect evals, traces, operational metrics, and user feedback into the production architecture.

• What to monitor

  Monitor quality, latency, cost, errors, retrieval behavior, tool usage, and model/provider performance.

• Explicit & implicit feedback

  Capture explicit feedback such as ratings as well as useful implicit behavioral signals.

• Failures become evaluation cases

  Turn production failures and user feedback into new evaluation cases.

• Versioning for traceable regressions

  Version models, prompts, retrieval configurations, and application changes so regressions can be traced.

• Deploy → observe → improve → redeploy

  Understand the loop: deploy → observe → collect feedback → evaluate → improve → redeploy.

Outcome · Design AI products that improve from real-world usage rather than remaining static after launch.

• Whiteboard and defend the choices

  Whiteboard several systems end-to-end and defend the choices.

Customer-support copilot

• Design around

  Design around:

• RAG & fresh knowledge

  RAG and fresh knowledge

• Citations

  citations

• Model routing

  model routing

• Caching

  caching

• Escalation to humans

  escalation to humans

• Latency & cost constraints

  latency and cost constraints.

Coding assistant

• Design around

  Design around:

• Large codebase context

  large codebase context

• Retrieval

  retrieval

• Tool & code execution

  tool/code execution

• Model selection

  model selection

• Sandboxing

  sandboxing

• Long-running tasks

  long-running tasks.

Deep research agent

• Design around

  Design around:

• Search & retrieval

  search and retrieval

• Planning

  planning

• Parallel tool calls

  parallel tool calls

• Context management

  context management

• Long-running execution

  long-running execution

• Citations

  citations

• Cost budgets & failure recovery

  cost budgets and failure recovery.

• Requirements → architecture → trade-offs

  For each case, reason about requirements → baseline → architecture → bottlenecks → failure modes → scaling → trade-offs.

Outcome · Defend architectural decisions rather than merely draw boxes and arrows.