Technology is your velocity engine. Architecture is how you make that velocity safe, repeatable, and economical. Together they determine whether ideas move from notebooks to production in days, whether rollbacks are rehearsed rather than improvised, whether token spend stays inside targets, and whether customer‑facing content is grounded, safe, and fast. This chapter focuses on the capabilities—not brand names—that let you ship traditional ML and generative AI at scale: clear reference patterns, automated pipelines, progressive delivery, observability end‑to‑end, guardrails embedded as code, and portable integrations that keep you out of vendor cul‑de‑sacs.
Readiness here is measured by outcomes you can see in logs and drills: deployment frequency and rollback reliability, latency and availability against SLOs, retrieval precision/recall on the corpora that power RAG, content‑safety pass rates, cache hit rates, and unit cost per task. The emphasis is on “policy‑as‑code” and “golden paths” that make the right thing the easy thing for every team. When these elements are in place, leaders can accept appropriate risk with confidence and your portfolio accelerates without trading away safety or economics.
8.1. Diagnostic Criteria
The assessment evaluates Technology and Architecture Readiness across fifteen sub‑dimensions. Each criterion describes what must be true in production (or demonstrably true in lower environments for drills) and the evidence you should have at hand.
Reference architecture and interoperability
A documented, adopted reference architecture spans data ingestion, transformation, feature management, model training and evaluation, model/prompt serving, retrieval (for RAG), guardrails, observability, and cost telemetry. Interfaces between components are standardized, versioned, and tested. Abstractions exist where you need vendor portability (model gateway, vector‑store interface, storage formats). Evidence includes the current architecture diagram, interface contracts, and examples where teams reused the reference patterns without bespoke glue.
SDLC, environments, and golden paths
Teams promote work through well‑defined environments (dev/test/stage/prod) with parity rules, seeded test data, and IaC for reproducible setup. There is a single “golden path” for shipping models/prompts/pipelines that embeds reviews and controls. Notebooks are traceably promoted (if allowed), not copy‑pasted. Evidence includes environment definitions, IaC repos, and a runbook that shows how a new service adopts the golden path.
CI/CD for ML and LLM (MLOps/LLMOps)
Pipelines automate tests (unit, data/feature checks, evaluation), approvals, security scans, and promotion. Required controls are gates—not email checklists. For GenAI, pipelines also manage prompt templates, retrieval evaluation, safety tests, and canary releases of filters and routing rules. Evidence includes CI/CD definitions, last three release logs with gate results, and failed‑gate examples that blocked unsafe releases.
Artifact management and registries
Model registry, prompt/template store, evaluation datasets, and data/feature/embedding versions are recorded with lineage and provenance. Artifacts are immutable and reproducible; checksums and signatures are used; SBOMs exist for runtime images. Evidence includes registry exports, provenance metadata, and rebuild proofs for at least one production artifact.
Serving, orchestration, and routing
There are reliable endpoints for inference and for RAG (retrieval + generation). Autoscaling protects latency and cost; batching and caching are used where appropriate. A model/LLM router supports A/B, shadow, and safety‑based routing (e.g., fallbacks on safety or latency breach). Evidence includes SLOs, router configs, and runtime dashboards showing scale‑up, cache hit rates, and routing decisions.
Performance engineering and SLOs
Each service defines availability and latency SLOs (p50/p95) tied to user‑perceived operations, not only model inference. Token budgets, context lengths, and memory footprints are engineered for target latency/cost. Load tests run before scale. Evidence includes SLO documents, load‑test results, and change logs where design choices were driven by performance/cost targets.
Observability end‑to‑end
You can see what matters: data freshness/quality; feature/embedding drift; model metrics and task quality; retrieval precision/recall; content‑safety signals; latency/availability; error rate; cost per task; and user feedback loops. Alerts route to owners with playbooks that define actions and rollbacks. Evidence includes dashboards, alert histories, and examples where alerts triggered automatic rollback or human intervention within SLA.
Reliability, release engineering, and rollback
Progressive delivery (canary/blue‑green) is the default. Automatic rollback triggers are defined on health signals (latency/error/safety/cost spikes) and tested. Chaos and failure‑mode drills are run and logged. Evidence includes rollout plans, rollback drill records from the last quarter, and post‑incident reviews that changed tests or gates.
Security, supply chain, and trust boundaries
Identity and authorization are least‑privilege; secrets are vaulted; network boundaries and egress controls are enforced; dependency scanning and vulnerability SLAs are active; SBOMs and image signing protect the supply chain. For LLMs, prompt‑injection and data‑exfiltration defenses exist (input sanitization, tool/connector sandboxes, output validators). Evidence includes IAM policies, vault configs, CVE triage logs, and LLM security test results.
Tenancy and isolation
Workloads and data are isolated by environment, business unit, and sensitivity where required. Resource quotas prevent noisy‑neighbor failures (especially on GPUs/accelerators); per‑tenant logs and costs are available. Evidence includes quota configs, isolation diagrams, and incident records showing containment.
RAG infrastructure and corpus governance (GenAI)
Vector stores, indexes, and retrieval orchestration follow standard patterns for chunking, embedding, indexing, filtering, and re‑indexing. Grounding and citation standards exist; retrieval is evaluated on representative tasks with agreed thresholds before external exposure. Evidence includes corpus registry, embedding/chunking standards, retrieval‑evaluation results, and vector‑store ACLs.
Guardrails and safety enforcement (GenAI)
Safety filters and output classifiers are configured by policy; topic restrictions and human‑in‑the‑loop thresholds are enforced in code; prompts and outputs are logged and protected; red‑team tests (prompt injection, jailbreaks, harmful content) run pre‑launch. Evidence includes policy‑as‑code rules, safety evaluation results, and logs of overrides/appeals.
Cost transparency and unit economics
Token, retrieval/query, storage, compute, and egress costs are visible per use case. Caching, model choice (distilled vs. frontier), and routing are used to keep unit cost per task within targets at the desired latency. Budgets and alerts exist; “cost regressions” fail CI in extreme cases. Evidence includes cost dashboards, budgets, and examples where design changed to meet targets.
Portability and exit paths
API abstractions, open formats, and testable exit plans reduce lock‑in. Portability drills are scheduled for high‑dependency components (LLM provider, embedding model, vector store) in lower environments. Evidence includes abstraction libraries, compatibility tests, and drill records with time to switch and defect counts.
Compliance and auditability
Decisions, approvals, artifacts, and runtime evidence are reconstructable in under an hour for any material release. For regulated contexts, explainability artifacts and controls are part of the pipeline. Evidence includes decision logs, model cards/data sheets, and exportable audit trails from CI/CD and runtime.
Measurable thresholds that indicate “ready”
Set explicit bars before you score; tune to your context, but make them visible and testable.
- Delivery and reliability
- Deployment frequency and lead time are measured; change failure rate ≤ 20% and trending down; mean time to restore within target.
- ≥ 90% of production releases use canary/blue‑green; rollback drills run quarterly for each critical service and complete within the target window.
- SLOs and performance
- 100% of user‑facing AI services have documented SLOs (availability and p95 latency) and error budgets; load tests run pre‑scale.
- For GenAI, token budgets and cache strategies are documented; cache hit rate targets exist for applicable flows.
- Observability and action
- 100% of critical services have dashboards that include data quality/freshness, drift, model/task quality, retrieval quality (if RAG), content safety (if GenAI), and cost.
- Alerts map to runbooks with owners; median alert‑to‑action time meets SLA.
- Security and supply chain
- 100% of secrets vaulted; image signing and SBOMs exist for all deployable artifacts; critical CVEs remediated or mitigated within SLA.
- LLM security tests (prompt‑injection, exfiltration via tools/connectors) pass before external exposure.
- GenAI guardrails
- Prompts/outputs are logged and protected; third‑party training opt‑out is enforced for sensitive workloads.
- Retrieval precision/recall thresholds are met on representative tasks prior to launch; content‑safety pass rates meet policy, with human‑review thresholds configured.
- Cost and portability
- Per‑use‑case unit cost visible; at least one meaningful design decision in the last quarter cites cost targets.
- One portability drill executed in the last six months for a high‑dependency component.
Domain‑specific gating rules (apply caps if any fail)
These are non‑negotiable safety bars. If one fails for in‑scope use cases, cap the Technology & Architecture domain per Chapter 3 and flag it on the heat map.
- No tested rollback path (or no ability to disable/roll back safety filters) for a production AI service → cap at 2.0.
- No CI/CD with enforced gates for required controls (privacy, security, model risk, retrieval quality, content safety) → cap at 2.0.
- No observability for data quality/drift and, for GenAI, retrieval quality and content safety → cap at 2.0.
- Secrets not vaulted or egress controls absent on LLM tool/connector pathways → cap at 2.0.
- Prompts/outputs not logged and protected, or third‑party training opt‑out not enforced where required → cap at 2.0.
- No restore/backup drill for vector indexes or model artifacts powering a top use case → cap at 2.0.
Minimum evidence required to score confidently
Have these in your repository before assigning more than a provisional score.
- Reference architecture and interface contracts; golden path runbook; IaC repos.
- CI/CD definitions with gate configs; last three release logs; failed‑gate example; rollback drill record.
- Model/prompt/embedding registries with lineage and provenance; SBOMs or image‑signing evidence.
- SLO documents; load‑test results; serving/router configs; autoscaling and cache dashboards.
- Observability dashboards and alert histories covering data quality/drift, model/task quality, retrieval precision/recall, content safety, latency/availability, and cost.
- IAM/vault configurations; CVE triage logs; LLM security test results (prompt injection, exfiltration).
- Corpus registry and retrieval‑evaluation results; content‑safety configurations; prompt/output logging policy.
- Cost dashboards (tokens, retrieval, storage, compute) with per‑use‑case attribution; example decision tied to cost targets.
- Portability drill record for one high‑dependency component.
Anti‑patterns to watch for
These patterns slow you down and increase risk—often invisibly until production.
- “Notebook‑to‑prod”: manual, opaque releases; no reproducibility or rollback.
- “Observability theater”: pretty dashboards with no runbooks or ownership; alerts ignored.
- Golden path bypassed “just this once”; side‑channel approvals.
- GPU/accelerator wild‑west: no quotas, preemption chaos, or opaque scheduling.
- RAG on a file share: indiscriminate indexing of unmanaged content; no retrieval evaluation.
- Single‑vendor cul‑de‑sac: proprietary interfaces with no abstraction or exit plan.
- Secrets in configs; permissive egress that lets tools/connectors leak data; no LLM security tests.
- Cost blindness: no token/retrieval visibility; no caching/routing; surprise bills drive design after launch.
Signals of excellence (what “great” looks like)
Releases are boring: canary succeeds, gates pass, SLOs hold, and rollback is a click when needed. Evaluation harnesses run in CI for both ML and GenAI; retrieval quality and content safety are treated like test coverage, not optional checks. Cost per task is visible and tuned by design—teams switch models, adjust context, enable caching, or route traffic automatically to hit targets. Abstractions make portability real; a quarterly drill proves you can switch a model provider, embedding function, or vector store in hours without rewriting applications. Most importantly, engineers use one golden path because it is faster than any workaround.
Satisfy these criteria with traceable evidence and you will have a technology and architecture spine that turns AI ambition into stable, economical, and safe production at speed.
8.2. Technology Assessment – Step-by-Step Guide
This guide converts the criteria into a short, decisive run that produces: a defensible baseline of your delivery machinery, evidence that safety and portability really work, and a 90‑day fix list tied to gates and funding. Assume you have survey/interview inputs and an artifact repository from Chapter 4. Aim to finish this sequence in 10 business days for a business unit, or 15 for an enterprise sweep.
Step 1 — Set the frame (scope, decisions, and the clock)
Write down the decisions this assessment must enable: which services are safe to scale, which controls must be automated, how to hit latency/cost targets, and what portability to prove. Name the in‑scope use cases and their production services. Fix a 10‑ or 15‑day timeline.
Acceptance tests: scope page lists services, environments, and decisions; end date is on calendars; “what’s out of scope” is explicit.
Step 2 — Assemble the minimum evidence pack
Pull live links (not screenshots) for the last 90 days: reference architecture, IaC repos, CI/CD definitions and release logs, failed‑gate examples, SLO docs and load‑test results, serving/router configs, observability dashboards (data, drift, model/task quality, retrieval quality, safety, latency, cost), vault/IAM policies, SBOMs and image‑signing, CVE triage logs, rollback drill records, backup/restore results (including vector stores), corpus registry and retrieval‑evaluation results, content‑safety configs, prompt/output logging policies, and cost dashboards (tokens, retrieval, storage, compute).
Acceptance tests: every artifact shows owner and last‑updated date; gaps logged with owners and due dates.
Step 3 — Map value cases to runtime assets
For each top use case, trace the runtime chain: APIs/services → models/prompts → pipelines → data/feature/embedding sources → corpora/vector indexes → controls (gates) → dashboards → cost centers. Keep this to one page per use case; it becomes your test plan.
Acceptance tests: each use case has a dependency map with owners, SLOs, and evidence links.
Step 4 — Confirm the golden path exists and is adopted
Describe the standard way to ship (environments, approvals, pipeline gates). Sample the last three releases per critical service and confirm they used the golden path. Document any bypasses and why they occurred.
Acceptance tests: one runbook explains the path end‑to‑end; ≥90% of critical releases followed it; bypasses are recorded with a fix.
Step 5 — Reconstruct two releases (truth from logs)
Pick (a) the most recent normal release and (b) the most recent release with a rollback or hotfix. Rebuild the timeline from CI logs, approvals, and alerts: what was tested, which gates fired, who approved, when it went live, and how it behaved.
Acceptance tests: timelines exist with links; discrepancies between “policy” and “what happened” are logged as constraints.
Step 6 — Verify CI/CD gates (policy‑as‑code over policy‑on‑paper)
List the mandatory checks by risk class: unit/data tests, privacy/security/model‑risk checklists, schema/contract checks, vulnerability scan, retrieval‑quality gate (RAG), content‑safety tests (GenAI), and cost regression guardrails where applicable. Prove gates fire by showing a failing build that blocked promotion.
Acceptance tests: each mandatory control appears as a pipeline gate with a log; at least one failing example exists for each class; no “email approvals.”
Step 7 — Check environments and IaC parity
Confirm environments (dev/test/stage/prod) are defined as code, seeded with safe test data, and parity rules exist (libraries, drivers, CUDA stacks, runtime images). Spin a fresh lower environment from IaC and deploy a sample service through CI.
Acceptance tests: a clean lower environment comes up from IaC; parity drift items are listed with owners and dates.
Step 8 — Security and supply chain hardening
Review IAM (least‑privilege), network egress controls, secrets vault usage, image signing, SBOM presence, CVE triage SLAs, and dependency pinning. For LLMs, test prompt‑injection and data‑exfiltration paths for tools/connectors.
Acceptance tests: secrets are vaulted (sample shows zero hard‑coded secrets); critical CVEs are within SLA; LLM security tests pass; egress controls documented on sensitive paths.
Step 9 — Serving, scaling, and routing
Inspect serving endpoints, autoscaling policies, batch/caching configs, and model/router rules (A/B, shadow, safety/latency fallbacks). Verify SLOs (availability and p95 latency) are defined on user‑perceived operations and monitored.
Acceptance tests: SLO docs exist for 100% of user‑facing AI services; router config supports shadow/canary; cache hit rate is measured where applicable.
Step 10 — SLOs, load, and cost engineering
Run or review the latest load tests against target latency. For GenAI, check token budgets, context windows, and cache/routing designs that achieve latency at reasonable cost.
Acceptance tests: pre‑scale load test results exist; cost vs. latency trade‑offs are documented; at least one design choice cites unit‑cost targets.
Step 11 — Observability that drives action
Confirm dashboards stitch together: data freshness/quality and drift; model metrics and task quality; retrieval precision/recall (RAG); content‑safety signals; latency/availability/error; cost per task; and user feedback loops. Trace two real alerts to the runbook steps taken.
Acceptance tests: every critical service has one owner‑named dashboard; median alert‑to‑action time meets SLA; playbooks exist and were followed.
Step 12 — Reliability and rollback drills
Run a low‑risk rollback drill in a lower environment for one service per use case. Confirm automatic rollback triggers exist (latency/error/safety/cost spikes). Review last quarter’s incidents and the changes they triggered (tests, gates, or runbooks).
Acceptance tests: a fresh drill record exists with time to restore; automatic triggers are configured; at least one post‑incident change is evidenced.
Step 13 — GenAI guardrails and retrieval quality
For RAG, examine corpus governance, chunking/embedding standards, index configs, and retrieval‑evaluation results on representative tasks. For safety, review policy‑as‑code filters, human‑in‑the‑loop thresholds, and override logging. Confirm prompts/outputs are logged and protected; third‑party training opt‑out is enforced where required.
Acceptance tests: retrieval metrics meet pre‑launch thresholds; safety tests pass; logging/retention are established for prompts/outputs.
Step 14 — Backup/restore and DR (include vector stores)
Validate backup policies and complete a restore drill for at least one warehouse/lake, one model registry, and one vector index powering a top use case. Confirm RTO/RPO alignment to business impact.
Acceptance tests: restore drills within six months for each store; RTO/RPO met; vector‑store restore proved.
Step 15 — Portability drill (one high‑dependency component)
Pick a dependency (LLM provider, embedding model, or vector store). In a lower environment, swap to an alternative using your abstraction layer. Measure time to switch, defects, and any code changes required.
Acceptance tests: drill completed; “time to switch” recorded; issues logged with owners; abstraction gaps identified.
Step 16 — Tenancy and isolation
Check resource quotas and isolation boundaries by tenant/BU/environment. Inspect GPU/accelerator scheduling, preemption, and noisy‑neighbor protections. Verify per‑tenant logging and cost attribution.
Acceptance tests: quotas configured; recent incidents show containment; per‑tenant cost and logs are available.
Step 17 — Compliance and auditability
Reconstruct the most recent material release (approvals, gates, artifacts, and runtime evidence) in under 60 minutes. For regulated contexts, ensure explainability artifacts and required attestations are generated in the pipeline.
Acceptance tests: end‑to‑end reconstruction completed on the clock; artifacts are traceable to source systems.
Step 18 — Quantify throughput and reliability (last 90 days)
Compute deployment frequency, lead time, change failure rate, mean time to restore, canary adoption rate, and percentage of releases with auto‑rollback configured.
Acceptance tests: a one‑page metric sheet exists; outliers highlighted with owners; trends inform the fix list.
Step 19 — Cost and unit‑economics view
Attribute tokens, retrieval queries, storage, compute, and egress by use case. Calculate cost per task at target latency and volume; identify routing/caching/model choices to hit targets.
Acceptance tests: per‑use‑case cost is visible; at least one cost‑driven change is committed to the backlog.
Step 20 — Score and apply caps
Use Chapter 3’s rubric to score each sub‑dimension with confidence tags and evidence links. Apply caps automatically where gating rules fail (no rollback path, no CI/CD gates, missing observability, secrets not vaulted, prompts/outputs not logged or opt‑out missing, no restore drill for vector indexes).
Acceptance tests: scored workbook exists; caps are explicit; heat map and rationale are ready for readout.
Step 21 — Decide and publish the 90‑day fix list
Translate constraints into short, funded remediations with acceptance tests. Typical high‑ROI moves: convert manual reviews into gates; stand up golden‑path scaffolding; implement LLM router and caching; add retrieval‑evaluation harness; enable automatic rollback; harden egress around tools/connectors; schedule a portability drill; publish per‑use‑case cost dashboards.
Acceptance tests: each fix has an owner, date, and “pass” definition; funding is tied to completion; gates reference these acceptance tests.
Step 22 — Stand up the technology scorecard and operating rhythm
Add a small set of KPIs to the weekly operating review: deployment frequency, lead time, change failure rate, MTTR, SLO attainment (availability and p95 latency), alert‑to‑action time, retrieval precision/recall (RAG), content‑safety pass rate, rollback drill cadence, cost per task, cache hit rate, and portability drill status.
Acceptance tests: scorecard appears in the next operating review; owners named for each KPI; actions captured in the decision log.
Probes you can run this week (lightweight but revealing)
- Gate reality check: Intentionally fail a retrieval‑quality test or content‑safety test in lower environment; confirm the pipeline blocks promotion and records why.
- Rollback in anger: Flip a health signal to breach threshold and observe automatic rollback.
- LLM security poke: Attempt benign prompt injection and tool‑exfiltration tests; confirm sanitization and sandboxing.
- Cost spike drill: Simulate higher token usage; verify budget alerts fire and routing/caching rules adapt.
- Portability smoke test: Swap embedding models for one endpoint via your abstraction; confirm no app rewrite needed.
Fast‑track (5 business days)
- Day 1: Scope, evidence pack, golden path confirmation.
- Day 2: Release reconstruction; CI/CD gate proof; IaC parity check.
- Day 3: SLOs, load‑test review; observability and alert‑to‑action trace; security vault and CVE spot‑check.
- Day 4: Rollback drill; GenAI retrieval/safety checks; cost snapshot.
- Day 5: Scores with caps; heat map; 90‑day fix list; executive readout.
Deep‑dive (15 business days)
Add multi‑service chaos drills, broader portability drills (LLM and vector store), GPU scheduler tuning, multi‑tenant isolation tests, explainability/documentation pipelines for regulated use cases, and a board‑ready technology risk statement.
Deliverables you should produce
- One‑page dependency map per top use case.
- Golden‑path runbook and adoption snapshot.
- Two release reconstructions with logs and lessons.
- CI/CD gate inventory with failing examples.
- SLO pack and load‑test summaries.
- Observability dashboards and alert‑to‑action traces.
- Rollback drill record; incident review notes and changes.
- GenAI retrieval‑evaluation and safety test results; prompt/output logging policy proof.
- Security/vault/SBOM/CVE notes; egress controls for tools/connectors.
- Restore drill record (including vector store); portability drill record.
- Cost and unit‑economics view; router/caching decisions.
- Scored workbook with caps; heat map; 90‑day fix list and funding memo.
Common schedule breakers—and how to preempt them
- “Shadow paths” around CI/CD. Enforce a single route to prod; disable side‑channel credentials; surface bypasses in the scorecard.
- Observability without ownership. Require an owner’s name on every dashboard tile and an alert‑to‑action SLA.
- Untested rollbacks. Put a drill on the calendar before you begin; tie scale decisions to drill success.
- RAG without curation. Block indexing of unmanaged shares; require corpus registration and retrieval evaluation.
- Secrets sprawl. Scan repos; rotate credentials; block builds that contain secrets.
- Cost blindness. Add token/retrieval dashboards early; set unit‑cost gates; test cache/routing.
- Single‑vendor cul‑de‑sacs. Build the abstraction now; run at least one portability drill.
Run this sequence with evidence, not aspiration. If any gating item fails, apply the cap immediately and fund the fix first. That discipline is what turns technology from a source of surprises into a compounding advantage—safe, fast, and economical at scale.
8.3. Tech-Stack Checklist
Use this checklist to confirm that your stack can ship ML and GenAI safely, quickly, and economically. It is vendor‑agnostic and outcome‑oriented: every line maps to evidence you can show today (logs, configs, dashboards, drills). Treat items as binary—checked if proven, or dated actions if not. Run it at two levels: (1) per top value case (pre‑flight before scaling) and (2) program‑wide (weekly health).
How to apply the checklist
- Anchor to your golden path (the standard way work moves from dev → prod with controls embedded).
- Classify each in‑scope use case by risk class (assistive internal, internal decision support, external content, automated decisions).
- For each section, attach live links to evidence in your governed repository; avoid screenshots where a system of record exists.
- If any gating item fails (called out below), record a cap in the readiness heat map and fund the fix first.
Core infrastructure and controls
- Cloud/org setup uses separate accounts/projects per environment; baseline guardrails (SCPs/org policies) in place.
- Network segmentation and egress controls defined; private links/peering for data planes; production endpoints behind WAF/CDN as needed.
- KMS and secrets: all secrets vaulted; automatic rotation; zero hard‑coded secrets; detection on commit.
- Identity: SSO with RBAC/ABAC; short‑lived tokens for services; periodic access recertification; least‑privilege verified on samples.
- IaC defines environments (dev/test/stage/prod); drift detection and remediation enabled.
Compute, scheduling, and accelerators
- Standard container orchestration with quotas and limits; autoscaling configured.
- GPU/accelerator scheduling supports preemption, priority, and per‑tenant quotas; runtime stacks (drivers/CUDA/etc.) are version‑pinned and parity‑checked across envs.
- Batch and real‑time workloads have distinct queues and SLOs; cost controls on bursty/spot usage.
Storage and data planes
- Stores by purpose (raw, curated, data products, features, vector, cache) declared with encryption at rest and lifecycle/retention rules.
- Cross‑region/zone strategy documented for P0 workloads; restore drills completed (including vector stores).
- Data contracts or schema registry enforced; backfill/replay runbooks exist and are tested.
Ingestion, orchestration, and change
- Standard connectors for batch/CDC/stream; idempotency and exactly/at‑least‑once semantics declared and tested.
- Orchestrator DAGs owned, versioned, and tested; CI fails on schema‑breaking changes; change calendar published.
Metadata, catalog, and lineage
- Catalog coverage ≥95% for P0/P1 products; owner, steward, classification, and last updated present.
- Automated lineage from sources to consumption (prefer field‑level) matches reality on spot‑checks.
Feature management (ML portfolios)
- Feature store (or governed pattern) with versioning, ownership, usage analytics.
- Training/serving skew checks active; offline/online parity measured for live models.
GenAI: corpora, embeddings, and vector stores
- Corpus registry: approved sources, inclusion/exclusion rules, dedupe/canonicalization, classification, retention.
- Chunking/embedding standards documented; embedding model/version pinned and recorded.
- Vector store supports filters/namespaces/ACLs; index (HNSW/IVF/etc.) chosen for workload; re‑indexing runbook exists and is tested.
- Retrieval evaluation harness runs on representative tasks with thresholds (precision/recall@k) set per use case.
Model/Prompt lifecycle and registries
- Model registry and prompt/template store with lineage, immutable versions, approvals, checksums, and provenance.
- Evaluation datasets and metrics tracked; reproducible builds proven for at least one production artifact.
Serving, routing, and APIs
- Managed inference endpoints with autoscaling; batch and real‑time supported; caching/batching configured when applicable.
- LLM router supports A/B, shadow, safety‑based and cost/latency‑based routing; fallbacks defined and tested.
- API gateway with authN/Z, quotas, rate limiting, and per‑tenant throttles; SLAs published (availability and p95 latency).
Safety, guardrails, and human‑in‑the‑loop (GenAI)
- Content safety filters configured by policy; topic restrictions and human‑review thresholds enforced in code.
- Prompt‑injection and tool/connector sandboxes in place; output validators for PII/PHI/unsafe content; override logging enabled.
- Prompts/outputs logged and protected; third‑party model training opt‑out enforced where required.
Observability and action
- Unified dashboards link data health (freshness/quality/drift), model/task quality, retrieval metrics, safety signals, latency/availability, errors, and cost per task.
- Alerts route to owners; alert‑to‑action SLA met; runbooks exist and are used; correlation IDs propagate across retrieval → generation → delivery.
SLOs, performance, and load
- Every user‑facing AI service has SLOs (availability and p95 latency) tied to user‑perceived operations; error budgets tracked.
- Load testing precedes scale; token/context budgets and cache strategy documented for GenAI.
CI/CD and release engineering
- Pipelines cover build → test → security → policy → evaluate → deploy; policy‑as‑code gates for privacy/security/model‑risk/retrieval/safety.
- Progressive delivery (canary/blue‑green) is default; auto‑rollback triggers defined on health signals (latency/error/safety/cost) and tested.
Security and software supply chain
- SBOMs and image signing for deployable artifacts; dependency pinning; CVE triage SLAs met.
- Egress policies on LLM tool/connector paths; secrets never in configs; SAST/DAST on critical services.
Portability and exit
- Abstractions for LLM providers, embedding models, vector stores, and storage formats; compatibility tests in CI.
- Portability drills scheduled (lower env) with “time to switch” recorded and issues tracked.
Cost and unit economics
- Per‑use‑case cost attribution (tokens, retrieval, compute, storage, egress); budgets and anomaly alerts.
- Design levers applied (routing, caching, quantization, context control, model selection) to hit unit cost per task targets at the required latency.
Developer experience (DX)
- Scaffolding/CLI templates embody the golden path; paved‑road examples; linting and pre‑commit hooks; self‑service docs live and current.
Operations, on‑call, and incident management
- Named on‑call rota; escalation ladder with SLAs; post‑incident reviews produce changes to tests/gates/runbooks.
- Change calendar visible; freeze windows declared where needed.
Compliance and auditability
- One source of truth contains decision memos, approvals, artifacts, and runtime evidence; reconstruct last material release in <60 minutes.
- Logging retention, access reviews, and separation of duties evidenced for regulated contexts.
GenAI‑specific addendum (apply in addition to core items)
- Retrieval grounding and citation standards defined; grounded‑answer and citation‑accuracy tracked for external content.
- Evaluation harness for task quality (human‑labeled set to calibrate any LLM‑as‑judge); red‑team tests for jailbreaks and prompt injection run pre‑launch.
- Human‑in‑the‑loop thresholds per task; review tooling provides latency and quality analytics.
Yes/No gating questions (must all be “Yes” before external scale)
- Can we roll back any AI service fast, and have we drilled it in the last quarter?
- Do CI/CD pipelines enforce policy‑as‑code gates for privacy, security, model‑risk, retrieval quality, and content safety?
- Are prompts and outputs logged and protected, with third‑party training opt‑out enforced where required?
- Do user‑facing services have SLOs (availability and p95 latency) and error budgets, with load tests completed?
- For RAG, have we met retrieval precision/recall thresholds on representative tasks before launch?
- Are secrets vaulted and egress controls enforced on tool/connector pathways?
- Has a restore drill succeeded for each P0 store (warehouse/lake, model registry, vector store) in the last six months?
- Is per‑use‑case unit cost visible, and do routing/caching/model choices keep us within target?
Evidence to attach (per value case)
- Dependency map from API → model/prompt → pipelines → data/features/embeddings/vector indexes → controls → dashboards → cost centers.
- Golden path runbook; last three release logs; one failed‑gate example; rollback drill record.
- SLO docs; load‑test summary; router and caching configs; safety policy and evaluator results.
- Corpus registry and retrieval evaluation; prompt/output logging policy; opt‑out proof for third‑party models.
- Vault/IAM policy excerpts; SBOMs/image signing; CVE triage snapshot; egress controls for tools/connectors.
- Restore drill evidence (including vector store); portability drill report with “time to switch.”
- Cost dashboards with unit‑cost per task; a design note citing cost‑driven choices.
Quantitative bars (defaults—tune to your context)
- Delivery: change failure rate ≤20%; ≥90% of production releases via canary/blue‑green; MTTR within target; quarterly rollback drills per critical service.
- SLOs: 100% of user‑facing AI services have availability and p95 latency SLOs; alert‑to‑action ≤ 15 minutes for P0.
- Observability: dashboards include data/drift, model/task quality, retrieval, safety, latency, error, cost; 100% of P0 alerts mapped to runbooks with owners.
- Security: 100% secrets vaulted; SBOM and signed images for all deployables; critical CVEs remediated/mitigated within SLA.
- GenAI: retrieval precision/recall ≥ task thresholds pre‑launch; safety block rate within policy; grounded‑answer rate tracked for external content.
- Cost: per‑use‑case unit cost visible; at least one design change per quarter cites cost targets.
- Portability: one drill in ≤6 months for a high‑dependency component (LLM, embedding, or vector store).
Fast triage (90 minutes)
- Pull golden path runbook; confirm last three releases used it; record any bypass.
- Open CI logs; verify at least one failing gate blocked promotion in the last 30 days.
- Check rollback drill record for a critical service; if missing, schedule now.
- For a GenAI use case, open the retrieval evaluation and safety results; if thresholds are absent, block external exposure.
- Verify prompts/outputs logging and opt‑out settings for third‑party models.
- Open the cost dashboard; compute unit cost per task; note gaps vs. target.
- Inspect vault/IAM for a sample service; confirm no hard‑coded secrets and least‑privilege scopes.
- Confirm a vector‑store restore drill exists; if not, log a gating gap.
Common anti‑patterns (and quick fixes)
- Notebook‑to‑prod (no reproducibility/rollback) → Enforce the golden path; disable side‑channel credentials; migrate with scaffolds.
- Observability theater (no runbooks/owners) → Add owners and alert‑to‑action SLAs to every dashboard tile.
- RAG on shared drives → Require corpus registration, classification, and retrieval evaluation thresholds before indexing.
- Secrets sprawl → Block builds with secret scanners; rotate credentials; centralize in vault.
- Single‑vendor cul‑de‑sac → Add abstraction layer; write a portability test; run a drill.
- Cost blindness → Instrument tokens/retrieval; set unit‑cost targets; add caching/routing; fail CI on extreme cost regressions.
- GPU chaos → Implement quotas, priority, and preemption; pin runtimes; report utilization per tenant.
30/60/90‑day stack hardening plan (assign owners and tests)
- 30 days: Golden path enforced; policy‑as‑code gates for privacy/security/model‑risk live; vault rollout and secret scanning; SLOs published for user‑facing services; schedule rollback and vector‑store restore drills.
- 60 days: Retrieval evaluation harness adopted across GenAI; router + caching tuned to cost/latency targets; lineage coverage ≥80% for model‑feeding flows; CI adds compatibility tests and SBOM/signing.
- 90 days: Portability drill executed and issues closed; per‑use‑case unit‑cost dashboards used in design reviews; incident postmortems driving automated tests and gate changes; on‑call and escalation SLAs met consistently.
Acceptance tests for “ready to scale” (program‑level)
- Golden path adoption ≥90% of critical releases; no “email approvals.”
- All gating questions are “Yes” with live evidence.
- Scorecard in the operating review shows deployment frequency, lead time, change failure rate, MTTR, SLO attainment, alert‑to‑action, retrieval/safety metrics, rollback drill cadence, unit cost per task, cache hit rate, and portability status—each with named owners.
Run this checklist with evidence, not aspiration. The moment a gating item fails (rollback, gates, observability, secrets, retrieval/safety, restore), stop scaling that use case, record the cap, and fund the fix. That discipline turns your tech stack into a compounding advantage—fast, safe, and cost‑aware by default.
8.4. Architecture Review Template
An architecture review is a decision factory, not a ceremony. Its purpose is to answer three questions with evidence: Is the design valuable (meets the business and user goals), safe‑to‑operate (controls and obligations wired in), and ready to scale (reliable, observable, portable, and economical)? Use this template to run crisp, one‑hour reviews that end in a recorded decision, dated conditions, and actionable follow‑ups. Keep it lightweight, link to live systems, and make the golden path the path of least resistance.
When to trigger a review
- Net‑new AI service or major revision to a production component.
- Any change that affects SLOs, data classifications, retrieval corpora, model class, safety posture, or unit economics by >20%.
- Movement across stage gates (Discovery → Pilot, Pilot → Production, Production → Scale).
- Material vendor change or portability drill outcomes that require redesign.
Who attends (and why)
- Chair (architecture lead) to drive decisions and timeboxes.
- Product owner to speak for the user and value.
- Tech leads for data, platform, ML/LLM, and security to own feasibility and controls.
- Risk partners (privacy, security, model risk; content safety for GenAI) to confirm “safe‑to‑operate.”
- SRE/operations to own SLOs, rollback, and runbooks.
- Optional: procurement/vendor management for third‑party risk, finance for unit economics.
1) Pre‑read package (links only)
Send 24 hours in advance. If the links below do not exist, create placeholders before the review.
- Problem and scope note (≤1 page) with non‑goals and user narratives.
- Current and target architecture diagrams (logical + runtime paths).
- Data/corpus map with ownership, classification, and SLAs.
- CI/CD pipeline and gates; IaC/environment plan; release/rollback strategy.
- SLOs (availability and latency), load‑test plan or results; observability plan.
- Risk class, “safe‑to‑operate” checklist, and required reviews (privacy, security, model risk, content safety).
- Cost model with unit‑cost targets and sensitivity; portability assumptions/abstractions.
- For GenAI: corpus registry, retrieval evaluation results, safety policy and evaluator plan, prompt/output logging policy.
Acceptance test: Every pre‑read links to a system of record (repo, registry, dashboard) and has a named owner.
2) Agenda (60‑minute decision flow)
- Purpose, scope, and decision requested (5 minutes).
- Architecture walk‑through: value flow → data/corpus → model/prompt → serving/routing → observability → safety/controls → cost (15 minutes).
- Reviewer probes by layer (20 minutes).
- Risks, trade‑offs, and alternatives considered (10 minutes).
- Decision, conditions, owners, and dates (10 minutes).
Accepted outcomes: Approve; Approve with conditions; Request changes; Reject (with rationale). All decisions are recorded in the decision log the same day.
3) Architecture Review Record (fill during the meeting)
- Decision ID and date; review stage (Discovery/Pilot/Production/Scale).
- System/service name; owner; environment(s) affected.
- Risk class (assistive internal, internal decision support, external content, automated decisions).
- Decision requested; options considered (max three) and why the chosen path wins.
- Evidence pack (links): architecture, data/corpus map, pipeline gates, SLOs and load tests, dashboards, safety results, cost model, portability abstractions.
- Decision and rationale; dissent noted; effective date; review date; success criteria.
- Conditions to proceed (what, owner, acceptance test, due date).
- Exceptions/waivers (control, mitigation, expiry) if any.
Acceptance test: The review record is complete and linked from the portfolio and decision log.
4) Reviewer prompts and checks (by layer)
Value and scope
- What KPI moves? What is out of scope in this phase? Which trade‑offs did you reject and why?
- Where does the design shorten lead time to value or reduce unit cost?
Data and corpora
- Which data products/corpora are critical? Are owners and SLAs named? Is classification current?
- Is access least‑privilege with masking? For GenAI corpora, are inclusion/exclusion rules enforced and documented?
Transformation and contracts
- How are schemas/contracts enforced (registry, CI)? What breaks consumers and how are they notified?
- What backfill/replay procedures exist and when were they last tested?
Model/prompt lifecycle
- Where do models/prompts live (registry/store)? Is lineage and provenance recorded? Can you reproduce a production artifact byte‑for‑byte?
Retrieval and grounding (GenAI)
- What are the chunking/embedding standards and versions? What retrieval thresholds (precision/recall@k) must be met for the tasks that matter?
- How will you ensure grounded answers with citations for external content?
Serving, routing, and performance
- SLOs: availability and p95 latency on user‑perceived operations—are they realistic, and where are the load‑test results?
- Routing: A/B, shadow, safety/latency/cost fallbacks—what are the triggers and where are they configured?
- Caching/batching: where applied, what are the target hit rates, and how do they affect cost and latency?
Observability and action
- Do dashboards include data freshness/quality/drift, model/task quality, retrieval metrics (if RAG), safety signals (if GenAI), latency/availability/errors, and cost?
- What is the alert‑to‑action SLA, who is paged, and what is the rollback trigger?
Security, privacy, and supply chain
- IAM and egress: where are the trust boundaries? Are secrets vaulted? Are image signing and SBOMs enforced?
- LLM security: prompt‑injection defenses, connector sandboxes, output validators for sensitive data.
Reliability and rollback
- Progressive delivery by default? Automatic rollback on what health signals? What did the last drill show?
- Backup/restore: when was the last successful restore for data stores, model registry, and vector indexes?
Compliance and responsible AI
- Which pre‑launch reviews apply? Are they embedded as pipeline gates with SLAs? Any explainability/fairness documentation required?
Cost and unit economics
- What is the unit cost per task at target volume/latency? Which levers (model class, context, cache, routing, quantization) keep you within target?
- What is the budget threshold that fails CI or blocks promotion?
Portability and exit
- Which abstractions shield you from lock‑in (LLM, embeddings, vector store, storage formats)? What portability drill will you run and when?
5) Go/No‑Go gates (must be true to move forward)
- Golden path: CI/CD with enforced gates for privacy, security, model risk, and—where applicable—retrieval quality and content safety.
- Rollback: canary/blue‑green configured with automatic rollback triggers; recent drill evidence exists.
- Observability: single owner‑named dashboard shows data health, drift, model/task quality, retrieval/safety (if GenAI), latency/availability, and cost; alert‑to‑action SLA defined.
- Data/corpora: owners and SLAs named; classification current; least‑privilege access with masking; GenAI corpora curated with retrieval thresholds set for representative tasks.
- Security: secrets vaulted; egress controls on tool/connector paths; SBOM and image signing in place.
- Restore: successful restore drills in the last six months for warehouse/lake, model registry, and vector store (if used).
- Cost: per‑use‑case unit‑cost visibility with design choices tied to targets; budget alerts configured.
- Portability: abstraction layer exists for at least one high‑dependency component with a dated drill.
Acceptance test: If any gate fails, the outcome is “Approve with conditions” or “Request changes,” and a cap is recorded in the readiness heat map until the condition is met.
6) Conditions catalog (pick and adapt)
- Convert [manual review] to [pipeline gate] by [date], evidenced by a failing build that blocks promotion.
- Complete rollback drill in [env] with [target time to restore]; attach logs by [date].
- Achieve retrieval precision@5 ≥ [X] and recall@20 ≥ [Y] on [task set]; attach evaluator report by [date].
- Enable prompt/output logging with [retention] and [access controls]; prove opt‑out for [provider] by [date].
- Reduce p95 latency to ≤ [ms] at [req/s]; attach load‑test results by [date].
- Bring unit cost per task within [target]; show design changes (routing/cache/model) and dashboards by [date].
- Execute portability drill for [component]; attach “time to switch” and gaps by [date].
7) Threat‑model quick sheet (AI‑specific)
- Data supply: poisoning risks, schema drift, compromised embeddings—what controls catch and quarantine?
- Prompt surface: injection vectors (user input, documents, tools), jailbreaks—what sanitization and detection?
- Retrieval: exposure of sensitive documents via over‑broad filters—what namespace/ACL and filter strategy?
- Output: harmful or ungrounded content—what filters, human‑in‑the‑loop thresholds, and override logging?
- Exfiltration: connectors/tools leaking data—what egress policies and audit hooks?
- Supply chain: dependency and container risks—what SBOM, signing, and CVE SLAs?
8) Evidence checklist (attach or link)
- Architecture diagrams; IaC repositories; CI/CD pipeline definitions and last three release logs; one failed‑gate example.
- SLOs and load‑test results; router and caching configs.
- Dashboards and alert routes; runbooks; last incident and post‑incident changes.
- Data catalog entries; lineage snapshots; access/masking policies; DPIA decisions (if required).
- Corpus registry; chunking/embedding standards; retrieval‑evaluation results; content‑safety configs; prompt/output logging policy.
- Vault/IAM extracts; SBOMs and signing policy; CVE triage logs; egress rules.
- Backup/restore drill records (including vector stores); portability drill record.
- Cost dashboards and a design note citing cost‑driven choices.
Acceptance test: Links resolve for reviewers without special favors; screenshots are allowed only where a system of record does not exist.
9) Anti‑patterns to flag immediately
- Notebook‑to‑prod or side‑channel releases bypassing gates.
- “RAG on a shared drive” without curation, classification, or retrieval evaluation.
- Missing rollback drill; untested restore for vector stores or model registry.
- Secrets in configs or permissive egress on tool/connector pathways.
- Observability without owners/runbooks; alerts ignored or not paged.
- Cost blindness; no unit‑cost targets or dashboards.
- Single‑vendor cul‑de‑sac with no abstraction or portability drill.
- Policy theater: beautiful standards, zero enforcement in pipeline.
10) Quality of the review (how you know this works)
- Decisions recorded the same day with conditions, owners, and dates.
- Time‑to‑decision for priority items ≤ 10 business days; ≥90% first‑pass decisions without deferral.
- Conditions closed on time ≥85%; exceptions with expiry <10% past due.
- At least one rollback drill and one portability drill executed per quarter across critical services.
- Post‑incident reviews result in new tests or gates within two sprints.
11) One‑page form you can copy into your workspace
Header
- Decision ID and date; stage; owner; chair; attendees.
- Decision requested (approve/approve with conditions/request changes/reject).
Context (≤150 words)
- Problem, user, KPI, non‑goals.
Design summary
- Diagram link; data/corpus map; model/prompt lifecycle; serving and routing; SLOs; observability; controls; cost; portability notes.
Risk and obligations
- Risk class; safe‑to‑operate checklist; required reviews; exceptions/waivers (with expiry).
Evidence links
- Architecture • IaC • CI/CD • SLOs/load tests • Dashboards • Data/corpus • Safety • Security • Restore • Cost • Portability.
Decision and rationale
- Why this path; alternatives considered; dissent (if any).
Conditions
- [Condition] — [Owner] — [Acceptance test] — [Due date].
Follow‑ups
- Backlog items and owners; next review date.
Use this template exactly once per material decision. Keep it short, link to the truth, and insist on gates over promises. Done this way, architecture reviews stop being a hurdle and become the mechanism that reliably converts clever designs into safe, fast, and economical production systems.
