Building an AI system is only the first half of the work. Operating it—reliably, safely, and economically over months and years—is the second half that determines whether the system becomes a durable capability or an expensive prototype. “Operations” here is broader than infrastructure uptime. It includes the lifecycle of data and models, observability for quality and safety, incident response and rollback, change management for both software and human workflows, and choices about what to build, buy, or partner for. This chapter lays out practical, repeatable patterns for running AI systems in production.
The key idea is treat models and their scaffolding as products. They have owners, roadmaps, budgets, service‑level objectives (SLOs), safety constraints, and customers. Their artifacts—datasets, prompts, indices, fine‑tunes, reward models, safety filters, and tools—are versioned, tested, and released with the same discipline as code. Their performance is measured not only by offline benchmarks but by on‑the‑ground outcomes, and their risk is managed like any other operational risk.
12.1 MLOps and Model Lifecycle
MLOps adapts software delivery practices to the peculiarities of machine learning and generative systems. The lifecycle is continuous and multidisciplinary: data engineering, model development, evaluation, deployment, monitoring, and improvement.
Artifacts and versioning
Traditional MLOps focused on models and feature code. Operating modern AI systems adds more artifacts:
Data assets: raw snapshots, curated training/eval sets, labels, weak labels, and datasheets documenting provenance and constraints.
Model artifacts: base checkpoints, fine‑tunes, adapters/LoRA weights, alignment/reward models, safety classifiers, calibration maps.
RAG assets: tokenizers, embedding models, vector indices, re‑rankers, and collection versions of documents.
Orchestration artifacts: prompts and templates, tool definitions and schemas, routing rules, guardrails, and business constraints.
Evaluation assets: golden sets, challenge sets, and test harnesses (offline and shadow).
Operational config: model routing policies, cost/latency budgets, escalation rules.
Every artifact should be immutable once released and tracked in a registry with metadata (owner, version, training/eval data IDs, intended use, limits, safety notes, and approval state). Store diffs for prompts and policies just as you do for code.
Environments and gates
Separate environments support safety and speed:
Dev for exploration and quick iteration.
Pre‑prod (or staging) for full‑stack tests with production‑like data and identities, but no customer impact.
Canary/Shadow for limited production exposure: either mirrored traffic with no side‑effects (shadow) or a small live slice (canary).
Prod for general availability.
Advancing between environments requires passing gates:
Data readiness: provenance documented; privacy checks passed; leakage audit completed.
Offline quality: target metrics met on golden and challenge sets; fairness slices reviewed; safety tests passed.
Operational readiness: SLOs defined; dashboards and alerts configured; runbooks and rollback plans in place.
Change approval: risk review completed (for regulated functions); sign‑offs recorded in the registry.
Continuous delivery with periodic training
Match update cadence to artifact type:
Indices and retrieval corpora: hours to days, tied to source system changes.
Prompts and routing policies: days to weeks; small, reversible deltas with canaries.
Safety filters and rules: weekly as policies evolve or new attack traces appear.
Parameter‑efficient fine‑tunes: monthly/quarterly after accumulating corrections.
Base model upgrades: quarterly or slower; treat as major versions with broad evaluation.
Automate the boring parts: dataset materialization, deduplication, index rebuilds, embedding jobs, and smoke tests. Keep a human in the loop for go/no‑go decisions on risky updates.
Reproducibility and lineage
For each released model or index, record:
Data and label versions (IDs, timestamps, source systems).
Code commit hashes for preprocessing and training.
Hyperparameters, random seeds, and hardware.
Evaluation results (overall and by slice) and sign‑offs.
Known limitations and approved use‑cases (“model card”).
This lineage enables audits, incident forensics, and compliance with model risk management norms.
Feature and tool stores
For prediction systems on tabular/structured data, a feature store ensures training/serving parity and avoids bespoke pipelines per model. For agentic systems, a tool store plays a similar role: cataloged, versioned APIs with schemas, preconditions, idempotency guarantees, permissions, and owner contacts. Both stores are crucial for operational safety.
Capacity planning and cost
Plan for tokens per second (or requests per second) and tokens per dollar. For serving, balance batch size and latency, and route to smaller models whenever quality allows. For training, schedule jobs to maintain high accelerator utilization, and checkpoint frequently to reduce risk of loss. Track cost by artifact (index rebuilds, embedding refreshes, fine‑tunes) to prevent surprise bills.
12.2 Observability: Data Drift, Performance Drift, Safety Drift
Observability for AI goes beyond infrastructure metrics. You need multi‑layer visibility: inputs, intermediate artifacts (retrieval context, tool calls), outputs, downstream outcomes, cost, and safety events. The goal is early detection of undesirable changes and enough context to diagnose causes.
Three kinds of drift
Data drift (covariate/prior shift). The distribution of inputs changes. In RAG systems, the corpus changes (new policies, formats) and the queries change (new product names, emerging issues). In predictive systems, feature distributions shift (new SKUs, unusual conditions).
Performance drift. The system’s ability to achieve its goals degrades. Accuracy falls on golden sets or in production proxies; latency creeps up; cost per task rises; acceptance/edit rates deteriorate. Often caused by data drift, code changes, or adversarial behavior.
Safety drift. The rate or severity of policy‑relevant events changes: toxic outputs, PII exposure, jailbreak success, tool misuse, permission violations, or fairness disparities. Safety drift can be subtle—e.g., a rising share of borderline outputs that pass filters but get flagged by reviewers.
What to measure—and where
Inputs:
Query mix (intents, languages, length).
Feature distributions (mean/variance, quantiles, missingness).
Retrieval context quality (recall@k proxies, re‑ranker scores).
Access control filter selectivity.
Processing:
Prompt and template versions; context lengths.
Tool call frequency, success/failure reasons, latency.
Safety classifier scores on inputs/outputs (histograms, tails).
Outputs:
Schema validity rates; citation presence and entailment checks.
Calibration (self‑reported confidence vs. correctness from samples).
Draft acceptance rate and edit distance (for assistive flows).
Abstention and escalation rates.
Outcomes:
Task success (resolution, accuracy) and quality (QA scores).
Latency (p50/p95/p99) and availability.
Cost (tokens/request, tokens/outcome; index queries; tool costs).
Fairness slices (accuracy/latency/cost by region, segment, device).
Events/Incidents:
Safety triggers (blocked/allowed, false pos/neg).
Policy exceptions, permission denials, and audit failures.
User‑reported issues.
Instrument with structured traces that tie a request to its retrieval bundle, model version, tool calls, and output. Sample a manageable subset for deep labeling each week; production is too noisy to infer everything from logs alone.
Detecting drift (without crying wolf)
Statistical tests: population stability index, Kolmogorov–Smirnov for numeric features; Jensen–Shannon divergence for embeddings; chi‑square for categorical. Use change‑point detection to avoid alert fatigue on small fluctuations.
Reference windows: compare current distributions to rolling baselines and seasonal peers (e.g., this week vs. the same week last year).
Golden canaries: run a fixed battery of curated prompts and documents daily; alert on regressions beyond control limits.
Business KPIs as sentinels: when re‑open rate or deflection rate moves, drill down to model‑level metrics.
Guardrail monitors: if abstention spikes, retrieval may be weak; if citations drop, prompt formatting may have regressed.
Tie alerts to runbooks. An alert without a documented next step is noise.
Fairness and subgroup observability
Monitor performance across relevant segments: geography, device class, channel, tenure, language, and, where appropriate, protected attributes. Keep privacy and legal guidance in mind when handling sensitive attributes—use approved proxies or audited processes. Investigate disparities; they often trace to data coverage gaps or policy misalignment.
Observability for cost and energy
Treat cost and energy as first‑class signals:
Tokens/request and tokens/outcome by model route.
Cache hit rates (retrieval and generation).
Index query counts, latency, and memory.
Accelerator utilization; batch size effectiveness; p95 tail waste.
Estimated energy per request (if available) and region mix.
Cost/energy drift frequently precedes performance drift (e.g., longer prompts, bloated contexts, inefficient routing).
12.3 Incident Response and Model Rollbacks
Incidents happen. A safe operator plans for them, reduces blast radius, and learns quickly.
Define severities and triggers
Agree on severity levels (e.g., Sev‑1 life/safety or major financial risk; Sev‑2 policy breach or material customer impact; Sev‑3 degraded service; Sev‑4 minor regression) and triggers (e.g., any verified PII leak is Sev‑1). Document who declares and who leads.
Prepare runbooks and controls
Kill switches: immediately disable a route, a tool, a model, or a feature flag.
Rollbacks: blue/green or canary infrastructure for instant reversion to prior artifacts (models, prompts, indices).
Quarantine: isolate bad content (poisoned documents, malicious prompts), and block their retrieval.
Safe mode: degrade gracefully—smaller model, stricter prompts, higher abstention threshold, “assist only,” or static FAQ.
Compensating controls: post‑filter outputs, require dual approval, or temporarily remove dangerous tools.
Ensure on‑call staff have permissions and single‑button pathways to execute these controls.
Investigate with the right evidence
Event timeline: detection time, first response, containment, communications.
Traces: full request context (prompts, retrieval, tool calls), model versions, cache hits.
Diffs: what changed recently? (index version, prompt tweak, model route, safety filter update)
Scope: how many users, tenants, or documents are affected?
Root cause: classify (data, code, model, policy, infra, adversarial).
Avoid “blame the model” as shorthand. Incidents often stem from system issues: bad indexing, broken permission filters, or a prompt change.
Communicate and remediate
Internal comms: concise situation report with impact, actions, ETA for next update.
External comms: if customers are affected, follow contractual and regulatory rules.
Remediation: patch prompts or filters, hotfix retrieval logic, revert or retrain models, restore permissions, purge caches.
Post‑incident review: within a week, run a structured review (what happened, why, how prevented/mitigated next time). Add tests and monitors. Track action items to closure.
Rollback strategy details
Artifacts: keep at least the last two production‑approved versions of each artifact warmed in serving. Rolling back a model without rolling back its prompt or index can make things worse; lock compatible sets (model + prompt + index + filters).
State: remember that indices and caches are state. Invalidate smartly; retain a map of doc‑ID to index version.
Data repair: when labels or documents are wrong, correct them and rebuild—partial repairs propagate confusion.
Humans: enable broad “assist mode” quickly. People can absorb spikes if they get context and a short‑term process.
Common incident classes—and fast paths
PII leakage or unsafe content: trigger kill switch for the offending route; raise safety thresholds; inspect safety classifier performance; identify and sanitize poisoned inputs.
Permission breach in retrieval: stop retrieval for affected tenants; restore permission filters; rebuild index with correct ACLs; audit access logs.
Hallucination spike: check index freshness; validate prompt tokens (e.g., lost “answer only from sources” instruction); tighten grounding; increase abstention; sample outputs for triage.
Latency/cost runaway: inspect prompt/context size growth; re‑enable batching; confirm KV cache behavior; fix regression in speculative decoding or quantization.
Tool misuse: restrict tool to allow‑list; add input validation; require confirmation; review tool schema clarity.
12.4 Change Management and Training End‑Users
AI systems are socio‑technical. Changing them means changing how people work. Good change management shortens time‑to‑value and reduces risk.
Release discipline without paralysis
Adopt a release train for AI changes (e.g., weekly prompts/indices; monthly fine‑tunes), with emergency paths for security fixes. Each train includes:
Release notes, written for non‑engineers: what changed, why, benefits, known risks.
Deprecation notices with dates and migration steps.
Change windows aligned with business cycles (avoid quarter‑end for finance).
Limit the surface area of each release; bundle small prompt updates together; avoid simultaneous upgrades to multiple artifacts unless required.
Golden tests and acceptance criteria
Before production, run golden traces: representative end‑to‑end scenarios with expected outputs (or ranges). For generative tasks, acceptance can be “all citations present,” “schema valid,” or “no safety triggers,” rather than exact text match. Keep a lightweight human review for grey areas.
Training and enablement
Train users on:
What the system is and is not. Clarify scope, confidence, and refusal behavior.
How to read outputs. Citations, confidence, and when to escalate.
How to give feedback. In‑product flags, categories (incorrect, incomplete, unsafe), and examples.
What changes. New buttons, drafts, or fields; where “assist mode” appears; what to do in edge cases.
Use brown‑bag sessions, short videos, and embedded champions. Provide microcopy inside the product that reminds users how to proceed (“Review citations before sending”).
Managing adoption
Start with volunteers and early adopters.
Publish adoption dashboards (share of tasks using assist, acceptance rates).
Collect qualitative wins and misses; share them to calibrate expectations.
Tackle sticking points (UI friction, incorrect defaults, missing tools) before wider rollout.
Policy alignment and legal review
Coordinate with legal, risk, and compliance on:
Approved uses and prohibited uses (documented in the model card).
Disclosure (when users/customers should be told an AI is assisting).
Records (what gets logged, retained, and produced in audits).
User agreements (acceptable use; data handling).
Capture sign‑offs in the registry for traceability.
Managing prompt and content sprawl
Prompts and knowledge proliferate. Prevent drift by:
Templates and prompt libraries with ownership and reviews.
Style guides encoded in prompts and checklists.
Content governance for the corpus: owners, freshness SLAs, deprecation policies.
Index hygiene as a product: de‑duplication, chunking standards, metadata completeness.
Organizational operating model
Successful teams formalize roles:
Product owner (use case outcomes, roadmap).
Model owner (artifacts, quality, safety).
Data steward (corpus, labels, indexing).
Platform/SRE (serving reliability, cost).
Safety lead (policy, red‑team, incidents).
Analytics/FinOps (cost and impact).
Customer enablement (training and support).
Meet regularly (e.g., fortnightly) to review metrics, incidents, and upcoming changes. Publish a brief model operations report monthly.
12.5 Build vs. Buy vs. Partner Strategies
Operating well also means choosing where to invest engineering time. Not every component needs to be built in‑house. The right mix depends on problem specificity, risk posture, and time‑to‑value.
What to build
Build when the component is:
Core to differentiation (your proprietary workflows, domain prompts, tool logic, or evaluation).
Entwined with sensitive data where control is paramount (e.g., on‑prem inference for PHI/PII; retrieval over confidential corpora).
An integration layer unique to your systems (tooling into your ERP/CRM/line‑of‑business applications).
Governance‑critical (policy enforcement, permission filters, audit trails).
Building yields control and adaptability but demands sustained ownership (roadmap, SLAs, on‑call).
What to buy
Buy when the capability is:
Commodity and not your differentiator (vector databases, schedulers, model hosting, observability platforms, transcription services).
Rapidly evolving where a vendor invests more than you can (frontier model access, compiler optimizations, inference kernels).
Cross‑cutting across teams (annotation tools, experiment platforms, feature stores).
Buying reduces time‑to‑value and spreads R&D cost, but introduces integration work and vendor risk (lock‑in, roadmap mismatch). Prefer vendors with clear SLAs, data handling terms, export paths, and usage transparency.
When to partner
Partner when:
You need co‑development in a regulated or niche domain (shared playbooks, safety data, and evaluation).
You seek knowledge transfer (standing up your platform with an experienced guide).
You face ecosystem dependencies (e.g., using a model vendor’s safety stack or tool ecosystem).
Partnerships should include exit plans (IP ownership, data portability), joint success metrics, and security reviews.
Open source considerations
Open models and tools offer transparency and control, especially for on‑prem or air‑gapped deployments, and can reduce unit cost. They require in‑house expertise for hardening, updates, and security patching. Evaluate:
License (commercial rights, copyleft obligations).
Community vitality (contributors, release cadence).
Compatibility with your stack (hardware/compilers).
Benchmarks on your tasks (not only public leaderboards).
A common pattern is hybrid: open‑source for infrastructure and models where fit is good, commercial APIs for niche high‑accuracy components (e.g., OCR languages, speech accents), with an internal routing layer.
Total cost and risk assessment
For each make/buy decision, quantify:
Build: engineering headcount, time to MVP, on‑call burden, security/compliance cost, and opportunity cost.
Buy: subscription/usage fees, integration, egress, data residency, vendor viability, and lock‑in (how easy to switch).
Risk: failure modes, incident liability, SLA penalties, and regulatory exposure.
Express the comparison in time‑to‑value and payback: speed often wins early; ownership pays back at scale if utilization is high and differentiation matters.
Avoiding lock‑in while being pragmatic
Abstraction layers for models (adapters that support multiple providers) and vector stores (portability tools).
Data gravity: keep embeddings and indices exportable; store raw corpora and labels in your systems of record.
Prompt portability: maintain prompts in your registry; avoid vendor‑specific syntax unless wrapped.
Contracts: include data deletion, training rights (or lack thereof), and termination assistance.
Lock‑in is not always bad; it is a risk to be priced and mitigated, not a categorical prohibition.
