AI creates value when its benefits exceed its total cost of ownership. That sounds obvious, yet many AI programs are launched without a clear view of what drives costs, what business models are viable, how to evaluate and contract with vendors, where markets are likely to concentrate, and what kinds of advantages are defensible. This chapter provides a pragmatic economics lens. We start with cost curves—how training and inference scale and how to bend those curves. We then compare business models, outline a rigorous vendor‑evaluation and contracting approach, discuss market structure (concentration versus open ecosystems), and close with the strategic moats that matter in practice: data, distribution, and integration.
15.1 Cost Curves: Training vs. Inference
Training and inference are economically different activities. Training is episodic, bursty, and risky (the run may not succeed). Inference is ongoing, demand‑driven, and tightly coupled to latency and reliability. The first is CapEx‑like even in the cloud; the second is OpEx with strong economies of scale in operations.
Training cost: what actually drives it
At a high level:
Training Cost = (Compute Hours × $/Hour) + Engineering + Data + Storage/Networking + Opportunity Cost
Under the hood, compute hours depend on four levers:
- Model size (parameters). For dense transformers, FLOPs per token scale roughly with parameter count. Bigger is not always better—see the current trend toward right‑sizing plus retrieval.
- Tokens seen (dataset size and epochs). Total training compute is roughly proportional to parameters × tokens for standard objectives. Data deduplication and quality control reduce wasted tokens.
- Parallelism efficiency. Poor scaling (communication overhead, topology mismatches, imbalanced pipelines) can double or triple the wall‑clock time, directly multiplying cost.
- Numerical precision and kernels. BF16/FP16, kernel fusion, and compiler optimizations materially raise throughput. Under‑optimized stacks squander hardware.
Hidden lines on the training bill.
- Data work. Curation, labeling, weak supervision, and filtering are frequently the largest non‑compute expense.
- Checkpoints. Frequent checkpointing lowers risk but increases I/O and storage.
- Retries and failures. A single fault near the end of a multi‑day run without robust checkpointing can erase days of spend.
- People time. Compiler specialists, distributed training engineers, and SREs are part of TCO.
How to bend the training curve.
- Start smaller; scale only with evidence. Use scaling‑law pilots to identify diminishing returns.
- Exploit parameter‑efficient tuning (PET). Adapters/LoRA reduce full‑fine‑tune needs; save base‑model refreshes for step‑change value.
- Prefer retrieval‑first over parameter bloat. Ground answers in corpora rather than enlarging models for knowledge.
- Quantization‑aware training where viable. Lower‑precision weights/activations reduce memory and improve throughput.
- Optimize parallelism. Keep high‑traffic collectives within the fastest interconnect; overlap compute and communication; right‑size micro‑batches.
- Use synthetic/augmented data judiciously. Fill rare‑case gaps without distorting distributions; validate on real hold‑outs.
Inference cost: where the money goes
Inference economics hinge on tokens and latency constraints:
Per‑Request Compute ≈ (Context Tokens + Generated Tokens) × (Compute per Token)
- Context length dominates. Long prompts and retrieval bundles multiply cost. Aggressively prune context to what is necessary; enforce token budgets.
- KV cache footprint. Generative models store key/value tensors per token; cache growth drives memory pressure and limits batch size, which in turn limits cost efficiency.
- Batching vs. latency. Dynamic batching (grouping tokens across requests) raises throughput and lowers per‑token cost, but increases tail latency if not tuned. The art is to set batch and scheduling policies that respect user expectations.
- Precision and quantization. INT8/INT4 inference reduces memory bandwidth and increases throughput when accuracy allows; combine with calibration and guardrails.
- Routing to the smallest acceptable model. Many requests do not need your largest model. A router that chooses the cheapest model meeting quality constraints can cut cost dramatically.
Techniques that move inference unit economics.
- Speculative decoding. A small “draft” model proposes tokens; the large model verifies fewer steps.
- Multi‑query attention and KV compression. Reduce memory and bandwidth per token.
- On‑device or near‑edge inference. For latency‑sensitive or privacy‑sensitive tasks, local inferencing reduces cloud spend and improves responsiveness—with constraints on model size.
- Response caching. For repeated prompts and deterministic tasks, cache entire outputs keyed by prompt + retrieval hash.
- Prompt discipline. The cheapest token is the one you never send. Trim system prompts, remove boilerplate, and design compact schemas.
Cost curves over time
Expect hardware improvement (more compute per dollar) and software improvement (better kernels, compilers, and quantization) to reduce per‑token costs year over year, but not uniformly. Memory bandwidth and energy can become binding constraints. Your best hedge is efficiency engineering and portfolio routing across model sizes and vendors.
Budgeting in practice
For each use case, produce a simple, reviewable model:
- Training: compute hours (with 20–30% contingency), data engineering hours, storage, and a risk reserve for retries.
- Serving: expected request mix, average and tail tokens per request, batch/latency settings, model‑route shares, cache hit rates.
- Sensitivity analysis: what happens to unit cost if request mix shifts, if context sizes drift up, or if routing degrades?
Instrument tokens per outcome (e.g., tokens per resolved case), not just tokens per request. This aligns engineering with business value.
15.2 Business Models: API, SaaS, On‑Prem, Hybrid
How you package and sell AI—or how you buy it—determines your revenue model, margins, and implementation risk. Four archetypes dominate.
API (Consumption‑based)
What it is. You expose model capabilities via a metered API (tokens, images, minutes, embeddings). Customers integrate the API into their products.
Strengths.
- Speed to market. Easy to adopt and scale.
- Elasticity. Demand spikes can be absorbed by the provider.
- Feature velocity. Customers benefit from the provider’s rapid model improvements.
Weaknesses.
- Gross margin sensitivity. Heavy compute costs and pass‑through expenses can keep margins thin.
- Platform risk. Customers may switch or multi‑home; differentiating beyond price and convenience is hard.
- Data constraints. Some customers cannot send sensitive data off‑prem.
How it wins. Developer experience, reliability, breadth of models/tools, compliance posture, and cost efficiency. Upsell via managed RAG, fine‑tuning, and observability.
SaaS (Application)
What it is. A complete product that solves a business task (support assistant, code assistant, FP&A copilot). Pricing is typically per seat, per task, or outcome‑based.
Strengths.
- Value capture. Closer to business outcomes; easier to justify ROI.
- Switching costs. Integration, workflows, and change management tie customers in.
- Differentiation. Domain prompts, retrieval corpora, and tool connectors become product features.
Weaknesses.
- Implementation work. Requires onboarding, content governance, and user training.
- Heterogeneous tenants. Ensuring performance and safety across varied corpora and tool sets is complex.
- Creep in cost of goods. As usage grows, inference cost can erode margins if not tightly managed.
How it wins. Focus, outcome metrics, retrieval over the customer’s sources of truth, strong safety and admin controls, and relentless UX iteration.
On‑Prem / Private Cloud
What it is. You deploy models and software into the customer’s environment (their VPC, data center, or sovereign cloud). Pricing may be subscription, license + support, or consumption within the customer’s account.
Strengths.
- Data control. Satisfies residency and confidentiality constraints.
- Performance predictability. Dedicated capacity and network control.
- Trust. Favored in regulated sectors and for mission‑critical systems.
Weaknesses.
- Sales and delivery cost. Longer cycles; custom integrations; complex support.
- Upgrade friction. Feature rollout lags; version sprawl.
- Hardware dependence. Customer’s fleet may be heterogeneous.
How it wins. Clear deployment patterns (reference architectures, Terraform modules), strong automation, and a product strategy that treats on‑prem as a first‑class citizen, not an afterthought.
Hybrid
What it is. Combine models and services across customer and provider environments. Examples: customer hosts RAG and indices; provider hosts models; or customer runs small models locally and bursts to provider for heavy tasks.
Strengths.
- Best of both worlds. Balance control and velocity.
- Cost optimization. Route by latency, sensitivity, and price.
- Resilience. Multi‑region/multi‑vendor strategies reduce outage risk.
Weaknesses.
- Complexity. Routing, observability, and policy enforcement across boundaries.
- Support surface area. More places for failures and misconfigurations.
How it wins. A strong orchestration layer, policy‑aware routing, and portable prompts and indices.
Pricing choices
- Tokens vs. seats vs. outcomes. Tokens align with provider cost but not customer value; seats align with budget processes but can mis‑align with usage; outcomes (e.g., cost per resolved case) are compelling if you can measure and guarantee quality.
- Bundles and tiers. Bundle retrieval, safety, and observability to lift ARPU. Offer tiers by latency/SLA and data residency.
- Minimums and commitments. Useful to de‑risk capacity planning; balance with elasticity to avoid deterring adoption.
15.3 Vendor Evaluation and Contracting
Procurement should be evidence‑based and portable. Build an evaluation that survives vendor turnover and internal audits.
A disciplined evaluation rubric
1) Fit for purpose.
- What task(s) does the product complete end‑to‑end?
- What quality/latency/cost metrics does it commit to?
- What is the failure mode (abstain, escalate, retry)?
- Does it support your data types (docs, tables, images, code) and tools (CRM/ERP/ticketing)?
2) Architecture and portability.
- Can you bring your own model or route to multiple providers?
- How are prompts, indices, and embeddings versioned and exported?
- Are APIs stable and well‑documented?
- Is there a path for on‑prem/VPC deployment if needed?
3) Security and privacy.
- Data handling (at rest/in transit encryption, key management, tenant isolation).
- Use of your data for training (default should be “no” unless contractually agreed).
- Logging and retention; DSAR and deletion workflows.
- Sub‑processor list and locations; breach notification and incident response.
4) Safety and governance.
- Content policies and moderation layers (input/output filters, citation/entailment checks).
- Red‑team program and regression suites.
- Human‑in‑the‑loop features and admin controls.
- Evaluation artifacts: model/system cards, slice metrics, and limitations.
5) Observability and SLOs.
- Metrics exposed (tokens, latency, error types, safety events, retrieval quality).
- Dashboards, alerting, and export to your telemetry stack.
- SLOs for latency and availability, with remedies.
6) Cost transparency.
- Clear unit prices and included features (e.g., embeddings, vector search, fine‑tuning).
- Egress fees, overage charges, regional pricing.
- Forecasting tools; soft/hard quotas; throttling behavior.
7) Roadmap and viability.
- Release cadence and backward compatibility.
- Financial and operational maturity; reference customers in your sector.
- Support SLAs and named technical contacts.
Contracting: clauses that prevent future pain
- Data processing and training rights. Specify that your data—inputs, outputs, embeddings, logs—will not be used for training or model improvement without explicit consent. If allowed, confine to anonymized aggregates with strict controls. Include deletion rights and evidence of deletion.
- IP and indemnities. Clarify ownership of outputs and fine‑tuned weights. Seek IP indemnity covering claims related to the vendor’s training data and model behavior, subject to your compliance with usage guidelines. Define caps and exclusions.
- Security exhibits. Require baseline controls (certifications, pen‑tests, vulnerability management) and incident SLAs. For regulated sectors, include sector‑specific controls.
- Availability and performance. Latency/uptime SLOs with credits; meaningful remediation beyond credits for chronic shortfalls.
- Change management. Advance notice for breaking API changes and model deprecations; long‑term support for previous versions.
- Termination assistance and portability. Upon exit, export of prompts, indices, embeddings, logs, and fine‑tune artifacts in open formats; cooperation for a fixed period; deletion of your data with attestations.
- Audit and assurance. Right to audit (directly or via independent assessor); access to model/system cards and evaluation summaries.
- Jurisdiction and data residency. Align with your regulatory obligations; name regions and restrict transfers where required.
A practical POC plan
- Run head‑to‑head evaluations on your data: quality (including faithfulness/citations), latency, and cost under realistic workloads.
- Include failure tests: ambiguous prompts, adversarial inputs, permission edge cases.
- Validate support responsiveness: time‑to‑fix for a realistic issue.
- Confirm observability integration with your logging/metrics.
- Document total effort: time and skills needed to reach production readiness.
15.4 Concentration vs. Open Ecosystems
The AI stack is stratifying. Understanding where markets concentrate and where they fragment helps you place bets and hedge risks.
Where concentration is likely
Compute and power.
Advanced accelerators, high‑bandwidth memory, leading‑edge packaging, and liquid‑cooled data center capacity are capital intensive and capacity constrained. Expect continued concentration among a small set of chipmakers and hyperscalers, with long lead times and pricing power cycling with demand. Energy availability and grid interconnection further constrain who can scale.
Frontier pretraining.
Training frontier‑scale foundation models remains a domain of a handful of labs with access to compute, data, and specialized talent. Scale confers learning benefits (infrastructure, research, data pipelines) and brand, which in turn attract customers and talent. This layer looks “winner‑take‑most”—for a while.
Where openness and fragmentation persist
Open models and mid‑scale.
Open‑weight models have improved rapidly and cover many enterprise needs, especially when paired with retrieval and fine‑tuning. Expect a barbell: a few frontier models at the top and a wide array of open and specialized models below, with fast iteration and healthy competition.
Applications and vertical solutions.
Customer value is realized in workflow change, not in raw tokens. Vertical apps differentiate through domain prompts, retrieval over proprietary corpora, tool connectors, and UX. Switching costs and integration depth favor competition and specialization rather than monopoly.
Developer tooling.
Vector databases, orchestration frameworks, evaluation harnesses, and safety tooling are competitive spaces with room for both open‑source and commercial offerings. Portability and API‑level standards will continue to improve.
Platform dynamics and risks
Aggregation vs. integration.
- Aggregators offer unified access to multiple models and tools with routing and billing. They lower switching costs but risk compressing vendor margins and becoming chokepoints.
- Integrators package models with data, workflows, and sector compliance into outcomes. They capture more value but assume more responsibility.
Commoditization pressure.
As models converge in capability on common tasks, value shifts to cost, safety, governance, and integration. Vendors without strong efficiency or differentiated workflows face margin pressure.
Standards and portability.
Emerging standards for AI management, documentation, and evaluation, alongside de facto standards for prompts and function calling, reduce lock‑in at higher layers. Your architecture should assume multi‑model routing and artifact portability.
Regulatory asymmetry.
New obligations (conformity assessments, incident reporting, training data documentation) may advantage large incumbents who can absorb compliance costs. Open ecosystems respond by sharing compliance tooling and templates. Expect both consolidation and innovation.
Practical hedge.
Design for optionality: model‑agnostic orchestration, exportable indices and prompts, and contracts with termination assistance. Avoid deep dependencies on proprietary prompt syntax or single‑vendor vector stores unless you price that risk explicitly.
15.5 Strategic Moats: Data, Distribution, Integration
“Moat” is often used loosely in AI. The durable advantages in practice are less about possessing the largest model and more about controlling the context in which AI delivers value.
Data moats (when they are real)
Proprietary, high‑quality, and refreshed data.
The strongest data advantages come from governed, authorized, and continually updated corpora that competitors cannot easily replicate. Examples: labeled defect images tied to a specific production line; longitudinal customer support transcripts with outcomes; domain‑specific ontologies and glossary‑driven chunking; permissioned financial or clinical datasets.
Structure over size.
Structured, well‑chunked, and richly tagged documents beat vast unstructured dumps. Retrieval quality (and thus answer quality) depends more on curation than on raw bytes. A company that invests in content life‑cycle, deduplication, and metadata achieves compounding returns.
Feedback loops.
Operational signals—accept/reject edits, escalations, error taxonomies—become high‑value training sets for preference optimization and task‑specific fine‑tunes. Owning the workflow means owning the feedback.
Limits of data advantage.
Public data rarely confer moat; competitors can obtain them. “Scraped the web” is not a moat. Similarly, hoarding uncurated internal documents without governance creates risk, not advantage.
Distribution moats
Existing channels.
Embedding AI into tools employees already use (ticketing, CRM, IDEs, office suites) lowers adoption friction and builds daily habit. If you control a channel—through existing software, service relationships, or marketplaces—you have a distribution edge.
Trust and compliance.
Approved‑vendor status, audit track records, and sector certifications are de facto distribution advantages. They reduce sales cycles and expand addressable markets.
Network effects.
In some domains, shared risk signals (e.g., fraud patterns) or benchmarking communities create network effects. These are strongest when participants contribute and benefit symmetrically, and when privacy‑preserving techniques allow aggregation without leakage.
Integration moats
Deep workflow integration.
Connecting AI to systems of record, approvals, and action tools creates switching costs. The value is not just the model but the orchestration: schema‑conformant outputs, validations, retries, and escalation. This work compounds and is hard to copy quickly.
Tool ecosystems.
A well‑maintained catalog of secure, schema‑validated tools (query, quote, book, create‑ticket) with clear idempotency and permissions is reusable across use cases. This tool fabric plus model routing becomes a platform moat.
Observability and SLO culture.
Reliable AI services require tracing, golden tests, drift monitors, and incident playbooks. An organization that can run AI—cheaply and safely—has an execution moat even if models are shared.
Where “model moat” still matters
- Ultra‑low latency or on‑device: custom small models tuned to your data and constraints.
- Special modalities (industrial time series, medical imaging): models trained on scarce, licensed data.
- Safety‑critical controls: models with certified performance under strict standards.
Even here, the moat is the system (data → model → safety → workflow), not just weights.
Building moats deliberately
- Own the corpus. Fund content cleanup, metadata, and access control. Treat the knowledge base as a product with owners and SLAs.
- Capture feedback. Bake rating/flagging and edit‑capture into UX. Curate hard‑case sets monthly.
- Right‑size models. Use retrieval and smaller models to reduce cost and make the unit economics robust; large models are easy to imitate but hard to afford.
- Invest in connectors. High‑quality, secure adapters to core systems are reusable assets.
- Governance as a feature. Policy controls, audit logs, and approvals aren’t just compliance—they are differentiators for enterprises.
- Measure outcomes. Tie value to business metrics (resolution rate, time saved, error reduction). Outcome evidence supports pricing power.
