Algorithms do not run in a vacuum. Every choice you make about model size, precision, batch size, or parallelism is constrained and enabled by the hardware beneath it. In practice, the dominant performance and cost drivers for AI systems are (1) the accelerator architecture you choose, (2) the memory hierarchy attached to it, (3) the quality of the interconnects that tie accelerators together, and (4) how effectively your software stack can parallelize computation and overlap it with communication and I/O. This chapter explains today’s compute landscape in plain language and provides practical guidance for leaders making investment and architecture decisions. We cover CPUs, GPUs, TPUs, and other specialized accelerators; the realities of memory and interconnect bottlenecks; core forms of parallelism; cost structures; and what semiconductor roadmaps and supply‑chain constraints imply for capacity planning.
6.1 CPUs, GPUs, TPUs, and Specialized Accelerators
CPUs: generalists with rich control logic.
Central Processing Units (CPUs) remain the control plane of most systems. They excel at tasks that involve branching, irregular memory access, and orchestration: scheduling, data loading, preprocessing, feature engineering on tabular data, and serving business logic in microservices. Modern CPUs include vector units (SIMD instructions) that accelerate dense numeric kernels, but their architectural budget favors caches and sophisticated control rather than massive numeric throughput. In AI pipelines, CPUs prepare data, manage kernels launched on accelerators, run retrieval and business rules, and handle I/O. For many classical ML tasks on structured data, a well‑provisioned CPU cluster is still the most economical option.
GPUs: parallel numeric engines.
Graphics Processing Units (GPUs) are optimized to perform the same arithmetic on many data elements in parallel—ideal for linear algebra, which underlies neural network training and inference. A single GPU contains thousands of lightweight arithmetic units (organized into blocks/warps), high‑bandwidth on‑package memory (often HBM), and specialized matrix math engines that accelerate low‑precision operations used in AI. GPUs shine when your workload can be expressed as large batches of matrix multiplications and convolutions with minimal branching. They struggle with heavy control flow or tiny, latency‑sensitive kernels that underutilize parallel units. For training, GPUs are the mainstream choice because they pair strong numeric throughput with a mature software ecosystem (libraries, kernel compilers, debuggers, profilers).
TPUs and systolic arrays: purpose‑built for matrix math.
Tensor Processing Units (and similar arrays in other ecosystems) implement matrix multiplication as a “systolic array,” where data stream rhythmically through a grid of multiply‑accumulate units. This delivers high utilization on dense linear algebra with predictable access patterns. TPUs rely on a compiler stack that maps models to this array with tight control over data movement. They are attractive for large‑scale training and inference when your models and tooling align with the compiler’s strengths and when you value consistent performance per watt across large pods.
Inference‑focused accelerators.
Dedicated inference chips trade some flexibility for efficiency, focusing on low‑precision matrix units, on‑chip SRAM to avoid off‑chip memory traffic, and deterministic latency. They can be compelling for high‑volume, steady‑state inference at known precisions (e.g., INT8, INT4) and batch sizes. Their economics depend on model fit (can you quantize without unacceptable accuracy loss?), the maturity of the compiler toolchain, and the effort to port and validate your models.
Wafer‑scale and spatial architectures.
A different approach is to spread compute over very large arrays with abundant on‑chip memory and local communication, reducing the need for off‑chip interconnect during training. These architectures aim to keep tensors “close to where they are used,” mitigating traditional memory bandwidth bottlenecks. They can simplify parallelization for very large models but require a software stack that maps layers onto a spatial fabric and careful model partitioning.
Edge NPUs and on‑device AI.
Smartphones, cameras, vehicles, and appliances now include Neural Processing Units (NPUs) designed for local inference. On‑device acceleration reduces latency, network cost, and privacy risk. NPUs are optimized for small to mid‑sized models at low precision with strict power envelopes. Techniques like quantization, pruning, and distillation are prerequisites for deploying to the edge. The business payoff is responsiveness and resilience; the constraint is model size and the difficulty of updating fleets.
How to choose.
If you train large deep learning models and want the broadest software support: GPUs.
If you train at scale within a specific compiler/runtime ecosystem and value consistent scaling: TPU‑like systems.
If your workload is steady-state inference at fixed precision and you can invest in model porting: inference accelerators.
If your models are small, structured, or mostly tabular: CPUs may suffice, potentially with vectorized libraries.
If low latency and privacy at the edge matter: NPUs on devices.
Whatever you choose, the machine’s usefulness is dominated by memory bandwidth, interconnect topology, and the maturity of the compiler stack. Raw FLOPS numbers are poor predictors of end‑to‑end throughput without those context details.
6.2 Memory, Interconnects, and I/O Bottlenecks
When a workload runs more slowly than expected, the culprit is often not arithmetic throughput but data movement. Three layers dominate: memory bandwidth and capacity, interconnects within and across servers, and I/O to storage and networks.
Memory hierarchy in brief.
Registers and caches (on‑core) are tiny but extremely fast.
On‑package memory (e.g., High‑Bandwidth Memory, HBM) offers very high bandwidth to the accelerator with moderate capacity.
Local DRAM (e.g., DDR on CPUs) has more capacity but lower bandwidth and higher latency.
Remote memory (via interconnect or CXL) increases capacity further but at significant latency/bandwidth penalties.
Storage (NVMe SSDs, object stores) is orders of magnitude slower and should be streamed and prefetched, not treated as random‑access memory.
Bandwidth vs. compute: the “roofline” intuition.
The speed of a kernel depends on how many arithmetic operations it can perform per byte moved from memory (arithmetic intensity). If the kernel’s intensity is low, it becomes memory‑bound: performance is capped by bandwidth, no matter how many FLOPS you have. Many attention and embedding operations fall into this category. If intensity is high (large matrix multiplies), the kernel can be compute‑bound and scale with accelerator throughput. Optimization means increasing intensity (fusing operations; reusing data in caches) or raising bandwidth (HBM, better interconnect). Profilers can reveal where your workload sits.
Attention to the KV cache.
Generative inference stores key/value (KV) tensors for each token to avoid recomputing attention over history. KV caches are large and bandwidth‑hungry; they often dominate memory use and memory traffic. Techniques like paged attention, KV quantization, multi‑query attention, and speculative decoding reduce the footprint and bandwidth pressure, raising throughput and lowering cost.
Interconnects: scale‑up and scale‑out.
Within a server (scale‑up): Accelerators connect via high‑speed links that allow rapid all‑reduce and tensor exchange (e.g., direct GPU‑to‑GPU links) and to the CPU via PCIe or similar. Topology (full mesh, ring, star) drives how well all‑reduce operations scale.
Across servers (scale‑out): Network adapters (NICs) connect nodes via InfiniBand or Ethernet with RDMA (remote direct memory access). Collective communication libraries implement all‑reduce, reduce‑scatter, and all‑gather over these links. For large training runs, the bisection bandwidth of the cluster fabric and oversubscription ratios at switches become limiting.
If your cluster’s fabric is oversubscribed or has a “long tail” of latency under load, training stalls waiting on collectives. The remedy is architectural: better topologies (fat‑tree, dragonfly), quality of service, congestion control, and traffic engineering; or algorithmic: reduce communication volume (gradient compression, larger batch sizes with careful tuning, optimizer state sharding).
CXL and memory expansion.
Compute eXpress Link (CXL) extends coherent memory across devices, enabling pooled or expanded memory beyond local DRAM/HBM. For AI, CXL can host large parameter or KV caches at lower cost per GB than HBM, trading bandwidth for capacity. It is not a panacea—latency is higher—but it is a useful tool for hosting large contexts or model shards when tight HBM is the bottleneck.
I/O and the data loader.
Starving accelerators for input is a chronic problem. Datasets must be sharded and prefetched from storage; preprocessing should be moved to the GPU where possible or parallelized on CPUs with pinned memory to accelerate host‑to‑device transfers. Compression reduces I/O but adds CPU load; caching frequently used shards on local NVMe eliminates network bottlenecks. Many “mysterious slowdowns” turn out to be data loader stalls or a few misbehaving storage nodes.
Storage choices.
Local NVMe provides high throughput and low tail latency for hot shards.
Distributed filesystems (parallel FS, object store gateways) simplify management but can show noisy neighbors under concurrent load.
Object stores are economical for cold data; add a caching layer for hot training shards.
Design for idempotence and retries; transient I/O failures should not kill multi‑day training runs.
6.3 Parallelism: Data, Model, and Pipeline
No single accelerator can train or serve the largest models at desirable speeds and costs. Practical systems rely on multiple forms of parallelism, often combined.
Data parallelism: copy the model, split the data.
Each accelerator holds a full copy of the model, processes a different mini‑batch, computes gradients, and participates in an all‑reduce to average gradients before the next step. Data parallelism is conceptually simple and scales until communication dominates compute. Techniques that help: larger batch sizes (within optimizer stability limits), gradient accumulation (simulate large batches with small memory), and overlapping gradient reduction with backpropagation.
Optimizer and state sharding (ZeRO/FSDP‑style).
Large models carry substantial optimizer state (e.g., momentum, variance) and gradients. Sharding these across devices reduces per‑device memory consumption, enabling larger models or batches. Fully Sharded Data Parallel (FSDP) and similar approaches also reshard parameters just‑in‑time for compute, then release memory, at the cost of extra communication. Correct configuration (shard granularity, auto‑wrap policies) is critical to avoid pathological communication patterns.
Model (tensor) parallelism: split layers across devices.
When a single layer is too large to fit on one accelerator, split its tensors across devices: each device computes a slice of the matrix multiply. This introduces all‑gather and reduce‑scatter operations inside each layer. Tensor parallelism demands fast intra‑node interconnect; otherwise communication dominates. Expert implementations fuse communication with compute to hide latency.
Pipeline parallelism: split the network depthwise.
Assign different layers (or blocks) to different devices in sequence; micro‑batches flow through like stages on an assembly line. Pipelines suffer a bubble (idle time) at the start and end of each iteration; using enough micro‑batches (1F1B or interleaved schedules) and balancing stage workloads minimizes idle time. Pipeline parallelism reduces per‑device memory but complicates optimizer state handling and load balancing.
Mixture‑of‑Experts (MoE) and expert parallelism.
MoE layers contain many “experts” (small sub‑networks); a learned router activates only a few per token. This increases parameter count without proportional compute—attractive for training efficiency. However, MoE demands expert parallelism (tokens for a given expert must meet on the same device), creating communication spikes and load‑balancing challenges (some experts get more tokens). Gating regularization and capacity constraints mitigate imbalance; the gain is better parameter efficiency for a given compute budget.
Sequence and context parallelism.
For very long sequences, split activations along the sequence dimension to fit memory. This requires attention variants or recomputation strategies to avoid quadratic memory growth.
Activation checkpointing and recomputation.
To trade compute for memory, drop intermediate activations during forward pass and recompute them during backpropagation. Checkpointing allows larger batch sizes or models at modest compute overhead.
Overlapping compute and communication.
Modern training stacks schedule reduce‑scatter while later layers compute, or prefetch weights for the next layer. The difference between a naive and a tuned schedule is dramatic: the latter can hide much of the communication behind compute and approach linear scaling on moderate cluster sizes.
Inference parallelism and scheduling.
Serving generative models introduces new dynamics: each request grows a KV cache over time; decoding is sequential per request but parallel across requests. Throughput depends on batching tokens from multiple requests at each step (dynamic batching), scheduling to keep warps busy (group similar sequence lengths), and memory‑aware admission control (reject or queue requests when KV cache is full). Multi‑model routing (choose the cheapest acceptable model) and speculative decoding (a small model drafts tokens the large model verifies) further improve cost and latency.
Putting it together.
Large training jobs often combine all of the above: data parallel across nodes; tensor parallel within a node; pipeline across groups of layers; activation checkpointing to fit in memory; and sharded optimizer state. The art is choosing the partitioning that matches your topology: keep high‑communication dimensions within the fastest links; keep low‑communication dimensions across nodes.
6.4 Cost Structures: CapEx vs. OpEx
Hardware strategy is not just about performance; it is about total cost of ownership (TCO) and the flexibility your business needs. AI cost has two primary components: training (episodic, bursty, high‑capex feel) and inference (continuous, demand‑driven, Opex‑heavy). Each pushes you toward different choices.
Cloud OpEx: speed and elasticity.
Cloud accelerators let you rent capacity as needed. Advantages: rapid access to new hardware generations; elastic scaling for bursts; managed networking and storage; and the ability to right‑size clusters to jobs. Downsides: higher per‑unit cost; possible scarcity for popular accelerator SKUs; egress fees; and dependence on a provider’s roadmap and quotas. For many teams, cloud is the fastest way to begin and the safest way to experiment.
On‑prem/colo CapEx: control and unit cost.
Owning hardware can lower cost per unit of compute and ensure availability if you can keep it busy. It also enables custom interconnects, data residency control, and predictable performance. Downsides: acquisition lead times; up‑front cash; depreciation risk if models or hardware generations move quickly; and the operational burden of power, cooling, firmware, spares, and staffing. Utilization is the make‑or‑break variable; idle accelerators erase any unit‑cost advantage.
Hybrid: match workload shape.
A pragmatic approach is to own a base load sized to steady inference and training needs, while bursting to cloud for spikes, large experiments, or when you need a hardware feature you do not own. Hybrid adds complexity (two toolchains, networking), but it hedges supply and pricing risk.
The utilization imperative.
Accelerators are expensive; value comes from high occupancy. For training, a scheduler that packs jobs, preempts low‑priority runs, and backfills gaps with smaller experiments raises utilization. For inference, multi‑tenant serving, dynamic batching, and model routing (send easy requests to smaller models) improve tokens per second per dollar. Measure tokens per joule and tokens per dollar; they align engineering with business outcomes.
Precision and thrift.
Lowering numeric precision (FP16 → BF16 → INT8/4) yields speed and capacity gains with careful quantization. Pruning, distillation, and low‑rank adaptation shrink models. Retrieval‑augmented generation reduces the need for very large models in knowledge tasks by grounding responses in your corpus. For many use cases, a well‑architected small model with retrieval outperforms a larger model without grounding at a fraction of the cost.
Hidden lines on the bill.
Networking. High‑performance fabrics and NICs are costly; cloud cross‑AZ traffic and egress add up.
Storage. Training at scale requires multiple petabytes of high‑performance storage and disciplined lifecycle policies.
Power and cooling. Accelerator racks have high power density; liquid cooling may be required. Energy prices, PUE (power usage effectiveness), and grid interconnection timelines affect TCO.
People and software. Compiler engineers, cluster schedulers, SREs, and data engineers are part of the cost. So are software licenses for orchestration, observability, or enterprise support for frameworks.
Budgeting a training run.
A sober budget decomposes into: (1) compute hours × price per hour (with a contingency factor for retries); (2) storage for checkpoints and datasets; (3) network for data staging and distributed training; (4) engineering time for data prep and tuning; and (5) opportunity cost of tying up scarce hardware. Include checkpoint cadence decisions: frequent checkpoints reduce risk of losing progress but consume I/O and storage.
Contracts and commitments.
Cloud providers offer discounts for committed use; negotiate terms that match your workload’s profile. For on‑prem, align procurement with product roadmaps and consider resale or repurposing plans. In either case, lock in support SLAs for firmware, drivers, and RMAs; downtime on a training cluster is expensive.
Sustainability and reporting.
Expect stakeholders to ask for energy and emissions accounting. Choose regions and facilities with cleaner grids; schedule non‑urgent jobs when renewable generation peaks if pricing supports it; prefer efficient precisions and architectures. Reducing energy often correlates with reducing cost.
6.5 Roadmaps and Constraints in the Semiconductor Supply Chain
Leaders often plan assuming compute will get faster and cheaper each year. It generally does—but not uniformly, and not in the same way for all components. Today’s bottlenecks are as likely to be packaging and memory as they are pure transistor density. Understanding the supply chain helps you set realistic expectations.
Moore’s law vs. system‑level gains.
Transistor density continues to improve, but Dennard scaling (which once kept power per transistor constant as they shrank) ended long ago. Performance and efficiency gains increasingly come from specialization (matrix engines), parallelism, lower precision arithmetic, and advanced packaging (bringing memory and compute closer). Expect incremental per‑core gains and step changes when a new architecture, interconnect, or memory generation lands.
HBM is a major constraint.
High‑Bandwidth Memory provides the throughput modern models need but is capacity‑constrained and expensive. Each accelerator’s HBM stacks impose a hard cap on model size per device and a supply‑chain cap on how many accelerators vendors can ship. As models and contexts grow, designers push for more HBM per device and more devices per node, but both stress packaging and power envelopes.
Advanced packaging capacity.
2.5D/3D packaging (e.g., chiplets on interposers, stacked memory) requires specialized substrates and equipment. Packaging capacity—not just wafer fabrication—often gates accelerator shipments. Lead times can be long; forecast accordingly. Chiplet designs help yields but demand sophisticated interconnects between tiles.
Interconnect evolution.
Within nodes, high‑speed links between accelerators keep tensor and optimizer shuffles fast. Across nodes, network fabrics push toward higher speeds and lower latency; optical interconnects continue to improve but face cost and power trade‑offs. System builders pursue topologies that deliver high bisection bandwidth with acceptable cost. If your training plan assumes global all‑reduce at massive scale, verify that your fabric can sustain it under load.
Compiler and software stack maturity.
New accelerators arrive with evolving compiler stacks. Performance hinges on kernel fusion, layout selection, and autotuning. Early adopters may see wide variance between “headline” FLOPS and end‑to‑end throughput. Factor engineering time for porting and tuning into any roadmap; toolchains typically mature rapidly but not instantly.
Foundry and geopolitical concentration.
Advanced nodes are produced by a few foundries; EDA tools, lithography equipment, and some materials have limited suppliers. Export controls, trade policy, and regional incentives can affect availability and pricing over multi‑year horizons. Diversifying your hardware portfolio and designing software that is portable across vendors hedge this risk.
Adjacencies matter: NICs, switches, optics, power.
Building a training cluster requires more than accelerators. You need high‑end NICs, top‑of‑rack and fabric switches, optical transceivers, precision timing, ample power delivery, and often liquid cooling. Shortages in ABF substrates, silicon photonics, or high‑speed optics can delay deployments even when accelerators are available. Treat the cluster as an integrated bill of materials, not as a single‑SKU purchase.
Lead times and procurement strategy.
For on‑prem, expect long lead times for cutting‑edge parts and longer for complete racks. Place orders early, in tranches aligned with staff ramp‑up and facility readiness. For cloud, capacity for top accelerators can be scarce during launch windows; plan model milestones around realistic allocation timelines.
What roadmaps imply for model strategy.
Right‑size models. Efficiency gains from quantization, distillation, and MoE can arrive faster than hardware. Avoid locking into monolithic architectures that demand only frontier‑class accelerators.
Exploit retrieval. Offload knowledge to corpora and retrieval rather than growing parameter counts for many enterprise tasks.
Embrace portability. Use frameworks and abstractions that target multiple back‑ends (libraries, compilers). Avoid kernel lock‑in unless you can sustain it.
Design for heterogeneity. Expect clusters with mixed generations of accelerators; build schedulers and serving stacks that can route by capability.
Executive questions that reveal readiness.
What is the memory bandwidth per device, and is my workload compute‑ or memory‑bound?
What is the intra‑node and inter‑node interconnect topology, and how do my chosen parallelism strategies map to it?
Can my compiler stack fuse the kernels I care about and target the precisions I need?
What is my plan to quantize or distill models to fit lower‑cost hardware without unacceptable loss?
How will I monitor tokens per second per watt/dollar and optimize for them?
What are my lead times for new capacity, and how do they align with product milestones?
If my preferred accelerator is unavailable, what is my fallback (alternative vendor, smaller models, retrieval‑first)?
Do my contracts guarantee training rights for my data and support SLAs for firmware and drivers?
