Artificial intelligence does not live in the abstract. It inhabits buildings—big, heavy, meticulously engineered sites that convert electricity into computation and heat, move bits across fiber with minimal loss, and maintain continuous service through utility disturbances, weather, and human error. If models are the heart of AI, data centers and their power systems are the circulatory and respiratory system. This chapter explains, in plain language, how modern AI data centers are put together; why cooling, power density, and location drive an unusual share of design decisions; how grid interconnection and energy procurement actually work; what “sustainability” means beyond marketing; and how to choose among edge, cloud, and on-prem options.
7.1 Architecture of Modern AI Data Centers
Modern AI facilities are purpose-built factories for linear algebra. While they resemble traditional hyperscale data centers, several differences are decisive: higher rack power, tighter coupling between compute and network fabric, and cooling solutions that assume sustained high utilization.
From site boundary to server. A canonical stack looks like this:
Utility interface and substation. The facility connects to the transmission or high-voltage distribution grid through a dedicated substation. Step-down transformers convert, for example, 115 kV or 69 kV to medium voltage (13.2–34.5 kV). Redundancy at this level—dual utility feeds, ring buses—reduces exposure to single contingencies, though genuine dual feeds from independent substations are rarer than marketing implies.
Switchgear and distribution. Medium-voltage switchgear distributes power to multiple power trains. Each train feeds uninterruptible power supplies (UPS) and then low-voltage switchgear that supplies power distribution units (PDUs) on the floor. Architectures vary: N, N+1, 2N, or distributed redundant topologies trade capital for resilience.
UPS and energy storage. Static UPS systems (double-conversion) with large lithium-ion banks are now common; rotary UPS with flywheels appear where operators prefer mechanical inertia. Battery energy storage systems (BESS) increasingly support ride-through, fast-frequency response, and limited peak shaving. Runtime targets are typically 2–10 minutes—enough to start generators.
Emergency generation. Diesel gensets remain standard because of energy density and reliability. Gas turbines or reciprocating gas engines appear where permitting discourages diesel, where fuel logistics favor gas, or where operators want low-carbon firm capacity. On testing and emissions: environmental permits limit annual runtime; fuel polishing and periodic load tests are essential.
White space and data halls. AI halls are laid out to minimize network hop count and maximize uniformity. Spine-leaf network blocks are co-located with the racks they serve. Hot aisle/cold aisle containment is universal; in AI halls, cold plates, rear-door heat exchangers (RDHx), or immersion tanks interrupt the traditional airflow picture.
Racks and enclosures. AI racks draw 30–80 kW today, with 100 kW+ designs in flight. Traditional open-frame racks give way to enclosed cabinets with liquid manifolds and leak detection. For immersion, tanks replace racks entirely, and the server board design is modified for dielectric fluid compatibility.
Compute and fabric. Accelerators are installed in baseboards (8–16 GPUs/TPUs per server) with NVLink, NVSwitch, or equivalent high-speed intra-node links. Nodes are aggregated into pods with fat-tree or dragonfly fabrics (InfiniBand or Ethernet/RDMA). The pod boundary is a performance boundary: most all-reduce traffic should stay inside it.
Storage tiers. Hot training shards live on local NVMe; shared parallel filesystems or object storage handle checkpoints, datasets, and logs. Metadata servers and object gateways are scaled to avoid thundering herds at job start.
Security and safety. Multi-factor perimeter access, mantraps, camera coverage, cage separation, gas or water-mist fire suppression, and leak detection are table stakes. AI halls add drip trays, quick-close valves, and fluid containment for liquid loops.
Uptime tiers and what they really mean. Uptime Institute’s Tier I–IV designations describe fault tolerance and maintenance levels. For AI training, the business tolerance for interruption differs from transactional workloads. A brief interruption is operationally expensive but may be acceptable if checkpoints are frequent. Many operators prefer concurrently maintainable designs (Tier III-ish) with strong operational discipline over maximal redundancy that increases complexity and failure modes.
Software meets building. The most successful AI sites treat power and cooling as software-defined systems: telemetry from PDUs, UPS, BESS, CRAC/CRAH units, pumps, and valves feed into a control plane that orchestrates load, temperature, and redundancy dynamically. On the compute side, cluster schedulers expose power and thermal budgets to job placement, avoiding local hotspots and derates.
7.2 Cooling, Power Density, and Site Selection
AI shifts the limiting factor from floor area to heat rejection. Understanding the options and their trade-offs is central to cost and risk.
Cooling options.
Air cooling (enhanced). Still viable up to ~20–30 kW/rack with aggressive containment, high-CFM fans, and raised supply temperatures. Beyond that, fan power and airflow become impractical. Air remains useful for ancillary racks and network gear.
Rear-door heat exchangers (RDHx). Water-cooled doors mounted on the rear of racks absorb most heat before it enters the room, extending air’s viability into the 50–80 kW/rack range. They require building chilled-water loops, quick-disconnects, and leak sensors.
Direct-to-chip (cold plate). Liquid circulates through cold plates on CPUs/GPUs, removing heat at source. Warm-water designs (e.g., 40–50 °C supply) enable free cooling for much of the year in temperate climates. This approach is the current mainstream for high-density AI.
Immersion cooling. Boards are submerged in dielectric fluid (single-phase or two-phase). Heat transfer is excellent; acoustic noise plummets; and server fans disappear, lowering parasitic load. Trade-offs: different service model, fluid management, compatibility, and supply chain maturity. Immersion shines at >80–100 kW/rack-equivalent densities or where water is scarce but heat sinks are available.
Plant configurations. Central chilled-water plants with evaporative cooling dominate large campuses; air-cooled chillers avoid water but raise power consumption. Dry coolers support warm-water loops. Economizers exploit cool ambient conditions to bypass compressors. The plant’s coefficient of performance (COP) sets the recurring energy penalty; design choices can swing PUE (power usage effectiveness) by material amounts over the life of the site.
Power density reality. Many AI deployments target 60–100 kW per rack footprint with direct liquid cooling, pushing hall densities into the 2–5 MW per row range. Planning for power transients matters: large training jobs can step load by megawatts at job start or failure. Electrical and thermal flywheels (UPS ride-through, hydraulic buffer tanks) damp the shocks.
Water and WUE. Water usage effectiveness (WUE) has joined PUE as a key metric. Evaporative cooling is energy-frugal but water-intensive; air-cooled chillers and dry coolers reverse the trade. In arid regions or where water politics are sensitive, designs favor air-cooled systems, warm-water loops, or heat rejection to district networks.
Site selection criteria.
Grid capacity and queue. The decisive factor is the ability to interconnect at tens to hundreds of megawatts within a rational timeline. Regions with congested queues can add years.
Climate. Cooler, drier climates favor free cooling and raise plant efficiency; hot, humid climates impose year-round mechanical cooling.
Water availability and policy. Access, cost, rights, and optics around water use are as important as technical feasibility.
Fiber routes and diversity. Multiple long-haul carriers, diverse paths to internet exchanges, and low-latency routes to paired sites are required for training clusters and replication.
Seismic and flood risk. Geotechnical surveys, flood plains, and building codes affect design and insurance.
Permitting and incentives. Local approvals for air, water, noise, and fuel storage determine schedule; tax abatements and energy tariffs influence TCO.
Community and optics. AI facilities concentrate power draw; community acceptance, workforce availability, and heat-reuse opportunities matter more than operators occasionally admit.
Campus strategy. Hyperscalers pursue campuses with shared high-voltage infrastructure, central plants, and modular expansion blocks. For AI, co-locating compute, storage, and network aggregation minimizes east-west traffic across long fiber paths and avoids fabric chokepoints.
7.3 Grid Interconnection and Energy Mix
The grid is not a tap that turns on and off at your convenience. Interconnection is a multi-year engineering and regulatory process, and the energy you ultimately consume is a portfolio choice.
Interconnection basics. To connect a large load, an operator submits an interconnection request to the transmission or distribution operator (ISO/RTO or utility). The system then runs feasibility, system impact, and facility studies to determine what upgrades are required and who pays. Backlogs are common. Even when the substation exists, upstream constraints (thermal limits, voltage stability, contingency criteria) can trigger expensive upgrades. Queue position and study scope drive timelines.
Firm vs. non-firm service. Some utilities offer non-firm or interruptible tariffs where, in exchange for lower cost or faster interconnection, the site agrees to curtail under specified conditions. Combining non-firm service with on-site BESS and generation yields a practical bridge while waiting for firm capacity. This shifts some risk and complexity onto the operator.
Power quality and harmonics. AI loads are power-electronics heavy and can inject harmonics; plant design must specify filters and harmonic limits. Rapid load changes demand coordination with the utility to avoid flicker and voltage sags; ramp-rate limiters and job schedulers that throttle start events mitigate impacts.
Energy procurement options.
Standard tariff supply. Default for many colocation tenants. Cost-predictable but with little control over carbon intensity.
Wholesale market participation. Large operators sometimes buy directly in organized markets or via retail choice. This requires trading capability and risk management.
Power Purchase Agreements (PPAs). Long-term contracts with renewable generators fix price and volume and underpin new wind/solar builds. Traditional PPAs are financial (virtual) swaps; sleeved PPAs route through a utility. PPAs deliver additionality but may not match time of use.
Time-matching and 24/7 carbon-free energy. A step beyond annual matching: procure clean energy that matches facility load hour by hour in the same grid region. This is harder and pushes portfolios toward wind+solar+storage, geothermal, hydro, nuclear, or clean firm contracts.
Behind-the-meter generation. On-site solar rarely moves the needle for AI loads; on-site gas generation can provide resilience and capacity but raises emissions accounting and permitting complexity. In some jurisdictions, combined heat and power (CHP) with heat reuse into district networks is attractive.
Capacity markets and demand response. In some regions, facilities can earn revenue by reducing load during system stress or by providing reserves. Participating demands telemetry, test events, and operational flexibility (e.g., pausing non-critical jobs, discharging BESS, or running generators). For AI, coupling the cluster scheduler with demand-response signals is a design opportunity: shift low-priority training, precompute embeddings during off-peak hours, and reserve capacity for inference SLAs.
The physics of “green power.” Electricity on the grid is fungible; electrons are not tagged. What changes your footprint is (1) reducing consumption, (2) shifting load to cleaner hours, and (3) enabling new clean generation that would not otherwise be built, ideally in your grid region. Certificates (RECs/GOOs) document attributes; their integrity and temporal/spatial matching rules determine whether claims are meaningful.
7.4 Sustainability Considerations and Reporting
Sustainability for AI data centers is broader than “we buy renewables.” It spans energy, water, materials, waste heat, and the social license to operate, converted into auditable disclosures.
What to measure.
PUE (Power Usage Effectiveness). Total facility power divided by IT power. Good modern sites achieve 1.10–1.25 depending on climate and cooling. AI pushes toward the low end when warm-water loops eliminate compressor use.
WUE (Water Usage Effectiveness). Annual site water use per IT energy unit (L/kWh). Track process water, make-up water, and municipal intensity. Air-cooled designs can target near-zero WUE at a PUE penalty.
Carbon intensity. Report Scope 2 (purchased electricity) location-based and market-based emissions; Scope 1 generator fuel; and relevant Scope 3 (embodied emissions in equipment, construction, and upstream energy). For credibility, apply hourly accounting in target regions and disclose methodology.
Embodied carbon (EC). Servers, accelerators, racks, batteries, and building materials carry EC. Vendor disclosures are improving; operators can influence EC by extending refresh cycles, designing for reuse, and choosing low-EC materials.
Circularity. Asset refurbishment, component harvesting, and certified recycling programs reduce waste and EC. For immersion, include fluid lifecycle impacts.
Heat reuse. Where district heating exists, warm-water loops can export low-grade heat. Economics depend on distance, temperature, and local policy; benefits include community goodwill and improved overall energy utilization.
Reducing impact in practice.
Right-size models and precision. Efficiency is sustainability. Quantization, distillation, mixture-of-experts, and retrieval grounding reduce compute per unit of value.
Schedule intelligently. Run non-urgent jobs when grid carbon intensity is low; align to renewable profiles; pre-cool thermal buffers before peaks; export reserves during stress.
Optimize the plant. Raise supply temperature setpoints; maintain coils and filters; fix valve authority issues; tune pump curves; eliminate chilled-water hunting. Subtle control issues squander megawatts.
Water strategy. In water-constrained regions, avoid evaporative systems; in temperate zones, design for seasonal switchover to dry operation. Meter by hall and reconcile.
Materials choices. Low-carbon concrete and steel, modular construction, and reuse of brownfield sites reduce Scope 3. For batteries, prefer chemistries with strong recycling value chains.
Reporting frameworks. Operators commonly report under the Greenhouse Gas Protocol (Scopes 1–3), disclose via CDP, set targets under SBTi, and offer TCFD-style (or equivalent) risk narratives. Some jurisdictions require audited climate disclosures; ensure data systems can trace from utility bills and metering to aggregated metrics. For customers, customer-allocable reporting (per cage/pod) is emerging: tenants want their attributable PUE, WUE, and emissions.
Avoiding greenwash. Be explicit about boundaries and methods: whether PUE includes network rooms, whether WUE excludes sanitary water, whether renewables are time-matched, and whether certificates are from the same grid region. Disclose generator testing hours and fuels. If claiming “carbon-free energy,” specify geography and temporal granularity.
7.5 Edge vs. Cloud vs. On-Prem Tradeoffs
Where you run AI workloads is a strategic decision touching latency, privacy, governance, and cost. No single model dominates; most enterprises adopt a portfolio.
Cloud: elasticity and velocity. Cloud accelerators provide immediate access to the newest hardware, global footprint, and managed fabrics. For AI, advantages include burst capacity for experiments, turnkey AI-optimized instances, and proximity to other managed services (storage, data pipelines). Drawbacks are unit cost, potential scarcity for flagship accelerators, data egress fees, and platform dependence. Cloud shines for experimentation, seasonal surges, and multiregion inference. It is also the easiest place to implement model routing—using the cheapest adequate model per request.
On-prem/colo: control and unit economics. Owning or leasing space for your hardware lowers long-run cost per token if—and only if—you maintain high utilization and have stable demand. You gain control over data locality, network topology, and supply chain choices (e.g., specific accelerators, interconnects, and cooling). You also inherit responsibility for power, cooling, firmware, spares, and compliance. On-prem is appropriate when you have steady inference loads, data residency mandates, or integration requirements that benefit from co-location with proprietary systems. Colocation provides a middle ground: your gear in someone else’s resilient facility, with you managing the computers and them managing the building.
Edge: latency, privacy, and resilience. Edge means inference close to where data are generated or decisions act—stores, factories, vehicles, hospitals, or handsets. Benefits include sub-100 ms latency without backhaul, reduced bandwidth cost, and privacy (raw data never leaves the site). Constraints are tight power/thermal envelopes, intermittent connectivity, and operational challenges of managing thousands of sites. Edge is right for closed-loop control, vision QA on lines, AR/VR, and mobile copilots. It is not a good fit for training beyond lightweight on-device personalization.
Design patterns across the portfolio.
Central training, distributed inference. Train or fine-tune centrally; distribute distilled or quantized models to cloud regions, on-prem clusters, or devices. Use federated analytics or periodic uploads for monitoring and drift detection.
Retrieval-first architecture. Keep models smaller by grounding with retrieval from regional knowledge stores. Retrieval is easier to update and govern than model weights and reduces cross-region data transfer.
Model routing and specialization. Use small models for routine requests; escalate to larger models only when uncertainty or complexity is high. Specialize models by region or product line where data allows.
Privacy tiers. Route sensitive inference to on-prem or edge; use cloud for de-identified workloads or public-facing channels. Align with data processing agreements and regulatory obligations.
Observability everywhere. Instrument token throughput, latency, error rates, and safety events across environments; centralize metrics; and enable fast rollback of models and prompts.
Financial lenses.
CapEx vs. OpEx. Cloud shifts cost to operating expense; on-prem demands capital plus depreciation. Compare on a tokens-per-dollar and tokens-per-joule basis, not per-GPU hour alone. Include people, software, facilities, and grid interconnection in the calculus.
Utilization and commitment. On-prem only wins if you can keep racks busy; cloud only wins if you avoid paying peak on-demand prices for steady baseload. In cloud, negotiate commitments that match your profile; on-prem, stage procurement with ramp-up.
Risk and option value. Cloud buys optionality: try new accelerators without buying them. On-prem buys predictability: you have capacity when clouds are oversubscribed. A hybrid approach—own base, burst in cloud—often maximizes real options.
Compliance and sovereignty. Certain sectors (healthcare, defense, regulated finance) require data to remain in specific jurisdictions or facilities. Cloud sovereign regions and dedicated instances help but may lag in hardware releases. On-prem simplifies audits but increases your responsibilities. Edge can satisfy constraints by keeping PHI/PII on site while sending only anonymized features centrally.
People and process. Cloud concentrates talent in FinOps and platform engineering; on-prem requires facilities engineering, SRE, cluster scheduling, and firmware/driver competence. Edge adds fleet management and secure update capabilities. Plan for staffing and training; many underperforming deployments are people problems dressed as hardware problems.
