Most organizations encounter AI through claims and demos—slides, benchmark charts, case studies, short pilots, and polished videos. These signals are noisy. Benchmarks can be saturated or cherry‑picked; case studies often hide confounders; demos follow golden paths; and pilots can be designed to succeed in ways that do not survive production. This chapter offers a disciplined way to read those signals. It explains how to interpret benchmarks and case studies; catalogs common pitfalls in vendor proofs and pilots; lays out how to design a meaningful proof of concept (POC) with business‑relevant KPIs; distinguishes substantive safety practice from “ethics washing”; and provides narrative due‑diligence guidance you can apply before committing time or money.
The theme is practical skepticism: trust, but verify—and design the verification yourself. Good vendors will welcome this. Weak vendors will try to change the subject.
19.1 Reading Benchmarks and Case Studies Carefully
Benchmarks and case studies are useful, provided you know what they measure—and what they do not.
Benchmarks: what numbers really say
1) Is the task your task?
Public benchmarks capture narrow skills (multiple‑choice Q&A, math word problems, coding puzzles). Your work likely involves retrieval from specific corpora, schema‑conformant outputs, tool use, and human oversight. A model’s score on a public benchmark is, at best, an upper bound on competence at parts of your workflow.
2) What was in scope?
Separately track capability dimensions:
- Knowledge vs. reasoning. Did the task require recall, or multi‑step reasoning with verifiable steps?
- Closed‑book vs. grounded. Did the model answer “from memory,” or did it use retrieval and cite sources?
- Single‑shot vs. tool‑using. Did the system call calculators, databases, or code interpreters?
- One‑turn vs. multi‑turn. Was long‑horizon planning or correction involved?
A score that conflates these hides where the strength lies (or does not).
3) Contamination and overfitting.
Benchmarks leaked into training corpora produce inflated scores. Ask whether the evaluation used fresh, held‑out items; whether paraphrases were filtered; and whether the vendor can run your private held‑out test set under NDA.
4) Aggregation, variance, and slices.
Macro‑averages hide tails. Request disaggregated results: by intent, difficulty, language, and format. Look for confidence intervals and p95/p99 behavior. A system that is excellent on average but fails unpredictably is hard to operate.
5) Metric choice and trade‑offs.
Accuracy alone can mislead. For extraction, precision and recall matter; for ranking, NDCG or MRR; for generation, faithfulness and citation entailment; for safety, false negatives matter more than false positives. For interactive systems, latency and cost are co‑equal metrics. Ask for calibration: does confidence correlate with correctness?
6) Units and budgets.
“Human‑level” under unlimited tokens and seconds is not “human‑level” at your latency and cost budget. Insist on budgeted evaluations: fixed max tokens, max latency, and tool limits comparable to your production envelope.
7) Plateau and saturation.
Some public benchmarks are saturated. High scores there tell you little. Prefer task families that can be kept fresh, especially those built from your domain.
8) Reproducibility.
Can you run the evaluation yourself with the vendor’s instructions and get similar results? If not, treat the claim as a marketing asset, not evidence.
Case studies: separating story from signal
Case studies are anecdotes with numbers. Read them like a forensic accountant.
Who wrote it and who benefits? If the vendor authored it, seek independent corroboration from the customer. If a consultancy wrote it, check incentives (were they paid?).
Baseline clarity. What was the pre‑AI process? What did the model replace or assist? Did any other changes occur (reorg, new incentives, content refresh) that could explain the outcome?
Outcome vs. proxy. “60% improvement in deflection” is meaningful only if customer satisfaction and re‑open rates did not worsen. Prefer bundle metrics: outcome + guardrails.
Population and sample. Was the case study one team, one product line, or the entire operation? Were the examples cherry‑picked? Ask for the denominator: how many cases, what period, and what exclusions.
Measurement discipline. Was there a control group? A pre/post with seasonality adjustments? Blind quality review? If not, treat effect sizes as upper bounds.
Operator load and safety. Did human review time go up or down? Were there incidents? How were PII and permissions handled?
Durability. Is the effect still present six months later? Gains can decay if content goes stale or if the organization reverts to old habits.
Transferability. Does the case study rely on idiosyncratic assets (a perfectly curated knowledge base; a bespoke tool only they have) that you lack? If so, discount.
19.2 Common Pitfalls in Vendor Proofs and Pilots
Demos and pilots can be engineered to look good while hiding fragility. Watch for these patterns.
1) Golden path demos.
The vendor shows a polished script where every tool returns clean results and the user intent is obvious. Real life includes typos, ambiguous questions, conflicting sources, and permission boundaries. Ask to run your own prompts and to see the system behave under messy conditions.
2) Prompt peeking and per‑example tuning.
Some demos secretly use per‑case prompt edits or hand‑picked few‑shot examples tuned to the test set. Inquire how prompts were chosen and whether the same prompt works across the entire intent distribution. Require a frozen prompt evaluated on a held‑out set.
3) Hidden human‑in‑the‑loop.
In pilots, vendors sometimes use humans to correct outputs in real time without disclosure. Clarify who can intervene and how often. Log edit rates and edit distance; disallow silent edits in evaluation unless they are part of your intended operating model.
4) Training on the test.
Beware of pilots where the vendor fine‑tunes on your labeled data and then reports performance on the same or near‑duplicate items. Enforce strict hold‑outs and de‑duplication checks using shingling or embeddings.
5) Retrieval illusions.
A common trick is pre‑loading a hand‑curated index with clean, de‑duplicated content that does not reflect your real corpus (stale pages, duplicative wikis, permissions). Demand pilots against your actual systems of record, with ACLs enforced, and check recall@k and citation entailment.
6) Cost and latency sleight of hand.
Demos are run with generous tokens, batch sizes, warm caches, and powerful hardware. Ask for per‑request token counts (prompt and generation), p95/p99 latency, and cost per outcome under realistic concurrency. Prohibit pre‑computation that would not exist in production.
7) Safety theater in pilots.
You see a content policy slide and one or two blocked prompts. Real safety requires input/output filters, refusal patterns, PII detection, and red‑team batteries. Include adversarial prompts and indirect prompt injection (malicious instructions embedded in retrieved content) in your pilot.
8) Tool misuse risk unexamined.
Agents may call tools without idempotency or preconditions. In a demo that is harmless; in production it can create duplicated orders or data corruption. Require schema validation, idempotency keys, allow‑lists, and least privilege in the tool catalog.
9) Multi‑tenant and permission leakage.
In cross‑tenant pilots, cached answers or loose filters can leak data. Test with look‑alike prompts from two tenants and confirm that results differ according to permissions.
10) Moving goalposts during the pilot.
Halfway through, the vendor changes models, prompts, or datasets without resetting baselines. Lock artifacts (model version, prompts, index version) for the evaluation window, or at least log changes and rerun baselines.
11) Overclaiming “human‑level.”
Anecdotes about matching “expert” performance often mask selection bias. Demand blinded review by your experts on representative samples.
12) Scoping games.
A vendor narrows the task mid‑pilot to boost performance (“We only handle Tier‑1 tickets”), quietly abandoning hard cases. Make scope a contracted artifact with penalties if changed unilaterally.
13) Security and privacy hand‑waving.
“Everything is encrypted” is not a security program. Ask where data live, whether your inputs or outputs are used for training, what sub‑processors exist, how long logs are retained, and whether deletion truly deletes embeddings and caches.
14) Benchmark transference.
A vendor cites a leaderboard they lead, then asserts superiority for your task. Treat domain transfer as an empirical question; most advantages compress when retrieval, tools, and latency budgets enter.
19.3 Designing Meaningful POCs and KPIs
A good POC tests your task with your constraints and produces evidence that maps to a go/no‑go decision. Design it yourself with vendor input, not the other way around.
Start with a task contract
Write a one‑page task contract:
- Users and triggers. Who initiates the task and why?
- Inputs. What documents, fields, or signals are provided?
- Output and format. Free text? JSON schema? Citations required?
- Constraints. Latency budget, cost budget, privacy/ACL boundaries, languages.
- Quality bar. Acceptance criteria and how they will be measured.
- Safety posture. What must be refused or escalated.
- Failure modes. Known errors, their severity, and acceptable rates.
- Ownership. Who approves deployment if the POC succeeds.
Make this the north star. Without it, a POC drifts into a show.
Build the right datasets
Representative samples. Pull a stratified sample of real cases across intent categories, difficulty, channels, and languages. Include edge cases and known hard examples. For extractions, include noisy OCR and variant formats.
Ground truth with references. For Q&A or summarization, have domain experts write answers with citations. For extraction, build gold JSON with field definitions. Use a second rater to measure inter‑rater agreement; if your humans disagree frequently, the task is under‑specified and the POC will be noisy.
Hold‑outs and leakage control. Split into dev, tuning, and final hold‑out sets. De‑duplicate across splits. If the vendor asks to fine‑tune, they use only dev/tuning; the hold‑out remains unseen until the end.
Stage the evaluation
1) Offline stage (fast iterate).
Run on the dev set to tune prompts, chunking, retrieval filters, and tool schemas. Track:
- Structural validity (schema adherence, required fields).
- Faithfulness (citation presence and entailment).
- Accuracy against gold labels.
- Latency and token counts (prompt vs. generation).
- Safety filter false negatives/positives.
2) Shadow stage (no user impact).
Deploy in a way that observes real traffic but does not act. Compare to human outcomes. Measure abstention rates, retrieval recall@k, tool success, and disagreements with humans categorized by severity.
3) Assist stage (human in control).
Show model outputs to operators who can accept/edit/reject. Measure adoption, edit distance, time saved, and quality via blind review. Capture edits to form a hard‑case set for future tuning.
4) Controlled experiment (if safe).
Run an A/B test: treatment = assist; control = status quo. Pre‑register metrics; check sample‑ratio mismatch; run long enough for steady state. For low volumes, use matched pairs or stepped‑wedge designs.
Choose KPIs that map to business value—with guardrails
Outcome metrics (primary).
- Resolution rate (support), time‑to‑merge (code), close time (finance), first‑pass yield (extraction), conversion (sales), recall@k (retrieval quality when that is the product), etc.
Quality guardrails.
- Accuracy or precision/recall vs. gold sets.
- Faithfulness: % of claims supported by citations; entailment pass rate.
- Schema validity: % outputs that parse and pass validators.
- Abstention: rate of “I don’t know” under uncertainty; this is good when justified.
Safety guardrails.
- PII leakage incidents, policy violation rate, tool misuse attempts blocked, jailbreak success rate (using a standard test battery), and moderation false‑negative rate.
Operational guardrails.
- Latency (p50/p95/p99), availability, error rates (by error type), tool latency, cache hit rate, retrieval index freshness.
Economics.
- Tokens per outcome (not just per request), cost per resolved case, cost per correct extract, inference cost share as % of revenue or savings.
Adoption and human factors.
- Adoption (% of eligible tasks using assist), edit distance, operator satisfaction, re‑open rate, escalation quality.
Tie go/no‑go to thresholds across the bundle: “Proceed if resolution improves ≥8% with no increase in re‑opens, p95 latency < 2.5 s, cost per resolution falls ≥20%, and safety incidents = 0 on the test suite.”
Instrument for learning, not just pass/fail
- Trace logging. Capture prompts, retrieval snippets, tool calls, and outputs for sampled cases; redact PII appropriately.
- Error taxonomy. Label failures: hallucination, retrieval miss, tool failure, schema violation, safety miss, latency breach.
- Hard‑case set. Curate the hardest 200 examples for regression testing.
- Drift checks. Track input distribution changes during the POC; ensure performance on new variants holds.
Guard against POC drift
Freeze artifacts (model versions, prompts, index version) per phase. If a change is necessary, re‑baseline or annotate results accordingly. Record time on task (engineering hours, SME hours); a pilot that “works” only with heavy babysitting does not scale.
19.4 Ethics Washing and Safety Theater
As scrutiny rises, so does the temptation to look responsible without doing the work. Learn to distinguish substantial safety practice from theater.
Common forms of ethics washing
1) Policy slide without mechanisms.
A vendor shows a content policy but cannot demonstrate input/output filters, refusal patterns, or escalation. Ask to see the filters trigger on adversarial prompts; ask for false‑negative/positive rates.
2) “Model card” with platitudes.
A glossy PDF cites generic risks but lacks training data summaries, known limitations, slice metrics, update history, or intended use constraints. A real model/system card includes these and links to evaluation artifacts.
3) One‑time red‑team stunt.
A vendor touts a “red‑team day” with a few screenshots. Substantive programs run regular batteries, record coverage, fix findings, and integrate them into CI tests.
4) Third‑party logo laundering.
They display a security or privacy certification that says little about AI‑specific risks (training data rights, retrieval permissioning, model evaluation). Certifications are valuable; the scope matters. Ask for reports and confirm that AI controls were in scope.
5) Safety as UI copy.
A “this may be inaccurate” banner is not safety. Real safety includes grounding (citations/entailment), schema constraints, abstention, and human‑in‑the‑loop where needed.
6) Ethical principles without accountability.
Principles are easy to publish. Evidence is ownership, budgets, SLAs, kill switches, and post‑incident reports.
7) “Fairness” without slices.
Claims of fairness must include which slices, metrics used, disparity thresholds, and mitigations. Beware fairness claims based on overall accuracy alone.
8) Privacy by press release.
“Your data is safe” means little without training‑use clauses, retention limits, deletion proofs, sub‑processor lists, and regional controls.
What good looks like
- Defense in depth. Input screening, output moderation, retrieval entailment checks, schema validation, tool allow‑lists, and least privilege—demonstrated live.
- Measurable safety. Test suites for disallowed content, jailbreaks, PII leakage, and tool misuse, with pass/fail thresholds and trend lines.
- Incident readiness. Named on‑call, kill switches, rollback plans, and post‑incident templates.
- Governance integration. Safety woven into change management: no promotion of a new model/prompt/index without passing safety gates.
- Transparency with substance. Model/system cards tied to artifacts; data provenance and rights documented; evaluation summaries with slices.
- Human oversight design. Clear escalation paths, reviewer UX, and training; refusal/abstention behavior defined in prompts and verified in tests.
19.5 Due Diligence Checklists (Narrative Guidance)
What follows is a narrative set of questions and checks to guide diligence. Use it as a script; adapt to your sector and risk posture.
A. Capability and fit
- Problem fit. Can the vendor articulate your task contract in their own words? Do they propose retrieval, schemas, and tools that match your systems, or do they revert to generic chat demos?
- Bounded claims. Do they state what the system will not do? Are abstention and escalation part of the design?
- Evidence on your data. Have they run on representative samples from your corpus with ACLs enforced? Can they show citation entailment or other grounding checks?
- Tool competence. For agentic use cases, do they define schemas, preconditions, idempotency, timeouts, and error classes? Can they demonstrate recovery from tool failures?
B. Evaluation and measurement
- Datasets and splits. Who created the gold sets? How was inter‑rater agreement ensured? How were hold‑outs protected against leakage? Are hard‑case sets maintained?
- Metrics. Which outcome and guardrail metrics are tracked? Are there confidence intervals and slice analyses? Are latency and tokens per outcome measured?
- Reproducibility. Can you re‑run their evaluation? Are artifacts locked (model, prompt, index) and versions recorded?
- Drift and robustness. How do they detect and respond to input drift (new intents, formats, languages)? Do they have golden canaries (fixed daily tests)?
C. Safety and governance
- Policy. Is there a written content and safety policy with examples? Are refusal templates and abstention rules encoded in prompts?
- Filters and tests. Which classifiers are used? What are their false‑negative rates? What is the jailbreak test pass rate? How are prompt injections via retrieved content handled?
- Human oversight. What requires review? Is the reviewer UI designed for citations, diffs, and quick approvals? Is feedback captured as training data (with consent)?
- Incident response. Who is on call? What are the kill switches? How long does rollback take in practice? Are there post‑incident templates and recurring reviews?
D. Data rights and privacy
- Training rights. Will your inputs/outputs/embeddings be used to train vendor models? If yes, under what terms? Prefer no training without explicit permission.
- Provenance. What are the sources and licenses of the vendor’s own training data? How do they handle takedowns and opt‑outs?
- Retention and deletion. How long are logs and embeddings stored? Can they prove deletion (including caches and backups) on request?
- Residency and access. Where is data processed and stored? Who can access it? Are sub‑processors and locations disclosed? Can the system run in your VPC or region?
E. Architecture and portability
- Model‑agnostic design. Can the solution route to multiple models? Are prompts portable? Can you bring your own model?
- Artifact export. Can you export prompts, indices, embeddings, and logs in open formats? Is there termination assistance in the contract?
- Performance engineering. Are batching, KV caching, quantization, and speculative decoding used appropriately? Are latency and cost predictable at your expected concurrency?
- Observability. What traces and metrics are exposed? Can they integrate with your telemetry? Are there dashboards for cost, safety, and drift?
F. Security
- Identity and access. How are users and services authenticated? Are tokens short‑lived? Is least privilege enforced for tools and data?
- Secrets. How are keys and credentials stored? Are there egress controls and sandboxing for code execution tools?
- Supply chain. How do they vet dependencies? Are model weights and datasets signed? Is there a software bill of materials?
- Pen testing and audits. Are regular tests performed? Can you review reports? Are AI‑specific risks in scope?
G. Operations and SLOs
- SLOs. What are latency/availability targets? Are there remedies beyond credits for chronic shortfalls?
- Support. Do you get named technical contacts? What is the support SLA? Is there 24/7 coverage for incidents?
- Change management. How are model/prompt/index updates rolled out? Are there canaries and rollbacks? Will you get advance notice?
- Capacity and energy. Can they scale with your growth? Do they disclose capacity constraints or rate limits?
H. Cost and commercials
- Pricing clarity. Is pricing per token, per seat, per outcome, or a mix? What features are included (embeddings, vector search, fine‑tuning)? Any egress fees?
- Forecasting. Do they provide tools to estimate tokens per outcome under your workload? Are there soft/hard quotas and behavior on overage?
- Commitments and flexibility. Are there minimums? Volume discounts? Can you step down if your usage profile changes?
- Value realization. How will success be measured in dollars saved or revenue gained? Will the vendor align payment milestones to POC exit criteria?
I. Vendor viability and roadmap
- Team and finances. Is the vendor likely to be around in two years? Are they dependent on a single upstream model provider?
- Roadmap. What is the release cadence? How are breaking changes handled? Are there plans for on‑prem/VPC support, new regions, or new safety features?
- References. Can they provide customers in your sector who will discuss failures as well as successes?
J. Contract hygiene
- Data processing addendum. Lock down training rights, retention, deletion, and sub‑processor transparency.
- IP and indemnities. Clarify ownership of outputs and tuned weights; seek indemnity for IP claims arising from the vendor’s training data/model behavior (with reasonable caps).
- Termination assistance. Ensure a period of cooperation and artifact export upon exit.
- Audit rights. Include right to audit or to receive independent attestation focused on AI controls.
- Region and residency. Name permitted regions; restrict transfers without consent.
