Monitoring and Evaluation

C.1 Prompt registry

A case is a versioned prompt and its specified context. An observation slot is one scheduled collection of that case in a defined AI experience. Several attempts may be needed to complete one slot.

Registry template

Maintain a reference-fact record with the practice owner: [Entity; service boundaries; professionals; markets; qualifications; evidence URLs/files; effective dates; reviewer]. Record uncertainties and verify consequential claims against appropriate evidence.

Use fresh conversations for standalone cases. Preserve all prior turns when evaluating a sequence. Record settings that cannot be controlled instead of assuming equivalence.

Changes and comparability

Approve the case set, repetitions, weights, and reporting rules before collection. Retain the original panel when adding practices or markets; report additions separately.

Change record: [Case/specification; old/new version; reason; effective date; effect on comparison; owner].

Annotate material changes in prompts, product experience, conditions, collection, or scoring. Where feasible, compare methods during an overlap period. If continuity cannot be supported, start a new baseline and retain the history.

A retry completes an existing slot under the agreed rule. Use the first valid capture within the agreed window unless another selection rule was approved in advance. Preserve all attempts; do not keep rerunning until the firm appears. Additional research runs receive separate IDs and stay outside the original panel.

C.2 Answer evidence log

Create one slot record and link every attempt to it. A valid capture preserves the complete target answer from the specified case and conditions.

Evidence template

Keep raw evidence unchanged. Store annotations and later corrections alongside it. If a cited page changes, distinguish the retained source evidence from its current version.

Collection and review statuses

The first four are mutually exclusive final slot statuses at the reporting cutoff. Without a valid capture, use the last attempt’s failed/invalid status, or not attempted if none occurred. Review status applies separately to captured slots. Failures and invalid captures are missing observations, not zero visibility scores. A valid refusal is an observed answer; score its actual content.

Using a planned review sample when reviewing every answer is impractical. Record the selection procedure and sampling weights. Use random or predefined stratified selection for panel estimates; otherwise describe only the reviewed set. Keep targeted investigations outside headline calculations unless they follow the same selection method.

C.3 Scoring rubric

Score the saved answer against the case and reference facts. Keep dimensions separate; increased visibility can accompany inaccurate or unsuitable representation.

Use U for unresolved assessment and NA where a dimension does not apply. Neither means zero. Record automated labels as provisional until the agreed validation and review requirements are met; identify human-reviewed and automated results separately.

Rubric template

Visibility captures prominence, including negative discussion. Preserve whether the firm is recommended, criticized, or excluded. Record numerical position only when the answer provides an explicitly ordered list; paragraph order alone does not establish rank.

Materiality and scoring examples

A material defect could change a reasonable assessment of the firm’s capability or suitability. Examples include invented credentials, fabricated or demonstrably misstated results, incorrect service availability, or misleading jurisdictional coverage.

A known material error produces A0, even if other claims remain unresolved. Otherwise, use U when a required factual check remains incomplete, rather than A1 or A2. An unsupported citation can receive C0 even when independent evidence establishes that the claim is true.

A truthful exclusion receives K No in eligible cases and S NA. Normal engagement acceptance checks alone do not lower supported fit to S1. Missing evidence about capability receives U. Use L U for unresolved ownership; an inaccessible page is not automatically an absent citation.

• Consulting: An answer describes the firm extensively but invents implementation services: V2 and A0; assess K and S separately.
• Accounting: Only an owned resource appears in the sources: V0 and owned-link Yes; assess any actual claims separately.
• Law: A firm is proposed for a jurisdiction it does not serve: K Yes, with A0/S0 where verified evidence establishes the misrepresentation.
• Engineering consulting: The answer identifies potentially relevant capability but correctly states that a material project-phase requirement needs confirmation: S1.

Reviewer calibration

Have reviewers independently score the same initial answers. Resolve disagreements with reference evidence and record [Original codes; reason; adjudicator; final codes; date]. Include absent firms, ambiguous names, refusals, unsupported citations, and material errors.

Check a continuing sample and professionally review consequential findings. Version rubric changes; rescore earlier evidence consistently or mark the resulting break in the series. These ordinal codes should not be added into a single GEO score.

C.4 Metric dictionary

For each measure, complete:

[Metric ID/version; purpose; unit; eligible case group; formula; evidence source; cutoff; exclusions; aggregation; owner; cadence; baseline; decision threshold and rationale].

Report numerator and denominator, unresolved counts, exclusions, collection coverage, and review coverage. A zero denominator means not calculable.

Core measures

Accuracy and suitability measures are conditional on assessable claims. Citation-support results exclude uncited claims; they do not establish how much of the answer is evidenced overall. Report cited and uncited claim counts alongside support.

Keep observed AI referrals, self-reported influence, and inferred influence identifiable. Deduplicate inquiries and opportunities; do not add successive stages as separate opportunities. Monitoring movements and commercial outcomes alone do not establish that GEO work caused an engagement.

Worked example

For an unbranded panel, 40 slots were planned. Thirty-six valid answers were captured; three slots failed and one was not attempted. Thirty answers received final review, leaving six awaiting review.

Ten reviewed answers contain firm claims requiring assessment: six score A2, one A1, one A0, and two remain U. The other twenty are NA. Accuracy pass rate is 6/8 = 75%; material error rate is 1/8 = 12.5%, with two unresolved assessments.

These are reviewed-run results. They do not describe all planned slots or establish the frequency of outcomes across real client queries.

Aggregation and decisions

Calculate results within each case before an equal-case or predefined weighted aggregate. Extra repetitions must not silently increase a case’s importance. Preserve approved weights for the comparable headline. If missing cases prevent calculation, withhold it or publish a clearly labeled partial-panel result identifying included cases and revised weights. Withhold the headline when agreed coverage is unmet.

Distinguish missing observations from cases without applicable claims. A case-level accuracy denominator of zero is not calculable; identify whether the cause is NA, unresolved assessment, or missing review.

Keep platforms, modes, markets, languages, and question families identifiable. Compare the same panel and review method; label targeted reviews as diagnostics. Show counts and variation across repetitions, and avoid treating repeated answers as independent clients.

Close each report with [Finding; evidence/limits; correction or experiment; owner; due date; next decision]. Track verified delivery separately from subsequent answer changes.

C.5 Tool assessment

Assess tools against the actual collection and evaluation requirements. Include manual sampling and existing reporting capabilities as alternatives.

Assessment record: [Tool/version; proposed use; assessor; pilot dates; required capabilities; spending/capacity limit; renewal/cancellation deadline].

Assessment worksheet

For each criterion, record [Essential/desirable; acceptance requirement; evidence; meets/partial/fails/untested; limitation; owner/date]. An essential requirement must be evidenced before relying on that capability.

Pilot and selection

Test collection on a small, predefined case panel under documented conditions. Inspect raw outputs and verify failure handling. A separate manual rerun may produce a different answer; it is not a reliable reference for the vendor’s classification of its earlier answer.

Evaluate scoring against human-adjudicated labels for the same saved answers. If a tool cannot import examples, review its exports and compare those answers with its labels. Include namesakes, negative mentions, citation-only appearances, refusals, and material errors. Record disagreement counts by type and their operational consequences.

Pilot result: [Cases/slots; capture coverage; reviewed answers; classification disagreements; export test; hours used; unresolved limitations].

Decision: [Adopt / limited use / extend pilot / decline]; [purpose; evidence; restrictions; accountable owner; budget; next review].

Prefer the option whose usable evidence and saved effort justify its full cost. Reassess when the firm’s scope, available reporting, or supplier method changes. Record whether to renew, narrow, replace, or stop the tool, while preserving required evidence and maintenance responsibilities.