Run GEO Experiments and Learn What Works

17.1 Turn Proposed Changes Into Specific Hypotheses and Observable Outcomes

Start with a decision: What would we do differently if this test produced convincing evidence? A useful answer might be, “Extend a new approach to describing engagements across six more practice areas.” If neither a positive nor a negative result would change a decision, routine monitoring may be sufficient.

Write the hypothesis before publishing: If we make this defined change, for these services and questions, we expect this observable outcome to improve relative to this comparison, during this period, because of this proposed mechanism, without this unacceptable consequence.

A consulting example

A consulting firm’s industry pages describe broad expertise but provide little detail about how engagements work. The proposed change adds a concise, verified explanation of the problem addressed, the firm’s role, typical deliverables, and engagement boundaries. The hypothesis is that this information will help AI services describe the firm as a plausible option for relevant assignments.

The primary outcome could be the frequency of accurate, qualified inclusion in answers to a fixed set of unbranded selection questions. A qualifying answer must name the firm as a plausible option, correctly explain a relevant capability, and avoid a material misstatement about that capability. Supporting measures could include citations to the changed pages and correct descriptions of deliverables.

The guardrail is equally specific: the answers must not increasingly imply services, geographic coverage, guaranteed outcomes, or credentials the firm cannot substantiate. Record inappropriate recommendations on a separate set of questions outside the firm’s scope.

Choose the question the test can answer

Use public, professionally approved information throughout the test. An experiment is not a reason to publish an unverified claim, disclose a confidential matter, or give bots a misleading version of the firm’s capabilities. Collect observations through permitted access methods.

These are separate tests. Anthropic, for example, documents that Claude can retrieve material from a supplied URL. A correct answer after you provide the page is useful diagnostic evidence, but it does not show that an ordinary prospective client’s question would have led to that page. 

Keep the intervention sufficiently bounded to interpret. If you simultaneously rewrite pages, add evidence, change navigation, and secure media coverage, you are testing the combined program. That can be a legitimate decision, but the result cannot identify which component contributed.

Published research can suggest hypotheses. The original GEO study evaluated changes to source content in a defined retrieval-and-generation setup; its Perplexity experiment supplied source text through file uploads. The authors also said they had not evaluated effects on search rankings. Those design choices limit what the findings establish about natural discovery after publishing a change on a live website. 

17.2 Choose Practices, Expert Pages, Topics, or Markets as Test Units and Comparisons

The test unit is the thing that receives the intervention: an expert biography, a service page, a group of related pages, or a practice’s public information. A prompt is a way of observing the outcome.

Keep three counts separate: the units receiving the change, the distinct questions used to measure them, and the repeated observations collected over time. Six changed page groups and 600 recorded answers still represent six changed groups. Repeated measurements cannot substitute for independent replication of the intervention; this is the general experimental-design problem known as pseudoreplication.

Build the strongest feasible comparison

Where the firm has enough suitable units, group them by relevant characteristics: service type, audience, market, existing content, and baseline performance. Randomly choose which units within each group receive the change first. Observe changed and comparison units over the same calendar periods. This applies the established principles of blocking and randomization to a publishing rollout.

A comparison should help estimate what might have happened without the intervention. An established international-tax practice is a weak comparison for a newly launched bookkeeping service. Similar baseline scores alone do not make two practices comparable.

If random assignment is impractical, select the comparison before seeing post-change results and explain the choice. Call it a matched comparison. For a small firm with one relevant practice, use several baseline and follow-up waves and describe the result as a before-and-after case. It can inform a decision, with limited causal confidence.

Account for overlap

Ask whether information changed in one unit could also affect another unit’s answers. Updating a shared expert biography or the firm’s central service description could influence questions assigned to several practices. In experimental methods, one unit’s treatment affecting another unit’s outcome is called interference. 

As a practical precaution, group closely related pages and questions together. A site-wide change leaves no unchanged within-site comparison. Distinct URLs, offices, or practice names do not by themselves establish independence. Record suspected overlap; do not discard comparison results because they improved too.

A staged rollout can preserve a comparison among units that have not yet received the change. Once a comparison unit is updated, it ceases to be an untreated comparison for subsequent observations. Record both calendar date and time since publication.

An accounting example

An accounting firm wants to improve explanations of its advisory work. It tests adding a clearly attributed methodology summary and links to current public technical sources on selected advisory topics. Comparable topics retain their existing, accurate explanations during the first stage. The primary outcome is the correct description of the advisory service and its limits; citation frequency is secondary.

Before choosing the topics, the practice leader checks whether a regulatory announcement, a reporting deadline, or sharply different client needs would make the comparison misleading. A reviewer also checks that the new material does not imply an assurance conclusion or a preferred accounting treatment.

Do not delay a necessary factual correction, credential update, or confidentiality repair to preserve a comparison. Make the correction, record it, and change the experimental claim if necessary.

17.3 Set Baselines, Success Criteria, Observation Windows, and Decision Thresholds

Freeze an experiment version of the monitoring program before launch. Keep the prompt wording, scoring rules, product settings, and planned observations together so another person could repeat the procedure.

Define the measurement conditions

For each observation, record the product and interface, mode, displayed model or version where available, date and time, language, geography, account or personalization conditions, conversation state, supplied files or URLs, and visible search activity. Mark unavailable information as unknown.

Keep product experiences separate. Google states that AI Overviews and AI Mode may use different models and techniques, produce different responses and links, and issue multiple related searches. An AI Overview may also fail to trigger. These are reasons to retain separate results for each experience. 

Ask a monitoring supplier whether it observes the consumer application, an API, or a custom search pipeline. Anthropic’s API, for example, exposes controls for search limits, domain restrictions, location, and search-tool behavior. A result from one configuration should not silently stand in for another product experience. 

Use fresh conversations for a test of first-contact discovery. If the hypothesis concerns a follow-up conversation, specify the preceding turns. Keep questions that name the firm separate from questions that do not. Include realistic wording variations selected before launch, and reserve some new questions for later replication.

Specify the numerator and denominator

For the consulting example, calculate qualified inclusion rate = qualifying observations ÷ valid completed checks in the predefined, relevant question set. This is a score for the test set, not an estimate of all prospective clients’ behavior.

A completed check in which an answer omits the firm is a non-inclusion. A valid search result without the AI feature being tested is also a non-inclusion for this visibility measure; record feature absence separately. Keep substantive refusals as observed outcomes. Technical failures and timeouts are missing observations: apply the predefined retry rule and report completion coverage.

Report raw counts alongside rates. Score wrong-fit recommendations in their own predefined question set. Do not remove unsuccessful questions, combine platforms differently, or change scoring standards after seeing results.

If topic groups have different numbers of observations, decide their weighting in advance. One approach is to calculate each group’s rate and average the group rates equally. Otherwise, a heavily monitored practice can dominate the result merely because it has more prompts.

Set the clocks and the stopping rule

Collect baseline observations across several planned dates so you can see existing variation. Choose a deployment date, a processing allowance where relevant, a follow-up schedule, and an evaluation date. Set a maximum duration and budget.

Google says crawling can take days to weeks and that a crawl request does not guarantee inclusion in results. This is platform-specific guidance, not a universal GEO testing interval. Google also distinguishes eligibility from any guarantee of indexing or serving. 

Record publication, observable processing, and answer measurement separately. Keep an evaluation from the original deployment date. If you also plan a later window based on a neutral processing milestone, define it beforehand and report both. Never restart the clock at the first favorable answer.

A pilot might use three weekly baseline waves, a predefined processing allowance, and four weekly follow-up waves. That is an illustrative operating schedule, not evidence that the sample is sufficient or that the intervention should work within those weeks.

Before launch, specify the smallest improvement worth the implementation and maintenance effort. For one pilot, management might require a ten-percentage-point improvement beyond the comparison group, a positive pattern across most tested topics, and no material deterioration in the guardrails before funding a second pilot. This is a business decision threshold, not a statistical guarantee.

Review during the experiment for inaccurate content, inappropriate recommendations, and broken collection. Those may justify an early correction or stop. A favorable interim score alone should not end a fixed-window test. If you need a formal rule for decisions made while results accumulate, have it designed before launch.

No single prompt count or test duration makes an experiment conclusive. The number of treatment units, baseline variation, overlap, outcome frequency, and size of the effect all matter. For a material investment decision, have an analyst assess the design and uncertainty before launch. Set a review workload the firm can actually sustain.

17.4 Record Changes Alongside News, Competitor Activity, Seasonality, and Platform Updates

Maintain one dated change log for the experiment. Preserve the exact published versions and approval record, not merely a note saying “content optimized.” If several people can edit the website, tell them which pages are being tested and ask them to log relevant changes.

Separate what happened from when you noticed it. Link to the announcement or retain the relevant public evidence. “Provider update announced on this date” is stronger documentation than “the model seemed different.” The absence of an announced update does not establish that the system stayed unchanged.

If an event affects one group more than the other, flag the comparison as compromised. For example, new trade-press coverage of the treated consulting practice could contribute to improved answers. Report the overlap rather than assigning the entire change to the revised page.

If the monitoring method changes, preserve the old series and identify the break. Run the old and new methods in parallel briefly when feasible. Do not stitch incompatible scores together as though nothing changed.

Keep departures from the plan visible, including missed checks, urgent corrections, additional publications, and incomplete deployment. A change log explains the limits of the finding; it should not become a retrospective collection of excuses for disappointing results.

17.5 Interpret Repeated Observations Without Confusing Variation or Correlation With Causation

Evaluate in order: implementation, measurement quality, observed outcomes, comparison, and uncertainty. This prevents a plausible-looking percentage from outrunning the evidence.

Check the evidence before calculating the effect

Confirm which units received the approved change and whether the relevant pages remain accessible. Retain units assigned to the intervention even when deployment was incomplete, with their status marked. Quietly dropping failed deployments can make the intervention appear more successful than the actual program.

Check a sample of the answer scoring against the full responses and cited pages. Have a professional reviewer assess technical accuracy and service fit. Where possible, conceal whether an answer came from before or after the change while it is scored. Resolve disagreements against a written rubric.

AI-assisted scoring can help organize the work, but a flattering tone is not evidence of correctness. Verify whether a citation supports the associated claim. NIST’s generative-AI guidance specifically recommends verifying sources and citations and cautions against extrapolating performance from narrow, anecdotal assessments. 

A worked example

Suppose a consulting firm compares six changed topic groups with six matched groups that remain unchanged. On one product experience, each group is measured using five fixed questions over four baseline waves and four follow-up waves. Each arm therefore has 120 observations in each period. All checks complete, and equal sampling gives each topic group equal weight.

The changed topics improved by 15 percentage points. The comparison topics improved by five. The difference between those changes is 15 − 5 = 10 percentage points.

This is a descriptive difference-in-changes calculation. Interpreting it as a causal effect requires assumptions, including that the groups would have followed comparable trends without the intervention and that the comparison was not affected by the change. Difference-in-differences research makes those assumptions explicit; subtraction alone does not establish them. 

Review the individual topic results and each observation wave. Did the pattern occur across several topics, or did one topic account for most of it? Did both groups move together after an external event? Similar pre-change trends can support the comparison’s plausibility, but cannot prove what would have happened afterward.

The 480 recorded observations do not turn this into 480 independently assigned interventions. Statistical analysis must reflect the assignment of the twelve topic groups and the repeated structure of the measurements. Ask for uncertainty estimates appropriate to that design rather than a significance test that treats every answer as an independent treatment unit. 

With no uncertainty calculation and only a small matched pilot, the management finding should remain bounded: “The observed pattern supports another test of the revised content on this product experience.” The ten-point difference does not establish a durable effect across every platform or practice.

Distinguish lack of evidence from evidence of no useful effect

If the results are too variable, the comparison is weak, or delivery is uncertain, call the test inconclusive. An unchanged score does not identify the reason. The content may not have been encountered, the change may have had little effect, or the measurement may be too limited to distinguish the possibilities.

Conversely, if a well-executed test with adequate precision rules out the improvement management considers worthwhile, that supports stopping further investment in that intervention for the tested setting.

Keep exploratory findings separate from the original success criterion. A positive result discovered among many prompts, metrics, or platforms deserves a new hypothesis and an independent test. Do not rewrite the original hypothesis to match the best-looking result.

17.6 Decide Whether to Expand, Revise, Reverse, or Stop an Intervention

End the experiment with a decision record, including the limits of the claim. “More data needed” should identify which uncertainty matters and what further observation could resolve it.

Stopping a GEO experiment does not require removing useful, accurate content. A clearer explanation may deserve to remain because it helps prospective clients, even when its incremental GEO effect is unproven. Record that editorial decision separately.

A law firm example

A law firm publishes a clearer explanation of the work its employment practice undertakes, the jurisdictions relevant to that work, and the limits of an initial inquiry. Because the explanation appears in shared firm information, the team cannot identify a credible unaffected internal comparison.

The firm therefore treats the evaluation as a before-and-after case. Its outcome measure asks whether answers accurately describe the practice’s scope. A separate question set checks whether the firm is being presented as suitable for matters outside that scope. The review also checks for unsupported promises of outcomes or immediate representation.

Suppose the post-change answers describe scope more accurately, while overall mentions remain flat. That may support retaining the explanation and testing the approach elsewhere. It does not establish a visibility gain. If mentions fall on unsuitable questions, review whether that represents more accurate matching before treating it as a loss.

Also check why a guardrail improved. If the firm disappears from every answer, the rate of wrong recommendations may fall to zero without any evidence that the service now understands the firm better. Read accuracy, appropriate inclusion, and omission together.

A reversal has limits too. Restoring website copy is an action the firm controls; immediate disappearance of earlier information from every AI answer is not an outcome it can promise. Preserve any necessary factual corrections and continue proportionate monitoring.

Maintain a concise learning record: what was tested, where, when, what happened, how strong the evidence was, and what decision followed. Preserve negative and inconclusive findings as carefully as positive ones. They prevent repeated spending on ideas the firm has already examined.

Practical Output: An Experiment Brief, Change Log, Evaluation Template, and Decision Record

Use the following four records together. Link to the existing prompt registry, firm-facts register, and answer evidence log instead of duplicating them.

1. Experiment brief

• Identity and decision: Experiment ID, owner, practice sponsor, professional reviewer, decision to be informed, and effort limit.
• Hypothesis and intervention: Exact change, proposed mechanism, target audience and questions, approved claims, affected URLs, and content version.
• Design: Units, eligibility, assignment method, comparison rationale, suspected overlap, and what the design cannot establish.
• Measures: Frozen product and collection configuration, or a link to that protocol; one primary outcome, numerator and denominator, weighting, secondary measures, accuracy and fit guardrails, scoring rubric, and missing-data rule.
• Schedule and decision rules: Baseline dates, deployment, processing allowance, follow-up dates, minimum useful improvement, evaluation date, extension conditions, and early-stop triggers.

2. Change log

Use one row per event: event date and time | date noticed | affected units | change or external event | evidence link or version | owner | possible consequence for interpretation | action taken. Enter events during the test.

3. Evaluation template

• Execution: Planned and actual deployment; processing evidence; deviations; completed versus planned observations.
• Results: Counts and rates before and after, by group and product experience; unit-level and wave-level results; guardrail findings.
• Interpretation: Comparison-adjusted description where appropriate; uncertainty; overlap; alternative explanations; exploratory findings kept separate.
• Conclusion: What the evidence supports, what it does not establish, and whether it is sufficient for the decision.

4. Decision record

Record expand, revise, reverse, or stop, the rationale, the responsible person, the next action, its due date, and any maintenance obligation. Distinguish the decision to retain useful content from a claim that it improved GEO.

For the consulting calculation above, if the topic-level results meet the predefined consistency requirement, the comparison remains credible, and professional review confirms that the guardrails were met, a defensible record might read: “Fund one further rollout with a fresh comparison and question set. The observed difference in changes was ten percentage points on the tested experience. Causal confidence remains limited. Retain the accurate engagement descriptions; do not claim a proven platform-wide uplift.”

The purpose of experimentation is to make the next decision better informed. A successful program learns which changes help users understand the firm, how reliably they help, and when the evidence is too weak to justify further investment.