Audit What AI Systems Say About Your Firm and Professionals

5.1 Run an Initial Audit Across the AI Experiences Relevant to Your Clients

Start with the program scope from Chapter 2, the question map and prompt library from Chapter 3, and the information-handling rules from Chapter 4. Assign an operator to run and preserve the tests, a practitioner to resolve factual questions, and a program owner to accept the findings. One person may fill more than one role.

Select the experiences because they matter to the chosen audience. Client interviews, referral conversations, and observed buying behavior can inform that decision. Where evidence is limited, record the selection as a working assumption. You do not need to test every available assistant before obtaining a useful first picture.

Name the experience precisely

Identify the product and interface, such as ChatGPT in a browser, Claude in its application, Gemini Apps, Google AI Mode, or an AI Overview within Google Search. Google documents that AI Mode and AI Overviews may use different models and techniques, producing different answers and links. It also says that AI Overviews often do not appear. These are separate experiences with separate observable outcomes. 

If a supplier collects responses through an API, label those observations as API results with the known configuration. They can be useful evidence about that setup. Do not automatically report them as the answers clients saw in the consumer application. The monitoring design in Chapter 6 will address how to select and validate collection methods.

Choose a manageable first set

Include questions covering discovery without naming the firm, evaluation of a named firm, evaluation of named professionals, and circumstances in which the firm is unsuitable. Include a comparison or follow-up conversation where it represents an important buying decision. Keep these question types identifiable in the records.

For a small pilot, consider 12 cases across three selected experiences, with two planned passes through the same cases. That creates 72 attempts to capture and inspect. This is a workload example, not a statistically representative sample or a required standard. Reduce the scope if you cannot review it properly, and document what you have left out.

Define a case as the exact prompt plus any specified preceding conversation or supplied material. Define a run as one attempt at that case in a recorded environment. Anthropic’s evaluation guidance similarly separates defined tasks from repeated trials and notes that outputs vary between runs. Applying those distinctions makes the audit easier to interpret. 

Gather a limited reference pack for the selected questions: current service descriptions, professional affiliations, relevant credentials, explicit service exclusions, and supporting records approved for the reviewer. You can verify additional claims as they arise. The pack helps assess accuracy; do not attach it to a discovery prompt unless supplied information is part of the case.

Preserve the first observation

Run the approved wording and save the response before asking corrective questions or supplying additional information. If a case calls for web search, record that instruction. Do not quietly add it only when the first answer is disappointing. A revised prompt or extra document creates a different condition.

Consulting example. An assistant correctly describes a firm’s procurement diagnostic when asked about the firm by name, but omits it from a question about consultants who help reduce supplier costs. Record both observations. The first demonstrates recognition under a named-firm question; the second records absence from one discovery answer.

Capture the baseline before planned content changes where practical. If a material problem requires immediate attention, preserve the available evidence, record the intervention, and proceed under the firm’s policy. Completing a tidy baseline should not become a reason to delay an important correction.

5.2 Record Prompts, Dates, Interfaces, Search Modes, Locations, and Relevant Context

Create one run record for each attempt, whether or not it produces an answer. Link it to its case. Use an identifier such as B01-E02-C04-R02 for baseline 1, experience 2, case 4, run 2. Use the same identifier across transcripts, source checks, and findings.

Record the conditions while running the test. Reconstructing them later from a screenshot is often impossible. The minimum record should contain:

• Case and input. Case ID and version; exact prompt; relevant preceding messages; names, URLs, files, or other information supplied.
• Date and environment. Date, time, time zone, provider, product, interface, account or workspace type, and model label if displayed. An undisclosed model remains undisclosed.
• Mode and activity. Selected mode or search setting; any explicit search instruction; visible search indicators, tool activity, and source displays.
• Geography and language. Location stated in the prompt, known tester or configured location, location permissions where relevant, language, and locale. Keep unknown signals marked unknown.
• Personalization and context. Whether this is a new conversation; relevant memory, custom instructions, project context, connected sources, and other known personalization settings.
• Observed outcome and evidence. Answer text, citations and destination URLs, presentation evidence, execution status, and the location of saved records.
• Review. Reviewer, review date, judgments, verification sources, and unresolved questions.

Distinguish a new conversation from an unpersonalized test

A founder’s account may contain extensive prior information about the firm. A fresh conversation does not establish that this information is unavailable. OpenAI documents that ChatGPT search may use saved memories when rewriting search queries and may use location information. Those features make account context relevant to the observation. 

Check the actual controls available on the audit account. As checked in October 2026, OpenAI’s Temporary Chat documentation distinguishes personalized and unpersonalized temporary conversations. Temporary status alone therefore does not establish an absence of personalization. Record the selected configuration rather than relying on an assumption about the feature’s name. 

For an initial public-discovery baseline, use a configuration that avoids introducing private firm information, as far as the available controls permit. Follow the permissions policy from Chapter 4. A deliberately unpersonalized test is still one documented condition; prospective clients may have their own relevant history and context.

Record what is visible without inventing what is hidden

A selected search setting and visible search activity are different observations. Claude’s documentation, for example, describes an indicator displayed during web search and the citations and links accompanying its response. Record such evidence when available. If the interface does not expose activity, write “search activity not established from the saved record,” rather than asserting that no retrieval occurred. 

Likewise, a city named in the prompt and the tester’s location are different fields. Preserve both when known. A visible model label records what the interface displayed; it is not a complete description of all the components that produced the answer.

Save copyable answer text and the citation destinations. Add screenshots or a saved page when placement, source cards, tables, or qualifiers matter. Keep the full relevant conversation for a follow-up case. Store evidence in the approved internal location; creating a public shared-chat link is unnecessary.

Log whether the run produced a substantive answer, a clarifying question, a decline, ordinary links without an AI narrative, or a technical failure. These outcomes should remain distinguishable. A failed request provides no answer to assess, while a clarifying question may be an appropriate response to an underspecified need.

5.3 Capture Firm and Expert Mentions, Recommendations, Citations, Omissions, and Competitors

First describe what the answer visibly did. Leave judgments about truth, suitability, and significance to the next step. Use agreed event definitions so two reviewers do not turn the same response into different visibility stories.

Firm mention. The firm is named or unambiguously identified in the answer. Record the wording and check the entity: a shared surname, abbreviation, network brand, or similarly named company may refer to someone else. Preserve the displayed name even when your record also uses a standardized firm name.

Professional mention. A named person appears. Record the person and the stated affiliation separately. Individual visibility should not automatically become current-firm visibility. The person may be described through work at an earlier employer, an independent practice, or an outdated profile.

Recommendation. The answer proposes the firm or professional as an option for the stated need. Save the recommended passage and its qualifications. An explicit recommendation, a conditional suggestion, an illustrative example, and a provider mentioned only to rule it out are different events.

Citation or source link. The answer displays a reference to material. Record the destination, the associated passage where identifiable, and whether the link points to the firm’s own site or elsewhere. A source card may identify a publisher whose name never appears in the answer’s narrative. Keep that source appearance distinct from a narrative firm mention.

Omission. Identify what is absent. A firm missing from a general shortlist, an unanswered question about a named professional’s qualifications, and a missing condition on a recommendation are different findings. A reasonable shortlist need not contain every suitable provider. Absence from it is an observation, not automatically an error.

Competitor presence. Record the other providers named, their apparent role, and what the answer says makes them relevant. A large multidisciplinary firm, a specialist boutique, and an independent practitioner may all appear in the same response. Do not assume they compete for identical work or that every description of them is correct.

Keep appearance separate from preference

When the answer uses an ordered list, preserve the order. Do not automatically call the first entry the assistant’s strongest recommendation unless the answer says so. List order alone does not establish a durable ranking across questions, conversations, or users.

For a named-firm case, record what the user already supplied. Repeating that name shows something different from introducing it in an unbranded question. Similarly, an answer that reads a supplied service page can demonstrate interpretation of that page without demonstrating independent discovery.

Keep the original event record intact if the operator asks a follow-up such as “What about our firm?” That may be useful diagnostic work, but the later mention was obtained after the firm was introduced. Label it as a follow-up observation and retain the preceding answer.

Law firm example

An answer names a lawyer, identifies the lawyer’s former employer as the current firm, and cites an old conference biography. Record a professional mention, the stated affiliation, and the conference-page citation. After checking current records, add the affiliation error to the assessment. Do not count the lawyer’s appearance as a successful mention of the current firm unless that firm is actually identified.

Automatic name matching can help locate candidate mentions, but a reviewer should resolve ambiguous names and relationships. Preserve the underlying passage so a later reviewer can understand the classification without trusting the detector’s label.

5.4 Assess Factual Accuracy, Completeness, Suitability, and Favorability Consistently

Review each answer against the exact question and a dated reference pack. Evaluate material claims individually where necessary. An answer can correctly locate an office, misidentify a partner, and recommend an unsuitable service in the same paragraph.

Use the following rubric as a starting point. These are the playbook’s review categories, not an industry-standard GEO score. Preserve separate judgments instead of combining them into a single number that could hide a serious problem.

Verify facts and preserve uncertainty

For each important factual claim, save the answer passage, the verification source, the conclusion, and the reviewer. Refer unfamiliar qualifications or regulated capabilities to the appropriate professional. A marketing operator should not resolve a disputed admission, license, or service authorization through guesswork.

Unverified does not mean false. Failure to find a professional in one directory may reflect incomplete coverage, a name variation, or a search problem. A claim about current availability may require direct confirmation. Record the uncertainty and the next check rather than forcing a confident judgment.

The firm’s own website may also be outdated or unsupported. Reconcile conflicting evidence with the responsible owner. If confidential records establish a fact, use them within the approved review process and keep them restricted. Their existence does not establish that an assistant had access to them.

Judge completeness against the client’s task

Before reviewing, identify the few facts or conditions the question actually requires. A request to compare two providers on implementation responsibility needs that comparison. A general request for an explanation of commercial due diligence does not need to reproduce the firm’s entire service brochure.

Do not penalize a response for failing to settle something the available information cannot establish. An answer that identifies a missing fact and suggests confirming it may serve the client well. A clarifying question can also be useful when the initial request lacks a decisive requirement.

Treat suitability as a separate professional-services judgment

Check the requirements that could change the provider choice: service type, jurisdiction, qualifications, industry experience, client size, implementation responsibility, or another stated constraint. Distinguish a verified exclusion from a requirement that remains unresolved.

Some engagement conditions, such as current capacity or a matter-specific conflict, may need confirmation directly with the firm. An appropriately conditional recommendation can acknowledge that. Do not demand an unconditional recommendation where the answer lacks the information needed to make one.

Accounting example. The reference pack establishes that a firm provides outsourced finance and advisory services but does not perform statutory audits. An answer praises its audit team and recommends it for a statutory audit. Record favorable characterization, a material inaccurate capability claim, and failure of a verified suitability requirement. Positive language does not offset the other findings.

Conversely, an accurate explanation that the firm does not provide the requested service is useful to a prospective client. The GEO program should preserve that boundary. Favorability concerns the way the firm is characterized; it cannot determine whether the answer is true or helpful.

Check the reviewers as well as the answers

Have a second qualified reviewer assess a small shared set, including an ambiguous answer and a material error. Resolve differences in how the rubric is applied and record remaining disagreements. Anthropic’s evaluation guidance recommends explicit criteria and calibration of model-based assessments against expert human judgment; that principle is useful here even though its guide addresses agent evaluation more broadly. 

AI assistance can extract candidate claims or suggest classifications under the approved information policy. Retain human responsibility for material findings, and preserve enough evidence to challenge an automated score. Mark unreviewed records as unreviewed.

5.5 Inspect Cited Sources and Distinguish Observed Evidence From Assumptions About Answer Origins

Open the sources attached to material claims about the firm, its professionals, and the stated reasons for choosing a provider. A familiar publication name or plausible title does not establish that the linked page supports the claim. Inspect the destination and the relevant passage.

Research on generative search distinguishes citation support from citation coverage: whether a reference supports its associated statement, and whether the answer’s statements receive adequate support. Liu, Zhang, and Liang made that distinction in their 2023 evaluation. Use the distinction as an audit method; their historical results are not current failure rates for the products in your baseline. 

Check the claim against the page

1. Confirm the destination. Record the URL, publisher, page title, access date, and any visible publication or update date. Note redirects and inaccessible pages.
2. Locate the supporting passage. Identify the text that supports, qualifies, or contradicts the answer. A citation beside a paragraph may cover only part of it.
3. Check identity and scope. Confirm the relevant firm or person, service, geography, time period, and attribution. A former employer’s project may not establish the current firm’s experience.
4. Classify the support. Record supports, partly supports, does not support, inaccessible, or ambiguous, with a short explanation.
5. Keep the checks distinct. A claim can be true even when its citation is inadequate. A cited page can support wording that is itself outdated or inaccurate.

If you cannot inspect a page, keep that limitation visible. A search snippet is not a substitute for reviewing the full context. If the source changes between the answer capture and your review, record the timing and avoid claiming that the current page is exactly what was available during the run.

Build a source inventory

Consolidate repeated URLs into a source inventory linked to the answer records and claims. Retain the publisher, source type, relevant entity or service, dates, support assessment, and any known relationship to the firm. Preserve separate dated captures where a page changes.

Useful source types include firm-owned pages, professional or regulatory records, independent reporting, directories, event biographies, and paid or affiliated publications where that relationship is known. Classification guides the next question; it is not an automatic credibility ranking. A directory may repeat the firm’s own copy, while an official record may answer only a narrow credential question.

Do not turn citations into a complete origin story

Google describes Gemini’s displayed references as sources and related content, which may include public pages, uploaded files, or connected Workspace material. It also says that some responses provide no links. A missing citation therefore does not establish that an answer came entirely from training data, and a related link should not automatically be treated as support for every nearby claim. 

Wallat and colleagues distinguish a citation that supports a statement from evidence that the cited document actually drove its production. That research supports a narrow conclusion for this audit: a relevant citation is something to inspect, not a complete account of the system’s reasoning or information inputs. 

Consulting example. An answer recommends an assessment-only practice for implementation work and cites an older directory with broad wording. The current approved service description excludes implementation. Record the inaccurate capability claim, what the directory says, and the conflicting current evidence. “The directory caused the recommendation” remains a hypothesis.

A follow-up asking an assistant why it omitted a firm may suggest useful checks. Preserve that response as diagnostic material. Do not treat its explanation as a verified execution record, or infer from a mention that the firm was included in training data. Likewise, an omission cannot establish that the provider has never encountered the firm.

5.6 Separate Isolated Answers From Recurring Gaps and Material Misrepresentations

Group observations only after reviewing the individual records. Look for the same factual discrepancy, recurring unanswered requirement, repeated unsuitable recommendation, or absence from relevant discovery answers. Keep question types and product conditions visible when describing a pattern.

A defensible statement is: “The former affiliation appeared in three of six completed runs of this case in this interface across the two recorded dates.” It tells the reader what happened and where to find the evidence. It does not establish what proportion of all clients receive that description.

Repeated runs make variation and recurrence observable within the audit. They do not automatically create independent observations or a representative sample of buyer behavior. Record all planned attempts, including failures and less interesting answers. Do not keep trying until the preferred outcome appears and then retain only that result.

Create a gap register with evidence and owners

Give each issue an ID. Record the affected practice or person, exact claim or omission, linked runs and conditions, verification evidence, source findings, recurrence, significance, owner, and unresolved question. Keep a possible explanation separate from a verified fact. The following entries are independent hypothetical illustrations, with sample sizes chosen for each issue; they are not results from the pilot described earlier.

Frequency and seriousness are different considerations. One verified false claim about a qualification, an engagement result, or a service the firm excludes can warrant prompt attention. A firm missing from one plausible shortlist may warrant further observation. Use the professional review and escalation responsibilities established in Chapter 4.

Treat uncertainty explicitly. “The answer alleges a problem; supporting evidence has not been established” is different from “the allegation is false.” Assign someone to verify the matter before deciding how to respond. An unfavorable but accurate observation is not an error to be removed from the record.

Preserve strengths as well as gaps. Note cases where the firm is described accurately, the right professionals are associated with the work, citations support material claims, and service limitations are handled well. These findings help the team retain useful information while addressing deficiencies.

Close the baseline as a dated version. Keep later diagnostic runs and answers collected after interventions distinguishable from the original set. Chapter 6 will establish ongoing monitoring, and Chapter 7 will turn the verified findings and open questions into a prioritized work plan.

Practical Output: A Documented GEO Baseline

Assemble four linked components. They can sit in one controlled workbook or document index with evidence stored alongside it; separate software systems are unnecessary.

1. Baseline summary. State the practices, audiences, question types, experiences, dates, and conditions covered. Report planned attempts and actual outcomes, review coverage, principal findings, and important exclusions.
2. Saved answer evidence. Retain exact inputs, relevant context, full answers, citations, capture records, and reviewer judgments. Link every material finding to the record that supports it.
3. Source inventory. Record the inspected pages, publishers, dates, relevant claims, support assessments, and known relationships. Identify sources that could not be checked.
4. Gap register. List verified discrepancies, observed omissions, suitability problems, uncertain claims, evidence of recurrence, responsible owners, and the next questions to resolve.

Ask a colleague who did not run the audit to inspect one favorable answer, one unfavorable answer, one omitted-firm case, and one uncertain claim. Can that person reconstruct the test conditions, distinguish observation from judgment, and locate the evidence? Resolve missing records or ambiguous classifications before accepting the baseline.

The program owner should be able to explain what the firm learned, what remains unknown, and which findings need prompt professional attention. A dashboard can come later. The immediate deliverable is a reliable record of what was observed and a clear basis for the next decisions.