Is AI Visibility a Vanity Metric? Ask What Decision It Changes

An AI visibility report can be accurate and still be useless. “Your brand appeared in 23 percent of answers” is a number you can put in a presentation. It is not yet something you can put on a task list.

I asked B2B marketers on Reddit whether appearing in AI answers helps them make decisions or just creates another metric to report. This was a small discussion, not a representative survey. But the replies exposed a useful distinction: people need different levels of evidence for different decisions.

One marketer said AI was their leading source of leads, while still warning against tying individual deals to it. Another would use missing buyer questions to change content, but wanted pipeline evidence before changing a budget. A third had lost trust in visibility tools after vendors waited until several sales calls in to explain how variable AI answers are. Read the lead source comment, the content versus budget reply, and the tool trust complaint.

All three positions make sense. They are asking the metric to do different jobs.

Disclosure: we build Alice at TranX. I asked the question because this is the measurement problem we are working on, and because a confident looking score is easy to build. A score somebody can trust and act on is harder.

A mention count cannot tell you what to fix

Suppose your brand appears whenever someone asks what your company does, but disappears when they ask which tools solve the problem you sell a solution for. Your mention count might look healthy. Your buyer discovery might not be.

That is why the underlying question matters more than the total. So does the difference between being named and being cited. An answer might mention your company but link to a competitor's comparison page. If you only count mentions, you miss the most useful part of the result: the source the buyer can actually visit.

A useful finding names the buyer question, saves the answer, distinguishes mentions from citations, and shows who appeared instead. Then you can investigate the gap. Is your relevant page hard to read? Does it answer a different question? Is the cited competitor page clearer about the use case? Those possibilities lead to different fixes.

One respondent made this point plainly: understanding why you are absent is often more valuable than watching the score change. Read the comment.

Three decisions, with three evidence thresholds

What should we investigate? A competitor repeatedly appearing for a buyer question you care about is enough to warrant a closer look. You do not need to prove lost revenue before reading the cited pages and checking your own.

Did our change help? Keep the question, market, and checking method consistent. Save the answers over time. If you improve a page and later appear more often for that question, that is encouraging. It is not proof the page change caused the movement. AI answers vary, and a single run is too fragile to carry that claim.

Should we spend more? That needs stronger evidence. Look for identifiable visits from AI tools, relevant landing page activity, qualified leads, and what prospects tell your sales team. One commenter wanted to see a consistent pattern across important questions, then visits or branded searches, and eventually qualified leads before calling visibility actionable for the business. Read their reply.

A visibility gap can justify a content investigation. It should not, on its own, justify a new budget line.

GSC and GA4 are useful, but they do not close every gap

Several people pointed to Search Console and GA4 as the most concrete measurements available. We use both.

Search Console's generative AI report shows impressions and pages appearing in Google's AI features, including AI Overviews and AI Mode. That is first party evidence of visibility on Google. The report does not show the full buyer question, the competitor beside you, or what another AI product answered. Google's report documentation lists the dimensions it provides.

GA4 can show visits with an identifiable AI referral source and what those visitors do on your site. But a person might see your name without a link, search for you later, and arrive as branded search or Direct. Missing referral information can also leave a visit classified as Direct. Neither case gives you a reliable way to tie one AI answer to one later deal. Google documents the traffic acquisition report and the limits of Direct traffic.

That does not make the data worthless. It means the honest report has separate lines for observed answer visibility, identifiable visits, and business outcomes. They can move together without pretending they are the same measurement. Our guide to measuring AI search traffic in GA4 goes deeper on where the referral number falls short.

The trust test I would use

Before putting an AI visibility metric in a report, try to finish this sentence:

For this buyer question, in this market, across these dated checks, we appeared this often. These competitors and sources appeared instead. We will make this specific change, then check the same question and the measurable visits or leads again.

If you cannot fill in the question, the checks, and the proposed action, the score is probably decoration. If you can, visibility has become a working signal, even if revenue attribution remains incomplete.

Also ask how the answers were collected. Was the tool checking a consumer AI product, or an API model with web search? Which prompts did it use? What happened when an engine returned no answer or no citations? A vendor should make those distinctions clear before the fourth sales call.

For a manual baseline, our guide to checking AI recommendations walks through the questions and the record to keep.

The point is a better decision

AI visibility is a vanity metric when the goal is simply to make the number go up. It becomes useful when it tells you which buyer question you are missing, why a competitor is appearing, what you can change, and what evidence you will check afterward.

That is the standard we are building toward with Alice. The answer, the gap, the next action, and the measurable outcome should be visible together. The uncertainty should be visible too.