An AI growth agent is software that connects to your analytics sources, decides on its own schedule what is worth your attention, and tells you the single thing to do about it. Not a dashboard you interrogate. Not a chat box waiting for the right question. Something that reads your GA4, Search Console, ads and billing data, works out which part of your funnel is actually holding growth back, and hands you one action.
That is the promise. The reason the category is worth writing about carefully is that the promise is unusually easy to fake. Any language model can produce a paragraph that sounds exactly like a growth diagnosis. Whether the numbers in it are true, and whether the recommendation would survive contact with your actual data, is a completely separate engineering problem, and it is the one that separates the products.
What makes it an agent and not a dashboard
Four properties, and a tool has to have all four before the word means anything.
It runs unprompted. A dashboard waits for you to open it. An agent shows up every morning whether or not you remembered to look, which matters because the weeks you stop checking are exactly the weeks something drifts.
It decides, it does not describe. Twenty charts that are all technically true is not a decision. An agent has to be willing to say this is the constraint right now and everything else can wait, and be wrong sometimes as the price of being useful.
It reads across sources. Your funnel does not live in one tool. Search Console knows whether anyone can find you, GA4 knows what they did next, your billing system knows whether it mattered. An agent that only sees one of those can only ever diagnose one layer.
It is accountable to its last answer. If it told you to do something last week, this week it should say whether that worked. Advice with no memory is horoscopes.
The trust problem is specific, and it is not hallucination
When people worry about AI and data, they picture invented numbers. That is the easy case. A number that appears nowhere in your account is trivial to catch in code, and it usually looks wrong to a human too.
The dangerous failures are different: real numbers in the wrong sentence. These are the ones we hit repeatedly while building our own agent, on our own data:
- Right number, wrong owner. “59 of your 106 sessions came from accounts.google.com.” Both figures were real. The 59 belonged to a different channel. The true answer was 32. A checker that only asks “does 59 appear in the data” waves this straight through.
- Right numbers, wrong direction. “5 clicks this week, down from 4.” Both true, and the story is backwards.
- Right analysis, useless instruction. Holding 25 rows of query data, it told us to go export the query data. Homework it had already done.
- Right data, wrong headline. Our own written rule says a measurement defect is never the verdict, because fixing it changes what you know rather than what you get. One run headlined the measurement defect anyway. Two copies of the same system, reading the same week of the same business, disagreed about which layer was failing.
None of those read as errors. They read as confident analysis, which is precisely why they do damage: you act on them, or you quietly stop trusting the whole surface.
Why prompting is not the fix
The instinct is to write a better prompt. We did that first, and we had rules covering every failure above, written plainly and repeated. They held most of the time, and most of the time is the problem: a rule that holds on four runs out of five is fine for tone and unusable for facts, because the fifth looks exactly like the other four and lands on the same card.
We could only see the real rate once something was checking mechanically. In the first seven briefings after the checker went live across both our environments, six tripped at least one flag. That is a small sample and we are two days into it, so treat it as a smoke alarm rather than a statistic. Some of those flags turned out to be the checker's own false alarms, which we then had to go and fix. The rest were claims that would have shipped.
A rule in the prompt is a preference. A rule in code is a guarantee. Anything your product promises has to live on the second side of that line.
The practical version: work out which parts of the output your promise depends on, and take those away from the model entirely. Compute them in code before generation, hand them over as givens, then verify the finished text against them and rewrite when it drifts.
Five questions to ask any AI growth agent
These work as an evaluation checklist whether or not you ever look at ours. Each one maps to a failure we actually shipped and then had to fix.
1. Does anything check the numbers after the model writes them?
Ask directly. The answer is usually no, or “the prompt tells it to be accurate,” which is the same as no. The follow-up that separates real answers from marketing: does the check know which source a number belongs to, or only that the number exists somewhere in the account? Only the first catches the 59.
2. Does it commit to one thing, or hand you a list?
A ranked list of eight opportunities is a dashboard with better manners. Ask what it would have you do today, and see whether you get one answer with a reason attached.
3. Does the verdict hold still?
Refresh it twice over the same period. If the top problem changes when the data did not, the ranking is being improvised by a language model each time, and next week's advice will contradict this week's.
4. Is the recommendation an action or homework?
“Export your top queries and analyze which ones underperform” is a task list for you. “Rewrite the title on this page, which ranks ninth for this query and got zero clicks on 30 impressions” is an action. The second one requires the tool to have actually done the work.
5. Does it tell you what it cannot see?
Every analytics setup has blind spots: consent-denied sessions, login redirects that erase the acquiring channel, conversion data that has not settled yet. An agent that never mentions uncertainty is not confident, it is unaware. In ours the sample floors are constants in code rather than requests in a prompt, so a layer with too little traffic to judge comes back as untested rather than as a confident failure. Below thirty genuine sessions we will not call activation broken, because at that volume nobody can.
A ten-minute test you can run on any of them
Connect one source. Then, in order: read the headline and ask whether it names a specific part of your funnel or just reports traffic. Take one number from the summary and find it yourself in GA4 or Search Console, paying attention to whether the channel attribution matches. Refresh and see whether the verdict survives. Finally, look at the recommended action and ask whether you could do it in the next hour without opening another tool. Most products fail on the second or the fourth.
What we built, and what we still get wrong
Alice is our answer to this. Every number on the morning briefing is checked in code against the data it came from before you see it, scoped to the right source, verified for direction, and rewritten once with the exact violation named when a claim does not hold. The funnel verdict is computed by code, not chosen by the model, so identical data produces an identical diagnosis. Recommendations that ask you to go fetch data the system already holds are rejected outright.
The honest part: our checker has been wrong too. It once read the word “directly” as the Direct traffic channel, decided the sentence was misattributed, and threw away the best briefing the system had ever produced in favour of a weaker one. False positives destroy good output as efficiently as hallucinations ship bad output, so most of our test suite now exists to prove that correct sentences pass. Verified numbers also cannot make an interpretation wise, and we are explicit about that rather than pretending the checker is a truth oracle.
If you want the underlying mechanics of the failures above, the three GA4 attribution bugs post covers why login redirects corrupt exactly the sessions that convert, and SEO vs GEO vs AEO covers the discovery-side measurement that most funnels are missing entirely.