What can I honestly claim from an SEO before-and-after analysis?
Short answer
You can claim the change happened and that it's consistent with your intervention, with the competing explanations listed. You cannot claim causation from a before-and-after alone, because a single group observed at two points has no control for seasonality, algorithm updates, competitor moves, or category growth. Adding an unaffected control group is what upgrades the claim from "consistent with" to "caused by."
The structural problem
A before-and-after design observes one group at two times. Any difference could come from your intervention or from everything else that happened between the two dates.
The five confounders that account for most false claims:
Seasonality. Almost every B2B category has an annual shape, and comparing Q4 to Q3 without year-over-year context mistakes the calendar for a result.
Algorithm updates. Google ships changes constantly. A ranking gain in the same window as a core update is not obviously yours.
Competitor movement. A competitor deindexing, redesigning, or being acquired moves your numbers with no action from you.
Category growth. Rising demand lifts everyone. If the category grew 20% and you grew 18%, you lost share while your chart went up.
Your other work. SEO teams rarely change one thing. Whatever else shipped that quarter is a candidate too.
Language that survives scrutiny
The claim you make should match the design you ran.
| You ran | You can say | You cannot say |
|---|---|---|
| Before-and-after only | "Traffic rose 34% following the refresh, consistent with the intervention" | "The refresh caused a 34% increase" |
| Before-and-after plus control group | "Refreshed pages rose 34% while comparable unrefreshed pages rose 6%" | Still avoid "proves," but this is a strong causal claim |
| Geographic or temporal holdout | "Pausing the channel reduced branded search 12% against control" | Nothing much, this is close to experimental |
| Year-over-year with control | "Adjusted for seasonality and category movement, the lift was 19%" | Precise attribution to a single change |
The phrase doing the work in row one is "consistent with." It's honest, it's still persuasive, and it doesn't collapse when someone points out there was a core update that month.
The control group you can almost always build
Most teams believe they can't run a control. Usually they can.
Take the pages you refreshed and find comparable pages you didn't: similar intent, similar traffic band, similar age. Track both groups across the same window. If refreshed pages rose 34% and the comparison set rose 6%, the difference is attributable in a way the raw 34% never was.
Build the comparison set before you intervene, not after. Selecting it afterwards invites choosing pages that make the result look good, and you'll know you did it even if nobody else does.
For the fuller isolation method, see How do I test whether an organic lift was real and not seasonal or brand-driven?.
The confounder that's new
There's now a sixth item on the list, and most before-and-after analyses don't include it.
A page can lose clicks because an AI surface began answering its query directly, with nothing about the page having changed. Conversely a page can gain branded search weeks after earning citations, with the credit landing on whatever shipped most recently.
Our own state of AI search research, covering more than 22,000 answers, found Google AI Mode returns a citation-rich answer in 97.4% of matched B2B queries and Perplexity in 93.2%. It also found presence is path-dependent, with next-observation mention probability at 83.2% on ChatGPT after a mention versus 9.9% after a miss.
Which is why a single reading proves nothing in either direction. Researchers make the same point directly, arguing visibility should be measured as a distribution across runs, prompts, and time rather than as one observation.
Report the confounders you couldn't rule out
The strongest before-and-after reports include a short section listing what else happened in the window and what you did to account for it. Something like: a core update landed in week three, two competitor pages were refreshed, and category search volume rose 7% year-over-year.
Counterintuitively, listing the things you couldn't control makes the claim more credible rather than less. It signals that the number you're presenting has already survived your own scrutiny.
How Analyze AI keeps the comparison honest
The risk in a before-and-after is that the method quietly changes between the two readings, so the difference reflects your measurement rather than your work.
Agent self-inspection recipes address this directly. workflow-memory returns outputs from prior successful runs of the same agent and last-run-output returns the most recent, which means the after-reading is computed by the identical workflow that produced the before-reading. failed-steps-log surfaces where a run partially failed, which matters because a silently incomplete data pull looks like a performance change.
A before-and-after agent that carries its own controls:
Start (schedule, monthly) → GSC Page Performance Summary on the intervention page set (daily series) → GSC Page Performance Summary on the pre-registered control set → Get Visibility Events (AI visibility movement per provider) → workflow-memory (retrieve the baseline run and the last three comparisons) → Code (compute lift for both groups, difference the two, adjust for year-over-year seasonality) → Prompt LLM (write the claim at the strength the design supports, list unresolved confounders including AI visibility movement) → DOCX export → Send Email.
The instruction worth putting in that Prompt LLM step is to write the claim at the strength the design supports. Given a control group it should say "compared against control." Without one it should say "consistent with." Encoding that rule in the agent stops the language inflating in the months when the result is good.
FAQ
Related answers
- How do I test whether an organic lift was real and not seasonal or brand-driven?
- How do I set an attribution baseline before changing our content strategy?
Want before-and-after reports that carry their own control set? Start a free Analyze AI trial and build the comparison once.