A/B testing without an experimentation platform: how AI agents run B2B experiments

Google Optimize and Optimize 360 ended on September 30, 2023. Google did not replace them with a product of its own; it pointed users to third-party testing providers, naming AB Tasty, Optimizely and VWO, and opened its APIs so any testing vendor can connect to Google Analytics. Many B2B teams that had used the free tool were left choosing between a new licence and no testing at all.

The licence is not small. Vendr’s transaction data puts the median Optimizely contract at $81,447 a year, and Optimizely itself publishes no prices. Open-source tools lower the bill: GrowthBook offers a free self-hosted open-source plan limited to one project, and a cloud Pro plan at $40 per seat a month. Both still need someone to build variants, wire tracking and read the result, which is where most of the cost of a testing program sits.

There is a third option. A testing platform does four things: it assigns visitors to variants, renders the variants, records outcomes and computes the result. On a headless site built and run by agents, each of those is a small piece of code or analysis you already have the means to build. What a platform adds is a visual editor and a statistics engine, and on low-traffic B2B pages the statistics are the part you most need to get right yourself.

This is one piece of the argument in our pillar on enterprise headless without a DXP: the expensive part of a website is the work around it, and agents can run that work.

Every test starts as a written hypothesis with a reason, a metric and a size. The experiments agent keeps the backlog, scores it and proposes the next test; the marketing lead approves the order. Use this template for each entry. The examples are illustrative, not client data.

Browser-side testing tools load a script and then change the page after it renders. That costs performance and can make the original flash before the variant appears. On a static or server-rendered site, the better approach is to build both variants as complete pages and decide which one each visitor gets before the page is sent.

There are two common patterns. With an edge function or CDN worker, assign the visitor to a variant, set a cookie so they keep it and serve the matching prebuilt page. With server rendering, make the same assignment on the server. Either way, crawlers and visitors receive complete HTML, and the variant ID is written to a hidden form field and an analytics event. Because both variants are complete pages, a crawler sees a real page whichever variant it gets, and a speed test measures the page visitors actually receive.

Follow Google’s website testing guidance so a test never hurts search. Use rel="canonical" on alternate URLs pointing to the original, rather than noindex; use a 302 temporary redirect, not a 301, if you redirect; never show crawlers different URLs from people; and remove all elements of the test as soon as it ends.

Check your host’s current options before you design around them. Netlify, for example, documents a branch-based Split Testing feature that divides traffic between deployed Git branches using an nf_ab cookie, but notes that since August 26, 2026 it can no longer be added to new projects and is not receiving further development. An edge function gives you the same control in your own code.

Peeking is the most common way B2B tests mislead. Evan Miller showed that if you check a test repeatedly and stop at the first significant result, the reported significance levels become meaningless. In his worst-case simulation, checking after every observation produced a false positive rate of 26.1% instead of the 5% the tester assumed. His remedy is to commit to a sample size before the test starts.

He gives a rule of thumb for the sample per variant: n = 16σ²/δ², where δ is the smallest effect you want to detect. For a conversion rate p, σ² is p(1 - p). As a worked illustration, not a benchmark: with a 3% baseline and a smallest effect worth detecting of 0.6 percentage points (3.0% to 3.6%), n = 16 × 0.0291 / 0.000036, or about 12,900 visitors per variant.

Many B2B pages will not reach that within a quarter. That is worth knowing before anything is built: test bigger changes, pool similar pages into one test, or use an earlier metric such as form starts and read the pipeline metric as directional. Sequential designs with planned checkpoints, which Miller also points to, are an option when you must be able to stop early.

In B2B, a variant that lifts form fills can lower pipeline if it attracts students, job seekers and competitors. Decide before launch which CRM outcomes count and how you will read them, then write it into the test plan.

• Pass the variant ID into a hidden form field, store it on the contact and copy it to the deal.
• Primary metric: the on-site conversion the sample size was calculated for.
• Pipeline metric: sales-accepted meetings and qualified opportunities by variant, read after a window you set up front, such as one sales stage later.
• Quality guardrail: share of conversions from target accounts or companies that fit your ideal customer profile.
• Freeze the plan at launch. Changing metrics after you see data turns a test into a story.

Pipeline counts per variant will usually be too small for significance. Treat them as a veto: if the variant that wins on the primary metric produces visibly worse pipeline quality, do not ship it.

This is where agents change the economics. The work that used to need a developer, a CRO specialist and an analyst, plus a testing licence, is split across agents, with a person approving at two points: before launch and before shipping. A test that took a sprint to build becomes a reviewed pull request.

1. The experiments agent drafts the test plan from the backlog entry, including the sample size and end date.
2. The builder agent creates the challenger page and the edge assignment as a pull request, with previews of both variants.
3. The review board checks accessibility, design, performance, SEO (canonical on the challenger) and claims.
4. The tracking QA agent confirms in a test submission that the variant ID reaches analytics and the CRM.
5. The marketing lead approves the launch. Nobody reads results before the planned end date.
6. At the end date the experiments agent writes the readout: the primary metric with its confidence interval, the guardrails, the pipeline metric and a recommendation.
7. The marketing lead decides to ship, iterate or stop, and the builder agent removes the losing variant and the assignment code.

The roles, gates and notifications behind these steps are set out in the AI agent web team operating model.

We still recommend a commercial experimentation platform in some situations. Optimizely, for example, describes its experimentation product as covering A/B, multi-page, feature and server-side testing, and that breadth is worth paying for when you use it.

• High-traffic sites running many tests at once that need mutual exclusion, holdouts and a shared statistics engine.
• Feature flags and product experiments inside a logged-in application.
• Teams where marketers must launch visual tests without code review, and accept the performance trade-off.
• Personalization programs fed by a CDP, where testing and targeting share one audience model.
• Companies that already own a licence and use it well.

For a typical B2B marketing site with modest traffic to its key pages, the binding constraint is traffic, not tooling. For the wider platform picture, see our comparison of agent-run headless vs Contentful, Storyblok and Sanity.

On planconversion.com, tests follow the steps above: written hypotheses, splits that never rewrite the page in the browser, sample sizes fixed in advance and readouts that include the CRM. Speed matters here, because a test on a slow page partly measures the slowness; we explain the build in how we scored 100 on PageSpeed Insights.

For clients, this is how our conversion rate optimization work runs on an agent-built site.

The site itself is built and maintained through our website development service.

Winning pages are written answer-first so they can be cited in AI answers, which is the generative engine optimization side of the same work.

This is how we built and run our own site. If you want the same results, talk to our team and we will come back with a full solution for your stack.