Testing a form flow that depends on bot checks and AI

Testing a form flow that depends on a bot-check widget and an AI endpoint means deciding, deliberately, which parts get replaced and which stay real. A project interview touches three things you don’t control: the bot-check widget, a submission endpoint that sends real emails, and, in our case, an optional AI endpoint. The rule we settled on: replace everything that sends, charges, or calls out; drive everything else for real; and wait for visible signals, never for time.

The runnable examples below are a small stand-in page written for this note, not our production interview. Versions: @playwright/test 1.63.0, Chromium 141, Node 22.

What do you replace in an end-to-end test?

The submission endpoint. Tests must never send a real inquiry or email. The browser test intercepts the request to it and fulfills it with a canned success, then asserts on what was sent: the verification token, the visitor’s answers, and any AI-written brief that traveled with it.

The AI endpoint. The tests intercept it too, with scripted responses. This is what makes the interesting cases testable on demand: a follow-up question appearing, a delayed response that proves the “thinking” state, a malformed reply, an unavailable service. A model is never involved, so the tests are fast, free, and identical on every run.

The bot-check challenge. The real widget loads from the vendor’s servers, which isn’t something a test should depend on. We block that origin and inject a hidden field with the same name the widget would create, filled with a fake token. This verifies that our own code carries a verified token through every step.

Here is that setup as Playwright code. The page, the routes and the helper are all in the stand-in demo:

async function setup(page) {
  const sent = [];
  await page.route('https://challenges.example/**', (r) => r.abort());   // vendor origin blocked
  await page.route('**/api/probe', async (r) => {                        // scripted AI step
    await new Promise((res) => setTimeout(res, 300));
    r.fulfill({ status: 200, body: 'ok' });
  });
  await page.route('**/api/submit', (r) => {                             // never a real send
    sent.push(JSON.parse(r.request().postData()));
    r.fulfill({ status: 200, body: 'ok' });
  });
  await page.addInitScript(() => addEventListener('DOMContentLoaded', () => {
    const i = document.createElement('input');                           // fake token, widget's field name
    i.type = 'hidden'; i.name = 'cf-turnstile-response'; i.value = 'FAKE.TOKEN';
    document.getElementById('f').appendChild(i);
  }));
  return sent;
}

What do you leave real?

The page itself, its real scripts, and its real styles. The more of the actual product the test drives, the more the test is worth. Mocking our own interview engine would prove nothing.

Playwright test
   |  drives real page, real scripts
   v
 [form] --click Next--> [step engine] --POST /api/probe--> ROUTE (scripted, delayed)
                              |                                  |
                              |<--------------- 200 ------------+
                              v
                      show next step, counter changes   <-- the signal the test waits for
                              |
 [form] --Submit + token--> POST /api/submit --> ROUTE (records body, returns 200)

Bug one: why can’t a mock prove the browser’s behavior?

The injected token proved the form kept it correctly. It could not prove that the real widget would keep its own. In production, “Edit answers” followed by “Review” left a checkmark with no valid token underneath, because we had hidden the widget’s container with display: none, and the vendor invalidates a verified token when its container disappears.

A mock can’t catch that, because the mock doesn’t behave like the vendor. So the fix came with a different kind of test: a mutation observer on the form that records its computed display value across the edit-and-review round trip, asserting it never becomes none. When you can’t reproduce a third party’s behavior, pin the condition that triggers it.

We rebuilt the mechanism in the stand-in page, with a Review button that hides the container briefly, once with display: none and once with visibility: hidden. The test:

await page.evaluate(() => {
  window.__seen = new Set();
  const ts = document.getElementById('ts');
  const rec = () => window.__seen.add(getComputedStyle(ts).display);
  rec(); new MutationObserver(rec).observe(ts, { attributes: true, attributeFilter: ['style'] });
});
await page.click('#review');
await page.waitForTimeout(150);
expect(await page.evaluate(() => [...window.__seen])).not.toContain('none');

The bad variant fails with Received array: ["block", "none"], and the good variant passes. (The fixed waitForTimeout(150) is acceptable here only because the demo’s hide-and-restore runs on a known 50ms timer; in a real flow, wait on a visible signal.)

Bug two: why does an async step break the test, not the product?

When we added an AI follow-up probe after one interview step, two older tests started failing. Their helper clicked Next and immediately checked which input was visible. For one step the click now awaited a network call, so the helper read the previous step’s state and tried to fill a field that was about to disappear.

The product was fine. The test had silently relied on every step completing synchronously. The fix was to wait for an observable fact: read the step counter before clicking, and wait for it to change afterward. That version works whether a step takes a frame or a second, and it will keep working when the next asynchronous step arrives.

We reproduced it. Two helpers, the same three-step flow (name, project, budget):

// NAIVE: assumes every Next completes synchronously
async function naiveStep(page, value) {
  await page.locator('.step.active input').fill(value);
  await page.click('#next');
}
// FIXED: wait for the step counter to change after the click
async function step(page, value) {
  await page.locator('.step.active input').fill(value);
  const before = await page.locator('#counter').textContent();
  await page.click('#next');
  await expect(page.locator('#counter')).not.toHaveText(before);
}

Step 2 awaits the mocked probe before step 3 appears. We ran each test 10 times with --repeat-each=10:

Step 2 behavior	Naive helper	Counter-wait helper
Synchronous (no probe)	10 pass, 0 fail	10 pass, 0 fail
Async, probe answers instantly	0 pass, 10 fail	10 pass, 0 fail
Async, probe delayed 50 ms	0 pass, 10 fail	10 pass, 0 fail
Async, probe delayed 300 ms	0 pass, 10 fail	10 pass, 0 fail

Note the shape. The naive helper was fine until the step became asynchronous, then failed every time, even when the mock answered immediately, because any network round trip is slower than the next line of the test. A single failing run looks like this: right after the third naive call the page showed Step 2 of 3, the project field held 5000 (the budget answer, typed into the previous step’s field), and the budget field was empty. No submission ever reached the mock.

The lesson generalizes. Tests that sample state right after an action are encoding an assumption about timing. Tests that wait for a visible signal that the action completed encode only the behavior.

A mistake we made while writing the fix

Our first counter-wait helper failed on the last step. Submitting does not change the counter, so not.toHaveText(before) waited out its timeout on Step 3 of 3. The signal has to be one the action actually produces. For the final step we assert on the recorded submission instead, using expect.poll(() => sent.length).toBe(1).

Failure paths are first-class tests

For any optional feature, the tests worth writing first are the ones that disable it: the endpoint returns an error, the endpoint is unconfigured, the response is garbage. Each should end with the visitor completing the flow and the submission going through. If those pass, the feature can’t hold the product hostage, and everything else is a bonus.

The practical checklist

Intercept every endpoint that sends, charges, or calls out. Assert on what would have been sent.
Drive the real page, not a mocked copy of its logic.
When a third party’s behavior caused a bug, write a test for the condition that triggers it, since you can’t mock the vendor faithfully.
Wait for visible signals, not for time or immediate state.
Test the “off” state of every optional feature before the “on” state.

Frequently asked questions

How do you test a form with a Cloudflare Turnstile or other bot-check widget?

Block the vendor's origin in the test, then inject a hidden field with the same name the widget would create, filled with a fake token. Intercept the submission endpoint and assert that the token arrived in the request. This proves your code carries a verified token through every step, though it cannot prove the real widget's own behavior.

Why does my Playwright test fail after I add an async step?

The test probably samples state immediately after a click and assumed every step completes synchronously. In our reproduction, a helper that filled the visible input after each Next click overwrote the previous step's field when one step awaited a network call, and failed 10 of 10 runs. Waiting for the step counter to change passed 10 of 10.

What should you mock in an end-to-end test and what should you leave real?

Mock whatever sends, charges, or calls out: the submission endpoint, the AI endpoint, and the vendor widget. Leave real the page, its scripts and its styles. Mocking your own interview engine proves nothing, while driving the real page tests the code visitors actually run. Assert on the requests your mocks receive.

How do you test a bug that only a third-party widget causes?

Pin the condition that triggers it, because you cannot mock the vendor faithfully. Our bug came from hiding the widget container with display none, which invalidates the vendor's token. A mutation observer recording the container's computed display across the edit-and-review round trip, asserting it never equals none, caught the bad variant and passed the good one.

How do you test the failure path of an optional AI feature?

Write the disabled-state tests first: the endpoint returns an error, is unconfigured, or returns garbage. Each test should end with the visitor completing the flow and the submission going through. If those pass, the optional feature cannot hold the product hostage, and the working state is a bonus on top.