Flaky tests: root causes and durable fixes

01Flake is a defect class, not bad luck

When the same commit passes on one run and fails on the next, teams learn the worst possible habit: re-run until green. Once that habit forms, the suite stops being a gate and becomes weather — people check it, nobody trusts it. The economic damage is quiet but compounding: engineers buffer their estimates for “CI roulette”, real regressions hide among false ones, and eventually someone proposes deleting the suite entirely.

The useful reframe is that flakiness is not random. Empirically, almost every intermittent failure falls into one of five buckets: timing races, shared state, environment drift, third-party failure, and selector brittleness. Each bucket has a specific, permanent fix. Diagnose the bucket first; reaching for retry before diagnosis just moves the problem from your failure report into your blind spot.

02Timing races: sleeps are guesses

The classic flake: the test works on your fast laptop, fails in CI where eight containers share four cores. The app takes 400ms to render instead of 200ms, and a fixed sleep that was tuned to “usually work” now loses the race. Any test that waits a duration is really waiting for luck.

// Fragile: sleeps guess at timing
await page.waitForTimeout(3000);
await page.click('#submit');

// Durable: act, then wait on observable state
await page.getByRole('button', { name: 'Pay now' }).click();
await expect(page.getByText('Payment confirmed')).toBeVisible();

Web-first assertions flip the model: instead of sleeping then checking once, they poll observable state until it’s true or a timeout expires. Two companion rules close the gap entirely. First, treat unhandled promise rejections and unexpected network errors as test failures rather than console noise — they are tomorrow’s flakes announcing themselves today. Second, freeze sources of variance you don’t need: animations off, fake clocks for anything time-based, deterministic IDs for anything paginated.

03Shared state: two tests, one row

The second bucket shows up when tests collide through something they both touch: the same seeded user, the same database table, the same localStorage profile. Symptoms include assertions on counts (“cart should have 3 items” seeing 7), duplicate-key errors that appear only under parallelism, and tests that pass alone but fail in the full run. Parallel execution doesn’t create these bugs — it reveals them.

Durable fixes, in order of preference: give every test its own data (factories that mint unique users and orders per invocation, then clean up); scope expensive shared fixtures per worker rather than globally; and where isolation must be absolute, roll back via transactions or run each worker against its own schema or container. If a test needs a specific global to exist, that dependency belongs in its setup — explicit, owned, and visible — never in the residue of whichever test ran last.

04Environment drift: it passed on my machine

Some flakes aren’t in your code at all. The staging server was mid-deploy. A feature flag flipped. CI runs in a region where the page renders in another locale, or under a timezone that makes your “end of month” invoice test compute a different month. These failures look mysterious precisely because the cause lives outside the test.

Countermeasures are organizational more than technical. Pin everything your tests implicitly depend on: timezone, locale, feature-flag state, seed data version. Prefer short-lived preview environments created per pull request over a long-lived shared staging box that accumulates entropy — see our guide to staging environment alternatives. And add a pre-flight health check so a half-deployed app produces “environment unhealthy”, not forty misleading test failures.

05Third-party failures: someone else’s uptime

Checkout tests that hit a live payment sandbox inherit its outage schedule. Email tests wait on real SMTP relay latency. Rate limits turn a burst of parallel tests into intermittent 429s. None of this is your product breaking, yet your build goes red and your team burns an afternoon proving it.

The durable fix is to stop sharing fate with third parties during routine runs. Stateful substitutes that speak the vendor’s API — returning deterministic IDs, replaying webhooks, delivering email to an inspectable inbox — remove the network, the latency, and the outages while exercising your real code paths. We compare the options in API mocking versus service clones, and walk a payments example in testing Stripe without real cards. Keep exactly one scheduled smoke against the real sandbox as a drift alarm, and you get hermetic daily runs without blinding yourself to contract changes.

06Selector brittleness: the slow rot

The fifth bucket degrades gradually rather than intermittently, but it manufactures flakes all the same: a selector like div:nth-child(3) > span.btn-primary survives until a designer wraps one node in a new div, and suddenly the locator matches nothing — or worse, matches a different button. Ambiguous text selectors produce the intermittent flavor: they pass while unique, then silently bind to a newly added element.

The fix ladder is well established: prefer role plus accessible name (getByRole('button', { name: 'Checkout' })), then label associations, then purpose-built data-testid, then unique text, and treat structural CSS as a code smell. When the UI legitimately changes, a disciplined self-healing layer can re-rank locators and propose the update for review instead of failing the build — provided intent was captured at authoring time.

07Retries and quarantine: policy, not vibes

Retries deserve nuance. Used deliberately — one retry, both outcomes reported — they absorb genuine environmental noise and generate a flake-rate metric you can manage down. Used silently, they convert your suite into a lottery where the prize is ignorance. The difference is whether a retry that flips red to green triggers investigation or celebration.

A workable policy: cap retries at one or two; tag any test that ever needed one; move repeat offenders to quarantine — still running and reported, but non-blocking, with a named owner and a hard expiry date so quarantine stays a hospital rather than a cemetery. Budget flake rate explicitly (under 1% of runs is achievable) and review it like any other SLO. Tooling should make this state visible: Molar marks flaky as its own run status — distinct from passed or failed — so intermittent behavior surfaces in dashboards without blocking merges. For the CI wiring that puts this policy into practice, see our GitHub Actions setup guide.