Continuous testing in your CI/CD pipeline

01The feedback budget comes first

Before deciding what to run where, decide what developers are allowed to wait for. A workable budget for most teams:

• Pre-push / commit: under 2 minutes, or people stop running it locally.
• Pull request: under 10 minutes wall-clock for required checks, or reviewers context-switch and merges stall.
• Deploy: under 5 minutes for smoke, because it blocks the release itself.
• Scheduled (production): unbounded runtime is fine — nobody is waiting.

The budgets are the strategy. Anything that can’t fit a stage’s budget moves right — to a later stage where waiting is acceptable — instead of slowing the earlier one. This is the core of the shift-left discipline we describe in shift-left testing for shipping teams: catch bugs early, but only pay for early detection where it’s cheap.

02Commit stage: seconds, not minutes

What runs before code even reaches shared branches should be nearly free: lint, typecheck, unit tests, maybe a fast subset of integration tests against containers. The goal is not coverage — it’s a tight loop that catches the majority of mechanical mistakes.

# package.json — the local loop stays under ~90s
{
  "scripts": {
    "check": "npm run lint && npm run typecheck && vitest run --changed HEAD"
  }
}

Note --changed: running only tests affected by the diff keeps the loop honest as the suite grows. If your “fast” stage regularly exceeds its budget, the fix is almost always sharding, caching, or moving slow tests right — never asking developers to wait longer.

03PR stage: critical-path E2E behind required checks

The pull request is where behavior verification belongs. Run end-to-end tests over the critical paths your change could plausibly affect — signup, login, checkout, the top three workflows by traffic — plus targeted regression tests derived from the diff.

Two mechanisms make this stage real rather than decorative. First, the checks must be required, so a red mark actually blocks merge — we walk through that configuration in required status checks as a merge gate. Second, flaky failures must be quarantined within hours, not weeks, because a gate that fails randomly gets admin-overridden into oblivion. The Playwright plumbing itself is covered step by step in Playwright in GitHub Actions.

# .github/workflows/pr-e2e.yml (excerpt)
jobs:
  critical-path:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright install --with-deps
      - run: npx playwright test e2e/critical --shard=${{ matrix.shard }}/4
        env:
          STRIPE_BASE_URL: http://localhost:12111   # clone, not sandbox
      - uses: actions/upload-artifact@v4
        if: failure()
        with: { name: traces, path: e2e/test-results }

04Deploy stage: smoke, then promote

After deploy to an environment, run a smoke suite: one pass through each critical flow against the deployed artifact, using cloned third parties so it neither spends money nor sends email. Smoke tests assert “the release can serve users,” not “every feature works.” Five minutes, hard limit; anything deeper belongs to scheduled production testing.

Promote only on green. The value here isn’t catching logic bugs — those were caught at PR time — it’s catching deployment bugs: missing env vars, wrong migrations, CDN misconfig, the staging-parity gaps described in why staging lies to you.

05Production stage: schedules, not releases

The last stage runs continuously against production with read-only or marked identities: synthetic logins, search, checkout up to (but not through) payment, API contract probes. This is the safety net for everything upstream can’t see — real data shapes, real CDN behavior, real vendor drift. We cover the design in synthetic monitoring for production; the broader cadence map is in our guide to regression testing strategies.

# Every 15 minutes, from three regions
- name: login → dashboard loads < 2s
- name: search returns seeded result
- name: api/v1/orders schema matches OpenAPI
# Alert routing: page only on 2 consecutive failures, same region

06Artifacts close the loop

A failed check without evidence is a support ticket; a failed check with evidence is a diagnosis. Attach trace links to the PR — video, DOM snapshots, network log, console output, and the exact cloned third-party interactions — so the author debugs from the comment thread instead of reproducing locally. Molar posts these artifacts automatically: Guard runs the suites as required GitHub checks and attaches a Trace replay to every failure, which cuts most “can’t reproduce” exchanges out of review entirely.

07The three ways pipelines rot

• Budget creep. Each new feature adds “just one more” PR test until median check time doubles and engineers start merging on red. Re-audit against the budget quarterly and move tests right.
• Flake debt. Retries hide flaky tests instead of fixing them; the retry count becomes your real reliability metric. Track first-run pass rate per suite and quarantine anything below it.
• Orphaned stages. Deploy smoke keeps running for an app whose flows changed months ago, asserting a UI that no longer exists. Every scheduled stage needs an owner who edits its scenarios as the product changes.

None of these are tooling problems; all three are policy problems with tooling symptoms. Write the policy down — budgets, quarantine rules, stage owners — and revisit it whenever someone says “CI is slow again.”