Synthetic monitoring for production, beyond uptime pings

01Pings measure liveness, not correctness

Every outage postmortem has a version of the same line: “monitoring was green.” Uptime pings answer one narrow question — did the server return an HTTP success code within N milliseconds? They don’t exercise the code path your revenue depends on. A broken signup form behind a bad JS bundle, an expired OAuth client secret, a payment provider whose schema changed silently, a feature flag that renders an empty dashboard — all invisible to a ping, all catastrophic for users.

The industry’s answer is synthetic monitoring: scripted checks that behave like real users, running continuously against production. Done right, this is just end-to-end testing executed on a schedule against live infrastructure. Done badly, it’s a pile of flaky checks that page you at 3am because a CDN hiccuped. The difference is entirely in the design decisions below.

02What a good synthetic check looks like

A flow-level check walks one critical user path end to end, asserts on outcomes (not pixels), and cleans up after itself. Model each check on a named business flow rather than a URL list:

# synthetic-checks/signup.yaml
flow: signup-and-first-login
schedule: "*/5 * * * *"        # every 5 minutes
regions: [us-east, eu-west]
steps:
  - open: /signup
  - fill_email: "{{ synthetic.identity }}@mail.example.com"
  - fill_password: "{{ synthetic.password }}"
  - click: Create account
  - expect_text: "Check your inbox"
  - extract_code_from_email: inbox={{ synthetic.inbox }}   # via mail clone
  - submit_code
  - expect_url: /dashboard
assert_timeout_s: 30

Three properties matter more than coverage breadth. First, every check has an owner — unowned alerts get ignored. Second, assertions are semantic (“dashboard heading renders”) not visual-only, which keeps them out of flake territory. Third, the check is hermetic: its email, SMS, and payment interactions hit clones, not live vendors, so runs cost nothing side-effect-wise.

03Frequency and regions: match cadence to blast radius

Run everything every minute and you’ll spend real money generating load and drown yourself in noise. Run things hourly and your mean time to detection is an hour. Calibrate by blast radius:

• Revenue path (checkout, paywall): every 1–5 minutes, from multiple regions.
• Auth path (login, SSO): every 5–10 minutes. Auth breakage blocks everything else.
• Core UX (search, profile, upload): every 15–30 minutes is usually fine.
• Long-tail pages: daily crawl-style sweeps instead of dedicated checks.

Regions matter for two reasons: CDNs and edge functions fail regionally, and latency-based routing can serve different backends. Two or three regions for revenue-critical checks is a sane default; one region everywhere else. Guard schedules in Molar work this way — you attach a cadence and region set to a flow, and the platform handles the run distribution, evidence capture, and alert routing so you’re not gluing cron to Playwright by hand.

04Marked synthetic identities, always

Every synthetic run should act through clearly labeled identities: emails like +synth--tagged addresses, accounts flagged in your database, order notes marking automation. This pays off repeatedly:

• Analytics stay clean — dashboards exclude synthetic traffic instead of showing phantom conversion dips.
• Support can recognize synthetic orders instead of investigating “customers” who never respond.
• You can audit check behavior after the fact: every artifact ties back to a known identity.

Unmarked synthetic data is how you end up emailing a fake customer a real invoice. If your checks need OTP codes or magic links, route them through a controlled mail system rather than a shared inbox — the same mechanics described in our guide to testing email workflows.

05Alert dedup: the difference between signal and noise

A single broken deployment fails every scheduled run until someone fixes it. At 5-minute cadence across three regions, one incident generates hundreds of failures. Naive paging on each failure trains everyone to ignore the channel within a week. Dedup before notifying:

# Grouping rule (pseudo-config)
group_by: flow_id
window: 15m
notify_when: first_failure_in_group
escalate_when: failures >= 3 consecutive   # ~15 min sustained
resolve_when: 2 consecutive passes
suppress_during_deploy_window: true          # known-change quieting

The key ideas: notify on group-open, not every failure; require sustained failure before paging humans (transient network blips self-heal); auto-resolve on consecutive passes; and suppress alerts during an active deploy window when a change is already being rolled back. Route sustained failures into the same channel your engineers already watch — for most teams that means opening a GitHub issue or firing the on-call webhook, not yet another dashboard nobody has open. This pairs naturally with PR-gated E2E in CI: if a change was going to break the flow, the gate should catch it before production ever sees it.

06Tie failures to diffs, not mysteries

An alert that says “signup flow failed” starts an investigation. An alert that says “signup flow started failing at 14:32 UTC, first failing run followed deploy a71f0e3 by 90 seconds; trace shows the submit button no longer enables” starts a fix. Correlating check failures with recent deploys and attaching full execution evidence — DOM snapshot, network log, console output — turns production incidents into ten-minute fixes.

This deploy-correlation habit also feeds your broader regression strategy: flows that keep breaking in production are telling you your pre-merge suite has a gap at exactly that spot. Treat every synthetic escape as a missing test case, write it, and add it to the PR gate.