Safeguards need safeguards too.
We automate tests and checks so we don't have to rely on humans catching every regression. But there's another problem we don't talk about as much:
What happens when the safeguard itself becomes wrong?
How safeguards drift
Every kind of automated check has a drift mode:
| Safeguard | How it drifts |
|---|---|
| Visual regression baseline | Screenshots captured against an old design; every legitimate redesign now "fails" |
| Contract test | Enforces a response shape or behavior the product no longer needs |
| E2E test | Becomes brittle — fails on selectors, timing, and environment noise unrelated to the feature |
| Accessibility rule | Configured with the wrong standard, or with exclusions that silently swallow real issues |
| Lint / policy check | Rules inherited from a past team, blocking patterns nobody remembers deciding against |
The endgame of all five is the same: false-positive fatigue. Teams get so used to red checks that mean nothing that they start ignoring the very systems designed to protect them. And I think that's dangerous — a safeguard everyone ignores is worse than no safeguard, because it consumes trust and attention while catching nothing.
Re-baselining is a decision, not a chore
The answer isn't to automatically re-baseline everything every few weeks.
A baseline represents something we've decided is correct. Changing it should be intentional.
For something like visual regression, I'd rather have the relevant engineers, design-system owners, and designers look at the change and agree: yes, this difference is expected — this is now our new accepted state.
In practice that looks like:
- A diff review, not a rubber stamp. Whoever approves the new baseline looks at the actual pixels, in context.
- A named owner per baseline. When it's everyone's job to accept diffs, it's no one's.
- A record of why. "Accepted 2026-09-01: spacing token change from DS-42" — so six months later, the baseline has a history instead of a mystery.
That third line is the whole point of intentional re-baselining: the process exists so that the unexpected diff has somewhere to surface.
A blocking safeguard is a signal too
And if a safeguard continually blocks legitimate changes? That's useful information too.
Maybe the safeguard itself needs maintenance. A contract test that fails on every valid product change isn't protecting the contract — it's advertising that the contract no longer matches the product. The same logic applies to a flaky E2E suite: every hour spent re-running it on green-noise failures is an hour the safeguard costs you instead of saving you.
The audit question
This has made me think differently about automated verification in general.
We spend a lot of time asking: did the test pass?
Maybe we also need to periodically ask: is this still the right test?
A quarterly safeguard audit doesn't have to be heavy. For each check in CI, ask:
- Is this passing cleanly? (Failing constantly = fix or delete it.)
- When it last caught a real problem, would anything else have caught it?
- Does it still encode behavior the product wants today?
- Is anyone still acting on its failures?
- If we deleted it today, would anything get worse?
If the answers are "no, never, no, no, nothing" — delete it. A dead check isn't free; it's a standing tax on trust in every red build.
Because a green checkmark only means something if we still trust the mechanism producing it.
The more we automate engineering safeguards, the more important that question becomes.
Who verifies the verification?