Blog
What automated accessibility testing misses (and why it still matters)
2026-07-15
Published July 2026.
Every accessibility tool vendor, Gangway included, will tell you automated scanning is fast, cheap, and repeatable. Fewer will tell you plainly what it can't do — so here's the honest version, from a company that sells the scanner.
The number that matters: roughly 30–40%
Independent estimates put the share of WCAG 2.2 success criteria that
automated tools can reliably evaluate at somewhere around 30–40%. Axe-core —
the open-source engine Gangway runs, also used by Google Lighthouse and most
serious competitors — is widely regarded as the most accurate automated
engine available, and that ceiling still holds. It's not a limitation of any
one tool; it's structural. A machine can check whether an <img> has an
alt attribute. It cannot check whether that alt text actually describes
the image usefully. It can check color contrast ratios. It cannot tell you
whether your focus order makes sense to someone tabbing through with a
keyboard, or whether your custom dropdown is usable with a screen reader, or
whether your form's error messages are actually understandable.
What that means concretely
A clean automated scan is evidence of absence of a specific, checkable class of problems — not evidence of an accessible site. The gap is filled by things a machine genuinely can't do:
- Keyboard-only walkthroughs — can you complete every core flow without a mouse, in a sane order, without getting trapped?
- Screen reader testing — does the experience with NVDA/VoiceOver actually make sense, not just technically parse?
- Plain-language and cognitive load review — is the content understandable, not just structurally correct?
- Judgment calls — is this alt text accurate? Is this error message actually helpful?
Why this isn't an argument against automated tools
It's an argument for using them honestly. Automated scanning is still the right first pass: it's fast, catches the mechanical issues at scale across every page of a large site (a manual audit of 500 pages isn't happening), and — critically — it can be rerun on every change to catch regressions a one-time manual audit never will. The mistake isn't running a scanner; it's stopping there and calling the result "compliant."
What "good-faith effort" actually looks like
Run the automated scan broadly, fix what it finds, then spend the manual review budget where it matters most — the flows clients actually use, not every page equally. Document what was checked automatically, what was checked by a human, and what's still open. That's the model Gangway is built around: the scan is the first pass, the audit trail and documents (remediation report, accessibility statement, VPAT draft) are honest about what's covered and what isn't — never a green checkmark standing in for a claim nobody can actually back up.