Why does a site pass axe-core but fail real screen reader users?
Because an automated scan evaluates a page's markup at one moment against rules a machine can decide, while a screen-reader user depends on what happens as they use it. A rule can confirm a button has a name; it cannot confirm that focus moves into the dialog the button opens, that the result is announced, or that the checkout can be finished.
Those behavioral failures are invisible to a static scan and can stop a real user completely — which is how a site with a clean report still loses the customer.
What an automated scan actually checks
Tools like axe-core, WAVE and Lighthouse load a page, read the DOM and the computed styles, and run rules against them. Each rule is a question with a deterministic answer: does this image have an alt attribute, does this text meet a 4.5:1 contrast ratio, does this ARIA role allow that attribute, is every form input named.
That is genuinely valuable. These failures are common — in 2026 the WebAIM Million found detectable WCAG failures on 95.9% of the top million home pages, an average of 56.1 per page, led by low-contrast text (83.9% of pages), missing alternative text (53.1%) and missing form labels (51%) (WebAIM, 2026). A scanner finds them on every page, in seconds, every release.
But a scan has three structural limits:
- It sees one state. The page as it loaded — not after a click opens a menu, a form fails validation, or a single-page app changes route.
- It checks presence, not meaning. It can tell that a link has a name, not that “Read more” repeated twelve times tells a listener nothing.
- It evaluates pages, not tasks. Nothing in a rule result says whether someone can get from the product page to a confirmed order.
WebAIM, whose report depends on automated testing, puts the consequence plainly: “Absence of detected errors does not indicate that a page is accessible or conformant.”
What the studies show
You will often see a single percentage for how much automated testing catches. The honest answer is a range, because studies count different things.
Independent: one page, ten tools
In 2017 the UK Government Digital Service built a deliberately inaccessible page with 143 failures and ran ten automated tools against it. The best single tool found 37% of the barriers; counting prompts for manual inspection as findings, the most effective tool reached 41%. All ten tools combined found 71%, and 29% were not picked up by any of them. GDS concluded the tools are “most effective when combined with manual testing” (UK Government Digital Service, 2017). It is one test page, and the tools have changed since, so treat the figures as a shape rather than a current measurement.
Vendor: issues by volume
In 2021 Deque, which maintains axe-core, analysed more than 2,000 audits covering over 13,000 pages and reported that 57.38% of total issues were identified by its automated tests (Deque Systems, 2021). Two caveats matter: it is self-reported by the tool's maker, and it counts issues by volume, so the most common, most automatable failures — contrast, missing labels — weigh heavily.
By success criteria
Counted the other way — how many WCAG success criteria automation can fully test — the numbers drop. Deque describes the answer as usually varying “anywhere between 20 and 30 percent” (Deque Systems, 2021). That is an industry estimate a vendor is summarizing, not a study, and we could not find a primary source for the often-quoted 30–40% figure, so we do not use it.
How to read these together: automation catches a large share of the failures that occur most often, and a small share of the kinds of failure WCAG describes. The ones it misses are disproportionately the ones that stop a task outright.
Seven failures a scan cannot see
Each of these can sit on a page with a clean automated report. They share one trait: they only appear when someone uses the page.
Focus that does not follow the interface
WCAG 2.4.3 Focus Order
- What the scan sees
- A dialog, drawer or menu with valid roles and labels.
- What the user gets
- The dialog opens and focus stays on the button behind it, so the next Tab moves through the page underneath. On close, focus jumps to the top of the page.
Focus lost after a route change
WCAG 2.4.3 Focus Order
- What the scan sees
- Each route of a single-page app, loaded fresh, with no violations.
- What the user gets
- Clicking a link swaps the content without a page load. Focus drops to the page body, nothing is announced, and the user has no idea anything changed.
Keyboard traps
WCAG 2.1.2 No Keyboard Trap
- What the scan sees
- A date picker, embedded player or chat widget whose markup passes.
- What the user gets
- Tab moves into the widget and never comes out. Keyboard-only users are stuck until they reload the page.
Status messages that are never announced
WCAG 4.1.3 Status Messages
- What the scan sees
- A cart page, a search page, a form — none of them contain an error.
- What the user gets
- "Added to cart", "3 results" and "Saved" appear on screen without a live region. A sighted user sees confirmation; a screen-reader user hears nothing.
Errors that only exist after you submit
WCAG 3.3.1 Error Identification
- What the scan sees
- An empty form in its initial state, correctly labelled.
- What the user gets
- After a failed submit, the errors appear in red but are not tied to their fields or announced, and focus stays on the submit button.
Names that exist but say nothing
WCAG 2.4.4 Link Purpose · 1.1.1 Non-text Content
- What the scan sees
- Every link and image has an accessible name, so the rule passes.
- What the user gets
- Twelve links announce "Read more", an icon button announces "button", and a product photo announces "IMG_2041". Present, and useless.
A task that cannot be finished
WCAG Several, in sequence
- What the scan sees
- Each page of the checkout, scanned separately, with a handful of minor findings.
- What the user gets
- Four payment fields announce "edit, blank". Nothing distinguishes the card number from the expiry. The purchase ends there — and no single rule result says so.
How to close the gap
The W3C's guidance on evaluation tools is blunt: they “can not determine accessibility, they can only assist in doing so” (W3C Web Accessibility Initiative, 2024). Closing the gap means adding checks that use the site the way people do.
- Keep scanning everything. Automated scans on a schedule and on every deploy catch detectable failures and regressions across every page. See automated scanning.
- Attempt the tasks that matter. Sign-up, search and checkout, performed keyboard-only while listening to what is announced. A simulated screen-reader journey does this on every release and reports the exact step that breaks.
- Monitor real sessions. Real user monitoring checks the states no scan or script reaches, and catches focus traps and lost focus as real visitors hit them.
- Test with people. Periodic review by experienced screen-reader and keyboard users catches the judgments no automation can make — whether content makes sense, whether a flow is reasonable, not just possible.
For a side-by-side view of what each automated layer catches and misses, read static scans vs. simulated journeys vs. real user monitoring.
Sources
- The WebAIM Million: The 2026 report on the accessibility of the top 1,000,000 home pages (opens in a new tab). WebAIM, March 30, 2026. Independent. Automated testing of home pages only.
- What we found when we tested tools on the world’s least-accessible webpage (opens in a new tab). UK Government Digital Service, February 24, 2017. Independent. One deliberately inaccessible test page, 143 failures, 10 tools.
- Deque Study Shows Its Automated Testing Identifies 57 Percent of Digital Accessibility Issues (opens in a new tab). Deque Systems, March 10, 2021. Vendor self-reported (Deque maintains axe-core). Measured by volume of issues, not by success criteria.
- Automated Accessibility Coverage Report (opens in a new tab). Deque Systems, 2021. Vendor. Describes the industry’s criteria-based estimate, not a study of it.
- Selecting Web Accessibility Evaluation Tools (opens in a new tab). W3C Web Accessibility Initiative, May 13, 2024.