Coveo Atomic is our open-source library of web components for building Coveo-powered search interfaces. It has 170+ components, and we ship new versions constantly. Keeping all of them accessible is one challenge. Keeping the evidence from that work current, reviewable, and close to the code is another. That internal operations problem is what this post is about.
One output of that work is an Accessibility Conformance Report (ACR): a structured report showing how a product holds up against WCAG, Section 508 in the US, or EN 301 549 in Europe. The ACR is the published artifact. The evidence pipeline behind it is the part that made the work useful to us internally.
That distinction matters because the post is about the engineering loop. The value is that accessibility evidence is generated from the same checks and review process as the code it describes.
Note:
“VPAT” and “ACR” aren’t the same thing. A VPAT (Voluntary Product Accessibility Template) is the empty form, published by ITI. Fill it in with your product’s results and the finished document is an Accessibility Conformance Report (ACR), the version you hand to a buyer.
Most people say “VPAT” for both. I’ll say “ACR” when I mean the completed report from here on.
What the Pipeline Covers
Atomic’s ACR describes the Atomic package. Atomic is a component library, so application teams still decide page structure, content, configuration, user flows, implementation choices, and end-to-end experience. The pipeline keeps Atomic’s layer of that experience under continuous review: component behavior, automated rules, keyboard checks, manual findings, and scoped exceptions.
A “Supports” row in Atomic’s ACR means the evidence we have for the Atomic package supports that conformance claim within that package’s scope.
The WAI-ARIA Authoring Practices Guide (APG) is a different kind of resource. It’s a guide to how common UI patterns can be built, not a standard with its own conformance rules. We use it as a reference when designing some of our keyboard checks, but a component is always measured against WCAG, never against the APG.
The goal is to keep an accessibility quality bar checked continuously across releases. Automated checks, keyboard interaction tests, manual findings, reviewed exceptions, and the published report all move through the same release process.
Point-in-time accessibility evidence is useful, but it can drift as the product changes. For a library that ships weekly, that drift adds up fast.
So we built a pipeline that produces the ACR as a by-product of the same process we use for other release artifacts: from test results and manual evidence, checked by CI, reviewed as a diff, and republished to our CDN on every release.
We also tried to hand the hardest part, the judgment calls automated tools can’t make, to AI. That experiment didn’t survive, and scrapping it shaped everything else. More on that below.
The Drift Problem
Re-testing 170+ components manually on every release would not scale. But publishing an ACR that only gets refreshed occasionally would create another problem: the report could fall out of sync with the product.
So we made the evidence pipeline part of the release process. The accessibility tests already run in CI on every change, and the ACR is generated from those results, plus manual evidence and engineering-reviewed exceptions.
That does not make accessibility free, and it does not remove human review. It changes where the human effort goes. People spend less time rebuilding the same report and more time on the places where judgment matters: ambiguous criteria, screen reader experience, scoped exceptions, release-risk review, and periodic spot checks of the automation itself.
The Sweet Spot: Three Signals and a Worst-Wins Rule
Automated accessibility testing has a ceiling. It can verify a large portion of the WCAG success criteria: axe-core will reliably flag a missing alt attribute or text that fails a contrast ratio. But it can’t tell you whether there’s a keyboard trap three steps into a flow, or whether a screen reader announcement makes sense to the person hearing it.
So full automation isn’t realistic, and a fully manual audit doesn’t scale to the entire list of components. The useful work sits in between. We resolve each criterion from up to three signals, with one override on top:
- Static (axe-core): Catches the low-hanging fruit at scale. We run axe-core against every Storybook story to flag structural issues (missing roles, broken labels, contrast failures) across all components with zero human effort.
- Interactive: Verifies keyboard behavior axe-core can’t see. These tests are automated too; the difference is that they drive the component. Scripted checks follow the WAI-ARIA Authoring Practices Guide (APG), the W3C’s catalog of how each UI pattern should behave, to confirm components actually work the way a keyboard user expects.
- Manual: Covers the criteria that need human judgment, things like whether a screen reader announcement makes sense, or whether a flow is understandable without vision. Results are stored in a single JSON file, not scattered per component.
- AI (dropped): We tried having a model judge the criteria axe-core can’t reach. It would have increased our maintenance load and made our CI slower and non-deterministic, so we dropped it. The full story is below.
- Overrides: Lets engineering document intentional, by-design exceptions with a written reason. Used sparingly for cases where a criterion does not technically apply to the library.
An override always wins, and it always carries a written reason. When there’s no override, the verdict is the worst of whatever signals we have:
┌──────────────────────────────────────────────────────────┐ │ How we resolve each criterion: │ │ │ │ 1. Override exists? → use it (authoritative) │ │ 2. Otherwise, take the → worst result wins: │ │ worst across: │ │ • manual audit │ │ • interactive test │ │ • static (axe-core) │ │ │ │ 3. No layer has evidence? → "Does Not Support" │ │ (flagged for manual audit) │ │ │ │ Severity scale: │ │ does-not-support > partially > supports > n/a │ └──────────────────────────────────────────────────────────┘
A few things to note. If axe-core doesn’t cover a criterion, it contributes nothing, so another layer (manual or interactive) can fill the gap. But if no layer covers it at all, the ACR defaults to “Does Not Support” with a note that a manual audit is required. A manual “pass” can’t hide an axe-core violation: the violation still wins, and we must either fix the code or write an override with a reason.
Rule of thumb:
Our rule of thumb for deciding how each criterion is tested is simple: automate the checks that are reliable and worth automating; give a human the ones that are flaky or expensive to code.
The pipeline has three stages, each feeding the next.
1. Test (runs in CI on every PR)
Storybook stories are tested by both axe-core (static checks) and Vitest keyboard helpers (interaction checks). Some helpers are informed by APG pattern guidance, but they test Atomic’s production implementation, not APG example code. A custom Vitest reporter collects all results into a single JSON report.
2. Assemble (merge all evidence, worst-wins)
The JSON report is combined with manual audit results and engineering overrides. The transform resolves each criterion using worst-wins and outputs a single OpenACR YAML.
3. Publish (on every release)
The committed OpenACR YAML is rendered through the official VPAT 2.5 (International Edition) into markdown, then converted to a PDF that gets pushed to our CDN.
Static and Interactive Checks
We didn’t build a separate harness for any of this. Every Atomic component already has Storybook stories running under Vitest in CI, so flipping on Storybook’s accessibility addon got us axe-core on every story.
On top of that, we built a custom Vitest reporter, VitestA11yReporter. Its job is to gather every axe-core result across the whole run, map each rule to the WCAG criteria it covers, and write a tidy JSON report, stitching together sharded CI runs along the way. Nothing about it is Coveo-specific.
Axe-core is great at the static checks: roles, attributes, contrast. What it can’t tell you is whether a component actually works when you drive it with a keyboard.
APG’s role in those keyboard checks needs to be precise. APG is useful, but it is not the conformance baseline for the ACR. W3C describes APG as informative, not normative; it has no conformance model. W3C also says APG is not a UI design system, and APG example pages warn that their code is not intended for production environments.
So we do not compare Atomic against APG sample code, and we do not mark a WCAG success criterion as “Supports” merely because an interaction resembles an APG example. We use APG as a reference when designing deterministic keyboard assertions: which keys to consider, which state changes should be observable, which focus behavior should be checked, and where common ARIA mistakes tend to happen. The actual evidence comes from running those assertions against Atomic’s own implementation and recording the result against the specific WCAG success criterion it exercises, then validating higher-risk or judgment-heavy behavior manually.
So we wrote one small Vitest helper per interaction pattern Atomic uses. Each helper drives the component, checks the behavior a keyboard user relies on, and records the WCAG success criterion it covered. When Atomic intentionally differs from an APG example, the helper checks the accessible outcome rather than the APG sample’s exact markup or roles:
Eleven helpers in all. Nine are informed by interaction patterns that APG documents (combobox, dialog, tabs, radio group, switch, disclosure, carousel, checkbox, and table). The other two catch things axe-core can’t watch for, like content that appears on hover (1.4.13) and live-region announcements (4.1.3).
Those helpers are not a replacement for human keyboard testing or assistive technology testing, and a passing helper is not an APG certification. They are repeatable regression checks. When a helper changes, a component’s interaction model changes, or a release touches high-risk UI, we can manually spot-check the behavior against the automated helper. The automation is a floor, not a ceiling.
The AI Experiment We Deleted
Axe-core and the keyboard helpers cover the criteria a machine can check reliably. The rest needs a person: does this screen reader announcement make sense? Is this flow usable without vision? That’s expensive. So before we accepted a human in this loop for good, we tried to shrink their role with AI.
The idea was simple: let a model handle the criteria axe-core can’t reach, the ones that need something to interact with a component and judge whether it behaves correctly. We built an orchestrator that drove a real browser, captured the accessibility tree and live-region announcements, and fed all of it to a vision-capable model with WCAG knowledge baked into its prompts. It could see the UI, read the DOM, and judge conformance against the spec. Thousands of lines of code. It worked, sometimes.
That “sometimes” was the problem. It was non-deterministic: a component could pass on one run and fail on the next, sometimes with different reasons, which is useless for a CI gate. It was slow. And keeping the prompts and orchestration working cost far more than the coverage we got back. So we deleted it and went back to plain, deterministic tests. I’m not saying AI can’t do this; for us, right now, it didn’t pay off. The judgment calls went back to people.
Keeping Humans in the Loop
Everything axe-core and the keyboard tests can’t judge goes to a person. We store those results as plain JSON in a single file. We started out auditing one component at a time, then realized a component only means something once it’s composed into a working experience. A facet isn’t accessible or inaccessible on its own; it’s accessible inside a search page. So manual audits look at the whole experience, not individual widgets.
{
"wcag22Criteria": {
"2.4.7-focus-visible": "pass",
"2.4.6-headings-and-labels": {
"conformance": "fail",
"remarks": "Facet group headings all read 'Filters'; users can't tell which attribute each one controls."
}
}
} From Signals to a Published Report
The merged result is a single OpenACR YAML file, the machine-readable ACR format maintained by the U.S. General Services Administration (GSA). That file is the source of truth, and from there two things happen to it.
We protect it. The YAML is committed to the repo, and a CI check regenerates it from the latest test results and fails if it doesn’t match what’s committed. Any shift in conformance, a new violation, a fix, a fresh manual result, or a scoped exception shows up as a diff in a pull request someone approves before it ships.
The important control is not just that the file is generated. It is that conformance changes become reviewable code changes. A new failure, a remediation, a manual audit update, or an exception all produce a diff. That diff has to be reviewed before it becomes the published report.
We publish it. On every release, we render the YAML through the official VPAT 2.5 (International Edition) and use Playwright to turn it into the PDF that lands on our CDN, the copy a buyer actually downloads. Each row carries its own evidence:
- num: 2.1.1
components:
- name: web
adherence:
level: supports
notes: >-
Supports — automated axe-core found no violations across 173
applicable components; interactive keyboard testing passed across
34 applicable components.
- num: 2.4.5 # Multiple Ways
components:
- name: web
adherence:
level: not-applicable
notes: >-
Multiple navigation ways are a host-application concern, not a
component-library concern. “Not Applicable” is not a pass. It is a scope statement. For a component library, some WCAG criteria belong to the consuming application rather than the component package itself: page titles, page-level navigation, language of page, repeated blocks, authentication flows, and other full-experience concerns. We still require a written rationale for those rows so the scope stays explicit.
Limits of This Approach
This pipeline gives us fresher evidence, not perfect certainty.
Atomic covers the component layer. Complete search pages still depend on how teams compose components, structure content, and configure flows. Axe-core cannot judge criteria that require human interpretation. Our keyboard helpers do not cover every possible path through every component. APG remains an informative reference, not a normative conformance model or a source of production-ready code.
That is why the pipeline is built as a quality bar plus human review so it gives a repeatable way to surface regressions, review conformance changes, document scope decisions, and keep the published report aligned with the code.
What We Learned
A few things stuck with us.
Automation has a point of diminishing returns, and finding it is most of the work. The thousands of lines we deleted taught us more than the code we kept. The useful question was never “can we automate this?”, but rather “is automating this cheaper and more reliable than having a person do it?” For many criteria, the answer was “no.” Designing around that reality, instead of fighting it, is what keeps this maintainable.
Model the audit the way the product is used. Auditing individual components in isolation didn’t work. Accessibility is a property of a whole experience, not a single widget, and the data should be shaped that way.
Commit the baseline and diff it in CI. Turning openacr.yaml into a checked-in file that CI validates is what moves you from “we have some accessibility tests” to “the report can’t drift without someone signing off.”
Take It With You
We set out to stop relying on a document that could fall behind on every release. What we have instead is an accessibility evidence pipeline that keeps a broad baseline checked with the code. One output is Atomic’s live ACR, and you can dig through the pipeline in the open-source coveo/ui-kit monorepo. If you run Storybook, Vitest, and axe-core, the reporter is the piece I’d most like to extract into a standalone package. If that’s something you’d use, tell us.
Accessibility work is never really finished. The useful change is operational: when the product changes, the tests run, the evidence updates, and meaningful shifts show up in review.
That gives us a better internal feedback loop and a clearer diff when something changes.
If you like building accessible, well-tested software with people who care about the details, take a look at our careers page and come work with us.