Coveo Atomic is our open-source library of web components for building Coveo-powered search interfaces. It has 170+ components, and we ship new versions constantly. Keeping all of them accessible is one challenge. Proving it, on every release, in a document a procurement team will read, is another. That’s what this post is about.
That document is called an Accessibility Conformance Report (ACR). If you sell software to a large enterprise or a public-sector buyer, someone will eventually ask for one: a structured report showing how your product holds up against WCAG, Section 508 in the US, or EN 301 549 in Europe.
Note:
“VPAT” and “ACR” aren’t the same thing. A VPAT (Voluntary Product Accessibility Template) is the empty form, published by ITI. Fill it in with your product’s results and the finished document is an Accessibility Conformance Report (ACR), the version you hand to a buyer.
Most people say “VPAT” for both. I’ll say “ACR” when I mean the completed report from here on.
The usual way to produce an ACR is to bring in a third party, run an audit once, and publish a PDF. That PDF is accurate on the day it’s signed, but every release that touches UI without re-testing may drift a little further from reality. For a library that ships weekly, that drift adds up fast. So we built a pipeline that produces the ACR the same way we produce any other build artifact: from real test results, checked by CI, and republished to our CDN on every release.
We also tried to hand the hardest part, the judgment calls automated tools can’t make, to AI. That experiment didn’t survive, and scrapping it shaped everything else. More on that below.
The Cost Problem
Re-testing 170+ components for accessibility on every release would be prohibitively expensive. So we made the report a by-product of the build instead: the accessibility tests already run in CI on every change, and the ACR is generated from their results. Keeping it up-to-date costs nothing extra.
Automation carries everything it can. A person steps in only for the criteria a machine can’t judge.
The Sweet Spot: Three Signals and a Worst-Wins Rule
Automated accessibility testing has a ceiling. It can verify a large portion of the WCAG success criteria: axe-core will reliably flag a missing alt attribute or text that fails a contrast ratio. But it can’t tell you whether there’s a keyboard trap three steps into a flow, or whether a screen reader announcement makes sense to the person hearing it.
So full automation isn’t realistic, and a fully manual audit doesn’t scale to the entire list of components. The useful work sits in between. We resolve each criterion from up to three signals, with one override on top:
- Static (axe-core): Catches the low-hanging fruit at scale. We run axe-core against every Storybook story to flag structural issues (missing roles, broken labels, contrast failures) across all components with zero human effort.
- Interactive: Verifies keyboard behavior axe-core can’t see. These tests are automated too; the difference is that they drive the component. Scripted checks follow the WAI-ARIA Authoring Practices Guide (APG), the W3C’s catalog of how each UI pattern should behave, to confirm components actually work the way a keyboard user expects.
- Manual: Covers the criteria that need human judgment, things like whether a screen reader announcement makes sense, or whether a flow is understandable without vision. Results are stored in a single JSON file, not scattered per component.
- AI (dropped): We tried having a model judge the criteria axe-core can’t reach. It would have increased our maintenance load and made our CI slower and non-deterministic, so we dropped it. The full story is below.
- Overrides: Lets engineering document intentional, by-design exceptions with a written reason. Used sparingly for cases where a criterion does not technically apply to the library.
An override always wins, and it always carries a written reason. When there’s no override, the verdict is the worst of whatever signals we have:
┌──────────────────────────────────────────────────────────┐ │ How we resolve each criterion: │ │ │ │ 1. Override exists? → use it (authoritative) │ │ 2. Otherwise, take the → worst result wins: │ │ worst across: │ │ • manual audit │ │ • interactive test │ │ • static (axe-core) │ │ │ │ 3. No layer has evidence? → "Does Not Support" │ │ (flagged for manual audit) │ │ │ │ Severity scale: │ │ does-not-support > partially > supports > n/a │ └──────────────────────────────────────────────────────────┘
A few things to note. If axe-core doesn’t cover a criterion, it contributes nothing, so another layer (manual or interactive) can fill the gap. But if no layer covers it at all, the ACR defaults to “Does Not Support” with a note that a manual audit is required. A manual “pass” can’t hide an axe-core violation: the violation still wins, and we must either fix the code or write an override with a reason.
Rule of thumb:
Our rule of thumb for deciding how each criterion is tested is simple: automate the checks that are reliable and worth automating; give a human the ones that are flaky or expensive to code.
The pipeline has three stages, each feeding the next.
1. Test (runs in CI on every PR)
Storybook stories are tested by both axe-core (static checks) and APG keyboard helpers (interaction checks). A custom Vitest reporter collects all results into a single JSON report.
2. Assemble (merge all evidence, worst-wins)
The JSON report is combined with manual audit results and engineering overrides. The transform resolves each criterion using worst-wins and outputs a single OpenACR YAML.
3. Publish (on every release)
The committed OpenACR YAML is rendered through the official VPAT 2.5 (International Edition) into markdown, then converted to a PDF that gets pushed to our CDN.
Static and interactive checks
We didn’t build a separate harness for any of this. Every Atomic component already has Storybook stories running under Vitest in CI, so flipping on Storybook’s accessibility addon got us axe-core on every story for free.
On top of that, we built a custom Vitest reporter, VitestA11yReporter. Its job is to gather every axe-core result across the whole run, map each rule to the WCAG criteria it covers, and write a tidy JSON report (stitching together sharded CI runs along the way). Nothing about it is Coveo-specific.
Axe-core is great at the static checks: roles, attributes, contrast. What it can’t tell you is whether a component actually works when you drive it with a keyboard. For that we lean on the APG, the WAI-ARIA Authoring Practices Guide, which spells out how each pattern should behave: a dialog traps focus and hands it back on Escape, tabs move with the arrow keys, a switch toggles on Space. That’s a contract, and a contract is testable.
So we wrote one small helper per APG pattern Atomic uses. Each one drives the component the way the spec says it should behave and records the criteria it covered. When Atomic does its own thing on purpose, the helper checks the behavior a keyboard user relies on rather than the exact roles:
// Verifies the WAI-ARIA APG Combobox keyboard contract.
// https://www.w3.org/WAI/ARIA/apg/patterns/combobox/
export const COVERED_CRITERIA = ['2.1.1'] as const; // Keyboard (Level A)
export async function testComboboxA11y(context: StoryContext) {
const input = await within(context.canvasElement).findByShadowRole('textbox');
let status: 'passed' | 'failed' = 'passed';
try {
input.focus();
await userEvent.keyboard('test{ArrowDown}'); // open and move into suggestions
await waitFor(() =>
expect(input.getAttribute('aria-activedescendant')).toBeTruthy()
);
await userEvent.keyboard('{Escape}'); // dismiss; focus must stay on the input
// ...
} catch (error) {
status = 'failed';
throw error;
} finally {
context.reporting?.addReport?.({
type: 'a11y-interactive',
version: 1,
status,
result: {criteriaCovered: [...COVERED_CRITERIA]},
});
}
} Eleven helpers in all. Nine cover APG patterns (combobox, dialog, tabs, radio group, switch, disclosure, carousel, checkbox, and table); the other two catch things axe-core can’t watch for, like content that appears on hover (1.4.13) and live-region announcements (4.1.3).
The AI Experiment We Deleted
Axe-core and the keyboard helpers cover the criteria a machine can check reliably. The rest needs a person: does this screen reader announcement make sense? Is this flow usable without vision? That’s expensive. So before we accepted a human in this loop for good, we tried to shrink their role with AI.
The idea was simple: let a model handle the criteria axe-core can’t reach, the ones that need something to interact with a component and judge whether it behaves correctly. We built an orchestrator that drove a real browser, captured the accessibility tree and live-region announcements, and fed all of it to a vision-capable model with WCAG knowledge baked into its prompts. It could see the UI, read the DOM, and judge conformance against the spec. Thousands of lines of code. It worked, sometimes.
That “sometimes” was the problem. It was non-deterministic: a component could pass on one run and fail on the next, sometimes with different reasons, which is useless for a CI gate. It was slow. And keeping the prompts and orchestration working cost far more than the coverage we got back. So we deleted it and went back to plain, deterministic tests. I’m not saying AI can’t do this; for us, right now, it didn’t pay off. The judgment calls went back to people.
Keeping Humans in the Loop
Everything axe-core and the keyboard tests can’t judge goes to a person. We store those results as plain JSON in a single file. We started out auditing one component at a time, then realized a component only means something once it’s composed into a working experience. A facet isn’t accessible or inaccessible on its own; it’s accessible inside a search page. So manual audits look at the whole experience, not individual widgets.
{
"wcag22Criteria": {
"2.4.7-focus-visible": "pass",
"2.4.6-headings-and-labels": {
"conformance": "fail",
"remarks": "Facet group headings all read 'Filters'; users can't tell which attribute each one controls."
}
}
} From Signals to a Published Report
The merged result is a single OpenACR YAML file, the machine-readable ACR format maintained by the U.S. General Services Administration (GSA). That file is the source of truth, and from there two things happen to it.
We protect it. The YAML is committed to the repo, and a CI check regenerates it from the latest test results and fails if it doesn’t match what’s committed. Any shift in conformance, a new violation, a fix, a fresh manual result, shows up as a diff in a pull request someone approves before it ships.
We publish it. On every release, we render the YAML through the official VPAT 2.5 (International Edition) and use Playwright to turn it into the PDF that lands on our CDN, the copy a buyer actually downloads. Each row carries its own evidence:
- num: 2.1.1
components:
- name: web
adherence:
level: supports
notes: >-
Supports — automated axe-core found no violations across 173
applicable components; interactive keyboard testing passed across
34 applicable components.
- num: 2.4.5 # Multiple Ways
components:
- name: web
adherence:
level: not-applicable
notes: >-
Multiple navigation ways are a host-application concern, not a
component-library concern. When that check does flag a difference, fixing it doesn’t mean re-running the slow Storybook suite. A small script pulls the latest report straight from the CI run and rebuilds the YAML locally for human sign off.
What We Learned
A few things stuck with us.
Automation has a point of diminishing returns, and finding it is most of the work. The thousands of lines we deleted taught us more than the code we kept. The useful question was never “can we automate this?”, but rather “is automating this cheaper and more reliable than having a person do it?” For many criteria, the answer was “no.” Designing around that reality, instead of fighting it, is what keeps this maintainable.
Model the audit the way the product is used. Auditing individual components in isolation didn’t work. Accessibility is a property of a whole experience, not a single widget, and the data should be shaped that way.
Commit the baseline and diff it in CI. Turning openacr.yaml into a checked-in file that CI validates is what moves you from “we have some accessibility tests” to “the report can’t drift without someone signing off.”
Take It With You
We set out to stop hand-writing a document that fell behind on every release. What we have instead is an ACR that keeps pace with the code and ships to our CDN with it. You can read Atomic’s live ACR and dig through the whole pipeline in the open-source coveo/ui-kit monorepo. If you run Storybook, Vitest, and axe-core, the reporter is the piece I’d most like to extract into a standalone package. If that’s something you’d use, tell us.
Accessibility work is never really finished. As long as the product keeps evolving, the report has to evolve with it. Now it does that mostly on its own. That’s the part I’m glad about: nobody has to sit down and build this thing from scratch again.
If you like building accessible, well-tested software with people who care about the details, take a look at our careers page and come work with us.