How to Troubleshoot Problems

Watcher Only Supports Chrome

Although the axe Developer Hub website supports several browsers, the @axe-core/watcher package only supports Google Chrome. If you try to use Cypress's Electron browser for testing, for instance, you will receive an error. See System Requirements for more information about supported software with axe Developer Hub and @axe-core/watcher.

Parallel Test Runners

If your test suite uses multiple test runners that run in parallel, you must ensure each test runner uses the same non-null buildID. Otherwise, each test runner's results will overwrite other test runners' results for the same commit SHA.

You typically set buildID in your test suite's AxeConfiguration. For more information about using parallel test runners with different CI/CD platforms, see buildID in the API Reference.

Duplicate Accessibility Errors or New Issues Count Wrong

If your website uses dynamic IDs or class names that change whenever the page is refreshed, you will likely see duplicate accessibility errors, particularly issues marked as new when previous test runs exhibit the same issue on the same element. (Axe Developer Hub uses IDs and classes to identify the same element between test runs.) To solve this problem, you must set the ancestry property in the runOptions object in your configuration to true. The example below shows how to set the option in your configuration:

axe: {
  runOptions: {
    ancestry: true
  }
}

See runOptions for more information.

Results Not Appearing

If you've run your test suite and no results are showing up in axe Developer Hub for a certain project, the cause could be found among the reasons in the following sections:

Not Configuring axe Watcher

Your modified test suite needs to call the appropriate configuration function for your test framework before running your tests. If you don't configure axe Watcher properly, it might be difficult to identify this problem. For example, you might see a problem like one of the following:

1. Your test suite could fail with an unreturned promise error.
2. Your test suite could silently fail.
3. Your test suite could time out.

Consult the configuration instructions for configuration examples for your language and browser test framework.

Not Flushing Results

You need to call the flush() function (or the custom command axeWatcherFlush() in Cypress) to send the collected results back to Deque's servers so the results can be presented on the axe Developer Hub website. Usually, you call the flush() function in your automation platform's cleanup hook.

For example, in the support/e2e.js file in Cypress you add the call to afterEach():

// Flush axe-watcher results after each test.
afterEach(() => {
  cy.axeWatcherFlush()
})

Using the --incognito Option

You cannot use the --incognito command-line option with Chrome; otherwise, your tests will silently fail. If you're using incognito mode to avoid writing cached files to disk (cached files are kept in memory only in incognito mode), use your testing suite's caching methods instead.

Not Setting the Required Environment Variable

The samples in the axe Developer Hub sample repository on GitHub use an environment variable for setting the API key, API_KEY.

You can set it in the environment or specify it on the command line when running tests.

Test Running Too Quickly

Your tests might run too quickly, unloading the page and freeing up its resources before Watcher can analyze it. To fix this problem, you can add a delay at the end of the test to allow time to analyze the page.

For example, in Cypress, you can add a delay of 10 seconds (10,000 milliseconds) with the cy.wait() method:

describe('Visitor', () => {
  it('should visit example.com', () => {
    cy.visit('https://www.example.com')
    cy.wait(10000);  })
})

Missing or Invalid API Key

An invalid or missing API key appears as an invalid configuration file in Cypress. The stack trace will reveal whether it is invalid or missing. A missing key results in the following:

AssertionError [ERR_ASSERTION]: API key is required
    at validateApiKey ...

(Many lines of the stack trace were deleted for brevity.)

An invalid key results in the following stack trace (shorted):

Error: Server responded to https://axe.deque.com/api/api-keys/test/validate/axe-devtools-watcher with status code 404:
{"error":"Invalid API key"}
    at Response.getBody
...

Help

If you can't resolve your problem, please email us so we can help.