Sauce Labs with XCUITest

Using axe DevTools Mobile, you can find accessibility issues with your UI tests. Integrate your test with Sauce Labs and gain insights into the audience's experience of your app - before they've even seen it! Sauce Labs is the largest cloud-based testing platform, hosting various device and simulator configurations to build your digital confidence. Configure your tests to run on Sauce within your current build automation, to catch issues before they reach production.

Below is a guide for running your UI tests integrated with axe DevTools Mobile and Sauce Labs.

Prerequisites

1. Prepare your app for distribution. Since testing on Sauce Labs is performed with real devices, having the app and UITest targets provisioned before sending it to their service to be resigned will result in a smoother process.
2. Install the saucectl command-line interface, and set up your Sauce Labs credentials. Follow the instructions from Sauce Labs.

Running Tests on Real Devices

Add Sauce Labs Configuration

Sauce Labs uses a yaml file, located at at the project root, .sauce/config.yml. See the sample config below for testing on real devices.

apiVersion: v1alpha
kind: xcuitest
sauce:
  region: us-west-1
  concurrency: 1

xcuitest: 
  # file names for real devices have the `.ipa` extension
  app: "<app_name>.ipa"
  testApp: "<app_name>UITests-Runner.ipa"


suites:
  - name: "<app_name>UITests"
    devices:
      - name: "iPhone.*"
        platformVersion: "17.2"

Build Setup

From your project's root folder in the Terminal, call the following command to create device builds without needing to have a device plugged into your computer:

xcodebuild build-for-testing -configuration Debug \
                             -scheme "$APP_SCHEME" \
                             -target "$APP_UITEST_TARGET" \
                             -sdk iphoneos \
                             -derivedDataPath "./DerivedData"

This command will generate two app files located at ./DerivedData/Build/Products/Debug-iphoneos/: Your iOS application and your UI tests (ending in UITests-Runner).

Next, convert both application files to .ipa files. Follow Sauce Labs' instructions on creating .ipa files.

Once the .ipa files are created, create an apps folder in the project's directory and add both .ipa files to it.

Run the following command to push the .ipa files to Sauce Labs and begin testing:

saucectl run

Automation

Once you've confirmed your configurations are set up correctly, you're ready to automate this process! We've included a script below to automatically build .ipa files for devices, upload them to Sauce Labs, and run your XCUITests. Your team can also use this script to invoke a test run on Sauce Labs manually.

APP_LOCATION="DerivedData/Build/Products/Debug-iphoneos"
APP_NAME="<app_name>"

rm -rf *.ipa

xcodebuild build-for-testing -configuration Debug \
       -scheme "<scheme_name>" \
       -target "<UITests target>" \
       -sdk iphoneos \
       -derivedDataPath "./DerivedData" \
       -quiet

mkdir Payload

mv $APP_LOCATION/$APP_NAME.app Payload
zip -r -qq "$APP_NAME.ipa" Payload
rm -rf Payload/*

mv $APP_LOCATION/${APP_NAME}UITests-Runner.app Payload
zip -r -qq "${APP_NAME}UITests-Runner.ipa" Payload

saucectl run

rm -rf Payload
rm -rf DerivedData

Running Tests on Virtual Devices

You can run automated XCUITest tests with Virtual Devices on Sauce Labs! Follow the steps below to get started.

Add Sauce Labs Configuration

When testing with simulators, your configuration will be located at the project root, .sauce/config.yml. If you have done testing with real devices before, you should note that the configuration for simulators is different from the config you use for real devices. The .zip extension is used in the file names for app and testApp, and suites lists 'simulators' rather than 'devices'.

apiVersion: v1alpha
kind: xcuitest
sauce:
  region: us-west-1
  concurrency: 1

xcuitest:
  # file names for simulators use the `.zip` extension
  app: "<app_name>.zip"
  testApp: "<app_name>UITests-Runner.zip"

suites:
  - name: "<app_name>UITests"
    simulators:
      - name: "iPhone 12 Simulator"
        platformVersions:
          - "16.2"

Build Setup

You will build the app and the test runner for simulators specifically, using the snippet below as your guide. If you have done testing with Real Devices before, you may note a few differences here - -sdk iphonesimulator, EXCLUDED_ARCHS="", and CODE_SIGN_IDENTITY...

xcodebuild build-for-testing -configuration Debug \
  -scheme "$APP_SCHEME" \
  -target "$APP_UITEST_TARGET" \
  -sdk iphonesimulator \ 
  -derivedDataPath "./DerivedData" \
  EXCLUDED_ARCHS="" \
  CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO

Now, zip the .app files to prepare for upload to Sauce Labs.

  zip -r testApp.zip testApp.app
  zip -r testRunner.zip testRunner.app

Finally, run the following command to upload your .zip files and begin testing.

  saucectl run

Automation

Once you've set up the configuration for testing on simulators, you'll be able to automate this process! Use the script below to automatically build .app files for simulators, zip them up, upload them to Sauce Labs, and run your XCUITests.

APP_LOCATION="DerivedData/Build/Products/Debug-iphonesimulator"
APP_NAME="<app_name>"
RUNNER_NAME="<testApp_name>"

rm -rf $APP_NAME.zip $RUNNER_NAME.zip

xcodebuild build-for-testing -configuration Debug \
  -scheme "<scheme_name>" \
  -target "<UITests target>" \
  -sdk iphonesimulator \
  -derivedDataPath "./DerivedData" \
  -quiet \
  EXCLUDED_ARCHS="" \
  CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO

zip -r -qq $APP_NAME.zip $APP_LOCATION/$APP_NAME.app
zip -r -qq $RUNNER_NAME.zip $APP_LOCATION/$RUNNER_NAME.app

saucectl run

rm -rf DerivedData