Blog

Posted April 7, 2023

How to Run Espresso Tests on CI with Real Mobile Devices

Learn how to achieve confidence for our SDK with the help of Sauce Labs RDC and GitHub Actions.

Here is a crazy idea: let’s improve git push to make our CI run Espresso tests on a real device, powered by Sauce Labs Real Device Cloud (RDC). That’s right. Every time we have a change in our app code, a real Google Pixel phone will launch our test suite and show us the results, all without leaving the comfort of our GitHub repo. This is how we test our mobile SDKs, and it is how you should test your app too!

Ingredients

Where is the phone?

Sauce Labs data centers host a vast number of real mobile devices, accessible by developers to ensure their apps can confidently run across all Android phones in the wild. You can harness Real Device Cloud devices with any testing framework of your choice.

As of 2021, Google Play shows there are at least 15,000 different Android device models people use all around the world. We can’t simply buy all of those devices and test TestFairy SDK in each one of them. It is not a race we can ever hope to win without the help of a robust service. If we want to be confident that TestFairy SDK will run without bugs on any device, going cloud is our best shot. Below, you will find the exact code for how we achieve confidence for our SDK with the help of Sauce Labs RDC and GitHub Actions.

Steps

Let’s start with configuring our repo for saucectl. It is the tool we use under the hood to get access to RDC. saucectl cli takes a look at our current directory (i.e., in CI) to decide what kind of tests to run, on which devices with the criteria we define.

Create a new folder named “.sauce” in the project root and put a configuration file named “config.yml” in it.

We will need to define the following items: testing framework, data center region, app paths, real phone specification.

1
apiVersion: v1alpha
2
kind: espresso
3
sauce:
4
 region: us-west-1
5
# Controls how many suites are executed at the same time (sauce test env only).
6
 concurrency: 2
7
 metadata:
8
   name: Testing My Demo App Android with espresso
9
   tags:
10
     - e2e
11
     - release team
12
     - other tag
13
   build: Release $CI_COMMIT_SHORT_SHA
14
espresso:
15
 app: ./app/build/outputs/apk/debug/app-debug.apk
16
 testApp: ./app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
17
suites:
18
 - name: "Main suite"
19
   devices:
20
     - name: "Google Pixel 3a"
21
       platformVersion: 11
22
     - id: Google_Pixel_3a_real
23
# Controls what artifacts to fetch when the suite on Sauce Cloud has finished.
24
artifacts:
25
 download:
26
   when: always
27
   match:
28
     - junit.xml
29
   directory: ./artifacts/

Once you finish, you can execute saucectl run locally and see your test results on your Sauce Labs dashboard in real time.

This is already really cool but we want to do more. Let’s use GitHub Actions to run these tests, so that every time a developer makes an update to our repo, the tests are automatically executed.

Our GitHub repo doesn’t know that we use Sauce Labs yet, so let’s add that.

Navigate to your project secrets: https://github.com/MY_ORGANIZATION/MY_APP/settings/secrets/actions

Add a repository secret named SAUCE_USERNAME and enter your saucelabs.com username in it.

Similarly, add another repository secret named SAUCE_ACCESS_KEY and paste your access key in it.

Now that our repo knows about our Sauce credentials, we can connect to RDC inside all of our GitHub workflows. For a basic “test on push” scenario, we define the following workflow file (.github/workflows/test-with-saucectl.yml).

1
name: Test with saucectl
2
on: [push]
3
env:
4
 SAUCE_USERNAME: ${{secrets.SAUCE_USERNAME}}
5
 SAUCE_ACCESS_KEY: ${{secrets.SAUCE_ACCESS_KEY}}
6
jobs:
7
 main:
8
   timeout-minutes: 20
9
   runs-on: ubuntu-latest
10
   steps:
11
     - name: Checkout
12
       uses: actions/checkout@v2
13
     - name: Setup Java
14
       uses: actions/setup-java@v1
15
       with:
16
         java-version: 11
17
     - name: Setup Android SDK
18
       uses: android-actions/setup-android@v2
19
     - name: Compile
20
       run: |
21
         ./gradlew app:assembleDebug
22
         ./gradlew app:assembleDebugAndroidTest
23
       shell: bash
24
     - name: Test espresso with saucectl
25
       uses: saucelabs/saucectl-run-action@v1
26
       with:
27
         testing-environment: ""
28
         concurrency: 10
29

Finally, we commit our changes and push.

Once our changes are merged to the main branch for the first time, GitHub will be able to start scheduling actions whenever a new contribution arrives.