Running Selenium Tests in parallel with GitHub Actions

How to run Selenium tests in parallel with GitHub Actions

Introduction

In our ongoing quest for efficiency and modernisation, our company recently moved from using Teamcity to using GitHub Actions for our Continuous Integration (CI) processes. This significant change brought with it a number of challenges, particularly for the Quality Assurance (QA) team. As QAs, we were faced with the critical task of adapting our test automation suites to integrate with the new workflow. This included a complex suite that used a mixture of Geb, Selenium, TestNG and Allure and was dedicated solely to UI testing. This transition wasn't just a change of tools. It was a strategic move towards more agile, collaborative and efficient development practices, which led us to rethink and refine our approach to software testing.

Beginning

Our journey started with the initial configuration based on the standard implementation - Selenium hub and nodes (Firefox and Chrome). The whole environment was running on a specific Teamcity agent (virtual machine) that was pre-configured. By preconfigured I mean that all Docker images were stored on this agent, so there was no need to download them. Also, compared to other Teamcity agents, our Teamcity Selenium agents had more RAM to handle running parallel tests. So the whole process was quite simple:

• Set up the Selenium hub using Docker Compose
• Set up Chrome and Firefox nodes with docker compose
• Scaling nodes up to 6 (to run parallel tests) using docker compose
• Running tests through TestNg

We ran all the tests through the TestNg test group option. In each test we added specific groups such as

@Test(groups = ["apples"])

Performance: ~11 min,

Debugging: Hard (need to rerun all the suite)

Approach 1: Lets do the same

We started by adapting the test automation suite to GitHub Actions with a simple approach. Do the same configuration as in Teamcity.

Right from the start we noticed some problems. First, we couldn't have pre-installed Docker images on the workers (for security reasons), so we had to download the image each time. The second problem was when we wanted to scale nodes. We couldn't use Docker Compose to set up the Selenium hub and nodes. Scaling this way wasn't very good either. The GitHub Actions workers were much smaller than the Teamcity workers. Performance dropped significantly when scaling to 6 separate nodes.

We also tried to change RAM size but we couldn't directly manipulate the 'power' of runners (workers). They had strict RAM policy.

Performance: ~19 min

Debugging: Hard (need to rerun all the suite)

Approach 2: Lets run every test per worker

The problem with the previous approach was mainly performance. We were experiencing a significant drop in performance (almost 72%, from 11 minutes to 19 minutes). It was at this point that we needed to focus on improving performance.

And a new problem appeared. How to run tests in parallel with GitHub Actions? Previously we were running tests through TestNg group option and Selenium hub & nodes. But how to run Selenium tests in parallel without this architecture. We would need to split tests in previous steps somehow and then pass list of tests to next jobs.

We would need something like this:

• Specify which group of tests the user wants to run
• Pass the name of the test group to GitHub Actions
• Split tests from this group
• Pass tests to separate GitHub Actions workers
• Run tests with new method (could not use TestNg group run anymore)

We decided to run each test on a separate worker. First, we need a simple shell script to get a list of all available tests from the test group (directory in this case):

declare -a array=(./src/test/groovy/com/airhelp/geb/tests/$1/*)

count=0
for i in ${array[@]}
do
  array[$count]=$(sed 's/.\/src\/test\/groovy//g;s/.groovy//g;s#/#.#g;;s#.com#com#g;' <<< $i)
  count=$((count + 1))
done

printf '%s\n' "${array[@]}" | jq -R . | jq -cs .

What does this script do?

• Declare array with path to directory containing tests
• Get all the tests and modify the text (in this case, remove .groovy and change "/" to ".")
• Return stringified JSON

As a parameter, it takes the test path to the directory with tests. So if we run it on the test directory 'apples', it should return a list of all the tests in the apples directory.

Lets present now the part of preparation workflow:

preparation-tests:
  runs-on: [self-hosted, utils]
  steps:
    - name: Checkout tests repo
      uses: actions/checkout@v3

    - name: List all tests
      id: set-matrix
      run: |
        echo "TESTS=$(bash get_list_of_test.sh ${{ inputs.testPath }})" >> $GITHUB_OUTPUT
    - name: Get tests
      run: echo "The selected tests are ${{ steps.set-matrix.outputs.TESTS }}"
  outputs:
    matrix: ${{ steps.set-matrix.outputs.TESTS }}

In next steps we will use the output from this step.

- name: Run tests with test path
  shell: bash
  run: |
    docker run --rm -u 0\
    --network=host \
    -v $(pwd):/tests -w /tests \
    -Dah.max_retries=2 -Dah_hub_port=4444 -DmaxParallelForks=1 \
    -DtestPath=${{ matrix.test-path }} -s clean remoteTest

Succeed! We run parallel tests even faster than before. But a new problem has arisen.

We have 30 tests. For each test we need a separate worker. This means that for 30 tests we would need 30 workers. It's quite a big effort.

• We have a certain number of available workers (e.g. 20). What if we want to run another test suite? Another suite would wait for the first 20 workers to finish their work. A large queue of waiting workers would appear.
• We have to set up a separate environment for each worker. This means, for example, downloading Docker images, which are sometimes quite large. Imagine 30 separate GitHub Actions workers concurrently downloading 30 images, each approximately 1GB in size.

Performance: ~7 mins

Debugging: Good (after fail we can rerun specific test not whole test suite)

Approach 3: multi groups

The final approach is to use subgroups and it looks like this:

• Identify all tests with a specific test group (e.g. documents)
• Manually split it into subgroups like documents_1, documents_2, documents_3 (just change the group name from documents to documents_1, documents_2)
• Define the list of subgroups in a separate JSON. Name this JSON as the main test group name (e.g.: documents, this will be used in the GitHub Actions workflow)

{
  "test_group": ["documents_1", "documents_2"]
}

And how does it look step by step in the GitHub Actions flow?

• In the GitHub Actions workflow, use the workflow_dispatch event trigger to manually run tests with parameters (in my case, includeGroups).

on:
  workflow_call:
    inputs:
      includeGroups:
        type: string
        description: "includeGroups"
        required: true
  workflow_dispatch:
    inputs:
      includeGroups:
        description: "includeGroups"
        type: choice
        required: true
        default: "default_group"
        options:
          - documents

• List tests subgroups from JSON and output in preparation step:

preparation-tests:
  runs-on: [self-hosted, tests]
  steps:
    - name: Checkout tests repo
      uses: actions/checkout@v3

    - name: List test groups
      id: set-matrix
      run: |
        echo "matrix=$(sed -e 's/^ *//' < ./tests/${{ inputs.includeGroups || 'default_group' }}.json | tr -d '\n')" >> $GITHUB_OUTPUT
    - name: Get tests
      run: echo "The selected tests are ${{ steps.set-matrix.outputs.matrix }}"
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}

• Use matrix to split GitHub Actions workers for each subgroup

selenium:
  name: Run Selenium tests for group - ${{ inputs.includeGroups }}
  runs-on: [self-hosted, tests]
  timeout-minutes: 120

  needs: [preparation-tests]
  strategy:
    fail-fast: false
    matrix:
      test-group: ${{fromJson(needs.preparation-tests.outputs.matrix).test_group}}

• Running tests using the TestNg group method with Docker

- name: Run tests with test group - ${{ matrix.test-group }}
  shell: bash
  run: |
    docker run --rm -u 0\
    --network=host \
    -v $(pwd):/tests -w /tests \
    -Dah.max_retries=2 -Dah_hub_port=4444 -DmaxParallelForks=1 \
    -DincludeGroups=${{ matrix.test-group }} -s clean remoteTest

There are also some drawbacks to this approach:

• We need to manually add a specific subgroup name to each test. Otherwise, the tests will not run and we will not know about them.
• We need to declare JSON with the list of subgroups and maintain it.
• Manual test load balancing. We need to consider how large the tests are and how to split them properly into subgroups. If we have 15 tests, should we split them into 3 subgroups like documents_1, documents_2, documents_3 for 5 tests each? Or maybe there are some tests that are much longer than others.

Performance: ~9 min

Debugging: Medium (need to rerun specific subgroups with tests)

Summary

There are no perfect ways to make parallel testing in GitHub Actions, as it greatly depends on individual project requirements, the specifics of the current infrastructure setup, security considerations, and budgetary constraints. This variability means that each project may require a unique approach to effectively implement parallel testing.

Modern automation frameworks like Playwright and Cypress are addressing these varied needs. Playwright lets developers run multiple tests at once, which makes everything faster. Cypress has a special feature that spreads out tests evenly across different machines so that no single machine is overwhelmed.

Despite the growing popularity of Cypress and Playwright, Selenium remains a top used in the world of test automation frameworks. Also it's important to remember how important it is to get quick feedback on the state of our software. Therefore, we should aim to set up Selenium tests to run in parallel to get feedback as soon as possible. Fortunately, with GitHub Actions, there's a way to do this using matrix strategy, which allows us to run Selenium tests in parallel, efficiently addressing the need for fast testing and reliable results.