Running Cypress tests with nested parallelization and GitHub Actions

Running Cypress tests in parallel with GitHub Actions

At some point, as our test suite grows larger, we will face the issue of our tests taking longer to execute. This problem can affect suites written in various frameworks to different degrees. This issue can be particularly relevant for Cypress, which does not perform very well in benchmarks related to the speed of test execution.

A natural reaction at this point might be to switch to a different technology and rewrite the tests to improve their speed. However, it's important to remember that speed is just one of the factors for which a particular framework is chosen, among others, like ease of debugging, stability, and community support.

Nevertheless, speed is one of the key elements in test execution. Every time we run tests, we would like to receive feedback about the state of our software as quickly as possible. Therefore, I would like to share one of the ways we managed to reduce execution time of Cypress tests.

There are plenty of ways to run Cypress tests in parallel:

Most of these methods are quite simple to configure. Typically, it's enough to use the appropriate actions within GitHub Actions and specify the number of workers that should run concurrently, just like in cypress-split.

Problems arise when you cannot use simple solutions in your current environment, such as on a GitHub Actions worker where you may have restrictions on installing external dependencies or passing secrets to Cypress, or when you need to use Docker to run Cypress.

In such cases, you can consider using a custom solution tailored to your specific environment constraints.

The method I am going to show you consists of combining the two methods mentioned above: manual splitting and the usage of the package cypress-parallel npm package. This approach is totally free.

GitHub Actions parallel workflow

The approach

Let's begin by outlining the structure of our approach step by step:

By organizing the tests into main groups and subgroups, we can achieve better test management and parallelization.

Division into main groups

First, we split our Cypress tests into groups based on the main areas of testing. This can be divided based on our applications or main functionalities.

For example, if our application is an online store, the main testing areas would be:

Therefore, in the folder where Cypress is located, we create directories corresponding to our main areas. Example structure of directories:

- registration
- login
- cart
- product_page

In the next step, you can divide your test structure even more granularly. The decision depends on how many automated tests you have and how complex your application's structure is. For example, you might want to divide the registration test group into smaller subgroups based on the affiliate partner or the source from which the user comes.

You can also skip this step and move to the step with the GitHub Actions workflow.

Split into subgroups

We divide the main testing groups into subgroups of testing. For registration, this could be subgroups like:

Similar to the first step, we create new directories. In each subgroup, there are tests verifying the operation of the application/components. At this stage, the structure of our tests could look like this:

- registration
	- registration_redirected
		- registration_partner_a.cy.js
		- registration_partner_b.cy.js
	- registration_direct
		- registration_partner_a.cy.js
		- registration_partner_b.cy.js
	- registration_other
		- registration_with_partner.cy.js
		- registration_without_partner.cy.js

This approach has its advantages and disadvantages. In this case, we are limited by the naming structure, which can significantly affect the execution time of tests. Why? Because here we have two factors that influence the final execution time of a group of tests:

For example, one group of tests may have more tests that take longer than others. For instance:

- registration
	- registration_redirected
		- registration_partner_a.cy.js (~8500 ms)
		- registration_partner_b.cy.js (~7700 ms)
		- registration_partner_c.cy.js (~9000 ms)
		- registration_partner_d.cy.js (~7800 ms)
		- registration_partner_e.cy.js (~8900 ms)
	- registration_direct
		- registration_partner_a.cy.js (~11000 ms)
	- registration_other
		- registration_with_partner.cy.js (~14000 ms)
		- registration_without_partner.cy.js (~15000 ms)

The test execution times for the 'registration_redirected' group will be significantly longer compared to those for the 'registration_direct' group. This discrepancy will impact the overall duration of the test suite. It's important to note that the total execution time for a test suite, such as 'registration', is determined by the group with the longest execution time, which in this case is 'registration_redirected'.

To mitigate this issue, consider reorganizing the tests into groups with more generalized names, as this approach can help distribute the tests more evenly and potentially reduce the total execution time.

- registration
	- group_1 (total execution time: ~23,500 ms)
		- registration_redirected_partner_a.cy.js (~8500 ms)
		- registration_other_without_partner.cy.js (~15000 ms)
	- group_2 (total execution time: ~30,600 ms)
		- registration_redirected_partner_b.cy.js (~7700 ms)
		- registration_redirected_partner_e.cy.js (~8900 ms)
		- registration_other_with_partner.cy.js (~14000 ms)
	- group_3 (total execution time: ~27,800 ms)
		- registration_redirected_partner_d.cy.js (~7800 ms)
		- registration_redirected_partner_c.cy.js (~9000 ms)
		- registration_direct_partner_a.cy.js (~11000 ms)

Thanks to such reorganization, you can influence the final execution time of a given group. This will ensure that the test groups run in approximately the same amount of time.

At first glance, this may seem labor-intensive as it requires manually sorting tests based on their execution time. However, this requires effort only at the very beginning of the journey – when creating the tests structure. From my own experience, I have noticed that the execution time of tests over the application's development lifecycle tends to not change a lot (although there are some exceptions). Manual rebalancing can also be done when adding new tests.

GitHub Actions workflow

The next step is to prepare a GitHub Actions workflow that will execute tests from each subgroup (e.g., registration_direct, registration_redirect) on separate workers. As a result, this workflow will run concurrent workers depending on the number of subdirectories (in this case 3 subdirectories). We will run our Cypress through docker with standard Cypress command:

cypress run --spec "cypress/e2e/main_group/sub_group/**"

But in our case, we need to run 3 separate workers to execute the commands:

What would such a workflow look like?

First, we want to run this workflow manually with workflow_dispatch:

workflow_dispatch:
  inputs:
    include_groups:
      description: "Allows to specify which group of test to run"
      type: choice
      required: true
      default: "registration"
      options:
        - registration
        - add_to_cart
        - login
        - purchase

On this option, we will choose which main group we want to run.

Next, we want to prepare a JSON output of subgroups (like registration_redirect). For this, we need to have the preparation_job.

preparation-tests:
  runs-on: [self-hosted, utils]
  steps:
    - name: Checkout tests repo
      uses: actions/checkout@v3

    - name: List test groups
      id: set-matrix
      run: |
        echo "matrix=$(find ./cypress/cypress/e2e/${{ inputs.include_groups }} -mindepth 1 -type d | awk -F'/' '{print $NF}' | jq -R . | jq -s '{ directories: . }' | tr -d '[:space:]'))" >> $GITHUB_OUTPUT
    - name: Get tests
      run: echo "The selected tests are ${{ steps.set-matrix.outputs.matrix }}"
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}

What does this part do?

{
  "directories": [
    "registration_redirected",
    "registration_direct",
    "registration_other"
  ]
}

In the next step, we need to define Cypress multi_jobs and matrix strategy which will use groups from the previous step.

cypress_multi_jobs:
  runs-on: [self-hosted, tests]
  timeout-minutes: 25
  needs: [preparation-tests]
  strategy:
    fail-fast: false
    matrix:
      directories: ${{fromJson(needs.preparation-tests.outputs.matrix).directories}}

and then:

- name: Run Cypress
        shell: bash
        run: |
          docker build . --file Dockerfile-cypress -t your-project-cypress:latest
          docker run --rm -w /e2e your-project-cypress:latest cypress run --spec "cypress/e2e/${{ inputs.include_groups }}/${{ matrix.directories }}/**"

At this stage, you should already have tests that will be executed in parallel. Depending on the number of workers, you will now be able to significantly reduce the test execution time.

Run Cypress tests on GitHub Actions - manual split
Run Cypress tests on GitHub Actions - manual split

Moreover, this approach unlocks one great advantage – in case of failing tests in a group, you will not have to rerun the whole suite but only a specific subgroup of tests.

Package cypress-parallel + GitHub Actions parallel workflow

To take it a step further within subgroups, we can utilize a tool called cypress-parallel. This tool enables us to define and execute tests in additional threads. For instance, we can specify that within a single worker, cypress-parallel will run tests in 3 separate threads. As a result, by combining it with the GitHub Actions parallel workflow, we can have 3 workers operating with 3 separate threads, allowing us to execute 9 tests simultaneously.

At this point, some questions might arise:

To install cypress-parallel, we just need to run the following command:

npm i cypress-parallel

and then in package.json, you need to specify the scripts:

"cy:run": "cypress run",

"cy:parallel": "node_modules/.bin/cypress-parallel -s cy:run -t 3 -d",

"cy:parallel:path": "npm run cy:parallel --spec"

After this, you can update the step in the GitHub Actions workflow:

name: Run Cypress
        shell: bash
        run: |
          docker build . --file Dockerfile-cypress -t your-project-cypress:latest
          docker run --rm -w /e2e your-project-cypress:latest npm run cy:parallel --spec "cypress/e2e/${{ inputs.include_groups }}/${{ matrix.directories }}/**"

You should be able to run tests in parallel using GitHub Actions and the cypress-parallel package. By implementing nested parallelization with both manual splitting and the cypress-parallel package, the execution time of the test suite, originally around 11 minutes, can be significantly reduced.

Run Cypress tests on GitHub Actions - cypress parallel
Run Cypress tests on GitHub Actions - cypress-parallel

I hope you found value in my article on enhancing your testing process by leveraging nested parallelization with Cypress and GitHub Actions. Let's take your testing to the next level by implementing these strategies for faster and more efficient workflows.

Mateusz Banasik's Profile

Mateusz Banasik

Quality Assurance Engineer