Cypress Parallelization — Case Study + Results

March 25, 2019

All programmers are faced with bugs. If your team ignores tests or you personally do not test the code, issues in the project become evident and over time they could become a total blocker and eventually kill your project. Therefore, in many teams there is a tester who catches bugs and report them to the team. We, as a company, didn’t want to undermine customer confidence in us, and so we incorporated Cypress.io, a platform for automatic testing in order to find and correct errors in a timely manner.

Table of contents
  1. A few words about why we love Cypress
  2. Parallelization task
  3. Instruments, Source data, Implementation plan
  4. Solution: Green phase
  5. Solution: Yellow phase
  6. Results

A few words about why we love Cypress

There are many ways to test, but in this article, we would like to limit ourselves to two of them - e2e and integration.
For testing web application interfaces, the Cypress.io platform is excellent. Its main task is to check how the client part interacts with the server part and the individual components of the page with each other.

Platform benefits:

  • Easily built into the project;
  • Support all levels of tests (e2e, integration, unit)
  • Take snapshots for each test step for easy debugging ;
  • Able to access every web-page object in the DOM;
  • When expecting any condition, Cypress will try to check it a few times, before sending failures after the first try
  • We were able to use our favourite continuous integration system (TravisCI).

Cypress works on a predetermined scenario. Even if you did not have to write tests before, you will be able to figure it out thanks to clear documentation.

Parallelization

Now to the most interesting part — parallelization. Here is a brief extract from the official documentation explaining what parallelization is and why it is needed:

If your project has a large number of tests, it can take a long time for tests to complete running serially on one machine. Running tests in parallel across many virtual machines can save your team time and money when running tests in Continuous Integration (CI)

As we said earlier, Cypress has great docset, even with examples, and parallelization is not an exception, official documentation provides you with step by step description about the parallel run.
But because of our specific basic project data, which includes a lot of instances, types of tests and scopes, implementation was a tough process. Let’s move through the main steps and issues, which we’ve faced, and find a solution for each of them :)

Goal: Decrease time to run e2e tests, speedup CI builds

Instruments:

Source data:

There are two scopes of e2e auto-tests in the project:

  • Integration (~70 tests)
  • Full (~450 tests at the moment, when the article was started to write)

There are also a few instances, which should be tested by e2e tests:

Parallelization implementation plan:

  • Integration e2e tests running for each PR for the local build
  • In addition, Integration + full e2e tests running for each merge to `development` branch at automatically deployed build into 2 different instances (2 diff URLs)
  • In addition, Integration + full e2e tests running for each new release for a production instance (3d URL)

This plan could be visually presented as a timeline. And will be implementing it in 2 phases: green and yellow (see image below).

Solution: Green phase

Green phase consists of two steps, so there shouldn’t be any complications:

  1. Firstly, we need to add a few additional settings to the cypress.json:
"video": false, (for quick run)
"projectId": "yourID", (for link CY dashboard service to each run)

  1. Make a command for CI run with additional params:
- your command -- --record --key $CY_KEY --parallel

Aa-and that's it, that's the whole settings which needed to be added (according to the documentation). But in real life.. in real life, we got a huge travis.yml and a little misbehaviour.
Our Travis configuration before parallelization looked like:

- script:
ng serve --prod & $(npm bin)/wait-on http-get://$URL    
npm run cy:run:smoke 			 
kill $(jobs -p) || true
name: "Run cypress smoke testing"
env: URL=localhost:4200/#

To be clear, this script will: build project locally, wait until the app is up and running, run tests and finally kill all process when they finished. For these needs you can also use start-server-and-test library.

After adding parameters for parallelization, our travis config looks like:

In this case, the number of our threads will be 3. And we need to have 3 machines on Travis for running all these tests. When our CI run will finish, we’ll see list of scripts which we’ve ran and amount of time spent by each Travis machine during start, test run and kill:

Time, which displayed on Travis was spent on: start machine, installing development packages, building library and Demo application and test run.

Looks like there's no problem at all, but on the CY dashboard we’ll see, that only 2 machines took part in the test run. Dashboard also show us time, which was spent for the direct test run: 2:26 minutes.

If we'll deeply dive to the job on the 3d Travis machine, we’ll see that the job actually started, but not in time:

What did just happened?
Travis machines are ready to run the tests, but not simultaneously.
The first machine can start at 01:00:00, the second one at 01:02:00 and the third will start at 01:05:00. And if your test run takes less than 5 minutes, then the third thread will be empty. It didn’t even get to run 1 little test :”(  

Solution: Yellow phase

At this stage all we have to do is to add similar commands and configuration for other instances. Because of the similarities of the test scope, we just set an additional parameter for changing base URL, and add a new function for running tests which we’ll reuse in each stage:

testPostDeploy: &testPostDeploy
script: CYPRESS_baseUrl=$BASE_URL npm run cy:run:all -- --record --key $CY_KEY --parallel --group $GROUP_NAME

This is a very common function, which applies the next parameters:

  • CYPRESS_baseUrl — one of our instances from the matrix above;
  • --record — send results to the Dashboard;
  • --key — determine which account from the dashboard we will use;
  • --parallel — runs our tests in parallel
  • --group — combines tests in different groups

In each script we need to define BASE_URL and GROUP_NAME parameters, as other parameters are defined globally already.
And you may be surprised with following command:

‍npm run cy:run:all

This command unfolded means the following:
cy:run:all": "cypress run --config integrationFolder=cypress"

“Why surprised?” you ask, ok, look:
Because each test scope lives in a separate directory with the appropriate name, i.e  ‘integration’ and ‘full’. To run these tests all together, we just set the global directory for the run, and make sure that we’ve excluded all unnecessary directories and files from this run (using "ignoreTestFiles" param in cypress.json), or otherwise, you could get an error like: “The tests were unable to run:  Oops...we found an error preparing this test file: ...”
Instead of specifying folder, you could use --specs parameter with glob pattern, like:

cypress run --config integrationFolder=cypress --spec '**/*_spec.ts

Our configuration for the first instance looks like this now:

name: "Cypress suit run on SSR 1thread"
env:
- GROUP_NAME=3x-electron
- BASE_URL=https://ngx-universal.herokuapp.com/#/
<<: *testPostDeploy
- script:
name: "Cypress suit run on SSR 2thread"
env:
- GROUP_NAME=3x-electron
- BASE_URL=https://ngx-universal.herokuapp.com/#/
<<: *testPostDeploy
- script:
name: "Cypress suit run on SSR 3thread"
env:
- GROUP_NAME=3x-electron
- BASE_URL=https://ngx-universal.herokuapp.com/#/
<<: *testPostDeploy

Just to remind you, Travis stages runs consistently, and Travis jobs inside 1 stage run in parallel. And this provides you with a possibility to run tests in different machines inside first stage. You can add as many scripts as you need, and each of the scripts will run in their own separate thread.
In our example above, integration and the full test scope will be run in 3 threads.
To finish what we have started, we need to add similar configuration for the two remaining instances.
Ok, imagine that’s done, but what do we have now? Of course an error:

The run you are attempting to access is already complete and will not accept new groups.

What is the reason of this mess?
Cypress uses ci-build-id and groups flags for grouping our tests into test runs. And this means that you need to set unique build-ids and unique groups inside each Travis stage.
To resolve this inconvenience, set different values for --group flag and --ci-build-id flag inside each Travis stage. As a result, our function should look something closer to the following:

testPostDeploy: &testPostDeploy
script: CYPRESS_baseUrl=$BASE_URL npm run cy:run:all -- --record --key $CY_KEY --parallel --group $GROUP_NAME --ci-build-id PostDeploy-$GROUP_NAME-$TRAVIS_BUILD_ID

Another way to solve this issue - is to set a "run completion delay". This is the time that the run will wait for new groups before completing (in seconds). All you need - is to go to the Dashboard => Project => Settings and set this time into "Parallelization" section:

And when we thought that we were done, an unpredictable phase of stabilization emerges
In our case, when the number of tests reached 400+, we were greeted with a “FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory” issue.

To deal with this issue, we went to Cypress GitHub bug tracker and found a solution there! According to provided information, we need to add an additional option for ts-loader, like so:

{
loader: 'ts-loader',
options: {
transpileOnly: true
}
}

And magic happens.. It a.c.t.u.a.l.l.y works, phew!

According to our results from the screenshot under, our 522 tests took about 10 minutes. And pay attention, that the biggest specs were run first and the smallest were latest.

Results

While we were implementing parallelization, the number of our tests increased several times. But let’s count the average time (in minutes) that it takes to run them (this statistic actual for the period of time from February to March of 2019).

These are just words. Where is the proof?

Useful information
*Parallelization example, provided officially by CY

Thank you words
The biggest kudos goes to the Cypress team for supplying us with a free plan to try out the Cypress Parallelization feature for our ngx-bootstrap open source library and a huge thank you to everyone who contributed!

Subscribe to find out more

Thank you!

Your submission has been received!
Oops! Something went wrong while submitting the form.

More articles

Want to work with us?

Let's discuss how Valor Software can help with your development needs!