March 25, 2019
Cypress testing: Running tests in parallel
Facing bugs is usual for web application development. If your team ignores tests or you do not test the code, issues in the project become evident and over time they can become a total blocker and eventually kill your project. Therefore, in many teams, there is a testing engineer who catches bugs and reports them to the team.
We, as a reliable software vendor, don’t want to undermine customer confidence in us, so we incorporated Cypress.io - a platform for automatic testing to find and correct errors on time.
Table of contents
-
Why we love Cypress
-
Parallelization task
-
Instruments, Source data, Implementation plan
-
Solution: Green phase
-
Solution: Yellow phase
-
Results
Why we love Cypress
There are many ways to test, yet in this article, we would like to limit ourselves to two of them - e2e and integration testing. The Cypress.io platform is excellent for testing web-application interfaces. Its main task is to check how the client part interacts with the server part, and the individual components of the page interact with each other.
Platform benefits:
-
Can be quickly built into the project.
-
Supports all levels of tests (e2e, integration, unit).
-
Takes snapshots for each test step for easy debugging.
-
Able to access every web-page object in the DOM.
-
When expecting any condition, Cypress tries to check it a few times, before sending failures after the 1st try.
-
Able to work with our favorite continuous integration system (TravisCI).
Cypress works on a predetermined scenario. Even if you haven’t written tests before, clear documentation will help you start.
Parallel testing
Now to the most exciting part - parallelization. Here’s a brief extract from the official documentation explaining what parallelization is and why it is needed:
If your project has a large number of tests, it can take a long time for tests to complete running serially on one machine. Running tests in parallel across many virtual machines can save your team time and money when running tests in Continuous Integration (CI).
As we said earlier, Cypress has great docset, even with examples, and parallelization is not an exception: official documentation provides you with step by step description about the parallel run. However, because of our specific basic project data, which includes many instances, types of tests, and scopes, implementation has been a tough process for us. Let’s move through the main steps and issues, which we’ve faced, and find a solution for each of them :)
Goal: Decrease time to run e2e tests, speed up CI builds.
Instruments, Source data, Implementation plan
Instruments:
-
TypeScript for test code
-
cypress/webpack-preprocessor - a plugin for compiling
-
ts-loader - TypeScript loader for Webpack
-
Travis - CI tool.
Source data:
There are two scopes of e2e automated tests in the project:
-
Integration (about 70 tests)
-
Full (about 450 tests at the time of this writing).
There are also a few instances, which should be tested by e2e tests:
Parallelization implementation plan:
-
Integration of e2e tests running for each PR (pull request) for the local build
-
Integration and full e2e tests running for each merge to
developmentbranch at automatically deployed build into two different instances (two different URLs) -
Integration and full e2e tests running for each new release for a production instance (the third URL).
This plan can be visually presented as a timeline:
We’re going to describe parallelization implementation in two phases: green and yellow.
Solution: Green phase
The green phase consists of two steps, so there don’t seem to be any complications:
-
Add a few extra settings to the cypress.json:
"video": false, (for quick run) "projectId": "yourID", (for link CY dashboard service to each run)
-
Make a command for CI run with additional parameters:
your command — --record --key $CY_KEY --parallel
A-and that’s it, these are the whole settings to be added (according to the documentation). However, in real life… in real life, we got a huge travis.yml and a little misbehavior.
Before parallelization, our Travis configuration looked like this:
- script:
ng serve --prod & $(npm bin)/wait-on http-get://$URL
npm run cy:run:smoke
kill $(jobs -p) || true
name: "Run cypress smoke testing"
env: URL=localhost:4200/#
To be clear, this script builds project locally, waits until the app is up and running, runs tests, and finally kills all processes when they are finished. For these needs you can also use the start-server-and-test library.
After adding parameters for parallelization, our Travis config looks like:
In this case, the number of our threads will be three. So we need to have three machines on Travis for running all these tests. When our CI run finishes, we’ll see list of scripts which we’ve ran and amount of time spent by each Travis machine during start, test run, and kill:
Time, which is displayed on Travis, was spent on starting the machine, installing development packages, building the library and Demo application, and running test.
It looks like there’s no problem at all, but on the CI Dashboard we’ll see that only two machines took part in the test run. The Dashboard also shows us time, which was spent for the direct test run: 2:26 minutes.
If we dive deeper into the job on the third Travis machine, we’ll see that the job actually has started, but not in time:
What happened?
Travis machines are ready to run the tests, but not simultaneously.
The first machine can start at 01:00:00, the second one at 01:02:00, and the third one will start at 01:05:00. And if your test run takes less than five minutes, then the third thread will be empty. It didn’t even get to run one little test :"(
Solution: Yellow phase
At this stage, all we have to do is to add similar commands and configuration for other instances. Because of the similarities of the test scope, we just set an additional parameter for changing base URL, and add a new function for running tests which we’ll reuse in each stage:
testPostDeploy: &testPostDeploy script: CYPRESS_baseUrl=$BASE_URL npm run cy:run:all -- --record --key $CY_KEY --parallel --group $GROUP_NAME
This is a very common function, which is applied to the next parameters:
-
CYPRESS_baseUrl - one of our instances from the matrix above
-
--record - sends results to the Dashboard
-
--key - determines which account from the dashboard we will use
-
--parallel - runs our tests in parallel
-
--group - combines tests in different groups.
In each script we need to define BASE_URL and GROUP_NAME parameters, as other parameters are defined globally already.
And you may be surprised with following command:
npm run cy:run:all
This command unfolded means the following:
cy:run:all": "cypress run --config integrationFolder=cypress"
"Why surprised?" you ask, okay, look:
Because each test scope lives in a separate directory with the appropriate name: 'integration' and 'full.' To run these tests altogether, just set the global directory for the run. Also, make sure to exclude all unnecessary directories and files from this run (using "ignoreTestFiles"parameter in cypress.json), or, otherwise, you could get an error like: "The tests were unable to run: Oops…we found an error preparing this test file: …"
Instead of specifying the folder, you could use --specs parameter with a global pattern, like:
cypress run --config integrationFolder=cypress --spec '**/*_spec.ts
Our configuration for the first instance looks like this now:
name: "Cypress suit run on SSR 1thread" env: - GROUP_NAME=3x-electron - BASE_URL=https://ngx-universal.herokuapp.com/#/ <<: *testPostDeploy - script: name: "Cypress suit run on SSR 2thread" env: - GROUP_NAME=3x-electron - BASE_URL=https://ngx-universal.herokuapp.com/#/ <<: *testPostDeploy - script: name: "Cypress suit run on SSR 3thread" env: - GROUP_NAME=3x-electron - BASE_URL=https://ngx-universal.herokuapp.com/#/ <<: *testPostDeploy
Just to remind you, Travis stages run consistently, and Travis jobs inside one stage run in parallel. This provides the possibility to run tests on different machines inside the first stage. You can add as many scripts as you need, and each of the scripts will run in their separate thread.
In our example above, integration and the full test scope will run in three threads.
To finish what we have started, we need to add a similar configuration for the two remaining instances.
Okay, imagine that’s done, But what do we have now? Of course, an error:
"The run you are attempting to access is already complete and will not accept new groups."
What is the reason for this mess?
Cypress uses ci-build-id and groups flags for grouping our tests into test runs. So you need to set unique build-ids and unique groups inside each Travis stage.
To resolve this inconvenience, set different values for --group flag and --ci-build-id flag inside each Travis stage. As a result, our function should look similar to the following:
testPostDeploy: &testPostDeploy script: CYPRESS_baseUrl=$BASE_URL npm run cy:run:all -- --record --key $CY_KEY --parallel --group $GROUP_NAME --ci-build-id PostDeploy-$GROUP_NAME-$TRAVIS_BUILD_ID
Another way to solve this issue is to set a "run completion delay" - the time that the run waits for new groups before completing (in seconds). All you need is to go to the Dashboard => Project => Settings and set this time into "Parallelization"section:
However, when we thought that we were done, an unpredictable phase of stabilization emerges.
In our case, when the number of tests reached 400+, we were greeted with a "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory" issue.
To deal with this issue, we’ve checked Cypress GitHub bug tracker and found a solution there! According to provided information, we need to add an extra option for ts-loader, like so:
{
loader: 'ts-loader',
options: {
transpileOnly: true
}
}
And magic happens.. It a.c.t.u.a.l.l.y works, phew!
According to our results from the following screenshot, our 522 tests took about 10 minutes. Note that the biggest specs were run first and the smallest were latest.
Results
While we were implementing parallelization, the number of our tests increased several times. However, let’s count the average time (in minutes) that it takes to run them (this statistic actual for the period from February to March of 2019).
These are just words. Where is the proof?
Useful information
Parallelization example provided officially by CY.
Thank you words
The biggest kudos go to the Cypress team for supplying us with a free plan to try out the Cypress Parallelization feature for our ngx-bootstrap open-source library and a huge thanks to everyone who contributed!
