Case study of Applitools or not only Cypress cross-browser testing

The world of testing is moving further and deeper towards automated tests. Naturally more and more new tools and approaches are emerging. To keep up with innovation and at the same time understand which of the approaches or tools is best suited precisely in this situation, you need to constantly experiment, try new applications, libraries, and technologies.

As a QA engineer in automation field, I understand that most functional testing types could and should be automated. But what about non-functional? UI testing, pixel-perfect, usability testing - could be also automated. Surprised? Then this article is for you, because it’s a story about automated testing of graphic user interface, including pixel-perfect and cross-browser Cypress testing, using Applitools.

According to official documentation, Applitools is a multi-functional application which provides a lot of interesting features for test engineers, developers, product owners, managers, etc. But we wanted to implement just a few of the provided features to our ngx-bootstrap library, specifically:

Screenshot comparisons using artificial intelligence;
Applitools dashboard for managing snapshot tests;
Cross-browser and cross-device testing for already implemented Cypress tests;

All tasks above are inter-dependent on each other and therefore should be done at one time. On a side note: we're implying that Cypress is already installed and that all functionality is covered by e2e-Cypress tests. If you need help with the basic setup and configuration of Cypress, our advice would be to follow the official Getting Started article. To keep things straight, let’s move through all steps and setup the Applitools plugin for Cypress:

Basic setup

We already had a large scope of Cypress tests, so they just needed several additions:

Add the Applitools plugin to the cypress/plugins/index.js file by specifying eyes.cypress module:

require('@applitools/eyes.cypress')(module);

Import all possible Applitools commands to the cypress/support/index.js file:

import './commands'

NOTE: In the basic setup, you can do steps #1 and #2 by just running `npx eyes-setup` after installing the Eyes Cypress npm package.

‍

Add applitools eyes as an additional library to the dependencies and run npm i command in the terminal:

"@applitools/eyes.cypress": "3.4.2"

Add the appropriate command to the package.json in the src folder (we'll explain what it does, don't be scared):

"cy:run:snapshot": "APPLITOOLS_SHOW_LOGS=1 APPLITOOLS_CONCURRENCY=100 cypress run --config integrationFolder=cypress/snapshot"

APPLITOOLS_SHOW_LOGS=1 - will throw additional logs to the console (could be helpful when you need to investigate the reason for failures locally and on the CI)
APPLITOOLS_CONCURRENCY=100 - specifies the amount of available concurrent sessions in your price plan. Note: Default value is 0
--config integrationFolder=cypress/snapshot - makes Cypress run only snapshot tests in this scope.

If you want to run Cypress tests without Applitools and you don’t want to get any warnings from Applitools such as in the image below, just add the APPLITOOLS_CONCURRENCY=100 parameter to the appropriate Cypress command.

Important notice: the Applitools visual tests are currently running with a concurrency value of 1. This means that the visual tests don't run in parallel, and therefore are slower. This is the default behavior for free accounts.
If your account does support a higher level of concurrency, it's possible to pass a different value by specifying `concurrency:X` in the applitools.config.js file.
For more information on how to configure the concurrency level, visit the following link: https://www.npmjs.com/package/@applitools/eyes.cypress#concurrency.
If you are interested in speeding up your visual tests, contact [email protected] to get a trial account and a higher level of concurrency.

Write your test for visual testing

To begin with, let's look how the fully functional test looks like:

import { DatepickerPo } from '../support/datepicker.po';
import { DropdownsPo } from '../support/dropdowns.po';
import { ModalsPo } from '../support/modals.po';
import { TabsPo } from '../support/tabs.po';
import { TypeaheadPo } from '../support/typeahead.po';describe('Snapshot test', () => {
const componentsArray = [
 new DatepickerPo(),
 new DropdownsPo(),
 new ModalsPo(),
 new TypeaheadPo(),
 new TabsPo()
];componentsArray.forEach(page => {‍
 it(`navigate to each Demo and check example: ${page.pageUrl}`, () => {
  page.navigateTo();
  cy.get('ng-sample-box').each(demo => {
   const subtitle = demo.parent().find('h3').text();   cy.wrap(demo).find(`.bd-example`)
    .eyesOpen({
     appName: 'NGX-bootstrap',
     concurrency: 5,
     matchLevel: 'Strict',
     testName: `${page.pageUrl} - ${subtitle}`,
     browser: [{
      name: 'chrome',
      width: 360,
      height: 640
     }, {
      name: 'firefox',
      width: 360,
      height: 640
     },
      {
      name: 'firefox',
      width: 1366,
      height: 768
     }]
    })
    .eyesCheckWindow({
     sizeMode: 'selector',
     selector: `.bd-example`,
     tag: `${page.pageUrl}-${subtitle}`,
     sendDom: false,
    })
    .eyesClose();
  });
 });
});
});

‍

Let’s analyze it step by step, because it's a lot of code to grasp in one sitting. Firstly, we need to declare an array of objects. Each Page Object has its own unique URL, which we’ll use for navigation and the set of methods we’ll use in our test.

const componentsArray = [
new DatepickerPo(),
new DropdownsPo(),
new TabsPo()
];

Take each object using forEach loop

componentsArray.forEach(page =>

Invoke URL name, which will make a descriptive test name and tell you what the name of the current testing page is

it('navigate to each Demo and check example: ${page.pageUrl}', () => {

Navigate to a page to start testing:

page.navigateTo();

Method navigateTo() is implemented using cy.visit command, which will take our pageUrl, add it to baseUrl and form an appropriate URL to navigate.

navigateTo() {
const bsVersionRoute = Cypress.env('bsVersion') ? `?_bsVersion=bs${Cypress.env('bsVersion')}` : '';
cy.visit(`${ this.pageUrl }${bsVersionRoute}`);
}

Ok, but what is bsVersionRoute doing there? This is just a specification of the testing process for the ngx-bootstrap library, due to the extended range of supported Bootstrap versions.

bsVersionRoute parameter will tell Cypress which bootstrap version should be tested:

Bootstrap 4 version (default one)
Bootstrap 3 version

Find each demo snippet, which consists of the demo component itself and 2 tabs with additional info:

cy.get('ng-sample-box').each(demo => {

Now we can define the sub-title of each Demo, which we’ll test. Let’s turn to the parent of the component that we've found and find 'h3' header there.

const subtitle = demo.parent().find('h3').text();

The ng-sample-box component contains template and component source. In our test, we don’t need any information from tabs, so, we’ll work solely with the .bd-example class

cy.wrap(demo).find(`.bd-example`)

Oooh! Now we can open Applitools eyes and begin "staring" at our demo. Remember we said that these tasks are inter-dependent at the beginning of an article? Here we can also define settings for our next task, in which we want to see the results on the Applitools Dashboard:

eyesOpen({
appName: 'NGX-bootstrap',
matchLevel: 'Strict',
testName: `${page.pageUrl} - ${subtitle}`,

1. appName as the name suggests, is an Application Name that'll be shown on the Dashboard in 'Apps & Tests' menu. And all our test results will be grouped together according to this parameter.

2. matchLevel parameter sets the level of image comparison. The default match level is 'Strict', if you would like to use a different match level, you should specify it right here.

Some other comparison levels:

Exact - pixel to pixel comparison;
Strict - comparison for everything, what the user can visually detect, including content and color;
Content - same as Strict, but without color comparison;
Layout - compares elements structure

If that's not enough, you can get more information about the matchLevel parameter at official Applitoos blog.

3. testName parameter helps you understand which tests failed and what functionality should be rechecked/fixed. This info will be shown in the Applitools Dashboard according to our configuration:

Also let's not forget about cross-browser testing, so don't forget to set the browsers:

browser: [{
name: 'chrome',
width: 360,
height: 640
}, {
name: 'firefox',
width: 1366,
height: 768
}]

Just to be clear, as of now, the Applitools plugin provides the possibility to test in Firefox, Chrome, IE, Edge, but in nearest future Safari and a lot of other browsers are going to be supported. But in the meantime you can already set different screen resolutions using width and height settings.

Now, when we're almost done, let Applitools know that we're ready to start testing:

.eyesCheckWindow({
sizeMode: 'selector',
selector: `.bd-example`,
tag: `${page.pageUrl}-${subtitle}`,
sendDom: false,
})

sizeMode parameter defines which part of our window will be checked. List of available options: full-page, viewport, selector, region. The official description of each option can be found in the appropriate section of official documentation.
selector parameter will work only if you set the sizeMode: 'selector'. The value should be a valid CSS-selector or XPath to an appropriate component/block/part which you want to test.
tag parameter grouping test results by tag in the dashboard.

Once our tests are finished, we need to tell Applitools about it:

.eyesClose();

Run our test and observe the results

APPLITOOLS_API_KEY=myKey npm run cy:run:snapshot

Applitools Dashboard will show you the baseline images (which will be taken at first test run and will be considered as a reference result) and images from the current test run.

If there are some differences, then the comparison logic, based on AI, will detect changes and show them to you:

If we want to update our baseline images, we need to go to the Dashboard and accept the differences which were found during latest test run. This helps you keep the latest reference images versions up to date.

Summary

We’ve implemented cross-browser and cross-device testing for our demo application, using our default testing library (Cypress) with the additional Applitools plugin. Now we can be sure that each part of our UI works properly. Ok, but everything couldn't be THAT good and painless, could it? Let’s take a look at some of the pros and cons:

PROS

Easy to setup, short time for implementation;
AI-powered image comparison technology;
Wide spectrum of available settings for image comparison;
Possibility to manage reference images in the dashboard;
Great test result management: find them, group them, remove, etc.;
Responsive support team, whom took all our wishes into account.

CONS

Not as many browsers available for testing as we would like (BUT, while our article was passing through several levels of verification, the Applitools development team has already made support for IE and Edge and will soon have Safari as well)
Typescript isn't supported yet (Thanks to the Applitools development team, Typescript support is coming (probably in a few weeks). Check the list of tutorials, maybe it already exists)
From time to time error logs in the console aren't that descriptive

Afterwords

A big THANK YOU to Applitools and This Dot Labs teams for their ongoing support during our testing journey and for providing the possibility to test all this out. It was a blast!