The world of testing is moving further and deeper towards automated tests. Naturally, more and more new tools and approaches are emerging. To keep up with innovation and at the same time understand which of the approaches or tools are best suited precisely in this situation, you need to continually experiment, try new applications, libraries, and technologies.
As a testing engineer in the automation field, I understand that most functional testing types could and should be automated. But what about non-functional? UI testing, pixel-perfect, usability testing—can also be automated. Surprised? Then this article is for you, because it’s a story about automated testing of the graphic user interface, including pixel-perfect and cross-browser Cypress testing, using Applitools.
According to official documentation, Applitools is a multi-functional application that provides a lot of interesting features for test engineers, developers, product owners, managers, etc. We wanted to implement just a few of the provided features to our ngx-bootstrap library, specifically:
All tasks above are interdependent on each other and therefore should be done at one time. On a side note: we're implying that Cypress is already installed and that all functionality is covered by e2e-Cypress tests. If you need help with the basic setup and configuration of Cypress, we advice to follow the official Getting Started article. To keep things straight, let’s move through all steps and set up the Applitools plugin for Cypress:
We already had a large scope of Cypress tests, so they just needed several additions:
require('@applitools/eyes.cypress')(module);
import './commands'
Note In the basic setup, you can do steps #1 and #2 by just running `npx eyes-setup` after installing the Eyes Cypress npm package.
"@applitools/eyes.cypress": "3.4.2"
"cy:run:snapshot": "APPLITOOLS_SHOW_LOGS=1 APPLITOOLS_CONCURRENCY=100 cypress run --config integrationFolder=cypress/snapshot"
If you want to run Cypress tests without Applitools and you don’t want to get any warnings from Applitools such as in the image below, add the APPLITOOLS_CONCURRENCY=100 parameter to the appropriate Cypress command.
Important notice: the default behavior for free accounts is that the Applitools visual tests are run with a concurrency value of 1. This means that the visual tests don't run in parallel, and therefore are slower.
If your account does support a higher level of concurrency, it's possible to pass a different value by specifying `concurrency:X` in the applitools.config.js file.
Check more information on how to configure the concurrency level.
If you are interested in speeding up your visual tests, contact [email protected] to get a trial account and a higher level of concurrency.
To begin with, let's look how the fully functional test looks like:
import { DatepickerPo } from '../support/datepicker.po';
import { DropdownsPo } from '../support/dropdowns.po';
import { ModalsPo } from '../support/modals.po';
import { TabsPo } from '../support/tabs.po';
import { TypeaheadPo } from '../support/typeahead.po';describe('Snapshot test', () => {
const componentsArray = [
new DatepickerPo(),
new DropdownsPo(),
new ModalsPo(),
new TypeaheadPo(),
new TabsPo()
];componentsArray.forEach(page => {
it(`navigate to each Demo and check example: ${page.pageUrl}`, () => {
page.navigateTo();
cy.get('ng-sample-box').each(demo => {
const subtitle = demo.parent().find('h3').text(); cy.wrap(demo).find(`.bd-example`)
.eyesOpen({
appName: 'NGX-bootstrap',
concurrency: 5,
matchLevel: 'Strict',
testName: `${page.pageUrl} - ${subtitle}`,
browser: [{
name: 'chrome',
width: 360,
height: 640
}, {
name: 'firefox',
width: 360,
height: 640
},
{
name: 'firefox',
width: 1366,
height: 768
}]
})
.eyesCheckWindow({
sizeMode: 'selector',
selector: `.bd-example`,
tag: `${page.pageUrl}-${subtitle}`,
sendDom: false,
})
.eyesClose();
});
});
});
});
Let's analyze it step by step because it's much code to grasp at one sitting.
Firstly, we need to declare an array of objects. Each Page Object has its unique URL for navigation and the set of methods for our test.
const componentsArray = [
new DatepickerPo(),
new DropdownsPo(),
new TabsPo()
];
Take each object using forEach loop.
componentsArray.forEach(page =>
Invoke URL name, which will make a descriptive test name and tell you what the name of the current testing page is.
it('navigate to each Demo and check example: ${page.pageUrl}', () => {
Navigate to a page to start testing:
page.navigateTo();
Method navigateTo() is implemented using cy.visit command, which will take our pageUrl, add it to baseUrl, and form an appropriate URL to navigate.
navigateTo() {
const bsVersionRoute = Cypress.env('bsVersion') ? `?_bsVersion=bs${Cypress.env('bsVersion')}` : '';
cy.visit(`${ this.pageUrl }${bsVersionRoute}`);
}
Okay, but what is bsVersionRoute doing there? This is just a specification of the testing process for the ngx-bootstrap library, due to the extended range of supported Bootstrap versions.
bsVersionRoute parameter will tell Cypress which bootstrap version should be tested:
Find each demo snippet, which consists of the demo component itself and two tabs with additional info:
cy.get('ng-sample-box').each(demo => {
Now we can define the sub-title of each Demo, which we’ll test. Let’s turn to the parent of the component that we've found and find 'h3' header there.
const subtitle = demo.parent().find('h3').text();
The ng-sample-box component contains a template and component source. In our test, we don’t need any information from tabs, so we'll work solely with the .bd-example class.
cy.wrap(demo).find(`.bd-example`)
Oooh! Now we can open Applitools eyes and begin "staring" at our demo. Remember we said that these tasks are interdependent at the beginning of an article? Here we can also define the settings for our next task, in which we want to see the results on the Applitools Dashboard:
eyesOpen({
appName: 'NGX-bootstrap',
matchLevel: 'Strict',
testName: `${page.pageUrl} - ${subtitle}`,
1. appName, as the name suggests, is an application name that'll be shown on the Dashboard in “Apps & Tests” menu. Also, we’ll group all our test results according to this parameter.
2. matchLevel parameter sets the level of image comparison. The default match level is “Strict.” If you would like to use a different match level, you should specify it right here.
Some other comparison levels:
If that's not enough, you can get more information about the matchLevel parameter at official Applitoos blog.
3. testName parameter helps you understand which tests failed and what functionality you should recheck or fix. You’ll see this info in the Applitools Dashboard according to our configuration:
Also, let's not forget about cross-browser testing, so be sure to indicate the browsers:
browser: [{
name: 'chrome',
width: 360,
height: 640
}, {
name: 'firefox',
width: 1366,
height: 768
}]
To be clear, as of now, the Applitools plugin provides the possibility to test in Firefox, Chrome, IE, Edge. In the nearest future, Safari and many other browsers are going to be supported. In the meantime, you can already set different screen resolutions using width and height settings.
Now, when we're almost done, let Applitools know that we're ready to start testing:
.eyesCheckWindow({
sizeMode: 'selector',
selector: `.bd-example`,
tag: `${page.pageUrl}-${subtitle}`,
sendDom: false,
})
Once our tests are finished, we need to tell Applitools about it:
.eyesClose();
APPLITOOLS_API_KEY=myKey npm run cy:run:snapshot
Applitools Dashboard shows the baseline images (which will be taken at the first test run and will be considered as a reference result) and images from the current test run.
If there are some differences, then the comparison logic, based on AI, will detect changes and show them to you:
If we want to update our baseline images, we need to go to the Dashboard and accept the differences which were found during the latest test run. This helps you keep the latest reference images versions up-to-date.
We’ve implemented cross-browser and cross-device testing for our demo application, using our default testing library (Cypress) with the additional Applitools plugin. Now we can be sure that each part of our UI works properly. Okay, but everything couldn't be that good and painless, could it? Let’s take a look at some of the pros and cons:
A big THANK YOU to Applitools and This Dot Labs teams for their ongoing support during our testing journey and for providing the possibility to test all this out. It was a blast!
Let's discuss how Valor Software can help with your project's development needs!