January 18, 2022

Performance Testing via Artillery.io

Dmitriy Zhemchugov

Dmitriy Zhemchugov

QA Engineer

Today, I’m going to tell you all about setting up and using Artillery.io on your projects. We have a lot of popular testing tools like JMeter, Gatling, LoadNinja, etc. So you might be wondering why I’ve chosen Artillery. Well, I’m going to show its advantages and share handy usage tips later in the article.

Artillery.io is a modern performance testing software that is powerful yet quite simple to navigate. It suits different types of performance testing, on local machines and as part of CI. What’s even better? – Artillery provides a free ‘Dev’ option if you want to test the waters before diving in. So, let’s make some exploration!

61e6b6dbef2e1c7bbd4aa6a7 biofiRwiFxpiSWFpLnazIYPCle0dSTw7n o 5JqBXU2M3gBTTzM tkByXqjU50wy lh3HE M6odcvhrltnNTq dHMT3cdd2RBMFJE17Dj XkJwXJAXqSrIY6un HfQg wzNiovD 3 FW XftCg

Why Artillery.io?

Artillery offers many services that make performance and load testing faster and more reliable. Now let me outline a few reasons why I choose to use it over the better-known software.

1. Node.js tool

Java-based tools can be complicated and time-consuming to work with. Artillery.io is a Node.js tool with a basic СLI, which is a far simpler way to run performance tests.

2. Broad support of protocols, frameworks, and libraries

Out of the box, Artillery.io will support:

However, you can use Artillery.io to test any stack, thanks to the plugin interface. Here are a few examples:

  • Apache Kafka

  • Amazon Kinesis

  • HLS (HTTP Live Streaming)

3. YAML format

Unlike popular competitors’ testing tools, Artillery.io writes tests in YAML format which makes them readable and easy to understand. Artillery also supports a JSON format for testing.

4. Accessible scenarios

With Artillery’s scenarios and phases, you can build more flexible tests and cover multiple use cases in parallel. Also, it takes you just a few steps and conditions to set up tests for complex user behaviours. Finally, it’s easy to reuse scenarios for comparison and future testing.

5. Easy integration

Artillery.io seamlessly integrates with CI (for example, check the integration guide for GitHub Actions), making the process very organized. It also works with Docker, so you can run your Artillery tests inside a Docker container.

6. Detailed reports

After running tests, you can create a JSON report through the --output flag. Also, you can generate an HTML report out of JSON to save it as a file or export it to your browser. Find details on reporting in the Scenario section below.

7. Artillery Docs

Artillery provides numerous documents to browse for instructions and other useful info. This helps you get the performance testing software up and running much smoother than with other similar tools. Also, you can look up support from other developers on community forums.

61e6b6dac135a26d7c224c86 FEcA8iQh4priOp2EIgpa41pwJFHtkcxiPlfx3Z1YRrUWU7tvH9Ddve5UY9cWYyGrullTwo97KCHgZlRZqFxCZEg5ZsjWa3U7JF nXQ9qTiNak2NGp1VBAI4GlJlPrslAFZRUoWSztiw7iLknhw

How does Artillery.io work?

I’m going to walk you through some basic commands to get started with Artillery.io using ngx-bootstrap in my examples. Ngx-bootstrap, together with other community libraries, were created by Valor Software, the company where I’m working. And as we have hundreds of thousands of engineers using our products monthly, we need scalable performance testing to make sure everything runs smoothly. So, let’s get started!

First, you need to install the latest version of the software.

Install

To install the latest version, enter:

npm install -g [email protected]

Fast test

To run a fast test, enter:

artillery quick --count 15 -n 30 https://valor-software.com/ngx-bootstrap/

With this command, the test will run 15 virtual users with each of them making 30 HTTP GET requests to the URL address that you specified.

This command also supports other flags to configure your fast tests:

61e6dc58268ada7b2baa7189 n3n85QBpLJnNRmjf2KmGI6Vd 0JXn6sZ4Pt Jf3TihpGe9xpzK Wmr icfmr1P6GTAg9FYkctA1jx8N gREo4dv9UxUOAZ2IlHXmzBXPy7PtsMEL3n7wDT0wzyuknHe2GBVN 5i9

Below is a list of possible configurations of the run command:

61e6dcc8f56ef40c91343a66 D1wfiDO76AqAbf8YFECjCHzfM9qEMex5fwDqiesnrSbYTig01ZrrUzcRGEHaUDVNpoKzdOb3MmY5ItdD2k3dUJgYIa68uPmJ3uZRaIpP7o47rbJzZV1XPVJDh3qm G5AC7seiycE

Config

For standard test scenarios, you should create a ‘my.yml’ file with config.

620a6caabc207815e5fd9851 B9QLpv1Xk2qcqHSwmTRK0jGv2uYYmLXg5kcfM rLZAxoeW51olKqZ gjZqWNvZLe9c CAKFo3KluK6pEdblP7WYNgKwNZZHqI7CoIevcbLaGD8vM3iPfOlyE6u4FdJZejlnJF9GR

The my.yml file should contain ‘config:’ paired with the target URL. To set up environments for your tests, you can write environments into the command after the target URL has been entered.

620a6cdfcec6669ba3cb933a 2OBgp8 wSgsPmKlIKoaE8IgftkHZxI8U Jir1xtLQqzXbsjjTIiXBxfTpt8VNBKqyo5SSZkR JRXI1uoU2SiuHisygXLl SuCcXYFzTToIluLozfi3DJe5Lp2 CXGvIpaNwaqYek

Phases in the config

Artillery.io allows you to enter multiple sequential load options for the application. The testing phase consists of several aspects:

  • duration: the time of one phase;

  • arrivalRate: the number of users added each second;

  • rampTo: up to how many users per second the load will grow by;

  • name: a name of the phases.

620a6d49351453e1970bd6df v2FiTu ndt8HAMbHRAjIrf 6jXsA fNg6BHksT1XywImEog5BjNvOEiK1UXmcBE0 mU Yw3mDpUBT6hLaK 4yBl1pGB33iTZ35tNH2F 13n2E18w6aHzhwtCwCtCcSjUNe8JVO8

If you only have one target URL, the different phases of the performance testing are placed right after it. If there are multiple environments, then you should add phases to each environment variable.

Plugins

In the Artillery npm utility, you can find lots of plugins that can help you in your performance testing. Install artillery-plugin-expect to compare the expected result with the actually received result.

Then, after the phases inside the config, enter:

plugins:
    expect: {}
620a6f1f351453cf580c72ab WW4O0hUrQygtMKCL41Y28O QtSu A4xItcAFeIUiNz7hCfipkstp6MpQW6dYvP4843dtb2JvwXQNSUT2ZXxfmQRG2ObvDBlzDtJtxu2jJUBdKnskDTkvYt uGvLiAoQUONeTw 9O

Now, with the config completed, we can finally write some tests!

Scenario

All tests should be written in the scenarios section and should contain:

  • GET, POST, PUT, DELETE, and some other commands;

  • a URL for every endpoint;

  • the body text in JSON format;

  • all checks you want to run.

620a6e00b9149a62042fd21e BaWRTn2B9CtF3V0o2f4fuvZ34cxOvNTQeOByTklpYQ5KeY SxgSEe2Rw5oZryDz1HaX9SlB3syP64qTHxPURlatPHPdMzqWr363JcLgzG3X4DwLGUvsejTsjg3Zt2WYq F3Nd5kn

To run tests, you should enter:

artillery run my.yml

To run a test and then generate a report in a .txt file, run:

artillery run -o first.txt my.yml

After generating a .txt report, to generate a second report in HTML, run the command:

artillery report first.txt

Here’s the report example:

620677d1e5b37a5a34ac81b8 Artillery report Valor

Authentication

With Artillery.io, you can use basic authentication or get authorized by tokens uploading CSV with your credentials. Just pick an option that suits your project best.

Check their official documentation to find information on authentication and many other aspects of Artillery.io.

620a317e6e04332776bf7cb9 JsWFVRbBNI RNrtp OeJl3b y6LcxHFlEvmXqAqn8zZIKzTnOZrQhPWVs0K7eBjtntToigptblOB3ZPic7ot0Z xcupUM T  U2oS NwljelN1fGeAJ26wwm0dgn9w0UTvvmkdQz

Let’s first revise how you can get authorized by tokens. To upload a CSV file with credentials, you should add the following lines into the config file:

  1. payload – to use the payload functionality;

  2. path – to write a path for the CSV file that Artillery needs to use;

  3. fields – names of fields that you need to use.

61e6b6dcab56b80f45de406b ydhUhDNow7 XGKOXcFR z8iechX2JWuaC6DH7NnsDFsT6lbZIcdD0PVyxfZVbmM390aVwnVobdEtLSW6hpJKjRE81cLu7ewXpXK0xjTNwo6M2mWOl3hAVsOb9T8M6 WGrnlj44Y7

See an example with a CSV file:

61e6b6dcc135a22c44224d06 2C8yt MSmdA8iiOuIsj5zaczDHbc nDsfDlp3nelSyAn8KLgpRnIQUW8c5B7PYwgFRTawbEktVfhKzogHvXMkR05kVfZBo7fG6ofObMR77LxLnKE9HCv2tKnmJea74yyGLtzb5rV

The second way to get authorized is through environment variables. And we have several options for this. The first one is to execute an export command in your console as below:

61e6b6dcc9a859cf8bd3b0a5 t5VlpO3PEXY0fx4Gz90oF2VhpSsmzK j0Qli3mZeg4i2ZrhQ3RJvVCtrySPRwtpHUOc78fhISHne SmjgNZPLK1ruEU63n5ufIqgNWeqlxbTLB3xArmN3gAkcN3MWkDpB62jDgOB

And after it’s done, you can use your variable inside your yml file:

61e6b6ddc9a8593889d3b180 emp w2bZFybmHBqVjaig0JVYAMNpEF 9DVpUFTYQ0OuulveZ2qrR05mUZao fGcNzKK7IpfJB2Gr7hb5kZG3878EvW W75gku95b0KwqNK8rGEpWW0dyQGONPFiTbCJ0zbR3K558

Another option is to write a new script with an environment variable in your package.json file to execute your tests without typing the export variable in the terminal each time. It will look like this:

61e6b6dc33effcb4ebfc94af 1FPnguaIeeWsyUgBiyhBUVYd1siQlEGlBGObLF ae57qBDD7bpVhfb6NdyLCEfJ86mUvFcJtV4SdAdT7FIj43EgahM4g6kSJxfu8qs pDuXwsDB1uxExTdxfyme9m5n79ziueQOY

Then all you need to do is run this script in the terminal:

npm run artillery:run

Finally, you can upload CSV to pass parameters for your test requests: GET, POST, PUT, etc.

Check an example with POST:

61e6b6dd62547e71c134018e h KGFTQfl8aK 2o2gRMbK89FdZXCkfTCX6r2yjtTLx2kZfiE1J26NyWLOZX xyxJdGJ9So4u4lRSY4pKB2q0OhceVpSLwzYVBSfLTpVh obAowo1Tvpwp KEgbRt92MwipCWsc r

Final thoughts

Artillery.io is an accessible and comprehensive tool that I use and can surely recommend for performance testing.

With a free Artillery Core version, you can run tests from a local or virtual machine. And by switching to Artillery Pro, you’re getting a self-hosted AWS performance testing platform.

Other features of Artillery Pro include:

  • runs millions of tests per second across 13 geographical regions;

  • works with existing security systems;

  • no repeat charges or paid maintenance;

  • VPC internal test services.

I hope this article has given you some clarity on Artillery’s software, and helped you make a choice regarding the performance testing tool you want to work with.

In case you’re looking for help in software testing, or your project needs an advanced quality assurance pipeline – drop us a line!