Quality Assured: What It Really Takes To Test Open Source Libraries

February 18, 2019

We're living in the golden age of software development, and open-source libraries are huge part of software revolution that we're facing now, that’s why I felt the need to talk about it now. I, as an Automation QA, got experience in testing open-source libraries and wanted to share my thoughts to everybody, because such type of projects requires slightly different testing approach.

All examples shown in this article are based solely on my experience testing ngx-bootstrap. Ngx-bootstrap is a set of Bootstrap components, powered by Angular, which every developer could quickly integrate into his project. Each UI component has specific reconfigurable properties/inputs/outputs, which makes the library very interesting for us in terms of gaining testing experience. Ok, now it becomes a bit confusing with all those properties and inputs, to make things clear, let's firstly go through all project roles.

Project roles in open source universe

  1. Customer
    The main consumer of an open source library is usually a simple developer. In most cases developer is trying to implement not his own, but customers wishes. What it means for us, is that QA can’t directly get requirements from an appropriate person, who knows at 100% how new feature should function. This results a few pros/cons:

PROS:

  • You have a lot of customers (in our case — developers) and you can contact each of them directly.
  • Your customer is usually very smart and it can happen that from time to time he knows much more about new feature request/issue, that you can imagine. It gives QA possibilities to grow and to learn a lot of new stuff, especially related to development, different technologies, and so on.
  • All your customers are from different countries, who speak different languages, and most importantly — they have different wishes. All this gives QA the possibility to make unique and universal instrument, like Datepicker locales: everyone needs it, but we have so many different languages with different parameters.

CONS:

  • You have tons of feature requests/issues/bugs, all of which needs to structured and implemented properly.
  • Feature requests could be conflicting. They could be hard to implement and test
  • Customer could just create an issue and forget about it. This bug could be without any details, specific description, clarifying information, which QA usually expects for reproducing issue. And in the end this bug could be not a bug at all, but just a misunderstanding on how to use the library.
  1. Developer
    Not all opensource libraries are being developed/supported by developers, who's directly interested in the development, as it been said by Tracy Lee in one of our discussions
Companies now supporting open source. Like Valor for example, because a lot of people think open source is just random people doing it, but really it’s actually supported by companies”.

This is true, and the main point here is that the main development team knows quality assurance team, and we collaborate with each other on regular basis as in the usual project. But from time to time your developer could be an undefined person from the other side of the planet.

PROS:

  • Open source project grows rapidly and exponentially, depending on how many developers are involved in working on it.
  • QA can directly interact with developers who will share their experiences.

CONS:

  • Open source for developers, in most cases, is just an initiative, which means that the response to a question / comment may be delayed due to higher priorities of actual and paid job.
  • You need to have rules, style-guides, lint’s, which are perfectly described and easy to find, BUT, the initial setup to support common code/commit/doc style, etc. requires a lot of time and force. This may not be a minus if you already have have such rules or a standardised approach to their setup.
  1. QA
    Maybe, you will laugh now, but the main testers of an open source library are the developers :-) Yeah, well, that's because they’re the main user, and they will try to use your library with different OS’s, environments, browsers, type of app’s, etc.

PROS:

  • QA needs to act like a developer, use your lib in real app code, test all new changes related to library source in already prepared test-projects.
  • Firstly, QA needs to review the code, parts of which has been changed (those parts could be broken or their behavior could change)

CONS:

  • If QA sees himself only as a manual tester, this activity could be boring and very hard.
  • If the library is too small, then QA Engineer can take over the responsibilities of  an analyst, project manager, and even the developer.

Testing process overview

Based on differences in project participants it is logically to assume, that the testing process itself will be very specific. And it’s also true. We can describe the testing stages in a following way:

1.Testing New Issues. Before including issue/request/bug to the current iteration/milestone QA needs to investigate all other related cards (link them together if they exist) and perform a requirements testing;

2.Pull Requests Testing. When a new task has been completed from the developer side, QA needs to clone all new changes locally, build the library and perform testing of a new feature, which includes:

  • requirements testing;
  • manual feature testing;
  • smoke regression testing;
  • integration testing;

3.Automated testing. PR testing also could include any types of auto-tests (if they’re not included to the CI, duh). If new PR broke auto-tests, then QA teams up with a developer in even more tighter bond to investigate failures and fix them. QA helps to cover new functionality with unit-tests.

4.Integration pre-release testing is performed only before creating a new release. Includes smoke check for the whole DEMO application, checks of full workability of all components and integration testing of the whole library (including testing of a new version of the library in integration with other test projects). Here's a very useful tip for it.

5.Integration post-release testing is being performed only after release been published (could be @latest, @next, with/without breaking changes). Usually performed in test libraries (TODO-APP or HELLO WORLD apps that can be tested for compatibility, sometimes via ng new), which are using the latest version of current library.

Documentation examples for open source libraries

The main goal of a new QA on the project — make it maximum qualitative for the minimum period of time (as it could be something else :D). And the first step in qualitative direction is documentation. In a perfect world docset should include:

  • Issue Template  — helps new contributors in creating issues in a standardised way and describes what should be done before/now/after creating a new issue.
  • Pull Request Template — helps new contributors in applying appropriate changes to the codebase with the same code style, approach and described the rules.
  • Contributing Guide — describes what types of issues/feature requests/pull requests exist, how to pull/commit/push changes, contains useful links and answers for the frequently questions.  
  • README — contains important links, info on how to clone, install, build, use and make changes to the library. This info always shown on the main GitHub repo page, so it should contain maximum amount of useful information, treat it as your landing page. Our advice would be to follow a makeareadme.com instructions on how to maintain a great README for your project.
  • Changelog — contains information about versions, releases, changes which were included to each release. Changelog is very important to developers, who waits for some fixes/features, that’s why it should not be empty. Our advice would be to follow keepachangelog.com instructions on how to maintain a great changelog for your project.
  • Wiki — a great place to accumulate all links and other useful info.
  • Graphical examples of different flows:
  1. PR flow
  2. Issue flow
  3. Development flow
  • Test design/Use-cases/Test-cases — helps each user understand how specific functionality should work, helps QA’s to cover application with auto-tests and check test coverage percentage and to manually test library and then prepare a summary with testing results.

Tech Stack for automated testing

Unit tests — Karma + Jasmine are our tools of choice for automated tests. Our approach consists of two major rules:

  1. Unit tests should be written for each new change;
  2. There should be no Pull Requests merged to development branch without tests;

This two simple rules gives dev team at least some kind of confidence that the code will work properly (its development, we never can be sure about something working 100% right). If new contributors create PR’s without unit tests, then other participants or QA’s could (should, in a perfect world) help and do everything the right way. Codecov is a bot for GitHub that helps to get rid of developers PR’s without any tests attached, and karma-coverage-istanbul-reporter is a  code coverage tool.

E2E tests — Cypress. This type of tests covers the user behavior for each demo page and demo component. Full coverage with such tests helps to decrease time for manual pre-release and post-release testing.

Anything else what QA’s should pay attention to? Sure!

  • Don’t spend a lot of time for reproducing a poorly described issue. Much better approach would be to ask author about the details directly. Many contributors ignore any recommendations and issue templates, providing so little info without confirmation.
  • Use Stackblitz, Plunker or any other online IDE (dozens of them!), which can easily share projects between people for reproduction of issues/requests.
  • Build priorities for next releases according to the most popular issues/requests inside a repository, as fixing issues that users demand highly increases usage auditory.
  • Before creating a new release with huge change/new functionality, be sure that it doesn’t break other components/services. Ask your collaborators and users about this in your public channels (Slack, Twitter, Telegram, etc.)
  • Many of paid services/libraries/tools are ready to provide free plans for Open Source. It gives to the development team more possibilities for promotion. Don’t be afraid to ask related team for cooperation, we're all in the same boat.

PS:

Testing open source libraries is a little bit different from testing other projects, at minimum it requires additional knowledge, which based on those technologies, which are being used in current lib. But it gives each involved engineer a possibility to grow much faster, than in other projects.

P.P.S

Thank you, the one, who contributes to the Open Source. Together we can make world a better place :-)

Subscribe to find out more

Thank you!

Your submission has been received!
Oops! Something went wrong while submitting the form.

More articles

Want to work with us?

Let's discuss how Valor Software can help with your development needs!