We’re living in the golden age of software development, and open-source libraries are a huge part of the software revolution that we’re facing now. That’s why I felt the need to talk about it now. I, as an Automation QA, got experience in testing open-source libraries and wanted to share my thoughts with everybody, because such type of projects requires a slightly different testing approach.
All examples shown in this article are based solely on my experience testing ngx-bootstrap.
Ngx-bootstrap is a set of Bootstrap components, powered by Angular, which developers can quickly integrate into their projects. Each UI component has specific reconfigurable properties/inputs/outputs, which makes the library very interesting for us in terms of gaining testing experience. Okay, now it becomes a bit confusing with all those properties and inputs. To make things clear, let’s go through all project roles first.
Project roles in open source universe
Customer The primary consumer of an open-source library is usually a simple developer. In most cases, a developer is trying to implement not his own, but customer’s wishes. What it means for us, is that testing engineer can’t directly get requirements from an appropriate person, who knows 100% how new features should function. This results a few pros and cons:
PROS:
You have many customers (in our case - developers), and you can contact each of them directly.
Your customer is usually smart and can know much more about the new feature request/issue that you can imagine. It gives QA possibilities to grow and to learn much new stuff, especially related to web development, different technologies, new software solutions, and so on.
All your customers are from different countries, who speak different languages, and most importantly - they have different wishes. All this gives QA the possibility to make a unique and universal instrument, like Datepicker locales: everyone needs it, but we have many different languages with different parameters.
CONS:
You have tons of feature requests/issues/bugs, all of which need to be appropriately structured and implemented.
Feature requests could be conflicting. They could be hard to implement and test.
A customer could create an issue and forget about it. This bug could be without any details, specific description, clarifying information, which QA usually expects in order to reproduce the issue. Also, in the end, this bug could be not a bug at all, but just a misunderstanding on how to use the library.
Developer
Not all open-source libraries are developed and supported by developers, who are directly interested in the development, as Tracy Lee said in one of our discussions:
Companies now support open source. Like Valor, for example, because a lot of people think open source is just random people doing it, but really it’s actually supported by companies.
This is true, and the main point here is that the leading development team knows the quality assurance team, and we regularly collaborate as in the usual project. However, from time to time, your developer could be an undefined person from the other side of the planet.
PROS:
Open source project grows rapidly and exponentially, depending on how many developers are involved in working on it.
QA can directly interact with developers who share their experiences.
CONS:
Open source for developers, in most cases, is just an initiative: the response to a question or comment may be delayed due to higher priorities of an actual and paid job.
You need to have rules, style guides, linters, which are correctly described and easy to find. BUT, the initial setup to support typical code/commit/doc style, etc., requires a lot of time and force. This may not be a minus if you already have such rules or a standardized approach to their setup.
QA
Maybe you will laugh now, but the main testers of an open source library are the developers :) Yeah, well, that’s because they’re the primary users, and they will try to use your library with different OSs, environments, browsers, type of apps, etc.
PROS:
QA needs to act as a developer, use your lib in real app code, test all new changes related to library source in already prepared test projects.
Firstly, QA needs to review the code, parts of which have been changed (those parts could be broken, or their behavior could have changed).
CONS:
This activity could be boring and very hard for a manual tester.
If the library is too small, then the QA Engineer can take over the responsibilities of an analyst, project manager, and even the developer.
Testing process overview
Based on differences in project participants, it is logical to assume that the testing process itself will be very specific. It’s true. We can describe the testing stages in the following way:
1.Testing New Issues. Before including issue/request/bug to the current iteration/milestone QA needs to investigate all other related cards (link them together if they exist) and perform a requirements testing.
2. Pull Requests (PR) Testing. When the developer completes a new task, QA needs to clone all new changes locally, build the library, and perform testing of a new feature, which includes:
Requirements testing
Manual feature testing
Smoke regression testing
Integration testing.
3. Automated testing. PR testing also can include any automated tests (if they’re not included to the CI, duh). If new PR broke automated tests, then QA teams up with a developer in a tighter bond to investigate failures and fix them. QA helps to cover new functionality with unit tests.
4. Integration pre-release testing Integration pre-release testing is performed only before creating a new release. It includes:
Smoke checking for the entire DEMO application
Checking for full workability of all components
Integration testing of the whole library (including testing of a new version of the library in integration with other test projects). Here’s a very useful tip for it.
5. Integration post-release testing is performed only after release is published (could be @latest, @next, with/without breaking changes). Usually performed in test libraries (TODO-APP or HELLO WORLD apps that can be tested for compatibility, sometimes via ng new), which are using the latest version of the current library.
Documentation examples for open-source libraries
The main goal of a new QA on the project is to make it maximum qualitative for the minimum time (as it could be something else :D). And the first step in the qualitative direction is documentation. In a perfect world, a docset should include:
Issue Template - helps new contributors in creating issues in a standardized way and describes what should be done before/now/after creating a new issue.
Pull Request Template - helps new contributors in applying appropriate changes to the codebase with the same code style, approach, and describes the rules.
Contributing Guide - describes what types of issues/feature requests/pull requests exist, how to pull/commit/push changes, contains useful links and answers for the frequent questions.
README - contains essential links, info on how to clone, install, build, use, and make changes to the library. This info is always shown on the main GitHub repo page, so it should contain the maximum amount of useful information, treat it as your landing page. Our advice would be to follow makeareadme.com instructions on how to maintain an excellent README for your project.
Changelog - contains information about versions, releases, changes that were included to each release. A changelog is very important to developers, who wait for some fixes/features, that’s why it should not be empty. Our advice would be to follow keepachangelog.com instructions on how to maintain an excellent changelog for your project.
Wiki - a great place to accumulate all links and other useful info.
Graphical examples of different flows:
Test design/Use-cases/Test-cases - helps each user understand how specific functionality should work, helps QAs to cover application with automated tests, check test coverage percentage, and to manually test the library and then prepare a summary with testing results.
Tech Stack for automated testing
Unit tests: Karma + Jasmineare our tools of choice for automated tests. Our approach consists of two major rules:
Unit tests should be written for each new change.
There should be no Pull Requests merged to development branch without tests.
These two simple rules give dev team at least some kind of confidence that the code will work adequately (its development, we can never be sure about something working 100% right). If new contributors create PRs without unit tests, then other participants or QAs could (should, in a perfect world) help and do everything the right way. Codecov is a bot for GitHub that helps to get rid of developers PRs without any tests attached, and karma-coverage-istanbul-reporter is a code coverage tool.
E2E tests: Cypress. This type of test covers the user behavior for each demo page and demo component. Full coverage with such tests helps to decrease the time for manual pre-release and post-release testing.
Anything else what QA’s should pay attention to? Sure!
Don’t spend much time for reproducing a poorly described issue. A much better approach would be to ask the author about the details directly. Many contributors ignore any recommendations and issue templates, providing so little info without confirmation.
Use Stackblitz, Plunker or any other online IDE (dozens of them!), which can easily share projects between people for the reproduction of issues/requests.
Build priorities for next releases according to the most popular issues/requests inside a repository, as fixing issues that users demand highly increases usage auditory.
Before creating a new release with colossal change/new functionality, be sure that it doesn’t break other components/services. Ask your collaborators and users about this in your public channels (Slack, Twitter, Telegram, etc.).
Many paid services/libraries/tools are ready to provide free plans for Open Source. It gives the development team more possibilities for promotion. Don’t be afraid to ask the related team for cooperation "we’re all in the same boat.
PS:
Testing open source libraries is a little bit different from testing other projects, at a minimum it requires additional knowledge, which is based on those technologies, which are used in current lib. However, it gives each involved engineer a possibility to grow much faster than in other projects.
P.P.S
Thank you, the one who contributes to the Open Source. Together we can make the world a better place :)
