Crowdsourced testing
Crowdsourced testing is an emerging trend in software testing which exploits the
benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform. It differs
from traditional testing methods in that the testing is carried out by a number of
different testers from different places, and not by hired consultants and professionals.
The software is put to test under diverse realistic platforms which makes it more
reliable, cost-effective, fast, and bug-free.[citation needed] In addition, crowdsource testing
allows for remote usability testing because specific target groups can be recruited
through the crowd.
This method of testing is considered when the software is more user-centric: i.e.,
software whose success is determined by its user feedback and which has a diverse
user space. It is frequently implemented with gaming, mobile applications, when
experts who may be difficult to find in one place are required for specific testing, or
when the company lacks the resources or time to carry out the testing internally.
Crowdsource testing companies provide the platform for the testing cycles, generally
free of charge to the companies whose products are being tested. They then
crowdsource the product to a community of testers, who register for testing the
software voluntarily. These testers are generally paid per bug, depending upon the type
of bug and its market price. The crowdsource testing team is usually in addition to the
in-house quality assurance team, not a replacement. Depending on the task, some
platforms offer access to their crowd through a self-service model, a managed service
by the provider, or both.[1]
Advantages of crowdsource testing
There are several advantages to crowdsource testing:
The core testing team may not have all the resources to test the software in
different environments and in different situations (e.g. different Internet bandwidths,
devices, etc.), as it may not be possible to have all the resources to create different
environments in which the software should be tested.
It is cost effective, as the product company pays only for the valid bugs reported.
Usually time to test the software is comparably lesser, so it leads to better
productivity and thus is cheaper than hiring engineers, designers, and specialists.
Testers performing this form of testing are unbiased towards the internal
concerns of the company.
The pool of testers is diverse with variations in languages as well as locales. This
helps in testing applications which are based on localization.
As there are large number of testers testing a software simultaneously, testing
can be done quickly, resulting in less time to market.
Disadvantages of crowdsource testing
There are several disadvantages to crowdsource testing: [2]
Confidentiality must be managed closely as the number of non-internal
individuals looking at the system under test increases.
Immediate and prompt communication with a group of crowdsource testers can
be difficult.
Crowdsource testers who are compensated based on the number of bugs
detected may detect a larger number of less impactful bugs while skipping over
more critical or harder to replicate bugs.
Crowdsource testing will result in increased need for management oversight due
to differences in testers' time zones and locations, languages, and cultures.
Ensuring test coverage in crowdsource testing can be difficult as testing is not
planned or tracked the same way as traditional waterfall or Agile test efforts.
Crowdsource testing vs. outsource testing
The main difference between crowdsource testing and software testing outsourcing is
that, in crowdsource testing, testers may belong to different workplaces.
When outsourcingtesting, the testers are from the same company or workplace that is
responsible for outsourcing. In crowdsource testing, people voluntarily test software
with the possibility of not being paid (if no bugs are discovered). Outsource testers
always get paid for their work.
While for some projects it may be possible to get away with only using one approach or
the other, a more thorough approach would use a holistic software testing method,
[3]
which uses both a dedicated testing team in addition to the crowd. Crowdsource
testing is best for things like beta and compatibility testing, which are necessary final
steps for testing; however, most software is far too complex for late-stage testing like
this to cover all of the possible issues. A dedicated outsourced or in-house testing team
will give a better idea of the software's possible defects, but will not give anywhere near
the scope of crowdtesting. Therefore, a good solution is to integrate multiple test teams
into any development project (and also to develop with the principles of testability [4] in
mind from the very beginning.)
Crowdsourcing alone may not give the best feedback on applications. A diverse testing
approach that pools both crowdsource testing and a dedicated testing team may be
favorable. "Having this diversity of staffing allows you to scale your resources up and
down in a fluid manner, meeting tight deadlines during peak periods of development
and testing, while controlling costs during slow periods."[5]
