Crowdtesting

Crowdtesting or crowdsourced testing is part of the Croudsourcing concept. Its purpose is to use the crowd as tester for applications, softwares, etc. instead of a test team. "The information technology world is shifting from one driven by corporations to one driven by consumers, who demand more flexibility as they carry their devices through a myriad of environments and ecosystems." [Dieter Speidel, Mayank Mittal, Mithun Sridharan, Testing experience, Crowd Testing, march 2014]

Definition

Crowdtesting is a manner of managing a test. The crowd is used to test a software, a product, an application, etc. It has appeared as an opposing option to the well-known test team usually trained to that purpose. Crowdtesting achieves results that are generally much less focused that a test team. Always depending on the project, such an approach can be extremely useful. A crowd can cover much more development scenarios and patterns of use. "The advantages of QoE [Quality of Experience] crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings" [Tobias Hossfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, Phuoc Tran-Gia, Best Practices for QoE Crowdtesting : QoE Assessment With Crouwdsourcing]

People involved

• All this process is realized to satisfy the client who will in the end use the product.
• In order to solve every possible problems that the product might have, the producer hires someone who organizes a test (organizer).
• After that the crowd has to play its role by using the product and giving feedbacks. "A crowd is a gathering of people, of varying numbers, who act in an extempore and independent manner" [Mukesh Sharma, Rajini Padmanaban, Leveraging the Wisdom of the Crowd in Software Testing]. Depending on the project, the crowd may or may not be selected precisely. This is maid thanks previous questionnaires that of course contain a certain margin of error, since the crowd answers these questions at home without any supervision.
• In the end, the results have to be analyzed and the product corrected in order to be as optimal as possible (specialists).

Process

• The first step consists of the definition of the project, and its analysis, so that it is clear that crowdtesting is better adapted than a classical test team.
• After that, it is necessary to define the questions that will lead to the discovery of the possible bugs of the product.

Now that the test is ready, the crowd enters the process by using the product. People have to document their use, so that when a problem is found, it is possible to know exactly how to reproduce it and solve it. The documentation (texts, images, screenshots, etc.) is extremely important since the testers are not taking the test under the surveillance of the organizer, but at home or in the train or anywhere else.

• The next step concerns the results that have to be collected and analyzed.
• The last step is to solve all the problems found.

Each project implies a different way of managing each step, since the product can have variable purposes.

Once the product is on the market, it is almost impossible to correct the problems and still be successful which means that the testing process is extremely important !

Money

There are different options in managing the money :

• “Pay-per-bug” is the most used option and the person taking the test only gets money if founding a significant problem and proving that it exists.
• Another option is to be paid a certain amount of money for taking the test and in addition to that be paid again if finding a significant problem.
• The last option is to be paid per hour spent taking the test and in addition to that be paid again if finding a significant problem.

Motivations

For the producer, "crowd-sourced software testing helps achieve a high probability of consumer acceptance of software modules by ensuring that the software is defect-free to a very high degree." [Dieter Speidel, Mayank Mittal, Mithun Sridharan, Testing exprience, Crowd Testing, march 2014] The main motivation in participating in a crowdtest is the money. There are other influent factors, but they are far less mentioned than money : competence, altruism, fun, challenge oneself, etc.

When to use a crowdtsourced test?

A crowdsourced test is useful if the product concerns a global use, which means that it is designed to be used by a large amount of people and not just a specific community that would have specific knowledges. The crowd allows a product to be tested in realistic conditions, whereas such a total coverage cannot be obtained by a classical test team.

Benefits

The benefits of crowdtesting are for instance coverage, reduction of costs, availability, speed, impartiality, diversity, real conditions.

• Coverage is a very important factor since an extremely large amount of devices is used as well as system configurations, whereas a test team could not cover such a large diversity of devices and therefore notice all the possible problems.

• Reduction of costs also plays an important role for the enterprises that organize a test and are looking for results. As described above, there are three different ways to manage the money but the one that is most used is "pay-per-bug" which is often applied regardless of the importance of the bug.

• Availability concerns the time as well as the location. Indeed, a test taken by the crowd runs at any time of the days because it involves people from all over the world.

• Speed is a result of the availability but also of the fact that the amount of people is much larger than the one of a test team.

• Impartiability results of the fact that a crowd consists of people whose education, interests, knowledges, etc. are indefinitely variable.

• Diversity concerns as well the people as the devices. The crowd may help the developing process of a product by enlightening a more intuitive manner of using it.

• Real conditions refers to the fact that these products are created for the people, so using the people to test it is as close as possible to the real situation in which it is going to be after its release.

Risks

The risks or challenges in crowdtesting are for example the lack of ownership, the lack of control, the lack of measurement, the reward concept, the communication with the crowd, the security, the continuous involvement, etc.

• Security concerns the confidentiality the testers want to have in order to take the test, so that they can give any personal information that will be kept confidential. This criteria is generally well organized by the crowdtesting platforms.

Examples

Ebay, Amazon, Microsoft, Google, Facebook, Linux, IBM, Netflix, etc.

• For instance, nine millions people tested the beta-version of Microsoft Office 2010, with which they gathered two millions significant comments. This allowed them to improve their product. Today, "Microsoft (especially Windows) is the biggest beta tester in history" [Mukesh Sharma, Rajini Padmanaban, Leveraging the Wisdom of the Crowd in Software Testing].

• IBM was one of the first to apply a crowdsourced test back in the 1950' for a hardware. However, the concept of crowdtesting only appeared concretely this last decade.

Sources

• Dieter Speidel, Mayank Mittal, Mithun Sridharan, Testing exprience, Crowd Testing, march 2014
• Mukesh Sharma, Rajini Padmanaban, Leveraging the Wisdom of the Crowd in Software Testing
• James Surowiecki, The Wisdom of Crowds
• Rajini Padmanaban, A Technical Paper on Best Practices in Crowd Sourced Testing
• Tobias Hossfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, Phuoc Tran-Gia, Best Practices for QoE Crowdtesting : QoE Assessment With Crowdsourcing