by Thomas D. LaToza (@ThomasLaToza) and André van der Hoek (@awvanderhoek)
Associate Editor: Christoph Treude (@ctreude)
We value our professional software developers, hire them in teams (local or remote), and set off on our merry way producing projects. This has been the status quo for a long time. But does it need to be? Could the crowd supplant the need for explicitly hired teams? More strongly yet: should the crowd perhaps replace hired employees?
These are tantalizing questions, both in terms of excitement about possibilities and in terms of 'fear' regarding the significant disruption that may play out in the workplace. Here, we consider several aspects of the issue of crowds versus hired employees.
1. Isn’t this just open source?
According to Howe [1], crowdsourcing is “the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call”. In software, open source is one of the oldest and most successful crowdsourcing models. But all crowdsourcing is not open source. Platforms such as TopCoder employ a competition model, in which developers respond to a client request and clients select and reward a winner. Others such as trymyUI offer to the crowd short, well-defined tasks that, when taken together, compose into a larger whole – for instance comprehensively testing the usability of an application. Labor markets such as UpWork offer the promise of on-demand, fluid labor forces with the skills necessary for the job at hand, with workers ready to take on specialized work for an hour, a day, or maybe a week.
2. What about the speed of development?
As the old adage goes, many hands make light work. Fundamental to many crowdsourcing models is the decomposition of a large task into many small tasks. In principle, decomposed tasks enable work to be completed dramatically faster, as they enable parallel distribution to and completion of tasks by the crowd. But decomposition itself brings challenges: communicating tasks to developers, coordinating contributions made by the crowd, and ensuring smooth handoffs between tasks. In practice, these competing forces have given rise to a wide diversity of competing systems and platforms, using different granularities of tasks to crowdsource a variety of software development activities.
3. What about quality?
One common concern about opening contributions to anyone is the quality of the work that will result. As in traditional development, there are many approaches to managing quality. Work is almost always reviewed, either by the client, an agent of the client, or the crowd itself. Many platforms explicitly track reputation, allowing clients to identify those who have demonstrated quality work on similar tasks in the past and motivating developers to do high quality work to maintain their reputation. Competition platforms use the crowd to generate competing alternatives, allowing the client to select the highest quality solution. Through these mechanisms, it may, in fact, be possible to achieve higher quality through crowdsourcing, as some of our work has demonstrated [2]. On the other hand, it has been observed that, without proper management, quality may well suffer; care, thus, must be taken to focus on quality from the start [3].
4. What about crowdsourcing environments?
A significant innovation many crowdsourcing approaches bring is their dedicated support for performing crowdsourced work. Crowdsourcing platforms often provide, directly in the environment, support for contributors to browse and identify tasks matching their interests or that have the greatest chance of a reward. Clients may browse reputation information about potential workers, browse submitted contributions, and use the platform to make and manage payments. Systems that offer fine-grained tasks may go further still, offering workers self-contained tasks that can be quickly completed within the environment itself. For example, CrowdCode [4] offers an online environment for completing programming microtasks.
5. What about more complex development tasks?
Another common question is how a complex task such as architecture or design—requiring knowledge of the project as a whole or with dependencies making decomposition difficult—could ever be crowdsourced? Of course, one approach is simply to not decompose such tasks. For example, some TopCoder tasks are architectural in nature, asking competing workers to create design documents. An experienced crowd worker called a “co-pilot” is then responsible for creating, coordinating, and managing the tasks to implement the architecture. However, this can bring challenges in imposing a waterfall development process, where an architecture is built up-front, independent of future programming work [3]. Alternatively, it may be possible for the crowd itself to be more involved, decomposing architectures into networks of connected decisions [5].
6. Can I help?
Crowdsourcing is already penetrating software development practice today [6][7]. TopCoder has hosted more than 427,000 software design, development, and data science competitions. More than 100,000 testers freelance on uTest. But we believe that this may be just the beginning [8]. Just as open source changed how many organizations do software development and the nature of organizations themselves, crowdsourcing opens the door to new types of contributions and new ways stakeholders may interact that may lead to new models of software development. To help understand where these models may fail and where they may succeed, we have begun to study crowdsourcing approaches through a variety of experiments, from debugging to programming to designing. If you’d like to see how such approaches may work and help discover new ways of developing software, sign up here.
References
[1] James Surowiecki. (2005). The Wisdom of Crowds. Random House.
[2] Thomas D. LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and André van der Hoek. (2015). Borrowing from the crowd: a study of recombination in software design competitions. International Conference on Software Engineering, 551-562.
[3] Klaas-Jan Stol and Brian Fitzgerald. (2014). Two's company, three's a crowd: a case study of crowdsourcing software development. International Conference on Software Engineering (ICSE), 187-198.
[4] Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André van der Hoek. (2014). Microtask programming: building software with a crowd. Symposium on User Interface Software and Technology (UIST), 43-54.
[5] Thomas D. LaToza, Arturo Di Lecce, Fabio Ricci, W. Ben Towne, and André van der Hoek. (2015). Ask the crowd: scaffolding coordination and knowledge sharing in microtask programming. Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 23-27.
[6] Thomas D. LaToza and André van der Hoek. (2016). Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software, 33, 1 (January 2016), 74-80.
[7] Ke Mao, Licia Capra, Mark Harman, and Yue Jia. (2015). A survey of the use of crowdsourcing in software engineering. Technical Report RN/15/01, Department of Computer Science, University College London.
[8] Thomas D. LaToza and André van der Hoek. (2015). A vision of crowd development. International Conference on Software Engineering (ICSE), 563-566.
Associate Editor: Christoph Treude (@ctreude)
We value our professional software developers, hire them in teams (local or remote), and set off on our merry way producing projects. This has been the status quo for a long time. But does it need to be? Could the crowd supplant the need for explicitly hired teams? More strongly yet: should the crowd perhaps replace hired employees?
These are tantalizing questions, both in terms of excitement about possibilities and in terms of 'fear' regarding the significant disruption that may play out in the workplace. Here, we consider several aspects of the issue of crowds versus hired employees.
1. Isn’t this just open source?
According to Howe [1], crowdsourcing is “the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call”. In software, open source is one of the oldest and most successful crowdsourcing models. But all crowdsourcing is not open source. Platforms such as TopCoder employ a competition model, in which developers respond to a client request and clients select and reward a winner. Others such as trymyUI offer to the crowd short, well-defined tasks that, when taken together, compose into a larger whole – for instance comprehensively testing the usability of an application. Labor markets such as UpWork offer the promise of on-demand, fluid labor forces with the skills necessary for the job at hand, with workers ready to take on specialized work for an hour, a day, or maybe a week.
2. What about the speed of development?
As the old adage goes, many hands make light work. Fundamental to many crowdsourcing models is the decomposition of a large task into many small tasks. In principle, decomposed tasks enable work to be completed dramatically faster, as they enable parallel distribution to and completion of tasks by the crowd. But decomposition itself brings challenges: communicating tasks to developers, coordinating contributions made by the crowd, and ensuring smooth handoffs between tasks. In practice, these competing forces have given rise to a wide diversity of competing systems and platforms, using different granularities of tasks to crowdsource a variety of software development activities.
3. What about quality?
One common concern about opening contributions to anyone is the quality of the work that will result. As in traditional development, there are many approaches to managing quality. Work is almost always reviewed, either by the client, an agent of the client, or the crowd itself. Many platforms explicitly track reputation, allowing clients to identify those who have demonstrated quality work on similar tasks in the past and motivating developers to do high quality work to maintain their reputation. Competition platforms use the crowd to generate competing alternatives, allowing the client to select the highest quality solution. Through these mechanisms, it may, in fact, be possible to achieve higher quality through crowdsourcing, as some of our work has demonstrated [2]. On the other hand, it has been observed that, without proper management, quality may well suffer; care, thus, must be taken to focus on quality from the start [3].
4. What about crowdsourcing environments?
A significant innovation many crowdsourcing approaches bring is their dedicated support for performing crowdsourced work. Crowdsourcing platforms often provide, directly in the environment, support for contributors to browse and identify tasks matching their interests or that have the greatest chance of a reward. Clients may browse reputation information about potential workers, browse submitted contributions, and use the platform to make and manage payments. Systems that offer fine-grained tasks may go further still, offering workers self-contained tasks that can be quickly completed within the environment itself. For example, CrowdCode [4] offers an online environment for completing programming microtasks.
5. What about more complex development tasks?
Another common question is how a complex task such as architecture or design—requiring knowledge of the project as a whole or with dependencies making decomposition difficult—could ever be crowdsourced? Of course, one approach is simply to not decompose such tasks. For example, some TopCoder tasks are architectural in nature, asking competing workers to create design documents. An experienced crowd worker called a “co-pilot” is then responsible for creating, coordinating, and managing the tasks to implement the architecture. However, this can bring challenges in imposing a waterfall development process, where an architecture is built up-front, independent of future programming work [3]. Alternatively, it may be possible for the crowd itself to be more involved, decomposing architectures into networks of connected decisions [5].
6. Can I help?
Crowdsourcing is already penetrating software development practice today [6][7]. TopCoder has hosted more than 427,000 software design, development, and data science competitions. More than 100,000 testers freelance on uTest. But we believe that this may be just the beginning [8]. Just as open source changed how many organizations do software development and the nature of organizations themselves, crowdsourcing opens the door to new types of contributions and new ways stakeholders may interact that may lead to new models of software development. To help understand where these models may fail and where they may succeed, we have begun to study crowdsourcing approaches through a variety of experiments, from debugging to programming to designing. If you’d like to see how such approaches may work and help discover new ways of developing software, sign up here.
References
[1] James Surowiecki. (2005). The Wisdom of Crowds. Random House.
[2] Thomas D. LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and André van der Hoek. (2015). Borrowing from the crowd: a study of recombination in software design competitions. International Conference on Software Engineering, 551-562.
[3] Klaas-Jan Stol and Brian Fitzgerald. (2014). Two's company, three's a crowd: a case study of crowdsourcing software development. International Conference on Software Engineering (ICSE), 187-198.
[4] Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André van der Hoek. (2014). Microtask programming: building software with a crowd. Symposium on User Interface Software and Technology (UIST), 43-54.
[5] Thomas D. LaToza, Arturo Di Lecce, Fabio Ricci, W. Ben Towne, and André van der Hoek. (2015). Ask the crowd: scaffolding coordination and knowledge sharing in microtask programming. Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 23-27.
[6] Thomas D. LaToza and André van der Hoek. (2016). Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software, 33, 1 (January 2016), 74-80.
[7] Ke Mao, Licia Capra, Mark Harman, and Yue Jia. (2015). A survey of the use of crowdsourcing in software engineering. Technical Report RN/15/01, Department of Computer Science, University College London.
[8] Thomas D. LaToza and André van der Hoek. (2015). A vision of crowd development. International Conference on Software Engineering (ICSE), 563-566.
