This link will take you to the Google Doc used to create the chapter. Below is a copy lifted from the link on February 28, 2025. Formatting adjusted with Chat GPT
Artificial Intelligence and Collective IntelligenceDaniel S. WeldUniversity of Washington (CSE)Seattle, WAweld@cs.washington.edu
MausamIndian Institute of TechnologyDelhi, Indiamausam@cse.iitd.ac.in
Christopher H. LinUniversity of Washington (CSE)Seattle, WAchrislin@cs.washington.edu
Jonathan BraggUniversity of Washington (CSE)Seattle, WAjbragg@cs.washington.edu
The vision of artificial intelligence (AI) is often manifested through an autonomous software module (agent) in a complex and uncertain environment. The agent is capable of thinking ahead and acting for long periods of time in accordance with its goals/objectives. It is also capable of learning and refining its understanding of the world. The agent may accomplish this based on its own experience, or from the feedback provided by humans. Famous recent examples include self-driving cars (Thrun 2006) and the IBM Jeopardy player Watson (Ferrucci et al. 2010). This chapter explores the immense value of AI techniques for collective intelligence, including ways to make interactions between large numbers of humans more efficient.
By defining collective intelligence as "groups of individuals acting collectively in an intelligent manner," one soon wishes to nail down the meaning of individual. In this chapter, individuals may be software agents and/or people and the collective may consist of a mixture of both. The rise of collective intelligence allows novel possibilities of seamlessly integrating machine and human intelligence at a large scale – one of the holy grails of AI (known in the literature as mixed-initiative systems (Horvitz 2007)). Our chapter focuses on one such integration – the use of machine intelligence for the management of crowdsourcing platforms (Weld, Mausam, and Dai 2011).
Crowdsourcing is a special case of collective intelligence, where a third party (called the requestor) with some internal objective solicits a group of individuals (called workers) to perform a set of inter-related tasks in service of that objective. The requestor’s objective may be expressed in the form of a utility function to be maximized. For example, a requestor might wish to obtain labels for a large set of images; in this case, her utility function might be the average quality of labels subject to a constraint that no more than $X dollars be spent paying workers. We assume that the workers act independently, interacting only through the shared tasks. Each worker has an individual utility function, which is often different from the collective’s utility function. Furthermore, we assume that their utility functions are independent of each other. The AI subfield of multi-agent systems considers even richer models, in which individual agents may reason about the objectives of other agents, negotiate, and bargain with each other (Weiss 2013). We won’t discuss these techniques here, but the chapter on game theory explores some of these issues.
There are two natural points of connection between AI & crowdsourcing:
AI for crowdsourcing and
crowdsourcing for AI.
While this chapter centers on the former we note that in recent years crowdsourcing has had a significant impact on AI research as well – a great many projects use crowdsourcing to label training sets as input for data-hungry supervised learning algorithms (Snow et al. 2008a; Callison- Burch 2009; Hsueh, Melville, and Sindhwani 2009).
Why does crowdsourcing need AI? Crowdsourcing is an effective medium for congregating a large set of workers (usually virtually) who assist with a common goal. This allows for creative new applications that use the wisdom of crowds or the round-the-clock availability of people (e.g., (Bigham et al. 2010)). At the same time, the sheer volume of tasks, and highly varying skills and abilities of workers typically make it infeasible to manually manage the task allocation as well as quality control. Moreover, the design of crowdsourced interfaces and workflows to accomplish a new task remains cumbersome and expensive. For example, often a task may get routed to a worker not skilled enough or interested in it. Different tasks may require slightly different workflows to achieve high quality. Different task instances may be individually easier or more difficult, requiring less or more work (iterations) on them. These and other challenges necessitate the use of automated techniques for the design and management of crowdsourcing processes.
A long-term vision of AI for crowdsourcing is to enable optimal design of workflows and management of task instances, thereby making crowdsourcing platforms highly efficient, saving thousands of man-hours and millions of dollars, and also making crowdsourcing really easy to use for a novice requestor. AI is a natural fit for this vision because, in general, AI algorithms are great at building models, drawing inferences, and detecting outliers from the data. They are also effective in taking decisions in uncertain environments towards maximizing an objective. In this chapter, we discuss several uses of AI in this space – we describe learning algorithms that model the accuracy of crowd members, aggregation methods for predicting true answers from error-prone and disagreeing workers, and AI control algorithms that choose which tasks to request and which individuals should work on them.
(Original text formatting has been preserved while adjusting layout for better readability.)
Most recent version released by authors
Comments