By Ben Christensen
Director, Content Relevance
“When should I use crowdsourcing and when should I use a managed crowd?” This is the question anyone interested in staffing human annotation tasks should be asking, but many don’t because they don’t even know there are two different options. So let’s start there—defining the options. Assuming you need human annotation, for example for search relevance evaluation, there are two ways you can gather the necessary humans to do that work: 1. Crowdsourcing, where the task is made available to a large crowd without any training or management beyond a very limited set of task instructions and possibly a simple screening test; or 2. Managed crowds, where a smaller group is trained to complete the task accurately and overseen by a manager.
The power of crowdsourcing is in its numbers. You can accomplish a lot quickly because many hands make light work. A hundred thousand people can do quite a bit more than a hundred can. The cost is less because crowdsourcing typically pays only a few pennies per judgment. Most members of the crowd aren’t trying to make a living—they’re just trying to make a few extra bucks in their spare time. There’s usually little overhead involved in crowdsourcing because the crowd manages itself. You put the task out there, and if it’s interesting enough and pays enough, the crowd will get it done.
With these advantages come a few limitations: First, quality control is extremely limited. Without anyone overseeing the quality of the crowd, you must rely on clear instructions, automated understanding checks, and high overlap to get data you can trust. Overlap is important because there will always be noise in the crowd, so you’ll likely pay for at least […]