Managing the Crowd: When to Use Managed Crowds vs. Crowdsourcing

By Ben Christensen
Director, Content Relevance

“When should I use crowdsourcing and when should I use a managed crowd?” This is the question anyone interested in staffing human annotation tasks should be asking, but many don’t because they don’t even know there are two different options. So let’s start there—defining the options. Assuming you need human annotation, for example for search relevance evaluation, there are two ways you can gather the necessary humans to do that work: 1. Crowdsourcing, where the task is made available to a large crowd without any training or management beyond a very limited set of task instructions and possibly a simple screening test; or 2. Managed crowds, where a smaller group is trained to complete the task accurately and overseen by a manager.

The power of crowdsourcing is in its numbers. You can accomplish a lot quickly because many hands make light work. A hundred thousand people can do quite a bit more than a hundred can. The cost is less because crowdsourcing typically pays only a few pennies per judgment. Most members of the crowd aren’t trying to make a living—they’re just trying to make a few extra bucks in their spare time. There’s usually little overhead involved in crowdsourcing because the crowd manages itself. You put the task out there, and if it’s interesting enough and pays enough, the crowd will get it done.

With these advantages come a few limitations: First, quality control is extremely limited. Without anyone overseeing the quality of the crowd, you must rely on clear instructions, automated understanding checks, and high overlap to get data you can trust. Overlap is important because there will always be noise in the crowd, so you’ll likely pay for at least […]

By |November 17th, 2015|Press|0 Comments

Appen’s Dorota Iskra to Speak at LT-Accelerate 2015

Appen is proud to announce that Dorota Iskra will be speaking at LT-Accelerate 2015 in Brussels, Belgium on Monday, November 23rd. Dorota’s presentation is entitled “Crowdsourcing in language data collection and annotation”.

Appen has been active in the area of data collection and annotation for over 15 years. Starting with a traditional approach of working locally, we have gradually moved to using web-based tools. This presentation focuses on advantages of web-based tools such as greater reach, lower cost, and enabling us to build an extensive network/crowd. On the surface crowdsourcing appears an attractive alternative for data collection and annotation, but poses a lot of challenges. In order to address these challenges, we have built our own crowd in a controlled and tested environment.

Dorota has extensive experience in speech and language technology where, after a period research, she has moved towards applications and language resources. She has worked in the telecom and software development industries and is currently responsible for the European business at Appen, a company collecting speech and language data and providing a wide range of linguistic services.


By |November 11th, 2015|Press|0 Comments

Appen to Exhibit at INTERSPEECH 2015 Conference in Dresden

Contact: Rachael Pappas, Appen

Language Technology Service Provider Appen Announces Expanded Offerings in Custom Linguistic Solutions

Sydney, Australia – September 2, 2015: Appen is pleased to announce that it will be exhibiting at the INTERSPEECH 2015 Conference in Dresden, Germany, September 6-10, in Booth 4.

Appen will be launching its expanded offerings in the area of on-demand linguist support provided by their Custom Linguistic Solutions (CLS) department.

CLS connects Appen’s clients with a premier network of native-speaker linguists and subject matter experts around the globe, for development, testing and customization of language technologies and applications. Appen’s solutions are tailored to fit their clients’ needs, whether they need on-site or remote support, on an ongoing basis or simply on demand.

For more information about Appen’s products and services, visit us at

About Appen: Appen is a global leader in language, search and social technology. With capability in more than 150 languages, we work with leading technology companies to expand into new global markets. Appen has offices in the US, Australia, and the Philippines, with remote employees all over the world. Appen was named in Forbes Magazine as one of the “Top 100 Companies Offering Flexible Jobs in 2015.”

By |September 4th, 2015|Press|0 Comments

Appen and Best Doctors Partnering in the IBM Watson Ecosystem

Contact: Rachael Pappas, Appen and Justin Joseph, Best Doctors

Appen and Best Doctors Partnering in the IBM Watson Ecosystem

Sydney, Australia and Boston, Mass – June 30, 2015: Appen, a global linguistic services firm, and Best Doctors, a global health services company that helps people get the right diagnosis and treatment plan, are proud to announce they have partnered through the IBM Watson Ecosystem to enhance the Best Doctors knowledge base of medical information.

Appen is a member of the Watson Ecosystem as a Talent Hub partner – a group of skilled professionals who provide specialized support to app-building partners to help them turn their creative visions into market-ready products and services. Specifically, Appen provides linguistic support for Watson application developers, including the development of language resources, such as question and answer pairs, lexical data and annotated corpora for training and tuning purpose.

“I am so pleased with our partnership with Appen,” said Cara Samia, DirectorProduct Management, Best Doctors. “Together, with IBM Watson, we are laying the crucial groundwork to enhance the Best Doctors knowledge base so that we may continue to improve and update our offering to Best Doctors members worldwide.”

“Years ago I started my career in IBM’s research division,” said Lisa Braden-Harder, Appen CEO. “Today, I’m excited to be a part of the future of technology, working together with IBM Watson and Best Doctors to integrate data, language and healthcare.”

Members of the Watson Ecosystem are bringing a variety of cognitive based apps to market to transform industries – from healthcare, to music, shopping and beyond. Developers and entrepreneurs building Watson apps have access to a variety of ecosystem resources such as the Watson Developer Cloud, and the Watson Talent Hub, among other resources.

About Appen
Appen is a […]

By |June 29th, 2015|Press|0 Comments

Appen’s Dorota Iskra to Speak at CLIN25 in Antwerp

Appen is proud to announce that Dorota Iskra will be speaking at the 25th Meeting of Computational Linguistics in the Netherlands (CLIN25) at the City Campus of the University of Antwerp on February 5-6, 2015.  Dorota’s presentation is entitled  “Approach to Non-standardised Languages in Asian and African Markets.”

As a global provider of language resources and linguistic services, Appen often has to work with languages where no established standard is present for orthography, vocabulary or grammar. This problem, although it may concern some minorities’ languages in Europe, becomes only apparent in its full dimensions when entering Asian and African markets. Examples are Arabic, Pashto, Urdu and various Indian languages.

This presentation describes the problems we have encountered when moving to Asian and African languages and proposes a methodology for establishing internal standards for non-standardised languages. In the initial phase of the project when creating orthographic transcription from speech we end up with multiple spellings for the same word. These are all added to the dictionary since there is no spell checker. Our linguistic expert checks the dictionary and makes decisions about which words should actually have the same spelling. As a result a rough spell checker is created which can be used in the next phase of the project. But because it is far from comprehensive, an exception list is kept. In multiple iterations the linguistic expert goes through the list and adds words to the dictionary.

Language technology expects consistency in the data which can only be achieved through a systematic approach in the face of lacking standards. The approach we propose has been developed and tested in a number of languages and with extensive volumes of data, not only by ourselves, but also by our clients who are […]

By |January 29th, 2015|Press|0 Comments

Appen’s Phil Hall to Speak at Mobile Voice Conference in San Jose

Appen is proud to announce that Phil Hall, SVP Language Resources, will be speaking at the Mobile Voice Conference in San Jose, California on April 20 and 21, 2015. Phil will be presenting on the topic of training data for speech recognition in a talk entitled, “The Value Versus Volume Continuum.”

Interested in learning more about how Appen works with speech recognition technology? Click here to read our case study on our work for the automotive industry.

By |January 29th, 2015|Press|0 Comments

Appen Named in Forbes Magazine as One of the Top 100 Companies Offering Telecommuting Jobs in 2015

Exciting news for Appen! For the second year in a row, Appen has been named in Forbes Magazine as one of the Top 100 Companies Offering Telecommuting Jobs. Forbes sourced its data from a list published by FlexJobs in January, 2015.

Following is an excerpt from the story: “The positive impact remote work options can have on a company’s bottom line and culture is increasingly being acknowledged by smart employers. The organizations on this list are among those leading the charge to show that remote work options are a part of successful 21st century workplaces,” says Sara Sutton Fell, founder and CEO of FlexJobs.

To view the story and the list on Forbes, please click here.

By |January 28th, 2015|Press|0 Comments

TAUS Review, January 2015 is Live!

The January 2015 issue of the TAUS Review, the online magazine published by TAUS, is live! Appen is a proud member of TAUS, a think tank and resource center for the global translation industry. To learn more about TAUS, click here, and to view the online magazine, click here.

By |January 19th, 2015|Press|0 Comments

Appen Live on the ASX on January 7, 2015

Appen was listed on the Australian Stock Exchange (APX on ASX) on January 7. This is a wonderful milestone in our history. Back in the 1990s most startups launched from garages with small, isolated teams but for Appen the path was two kitchens on far-flung shores. Julie Vonwiller and I both saw at the dawn of the internet, tools that could bridge distances and connect like-minded people towards empowering cultures with language technology. Appen was always powered by strong research roots and roll-up-your-sleeves partnering with technology teams. The company grew steadily benefiting from the injection of strong financial and business know-how, first coming from Chris Vonwiller and later Anacacia Captial, Bill Pulver and Mark Byrne.

As we move into the public space we want to thank two exiting board members for their past service, Julie and Jon Shein. Julie has contributed to the company in leading roles for nearly 20 years from founder to the board always bringing her strong background in linguistics. Jonny provided a strong financial background and a deep belief in the importance of employees which we will miss as well.

We are a unique company powered by awesome teamwork. Many pizza-powered companies of the 1990s have come and gone with waves of new technology but we are still growing and still taking the latest technology to global markets.

Lisa Braden-Harder

By |January 8th, 2015|Press, Uncategorized|0 Comments

Appen Secures Three New Deloitte Technology Awards

Appen has just secured three new industry awards. Appen Holdings was named a winner in the 2014 Deloitte Technology Fast 50 Awards. A four-time winner, Appen was named 50th on the list.

Appen Holdings was also named a winner in the Leadership Category of the Deloitte Technology Fast 50 Awards. Please find a link to the detailed report, outlining the winners’ profiles as well as the full list of 2014 winners.

Additionally, Appen was named a winner in the Deloitte Asia Pacific Technology Fast 500 Awards. Appen and other Australian companies have featured strongly, comprising 15% of the regional ranking – up from 13% last year. Please visit the Asia Pacific Technology Fast 500 website for the more information, and click here to view the entire list.

Thank you, and congratulations to all at Appen whose hard work resulted in these incredible achievements.

By |December 2nd, 2014|Press|0 Comments