Designing a Scalable Crowdsourcing Platform

Chris Van Pelt

Agenda

Context

What is Crowdsourcing?

Microtasks

Online tasks, requiring little to no domain specific knowledge, that take a few minutes to complete.

Content
Moderation

Does this image meet the following requirements?

Content Analysis

Does this tweet mention McDonalds in a positive light?

Content Generation

What's a good question about back pain?

Categorization

What category does this product belong to?

Verification

Is this business still open, and what's their phone number?

Tools

Units, CML and aggregation

Units

Simple CML


<img src="{{url}}" />
<cml:checkbox label="Is the image appropriate?" 
  default="no" value="yes" />
  

Advanced CML


<img src="{{url}}" />
<cml:ratings label="How attractive is this young man" 
  from="Hot" to="Not" validates="required" golden="true" />
  

  <cml:checkboxes label="Choose category"
    aggregation="agg_3" />
    
    <cml:checkbox label="Tablet" />
    <cml:checkbox label="Laptop" />
    <cml:checkbox label="PC" />
    ...
    
  </cml:checkboxes>
      

Aggregation

Quality

Speed

Cost

Quality

Gold, peer review, and trust

Gold

Training and Feedback

Open-ended tasks

Peer review

Trust

Contributor agreement weighted by trust

Confidence

Cost

The Microtask marketplace

Competition

Profiles

Bonuses

Speed

Workflows, channels, elastic supply

Concurrency

Virtual Marketplaces

Cellphone minutes

In-Game Virtual Currency

Elastic supply

0

The Future

RPC's

Thanks!

Chris Van Pelt