Crowdsourcing a word-emotion association lexicon

  1. (PDF, 447 KB)
  2. Get@NRC: Crowdsourcing a word-emotion association lexicon (Opens in a new window)
DOIResolve DOI:
AuthorSearch for: ; Search for:
Journal titleComputational Intelligence
Pages436465; # of pages: 30
Subjectaffect; Crowdsourcing; emotion lexicon; emotions; Mechanical turks; polarity lexicon; Semantic orientation; Sentiment analysis; Artificial intelligence; Computational methods; Semantics
AbstractEven though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper, we show how the combined strength and wisdom of the crowds can be used to generate a large, high-quality, word-emotion and word-polarity association lexicon quickly and inexpensively. We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help to identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help to obtain annotations at sense level (rather than at word level). We conducted experiments on how to formulate the emotion-annotation questions, and show that asking if a term is associated with an emotion leads to markedly higher interannotator agreement than that obtained by asking if a term evokes an emotion.
Publication date
AffiliationInformation and Communication Technologies; National Research Council Canada
Peer reviewedYes
NPARC number21270400
Export citationExport as RIS
Report a correctionReport a correction
Record identifier4d626836-c350-421f-9775-f549024097fd
Record created2014-02-07
Record modified2016-05-09
Bookmark and share
  • Share this page with Facebook (Opens in a new window)
  • Share this page with Twitter (Opens in a new window)
  • Share this page with Google+ (Opens in a new window)
  • Share this page with Delicious (Opens in a new window)
Date modified: