Crowdsourcing a Gold Standard for Medical Relation Extraction with CrowdTruth


In this paper, we make the following contributions: (1) a comparison of the quality and efficacy of annotations for medical relation extraction provided by both crowd and medical experts, showing that crowd annotations are equivalent to those of experts, with appropriate processing; (2) an openly available dataset of 900 English sentences for medical relation extraction, centering primarily on the cause relation, that have been processed with disagreement analysis and by experts.