Addendum CSCW 2015 Submission
Submission details
Title: Crowdsourcing Local Entity Tagging for Artwork Annotation Tasks
Submission #735
Abstract
Cultural heritage institutions more and more provide their collections online. To make them suitable for online access, the collections need detailed annotations of different aspects of the collections. Creating such annotations requires a variety of knowledge and expertise which is not always possessed by the collection curators. The usage of crowdsourcing techniques could provide the cultural heritage institutions with new tools that are able to cater for the costly task (both in terms of time and money) to create high-quality annotations. Artwork annotation is an example of an important class of image annotation tasks with special requirements for the annotators in order to be successfully completed. The special difficulties in artwork annotation relate to the identification and labeling of the entities depicted in the artworks. In this paper, we study how the scope of annotation tasks affects the precision of entity identification and the quality of the provided labels in a crowdsourcing setting. The assumption is that assigning labels to localized regions in an image provides requesters with a more fine-grained description of the annotated image, and thus enables more powerful navigation and retrieval functionalities for their collections. Based on a real-life case study from Rijksmuseum Amsterdam, the contributions from this paper includes a detailed analysis of the entity identification and recognition performance of a crowd annotating artworks with different image annotation techniques, a novel method for the automatic aggregation of local image annotations generated by multiple workers, a comparison of the quality of crowd generated annotations with respect to the ones provided by domain-specific experts, and a dataset with local (i.e. bounding boxes) and global expert and crowd annotations to be used for further study.
Dataset
Dataset containing the prints and the results from the Global and Local annotation tasks.
CSCW 2015 submission 735 dataset
Algorithm for merging bounding boxes
https://github.com/yangjiera/mergeBB
CrowdFlower template files
These files in these archives can be used to set up a task exactly as we used in our experiments.