Evaluation is a mandatory task for Information Retrieval research. Under the Cranfield paradigm evaluation needs of document collections, topics and relevance judgments. The creation of those elements is a time and resource consuming process. At the same time, new tasks and models are continuously appearing, so it emerges the need of creating new test collections. In this paper we present a platform to easily and cheaply build datasets for Information Retrieval evaluation without the need of organizing expensive evaluation campaigns. In particular, we propose the simulation of participant systems and the use of intelligent pooling strategies to make the most of the assessor's work. Our platform is aimed to cover the whole process of building the test collection, from document gathering to judgment creation.
This work was supported by project RTI2018-093336-B-C22 (MCIU& ERDF), project GPC ED431B 2019/03 (Xunta de Galicia & ERDF), and accreditation ED431G 2019/01 (Xunta de Galicia & ERDF)