Evaluation Methodology

Systems will be evaluated mainly on informativeness and relevance, but readability and ergonomy will be also checked. Informativeness evaluation will rely on textual references established by experts in project GAFES, following the strict methodology
at CLEF-INEX tweet contextualization track (http://inex.mmci.uni-saarland.de/tracks/qa/). Readability and ergonomy would be carried out on the output for specific festivals based on questionnaires to be filled out by lab participants. Best systems will have the opportunity to be experimented in july 2016 for real with the support of the label French Tech Culture (http://frenchculture.org/digital-cultures).

Therefore, informativeness and relevance evaluation will be automatic and reproducible while readability and ergonomy would only be available for lab participants. All systems will be required to run on a dedicated LINUX server (allowing virtual machines) provided by organizers to will have to run in real time (maximum 5s per query). Access to full micro blog data will only be authorized for applications running on this server.