MC2 2018 Lab

Milestones and timetable 2018

sanjuan — 2018-02-21T13:03:43Z

Registration opens: 8 february 2018 (Task2)
Registration closes: 30 April 2018
End Evaluation Cycle: 19 May 2018
Submission of Participant Papers [CEUR-WS]: 31 May 2018
Submission of Lab Overviews [LNCS]: 8 June 2018
Notification of Acceptance Participant Papers [CEUR-WS]: 15 June 2018
Notification of Acceptance Lab Overviews [LNCS]: 15 June 2018
Camera Ready Copy of Lab Overviews [LNCS]: 22 June 2018
Camera Ready Copy of Participant Papers and Extended Lab Overviews [CEUR-WS]: 29 June 2018
CEUR-WS Working Notes Preview for Checking by Authors and Lab Organizers: 18-24 July 2018
September 10-14 2018 CLEF 2018 Conference

Task objectives and Evaluation process

Jean-valère, olivier, sanjuan — 2018-02-21T12:50:00Z

Objective

Vodkaster ( http://www.vodkaster.com/ ) is a French social network about movies where participants can share comments about movies under the form of microcritics not longer than a tweet. The main differences are the restricted cultural domain and the form.
The objective of the task is for a given movie and microcitic and each language among French, English, Spanish, Portuguese and Arabic to provide a summary of the related microblogs.
Microblogs included is a summary should provide relevant information about at least one of the following aspects:

the film mentioned in the microcritic including subject, genre, presence in festivals,
reception, audience, critics and opinions as well as actors and producers careers.
events like festivals mentioned in the microcritic if any, including opinions and narratives.
comments and critics in twitter similar to those in the microcritic if any.
Extended summaries can include microblogs about closely related films and events.
Promotional, automatic tweets or retweets are not considered as relevant. However, retweets by movie aficionados or movie makers are relevant.

Task description

Browsing the VodKaster website allows french readers to get personal short comments (microcritics)
about movies. You can get similar and/or complementary opinions on twitter but they are less specific to movies and harder to find. The use case is to display to the reader a concise summary of microblogs related to the microcritics he/she is reading, considering bilingual and trilingual users that would read microblogs in other languages than French.
Summaries are exclusively made of extracts from microblog contents and can include author names if considered as informative. They should be readable and codes like external URLs and references to multimedia objects should be removed. Three different summary lengths in words are considered: 50, 150 and 250.
Summaries are intended to provide an idea of all relevant information included in the corpus.
Diversity among top ranked microblogs is important. If the summary does not provide any microblog directly related to the topic it suggests that there is none in the corpus.

Evaluation process

Runs will be primarily evaluated on informativeness following INEX Tweet
Contextualization methodology [1] and based on the FRESA 2 [2] software extended to Arabic, French, Portuguese and Spanish. All FRESA metrics will be computed between runs top ranked microblog extracts and a textual reference to be provided by organizers. Following [1], this reference will based on both manual runs and pools from participant submissions.
Graded standard q-rels for microblogs will be automatically generated based on FRESA [2] scores to be used with standard TREC eval tools. However, due to the impact of microblog high redundancy and reposts over q-rels exhaustivity [3], these measures won't be considered as official.
Alternative Nugget-based Information Retrieval Evaluation references and scores [4] will be also tentatively provided and discussed at the Lab.
Readability of results provided by systems will be also manually checked, the user case requiring these results to be displayed to the user.

[1] INEX Tweet Contextualization task : Evaluation, results and lesson learned
Patrice Bellot , Véronique Moriceau , Josiane Mothe, Eric SanJuan , Xavier Tannier :- Inf. Process.
Manage. 52(5) : 801-819 (2016)
[2] http://fresa.talne.eu
[3] Philippe Mulhem, Lorraine Goeuriot, Nayanika Dogra, Nawal Ould Amer: TimeLine Illustration
Based on Microblogs: When Diversification Meets Metadata Re-ranking. CLEF 2017 Proceedings.
Lecture Notes in Computer Science 10456, Springer 2017 : 224-235
[4] http://www.ccs.neu.edu/home/jaa/IIS-1256172/

Submission

Submitted summaries should be in TREC like format, a tabulated file with five fields:

a run ID
an integer indicating its position in the summary
a float number as an estimation of its relevance
the main language of the microblog content (fr, en, es, pt or ar)
an extract of the microblog content with the author name if considered as relevant

Runs will be truncated at 50, 150 and 300 words, content will be concatenated and displayed to evaluators that will highlight relevant passages. Therefore, the concatenation of content in the last column should be readable by a human (i.e. this column needs to be readable on its own).

Each team can submit up to three runs in each language (Arabic, English, French, Portuguese and Spanish). Teams will be invited to share there queries in different languages. Organizers will facilitate running submitted sets of queries on the following baseline systems.

Baseline system

A baseline system powered by Indri is provided to participants to run complex focus nested queries.
In this index, microblogs have been merge into XML documents per autor to allow expansions. Indri XML index permits to retrieve XML elements based on nested content.










Example:
20666489
727389569688178688
soulsurvivornl
en
Twitter for iPhone
2016-05-03
RT @ndnl: Dit weekend begon het Soul Surivor Festival.

727944506507669504
soulsurvivornl
en
Facebook
2016-05-04
Last van een festival-hangover?

Wikipedia XML corpus for summary generation

sanjuan — 2016-10-18T16:44:45Z

Wikipedia is under Creative Commons license, and its contents can be used to contextualize tweets or to build complex queries referring to Wikipedia entities.

We have extracted an average of 10 million XML documents from Wikipedia per year since 2012 in the four main twitter languages:- en, es, fr and pt.

These documents reproduce in an easy-to-use XML structure the contents of the main Wikipedia pages: title, abstract, section and subsections as well as Wikipedia internal links. Other contents such as images, footnotes and external links are stripped out in order to obtain a corpus easier to process using standard NLP tools.

By comparing contents over the years, it is possible to detect long term trends

View online : Micro Blog Contextualization CLEF & Inex tracks data and tools

The festival galleries dataset

sanjuan — 2016-10-18T16:31:57Z

This data set allows to experiment microblog search and stream summarization.

Microblog collection

The document collection is provided to registered participants by ANR GAFES project. It consists in a pool of more than 50M unique micro-blogs from different sources with their meta-information as well as ground truth for the evaluation.

The microblog collection contains a very large pool of public posts on Twitter using the keyword festival since June 2015. These micro-blogs are collected using private archive services based on streaming API. The average of unique microblog posts (i.e. without re-twitts) between June and September is 2, 616, 008 per month. The total number of collected micro-blog posts after one year (from May 2015 to May 2016) is 50, 490, 815 (24, 684, 975 without re-posts). These micro-blog posts are available online on a relational database with associated fields.

Because of privacy issues, they cannot be publicly released but can be analyzed inside the organization that purchased these archives and among collaborators under privacy agreement. The CM2 lab provides this opportunity to share this data among academic participants. These archives can be indexed, analyzed and general results acquired from them can be published without restriction.

Linked web pages

66% of the collected micro-blog posts contain Twittert.co compressed URLs. Sometimes these URLs refer to other online services like adf.ly, cur.lv, dlvr.it, ow.ly that hide the real URL. We used the spider mode of the GNU wget tool to get the real URL, this process required multiple DNS requests.

The number of unique uncompressed urls collected in one year is 11,580,788 from 641,042 distinct domains.

Getting access to the data set for scholars

register your institution to CLEF
send a request by email to admin@talne.eu from the same domain as your institution with full contact information.
if accepted, you will receive a confidential agreement to be approved by your institution.
once we get back the agreement you will receive personal information to access lab data servers.

View online : Cultural micro-blog Contextualization 2016 Workshop Overview: data and pilot tasks

Microblog Cultural Contextualization 2017 lab introduction

sanjuan — 2016-10-18T12:38:54Z

These are the slides used to presented at CLEF 2016 in Evora to introduce the CM2 lab.

Overall Procedure

Take a microblog about an event with an url.
Identify its language.
Identify a related cultural event or filter it out.
Reveal When, Where, Who ...
Relate it to Wikipedia entities

2017 Organization

Task 1: language, filtering and localization lead by Toulouse, Montréal and Paris starts … now!
Task 2: entity extraction, summarization and linking starts in November 2016 lead by Avignon, London University and Syllabs.
Task 3: time-line illustration starts in January 2017 lead by Grenoble.

Microlog Data Set

sanjuan — 2015-11-02T08:08:38Z

The document collection provided by GAFES project consists a pool of more than 70M unique microblogs from different sources with their meta-information and expanded URLs on a MySQL server. Due to legal terms the access to this database is restricted to registered participants under privacy agreement.
Along with the microblog corpus, a clean simplified xml dump of wikipedia easy to index and to process with state of the art NLP tools is made available to participants. Ground truth (…)

- 2 - MicroBlog Search / data

Evaluation Methodology

sanjuan — 2015-11-02T07:40:36Z

Systems will be evaluated mainly on informativeness and relevance, but readability and ergonomy will be also checked. Informativeness evaluation will rely on textual references established by experts in project GAFES, following the strict methodology
at CLEF-INEX tweet contextualization track (http://inex.mmci.uni-saarland.de/tracks/qa/). Readability and ergonomy would be carried out on the output for specific festivals based on questionnaires to be filled out by lab participants. Best systems will have the opportunity to be experimented in july 2016 for real with the support of the label French Tech Culture (http://frenchculture.org/digital-cultures).

Therefore, informativeness and relevance evaluation will be automatic and reproducible while readability and ergonomy would only be available for lab participants. All systems will be required to run on a dedicated LINUX server (allowing virtual machines) provided by organizers to will have to run in real time (maximum 5s per query). Access to full micro blog data will only be authorized for applications running on this server.