MC2 2018 Lab

Dialect detection in Informal Arabic Text

Malek Hajjem — 2018-04-24T15:24:21Z

Content Analysis Results: Language identification 2017

Malek Hajjem — 2018-03-15T09:00:54Z

Results

Topics are a random selection of original microblogs posted in June 2016 without external links and with more then 80 characters.
Submissions and scores for the two best teams can be found here Syllabs and Lia.
The task paper can be found here

@inproceedings{DBLP:conf/clef/ErmakovaMS17, author = {Liana Ermakova and Josiane Mothe and Eric SanJuan}, title = {{CLEF} 2017 Microblog Cultural Contextualization Content Analysis task Overview}, booktitle = {Working Notes of {CLEF} 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14, 2017.}, year = {2017}, crossref = {DBLP:conf/clef/2017w}, url = {http://ceur-ws.org/Vol-1866/invited_paper_14.pdf}, timestamp = {Thu, 16 Nov 2017 14:36:59 +0100}, biburl = {https://dblp.org/rec/bib/conf/clef/ErmakovaMS17}, bibsource = {dblp computer science bibliography, https://dblp.org}
}

Evaluation process

The Evaluation process detects the reliability of the language on Twitter.
In fact, Tweet objects have a long list of ‘root-level' attributes, including fundamental attributes such as "lang". When present, this attribute indicates a BCP 47 language identifier corresponding to the machine-detected language from where the microblog was edited. Obviously the machine-detected language may be different from the microblog langage.
Scores in this evaluation are assigned by a human expert. Only the tweets where the results of participants' language detector systems differ from tweet's "lang" attribute were examined. Tweets in several languages have a graduated score describing how much a language is present on it.

Available ressources Clef 2018: detailed description

Malek Hajjem — 2018-03-14T16:09:54Z

The festival galleries dataset

A massive collection of microblogs and urls related to culture festivals are provided for registered participants here .
In order to deal with such large dataset we propose different format :

A CSV format : It is a tab-separated CSV file that could be useful in case of managing dataset via a Mysql database or python programming langague.

An XML format for Indri: This format could be smoothly indexed with Indri in case of need. With tweet textual content some metadata ( see description above ) is also provided. We note that XML files are grouped by author.

The festival galleries dataset is presented partially or totally. In case of a partial format, each csv file contains gathered tweets by month. Original tweets are separated from rediffused tweets to manage lighter files.

festival
Originals: Re posts: 1- 2015-05(72M) 2015-05(54M)
2- 2015-06(235M) 2015-06(190M)
3- 2015-07(220M) 2015-07(162M)
... ...
... ...
... ...
... ...
18- 2016-10(102M) 2016-10(148M)

HTML form to test queries: this form make you able to test the Microblog search baseline system using an Indri query

*Simple queries:
For a basic query, just type in the terms you wish to search on. Each term will be weighed equally and combined in an "or" fashion.
- hiphop jazz #combine(hiphop jazz ) *Phrase Matching: To search for a specific phrase (i.e. "hiphop jazz"), you can wrap your terms using the ordered window operator #n (where n is the window size of the number of terms). #1(hiphop jazz) Your search results would return only those documents where the terms "hiphop" and "jazz" appear in order. *Unordered Windows
The #uwN operator performs a search on terms that occur within a certain window size. For example, if we wanted to look for the terms "hiphop" and "jazz" that occured within 2 terms of each other, but we did not care if the term "hiphop" came before "jazz" or not, we would write this as: #uw2(hiphop jazz) *Boolean Searches By default, the Indri will return a document if any of the terms occur in the document; documents that contain more terms will generally be ranked above documents that contain fewer terms. If you wish to specify that all of your search terms must be included, you can use the "boolean and" operator (#band). For example, if you want to ensure that the terms "hiphop" and "jazz" both exist, use: #band(hiphop jazz)

PERL API used to interroge the web service locally with suitable query in Indri language
Indri parameter files : A parameter file in XML format useful to reindex the collection with Indri
Compressed Indri Indexes per month
Programs to generate xml repositories from CSV ordered data
Root of Indri indexes and data

Links

An uncompressed list of tweets url is available for participants in csv format. This metadata could be used to explore more the tweet content.

Milestones and timetable 2018

sanjuan — 2018-02-21T13:03:43Z

Registration opens: 8 february 2018 (Task2)
Registration closes: 30 April 2018
End Evaluation Cycle: 19 May 2018
Submission of Participant Papers [CEUR-WS]: 31 May 2018
Submission of Lab Overviews [LNCS]: 8 June 2018
Notification of Acceptance Participant Papers [CEUR-WS]: 15 June 2018
Notification of Acceptance Lab Overviews [LNCS]: 15 June 2018
Camera Ready Copy of Lab Overviews [LNCS]: 22 June 2018
Camera Ready Copy of Participant Papers and Extended Lab Overviews [CEUR-WS]: 29 June 2018
CEUR-WS Working Notes Preview for Checking by Authors and Lab Organizers: 18-24 July 2018
September 10-14 2018 CLEF 2018 Conference

Task objectives and Evaluation process

Jean-valère, olivier, sanjuan — 2018-02-21T12:50:00Z

Objective

Vodkaster ( http://www.vodkaster.com/ ) is a French social network about movies where participants can share comments about movies under the form of microcritics not longer than a tweet. The main differences are the restricted cultural domain and the form.
The objective of the task is for a given movie and microcitic and each language among French, English, Spanish, Portuguese and Arabic to provide a summary of the related microblogs.
Microblogs included is a summary should provide relevant information about at least one of the following aspects:

the film mentioned in the microcritic including subject, genre, presence in festivals,
reception, audience, critics and opinions as well as actors and producers careers.
events like festivals mentioned in the microcritic if any, including opinions and narratives.
comments and critics in twitter similar to those in the microcritic if any.
Extended summaries can include microblogs about closely related films and events.
Promotional, automatic tweets or retweets are not considered as relevant. However, retweets by movie aficionados or movie makers are relevant.

Task description

Browsing the VodKaster website allows french readers to get personal short comments (microcritics)
about movies. You can get similar and/or complementary opinions on twitter but they are less specific to movies and harder to find. The use case is to display to the reader a concise summary of microblogs related to the microcritics he/she is reading, considering bilingual and trilingual users that would read microblogs in other languages than French.
Summaries are exclusively made of extracts from microblog contents and can include author names if considered as informative. They should be readable and codes like external URLs and references to multimedia objects should be removed. Three different summary lengths in words are considered: 50, 150 and 250.
Summaries are intended to provide an idea of all relevant information included in the corpus.
Diversity among top ranked microblogs is important. If the summary does not provide any microblog directly related to the topic it suggests that there is none in the corpus.

Evaluation process

Runs will be primarily evaluated on informativeness following INEX Tweet
Contextualization methodology [1] and based on the FRESA 2 [2] software extended to Arabic, French, Portuguese and Spanish. All FRESA metrics will be computed between runs top ranked microblog extracts and a textual reference to be provided by organizers. Following [1], this reference will based on both manual runs and pools from participant submissions.
Graded standard q-rels for microblogs will be automatically generated based on FRESA [2] scores to be used with standard TREC eval tools. However, due to the impact of microblog high redundancy and reposts over q-rels exhaustivity [3], these measures won't be considered as official.
Alternative Nugget-based Information Retrieval Evaluation references and scores [4] will be also tentatively provided and discussed at the Lab.
Readability of results provided by systems will be also manually checked, the user case requiring these results to be displayed to the user.

[1] INEX Tweet Contextualization task : Evaluation, results and lesson learned
Patrice Bellot , Véronique Moriceau , Josiane Mothe, Eric SanJuan , Xavier Tannier :- Inf. Process.
Manage. 52(5) : 801-819 (2016)
[2] http://fresa.talne.eu
[3] Philippe Mulhem, Lorraine Goeuriot, Nayanika Dogra, Nawal Ould Amer: TimeLine Illustration
Based on Microblogs: When Diversification Meets Metadata Re-ranking. CLEF 2017 Proceedings.
Lecture Notes in Computer Science 10456, Springer 2017 : 224-235
[4] http://www.ccs.neu.edu/home/jaa/IIS-1256172/

Submission

Submitted summaries should be in TREC like format, a tabulated file with five fields:

a run ID
an integer indicating its position in the summary
a float number as an estimation of its relevance
the main language of the microblog content (fr, en, es, pt or ar)
an extract of the microblog content with the author name if considered as relevant

Runs will be truncated at 50, 150 and 300 words, content will be concatenated and displayed to evaluators that will highlight relevant passages. Therefore, the concatenation of content in the last column should be readable by a human (i.e. this column needs to be readable on its own).

Each team can submit up to three runs in each language (Arabic, English, French, Portuguese and Spanish). Teams will be invited to share there queries in different languages. Organizers will facilitate running submitted sets of queries on the following baseline systems.

Baseline system

A baseline system powered by Indri is provided to participants to run complex focus nested queries.
In this index, microblogs have been merge into XML documents per autor to allow expansions. Indri XML index permits to retrieve XML elements based on nested content.










Example:
20666489
727389569688178688
soulsurvivornl
en
Twitter for iPhone
2016-05-03
RT @ndnl: Dit weekend begon het Soul Surivor Festival.

727944506507669504
soulsurvivornl
en
Facebook
2016-05-04
Last van een festival-hangover?

More about use case, data and evaluation process

Chiraz Latiri, Julio Gonzalo, Malek Hajjem — 2018-02-09T09:39:44Z

Detailed description

use case

Given, a selected of festivals name from popular festivals on FlickR English and French language, participants have to search for the most argumentative tweets in a collection covering 18 months of news about festivals in different languages. The identified tweets have to be a summary of ranked tweets according to their probability of being argumentative tweets. This use case was proposed to help festival organiser treating such set of tweets on priority. That is why the more the summary of ranked tweets is variant the in term of argumentation the more the run is useful.
For each language English and French (English and French), a monolingual scenario is expected : Given a festival name from topics file, participants have to search, from the microblog collection, the set of the most argumentative tweets in the same query language.
Samples of argumentative Tweets are provided here: English_Sample, French_Sample

Topics

English and French contain respectively 12 and 4 festival name. They represent a set of some popular festivals on FlickR for which we have pictures. Topics were carefully selected by organizer to ensure that selected topics have enough related argumentative tweets in our corpus. Such manual selection was conduct to to ensure a possible evaluation.

The choice of FlickR as source of topic was motivated by the fact that such social media platform had a high quality amateur pictures. This personal involvement serves our goal as we are interested mainly to personal tweets.

Microblog Corpus

A login is required to access the data, once registered to CLEF

The complete stream of 70 000 000 microblogs is available here for registered participants. This document collection is provided by GAFES. Microblogs are provided with their meta-information and expanded URLs on a MySQL server.
Due to legal terms the access to this database is restricted to registered participants under privacy agreement.
An [indri Index with a web interface-https://mc2.talne.eu/data/clef/api] are available to query the whole set of microblogs

Evaluation

The official evaluation measure is NDCG.

This ranking measures will give a score for each retrieved tweet with a discount function over the rank. As we are mostly interested in top ranked arguments, this ranking measures meet our expectation.
This measure was also used in TREC Microblog Track [1]. A tweet is considered as highly relevant when it is a personal and contains an argument that directly refers to the festival (topic).

[1] Overview of the TREC-2015 Microblog Track
Jimmy Lin,Miles Efron, Yulu Wang, Garrick sherman, Ellen Voorhees

Result Submission
The runs must respect the classical trec top files format as describe above. Only the top 100 results for each query run must be given. Each run in each language, must contain 3 fields:
– Id : a long integer representation of the unique identifier of this Tweet
– Scores : The probability of being an argument tweet accorded by participant system
– Rank : The accorded position of the tweet in the grading list of argument tweets
– Content: Microblog textual content

Diversity criteria:
The more a run detects different arguments about a cultural event, the more it is interesting.

Exemples about "Cannes festival name:
– I ve seen some people saying they're boycotting Cannes because of the high heels rule. I'm not sure they'll notice.
– Not going to lie, one of my favorite things about the Cannes festival is all of these handsome men in tuxedos.
– Cannes is relevant because movies get timed standing ovations.

How to get the data?

To get an access to the Microblog corpus, email malek.hajjem@univ-avignon.fr or registered to CLEF
The English topics can be downloaded here
The French topics can be downloaded here.

Contact Information

If you have any question, email us through this address mail : malek.hajjem@univ-avignon.fr

Towards Argumentative Ranking

Chiraz Latiri, Julio Gonzalo, Malek Hajjem — 2018-02-08T18:17:15Z

Organizers:

Chiraz Latiri, Julio Gonzalo, Malek Hajjem

Task 2 participation deadline April 30, 2018

Argumentative Ranking of Microblogs

Argumentation mining is a new problem in corpus-based text analysis that addresses the challenging task of automatically identifying the justifications provided by opinion holders for their judgment. Several approaches of argumentation mining have been proposed so far in areas such as legal documents, on-line debates, product reviews, newspaper articles and court cases, as well as in dialogical domains.
With the popularization of social networks, argumentation mining is considered as an extension of the opinion mining issue from social network content. The aim is to automatically identify reason-conclusion structures that can lead to model social web user's positions about a service or an event expressed through social networks platforms like Twitter. Indeed, when we need to form an opinion on a new topic or make a decision, arguments will be all what we are looking for.
To make argumentation structures available, in case of Twitter, a robust automatic recognition of it is required, based on resources that should be created in a reproducible fashion to be reliable. However, the ambiguity of natural language text produced in social media, with different writing styles, implicit context and heterogeneous content make argumentation, on Twitter, very challenging.

Another possible way to pick up the argumentation structures, from a generic tweet corpus, is to use approaches based on information extraction. The idea is to perform a search process that focus on claims about a given topic out in a massive collection. This approach relates to the field of focused retrieval, that aims to provide users with direct access to relevant information in retrieved documents. In this task, relevant information is expressed in the form of arguments. [1]

Success of such argumentation ranking will require interdisciplinary approaches based on the combination of different research issues. In fact, to better understand a short text and be able to detect the argumentative structures within a microblog, we could restore a « text contextualization » as a way to provide more information on the corresponding text [2]. Providing such information in order to detect argumentative tweets, would highlight relevant ones, in other words, tweets expressed in the form of arguments. Thus, argumentation mining in this situation will tend to act in the same way of an Information Retrieval (IR) system where potential argumentative tweets had to come first. Similar approach that addresses such purpose is presented in [3], where the output of the priority task will be a ranking of tweets according to their probability of being a potential threat to the reputation of some entity.

[1] Argumentative Ranking
Marco Lippi and Paolo Sarti and Paolo Torroni DISI - Universita degli Studi di Bologna Proceedings of Natural Language Processing meets Journalism - IJCAI-16 Workshop (NLPMJ 2016), New York, (July 2016)
[2] INEX Tweet Contextualization task : Evaluation, results and lesson learned
Patrice Bellot, Véronique Moriceau, Josiane Mothe, Eric SanJuan, Xavier Tannier:- Inf. Process. Manage. 52(5): 801-819 (2016)
[3] Overview of RepLab 2013: Evaluating Online Reputation Monitoring Systems
Enrique Amigo, Jorge Carrillo de Albornoz, Irina Chugur, Adolfo Corujo Julio Gonzalo, Tamara Martin, Edgar Meij Maarten de Rijke and Damiano Spina

TimeLine Illustration based on Microblogs

Lorraine, Philippe — 2016-10-19T19:42:26Z

This paper by Nayanika DOGRA, Philippe MULHEM, Nawal OULD AMER, and Lorraine GOEURIOT presents the approach used by the LIG-MRIM research group to the participation of the pilot task TimeLine illustration based on Microblogs for the 2016 CLEF Cultural Microblog Contextualization WorkShop that lead to the 2017 lab.

View online : LIG at CLEF 2016 Cultural Microblog Contextualization: TimeLine Illustration based on Microblogs

Wikipedia XML corpus for summary generation

sanjuan — 2016-10-18T16:44:45Z

Wikipedia is under Creative Commons license, and its contents can be used to contextualize tweets or to build complex queries referring to Wikipedia entities.

We have extracted an average of 10 million XML documents from Wikipedia per year since 2012 in the four main twitter languages:- en, es, fr and pt.

These documents reproduce in an easy-to-use XML structure the contents of the main Wikipedia pages: title, abstract, section and subsections as well as Wikipedia internal links. Other contents such as images, footnotes and external links are stripped out in order to obtain a corpus easier to process using standard NLP tools.

By comparing contents over the years, it is possible to detect long term trends

View online : Micro Blog Contextualization CLEF & Inex tracks data and tools

The festival galleries dataset

sanjuan — 2016-10-18T16:31:57Z

This data set allows to experiment microblog search and stream summarization.

Microblog collection

The document collection is provided to registered participants by ANR GAFES project. It consists in a pool of more than 50M unique micro-blogs from different sources with their meta-information as well as ground truth for the evaluation.

The microblog collection contains a very large pool of public posts on Twitter using the keyword festival since June 2015. These micro-blogs are collected using private archive services based on streaming API. The average of unique microblog posts (i.e. without re-twitts) between June and September is 2, 616, 008 per month. The total number of collected micro-blog posts after one year (from May 2015 to May 2016) is 50, 490, 815 (24, 684, 975 without re-posts). These micro-blog posts are available online on a relational database with associated fields.

Because of privacy issues, they cannot be publicly released but can be analyzed inside the organization that purchased these archives and among collaborators under privacy agreement. The CM2 lab provides this opportunity to share this data among academic participants. These archives can be indexed, analyzed and general results acquired from them can be published without restriction.

Linked web pages

66% of the collected micro-blog posts contain Twittert.co compressed URLs. Sometimes these URLs refer to other online services like adf.ly, cur.lv, dlvr.it, ow.ly that hide the real URL. We used the spider mode of the GNU wget tool to get the real URL, this process required multiple DNS requests.

The number of unique uncompressed urls collected in one year is 11,580,788 from 641,042 distinct domains.

Getting access to the data set for scholars

register your institution to CLEF
send a request by email to admin@talne.eu from the same domain as your institution with full contact information.
if accepted, you will receive a confidential agreement to be approved by your institution.
once we get back the agreement you will receive personal information to access lab data servers.

View online : Cultural micro-blog Contextualization 2016 Workshop Overview: data and pilot tasks