2- Mining opinion argumentation

Objective

This task aims to automatically identify reason-conclusion structures that can lead to model social web user’s positions about an culture event expressed through the social networks platforms Twitter. The idea is to perform a search process that focus on claims about a festival name out in a massive collection. The goal is to provide users with direct access to relevant information. In this task, relevant information is expressed in the form of a summary of argumentative tweets about a culture event.

Task description

Use case

Given, a set of festivals name from most popular festivals on FlickR English and French language, participants have to search for the most argumentative tweets in a collection covering 18 months of news about festivals in different languages. The identified tweets have to be a summary of ranked tweets according to their probability of being argumentative tweets. Such sets of tweets could be treated easier, on priority, by a festival organiser.
For each language ( English and French ), a monolingual scenario is expected : Given a festival name from topics file, participants have to search, from the microblog collection, the set of the most argumentative tweets in the same query language.
Samples of argumentative Tweets are provided here: English_Sample, French_Sample

Topics

English and French are a respectively 12 and 4 festival name. They represent a set of some popular festivals on FlickR for which we have pictures. Topics were carefully selected by organizer to ensure that selected topics have enough related argumentative tweets in our corpus. Such manual selection was conduct to ensure a possible evaluation.

Microblog Corpus

A login is required to access the data, once registered to CLEF

• The complete stream of 70 000 000 microblogs is available here for registered participants.
• An indri Index with a web interface are available to query the whole set of microblogs

Evaluation

The official evaluation measures planned are: NDCG and Pyramid

1. NDCG : This ranking measures will give a score for each retrieved tweet with a discount function over the rank. As we are mostly interested in top ranked arguments, this ranking measures meet our expectation.
2. Pyramid evaluation protocol : This evaluation protocol was chosen to evaluate how much the identified set of argumentative tweets about a festival name is diversified. In fact, participant results are expected to cover a variety of arguments given by a user about a culture event. Such evaluation protocol will allow us to determine if the identified summary of ranked tweets express the same content in different words or involve different arguments about a given festival name.

Result Submission

The runs must respect the classical trec top files format. Only the top 100 results for each query run must be given. Each run in each language, must contain 4 fields:

• Id : a long integer representation of the unique identifier of this Tweet
• Scores : The probability of being an argument tweet accorded by participant system
• Rank : The accorded position of the tweet in the grading list of argument tweets
• Content: Microblog textual content

Important constraint: Participant must respect the variation criteria in their submission. The more a run detect different argument about a culture event the more it is interesting.

Schedule

• Registration closes: 30 April 2018
• End Evaluation Cycle: 04 June 2018
• Submission of Participant Papers [CEUR-WS]: 08 June 2018
• Notification of Acceptance Participant Papers: 15 June 2018
• Camera Ready Copy of Participant Papers: 29 June 2018
• September 10-14 2018 CLEF 2018 Conference

Task organizers (contact : malek.hajjem@univ-avignon.fr)

• Chiraz Latiri (ISAMM, Tunis)
• Julio Gonzalo (UNED, Madrid)
• Malek Hajjem (LIA, Avignon, malek.hajjem@univ-avignon.fr)