MC2 2018 Lab

Wikipedia XML corpus for summary generation

sanjuan — 2016-10-18T16:44:45Z

Wikipedia is under Creative Commons license, and its contents can be used to contextualize tweets or to build complex queries referring to Wikipedia entities.

We have extracted an average of 10 million XML documents from Wikipedia per year since 2012 in the four main twitter languages:- en, es, fr and pt.

These documents reproduce in an easy-to-use XML structure the contents of the main Wikipedia pages: title, abstract, section and subsections as well as Wikipedia internal links. Other contents such as images, footnotes and external links are stripped out in order to obtain a corpus easier to process using standard NLP tools.

By comparing contents over the years, it is possible to detect long term trends

View online : Micro Blog Contextualization CLEF & Inex tracks data and tools

The festival galleries dataset

sanjuan — 2016-10-18T16:31:57Z

This data set allows to experiment microblog search and stream summarization.

Microblog collection

The document collection is provided to registered participants by ANR GAFES project. It consists in a pool of more than 50M unique micro-blogs from different sources with their meta-information as well as ground truth for the evaluation.

The microblog collection contains a very large pool of public posts on Twitter using the keyword festival since June 2015. These micro-blogs are collected using private archive services based on streaming API. The average of unique microblog posts (i.e. without re-twitts) between June and September is 2, 616, 008 per month. The total number of collected micro-blog posts after one year (from May 2015 to May 2016) is 50, 490, 815 (24, 684, 975 without re-posts). These micro-blog posts are available online on a relational database with associated fields.

Because of privacy issues, they cannot be publicly released but can be analyzed inside the organization that purchased these archives and among collaborators under privacy agreement. The CM2 lab provides this opportunity to share this data among academic participants. These archives can be indexed, analyzed and general results acquired from them can be published without restriction.

Linked web pages

66% of the collected micro-blog posts contain Twittert.co compressed URLs. Sometimes these URLs refer to other online services like adf.ly, cur.lv, dlvr.it, ow.ly that hide the real URL. We used the spider mode of the GNU wget tool to get the real URL, this process required multiple DNS requests.

The number of unique uncompressed urls collected in one year is 11,580,788 from 641,042 distinct domains.

Getting access to the data set for scholars

register your institution to CLEF
send a request by email to admin@talne.eu from the same domain as your institution with full contact information.
if accepted, you will receive a confidential agreement to be approved by your institution.
once we get back the agreement you will receive personal information to access lab data servers.

View online : Cultural micro-blog Contextualization 2016 Workshop Overview: data and pilot tasks

Microlog Data Set

sanjuan — 2015-11-02T08:08:38Z

The document collection provided by GAFES project consists a pool of more than 70M unique microblogs from different sources with their meta-information and expanded URLs on a MySQL server. Due to legal terms the access to this database is restricted to registered participants under privacy agreement.
Along with the microblog corpus, a clean simplified xml dump of wikipedia easy to index and to process with state of the art NLP tools is made available to participants. Ground truth (…)

- 2 - MicroBlog Search / data