ensiwiki-2011 dataset for readability modelling

  Creators

Van der Sluis, Frans

  Item identifier

http://hdl.handle.net/20.500.12115/49

 Referenced by

https://doi.org/10.1002/asi.23095
https://doi.org/10.3990/1.9789036505673

 Date issued

2011-08-03

 Type

corpus

 Size

138790 articles

 Language(s)

English

 Description

The ensiwiki dataset contains Wikipedia pages sampled from Simple-English and regular English Wikipedia. For each Simple-English page, a paired page was sampled from the regular English Wikipedia if available. The result is a list of pairs between Simple-English and regular English pages. Only pages that form a pair were included. In total 138,790 pages were sampled from Simple-English Wikipedia and English Wikipedia from August, 2011. The purpose of this dataset is to train and test readability detection systems. The dataset is intended to be sufficiently large to detect intricate relations between different features of readability. The dataset is used for this purpose in Van der Sluis (2013, 2014) and is described in further detail in Van der Sluis (2013). The dataset furthermore contains plain text versions of the wiki-text pages. These were parsed using JWPL MediaWiki parser (see https://dkpro.github.io/dkpro-jwpl/JWPLParser/) and split to the level of articles, sections, and paragraphs. Only the oldest 38,955 wikitext pages were parsed in order to arrive at a more mature set of pages that more clearly distinguishes between different levels of readability, which proved superior for training readability models. Note: This data is a result of work done at the Human-Media Interaction group of the University of Twente, The Netherlands. It's release is in accordance with original licensing requirements and aligned with relevant parties.

 Publisher

University of Copenhagen

 Acknowledgement

7th Framework ICT Programme of the European Union.

Project code: FP7-ICT-2007-3

Project name: PuppyIR

 Subject(s)

readability textual complexity wikipedia simple english

 Collection(s)

CLARIN-DK-UCPH Repository

Show full item record