ParaCrawl : web-scale acquisition of parallel corpora

Bañón, Marta and Chen, Pinzhen and Haddow, Barry and Heafield, Kenneth and Hoang, Hieu and Esplà-Gomis, Miquel and Forcada, Mikel and Kamran, Amir and Kirefu, Faheem and Koehn, Philipp and Ortiz-Rojas, Sergio and Pla, Leopoldo and Ramírez-Sánchez, Gema and Sarrías, Elsa and Strelec, Marek and Thompson, Brian and Waites, William and Wiggins, Dion and Zaragoza, Jaume; (2020) ParaCrawl : web-scale acquisition of parallel corpora. In: ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Proceedings of the Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics (ACL), USA, pp. 4555-4567. ISBN 9781952148255 (https://doi.org/10.18653/v1/2020.acl-main.417)

[thumbnail of Banon-etal-ACL2020-ParaCrawl-web-scale-acquisition-parallel-corpora]
Preview
 Text. Filename: Banon-etal-ACL2020-ParaCrawl-web-scale-acquisition-parallel-corpora.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (618kB)| Preview

Abstract

We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.

  • Item type:Book Section
    ID code:87251
    Dates:
    Date
    Event
    10 July 2020
    Published
    3 April 2020
    Accepted
    Notes:Funding Information: This work has been supported in part by three projects funded by the Connecting Europe Facility of the European Union (paracrawl.eu), two Google Faculty Research Awards to Philipp Koehn, a Mozilla research grant to Kenneth Heafield, and a donation from eBay to Kenneth Heafield. Hosting is provided by the AWS Public Dataset Program. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (http://www.csd3.cam.ac.uk/), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/P020259/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). This paper is the authors' opinion and not necessarily that of the funders. Funding Information: This work has been supported in part by three projects funded by the Con necting Europe Facility of the European Union (paracrawl.eu), two Google Faculty Research Awards to Philipp Koehn, a Mozilla research grant to Kenneth Heafield, and a donation from eBay to Kenneth Heafield. Hosting is provided by the AWS Public Dataset Program. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (http://www.csd3.cam.ac.uk/), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council(capital grant EP/P020259/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). This paper is the authors’ opinion and not necessarily that of the funders. Publisher Copyright: © 2020 Association for Computational Linguistics Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-Scale Acquisition of Parallel Corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555–4567, Online. Association for Computational Linguistics. DOI: 10.18653/v1/2020.acl-main.417
    Subjects:Science > Mathematics > Electronic computers. Computer science
    Department:Faculty of Science > Computer and Information Sciences
    Depositing user: Pure Administrator
    Date deposited:09 Nov 2023 10:28
    Last modified:24 Apr 2024 08:44
    Related URLs:
    URI:https://strathprints.strath.ac.uk/id/eprint/87251
CORE (COnnecting REpositories)