Evaluation Campaign

IWSLT proposes challenging research tasks and an open experimental infrastructure for the scientific community working on spoken and written language translation. The IWSLT 2011 Evaluation Campaign will focus on the translation of TED Talks, a collection of public speeches on a variety of topics. Four tracks are proposed addressing different research tasks:

• ASR track : automatic transcription of talks from audio to text (in English).
• SLT track: speech translation of talks from audio (or ASR output) to text (from English to French)
• MT track : text translation of talks (from English to French, Arabic to English, and Chinese to English)
• SC track : system combination of ASR outputs (in English) and MT outputs (in English and French)

Main challenges of the proposed tracks are:

• Open domain ASR, clean transcription of spontaneous speech, detection and removal of non-words, and talk style and topic adaptation.
• Open domain SLT, translation of speech or ASR output into true-case punctuated text, and talk style and topic adaptation.
• Open domain MT between distant languages, and talk style and topic adaptation.

Training of MT systems and language models for ASR is constrained to data supplied by the organizers. As for ASR acoustic modeling no training data are distributed, participants are allowed to use any publicly available data recorded before 31 December 2010.

• Input format: pre-segmented SPHERE files
• Output format: CTM (description and example here, at section ”.ctm”), no case, no punctuation
• Coding: UTF-8
• Submission guidelines
• Evaluation: WER
• Training data: supplied monolingual texts for language modeling
• Development data:
  • dev2010:
    • audio+reference transcripts, 176Mb
    • automatic transcripts and lattices v0, 959MB
    • automatic transcripts v1
  • test2010:
    • audio+reference transcripts, 300Mb
    • automatic transcripts and lattices v0, 1.5GB
    • automatic transcripts v1
  • Scoring Package
• Test data 1: audio + sentence segmentation, 260Mb
• Test data 2: audio + sentence segmentation, 130Mb

• Input format: pre-segmented SPHERE files or supplied ASR outputs (transcript, SLF lattice)
• Output format: NIST XML format, true case with punctuation (example )
• Coding: UTF-8
• Submission guidelines
• Evaluation: BLEU and subjective ranking
• Training data: supplied monolingual and parallel texts
• Development data:
  • dev2010:
    • audio+reference transcripts, 176Mb
    • automatic transcripts and lattices v0, 959Mb
    • lattices by LIUM, 2.5Mb
    • automatic transcripts v1
  • test2010:
    • audio+reference transcripts, 300Mb
    • automatic transcripts and lattices v0 by KIT, 1.5Gb
    • lattices by LIUM, 4.0Mb
    • automatic transcripts v1
• Test data 2:
  • audio + sentence segmentation, 130Mb
  • automatic transcripts
  • lattices:
    • by FBK, 100Mb
    • by KIT (11Mb); in SLF format (726Mb)
    • by LIUM, 1.8Mb

• Input format: NIST XML format, true case with punctuation
• Output format: NIST XML format, true case with punctuation (example )
• Coding: UTF-8
• Submission guidelines
• Evaluation: BLEU and subjective ranking
• Training data: supplied monolingual and parallel texts
• Development data: TED en-fr,ar-en,zh-en
• Test data: TED en-fr,ar-en,zh-en

• ASR-SC: input-output format compliant ASR track output format
• MT-SC: input-output format compliant with MT tracks output format
• Evaluation: WER for ASR-SC, BLEU and subjective ranking for MT-SC
• Training data: supplied monolingual and parallel texts
• Development data: ASR runs, MT runs
• Test data:
  • ASR-SC: submitted ASR runs on "Test data 2"
  • MT-SC: submitted MT runs on test data

• To participate in one or more tracks please fill in the Registration Form and join the IWSLT2011 Mailing List in order to receive announcements and information about the IWSLT 2011 Evaluation Campaign.
• Participants are requested to submit a system paper describing their work and to present it at the workshop.

• TED talks data are copyright of TED Conference LLC and distributed under the Creative Commons Attribution-NonCommercial- NoDerivs 3.0 license
• Google Books ngrams are copyright of Google Inc. and distributed under a Creative Commons Attribution 3.0 license
• WMT11 data are kindly supplied by the EMNLP 2011 6th Workshop on Statistical Machine Translation
• MultiUN data are kindly supplied by the EuroMatrixPlus project

• Monolingual texts: TED en,fr (8Mb), WMT11 texts en,fr (7Gb), Google books ngrams en,fr,..
• Parallel texts: TED en-fr|ar-en|zh-en pure text (21Mb),TED en-fr|ar-en|zh-en with tags (22Mb) WMT11 news fr-en (12Mb), Europarl v6 fr-en (177Mb) , WMT11 Giga fr-en (2.3 Gb), WMT11 UN fr-en (1.1Gb), MultiUN ar-en-fr-zh (1.6Gb)

• 16 Jun 2011 - Release of training data
• 24 Jun 2011 - Release of dev data (all but SC track)
• 22 Aug 2011 - Submission of ASR/MT runs on dev data
• 25 Aug 2011 - Release of SC dev data
• 01 Sep 2011 - Release of ASR test data 1
• 14 Sep 2011 - Release of ASR test data 2
• 18 Sep 2011 - Submission of ASR runs (closing time 23:59:59 PDT) POSTPONED
• 19 Sep 2011 - Release of ASR-SC test data
• 22 Sep 2011 - Submission of ASR-SC runs (closing time 23:59:59 PDT)
• 20 Sep 2011 - Release of SLT and MT test data
• 25 Sep 2011 - Submission of SLT and MT runs (closing time 23:59:59 PDT)
• 27 Sep 2011 - Release of MT-SC test data
• 30 Sep 2011 - Submission of MT-SC runs (closing time 23:59:59 PDT)
• 15 Oct 2011 - Submission of system papers
• 31 Oct 2011 - Notification of acceptance
• 11 Nov 2011 - Camera ready version of system paper
• 8-9 Dec 2011 - IWSLT 2011 Workshop

• Marcello Federico (FBK, Evaluation Chair)
• Luisa Bentivogli (FBK/CELCT, evaluation & analysis)
• Mauro Cettolo (FBK, MT/SC-MT data processing)
• Sebastian Stueker (KIT, ASR/SLT/SC-ASR evaluation and analysis)
• Teresa Hermann (KIT, ASR/SLT/SC-ASR data processing)
• Giovanni Moretti (CELCT, crowdsourcing)
• Jan Niehues (KIT, ASR/SLT/SC-ASR data processing)
• Michael Paul (NICT, evaluation & analysis)