Ecological Validity and the Evaluation of Speech Summarization

The Association for Computational Linguistics Order copies of this and other ACL proceedings from: Association for Computational Linguistics (ACL) 209 N. Eighth Street Stroudsburg, PA 18360 USA Tel: +1-570-476-8006 Fax: +1-570-476-0860 acl@aclweb.org ISBN 978-1-937284-20-6 / 1-937284-20-4 ii Introduction Welcome to the NAACL/HLT 2012 Workshop on Evaluation Metrics and System Comparison for Automatic Summarization. One of the goals of the workshop is to give a retrospective analysis of evaluation methods employed at the Text Analysis Conferences (TAC) and its predecessor, the Document Understanding Conferences (DUC). The other goal is to set plans for the future

as we introduce the new task of summarization of scientific articles. We have planned two invited presentations. Dragomir Radev will talk about his own work on summarization of scientific articles, as well as provide us with some background on related work. Lucy Vanderwende will present the plans for the new summarization task, the evaluation and the time line for future shared tasks. We have reserved ample time for discussion. In the six regular presentations we will discuss a range of exciting topics in summarization and evaluation. These include task-based evaluations of summarization, assessments of the accuracy of current automatic evaluations, the benefits from using several automatic evaluation measures, case studies of differences between manual and automatic evaluation, cross-lingual summarization and steps towards abstractive summarization. We anticipate a lively and rewarding workshop. Thank you for your participation! John M. Conroy Hoa Trang Dang Ani Nenkova Karolina Owczarzak iii Organizers: John M. Conroy, IDA Center for Computing Sciences Hoa Trang Dang, National Institute for Standards and Technology Ani Nenkova, University of Pennsylvania Karolina Owczarzak, National Institute for Standards and Technology Program Committee: Enrique Amigo (UNED, Madrid) Giuseppe Carenini (University of British Columbia) Katja Filippova (Google Research) George Giannakopoulos (NCSR Demokritos) Dan Gillick (University of California at Berkeley) Min-Yen Kan (National University of Singapore) Guy Lapalme (University of Montreal) Yang Liu (University of Texas, Dallas) Annie Louis (University of Pennsylvania) Kathy McKeown (Columbia University) Gabriel Murray (University of British Columbia) Dianne O’Leary (University of Maryland) Drago Radev (University of Michigan) Steve Renals (University of Edinburgh) Horacio Saggion (Universitat Pompeu Fabra) Judith Schlesinger (IDA Center for Computing Sciences) Josef Steinberger (European Commission Joint Research Centre) Stan Szpakowicz (University of Ottawa) Lucy Vanderwende (Microsoft Research) Stephen Wan (CSIRO ICT Centre) Xiaodan Zhu (National Research Council Canada) Invited Speakers: Drago Radev (University of Michigan) Lucy Vanderwende (Microsoft Research) v Table of Contents An Assessment of the Accuracy of Automatic Evaluation in Summarization Karolina Owczarzak, John M. Conroy, Hoa Trang Dang and Ani Nenkova....................1 Using the Omega Index for Evaluating Abstractive Community Detection Gabriel Murray, Giuseppe Carenini and Raymond Ng ..................................... 10 Machine Translation for Multilingual Summary Content Evaluation Josef Steinberger and Marco Turchi ..................................................... 19 Ecological Validity and the Evaluation of Speech Summarization Quality Anthony McCallum, Cosmin Munteanu, Gerald Penn and Xiaodan Zhu.....................28 The Heterogeneity Principle in Evaluation Measures for Automatic Summarization Enrique Amig´o, Julio Gonzalo and Felisa Verdejo ........................................ 36 Discrepancy Between Automatic and Manual...

Website: aclweb.org | Filesize: -
No of Page(s): 63
Download Ecological Validity and the Evaluation of Speech Summarization ....pdf