Skip to main content
Article
Automatically Evaluating Content Selection in Summarization Without Human Models
Departmental Papers (CIS)
  • Annie Louis, University of Pennsylvania
  • Ani Nenkova, Univesity of Pennsylvania
Date of this Version
8-1-2009
Document Type
Conference Paper
Comments

Louis, A. & Nenkova, A., Automatically Evaluating Content Selection in Summarization Without Human Models, Conference on Empirical Methods in Natural Language Processing, Aug. 2009, doi: anthology/D09-1032

Abstract

We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, Jensen- Shannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations.

Disciplines
Citation Information
Annie Louis and Ani Nenkova. "Automatically Evaluating Content Selection in Summarization Without Human Models" (2009)
Available at: http://works.bepress.com/ani_nenkova/10/