Automatically Evaluating Content Selection in Summarization Without Human ModelsDepartmental Papers (CIS)
Date of this Version8-1-2009
Document TypeConference Paper
AbstractWe present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, Jensen- Shannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations.
Citation InformationAnnie Louis and Ani Nenkova. "Automatically Evaluating Content Selection in Summarization Without Human Models" (2009)
Available at: http://works.bepress.com/ani_nenkova/10/