Summarizing and Referring: Towards Cohesive Extracts
Patricia Nunes Lucia Rino
Gonçalves UFSCAR
PUCRS São Carlos - Brasil
Porto Alegre - Brasil lucia@dc.ufscar.br
patt.nunes@gmail.com
ABSTRACT
In this paper we propose and evaluate a system for summary
post-edition, which aims at replacing referential expressions,
trying to avoid referencial cohesion problems. To propose
expressions that best represent the evoked entity, the sys-
tem uses knowledge about coreference chains. We evaluate
the system both with knowledge provided by manual and
automatic annotation of coreference chains.
Categories and Subject Descriptors
J.5 [Computer Applications]: Linguistics; H.4.2 [Infor-
mation Systems Applications]: Decision support
General Terms
Algorithms,Experimentation
Keywords
Referencial Cohesion, Coreference Chains, Automatic Sum-
marization
1. INTRODUCTION
A common problem in extractive summaries is the occur-
rence of referential expressions which are of difficult interpre-
tation. In this paper we propose and evaluate a system for
summary post-edition, which aims at replacing referential
expressions, trying to avoid problems of broken referential
linkage. The system is based on a hybrid approach: it em-
ploys methods from deep approaches to verify and recover
quality of summaries which were first generated by a superfi-
cial approach. CorrefSum is thus a system developed to ver-
ify the referential cohesion of the extractive summaries using
knowledge about coreference chains. This paper presents
some experiments on the basis of this tool. This paper has
the following structure: in Section 2, we present concepts
related to referential cohesion and coreference. Section 3
describes the CorrefSum tool. Section 4 presents experi-
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
DocEng’08, September 16–19, 2008, São Paulo, Brazil.
Copyright 2008 ACM 978-1-60558-081-4/08/09 ...$5.00.
253
Renata Vieira
PUCRS
Porto Alegre - Brasil
renata.vieira@pucrs.br
ments using CorrefSum. Section 5 presents related work us-
ing coreference resolution and summarization. Conclusions
are presented in Section 6.
2. REFERENTIAL COHESION AND
COREFERENCE
A text is more than a simple sequence of words. There
are many inner-relationships among them in the discourse.
One way to establish such text cohesion is through referring
expressions [7]. When they address the same entity, they
signal a coreference chain. In the example below, extracted
from the corpus (Section 3), there is a coreference chain:“the
agronomist Miguel Guerra, from Universidade Federal de
Santa Catarina”, “Guerra”, “the agronomist”.
The discussion about national biotechnology, has
been understood as transgenic. That is the opin-
ion of the agronomist Miguel Guerra, from
Universidade Federal de Santa Catarina. Guerra
has cited the plant micro-propagation as an ex-
ample of low cost biotechnology. According to the
agronomist, Brazil must search the development
of transgenic.
This work considers that referential cohesion of extractive
summaries can be improved if the coreference chain of the
source-text is analyzed. Suppose the extractive summary
results in:“Guerra has cited the plant micro-propagation as
an example of low cost biotechnology” The reader cannot
identify the reference for the expression “Guerra”. If the
coreference chain is considered, the expression “Guerra” can
be replaced by “the agronomist Miguel Guerra, from Uni-
versidade Federal de Santa Catarina”. In this work we show
experiments where the linguistic knowledge is given both by
a corpus of manually annotated coreference chains and by
an automatic coreference resolution system [15].
3. SYSTEM ARCHITECTURE
The goal of the developed system is to treat problems
of referential cohesion found in extractive summaries, using
knowledge about coreference chains from the source text to
replace noun phrases whose referential interpretation may be
difficult. The system was implemented in Java1 , allowing
operating systems portability and free distribution. The
development and evaluation of this tool was based on the
Summ-it corpus [5] composed by 50 newspaper texts from
Folha de São Paulo, science section, written in Brazilian Por-
tuguese. The corpus has been processed by the PALAVRAS
parser [3]. This parser encodes a Constraint Grammar that
1
http://java.sun.com/
allows to encodes all annotation information as word based
tags. The corpus has been manually annotated with corefe-
rence information using the MMAX tool [12]. Each text has
been annotated and reviewed by 2 annotators using the same
annotation reference manual.Experiments were undertaken
with summaries produced by the Portuguese summarizers
GistSumm [13] and SuPor-2 [9]. GistSumm is a generic ex-
tractive summarizer. SuPor-2 is an extractive summarizer
developed on the basis of machine learning algorithms. We
have chosen these summarizers since they were available and
ready to use, they were easily applied to the texts from our
corpus. These systems were evaluated in previous work [10]
and [2].
The CorrefSum system has the following modules: file rea-
ding, reference score processing and summaries reviewing.
The goal of the reading file module is to read and store
information about the summaries and the source text.
The reference score processing module is designed to search
in the source text for the sentences included in the summary,
select all coreference chains related to noun phrases which
are present in the summary and apply a scoring scheme for
the coreference chain elements. The scores are based on the
following criteria, the presence of the criteria adds one point
to the score of the noun phrase:
Proper Name: if the noun phrase contains any proper
name.
Size: if the noun phrase is the longest one in the chain,
considering character length.
First: if the noun phrase is the first element in its chain.
Apposition: if the noun phrase contains commas (generally
used as an apposition mark).
All chain elements are scored on the basis of the above
features. The points are cumulative. They are the selec-
tion criteria for replacement of referring expressions in the
original summary. The summary recovery module uses the
scoring scheme to select the most complete chain element for
replacing the original summary term. If there are previous
sentences in the summary which contain referring expres-
sions common to those that appear in a coreference chain
under investigation, the expressions are not replaced. This
module is also responsible for maintaining the compression
rate as configured by the user. If the recovered summary
exceeds the rate, and there is an apposition in the noun
phrase to be replaced, then only the first part is selected.
Also parenthesis may be disconsidered when the compres-
sion rate is exceeded.
4. EXPERIMENTS AND EVALUATION
In this section we present experiments with the CorrefSum
system on the basis of Summ-it corpus. The recovered sum-
maries generated by CorrefSum were evaluated automati-
cally (Rouge), subjectively (human judges) and qualitatively
(author’s analysis). For the subjective evaluation, we have
used 5 human judges, native speakers of Portuguese. The
judges filled in a questionnaire about the extractive sum-
maries both recovered and non-recovered, aiming at eva-
luating informativity. On average, the source-texts convey
11.72 coreference chains whereas summaries contain 6.60.
For GistSumm summaries has performed a total of 89 re-
placements, with 1.78 on average per text. Out of the 330
chains (found in 50 summaries), 241 did not need replace-
ment. For SuPor-2 summaries we had 75 replacements (1.5
average per text). The largest number of replacements that
254
occurred in a summary alone was 4, there were also cases in
which no replacements were necessary, however, there were
replacements in most summaries. The average compression
rate for the original summaries were 25.30%/23.14% and the
summaries recovered after the application of CorrefSum sys-
tem got an average compression rate of 28.36%/25.52%, an
increase of about 3% when compared to their original size,
due to replacements of less informative referential expres-
sions by more complete ones. Although there is an increase
in the compression rate, it is still below the aimed 30%.
4.1 Quantitative Evaluation
Table 1 shows the Rouge measures for the original sum-
maries generated by GistSumm and SuPor-2 against the re-
covered summaries by CorrefSum.ROUGE [11] stands for
Recall-Oriented Understudy for Gisting Evaluation. It in-
cludes measures to automatically determine the quality of a
summary by comparing it to other (ideal) summaries created
by humans. The measures count the number of overlapping
units such as n-gram, word sequences, and word pairs bet-
ween the computer-generated summary to be evaluated and
the ideal summaries created by humans. Rouge-N are based
on N-gram co-occurrences between reference and candidate.
High order Rouge-N with n-gram length greater than 1 es-
timates the fluency of summaries. Rouge-L is based on the
longest common subsequence given two sequences X and Y
(a longest common subsequence of X and Y is a common
subsequence with maximum lenght.) We used summaries
manually generated by professional summarizers [4].
Table 1: Rouge
GistSumm SuPor-2
Original Recovered Original Recovered
Rouge-1 0.49249 0.52275 0.56085 0.56496
Rouge-2 0.32250 0.34320 0.42391 0.41735
Rouge-3 0.28382 0.29699 0.38510 0.36811
Rouge-4 0.26615 0.27218 0.36196 0.33566
Rouge-L 0.45497 0.47911 0.53368 0.53639
We noticed that by applying CorrefSum, the F-measure
increased in most of the cases specially for GistSumm. The
replacements performed with the goal of recovering the refe-
rential cohesion in the summaries indicate, according to this
measure, an improved informativity.
4.2 Subjective Evaluation
This kind of evaluation is difficult to perform because it
is based on human judgement, and it is often highly time
consuming. Due to these difficulties we made a subjective
evaluation only on a part of our corpus, 10 texts were chosen,
corresponding to 20% of the corpus. Since we wanted to con-
firm whether a difference in Rouge is actually perceived as
informativity improvement by the human readers, the texts
were those whose F-measure (Rouge) presented greater dif-
ferences before and after recovery. Five judges recived two
versions of the summaries: Original (O) and Recovered (R),
they were asked to indicate which one was more informa-
tive or else if Both (B) were considered equally informative.
Informativity was briefly defined as “Informativity is what
determines the information level of each summary”. So the
judges were working on their intuitive understanding of the
concept. Table 2 brings the subjective evaluation results on
informativity. In informativity evaluation, we have noticed a
common agreement that the recovered summaries are more
informative. This evaluation confirms the results indicated
by the Rouge measure. The recovered summaries are prefe-
red by the human judges also for SuPor-2, even through the the name “Allen”. Dealing with internal expressions in the
Rouge differences were not so evident for this summarizer. noun phrase is another aspect in which this work can be
Only judge n. 4 does not agree with the others, pointing that perfected, for example, the substitution of “The minister ”
both summaries (original and recovered) are informative. by the more complete noun phrase “The Minister of Justice
in the country, Elisabeth Guigou” has created a referential
cohesion problem, in which one cannot identify to which
Table 2: Subjective Evaluation on Informativity country the expression “the country” refers.
GistSumm SuPor-2
Judge
Judge 1
O
1
R
9 0
B O
1
R
8
B
1
4.4 Evaluation of CorrefSum with a
Judge 2 1 9 0 2 7 1 coreference resolution system
Judge 3 1 9 0 2 7 1 The coreference resolution system considered in this work
Judge 4 2 7 1 0 2 8
Judge 5 1 8 1 0 9 1 calls ACRoPos (Automatic Coreference ResOlution system
for POrtugueSe) [15]. The process consists in first identify-
ing pairs of expressions that are anaphoric and then grouping
4.3 Qualitative Evaluation the pairs that relate to each other thus forming the corefer-
In this section, we discuss the qualitative evaluation of the ence chains. The solution implemented uses machine lear-
substitutions performed by the system, based on an analy- ning to infer a classifier which receives as input noun phrases
sis conducted by the author. The analyzed replacements are pairs and their features. It is a first proposal of such a sys-
the 89 substitutions performed by the CorrefSum system in tem for the Portuguese Language, which considers all noun
the GistSumm summaries. To carry out this analysis the phrases types. The evaluation reported for the system is
following scoring criteria were adopted: of about Precision 97.14%, Recall 42.11% and F-measure
1 point: irrelevant replacement; for example, the replace- 57.96%. Here we present Rouge numbers for the original
ment of “human skin” by “regular human skin”. summaries compared against the recovered GistSumm sum-
2 points: to the replacement that, somehow, has helped in maries on the basis of our two coreference knowledge sources,
the summary comprehension, by adding extra information; first for the manual anotation corpus and then automatic
for example, the replacement of “Science and Technology” anotation. As the system has been trained on the same cor-
by “The Ministry of Science and Technology (MST)”. pus, we consider here the results regarding only the 20 texts
3 points: to the replacement that meaningfully contributes of the test set to avoid biased results. As expected, the
to the summary comprehension, and that solved a referential manual gold standard produces always a greater impact in
cohesion mistake; for example, the replacement of “Guerra” Rouge. We can see, however, that the coreference resolution
by “the agronomist Miguel Guerra, from UFSC (Universi- system produce chains that seem useful for summary reco-
dade Federal de Santa Catarina)”. very. Considering that this coreference resolution can still
be improved, we consider this a promising line of research.
Table 3: Qualitative Analysis Table 4: Rouge - ACRoPos and CorrefSum
Total Average Original Recovered Recovered
1-Point 36 40.45% Manual Annot. Autom. Annot.
2-Points 17 19.10% Rouge-1 0.51306 0.52735 0.52528
Rouge-2 0.34727 0.33950 0.33690
3-Points 36 40.45% Rouge-3 0.30840 0.29155 0.28954
Rouge-4 0.28881 0.26496 0.26420
Table 3 shows the analysis result. About 60% were mean- Rouge-L 0.47745 0.48163 0.47690
ingful replacements, whereas 40% did not have a major im-
pact on the summary informativity. Generally, in 1-point
replacements, the replaced noun phrase do not have any 5. RELATED WORK
anaphoricity feature, they are complete expressions whose Azzam et al [1] describe the use of coreference chains for
meaning is self-contained, and their interpretations are, some- summary production. Differently from us, Azzam’s paper
how, context-independent. We believe the system can still makes use of coreference chain information to select those
be improved with the development of an anaphoricity classi- sentences which are going to be part of the summary, consi-
fication module, in which the substitution will only be per- dering that the “best” chain refers to the most relevant topic
formed if the system identifies a need for it. in the text. Kashani et al [8] reports an algorithm for pro-
Other interesting cases refer to the substitutions in which noun generation in automatic summaries, aiming at avoiding
the expression chosen for replacement presented a referential repetition. While Kashani intends to prevent repetition by
cohesion problem in an internal noun phrase. For example: the inclusion of pronouns, our work tries to avoid the occur-
The minister → The Minister of Justice of the country, Elis- rence of an unresolved pronoun (that is, without a proper
abeth Guigou antecedent) by substituting such a pronoun for a complete
This insect → a being that breaks into bodies and controls expression, if necessary. Steinberger et al [16] propose two
other people’s minds, forcing them to do what he orders ways for using coreference in summarization. The first one
This study → Allen and his colleagues’ work is summary generation by exploring lexical information and
The noun phrase “a being that breaks into bodies and con- coreference resolution. Experiments comparing summaries
trols other people’s minds, forcing them to do what he or- generated by using lexical information only and by making
ders” turns out to be a problem, once it is impossible to use of both lexical information and anaphoric resolution
identify that the expression “being” refers to an “insect” . show an increase in Rouge-1 from 53,14% to 59,27%. The
Another problem is in the expression “Allen and his col- second approach, similarly to our proposal, verifies the sum-
leagues’ work ”, in which we cannot identify the referent to mary with the goal of recover referring expression. For the

255
coreference resolution, the GUITAR system [14] has been
used. The basic strategy is the change of anaphoric nouns
for the first element in the coreference chain. As an alter-
native, our work proposes a scoring scheme for choosing the
substitute, as described in Section 3. In [6] we presented a
first preliminary evaluation based only on Rouge-1 measure.
Here we present a more detailed evaluation considering se-
veral Rouge measures as well as an subjetive and qualitative
evaluation. We also evaluate the integration of our system
with a coreference resolution module.
6. CONCLUSIONS
Some of the most common problems in extractive sum-
maries is the occurrence of referential expression which are
of difficult interpretation. This informational gap, can often
cause misunderstandings. In this paper we have proposed
and evaluated a system for automatic summary post-edition,
which aims at rewriting referential expressions in the most
coherent possible way, trying to avoid problems of referen-
tial linkage. In order to achieve this, the noun phrases in the
summaries are analyzed on the basis of coreference chains,
with the goal of identifying expressions which best repre-
sent the evoked entity and performing the corresponding
substitution when appropriate. The experiments conducted
in this paper considered two summarizers previously evalu-
ated for Portuguese: GistSumm and SuPor-2. The results
show an increase in F-measure for both experiments. The
linguistic knowledge is given both by a corpus of manually
annotated coreference chains and by a coreference resolu-
tion system [15]. In the subjective evaluation, most judges
considered the recovered summaries more informative in re-
lation to both summarizers. Also, in order to improve the
performance of the system, we intend to consider the classifi-
cation of anaphoric expressions to verify the need for substi-
tution, to solve the referential cohesion problems in internal
noun phrase, and also to generate alternative referential ex-
pressions based on the coreference chains, instead of just
replacing them. A further step in this research is to build
and evaluate automatic summarizers which take into con-
sideration the coreference chains in the choice of relevant
sentences.
7. REFERENCES
[1] Saliha Azzam, Kevin Humphreys, and Robert
Gaizauskas. Using coreference chains for text
summarization. In Proceedings of The Relation of
Discourse/Dialogue Structure and Reference, 1999.
[2] Pedro Paulo Balage Filho, Thiago Alexandre Salgueiro
Pardo, and Maria das Graças Volpe Nunes.
Sumarização automática de textos cientı́ficos: Estudo
de caso com o sistema gistsumm. Technical report,
Departamento de Computação, Universidade Federal
de São Carlos. São Carlos-SP. NILC-TR-07-11, 2007.
[3] Eckhard Bick. The Parsing System ”PALAVRAS” -
Automatic Grammatical Analysis of Portuguese in a
Constraint Grammar Framework. PhD thesis,
Department of Linguistics, University of Århus, DK.,
2000.
[4] Jorge César Barbosa Coelho. Uso de informação de
correferência e anáfora para verificação da coesão e
coerência textual na sumarização automática.
Trabalho de Conclusão de Curso de Letras. Unisinos -
São Leopoldo, Junho 2007.
256
[5] Sandra Collovini, Thiago Carbonel, Juliana Thielsen
Fuchs, Jorge César Coelho, Lucia Rino, and Renata
Vieira. Summit: Um corpus anotado com informações
discursivas visando à sumarização automática. In 5ž
Workshop em Tecnologia da Informação e da
Linguagem Humana (TIL’2007), Rio de Janeiro, RJ,
2007. Proceedings of the SBC.
[6] Patricia Nunes Gonçalves, Renata Vieira, , and Lucia
Rino. Correfsum: Referencial cohesion recovery in
extractive summaries. In 8th Workshop on
Computational Processing of Written and Spoken
Language (PROPOR’2008), Aveiro, Portugal, 2008.
Springer.
[7] M.A.K. Halliday and R. Hasan. Cohesion in english.
Londres: Longman, 1976.
[8] Mehdi M. Kashani and Fred Popowich. Pronoun
generation for text summarization and question
answering. In Proceedings of 5th Slovenian and 1st
international Language Technologies Conference 2006,
2006.
[9] Daniel Leite and Lucia Rino. Supor: extensões e
acoplamento a um ambiente para mineração de dados.
Technical report, Departamento de Computação,
Universidade Federal de São Carlos. São Carlos-SP.
NILC-TR-06-07, 2006.
[10] Daniel Leite, Lucia Rino, Thiago Pardo, and Maria da
Graça Volpe Nunes. Extractive automatic
summarization: Does more linguistic knowledge make
a difference? In C. Biemann, I. Matveeva, R.
Mihalcea, and D. Radev (eds.), Rochester, NY, USA,
2007. Proceedings of the HLT/NAACL Workshop on
TextGraphs-2: Graph-Based Algorithms for Natural
Language Processing.
[11] Chin-Yew Lin. Rouge: A package for automatic
evaluation of summaries. In Workshop on Automatic
Summarization, Philadelphia,USA, 2000. Proceedings
of ACL-02.
[12] Christoph Müller and Michael Strube. Mmax: A tool
for the annotation of multi-modal corpora. In
Proceedings of the 2nd IJCAI Workshop on Knowledge
and Reasoning in Practical Dialogue Systems, pages
45–50, Seattle, Washington, 2001.
[13] T.A.S. Pardo. Gistsumm - gist summarizer: Extensões
e novas funcionalidades. Technical report,
NILC-TR-05-05. São Carlos-SP, 2005.
[14] Massimo Poesio and Mijail A. Kabadjov. A
general-purpose,off the shelf anaphoric resolver. In
Proceedings of International Conference on Language
Resources and Evaluation, Lisboa, Portugual, 2004.
[15] José Guilherme Souza, Patricia Nunes Gonçalves, and
Renata Vieira. Automatic coreference resolution
applied to portuguese. In 8th Workshop on
Computational Processing of Written and Spoken
Language (PROPOR’2008), Aveiro, Portugal, 2008.
Springer.
[16] Josef Steinberger, Massimo Poesio, Mijail Kabadjov,
and Karel Jezek. Two uses of anaphora resolution in
summarization. In Information Processing and
Management, 2007.