[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2013-elhadad-multi-document-multilingual-summarization-corpus/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish

# Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish

**Authors:** [Michael Elhadad](https://md.mova.elamur.ai/free/en/docs/papers/authors/michael-elhadad/); Sabino Miranda-Jiménez; Josef Steinberger; George Giannakopoulos · **Year:** 2013 · **Venue:** Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 13–19 **Link:** [aclanthology.org/W13-3102](https://aclanthology.org/W13-3102/) · library `aclbulk-2013-elhadad-multi-document-multilingual-summarization-corpus` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**14** citations](https://openalex.org/works?filter=cites:W2250833996) openalex · 2026-10-19[**6** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2013-elhadad-multi-document-multilingual-summarization-corpus/) of the 140673 papers whose bibliographies we read[Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization](https://aclanthology.org/volumes/W13-31/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W13-3102/)
- [PDF](https://aclanthology.org/W13-3102.pdf)
- [OpenAlex](https://openalex.org/W2250833996)

**Authors (4)**

1. [Michael Elhadad](https://openalex.org/A5013658527) [ORCID](https://orcid.org/0000-0002-5629-2351) · [Semantic Scholar](https://www.semanticscholar.org/author/1754680) · h 25 · 3695 citations · 137 works
2. Sabino Miranda-Jiménez
3. [Josef Steinberger](https://openalex.org/A5025269577) [ORCID](https://orcid.org/0000-0003-1707-1895) · [Semantic Scholar](https://www.semanticscholar.org/author/3292561) · h 22 · 1664 citations · 93 works
4. [George Giannakopoulos](https://openalex.org/A5048495248) [ORCID](https://orcid.org/0000-0003-2459-589X) · [Semantic Scholar](https://www.semanticscholar.org/author/144274464) · h 22 · 1396 citations · 104 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

This is the second part of the report on the MultiLing 2013 corpus, which covers ten languages; this part covers Czech, Hebrew and Spanish. A topic is a set of news documents describing an event sequence. Each language needed source documents (translated from English where needed), human summaries of 240–250 words, and human grades of the summaries for overall responsiveness on a 1–5 scale. For Hebrew, five clusters of ten articles were translated and summarised by three people; per article, reading took 2.5 minutes, translation 30 minutes and proofreading 18.5 minutes, and the average grades of the three summarisers were 4.80, 4.40 and 4.13. For Spanish, 15 topics of ten documents were translated by 30 students (104.5 man-hours), and the average human grade was 3.8 out of 5. The authors record that ordering texts by date gave the most fluent summaries, that translation needs expert knowledge, and that machine-translated drafts slowed proofreading.

## Ideas

- I1. Hebrew spelling variants: the team left plene and deficient spellings, and foreign words that have both a transliterated and a Hebrew form, to Wikipedia's usage. A rule for `languages/he` could map plene and deficient forms to one lemma for lookup while keeping both as surface forms.
- I2. Date-ordered event chains: the summarisers found that ordering by publication date gave the most fluent text, but background facts sometimes sat in a later document. In `world`, order the event chain by date, and let a rule move a background event to the front when the first event refers to it.
- I3. Translate from scratch when building a gold set: the paper found that editing machine output took longer than translating by hand with dictionaries, and that the first text of each cluster took longest. Any Hebrew or Aramaic gold set Mova builds should follow the same rule, and budget the first items of each topic higher.

## For Mova

I1 goes into [2. Script and spelling](https://md.mova.elamur.ai/free/en/docs/languages/he/02-script-and-spelling/). I2 is an idea for the event chain in `world`, with the date rule noted in [10. Historical layers: Biblical, Mishnaic, Medieval and Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/languages/he/10-historical-layers/) only as background. I3 goes into [12. Mova and Modern Hebrew: where it stands, what it would need](https://md.mova.elamur.ai/free/en/docs/languages/he/12-mova/) as a gold-set policy.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{elhadad2013multi,
  title = {{Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish}},
  author = {Elhadad, Michael and Miranda-Jiménez, Sabino and Steinberger, Josef and Giannakopoulos, George},
  booktitle = {Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization},
  year = {2013},
  url = {https://aclanthology.org/W13-3102/}
}
```

- [← previous Effect of Out Of Vocabulary Terms on Inferring Eligibility Criteria for a Retrospective Study in Hebrew EHR](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2013-cohen-effect-out-vocabulary-terms-inferring/)
- [next → Innovation of Verbs in Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2014-ornan-innovation-verbs-hebrew/)
- [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/)

## References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hoa Trang Dang and K. Owczarzak. 2009. Overview of the tac 2009 summarization track, Nov. [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang+and+K%2E+Owczarzak%2E+2009%2E+Overview+of+the+tac+2009+summarization+track%2C+Nov%2E)
2. Michael Elhadad, Sabino Miranda-Jiménez, Josef Steinberger, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 2: Czech, hebrew and spanish. In MultiLing 2013 Workshop in ACL 2013, Sofia, Bulgaria, August. [find](https://scholar.google.com/scholar?q=Michael+Elhadad%2C+Sabino+Miranda%2DJim%C3%A9nez%2C+Josef+Steinberger%2C+and+George+Giannakopoulos%2E+2013%2E+Multi%2Ddocument+multilingual+summarization+corpus+preparation%2C+part+2%3A+Czech%2C+hebrew+and+spanish%2E+In+MultiLi) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2013-elhadad-multi-document-multilingual-summarization-corpus/)
3. Brigitte Endres-Niggemeyer and Elisabeth Wansorra. 2004. Making cognitive summarization agents work in a real-world domain. In Proceedings of NLUCS Workshop, pages 86–96. Citeseer. [find](https://scholar.google.com/scholar?q=Brigitte+Endres%2DNiggemeyer+and+Elisabeth+Wansorra%2E+2004%2E+Making+cognitive+summarization+agents+work+in+a+real%2Dworld+domain%2E+In+Proceedings+of+NLUCS+Workshop%2C+pages+86%E2%80%9396%2E+Citeseer%2E)
4. Brigitte Endres-Niggemeyer. 2000. Human-style WWW summarization. Technical report. [find](https://scholar.google.com/scholar?q=Brigitte+Endres%2DNiggemeyer%2E+2000%2E+Human%2Dstyle+WWW+summarization%2E+Technical+report%2E)
5. G. Giannakopoulos, M. El-Haj, B. Favre, M. Litvak, J. Steinberger, and V. Varma. 2011. TAC 2011 MultiLing pilot overview. In TAC 2011 Workshop, Maryland MD, USA, November. [find](https://scholar.google.com/scholar?q=G%2E+Giannakopoulos%2C+M%2E+El%2DHaj%2C+B%2E+Favre%2C+M%2E+Litvak%2C+J%2E+Steinberger%2C+and+V%2E+Varma%2E+2011%2E+TAC+2011+MultiLing+pilot+overview%2E+In+TAC+2011+Workshop%2C+Maryland+MD%2C+USA%2C+November%2E)
6. E. Hovy, C. Y. Lin, L. Zhou, and J. Fukumoto. 2005. Basic elements. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2C+C%2E+Y%2E+Lin%2C+L%2E+Zhou%2C+and+J%2E+Fukumoto%2E+2005%2E+Basic+elements%2E)
7. Lei Li, Corina Forascu, Mahmoud El-Haj, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 1: Arabic, english, greek, chinese, romanian. In MultiLing 2013 Workshop in ACL 2013, Sofia, Bulgaria, August. [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Corina+Forascu%2C+Mahmoud+El%2DHaj%2C+and+George+Giannakopoulos%2E+2013%2E+Multi%2Ddocument+multilingual+summarization+corpus+preparation%2C+part+1%3A+Arabic%2C+english%2C+greek%2C+chinese%2C+romanian%2E+In+MultiLing+2)
8. C. Y. Lin. 2004. Rouge: A package for automatic evaluation of summaries. Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), pages 25–26. [find](https://scholar.google.com/scholar?q=C%2E+Y%2E+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+Proceedings+of+the+Workshop+on+Text+Summarization+Branches+Out+%28WAS+2004%29%2C+pages+25%E2%80%9326%2E)
9. Annie Louis and Ani Nenkova. 2012. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267–300, Aug. [find](https://scholar.google.com/scholar?q=Annie+Louis+and+Ani+Nenkova%2E+2012%2E+Automatically+assessing+machine+summary+content+without+a+gold+standard%2E+Computational+Linguistics%2C+39%282%29%3A267%E2%80%93300%2C+Aug%2E)
10. Annie Piolat, Thierry Olive, and Ronald T Kellogg. 2005. Cognitive effort during note taking. Applied Cognitive Psychology, 19(3):291–312. [find](https://scholar.google.com/scholar?q=Annie+Piolat%2C+Thierry+Olive%2C+and+Ronald+T+Kellogg%2E+2005%2E+Cognitive+effort+during+note+taking%2E+Applied+Cognitive+Psychology%2C+19%283%29%3A291%E2%80%93312%2E)
11. Horacio Saggion and Sandra Szasz. 2012. The concisus corpus of event summaries. In LREC, pages 2031–2037. [find](https://scholar.google.com/scholar?q=Horacio+Saggion+and+Sandra+Szasz%2E+2012%2E+The+concisus+corpus+of+event+summaries%2E+In+LREC%2C+pages+2031%E2%80%932037%2E)
12. H. Saggion, J. M. Torres-Moreno, I. Cunha, and E. SanJuan. 2010. Multilingual summarization evaluation without human models. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, page 1059– 1067. [find](https://scholar.google.com/scholar?q=H%2E+Saggion%2C+J%2E+M%2E+Torres%2DMoreno%2C+I%2E+Cunha%2C+and+E%2E+SanJuan%2E+2010%2E+Multilingual+summarization+evaluation+without+human+models%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computational+Lingui)