[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-1994-palmer-adaptive-sentence-boundary-disambiguation/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Adaptive Sentence Boundary Disambiguation # Adaptive Sentence Boundary Disambiguation **Authors:** David D. Palmer, Marti A. Hearst · **Year:** 1994 · **Venue:** Fourth Conference on Applied Natural Language Processing **Link:** [aclanthology.org/A94-1013](https://aclanthology.org/A94-1013/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**77** citations](https://openalex.org/works?filter=cites:W2133065881) openalex-snapshot · 2026-10-14[**14** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-1994-palmer-adaptive-sentence-boundary-disambiguation/) of the 140673 papers whose bibliographies we read[Fourth Conference on Applied Natural Language Processing](https://aclanthology.org/volumes/A94-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/A94-1013/) - [PDF](https://aclanthology.org/A94-1013.pdf) - [DOI 10.3115/974358.974376](https://doi.org/10.3115/974358.974376) - [OpenAlex](https://openalex.org/W2133065881) **Authors (2)** 1. [David D. Palmer](https://openalex.org/A5037867022) h 12 · 674 citations · 25 works 2. [Marti A. Hearst](https://openalex.org/A5019933387) [ORCID](https://orcid.org/0000-0002-4346-1603) · h 62 · 28774 citations · 279 works Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart. ## What it is about The authors decide whether a period, question mark or exclamation mark ends a sentence. Their method is trainable and does not use hand-made regular expressions. A lexicon gives the part-of-speech probabilities of the tokens around the punctuation mark, and a small neural network reads those probabilities over a window of context. Trained on a corpus of over 27,000 boundary marks, the method labels 98.5% correctly, which is 409 errors out of 27,294 marks, with both sensitivity thresholds at 0.5. Tested on uppercase legal text, it reaches 99.71% and 98.24% on two sets of 5,305 and 9,396 periods. The authors note that the results would probably be lower on lowercase data, and they analyse the errors by cause, such as abbreviations and quotation marks. ## Ideas - I1. `en/src/tok.rs` with `en/src/tag.rs`: replace the abbreviation list by the tag distribution of the token before the period, taken from the TnT prior, and a small table-driven score. Measure the boundary errors on EWT test, then count the errors per cause as the paper does. - I2. Error analysis by cause: group the boundary errors into abbreviation, quotation-mark and other cases, and print the groups with `en explain`, so each decision says which cause it belongs to. - I3. Test the thresholds: sweep the two decision thresholds on the dev part of EWT and report how many errors each threshold gives, as the paper does with its two cut-offs. ## For Mova The code is `en/src/tok.rs` and `en/src/tag.rs`, and the metric is `en tok-eval`. The chapter is [07. Punctuation, capital letters, abbreviations and numbers](https://md.mova.elamur.ai/free/en/docs/languages/en/07-punctuation-capitals-numbers/). The paper uses a 1990s corpus and a neural net, so its numbers are not Mova numbers and the method only gives the idea. Measure first on EWT test, twice, to check that the error count is stable. ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/974358.974376](https://doi.org/10.3115/974358.974376) ``` @inproceedings{palmer1994adaptive, title = {{Adaptive Sentence Boundary Disambiguation}}, author = {Palmer, David D. and Hearst, Marti A.}, booktitle = {Fourth Conference on Applied Natural Language Processing}, year = {1994}, doi = {10.3115/974358.974376}, url = {https://doi.org/10.3115/974358.974376} } ``` - [← previous Tokenization as the Initial Phase in NLP](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1992-webster-tokenization-as-initial-phase-nlp/) - [next → A Maximum Entropy Model for Prepositional Phrase Attachment](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1994-ratnaparkhi-maximum-entropy-model-prepositional-phrase/) - [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/) ## References 13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Leo Breiman, Jerome H. Friedman, Richard Olshen, and Charles J. Stone. 1984. Classification and regression trees. Wadsworth International Group, Belmont, CA. [find](https://scholar.google.com/scholar?q=Leo+Breiman%2C+Jerome+H%2E+Friedman%2C+Richard+Olshen%2C+and+Charles+J%2E+Stone%2E+1984%2E+Classification+and+regression+trees%2E+Wadsworth+International+Group%2C+Belmont%2C+CA%2E) 2. Kenneth W. Church and Mark Y. Liberman. 1991. A status report on the ACL/DCI. In The Proceedings of the 7th Annual Conference of the UW Centre for the New OED and Tezt Research: Using Corpora, pages 84-91, Oxford. [find](https://scholar.google.com/scholar?q=Kenneth+W%2E+Church+and+Mark+Y%2E+Liberman%2E+1991%2E+A+status+report+on+the+ACL%2FDCI%2E+In+The+Proceedings+of+the+7th+Annual+Conference+of+the+UW+Centre+for+the+New+OED+and+Tezt+Research%3A+Using+Corpora%2C+pages+8) 3. Kenneth W. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Second Conference on Applied Natural Language Processing, pages 136-143, Austin, TX. [find](https://scholar.google.com/scholar?q=Kenneth+W%2E+Church%2E+1988%2E+A+stochastic+parts+program+and+noun+phrase+parser+for+unrestricted+text%2E+In+Second+Conference+on+Applied+Natural+Language+Processing%2C+pages+136%2D143%2C+Austin%2C+TX%2E) 4. Doug Cutting, Julian Kupiec, Jan Pedersen, and Penelope Sibun. 1991. A practical part-of-speech tagger. [find](https://scholar.google.com/scholar?q=Doug+Cutting%2C+Julian+Kupiec%2C+Jan+Pedersen%2C+and+Penelope+Sibun%2E+1991%2E+A+practical+part%2Dof%2Dspeech+tagger%2E) 5. Training Error 1.52 0.75 0.043 Cross Error 2.36 2.01 1.88 Testing Errors 1424 409 877 Testing Error (%) 5.22% 1.50% 3.21% Testing Error (%) Training Error 1.05 1.08 0.39 0.27 Cross Error 1.61 2.18 2.27 1.42 Testing Errors 721 409 435 1343 2.64% 1.50% 1.59% 4.92% In The 3rd Conference on Applied Natu… [find](https://scholar.google.com/scholar?q=Training+Error+1%2E52+0%2E75+0%2E043+Cross+Error+2%2E36+2%2E01+1%2E88+Testing+Errors+1424+409+877+Testing+Error+%28%25%29+5%2E22%25+1%2E50%25+3%2E21%25+Testing+Error+%28%25%29+Training+Error+1%2E05+1%2E08+0%2E39+0%2E27+Cross+Error+1%2E61+2%2E18+2%2E2) 6. W. Francis and H. Kucera. 1982. Frequency Analysis of English Usage. Houghton Mifl]in Co., New York. [find](https://scholar.google.com/scholar?q=W%2E+Francis+and+H%2E+Kucera%2E+1982%2E+Frequency+Analysis+of+English+Usage%2E+Houghton+Mifl%5Din+Co%2E%2C+New+York%2E) 7. William A. Gale and Kenneth W. Church. 1993. A program for aligning sentences in bilingual corpora. Computational Linguistics, 19(1):75-102. [find](https://scholar.google.com/scholar?q=William+A%2E+Gale+and+Kenneth+W%2E+Church%2E+1993%2E+A+program+for+aligning+sentences+in+bilingual+corpora%2E+Computational+Linguistics%2C+19%281%29%3A75%2D102%2E) 8. John Hertz, Anders Krogh, and Richard G. Palmer. 1991. Introduction to the theory of neural computation. Santa Fe Institute studies in the sciences of complexity. Addison-Wesley Pub. Co., Redwood City, CA. [find](https://scholar.google.com/scholar?q=John+Hertz%2C+Anders+Krogh%2C+and+Richard+G%2E+Palmer%2E+1991%2E+Introduction+to+the+theory+of+neural+computation%2E+Santa+Fe+Institute+studies+in+the+sciences+of+complexity%2E+Addison%2DWesley+Pub%2E+Co%2E%2C+Redwood+City) 9. Susanne M. Humphrey. 1989. Research on interactive knowledge-based indexing: The medindex prototype. In Symposium on Computer Applications in Medical Care, pages 527-533. [find](https://scholar.google.com/scholar?q=Susanne+M%2E+Humphrey%2E+1989%2E+Research+on+interactive+knowledge%2Dbased+indexing%3A+The+medindex+prototype%2E+In+Symposium+on+Computer+Applications+in+Medical+Care%2C+pages+527%2D533%2E) 10. Martin Kay and Martin R6schelsen. 1993. Text-translation alignment. Computational Linguistics, 19(i):121-142. [find](https://scholar.google.com/scholar?q=Martin+Kay+and+Martin+R6schelsen%2E+1993%2E+Text%2Dtranslation+alignment%2E+Computational+Linguistics%2C+19%28i%29%3A121%2D142%2E) 11. Mark Y. Liberman and Kenneth W. Church. 1992. Text analysis and word pronunciation in text-to-speech synthesis. In Sadaoki Furui and Man Mohan Sondhi, editors, Advances in Speech Signal Processing, pages 791-831. Marcel Dekker, Inc. [find](https://scholar.google.com/scholar?q=Mark+Y%2E+Liberman+and+Kenneth+W%2E+Church%2E+1992%2E+Text+analysis+and+word+pronunciation+in+text%2Dto%2Dspeech+synthesis%2E+In+Sadaoki+Furui+and+Man+Mohan+Sondhi%2C+editors%2C+Advances+in+Speech+Signal+Processing%2C+pa) 12. Hans Miiller, V. Amerl, and G. Natalls. 1980. Worterkennungsverfahren als Grundlage einer Unl-versalmethode zur automatlschen Segmentierung yon Texten in Sgtze. Ein Verfahren zur maschineUen Satzgrenzenbestimmung im Englischen. Sprache und Datenverarbeitung, 1. [find](https://scholar.google.com/scholar?q=Hans+Miiller%2C+V%2E+Amerl%2C+and+G%2E+Natalls%2E+1980%2E+Worterkennungsverfahren+als+Grundlage+einer+Unl%2Dversalmethode+zur+automatlschen+Segmentierung+yon+Texten+in+Sgtze%2E+Ein+Verfahren+zur+maschineUen+Satzgrenz) 13. Michael D. Riley. 1989. Some applications of tree-based modelling to speech and language indexing. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 339-352. Morgan Kaufrnann. [find](https://scholar.google.com/scholar?q=Michael+D%2E+Riley%2E+1989%2E+Some+applications+of+tree%2Dbased+modelling+to+speech+and+language+indexing%2E+In+Proceedings+of+the+DARPA+Speech+and+Natural+Language+Workshop%2C+pages+339%2D352%2E+Morgan+Kaufrnann%2E)