[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2009-gillick-sentence-boundary-detection-problem-u/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Sentence Boundary Detection and the Problem with the U.S.

# Sentence Boundary Detection and the Problem with the U.S.

**Authors:** Dan Gillick · **Year:** 2009 · **Venue:** Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers **Link:** [aclanthology.org/N09-2061](https://aclanthology.org/N09-2061/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**85** citations](https://openalex.org/works?filter=cites:W2143230354) openalex · 2026-10-19[**25** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2009-gillick-sentence-boundary-detection-problem-u/) of the 140673 papers whose bibliographies we read[Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers](https://aclanthology.org/volumes/N09-2/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/N09-2061/)
- [PDF](https://aclanthology.org/N09-2061.pdf)
- [DOI 10.3115/1620853.1620920](https://doi.org/10.3115/1620853.1620920)
- [OpenAlex](https://openalex.org/W2143230354)

**Authors (1)**

1. [Gillick, Dan](https://openalex.org/A5110207594) [Semantic Scholar](https://www.semanticscholar.org/author/2396669) · h 16 · 1369 citations · 26 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

Sentence boundary detection is hard because the period is ambiguous, and most practical tools rely on hand-made abbreviation lists. The author builds a fully statistical system. For each period, the features look only at one token on each side of it, and a linear support vector machine classifies the boundary. On a standard news corpus of about 27,000 examples the system makes 67 errors, which is an error rate of about 0.25%. Of these, 23 involve the word "U.S.". The single feature of the token that ends with the period gives surprisingly good results on its own, and a simple Naive Bayes classifier is also good, so the choice of classifier matters less than the features.

## Ideas

- I1. `en/src/tok.rs`: a period classifier over the token before the period and the token after it, with binary features and a linear score written in Rust. Measure the error count on EWT test per candidate period, and report the errors on "U.S.", "Mr." and similar tokens separately.
- I2. Test the paper's ablation first: keep only the feature of the token ending with the period, then add the next-token feature. Count the errors each step removes on EWT test, to see which feature carries the weight in English.
- I3. Store the abbreviation decisions as a table of learned weights, not a hand list, and print for each decision the feature that drove it, so every boundary can be explained.

## For Mova

The code is `en/src/tok.rs`, and the metric is `en tok-eval`. The chapter is [07. Punctuation, capital letters, abbreviations and numbers](https://md.mova.elamur.ai/free/en/docs/languages/en/07-punctuation-capitals-numbers/). The paper's corpus is news text, so the 0.25% figure is not a Mova figure. The paper's data is not in Mova's build. Measure first on EWT test: errors per candidate period, with a two-fold check (`en folds`).

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1620853.1620920](https://doi.org/10.3115/1620853.1620920)

```
@inproceedings{gillick2009sentence,
  title = {{Sentence Boundary Detection and the Problem with the U.S.}},
  author = {Gillick, Dan},
  booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers},
  year = {2009},
  doi = {10.3115/1620853.1620920},
  url = {https://doi.org/10.3115/1620853.1620920}
}
```

- [← previous Task-oriented Evaluation of Syntactic Parsers and Their Representations](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2008-miyao-task-oriented-evaluation-syntactic-parsers/)
- [next → Descriptive and Empirical Approaches to Capturing Underlying Dependencies among Parsing Errors](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2009-hara-descriptive-empirical-approaches-capturing-underlying/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Aberdeen, J. Burger, D. Day, L. Hirschman, P. Robinson, and M. Vilain. 1995. MITRE: description of the Alembic system used for MUC-6. In Proceedings of the 6th conference on Message understanding, pages 141–155. Association for Computational Linguistics Morristown, NJ, USA. [find](https://scholar.google.com/scholar?q=J%2E+Aberdeen%2C+J%2E+Burger%2C+D%2E+Day%2C+L%2E+Hirschman%2C+P%2E+Robinson%2C+and+M%2E+Vilain%2E+1995%2E+MITRE%3A+description+of+the+Alembic+system+used+for+MUC%2D6%2E+In+Proceedings+of+the+6th+conference+on+Message+understanding%2C+)
2. T. Joachims. 1999. Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning. [find](https://scholar.google.com/scholar?q=T%2E+Joachims%2E+1999%2E+Making+large%2Dscale+support+vector+machine+learning+practical%2C+Advances+in+kernel+methods%3A+support+vector+learning%2E)
3. T. Kiss and J. Strunk. 2006. Unsupervised Multilingual Sentence Boundary Detection. Computational Linguistics, 32(4):485–525. [find](https://scholar.google.com/scholar?q=T%2E+Kiss+and+J%2E+Strunk%2E+2006%2E+Unsupervised+Multilingual+Sentence+Boundary+Detection%2E+Computational+Linguistics%2C+32%284%29%3A485%E2%80%93525%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-2006-kiss-unsupervised-multilingual-sentence-boundary-detection/)
4. E. Loper and S. Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 62–69. [find](https://scholar.google.com/scholar?q=E%2E+Loper+and+S%2E+Bird%2E+2002%2E+NLTK%3A+The+Natural+Language+Toolkit%2E+In+Proceedings+of+the+ACL+Workshop+on+Effective+Tools+and+Methodologies+for+Teaching+Natural+Language+Processing+and+Computational+Lingu)
5. M.P. Marcus, M.A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: the penn treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=M%2EP%2E+Marcus%2C+M%2EA%2E+Marcinkiewicz%2C+and+B%2E+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+the+penn+treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
6. A. Mikheev. 2002. Periods, Capitalized Words, etc. Computational Linguistics, 28(3):289–318. [find](https://scholar.google.com/scholar?q=A%2E+Mikheev%2E+2002%2E+Periods%2C+Capitalized+Words%2C+etc%2E+Computational+Linguistics%2C+28%283%29%3A289%E2%80%93318%2E)
7. D.D. Palmer and M.A. Hearst. 1997. Adaptive Multilingual Sentence Boundary Disambiguation. Computational Linguistics, 23(2):241–267. [find](https://scholar.google.com/scholar?q=D%2ED%2E+Palmer+and+M%2EA%2E+Hearst%2E+1997%2E+Adaptive+Multilingual+Sentence+Boundary+Disambiguation%2E+Computational+Linguistics%2C+23%282%29%3A241%E2%80%93267%2E)
8. J.C. Reynar and A. Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing, pages 16–19. [find](https://scholar.google.com/scholar?q=J%2EC%2E+Reynar+and+A%2E+Ratnaparkhi%2E+1997%2E+A+maximum+entropy+approach+to+identifying+sentence+boundaries%2E+In+Proceedings+of+the+Fifth+Conference+on+Applied+Natural+Language+Processing%2C+pages+16%E2%80%9319%2E)