[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/hbo-2016-shmidman-identification-parallel-passages-across-large/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/) / Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus

# Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus

**Authors:** [Avi Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/avi-shmidman/), [Moshe Koppel](https://md.mova.elamur.ai/free/en/docs/papers/authors/moshe-koppel/), Ely Porat · **Year:** 2016 · **Venue:** Journal of Data Mining and Digital Humanities; arXiv 1602.08715 **Link:** [arxiv.org/abs/1602.08715](https://arxiv.org/abs/1602.08715) · **Licence:** arXiv non-exclusive licence — copyrighted, summary only

[**6** citations](https://openalex.org/works?filter=cites:W2291996448) openalex · 2026-10-18[**7** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/hbo-2016-shmidman-identification-parallel-passages-across-large/) of the 140673 papers whose bibliographies we read[Journal of Data Mining & Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388](https://openalex.org/S2736708624)[licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

- [Publisher's page](https://doi.org/10.46298/jdmdh.1388)
- [PDF](https://arxiv.org/pdf/1602.08715)
- [DOI 10.46298/jdmdh.1388](https://doi.org/10.46298/jdmdh.1388)
- [arXiv 1602.08715](https://arxiv.org/abs/1602.08715)
- [OpenAlex](https://openalex.org/W2291996448)

**Authors (3)**

1. Avi Shmidman
2. Moshe Koppel
3. Ely Porat

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

A fast method for finding all approximate parallel passages (20+ words) in a large Hebrew/Aramaic corpus, where spelling (plene vs defective, final letters, א/י), particles and synonyms vary. Each word is reduced to its two rarest letters (by corpus letter frequency), keeping their order; this strips most prefixes and matres lectionis, so דילמא and דלמא both become דל. From every five-word window four "skip-grams" of four words are taken (each leaving one word out), packed into collision-free integers and indexed. A parallel is a cluster of at least 3 matching skip-grams, gaps of at most 8 words, spanning at least 20 words, found by binning matches along the diagonal of a dot-plot. On the Babylonian Talmud (1.8 million words) it finds 4 602 parallel pairs in about 11 seconds on one thread; on a test pair of tractates it finds all 41 parallels that exhaustive Levenshtein finds, plus 5 more, with no false positives seen.

## Ideas

- I1. The two-rarest-letters key is a cheap spelling-robust word signature for unvocalised Hebrew: use it to match Torah words against other witnesses (Samaritan Pentateuch, Qumran texts, Mishnah quotations of verses) where plene/defective spelling differs, before any morphological analysis.
- I2. Skip-gram clusters as the parallel finder for quotation detection: find every Torah verse quoted or paraphrased in the Mishnah corpus, giving aligned Biblical–Mishnaic pairs for the comparison chapter.
- I3. The whole method is integer-only and deterministic; a Rust implementation is small and fits Mova's rule of BTreeMap order and two identical runs. Measure recall on known inner-Torah parallels (Exod 20 / Deut 5) against a plain lemma-Jaccard method.

## For Mova

I1 is a helper for [1. Script and pointing](https://md.mova.elamur.ai/free/en/docs/languages/hbo/01-script/) (matres lectionis, plene/defective) and for any OCR or witness-alignment work. I2 feeds `11-mishnaic.md` and `12-biblical-mishnaic-modern.md` with aligned quotations. I3 is a tool to write beside `tools/macula`; the paper is read-only, so the method is reimplemented from its description.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.46298/jdmdh.1388](https://doi.org/10.46298/jdmdh.1388) · [arXiv 1602.08715](https://arxiv.org/abs/1602.08715)

```
@article{shmidman2016identification,
  title = {{Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus}},
  author = {Shmidman, Avi and Koppel, Moshe and Porat, Ely},
  journal = {Journal of Data Mining \& Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388},
  year = {2016},
  doi = {10.46298/jdmdh.1388},
  eprint = {1602.08715},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.46298/jdmdh.1388}
}
```

- [← previous An Alternative Approach to the Lexicon of Late Biblical Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo-2016-rezetko-alternative-approach-lexicon-late-biblical/)
- [next → Approaching the mountain of Exodus 19: thou shalt explore syntax first](https://md.mova.elamur.ai/free/en/docs/papers/hbo-2016-talstra-approaching-mountain-exodus-19-thou/)
- [↑ back to Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/)

## References

14 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Basile C., Benedetto D., Caglioti G., and Degli Esposti M. A Plagiarism Detection Procedure in Three Steps: Selection, Matches and Squares, PAN 2009:19-23. [find](https://scholar.google.com/scholar?q=Basile+C%2E%2C+Benedetto+D%2E%2C+Caglioti+G%2E%2C+and+Degli+Esposti+M%2E+A+Plagiarism+Detection+Procedure+in+Three+Steps%3A+Selection%2C+Matches+and+Squares%2C+PAN+2009%3A19%2D23%2E)
2. Barron-Cedeno A., Basile C., Degli Esposti M. and Rosso P. Word length n-grams for text re-use detection, in Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing, ser. CICLing’10. 2010:687-699. [find](https://scholar.google.com/scholar?q=Barron%2DCedeno+A%2E%2C+Basile+C%2E%2C+Degli+Esposti+M%2E+and+Rosso+P%2E+Word+length+n%2Dgrams+for+text+re%2Duse+detection%2C+in+Proceedings+of+the+11th+international+conference+on+Computational+Linguistics+and+Intellige)
3. Büchler M. Informationstechnische Aspekte des Historischen Text Re-use (Engl. Computational Aspects of Historical Text Re-use Detection), Ph.D. dissertation, Universitat Leipzig, 2013. [find](https://scholar.google.com/scholar?q=B%C3%BCchler+M%2E+Informationstechnische+Aspekte+des+Historischen+Text+Re%2Duse+%28Engl%2E+Computational+Aspects+of+Historical+Text+Re%2Duse+Detection%29%2C+Ph%2ED%2E+dissertation%2C+Universitat+Leipzig%2C+2013%2E)
4. Büchler, M., Burns, P. R., Müller, M., Franzini, E., and Franzini, G. Towards a Historical Text Re-use Detection, in: Biemann, C. and Mehler, A. (eds.) Text Mining, Theory and Applications of Natural Language Processing. 2014. [↗](https://doi.org/10.1007/978-3-319-12655-5_11)
5. Skolnik F. (ed.) Encyclopedia Judaica, second edition. Keter Publishing House (Jerusalem), 2007. [find](https://scholar.google.com/scholar?q=Skolnik+F%2E+%28ed%2E%29+Encyclopedia+Judaica%2C+second+edition%2E+Keter+Publishing+House+%28Jerusalem%29%2C+2007%2E)
6. Franzini G., Büchler M., Franzini E., and Moritz M. Scaling Historical Text Re-use. The IEEE International Conference on Big Data 2014 (IEEE BigData 2014), 2014:27-30. [find](https://scholar.google.com/scholar?q=Franzini+G%2E%2C+B%C3%BCchler+M%2E%2C+Franzini+E%2E%2C+and+Moritz+M%2E+Scaling+Historical+Text+Re%2Duse%2E+The+IEEE+International+Conference+on+Big+Data+2014+%28IEEE+BigData+2014%29%2C+2014%3A27%2D30%2E)
7. Grozea C., Gehl C., and Popescu M. ENCOPLOT: Pairwise Sequence Matching in Linear Time Applied to Plagiarism Detection. 3rd PAN Workshop. Uncovering Plagiarism, Authorship and Social Software Misuse. 2009:10. [find](https://scholar.google.com/scholar?q=Grozea+C%2E%2C+Gehl+C%2E%2C+and+Popescu+M%2E+ENCOPLOT%3A+Pairwise+Sequence+Matching+in+Linear+Time+Applied+to+Plagiarism+Detection%2E+3rd+PAN+Workshop%2E+Uncovering+Plagiarism%2C+Authorship+and+Social+Software+Misuse%2E+)
8. Grozea C., and Popescu M. The Encoplot Similarity Measure for Automatic Detection of Plagiarism Notebook for PAN at CLEF 2011. [find](https://scholar.google.com/scholar?q=Grozea+C%2E%2C+and+Popescu+M%2E+The+Encoplot+Similarity+Measure+for+Automatic+Detection+of+Plagiarism+Notebook+for+PAN+at+CLEF+2011%2E)
9. Guthrie D., Allison B., Liu W., Guthrie L. and Wilks Y. A Closer Look at Skip-gram Modelling. Proceedings of the 5th International Conference on Language Resources and Evaluation. 2006:1-4. [find](https://scholar.google.com/scholar?q=Guthrie+D%2E%2C+Allison+B%2E%2C+Liu+W%2E%2C+Guthrie+L%2E+and+Wilks+Y%2E+A+Closer+Look+at+Skip%2Dgram+Modelling%2E+Proceedings+of+the+5th+International+Conference+on+Language+Resources+and+Evaluation%2E+2006%3A1%2D4%2E)
10. Harris Z. Distributional structure. Word. 1954;10(23):146-162. http1 [https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe](https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe) [↗](https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe)
11. Karp R. and Rabin M. O. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development 1987; 31(2):249-260. [find](https://scholar.google.com/scholar?q=Karp+R%2E+and+Rabin+M%2E+O%2E+Efficient+randomized+pattern%2Dmatching+algorithms%2E+IBM+Journal+of+Research+and+Development+1987%3B+31%282%29%3A249%2D260%2E)
12. Klein E., Dershowitz N., Wolf L., Almogi O., and Wangchuk D. Finding Inexact Quotations Within a Tibetan Buddhist Corpus. Digital Humanities. 2014:486-488. [find](https://scholar.google.com/scholar?q=Klein+E%2E%2C+Dershowitz+N%2E%2C+Wolf+L%2E%2C+Almogi+O%2E%2C+and+Wangchuk+D%2E+Finding+Inexact+Quotations+Within+a+Tibetan+Buddhist+Corpus%2E+Digital+Humanities%2E+2014%3A486%2D488%2E)
13. Lee, J. A Computational Model of Text Reuse in Ancient Literary Texts. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 2007:472-479 [find](https://scholar.google.com/scholar?q=Lee%2C+J%2E+A+Computational+Model+of+Text+Reuse+in+Ancient+Literary+Texts%2E+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+Computational+Linguistics%2E+2007%3A472%2D479)
14. Lee L. Measures of distributional similarity. 37th Annual Meeting of the Association for Computational Linguistics. 1999:25-32. [find](https://scholar.google.com/scholar?q=Lee+L%2E+Measures+of+distributional+similarity%2E+37th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E+1999%3A25%2D32%2E)