Elamur: **Mova** · [Parse](https://mova.elamur.ai/free/en/parse/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Translate](https://md.translate.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · [MCP](https://mcp.elamur.ai/) · [Groupants](https://md.groupants.elamur.ai/free/en/) [Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/tools/mova-parse/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [tools](https://md.mova.elamur.ai/free/en/docs/tools/) / mova-parse # mova-parse A two-level multilingual parser. **Level 1** decides the language of every sentence and of every span inside it: it is `langseg` with its Latin table (English and French by dictionary, a character model for unknown words, a switch cost for spans; [tools/langseg](https://md.mova.elamur.ai/free/en/docs/tools/langseg/#latin-script-languages-and-dialects)), plus the dialect modifier. **Level 2** is purely language-specific: each span is parsed by its own language's model, a file of `en::annotate::Annotator` (`en.bin`, `fr.bin`). The models stay separate; nothing in level 2 knows about other languages. ## How it works 1. **Sentences.** Each input line is a paragraph, cut after `. ? ! …` followed by whitespace and a capital, a digit or an opening quote, except after a one-letter or listed abbreviation (*Dr., M., etc.*). 2. **Level 1.** `langseg::Segmenter::segment` gives word spans with languages; Hebrew transliteration goes with English. The sentence's language is the one with most letters (a sentence without letters takes the language of the sentence before). The spans become pieces that cover the sentence: the text between two pieces is cut at its last whitespace, an opening quote going with the piece it opens. 3. **Level 2.** The pieces in the sentence's language are tokenized by its model and parsed together as one sentence, the switched pieces taken out, so the matrix clause stays whole. A switched piece of `--min-span` words or more (default 2) is tokenized and parsed by its own language's model; its root is attached to the matrix root with `dep` and `CSAttach=root` in MISC (a placeholder, not a parse of the attachment). A shorter switched piece (one word: a loan, a name) stays in the matrix parse and is marked `Lang=xx|CSParse=matrix`. 4. **No model.** A language without a model file is reported on stderr and in `# parse`; its tokens get `_` in LEMMA, UPOS, XPOS, FEATS, HEAD and DEPREL and `Parse=None` in MISC. French is never parsed by the English model, except a one-word span under `--min-span`, which says so. Output: one CoNLL-U document. Every sentence has ``` # sent_id = doc-1 # text = He said « je ne sais pas » and left. # lang = en # dialect = unknown (or a label, with # dialect_confidence and # dialect_markers) # style = … (only when a style detector answers) # parse = en=en.bin; fr=fr.bin (or fr=none (no model for fr)) # spans = en 0-8, fr 8-29, en 29-38 (byte offsets into # text, when there is more than one piece) ... 6 sais … 2 dep _ Lang=fr|CSAttach=root ``` A label is (language, dialect, style). The dialect is `langseg::variety` (`US`/`GB`/`CA` for English, `FR`/`CA`/`BE`/`CH` for French, by marker words, `unknown` without enough evidence). The style is filled by an implementation of `langseg::variety::StyleTagger` set as `Dispatcher::style`; none is built yet (Mova's register lexicons `global/data/register-*.tsv` are the first candidates). ## Run ``` cargo build --release -p mova-parse mova-parse [--model xx=PATH]... [--min-span N] [--doc ID] [FILE...] > out.conllu ``` Models default to .bin for every language of the Latin table. Library use: ``` let mut d = mova_parse::Dispatcher::new(2); d.load(langseg::Lang::En, &mova_parse::default_model(langseg::Lang::En))?; let sents = d.parse(text); // Vec: lang, dialect, style, spans, tokens let doc = mova_parse::conllu("doc", &sents); ``` Readers that take raw text in more than one language (the Elephant's Wikipedia graph, the tree store's writers) call `Dispatcher::parse` and read `OutSent::lang` and the `Lang=` MISC values, or run `mova-parse` and read `# lang`; a sentence whose `# parse` says `none` has no tree. ## A new language in level 2 1. A `Lang` code in `langseg` (`Lang::code`, `Lang::parse`). 2. A lexicon for the Latin table (`langseg latin-lexicon ...` from open treebanks, licences checked; dev and test splits stay out) or, for another script, its own level-1 classifier; then a `LexLang` in `latin::default_table`, and a re-measured switch cost on a dev half. 3. A model file .bin: `en train` on the language's UD train after `en xpos-fill` ([What every language gets](https://md.mova.elamur.ai/free/en/docs/languages/PACKAGE/) item 6). Nothing else changes in `mova-parse`. 4. Dialect markers in `langseg::variety` (optional), from the language manual's chapter on national varieties. ## Measurements Level 1: see [langseg](https://md.mova.elamur.ai/free/en/docs/tools/langseg/#latin-script-languages-and-dialects) (2026-11-06: 97.77% of 3 455 UD dev sentences, 96.93% of the tokens of 3 158 code-switched sentences, switch-point F1 86.01%; shuffled lexicons 55.02% against a majority class of 57.28%). Level 2 adds no decision of its own: a sentence reaches the model of the language level 1 chose, so its parse quality is the model's (the baselines in [What every language gets](https://md.mova.elamur.ai/free/en/docs/languages/PACKAGE/)). The output of two runs on the same text is identical (2026-11-06, 60 Global Voices lines). ## Limits - The attachment of a switched span to its matrix is a placeholder (`dep` on the root). - Each model brings its own tokenizer; the French model's tokenizer comes from GSD train and keeps some elisions whole (*c'est*). - Hebrew-script spans are found but, without the Hebrew-script model of `langseg`, not told apart, and there is no Hebrew or Aramaic model in level 2 yet: they are reported as having no model. - Sentence cutting is rule-based and does not know every abbreviation. The Elamur team - [← previous mova-json](https://md.mova.elamur.ai/free/en/docs/tools/mova-json/) - [next → mova-serve](https://md.mova.elamur.ai/free/en/docs/tools/mova-serve/) - [↑ back to tools](https://md.mova.elamur.ai/free/en/docs/tools/)