Elamur: **Mova** · [Parse](https://mova.elamur.ai/free/en/parse/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Translate](https://md.translate.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · [Groupants](https://md.groupants.elamur.ai/free/en/)

[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/tools/parallel/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [tools](https://md.mova.elamur.ai/free/en/docs/tools/) / parallel — parallel texts in one TSV (English–French, English–Ukrainian, French–Ukrainian)

# parallel — parallel texts in one TSV (English–French, English–Ukrainian, French–Ukrainian)

Converts free parallel corpora to one aligned TSV (`id`, `en`, `fr`, `prov`) and aligns Project Gutenberg book pairs. Plain Rust, deterministic (two runs give identical bytes); the only dependencies serve `tr-run` (`mova-teacher`, `serde_json`).

## Build and run

```
cargo build --release -p parallel
parallel moses   <en.txt> <fr.txt> <out.tsv> <source-tag>      # OPUS/Moses line-aligned files
parallel tatoeba <links.tsv> <eng.tsv> <fra.tsv> <out.tsv> [la lb]  # Tatoeba exports (bzcat them first)
parallel bible   <en-vpl.txt> <fr-vpl.txt> <out.tsv> <tag> [la lb]  # verse-per-line files; chapters with different verse counts go to <out>.mismatch
parallel align   <out.tsv> <tag> <en.txt[,..]> <fr.txt[,..]>   # Gutenberg texts: chapters, paragraphs, sentences
parallel choices <list.tsv> <out.tsv> <examples.tsv> <la> <lb> <corpus.tsv[@count]>...   # count listed renderings, pick and verify examples
parallel choices-md <counted.tsv> <examples.tsv> <manual.md> <out.md> <date>   # render the entries into a manual
parallel choices-queue <out.tsv> <pair>:<list>=<counted.tsv>...   # the entries that still need an example
parallel tr-run <queue.tsv> <translation-dir> <work-dir> <date> [--teacher terra|sol] [--pair P] [--calls N] [--batch 20] [--parse-bin P --parse-en P --parse-fr P] [--verify hard|all|none] [--dry-run]
parallel tr-report <queue.tsv> <translation-dir> <work-dir>      # accepted/rejected/left per pair and teacher, tokens, rejection reasons
```

`tatoeba` and `bible` take two optional column labels (`en uk`, `fr uk`; default `en fr`): they name the text columns of the header and the keys in `prov`, so the same converters build the English–Ukrainian and French–Ukrainian corpora in en-uk and `fr-uk/`. Without them the output is byte-identical to before (checked on the Tatoeba EN–FR pairs, 2026-11-06).

Existing output files are never overwritten. Data and per-source READMEs live in en-fr (not in git).

## How the aligner works

Project Gutenberg boilerplate is cut at the `*** START/END OF` markers. Headings (`CHAPTER`, `CHAPITRE`, Roman numerals, ...) split the books into chapters; when both sides have the same number of chapters they are paired in order, otherwise the whole book is one block. Inside a block, paragraphs are aligned with the Gale–Church length model (1993): beads 1-1, 1-0, 0-1, 2-1, 1-2, 2-2, 3-1, 1-3 with their priors, a banded dynamic programme, and the ratio of the two sides' total length as the length constant. Each aligned paragraph bead is split into sentences (abbreviation list for `Mr.`, `M.`, `Mme`, initials) and aligned the same way. Every row keeps `para_bead`, `para_p`, `sent_bead`, `sent_p`; consumers filter on them (the corpus README names the filter used for `pairs-good.tsv`).

Cost: O(n · band) per block, memory O(n · band) bytes; Les Misérables (12,000 paragraph beads) aligns in seconds.

## Counting renderings: `choices`

`choices` serves the false-friend lists (`languages/translation/false-friends/<pair>.tsv`) and the context lists (`languages/translation/context/<pair>.tsv`). Each list row names a source word (`src_match`), optionally a context the source sentence must also contain (`ctx_match`), the recommended renderings (`good_match`) and the look-alike or rendering to avoid (`bad_match`), as patterns: `|` between alternatives, spaces between consecutive tokens, `*` for a prefix, matched on lowercased tokens (apostrophes stay inside Ukrainian words). For every row it counts the sentence pairs with the source word (`n_src`), and among them those whose target side contains a recommended rendering (`n_good`) or the avoided one (`n_bad`), per corpus too (`by_corpus`), and sorts the rows by `n_src`. An empty `example_id` is filled with the shortest quotable row (25–220 characters) that has the source word and a recommended rendering but not the avoided one; a given id is verified (`example_ok` = `yes`, `no-match` or `missing`). A corpus given as `path@count` is counted but never quoted (learn/test-only sources); Tatoeba rows with an unknown author are never quoted. It also writes `frequency_rank` (1 + the number of rows with a higher `n_src`), `example_status` (`corpus` for a verified id, `constructed` for an id `ours-tr:<pair>:<n>` of our own translated sentences, `needs_example`) and keeps the authored `difficulty` (`plain` or `hard:<reason>`; empty becomes `plain`). Rows whose text carries placeholder characters (DGT has `Surname: 愦灭※`) are never quoted. The examples file keeps the pair's column order (`id`, `<la>`, `<lb>`, `prov`), so one row can serve entries of both directions. On a rerun a picked id is verified, so `example_ok` turns from `picked` to `yes`; after that, reruns are byte-identical. `choices-md` fills the blocks `<!-- entries col=value ... -->` … `<!-- /entries -->` of a manual with the matching rows, most frequent first, and fails when a row lands in no block or in two. `choices-queue` lists the rows with `example_status` = `needs_example` of several lists, with the suggested teacher (Terra for `plain`, Sol for `hard`) for the planned example pipeline; it calls no teacher.

The counts are **co-occurrence, not word alignment**: a recommended rendering in the same sentence pair is evidence, not proof, that it translates the source word; duplicated rows (DGT) count each time. Cost: one pass over the corpora, O(tokens) with a first-token index; the English–French corpora (about 10 million pairs) take about two minutes on one core.

## Our own example pairs: `tr-run`

`tr-run` fills the `needs_example` entries of the false-friend and context lists with our own sentence pairs ([queue and plan](https://md.mova.elamur.ai/free/en/docs/languages/translation/false-friends/#3-examples-we-make)). It is the only subcommand that calls a teacher: Codex CLI through [`mova-teacher`](https://md.mova.elamur.ai/free/en/docs/teacher/), no Claude API, no retry. Terra (`gpt-6.1-terra`, the `mova-teacher` default) takes the queue's `plain` entries and Sol (`gpt-6.1-sol`) the `hard:*` ones, both at medium effort; `MOVA_TR_TERRA_MODEL` and `MOVA_TR_SOL_MODEL` override the names.

1. **Queue to rows.** `examples-queue.tsv` names (pair, list, entry key, context); the n-th queue entry takes the n-th matching list row, so duplicate keys stay distinct. Entries are grouped by pair and teacher, in frequency order, and cut into batches of 20 (`--batch`). A batch mixes directions and lists; each entry states its own.
2. **One call per batch.** The prompt gives, per entry, the headword, context, recommended and avoided renderings, the list's note, and its patterns (`src_match`, `ctx_match`, `good_match`, `bad_match`). The teacher answers JSON `{"items":[{"n","src","tgt","used","note"}]}`: one sentence of its own, its translation, the rendering used, a one-line note.
3. **Gates** (every failing reason is recorded): both texts non-empty, no tabs or line breaks, script of the language, one sentence, length and length-ratio bounds, the source matches `src_match` (and `ctx_match`), the translation matches `good_match` and not `bad_match`, no duplicate source. A pattern is a form list, so an inflected form outside it is rejected; that costs yield, not correctness.
4. **Mova's parse** of the English and French sides (`mova-parse`, `en.bin`, `fr.bin`; Ukrainian has no model): the findings `no-root`, `multi-root`, `parse-none`, `verb-not-root(root=X)`, `detected-<lang>` and `sentence-count` are recorded, never a rejection: a parse failure is a finding about the model.
5. **Second call for the hard entries** (`--verify hard`, default): Terra back-translates each Sol pair without being asked to match the source, then judges `verdict` (`same`, `close`, `different`), `trap_ok` and `natural`; `different`, a misused trap or an unnatural side rejects. `--verify all` also checks the plain ones, `none` checks nothing.
6. **Accept.** An accepted pair gets `ours-tr:<pair>:<n>` (one counter per pair across both lists), a row in `languages/translation/<kind>/<pair>.ours.tsv` (columns `id src_lang tgt_lang src tgt note headword context list teacher model verify parse prov`; Elamur Cosmic License) and its list row becomes `example_id` = that id, `example_ok` = `ours`, `example_status` = `constructed`. `choices-md` reads the sibling `<pair>.ours.tsv` and renders the pair in the entry; re-render the manuals after a run.

State lives outside the repo, in the work directory: `results.tsv` (append-only; the last row per entry wins; statuses `accepted`, `rejected`, `held`), `calls/NNN-<teacher>-<gen|verify>.json` (prompt, reply, token usage) and `codex-calls.jsonl`. A rejected entry is not requeued by a rerun. If a verification call fails after generation, the passed rows stay `held` and the next run verifies them first. `--calls N` caps the calls of one run; a batch starts only if its generation and verification fit. The first transport or limit error stops the run.

```
parallel tr-run languages/translation/examples-queue.tsv languages/translation $MOVA_RUNS/tr-examples-2026-11-06 2026-11-06 --teacher terra --pair en-uk --calls 1
parallel tr-report languages/translation/examples-queue.tsv languages/translation $MOVA_RUNS/tr-examples-2026-11-06
```

Checks: unit tests with a negative control for every gate (a gate that cannot fail is not a gate), the reply decoder, the parse-finding reader, `choices-md` with our pairs (and a pair of a foreign language refused), and two tests on the real lists: every list rewrites byte for byte, and the whole queue resolves to distinct rows. A run with a fake `codex` (`MOVA_CODEX_BIN`) returning one good, one wrong-context and one avoided-form answer accepted 3 and rejected 2 as expected.

## Checks

Unit tests: sentence splitting with abbreviations, 1-1 on equal lengths, 2-1 when a target merges two sources, and a negative control (a 5x length mismatch scores below 0.001); for `choices`, tokenisation (French apostrophes split, Ukrainian kept), exact, prefix and multi-word patterns, counts and example choice on a tiny corpus, and a negative control (a wrong pre-filled example id must show `no-match`).

## Limits

Length-only alignment: no lexical anchors, so a mistranslated or cut passage can drift until the next chapter anchor; books with different chapter counts or long notes align poorly (Candide, Zadig and The Black Tulip were measured and excluded). The dialogue dash and front matter survive in a few rows. Alignment quality is checked by eye on samples, not against a gold standard. The aligner and `choices.rs` have no groupant sidecars yet and are not registered in `models/contracts.json`: a known gap.

## Sources and citations

Gale and Church 1993 (the aligner); Koehn 2005 Europarl; Tiedemann 2012 OPUS; Steinberger et al. 2006 JRC-Acquis and 2012 DGT-TM; Eisele and Chen 2010 MultiUN; Ziemski et al. 2016 UN Parallel Corpus (ziemski-2016-united-nations-parallel-corpus-v1); Schwenk et al. 2021 WikiMatrix. Data licences: [Data and model licences](https://md.mova.elamur.ai/free/en/docs/licence/DATA/).

- [← previous pageviews — Wikipedia page views for every language at once](https://md.mova.elamur.ai/free/en/docs/tools/pageviews/)
- [next → parsecmp — many parsers against the reference trees, and parser agreement](https://md.mova.elamur.ai/free/en/docs/tools/parsecmp/)
- [↑ back to tools](https://md.mova.elamur.ai/free/en/docs/tools/)