Elamur: **Mova** · [Parse](https://mova.elamur.ai/free/en/parse/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Translate](https://md.translate.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · [MCP](https://mcp.elamur.ai/) · [Groupants](https://md.groupants.elamur.ai/free/en/)

[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/wordgraph/tamp/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [wordgraph](https://md.mova.elamur.ai/free/en/docs/wordgraph/) / wordgraph-tamp — the Elephant's tamping step over papers

# wordgraph-tamp — the Elephant's tamping step over papers

🐘 The Elephant keeps the word graph ([`wordgraph`](https://md.mova.elamur.ai/free/en/docs/wordgraph/), [the animals](https://md.mova.elamur.ai/free/en/docs/repo-docs/animals/)). Tamping is its pass over what papers say: statements that many papers say alike — how they train with Adam, what CIFAR-10 holds, that results are averaged over seeds — are collapsed into shared **approach nodes**, and each paper keeps what is its own. A node is a reverse distillation of what large models absorb from repetition, built natively: Mova reads, Rust counts, no language model writes or judges anything here.

It is a separate crate next to the graph because it reads papers through the Whales' reader ([`tools/whale`](https://md.mova.elamur.ai/free/en/docs/tools/whale/), `reader.rs`), whose dependencies stay out of the root workspace. It uses the graph (lemma ids, `Graph::freq`, typed neighbour profiles) through the `wordgraph` library.

```
CARGO_TARGET_DIR=… cargo build --release --offline --manifest-path wordgraph/tamp/Cargo.toml
elephant-tamp read STORE ROOT [--jobs N] [--limit N]           # Mova reads every ROOT/*/eprint.md once
elephant-tamp build STORE ROOT OUT [--graph WG] [--tau X] [--min-papers N] [--min-cos X]
                    [--min-typicality X] [--min-lift X] [--max-anchor-share X] [--min-score X] [--jobs N]
elephant-tamp controls STORE ROOT [--graph WG] [same options]  # the negative controls (JSON)
elephant-tamp probe STORE ROOT LEMMA                           # statements using a lemma (TSV)
```

Runs need `WORLD_EN_MODEL` (the `en` model) for `read`; `--graph` takes a word-graph file. Heavy runs go through `tools/jobs`. Outputs hold sentences of the papers: internal, under the papers' licences, never published.

## Model

**Statements.** `read` keeps, per paper, every admitted sentence of a read part (`reader::read`: references, acknowledgement headings and LaTeX math are not read) with its restated clauses and the parse's lemma and UPOS per token (`reader::Sentence::lemmas`). The record is cached under the text's SHA-256 and the reader version (`compose::WRITER` + record layout); a changed text or reader is read again.

**Normalization** (`norm.rs`). A statement's content sequence: lemma ids of the tokens its clauses restate (all tokens when none is restated), UPOS NOUN, PROPN, VERB, ADJ, ADV or NUM, case-folded, every number (and a lemma starting with a digit, `1e-4`) one id, light verbs and connectives left out. Dictionary lemmas keep the `en` dictionary number (the word graph's ids); others `OOV_TAG | fnv`. **Shingles**: ordered pairs at distance one or two. Statements with fewer than 5 content lemmas are scored but not clustered.

**Synonyms** (`synonym.rs`). WordNet proposes, the graph confirms: v in w's first-sense synonyms (same part of speech, both used by ≥ 2 papers) and the cosine of their typed neighbour profiles in the word graph (relation, direction, neighbour; PMI weights) ≥ 0.15; British/American spelling variants without the graph. Union-find, the smallest id represents a class.

**Template nodes** (`cluster.rs`): near-identical statements. MinHash LSH, 64 bands × 3 rows (candidate probability 0.40 at Jaccard 0.2, 0.83 at 0.3, 0.985 at 0.4), exact Jaccard verification, an edge at J ≥ τ = 0.4 between different papers. Star clustering, no chaining: sets in order of statement document frequency, an unassigned set is a centre and takes its unassigned neighbours; ≥ 3 papers make a node.

**Anchor nodes** (`anchor.rs`): what papers typically say around a term. For a noun/proper-noun lemma t used by 5 papers up to 40 % of them, with T(t) its statements: the profile Π(t) holds the shingles of T(t) that occur in at least max(3, 5 % of its papers) papers *and* are said in T(t) at least 10 times more often than in all statements (lift). Typicality τ_t(s) = |sh(s) ∩ Π(t)| / |sh(s)|. A member has τ ≥ 0.5 with ≥ 3 shared pairs; a statement typical for several anchors goes to the highest τ × salience, salience = ln(corpus share / Wikipedia share in the word graph). Anchors whose nodes take the same name merge. Name: the adjacent pair with the anchor that half the members use («Adam optimizer», «beam search»).

**The order test** (both kinds). A match must hold in the statement's own order: τ (or J) must beat the best of 8 seeded shuffles of the same words by 0.2 (`norm::order_holds`). Without it the shuffle control was red (below).

**Canonical explanation**: Mova's restated clauses of the canonical member — for an anchor a definitional member («t is/consists/computes …», «t (…»), else the most typical; for a template the centre — plus `core`, the member words at least half of the members share, in its order, and every variant with its fit and paper.

**Importance** (`importance.rs`): score = 0.35 rarity + 0.25 specificity + 0.20 own + 0.15 result + 0.05 part, × 0.25 for boilerplate. Rarity = 1 − ln(df)/ln(N) with df the papers holding a statement of J ≥ τ (0.5 when too short to cluster); specificity = mean of the three highest lemma idfs; own = first person or «this paper/work» with a contribution verb; result = a number with %, a metric or a comparison verb; part +1 abstract/contributions/conclusion, +0.5 introduction, −1 related work/background. A node member is boilerplate unless it carries a **payload**: a lemma outside the node's shared words used by at most 1 % of the papers (then it stays, and the node is still referenced). Important: score ≥ 0.5 and not boilerplate. The weights are a stated operating point, not fitted.

**Compression** per paper: each boilerplate member is replaced by one reference `[approach ID: NAME]`, written once per node. `ratio_refs` = words / words with references; `ratio_important` = words / (important statements + refs).

**Invariants.** Deterministic: sorted maps, seeded hashes, ties by text and index; output bytes do not depend on the thread count. A node member is within τ of its centre (template) or passes the anchor's tests (anchor); every reference names an existing node; a paper record is bound to its text hash and reader.

**Cost.** Reading ≈ 0.7 CPU-s per paper (Mova parse plus the grammar gate per clause); build O(U·(K + B log U)) for U statements, K = 192 hashes, B = 64 bands, plus per-anchor profile counting O(Σ|T(t)|·|sh|).

## Prospective predictions (written 2026-11-06 before the first build) and measurements

|  | predicted | measured (2026-11-06, 1,763 e-prints) |
| --- | --- | --- |
| statements | ≈ 440 k, 85 % clusterable | 486,401; 439,841 (90 %) |
| nodes (τ 0.5, 3 papers) | 1,000–5,000 | 378 (τ 0.5, template only, first build) |
| statements in nodes | 2–5 % | 0.4 % (first build); final 0.64 % |
| ratio_refs | 1.03–1.10 | final 1.003 (p50 1.001, p90 1.008, max 1.149) |
| shuffle control | ≈ 0 nodes | red before the order test (201 nodes); 11 after |
| build | < 3 min, < 3 GB | ≈ 60 s, 1.7 GB peak; read 220 s for all papers with 6 threads, 0.76 GB |

The predictions were wrong by an order of magnitude: in these papers repeated explanations are rarely near-identical statements. Statements mentioning «Transformer» (582 papers) have typicality p90 0.19, max 0.57 — each paper words it its own way — so no Transformer node forms; Adam, beam search, BLEU, CIFAR-10, COCO do.

Final build (defaults above, word graph `wiki-50k.wg`): 557 nodes (194 anchor, 363 template), 3,122 statements, 43,171 of 9,568,093 words boilerplate; 144,169 important statements; ratio_important 2.83. Coherence: template member pairs with J < 0.2: 0.6 %; anchor member pairs: 57 % (anchor members share the profile, not each other's words). Single linkage at the same τ would join 195 sets of 162 papers into one component; the star keeps nodes apart. Without the graph: 534 nodes, 3,039 statements (the graph adds 551 synonym pairs and the salience).

Hand check (2026-11-06, a small finite sample): every 10th of the first 200 nodes of the τ 0.4 / typicality 0.5 build before the order test, its canonical member and its two worst-fitting members from other papers (60 statements): 20/20 nodes on one topic; 1 node («FID») groups result statements rather than an explanation; 1 («leftmargin») was LaTeX residue, now excluded. Synonyms: 13 of 16 sampled pairs are synonyms in context.

Controls (`controls`, final code): shuffled-only corpus 11 nodes (4 anchor, 7 template) against 557 real; real plus a shuffled copy of every statement as another paper: 113 copies join a node (50 of them shuffles that left the order nearly unchanged, J ≥ 0.5 to the original; 385 shuffles are identities); foreign statements (from papers that never use the anchor) asked against each anchor's profile: 0 of 1,348 pass. Two identical runs and a run with 3 threads give byte-identical outputs.

## Writer hook

`mova_whale::compose::Approaches::from_record(record, paper_text)` reads a paper record (refused for another text or reader); `compose_with_approaches(reading, known, guard, max, &approaches)` writes the description with each node's repeated explanation replaced by one sentence naming the node, and over the word budget keeps the most important sentences first. API only: the Owls' configuration does not use it.

## Sources

Library ids are folders of the paper library; works marked «outside» have no card yet (`docs/usage.tsv`).

- Shingles and Jaccard resemblance: broder-1997 (resemblance and containment; syntactic clustering of the web).
- MinHash banding as locality-sensitive hashing: Indyk & Motwani 1998 (outside); NLP use ravichandran-2005; near-duplicates at web scale henzinger-2006, manku-2007; random-projection relative charikar-2002.
- Star clustering: Aslam, Pelekhov & Rus 2004 (outside).
- Term specificity and tf-idf: Sparck Jones 1972 (outside), salton-1975, robertson-2009 (BM25 as the probabilistic alternative); synonym profiles after baroni-2010.

## Limits

- Repeated *explanations* of an approach across papers are mostly paraphrases, not near-identical: at these thresholds the nodes are dominated by setup and reporting boilerplate (hyper-parameters, dataset sizes, «comparison with the state of the art», funding, checklists). The compression ratio is small (≈ 1.003); paraphrase-level merging needs frames or meaning, not shingles.
- Duplicate library entries (two versions of one paper) count as two papers.
- Anchors are single lemmas; «GPT» covers GPT-2/3/4 (the tokenizer splits «GPT-3»).
- Template + payload is decided by word rarity only.
- The shuffle control still lets short statements through (5–7 content lemmas keep runs under a random shuffle).
- No word senses: «Adam» the optimizer and Adam of Genesis share an anchor; the profile keeps the majority sense.
- [↑ back to wordgraph — the word graph](https://md.mova.elamur.ai/free/en/docs/wordgraph/)