Elamur: **Mova** · [Parse](https://mova.elamur.ai/free/en/parse/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Translate](https://md.translate.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · [MCP](https://mcp.elamur.ai/) · [Groupants](https://md.groupants.elamur.ai/free/en/)

[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/tools/pageviews/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [tools](https://md.mova.elamur.ai/free/en/docs/tools/) / pageviews — Wikipedia page views for every language at once

# pageviews — Wikipedia page views for every language at once

`pageviews` turns the Wikimedia monthly `pageview_complete` **user** dumps into per-language article view counts, so that every language's Wikipedia pass can walk its articles from the most viewed down. One pass over each monthly dump covers all `*.wikipedia` projects; the raw `.bz2` is never stored.

Source: Wikimedia, *Pageview complete dumps* ([https://dumps.wikimedia.org/other/pageview_complete/readme.html](https://dumps.wikimedia.org/other/pageview_complete/readme.html), data under [https://dumps.wikimedia.org/other/pageview_complete/monthly/](https://dumps.wikimedia.org/other/pageview_complete/monthly/)), CC0. Line format `project title page_id access count hourly`; documentation: [https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Pageview_actor](https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Pageview_actor) and the dump readme above. The English-only precursor is `pvtitle` of the Grand Spider.

## How it works

1. **`month`** streams `curl --limit-rate 40M` into `lbzip2`/`pbzip2` when installed, else `bzip2 -dc`, into the parser. The dump is sorted by project, so one project is aggregated at a time in a hash map and flushed when the project changes: memory is that of the largest project of the month. The grouping is checked on the real stream; a project that returns after it was left is an error (the tool stops; the external-bucketing fallback was not needed and is not implemented). Service titles are dropped (see below). Output per project and month, title-sorted, zstd level 6: `months/<ym>/<project>.tsv.zst`, rows `title<TAB>page_id<TAB>views` (page id 0 = unknown, as in the dump's `null`; views summed over access methods). `months/<ym>/_stats.tsv` lists per project titles, views, lines, dropped lines. The month appears under its final name only after the whole stream ended cleanly (until then it is `<ym>.part`).
2. **`merge`** per project: a k-way merge of the twelve monthly files by title sums views, an external sort (zstd spill runs of `--chunk-mb`, default 512 MB, removed afterwards) orders them, and `<project>-12m.tsv.zst` is written. The order is [`views_order`](https://md.mova.elamur.ai/free/en/docs/tools/pageviews/): **views descending, ties by title bytes ascending**. `summary-12m.tsv` lists project, articles, total views, months.
3. **`ns-fetch`** builds `data/namespaces.tsv` from each wiki's `action=query&meta=siteinfo` (local, canonical and alias names of every non-main namespace, and the main-page title); 364 Wikipedias on 2026-11-06. It is compiled in.

Titles are those of the dump (requested title, redirects included, underscores); `page_id` is the target page, so redirect titles share their target's id. Titles with no page (id 0) are scanner noise and 404s: filter on `page_id > 0` or resolve against the dump's article list. The Grand Spider's file counts every title the same way.

## Run

```
cargo build --release --manifest-path tools/pageviews/Cargo.toml
jobs run --name pv-2025-10 --mem-gb 4 -- pageviews month 2025-10 --root $MOVA_DATA/views
pageviews merge --root $MOVA_DATA/views [--projects fr.wikipedia,uk.wikipedia] [--threads 3] [--chunk-mb 512]
pageviews export $MOVA_DATA/views/fr.wikipedia-12m.tsv.zst > fr-views.tsv      # title<TAB>views, plain
pageviews compare A.tsv.zst B.tsv N                                           # top-N overlap
```

At most two downloads at a time (the server answers 429 to more). `cargo test` covers parsing, grouping (with a negative control for an ungrouped dump), determinism and the external sort against the in-memory path.

## Using the views order in a pass

`wikicorpus graph` already takes a plain `title<TAB>views` file (its `views` argument; the score is `ln(1 + inbound) + ln(1 + views)`, redirect views are added to their article). Replace the empty/dummy views file of a French or Ukrainian graph with `pageviews export <project>-12m.tsv.zst`. For a pure views order read the `.tsv.zst` in file order (it is already sorted by `views_order`) and keep the titles that are articles of the dump.

## Limits

- Service namespaces come from each wiki's namespace table as of 2026-11-06; a wiki founded later, or a namespace added later, falls back to the English canonical names. A main-namespace title that begins with a namespace name is dropped.
- Pageviews include automated traffic that the `user` agent type did not catch, and 404 titles; views are not ranks of importance. The 12 months are 2025-10 … 2026-09, as in the English file.
- Bots/`spider` and `automated` agent types are not in the `user` dump by design.

## Measurements (2026-11-06)

Box under load 38–47 on 20 cores, everything at `nice 19` under `tools/jobs` with a 4 GB limit; no multi-threaded bzip2 was installed, so each month is one `bzip2 -dc` core.

| step | result |
| --- | --- |
| `month`, 12 months 2025-10 … 2026-09 (three at a time) | 34–62 min each, 2.4 h in all (one month was retried after a network drop; nothing was published from the failed stream); peak 1.9–2.1 GB (curl + bzip2 + parser); 363–369 Wikipedias, about 114 M titles per month |
| `merge`, 378 projects, 3 threads, 400 MB chunks | 321 s, peak 1.2 GB |
| size on disk | monthly files 11 GB, 12-month files 1.8 GB (`en` 446 MB, `fr` 72 MB, `uk` 36 MB) |
| the dump is grouped by project | yes: no project came back in any of the 12 real streams |
| determinism | the first 120 MB of the 2025-10 dump read again gave 25 project files byte-identical to the full run's (the cut-off last one excluded); the merge's spill path equals its in-memory path (unit test) |
| `en.wikipedia` against the Grand Spider's `enwiki-12m.tsv` | top 1000: same titles, same views, same positions (1000/1000); top 10 000: 9999 common, same views for all of them, 2072 same positions (his ties are in hash order); top 100 000: 99 994 common. Mine has 38.4 M titles against his 44.2 M: the full namespace table drops more service titles |

Top Wikipedias of the 12 months (articles = distinct titles, views in billions): en 38.4 M / 75.6, ja 2.9 M / 10.1, de 5.8 M / 7.9, ru 5.5 M / 7.3, fr 5.3 M / 6.7, es 4.6 M / 5.9, it 3.7 M / 4.4, zh 6.9 M / 4.3, pl 2.6 M / 2.1, pt 2.5 M / 2.0; the full table is `summary-12m.tsv`.

The top of a pure views order contains automated and noise titles (`fr`: «Cookie_(informatique)» 172 M, «XXX»; `uk`: «Xnxx.com», `wiki.phtml` with page id 0). Filter `page_id > 0` and article status, or combine with in-links as the `wikicorpus graph` score does.

- [← previous Crab's deputy planner (mova-octopus)](https://md.mova.elamur.ai/free/en/docs/tools/octopus/)
- [next → parallel — parallel texts in one TSV (English–French, English–Ukrainian, French–Ukrainian)](https://md.mova.elamur.ai/free/en/docs/tools/parallel/)
- [↑ back to tools](https://md.mova.elamur.ai/free/en/docs/tools/)