[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/wilkinson-2016-gold-standard-scalar-adjectives/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / A Gold Standard for Scalar Adjectives

# A Gold Standard for Scalar Adjectives

**Authors:** Bryan Wilkinson and Tim Oates · **Year:** 2016 · **Venue:** Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) **Link:** [aclanthology.org/L16-1424](https://aclanthology.org/L16-1424/) · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (ELRA, LREC proceedings) — summary only; evaluation only, nothing copied into a release

[**10** citations](https://openalex.org/works?filter=cites:W2575264276) openalex-snapshot · 2026-10-14[**9** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/wilkinson-2016-gold-standard-scalar-adjectives/) of the 140673 papers whose bibliographies we read[Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)](https://aclanthology.org/volumes/L16-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L16-1424/)
- [PDF](https://aclanthology.org/L16-1424.pdf)
- [DOI 10.63317/5q95e9epid2n](https://doi.org/10.63317/5q95e9epid2n)
- [OpenAlex](https://openalex.org/W2575264276)

**Authors (2)**

1. [Bryan Wilkinson](https://openalex.org/A5073266752) h 3 · 17 citations · 4 works
2. [Tim Oates](https://openalex.org/A5114778025) [ORCID](https://orcid.org/0000-0002-8655-747X) · h 34 · 7465 citations · 308 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

The paper builds a gold standard for two things: which adjectives belong on one scale (for example size, or temperature), and in what order. Two Mechanical Turk tasks were run. In the first, 500 workers were shown three prompt words from one of 12 postulated scales and asked to list other adjectives that belonged. Free lists were aggregated with Cultural Consensus Theory (CCT), using a bias term for list length, and the first factor's eigenvalue ratio measured consensus (3:1 is the usual threshold). In the second task, 200 workers dragged the kept words into an order, and CCT again gave the order. The result is 12 ordered scales, with eigenvalue ratios from 29.47 (size) down to 3.52 (speed). Of the prompt words, 65% were judged correct, and 14 of 16 word groups showed a significant link between prompt status and membership. The gold standard was then used to score three automatic ordering methods on sliding windows of three words. The two methods that score highest are Kim and de Marneffe's (mean window accuracy 0.50 for the wider setting) and De Melo and Bansal's; the paper's Table 5 gives per-scale values up to 0.66. For speed and sameness no method was correct on any window.

## Ideas

- I1. `en/src/dict.rs` and `world/src/dictionary.rs`: store the 12 gold scales as ordered BTreeMap entries with a membership flag, so that word-sense lookups can answer "which is larger, medium or huge" by scale position. Keep the scale file small and build it from the lexicon through `build.rs`.
- I2. A deterministic consensus check for Mova's own scale lists: for each prompt set, count how often each word is listed, build the informant-by-word matrix, and take the first factor by power iteration. Report the eigenvalue ratio; a scale below 3:1 is marked as not a scale. Run twice and compare.
- I3. Evaluate Mova's scale order with the paper's three-word window test on the 12 scales. A shuffled order must score close to chance (one in three on these windows), as a negative control.
- I4. Store words that most informants reject (the paper lists "fresh", "difficult", "slow" as non-members of some scales) as explicit negative entries, so that the scale builder does not add them back.

## For Mova

The chapters are [15. Adjectives and adverbs](https://md.mova.elamur.ai/free/en/docs/languages/en/15-adjectives-adverbs/) (scalar adjectives and their order) and [12. Pronouns, determiners and numerals](https://md.mova.elamur.ai/free/en/docs/languages/en/12-pronouns-determiners-numerals/) (quantifiers such as "few", "some", "many"). The code is `en/src/dict.rs` for the lexical scales, `world/src/dictionary.rs` for word senses, and `prag/src/infer.rs` where the ordering feeds scalar inference. The measure: window accuracy on the 12 scales, run twice, against a shuffled-order negative control. Evaluation only: the gold standard is CC BY-NC and must not enter a release.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/5q95e9epid2n](https://doi.org/10.63317/5q95e9epid2n)

```
@inproceedings{wilkinson2016gold,
  title = {{A Gold Standard for Scalar Adjectives}},
  author = {Wilkinson, Bryan and Oates, Tim},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/5q95e9epid2n},
  url = {https://doi.org/10.63317/5q95e9epid2n}
}
```

- [← previous Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework](https://md.mova.elamur.ai/free/en/docs/papers/white-2017-inference-is-everything-recasting-semantic/)
- [next → Wiktionary as a Crowdsourced Lexicon for English Dialects](https://md.mova.elamur.ai/free/en/docs/papers/wong-2026-wiktionary-crowdsourced-lexicon-english-dialects/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)