[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/le-2022-referring-expressions-rational-speech-act/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach

# Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach

**Authors:** Hieu Le, Taufiq Daryanto, Fabian Zhafransyah, Derry Wijaya, [Elizabeth Coppock](https://md.mova.elamur.ai/free/en/docs/papers/authors/elizabeth-coppock/), Sang Chin · **Year:** 2022 · **Venue:** arXiv:2205.07795 (preprint) **Link:** [arxiv.org/abs/2205.07795](https://arxiv.org/abs/2205.07795) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (arXiv)

[**1** citations](https://openalex.org/works?filter=cites:W4307656474) openalex · 2026-10-18[**1** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/le-2022-referring-expressions-rational-speech-act/) of the 140673 papers whose bibliographies we read[arXiv:2205.07795](https://openalex.org/S4306400194)[licence](http://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://arxiv.org/abs/2205.07795)
- [PDF](https://arxiv.org/pdf/2205.07795)
- [arXiv 2205.07795](https://arxiv.org/abs/2205.07795)
- [OpenAlex](https://openalex.org/W4307656474)

**Authors (6)**

1. [Hieu Le](https://openalex.org/A5101977591) [ORCID](https://orcid.org/0000-0001-7855-2778) · h 3 · 65 citations · 12 works
2. [Taufiq Daryanto](https://openalex.org/A5051502019) h 1 · 1 citations · 3 works
3. [Fabian Zhafransyah](https://openalex.org/A5024229022) h 1 · 12 citations · 4 works
4. [Derry Wijaya](https://openalex.org/A5027174211) [ORCID](https://orcid.org/0000-0002-0848-4703) · h 19 · 1836 citations · 178 works
5. [Elizabeth Coppock](https://openalex.org/A5074610868) [ORCID](https://orcid.org/0000-0001-9987-344X) · h 17 · 1130 citations · 80 works
6. [Sang Chin](https://openalex.org/A5103044811) [ORCID](https://orcid.org/0000-0002-1913-4223) · h 14 · 942 citations · 70 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper generates referring expressions ("the dog with the frisbee") for objects in photographs with the Rational Speech Act (RSA) model instead of a neural speaker. A scene graph (object types, attributes, relations) is first extracted with pre-trained Detectron2 and Graph R-CNN, and ordinal relations (left, right, second from left) are added by sorting objects by x-coordinate. The speaker adds one descriptor at a time until the listener's entropy over objects falls below a threshold or four descriptors are used. On RefCOCO and RefCOCO+ the expressions are checked by human choice among boxes. The method does not beat an end-to-end neural model trained on full data, and the authors trace many failures (boxes with no match, or not highlighted) to the vision step rather than to the pragmatic one. The tables are hard to align in the text read here, so the card quotes only one clear figure: an adjusted accuracy of 52.54% on RefCOCO+.

## Ideas

- I1. `prag/src/rsa.rs`: a deterministic version of the speaker. At each step choose the descriptor that most reduces the set of candidate referents, with ties broken by the descriptor string in `BTreeMap` order, and stop when one candidate remains or after a fixed length. Compare length and success with the sampling version on the school lessons.
- I2. `world` thresholds: the paper fixes one threshold for attributes and one for relations by hand, and says it would learn them from data. Fit each threshold on a training split and report it on a held-out split (`cv`), never on the test set.
- I3. Ordinal descriptors as a rule in the story world: sort the same-kind objects by position in the text (first, second, last) and add "second from left"-style descriptors. Test on the school scene questions, with a negative control where the order is shuffled.
- I4. Keep the trace: store the descriptor sequence and the entropy after each step for each referring expression. This makes each choice explainable (the "why" the project needs), and the trace is the input to an error table like the paper's.

## For Mova

The speaker and listener belong with `prag/src/rsa.rs` and `prag/src/infer.rs`. The scene and facts are in `world/src/knowledge.rs` and `world/src/events.rs`. Referring expressions and the choice of descriptor relate to [35. Reference and coreference](https://md.mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/) and to the quantity implicature in [33. Implicature and presupposition](https://md.mova.elamur.ai/free/en/docs/languages/en/33-implicature-and-presupposition/). Measures: expression length and success on the school scene questions, with the random-order control, and the four error categories on a held-out scene set.

Topics: [Speech act theory](https://md.mova.elamur.ai/free/en/docs/papers/topics/speech-act-theory/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2205.07795](https://arxiv.org/abs/2205.07795)

```
@misc{le2022referring,
  title = {{Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach}},
  author = {Le, Hieu and Daryanto, Taufiq and Zhafransyah, Fabian and Wijaya, Derry and Coppock, Elizabeth and Chin, Sang},
  year = {2022},
  eprint = {2205.07795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.07795}
}
```

- [← previous Coordination of unmatched clause types and dynamic look-ahead](https://md.mova.elamur.ai/free/en/docs/papers/lasersohn-2026-coordination-unmatched-clause-types-dynamic/)
- [next → Scalar Implicatures: Pragmatic Inferences or Grammar?](https://md.mova.elamur.ai/free/en/docs/papers/lee-2008-scalar-implicatures-pragmatic-inferences-or/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. W. Monroe and C. Potts, “Learning in the Rational Speech Acts model,” CoRR, vol. abs/1510.06807, 2015. [↗](https://arxiv.org/abs/1510.06807)
2. S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, “ReferItGame: Referring to objects in photographs of natural scenes,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), (Doha, Qatar), pp. 787–798, ACL, Oct. 2014. [find](https://scholar.google.com/scholar?q=S%2E+Kazemzadeh%2C+V%2E+Ordonez%2C+M%2E+Matten%2C+and+T%2E+Berg%2C+%E2%80%9CReferItGame%3A+Referring+to+objects+in+photographs+of+natural+scenes%2C%E2%80%9D+in+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+)
3. M. C. Frank and N. D. Goodman, “Predicting pragmatic reasoning in language games,” Science, vol. 336, no. 6084, p. 998, 2012. [find](https://scholar.google.com/scholar?q=M%2E+C%2E+Frank+and+N%2E+D%2E+Goodman%2C+%E2%80%9CPredicting+pragmatic+reasoning+in+language+games%2C%E2%80%9D+Science%2C+vol%2E+336%2C+no%2E+6084%2C+p%2E+998%2C+2012%2E)
4. J. Degen, R. X. D. Hawkins, C. Graf, E. Kreiss, and N. D. Goodman, “When redundancy is rational: A Bayesian approach to ’overinformative’ referring expressions,” CoRR, vol. abs/1903.08237, 2019. [↗](https://arxiv.org/abs/1903.08237)
5. J. Andreas and D. Klein, “Reasoning about pragmatics with neural listeners and speakers,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, (Austin, Texas), pp. 1173–1182, ACL, Nov. 2016. [find](https://scholar.google.com/scholar?q=J%2E+Andreas+and+D%2E+Klein%2C+%E2%80%9CReasoning+about+pragmatics+with+neural+listeners+and+speakers%2C%E2%80%9D+in+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+%28Austin%2C+Texas%29%2C+pp)
6. R. Cohn-Gordon, N. D. Goodman, and C. Potts, “Pragmatically informative image captioning with character-level reference,” CoRR, vol. abs/1804.05417, 2018. [↗](https://arxiv.org/abs/1804.05417)
7. J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph R-CNN for scene graph generation,” CoRR, vol. abs/1808.00191, 2018. [↗](https://arxiv.org/abs/1808.00191)
8. Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” [https://github.com/facebookresearch/detectron2](https://github.com/facebookresearch/detectron2), 2019. [↗](https://github.com/facebookresearch/detectron2)
9. L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” CoRR, vol. abs/1608.00272, 2016. [↗](https://arxiv.org/abs/1608.00272)
10. G. Scontras, M. H. Tessler, and M. Franke, “Probabilistic language understanding: An introduction to the rational speech act framework,” 2018. [find](https://scholar.google.com/scholar?q=G%2E+Scontras%2C+M%2E+H%2E+Tessler%2C+and+M%2E+Franke%2C+%E2%80%9CProbabilistic+language+understanding%3A+An+introduction+to+the+rational+speech+act+framework%2C%E2%80%9D+2018%2E)
11. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems (NIPS), 2015. [find](https://scholar.google.com/scholar?q=S%2E+Ren%2C+K%2E+He%2C+R%2E+Girshick%2C+and+J%2E+Sun%2C+%E2%80%9CFaster+R%2DCNN%3A+Towards+real%2Dtime+object+detection+with+region+proposal+networks%2C%E2%80%9D+in+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+2015%2E)
12. W. Abdulla, “Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow.” [https://github.com/matterport/Mask_RCNN](https://github.com/matterport/Mask_RCNN), 2017. [↗](https://github.com/matterport/Mask_RCNN)
13. R. Luo and G. Shakhnarovich, “Comprehension-guided referring expressions,” CoRR, vol. abs/1701.03439, 2017. [↗](https://arxiv.org/abs/1701.03439)
14. D. Lassiter and N. Goodman, “Adjectival vagueness in a Bayesian model of interpretation,” Synthese, vol. 10, pp. 3801–3836, 2017. [find](https://scholar.google.com/scholar?q=D%2E+Lassiter+and+N%2E+Goodman%2C+%E2%80%9CAdjectival+vagueness+in+a+Bayesian+model+of+interpretation%2C%E2%80%9D+Synthese%2C+vol%2E+10%2C+pp%2E+3801%E2%80%933836%2C+2017%2E)
15. D. Edgington, “The philosophical problem of vagueness,” Legal Theory, vol. 7, pp. 371–378, 2001. [find](https://scholar.google.com/scholar?q=D%2E+Edgington%2C+%E2%80%9CThe+philosophical+problem+of+vagueness%2C%E2%80%9D+Legal+Theory%2C+vol%2E+7%2C+pp%2E+371%E2%80%93378%2C+2001%2E)
16. L. Yu, H. Tan, M. Bansal, and T. L. Berg, “A joint speaker-listener-reinforcer model for referring expressions,” 2017. [find](https://scholar.google.com/scholar?q=L%2E+Yu%2C+H%2E+Tan%2C+M%2E+Bansal%2C+and+T%2E+L%2E+Berg%2C+%E2%80%9CA+joint+speaker%2Dlistener%2Dreinforcer+model+for+referring+expressions%2C%E2%80%9D+2017%2E)
17. J. Kim, H. Ko, and J. Wu, “CoNAN: A complementary neighboring-based attention network for referring expression generation,” in Proceedings of the 28th International Conference on Computational Linguistics, (Barcelona, Spain (Online)), pp. 1952– 1962, International Committee on Computational Linguistics, Dec. 2020. [find](https://scholar.google.com/scholar?q=J%2E+Kim%2C+H%2E+Ko%2C+and+J%2E+Wu%2C+%E2%80%9CCoNAN%3A+A+complementary+neighboring%2Dbased+attention+network+for+referring+expression+generation%2C%E2%80%9D+in+Proceedings+of+the+28th+International+Conference+on+Computational+Linguis)
18. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the NLP world,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, (Online), pp. 6282–6293, Association for Computational Linguistics, July 2020. [find](https://scholar.google.com/scholar?q=P%2E+Joshi%2C+S%2E+Santy%2C+A%2E+Budhiraja%2C+K%2E+Bali%2C+and+M%2E+Choudhury%2C+%E2%80%9CThe+state+and+fate+of+linguistic+diversity+and+inclusion+in+the+NLP+world%2C%E2%80%9D+in+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+f)