[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Deep Biaffine Attention for Neural Dependency Parsing

# Deep Biaffine Attention for Neural Dependency Parsing

**Authors:** [Timothy Dozat](https://md.mova.elamur.ai/free/en/docs/papers/authors/timothy-dozat/), [Christopher D. Manning](https://md.mova.elamur.ai/free/en/docs/papers/authors/christopher-d-manning/) · **Year:** 2016 · **Venue:** arXiv:1611.01734 (ICLR 2017 version) **Link:** [arxiv.org/abs/1611.01734](https://arxiv.org/abs/1611.01734) · **Licence:** arXiv licence — summary only

[**481** citations](https://openalex.org/works?filter=cites:W2963571341) openalex · 2026-10-19[**626** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/missing-2016-dozat-deep-biaffine-attention-neural-dependency/) of the 140673 papers whose bibliographies we read[arXiv:1611.01734](https://openalex.org/S4306400194)[licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

- [Publisher's page](https://arxiv.org/abs/1611.01734)
- [PDF](https://arxiv.org/pdf/1611.01734)
- [arXiv 1611.01734](https://arxiv.org/abs/1611.01734)
- [OpenAlex](https://openalex.org/W2963571341)

**Authors (2)**

1. [Timothy Dozat](https://openalex.org/A5046428118) h 15 · 5002 citations · 35 works
2. [Christopher D. Manning](https://openalex.org/A5046006076) [ORCID](https://orcid.org/0000-0001-6155-649X) · h 140 · 185462 citations · 781 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The paper builds on the neural graph-based parser of Kiperwasser and Goldberg. It uses a larger and more heavily regularised BiLSTM encoder, passes the top recurrent states through dimension-reducing MLPs, and replaces the MLP attention and the affine label classifier with biaffine ones: one for arc scores over all head-dependent pairs and one for labels. Each arc and label score is a biaffine function of a head-side and a dependent-side vector, which keeps the classifier close in form to an affine one. On the English Penn Treebank the parser reaches 95.7 UAS and 94.1 LAS, which is 1.8 and 2.2 points above Kiperwasser and Goldberg and close to the best transition-based parser of the time (95.8 UAS). The paper also reports which hyperparameter choices change accuracy most.

## Ideas

- I1. gparse: check that the arc-head and arc-dependent representations, and the label-head and label-dependent representations, each pass through their own dimension-reducing MLP before the biaffine scorer, and add them if not; measure LAS on EWT dev (cv).
- I2. gparse: report UAS and LAS in the same run with separate loss sums for the arc and label heads, and check whether the label loss hurts the arc score on EWT dev, as the paper's joint training assumes it does not.
- I3. gparse: run the paper's hyperparameter sweep on dropout and the regularisers only (a small grid on EWT dev) before any architectural change, since its results say which settings move accuracy most; start only after the operator's go-ahead, because each GPU run is expensive.

## For Mova

The ideas touch the biaffine heads in `gparse` and its training loop, and they use EWT dev through the cv folds. The chapter that matters most is `languages/en/20-syntax-overview`, with `languages/en/23-complementation` for the label set. Measure first I2, which is cheap and tells whether the joint loss costs arc accuracy before any change to the architecture.

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 1611.01734](https://arxiv.org/abs/1611.01734)

```
@misc{dozat2016deep,
  title = {{Deep Biaffine Attention for Neural Dependency Parsing}},
  author = {Dozat, Timothy and Manning, Christopher D.},
  year = {2016},
  eprint = {1611.01734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.01734}
}
```

- [← previous A Corpus and Model Integrating Multiword Expressions and Supersenses](https://md.mova.elamur.ai/free/en/docs/papers/missing-2015-schneider-corpus-model-integrating-multiword-expressions/)
- [next → Characterizing the Errors of Data-Driven Dependency Parsing Models](https://md.mova.elamur.ai/free/en/docs/papers/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-based neural networks. In Association for Computational Linguistics, 2016. URL https://arxiv.org/abs/1603.06042. [↗](https://arxiv.org/abs/1603.06042)
2. Gabor Angeli, Melvin Johnson Premkumar, and Christopher D Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), 2015. [find](https://scholar.google.com/scholar?q=Gabor+Angeli%2C+Melvin+Johnson+Premkumar%2C+and+Christopher+D+Manning%2E+Leveraging+linguistic+structure+for+open+domain+information+extraction%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2014. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+International+Conference+on+Learning+Representations%2C+2014%2E)
4. Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A Smith. Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing, 2016. [find](https://scholar.google.com/scholar?q=Miguel+Ballesteros%2C+Yoav+Goldberg%2C+Chris+Dyer%2C+and+Noah+A+Smith%2E+Training+with+exploration+improves+a+greedy+stack%2DLSTM+parser%2E+Proceedings+of+the+conference+on+empirical+methods+in+natural+language+p)
5. Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. ACL 2016, 2016. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Jon+Gauthier%2C+Abhinav+Rastogi%2C+Raghav+Gupta%2C+Christopher+D+Manning%2C+and+Christopher+Potts%2E+A+fast+unified+model+for+parsing+and+sentence+understanding%2E+ACL+2016%2C+2016%2E)
6. Danqi Chen and Christopher D Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pp. 740–750, 2014. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Christopher+D+Manning%2E+A+fast+and+accurate+dependency+parser+using+neural+networks%2E+In+Proceedings+of+the+conference+on+empirical+methods+in+natural+language+processing%2C+pp%2E+740%E2%80%93750%2C+20) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2014-chen-fast-accurate-dependency-parser-neural/)
7. Hao Cheng, Hao Fang, Xiaodong He, Jianfeng Gao, and Li Deng. Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076, 2016. [find](https://scholar.google.com/scholar?q=Hao+Cheng%2C+Hao+Fang%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+and+Li+Deng%2E+Bi%2Ddirectional+attention+with+agreement+for+dependency+parsing%2E+arXiv+preprint+arXiv%3A1608%2E02076%2C+2016%2E)
8. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. Transition-based dependency parsing with stack long short-term memory. Proceedings of the conference on empirical methods in natural language processing, 2015. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Miguel+Ballesteros%2C+Wang+Ling%2C+Austin+Matthews%2C+and+Noah+A+Smith%2E+Transition%2Dbased+dependency+parsing+with+stack+long+short%2Dterm+memory%2E+Proceedings+of+the+conference+on+empirical+methods+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2015-dyer-transition-based-dependency-parsing-stack/)
9. Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, 2015. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+Dropout+as+a+bayesian+approximation%3A+Representing+model+uncertainty+in+deep+learning%2E+International+Conference+on+Machine+Learning%2C+2015%2E)
10. Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 2015. [find](https://scholar.google.com/scholar?q=Klaus+Greff%2C+Rupesh+Kumar+Srivastava%2C+Jan+Koutn%C4%B1%CC%81k%2C+Bas+R+Steunebrink%2C+and+Ju%CC%88rgen+Schmidhuber%2E+LSTM%3A+A+search+space+odyssey%2E+IEEE+Transactions+on+Neural+Networks+and+Learning+Systems%2C+2015%2E)
11. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587, 2016. [find](https://scholar.google.com/scholar?q=Kazuma+Hashimoto%2C+Caiming+Xiong%2C+Yoshimasa+Tsuruoka%2C+and+Richard+Socher%2E+A+joint+many%2Dtask+model%3A+Growing+a+neural+network+for+multiple+nlp+tasks%2E+arXiv+preprint+arXiv%3A1611%2E01587%2C+2016%2E)
12. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+International+Conference+on+Learning+Representations%2C+2014%2E)
13. Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327, 2016. [find](https://scholar.google.com/scholar?q=Eliyahu+Kiperwasser+and+Yoav+Goldberg%2E+Simple+and+accurate+dependency+parsing+using+bidirectional+LSTM+feature+representations%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A313%E2%80%93327) [our card](https://md.mova.elamur.ai/free/en/docs/papers/narrative-2016-kiperwasser-simple-accurate-dependency-parsing-bidirectional/)
14. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774, 2016. URL http://arxiv.org/abs/1611.05774. [↗](http://arxiv.org/abs/1611.05774)
15. Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL 2014, pp. 302–308, 2014. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+Dependency%2Dbased+word+embeddings%2E+In+ACL+2014%2C+pp%2E+302%E2%80%93308%2C+2014%2E)
16. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. Empirical Methods in Natural Language Processing, 2015. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+Empirical+Methods+in+Natural+Language+Processing%2C+2015%2E)
17. Ankur P Parikh, Hoifung Poon, and Kristina Toutanova. Grounded semantic parsing for complex knowledge extraction. In Proceedings of North American Chapter of the Association for Computational Linguistics, pp. 756–766, 2015. [find](https://scholar.google.com/scholar?q=Ankur+P+Parikh%2C+Hoifung+Poon%2C+and+Kristina+Toutanova%2E+Grounded+semantic+parsing+for+complex+knowledge+extraction%2E+In+Proceedings+of+North+American+Chapter+of+the+Association+for+Computational+Linguist)
18. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pp… [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Dan+Klein%2C+Christopher+D+Manning%2C+and+Yoram+Singer%2E+Feature%2Drich+part%2Dof%2Dspeech+tagging+with+a+cyclic+dependency+network%2E+In+Proceedings+of+the+2003+Conference+of+the+North+America)
19. Kristina Toutanova, Xi Victoria Lin, and Wen-tau Yih. Compositional learning of embeddings for relation paths in knowledge bases and text. In ACL, 2016. [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Xi+Victoria+Lin%2C+and+Wen%2Dtau+Yih%2E+Compositional+learning+of+embeddings+for+relation+paths+in+knowledge+bases+and+text%2E+In+ACL%2C+2016%2E)
20. David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. Structured training for neural network transition-based parsing. Annual Meeting of the Association for Computational Linguistics, 2015. [find](https://scholar.google.com/scholar?q=David+Weiss%2C+Chris+Alberti%2C+Michael+Collins%2C+and+Slav+Petrov%2E+Structured+training+for+neural+network+transition%2Dbased+parsing%2E+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2015%2E)