[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/findings-2021-wang-closer-look-into-robustness-neural/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / A Closer Look into the Robustness of Neural Dependency Parsers Using Better Adversarial Examples

# A Closer Look into the Robustness of Neural Dependency Parsers Using Better Adversarial Examples

**Authors:** Yuxuan Wang, [Wanxiang Che](https://md.mova.elamur.ai/free/en/docs/papers/authors/wanxiang-che/), [Ivan Titov](https://md.mova.elamur.ai/free/en/docs/papers/authors/ivan-titov/), [Shay B. Cohen](https://md.mova.elamur.ai/free/en/docs/papers/authors/shay-b-cohen/), Zhilin Lei, Ting Liu · **Year:** 2021 · **Venue:** Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 **Link:** [aclanthology.org/2021.findings-acl.207](https://aclanthology.org/2021.findings-acl.207/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**5** citations](https://openalex.org/works?filter=cites:W3174226126) openalex-snapshot · 2026-10-14[**3** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/findings-2021-wang-closer-look-into-robustness-neural/) of the 140673 papers whose bibliographies we read[Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021](https://aclanthology.org/volumes/2021.findings-acl/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.findings-acl.207/)
- [PDF](https://aclanthology.org/2021.findings-acl.207.pdf)
- [DOI 10.18653/v1/2021.findings-acl.207](https://doi.org/10.18653/v1/2021.findings-acl.207)
- [OpenAlex](https://openalex.org/W3174226126)

**Authors (6)**

1. [Yuxuan Wang](https://openalex.org/A5101664515) [ORCID](https://orcid.org/0009-0005-3508-3736) · h 23 · 8012 citations · 151 works
2. [Wanxiang Che](https://openalex.org/A5019108029) [ORCID](https://orcid.org/0000-0002-3907-0335) · h 49 · 8874 citations · 359 works
3. [Ivan Titov](https://openalex.org/A5086717154) [ORCID](https://orcid.org/0000-0002-2583-1893) · h 43 · 14399 citations · 188 works
4. [Shay B. Cohen](https://openalex.org/A5030503109) [ORCID](https://orcid.org/0000-0003-4753-8353) · h 28 · 4697 citations · 242 works
5. [Zhilin Lei](https://openalex.org/A5048234392) h 1 · 6 citations · 4 works
6. [Ting Liu](https://openalex.org/A5100418120) [ORCID](https://orcid.org/0000-0001-8904-8796) · h 31 · 2956 citations · 119 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

The paper attacks two English dependency parsers, a biaffine graph parser and a stack-pointer parser, by substituting words. It asks what makes an adversarial example valid: the sentence must stay fluent and keep its gold tree. Earlier work proposed substitutes with BERT alone. This paper draws candidates from four generators (BERT, embedding neighbours, sememes, WordNet synonyms) and filters them by POS, embedding similarity, a grammar checker and GPT-2 perplexity. The resulting examples are more fluent: their average perplexity is 139.99, against 267.96 for the earlier method and 127.67 for the clean test set. In a human check of 100 sentences they keep the syntax 85% of the time, against 75% for the earlier method. Out-of-vocabulary and out-of-training words drive the attacks, and the examples transfer poorly across seeds, embeddings and parser families. Adversarial training on all the examples, and an ensemble of seeds, both make the parsers more robust.

## Ideas

- I1. Build a substitution set from `en/seeds/lexicon` synonyms of the same UPOS (the paper's POS filter), keep the gold trees, and report the LAS drop of `en parse-eval` on EWT dev.
- I2. Measure how many substituted words are unknown to the dictionaries of `en/src/ptag.rs` and `en/src/parse.rs`, and report LAS separately for sentences with many unknown words.
- I3. Adversarial training for the `en` perceptron: add the I1 sentences at 10%, 50% and 100% of the training set, and choose the amount by clean dev LAS.
- I4. In `gparse`, ensemble models trained with different seeds, and measure how often the I1 attack succeeds before and after.

## For Mova

`en` is not a neural model, so the substitution test shows how much the perceptron leans on its dictionary and its word features. The ideas touch [10. Word classes](https://md.mova.elamur.ai/free/en/docs/languages/en/10-word-classes/) (the POS filter) and `19-lexicon-mwe-register.md` (the lexicon). Measure I1 and I2 on dev first.

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2021.findings-acl.207](https://doi.org/10.18653/v1/2021.findings-acl.207)

```
@inproceedings{wang2021closer,
  title = {{A Closer Look into the Robustness of Neural Dependency Parsers Using Better Adversarial Examples}},
  author = {Wang, Yuxuan and Che, Wanxiang and Titov, Ivan and Cohen, Shay B. and Lei, Zhilin and Liu, Ting},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.207},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.207}
}
```

- [← previous Using Lexical Dependency and Ontological Knowledge to Improve a Detailed Syntactic and Semantic Tagger of English](https://md.mova.elamur.ai/free/en/docs/papers/finch-2006-lexical-dependency-ontological-knowledge-improve/)
- [next → Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis](https://md.mova.elamur.ai/free/en/docs/papers/findings-2022-kulick-penn-helsinki-parsed-corpus-early/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Moustafa+Alzantot%2C+Yash+Sharma%2C+Ahmed+Elgohary%2C+Bo%2DJhang+Ho%2C+Mani+Srivastava%2C+and+Kai%2DWei+Chang%2E+2018%2E+Generating+natural+language+adversarial+examples%2E+In+Proceedings+of+the+2018+Conference+on+Empiri)
2. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+ELECTRA%3A+pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+8th+International+Conference+on+Learning+R)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
4. Zhendong Dong and Qiang Dong. 2006. Hownet And the Computation of Meaning. World Scientific Publishing Co., Inc., USA. [↗](https://doi.org/10.1142/5935)
5. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net. [find](https://scholar.google.com/scholar?q=Timothy+Dozat+and+Christopher+D%2E+Manning%2E+2017%2E+Deep+biaffine+attention+for+neural+dependency+parsing%2E+In+5th+International+Conference+on+Learning+Representations%2C+ICLR+2017%2C+Toulon%2C+France%2C+April+24%2D) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
6. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. [↗](https://doi.org/10.48550/arxiv.1412.6572)
7. Wenjuan Han, Liwen Zhang, Yong Jiang, and Kewei Tu. 2020. Adversarial attack and defense of structured prediction models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2327–2338, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wenjuan+Han%2C+Liwen+Zhang%2C+Yong+Jiang%2C+and+Kewei+Tu%2E+2020%2E+Adversarial+attack+and+defense+of+structured+prediction+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language)
8. Zhifeng Hu, Serhii Havrylov, Ivan Titov, and Shay B. Cohen. 2020. Obfuscation for privacy-preserving syntactic parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 62–72, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.09585)
9. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8018–8025. AAAI Press. [find](https://scholar.google.com/scholar?q=Di+Jin%2C+Zhijing+Jin%2C+Joey+Tianyi+Zhou%2C+and+Peter+Szolovits%2E+2020%2E+Is+BERT+really+robust%3F+A+strong+baseline+for+natural+language+attack+on+text+classification+and+entailment%2E+In+The+Thirty%2DFourth+AAAI+)
10. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681–691, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Xinlei+Chen%2C+Eduard+Hovy%2C+and+Dan+Jurafsky%2E+2016%2E+Visualizing+and+understanding+neural+models+in+NLP%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+f)
11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
12. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403–1414, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma%2C+Zecong+Hu%2C+Jingzhou+Liu%2C+Nanyun+Peng%2C+Graham+Neubig%2C+and+Eduard+Hovy%2E+2018%2E+Stack%2Dpointer+networks+for+dependency+parsing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+C)
13. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Bill+MacCartney%2C+and+Christopher+D%2E+Manning%2E+2006%2E+Generating+typed+dependency+parses+from+phrase+structure+parses%2E+In+Proceedings+of+the+Fifth+International+Conference+on) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2006-marneffe-generating-typed-dependency-parses-phrase/)
14. John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020. Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3829–3839, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Morris%2C+Eli+Lifland%2C+Jack+Lanchantin%2C+Yangfeng+Ji%2C+and+Yanjun+Qi%2E+2020%2E+Reevaluating+adversarial+examples+in+natural+language%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+)
15. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 142–148, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikola+Mrks%CC%8Cic%CC%81%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Blaise+Thomson%2C+Milica+Gas%CC%8Cic%CC%81%2C+Lina+M%2E+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2016%2E+Counter%2Dfitting+word+vectors+to+lingu)
16. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. [↗](https://doi.org/10.3115/v1/d14-1162)
17. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
18. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
19. Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085–1097, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shuhuai+Ren%2C+Yihe+Deng%2C+Kun+He%2C+and+Wanxiang+Che%2E+2019%2E+Generating+natural+language+adversarial+examples+through+probability+weighted+word+saliency%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+As)
20. Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 606–615, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yequan+Wang%2C+Minlie+Huang%2C+Xiaoyan+Zhu%2C+and+Li+Zhao%2E+2016%2E+Attention%2Dbased+LSTM+for+aspect%2Dlevel+sentiment+classification%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Languag)
21. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066–6080, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuan+Zang%2C+Fanchao+Qi%2C+Chenghao+Yang%2C+Zhiyuan+Liu%2C+Meng+Zhang%2C+Qun+Liu%2C+and+Maosong+Sun%2E+2020%2E+Word%2Dlevel+textual+adversarial+attacking+as+combinatorial+optimization%2E+In+Proceedings+of+the+58th+Annual)
22. Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5564–5569, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Huangzhao+Zhang%2C+Hao+Zhou%2C+Ning+Miao%2C+and+Lei+Li%2E+2019%2E+Generating+fluent+adversarial+examples+for+natural+languages%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lin)
23. Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Trans. Intell. Syst. Technol., 11(3). [↗](https://doi.org/10.48550/arxiv.1901.06796)
24. Xiaoqing Zheng, Jiehang Zeng, Yi Zhou, Cho-Jui Hsieh, Minhao Cheng, and Xuanjing Huang. 2020. Evaluating and enhancing the robustness of neural network-based dependency parsing models with adversarial examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6600–6610, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaoqing+Zheng%2C+Jiehang+Zeng%2C+Yi+Zhou%2C+Cho%2DJui+Hsieh%2C+Minhao+Cheng%2C+and+Xuanjing+Huang%2E+2020%2E+Evaluating+and+enhancing+the+robustness+of+neural+network%2Dbased+dependency+parsing+models+with+adversarial) [our card](https://md.mova.elamur.ai/free/en/docs/papers/zheng-2020-evaluating-enhancing-robustness-neural-network/)