[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/langstream-2025-ben-atya-agent-based-uncertainty-awareness-improves/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model # Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model **Authors:** [Hadas Ben-Atya](https://md.mova.elamur.ai/free/en/docs/papers/authors/hadas-ben-atya/); [Naama Gavrielov](https://md.mova.elamur.ai/free/en/docs/papers/authors/naama-gavrielov/); [Zvi Badash](https://md.mova.elamur.ai/free/en/docs/papers/authors/zvi-badash/); [Gili Focht](https://md.mova.elamur.ai/free/en/docs/papers/authors/gili-focht/); [Ruth Cytter-Kuint](https://md.mova.elamur.ai/free/en/docs/papers/authors/ruth-cytter-kuint/); [Talar Hagopian](https://md.mova.elamur.ai/free/en/docs/papers/authors/talar-hagopian/); [Dan Turner](https://md.mova.elamur.ai/free/en/docs/papers/authors/dan-turner/); [Moti Freiman](https://md.mova.elamur.ai/free/en/docs/papers/authors/moti-freiman/) · **Year:** 2025 · **Venue:** arXiv:2502.01691 **Link:** [arxiv.org/abs/2502.01691](https://arxiv.org/abs/2502.01691) · library `langstream-2025-ben-atya-agent-based-uncertainty-awareness-improves` · **Licence:** [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) (arXiv) — summary only [**1** citations](https://openalex.org/works?filter=cites:W4407184669) openalex · 2026-10-18[**2** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/langstream-2025-ben-atya-agent-based-uncertainty-awareness-improves/) of the 140673 papers whose bibliographies we read[arXiv:2502.01691](https://openalex.org/S4306400194)[licence](http://creativecommons.org/licenses/by-nc-nd/4.0/) - [Publisher's page](https://arxiv.org/abs/2502.01691) - [PDF](https://arxiv.org/pdf/2502.01691) - [arXiv 2502.01691](https://arxiv.org/abs/2502.01691) - [OpenAlex](https://openalex.org/W4407184669) **Authors (8)** 1. Hadas Ben-Atya 2. [Naama Gavrielov](https://openalex.org/A5093775918) [Semantic Scholar](https://www.semanticscholar.org/author/2281247851) · h 1 · 4 citations · 13 works 3. [Zvi Badash](https://openalex.org/A5115983654) [Semantic Scholar](https://www.semanticscholar.org/author/2341785458) · h 1 · 1 citations · 9 works 4. [Gili Focht](https://openalex.org/A5085773612) [Semantic Scholar](https://www.semanticscholar.org/author/35674267) · h 19 · 1349 citations · 131 works 5. Ruth Cytter-Kuint 6. [Talar Hagopian](https://openalex.org/A5097111482) [Semantic Scholar](https://www.semanticscholar.org/author/2341785595) · h 1 · 3 citations · 13 works 7. [Dan Turner](https://openalex.org/A5084520591) [ORCID](https://orcid.org/0000-0001-5160-868X) · [Semantic Scholar](https://www.semanticscholar.org/author/2366699632) · h 80 · 40069 citations · 701 works 8. [Moti Freiman](https://openalex.org/A5089641146) [ORCID](https://orcid.org/0000-0003-1083-1548) · [Semantic Scholar](https://www.semanticscholar.org/author/2281247283) · h 22 · 1415 citations · 161 works Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart. ## What it is about The paper asks an open-source LLM (Llama 3-8B-Instruct) whether a Hebrew Crohn's disease radiology report shows a given organ–finding pair, and asks for a JSON answer with an explanation. Each question is put in six paraphrased prompts (Bayesian Prompt Ensembles). Three entropy-based aggregators are compared with an "agent" that reads the six answers and explanations and assigns one of five confidence levels (definitely or likely yes or no, uncertain). The labels are automatic: the 9,683 reports were labelled by the paper's HSMP-BERT model, and only eight labels with Cohen's kappa above 0.7 were kept. The agent reaches F1 0.3967 and kappa 0.3006 on the held-out set. When items with uncertainty of 0.5 or more are dropped, F1 rises to 0.4787 and kappa to 0.4258. Hebrew is not among the languages Llama 3.1 officially supports. ## Ideas - I1. Use paraphrase agreement as a filter for teacher labels. When a large model is only the teacher, ask it the same question in several wordings and keep only the labels on which the answers agree; send the rest to a review list. This protects the training data, and the test sets stay separate. - I2. Let an answer be "unknown". The paper's clearest gain comes from abstaining on the uncertain items. In `prag`, an answer that cannot be decided should say so with its reason, and a test should show that precision on the answered items rises when the unknowns are excluded. - I3. Keep a short reason with every decision, together with a level. The agent's JSON format pairs each decision with an explanation. Mova already works this way: every step should say why, and the level plus the reason should be stored with each answer. ## For Mova I1 is a rule for the teacher-label step of the training pipeline, which uses the big model only as a teacher. I2 is a behaviour for `prag` answers and a test for it. I3 is a format note for `prag` output. ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2502.01691](https://arxiv.org/abs/2502.01691) ``` @misc{benatya2025agent, title = {{Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model}}, author = {Ben-Atya, Hadas and Gavrielov, Naama and Badash, Zvi and Focht, Gili and Cytter-Kuint, Ruth and Hagopian, Talar and Turner, Dan and Freiman, Moti}, year = {2025}, eprint = {2502.01691}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/2502.01691} } ``` - [← previous Hierarchical Section Matching Prediction (HSMP) BERT for Fine-Grained Extraction of Structured Data from Hebrew Free-Text Radiology Reports in Crohn's Disease](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2025-badash-hierarchical-section-matching-prediction-hsmp/) - [next → Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2025-hashiloni-building-patient-journeys-hebrew-language/) - [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) ## References 26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Nobel JM, van Geel K, Robben SG. Structured reporting in radiology: a systematic review to explore its potential. European Radiology. 2022:1-18. [find](https://scholar.google.com/scholar?q=Nobel+JM%2C+van+Geel+K%2C+Robben+SG%2E+Structured+reporting+in+radiology%3A+a+systematic+review+to+explore+its+potential%2E+European+Radiology%2E+2022%3A1%2D18%2E) 2. Hassanpour S, Langlotz CP. Information extraction from multi-institutional radiology reports. Artificial Intelligence in Medicine. 2016;66:29-39. [find](https://scholar.google.com/scholar?q=Hassanpour+S%2C+Langlotz+CP%2E+Information+extraction+from+multi%2Dinstitutional+radiology+reports%2E+Artificial+Intelligence+in+Medicine%2E+2016%3B66%3A29%2D39%2E) 3. Haendel MA, Chute CG, Robinson PN. Classification, ontology, and precision medicine. New England Journal of Medicine. 2018;379(15):1452-62. [find](https://scholar.google.com/scholar?q=Haendel+MA%2C+Chute+CG%2C+Robinson+PN%2E+Classification%2C+ontology%2C+and+precision+medicine%2E+New+England+Journal+of+Medicine%2E+2018%3B379%2815%29%3A1452%2D62%2E) 4. Langlotz CP, Allen B, Erickson BJ, Kalpathy-Cramer J, Bigelow K, Cook TS, et al. A roadmap for foundational research on artificial intelligence in medical imaging: from the 2018 NIH/RSNA/ACR/The Academy Workshop. Radiology. 2019;291(3):781-91. [find](https://scholar.google.com/scholar?q=Langlotz+CP%2C+Allen+B%2C+Erickson+BJ%2C+Kalpathy%2DCramer+J%2C+Bigelow+K%2C+Cook+TS%2C+et+al%2E+A+roadmap+for+foundational+research+on+artificial+intelligence+in+medical+imaging%3A+from+the+2018+NIH%2FRSNA%2FACR%2FThe+Acade) 5. Busch F, Hoffmann L, Dos Santos DP, Makowski MR, Saba L, Prucker P, et al. Large language models for structured reporting in radiology: past, present, and future. European Radiology. 2024:1-14. [find](https://scholar.google.com/scholar?q=Busch+F%2C+Hoffmann+L%2C+Dos+Santos+DP%2C+Makowski+MR%2C+Saba+L%2C+Prucker+P%2C+et+al%2E+Large+language+models+for+structured+reporting+in+radiology%3A+past%2C+present%2C+and+future%2E+European+Radiology%2E+2024%3A1%2D14%2E) 6. Doshi R, Amin KS, Khosla P, Bajaj SS, Chheang S, Forman HP. Quantitative evaluation of large language models to streamline radiology report impressions: A multimodal retrospective analysis. Radiology. 2024;310(3):e231593. [find](https://scholar.google.com/scholar?q=Doshi+R%2C+Amin+KS%2C+Khosla+P%2C+Bajaj+SS%2C+Chheang+S%2C+Forman+HP%2E+Quantitative+evaluation+of+large+language+models+to+streamline+radiology+report+impressions%3A+A+multimodal+retrospective+analysis%2E+Radiology%2E) 7. Reichenpfader D, Müller H, Denecke K. A scoping review of large language model-based approaches for information extraction from radiology reports. NPJ Digital Medicine. 2024;7(1):222. [find](https://scholar.google.com/scholar?q=Reichenpfader+D%2C+M%C3%BCller+H%2C+Denecke+K%2E+A+scoping+review+of+large+language+model%2Dbased+approaches+for+information+extraction+from+radiology+reports%2E+NPJ+Digital+Medicine%2E+2024%3B7%281%29%3A222%2E) 8. Nowak S, Wulff B, Layer YC, Theis M, Isaak A, Salam B, et al. Privacy-ensuring open-weights large language models are competitive with closed-weights GPT-4o in extracting chest radiography 2025;314(1):e240895. findings from free-text reports. Radiology. [find](https://scholar.google.com/scholar?q=Nowak+S%2C+Wulff+B%2C+Layer+YC%2C+Theis+M%2C+Isaak+A%2C+Salam+B%2C+et+al%2E+Privacy%2Densuring+open%2Dweights+large+language+models+are+competitive+with+closed%2Dweights+GPT%2D4o+in+extracting+chest+radiography+2025%3B314%281%29) 9. Shmidman S, Shmidman A, Cohen AD, Koppel M. Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities. arXiv preprint arXiv:240707080. 2024. [find](https://scholar.google.com/scholar?q=Shmidman+S%2C+Shmidman+A%2C+Cohen+AD%2C+Koppel+M%2E+Adapting+LLMs+to+Hebrew%3A+Unveiling+DictaLM+2%2E0+with+Enhanced+Vocabulary+and+Instruction+Capabilities%2E+arXiv+preprint+arXiv%3A240707080%2E+2024%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2024-shmidman-adapting-llms-hebrew-unveiling-dictalm/) 10. Rivera M, Godbout JF, Rabbany R, Pelrine K. Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation. arXiv preprint arXiv:2401.08694. 2024. [↗](https://arxiv.org/abs/2401.08694) 11. Xiong M, Hu Z, Lu X, Li Y, Fu J, He J, et al. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv preprint arXiv:2306.13063. 2024. [↗](https://arxiv.org/abs/2306.13063) 12. Zhou K, Jurafsky D, Hashimoto T. Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence arXiv:2302.13439. 2023. [↗](https://arxiv.org/abs/2302.13439) 13. Tonolini F, Massiah J, Aletras N, Kazai G. Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models. Amazon Science. 2024. [find](https://scholar.google.com/scholar?q=Tonolini+F%2C+Massiah+J%2C+Aletras+N%2C+Kazai+G%2E+Bayesian+prompt+ensembles%3A+Model+uncertainty+estimation+for+black%2Dbox+large+language+models%2E+Amazon+Science%2E+2024%2E) 14. Xi Z, Chen W, Guo X, He W, Ding Y, Hong B, et al. The rise and potential of large language model-based agents: A survey. arXiv preprint arXiv:230907864. 2023. [find](https://scholar.google.com/scholar?q=Xi+Z%2C+Chen+W%2C+Guo+X%2C+He+W%2C+Ding+Y%2C+Hong+B%2C+et+al%2E+The+rise+and+potential+of+large+language+model%2Dbased+agents%3A+A+survey%2E+arXiv+preprint+arXiv%3A230907864%2E+2023%2E) 15. Friedman MY, Leventer-Roberts M, Rosenblum J, Zigman N, Goren I, Mourad V, et al. Development and validation of novel algorithms to identify patients with inflammatory bowel diseases in Israel: an epi-IIRN group study. Clinical Epidemiology. 2018:671-81. [find](https://scholar.google.com/scholar?q=Friedman+MY%2C+Leventer%2DRoberts+M%2C+Rosenblum+J%2C+Zigman+N%2C+Goren+I%2C+Mourad+V%2C+et+al%2E+Development+and+validation+of+novel+algorithms+to+identify+patients+with+inflammatory+bowel+diseases+in+Israel%3A+an+epi) 16. Bruining DH, Zimmermann EM, Loftus Jr EV, Sandborn WJ, Sauer CG, Strong SA, et al. Consensus recommendations for evaluation, interpretation, and utilization of computed tomography and magnetic resonance enterography in patients with small bowel Crohn’s disease. Radiology. 2018;286(3):776-99. Affect Language Models. arXiv preprint [find](https://scholar.google.com/scholar?q=Bruining+DH%2C+Zimmermann+EM%2C+Loftus+Jr+EV%2C+Sandborn+WJ%2C+Sauer+CG%2C+Strong+SA%2C+et+al%2E+Consensus+recommendations+for+evaluation%2C+interpretation%2C+and+utilization+of+computed+tomography+and+magnetic+resonan) 17. Hazan L, Focht G, Gavrielov N, Reichart R, Friss C, Cytter Kuint R, et al. P269 Harnessing Natural Language Processing for Structured Information Extraction from Radiology Reports in Crohn’s Disease: A Nationwide Study From the epi-IIRN. Journal of Crohn’s and Colitis. 2024;18(Supplement 1):i633-4. [find](https://scholar.google.com/scholar?q=Hazan+L%2C+Focht+G%2C+Gavrielov+N%2C+Reichart+R%2C+Friss+C%2C+Cytter+Kuint+R%2C+et+al%2E+P269+Harnessing+Natural+Language+Processing+for+Structured+Information+Extraction+from+Radiology+Reports+in+Crohn%E2%80%99s+Disease%3A+) 18. Hazan L, Focht G, Gavrielov N, Reichart R, Hagopian T, Greer MLC, et al. Leveraging Prompt-Learning for Structured Information Extraction from Crohn’s Disease Radiology Reports in a Low-Resource Language. arXiv preprint arXiv:240501682. 2024. [find](https://scholar.google.com/scholar?q=Hazan+L%2C+Focht+G%2C+Gavrielov+N%2C+Reichart+R%2C+Hagopian+T%2C+Greer+MLC%2C+et+al%2E+Leveraging+Prompt%2DLearning+for+Structured+Information+Extraction+from+Crohn%E2%80%99s+Disease+Radiology+Reports+in+a+Low%2DResource+Langu) 19. AI@Meta. Llama 3 Model Card; 2024. Available from: [https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md). [↗](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md) 20. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention Is All You Need. arXiv preprint arXiv:1706.03762. 2017. [↗](https://arxiv.org/abs/1706.03762) 21. Sun H. Supervised Fine-Tuning as Inverse Reinforcement Learning. arXiv preprint arXiv:2403.12017. 2024. [↗](https://arxiv.org/abs/2403.12017) 22. Christiano PF, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems. 2017. [find](https://scholar.google.com/scholar?q=Christiano+PF%2C+Leike+J%2C+Brown+T%2C+Martic+M%2C+Legg+S%2C+Amodei+D%2E+Deep+Reinforcement+Learning+from+Human+Preferences%2E+Advances+in+Neural+Information+Processing+Systems%2E+2017%2E) 23. Stiennon N, Ouyang L, Wu J, Ziegler DM, Lowe R, Voss C, et al. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325. 2022. [↗](https://arxiv.org/abs/2009.01325) 24. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022. [↗](https://arxiv.org/abs/2203.02155) 25. OpenAI. ChatGPT (GPT-4); 2024. Available from: [https://chat.openai.com](https://chat.openai.com). [↗](https://chat.openai.com) 26. Zeng F, Lyu Z, Li Q, Li X. Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System. arXiv preprint arXiv:241206828. 2024. Figure captions: Fig. 1: (a) Distribution of organ-specific findings and (b) filtered organ-finding combinations with more than 15 positive cases. Fig. 2: Illustration of the proposed Bayesian Prompt Ensemble pipeline for uncertainty-aware predictions. A radiology report and multiple semantically equivalent prompts are fed into an LLM, generating one prediction per prompt. These predictions are then aggregated to yield a final decision and uncertainty estimation. The aggregation function can be either an LLM agent or an entropy-based function that applies uniform weights, learned fixed weights, or weights predicted by a pre-trained MLP model. Fig. 3: Uncertainty Histograms for Ileum stenosis Computed by: (a) Uniform Weights, (b) Linear-Optimized Weights, (c) Learnable- MLP Weights, and (d) Agent-based decision. Fig. 4: Uncertainty Histograms for Ileum inflammation Computed by: (a) Uniform Weights, (b) Linear-Optimized Weights, (c) Learnable- MLP Weights, and (d) Agent-based decision. Table 1: Comparison of average results across all labels for different methods before applying the uncertainty threshold. Model Accuracy Baseline 0.8243 Uniform 0.832 Linear 0.8454 MLP 0.8605 Agent 0.8022 F1 0.2699 0.3873 0.3847 0.3643 0.3967 Precision 0.3046 0.336 0.3463 0.3772 0.3131 Recall 0.3097 0.5336 0.4878 0.3977 0.6437 Kappa 0.179 0.2979 0.2988 0.2863 0.3006 Table 2: Comparison of average of the median uncertainty values, across all 23 labels, for correct and incorrect predictions across different methods. Median Uncertainty Correct Predictions Incorrect Predictions 0.4081 0.6153 0.2668 0.4694 0.2261 0.5717 0 0.5108 Method Uniform Linear MLP Agent Table 3: Comparison of average results across all labels for different methods after applying uncertainty threshold ≥0.5. Model Baseline Uniform Linear MLP Agent Accuracy 0.8243 0.9188 0.897 0.9242 0.8947 F1 0.2699 0.4634 0.4261 0.4032 0.4787 Precision 0.3046 0.4458 0.4008 0.4584 0.3976 Recall 0.3097 0.5327 0.508 0.4079 0.6614 Kappa 0.179 0.4203 0.3702 0.3643 0.4258 Excluded Cases 0 200 (43.31%) 96 (20.97%) 120 (26.07%) 153 (33.16%) Table 4: Comparison of average results across all labels for different methods after applying uncertainty threshold up to ≥ 0.5, with case exclusions limited to a maximum of 20%. Model Baseline Uniform Linear MLP Agent Accuracy 0.8243 0.8789 0.8905 0.9087 0.8285 F1 0.2699 0.4267 0.212 0.3933 0.4494 Precision 0.3046 0.3873 0.3938 0.4492 0.3529 Recall 0.3097 0.5304 0.5079 0.4004 0.7295 Kappa 0.179 0.3615 0.3613 0.345 0.3683 Excluded Cases 0 92 (19.91%) 80 (17.5%) 90 (19.5%) 92 (19.91%) [find](https://scholar.google.com/scholar?q=Zeng+F%2C+Lyu+Z%2C+Li+Q%2C+Li+X%2E+Enhancing+LLMs+for+Impression+Generation+in+Radiology+Reports+through+a+Multi%2DAgent+System%2E+arXiv+preprint+arXiv%3A241206828%2E+2024%2E+Figure+captions%3A+Fig%2E+1%3A+%28a%29+Distribution+o)