A Review of the Opportunities and Challenges with Large Language Models in Radiology: The Road Ahead

References

1. 1.↵
   1. Mikolov T,
   2. Sutskever I,
   3. Chen K, et al

   . Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 2013;26:3111–3119
2. 2.↵
   1. Vaswani A,
   2. Shazeer N,
   3. Parmar N, et al

   . Attention is all you need. Adv Neural Inf Process Syst 2017;30:6000–10
3. 3.↵
   1. Devlin J,
   2. Chang M-W,
   3. Lee K, et al

   . BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2019:4171–86
4. 4.↵
   1. Lewis M,
   2. Liu Y,
   3. Goyal N, et al

   . BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2020:7871–80
5. 5.↵
   1. Brown T,
   2. Mann B,
   3. Ryder N, et al

   . Language models are few-shot learners. Adv Neural Inf Process Syst 2020;33:1877–1901
6. 6.↵
   1. Chowdhery A,
   2. Narang S,
   3. Devlin J, et al

   . Palm: Scaling language modeling with pathways. J Mach Learn Res 2023;24:11324–43
7. 7.↵
   1. Zhao WX,
   2. Zhou K,
   3. Li J, et al

   . A survey of large language models. arXiv preprint arXiv:230318223 2023
8. 8.↵
   1. Ghosh A,
   2. Acharya A,
   3. Saha S, et al

   . Exploring the frontier of vision-language models: a survey of current methodologies and future directions. arXiv preprint arXiv:240407214 2024
9. 9.↵
   1. Akinci D’Antonoli T,
   2. Stanzione A,
   3. Bluethgen C, et al

   . Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions. Diagn Interv Radiol 2024;30:80–90 doi:10.4274/dir.2023.232417 pmid:37789676

   CrossRefPubMed
10. 10.↵
    1. Bhayana R,
    2. Krishna S,
    3. Bleakney RR

    . Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology 2023;307:e230582 doi:10.1148/radiol.230582 pmid:37191485

    CrossRefPubMed
11. 11.↵
    1. Liu Z,
    2. Zhong A,
    3. Li Y

    . Radiology-GPT: a large language model for radiology. Meta-Radiology Epub ahead of print May 16, 2025
12. 12.↵
    1. Gallifant J,
    2. Fiske A,
    3. Levites Strekalova YA, et al

    . Peer review of GPT-4 technical report and systems card. PLOS Digital Health 2024;3:e0000417 doi:10.1371/journal.pdig.0000417 pmid:38236824

    CrossRefPubMed
13. 13.↵
    1. Anil R,
    2. Dai AM,
    3. Firat O, et al

    . Palm 2 technical report. arXiv preprint arXiv:230510403 2023
14. 14.↵
    1. Touvron H,
    2. Lavril T,
    3. Izacard G, et al

    . LLaMA: open and efficient foundation language models. arXiv preprint arXiv:230213971 2023
15. 15.↵
    1. Joshi G,
    2. Jain A,
    3. Araveeti SR, et al

    . FDA-approved artificial intelligence and machine learning (AI/ML)-enabled medical devices: an updated landscape. Electronics 2024;13:498 doi:10.3390/electronics13030498

    CrossRef
16. 16.↵
    1. Banerjee S,
    2. Agarwal A,
    3. Singla S

    . LLMs will always hallucinate, and we need to live with this. arXiv preprint arXiv:240905746 2024
17. 17.↵
    1. Wiggins WF,
    2. Tejani AS

    . On the opportunities and risks of foundation models for natural language processing in radiology. Radiol Artif Intell 2022;4:e220119 doi:10.1148/ryai.220119 pmid:35923379

    CrossRefPubMed
18. 18.↵
    1. Paaß G,
    2. Giesselbach S

    . Foundation Models for Natural Language Processing: Pre-Trained Language Models Integrating Media. Springer-Verlag Nature; 2023
19. 19.↵
    1. Denecke K,
    2. May R,
    3. Rivera Romero O

    . Potential of large language models in health care: Delphi study. J Med Internet Res 2024;26:e52399 doi:10.2196/52399 pmid:38739445

    CrossRefPubMed
20. 20.↵
    1. Yang X,
    2. Chen A,
    3. PourNejatian N, et al

    . Gatortron: a large clinical language model to unlock patient information from unstructured electronic health records. arXiv preprint arXiv:220303540 2022
21. 21.↵
    1. Bordes F,
    2. Pang RY,
    3. Ajay A, et al

    . An introduction to vision-language modeling. arXiv preprint arXiv:240517247 2024
22. 22.↵
    OpenVLM Leaderboard. 2024
23. 23.↵
    1. Radford A,
    2. Kim JW,
    3. Hallacy C

    , et al. Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning. Proc Mach Learn Res 2021;139:8748–63
24. 24.↵
    1. Rombach R,
    2. Blattmann A,
    3. Lorenz D, et al

    . High-resolution image synthesis with latent diffusion models. Paper presented at: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); June 21–24, 2022; New Orleans, LA
25. 25.↵
    1. Berger E

    . Grounding LLMs. Eleanor on Everything. Accessed January 6, 2025. https://everything.intellectronica.net/p/grounding-llms
26. 26.↵
    1. Mukherjee P,
    2. Hou B,
    3. Lanfredi RB, et al

    . Feasibility of using the privacy-preserving large language model Vicuna for labeling radiology reports. Radiology 2023;309:e231147 doi:10.1148/radiol.231147 pmid:37815442

    CrossRefPubMed
27. 27.↵
    1. Qiu Z-H,
    2. Guo S,
    3. Xu M, et al

    . To cool or not to cool? Temperature network meets large foundation models via DRO. arXiv preprint arXiv:240404575 2024
28. 28.↵
    1. López-Úbeda P,
    2. Martín-Noguerol T,
    3. Juluru K, et al

    . Natural language processing in radiology: update on clinical applications. J Am Coll Radiol 2022;19:1271–85 doi:10.1016/j.jacr.2022.06.016 pmid:36029890

    CrossRefPubMed
29. 29.↵
    1. Gertz RJ,
    2. Bunck AC,
    3. Lennartz S, et al

    . GPT-4 for automated determination of radiological study and protocol based on radiology request forms: a feasibility study. Radiology 2023;307:e230877 doi:10.1148/radiol.230877 pmid:37310247

    CrossRefPubMed
30. 30.↵
    1. Nazario-Johnson L,
    2. Zaki HA,
    3. Tung GA

    . Use of large language models to predict neuroimaging. J Am Coll Radiol 2023;20:1004–09 doi:10.1016/j.jacr.2023.06.008 pmid:37423349

    CrossRefPubMed
31. 31.↵
    1. Kuckelman IJ,
    2. Paul HY,
    3. Bui M, et al

    . Assessing AI-powered patient education: a case study in radiology. Acad Radiol 2024;31:338–42 doi:10.1016/j.acra.2023.08.020 pmid:37709612

    CrossRefPubMed
32. 32.↵
    1. Laukamp KR,
    2. Terzis RA,
    3. Werner JM, et al

    . Monitoring patients with glioblastoma by using a large language model: accurate summarization of radiology reports with GPT-4. Radiology 2024;312:e232640 doi:10.1148/radiol.232640 pmid:39041936

    CrossRefPubMed
33. 33.↵
    1. Antonelli M,
    2. Reinke A,
    3. Bakas S, et al

    . The medical segmentation decathlon. Nat Commun 2022;13:4128 doi:10.1038/s41467-022-30695-9 pmid:35840566

    CrossRefPubMed
34. 34.↵
    1. Kirillov A,
    2. Mintun E,
    3. Ravi N, et al

    . Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision; October 1–6, 2023; Paris, France: 4015–26
35. 35.↵
    1. Huang Y,
    2. Yang X,
    3. Liu L, et al

    . Segment anything model for medical images? Med Image Anal 2024;92:103061 doi:10.1016/j.media.2023.103061 pmid:38086235

    CrossRefPubMed
36. 36.↵
    1. Liu Y,
    2. Zhang J,
    3. She Z, et al

    . SAMM (segment any medical model): a 3D slicer integration to SAM. arXiv preprint arXiv:230405622 2023
37. 37.↵
    1. Ma J,
    2. He Y,
    3. Li F, et al

    . Segment anything in medical images. Nat Comm 2024;15:654 doi:10.1038/s41467-024-44824-z pmid:38253604

    CrossRefPubMed
38. 38.↵
    1. Ma J,
    2. Wang B

    . Towards foundation models of biological image segmentation. Nat Methods 2023;20:953–55 doi:10.1038/s41592-023-01885-0 pmid:37433999

    CrossRefPubMed
39. 39.↵
    1. Ravi N,
    2. Gabeur V,
    3. Hu Y-T, et al

    . SAM 2: segment anything in images and videos. arXiv preprint arXiv:240800714 2024
40. 40.↵
    1. Kiraly A,
    2. Traverse M

    ; Google research. Taking medical imaging embeddings 3D. October 21, 2024. Accessed February 28, 2025. https://research.google/blog/taking-medical-imaging-embeddings-3d/
41. 41.↵
    1. Sonoda Y,
    2. Kurokawa R,
    3. Nakamura Y, et al

    . Diagnostic performances of GPT-40, Claude 3 Opus, and Gemini 1.5 Pro in “diagnosis please” cases. Jpn J Radiol 2024;42:1231–35 doi:10.1007/s11604-024-01619-y pmid:38954192

    CrossRefPubMed
42. 42.↵
    1. Wada A,
    2. Akashi T,
    3. Shih G, et al

    . Optimizing GPT-4 turbo diagnostic accuracy in neuroradiology through prompt engineering and confidence thresholds. Diagnostics (Basel) 2024;14:1541 doi:10.3390/diagnostics14141541 pmid:39061677

    CrossRefPubMed
43. 43.↵
    1. Suh PS,
    2. Shim WH,
    3. Suh CH, et al

    . Comparing diagnostic accuracy of radiologists versus GPT-4V and Gemini Pro Vision using image inputs from diagnosis please cases. Radiology 2024;312:e240273 doi:10.1148/radiol.240273 pmid:38980179

    CrossRefPubMed
44. 44.↵
    1. Abdullahi T,
    2. Singh R,
    3. Eickhoff C

    . Learning to make rare and complex diagnoses with generative AI assistance: qualitative study of popular large language models. JMIR Med Educ 2024;10:e51391 doi:10.2196/51391 pmid:38349725

    CrossRefPubMed
45. 45.↵
    1. Liu C,
    2. Wan Z,
    3. Wang Y, et al

    . Benchmarking and boosting radiology report generation for 3D high-resolution medical images. arXiv preprint arXiv:240607146 2024
46. 46.↵
    1. Wang Z,
    2. Liu L,
    3. Wang L, et al

    . R2GenGPT: radiology report generation with frozen LLMs. Meta-Radiology 2023;1:100033 doi:10.1016/j.metrad.2023.100033

    CrossRef
47. 47.↵
    1. Pinto Dos Santos D,
    2. Cuocolo R,
    3. Huisman M

    . O structured reporting, where art thou? Eur Radiol 2024;34:4193–94 doi:10.1007/s00330-023-10465-x pmid:38010379

    CrossRefPubMed
48. 48.↵
    1. Hartung MP,
    2. Bickle IC,
    3. Gaillard F, et al

    . How to create a great radiology report. Radiographics 2020;40:1658–70 doi:10.1148/rg.2020200020 pmid:33001790

    CrossRefPubMed
49. 49.↵
    1. Adams LC,
    2. Truhn D,
    3. Busch F, et al

    . Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: a multilingual feasibility study. Radiology 2023;307:e230725 doi:10.1148/radiol.230725 pmid:37014240

    CrossRefPubMed
50. 50.↵
    1. Hasani AM,
    2. Singh S,
    3. Zahergivar A, et al

    . Evaluating the performance of Generative Pre-trained Transformer-4 (GPT-4) in standardizing radiology reports. Eur Radiol 2024;34:3566–74 doi:10.1007/s00330-023-10384-x pmid:37938381

    CrossRefPubMed
51. 51.↵
    1. Gertz RJ,
    2. Dratsch T,
    3. Bunck AC, et al

    . Potential of GPT-4 for detecting errors in radiology reports: implications for reporting accuracy. Radiology 2024;311:e232714 doi:10.1148/radiol.232714 pmid:38625012

    CrossRefPubMed
52. 52.↵
    1. Matsuo H,
    2. Nishio M,
    3. Matsunaga T, et al

    . Exploring multilingual large language models for enhanced TNM classification of radiology report in lung cancer staging. arXiv preprint arXiv:240606591 2024
53. 53.↵
    1. Yan A,
    2. McAuley J,
    3. Lu X, et al

    . RadBERT: adapting transformer-based language models to radiology. Radiol Artif Intell 2022;4:e210258

    CrossRefPubMed
54. 54.↵
    1. Doshi R,
    2. Amin KS,
    3. Khosla P, et al

    . Quantitative evaluation of large language models to streamline radiology report impressions: a multimodal retrospective analysis. Radiology 2024;310:e231593 doi:10.1148/radiol.231593 pmid:38530171

    CrossRefPubMed
55. 55.↵
    1. Park J,
    2. Oh K,
    3. Han K, et al

    . Patient-centered radiology reports with generative artificial intelligence: adding value to radiology reporting. Sci Rep 2024;14:13218 doi:10.1038/s41598-024-63824-z pmid:38851825

    CrossRefPubMed
56. 56.↵
    1. Berigan K,
    2. Short R,
    3. Reisman D, et al

    . The impact of large language model-generated radiology report summaries on patient comprehension: a randomized controlled trial. J Am Coll Radiol 2024;21:1898–903 doi:10.1016/j.jacr.2024.06.018 pmid:38964446

    CrossRefPubMed
57. 57.↵
    1. Li C,
    2. Wong C,
    3. Zhang S, et al

    . LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. In: Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23). Curran Associates Inc; 2023;1240:28541–64
58. 58.↵
    1. Moor M,
    2. Huang Q,
    3. Wu S, et al

    . Med-flamingo: a multimodal medical few-shot learner. Proceedings of the 3rd Machine Learning for Health (ML4H). Proc Mach Learn Res 2023;225:353–67
59. 59.↵
    1. Haemmerli J,
    2. Sveikata L,
    3. Nouri A, et al

    . ChatGPT in glioma adjuvant therapy decision making: ready to assume the role of a doctor in the tumour board? BMJ Health Care Inform 2023;30:e100775 doi:10.1136/bmjhci-2023-100775 pmid:37399360

    Abstract/FREE Full Text
60. 60.↵
    1. Tippareddy C,
    2. Jiang S,
    3. Bera K, et al

    . Radiology reading room for the future: harnessing the power of large language models like ChatGPT. Curr Probl Diagn Radiol 2023 August 30. [Epub ahead of print] doi:10.1067/j.cpradiol.2023.08.018 pmid:37758604

    CrossRefPubMed
61. 61.↵
    1. Liang W,
    2. Zhang Y,
    3. Cao H, et al

    . Can large language models provide useful feedback on research papers? A large-scale empirical analysis. N Eng J Med AI 2024;1:AIoa2400196 doi:10.1056/AIoa2400196

    CrossRef
62. 62.↵
    1. Hwang SI,
    2. Lim JS,
    3. Lee RW, et al

    . Is ChatGPT a “fire of Prometheus” for non-native English-speaking researchers in academic writing? Korean J Radiol 2023;24:952–59 doi:10.3348/kjr.2023.0773 pmid:37793668

    CrossRefPubMed
63. 63.↵
    Scispace. Scispace. The Fastest Research Platform Ever
64. 64.↵
    1. Ullah E,
    2. Parwani A,
    3. Baig MM, et al

    . Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology–a recent scoping review. Diagn Pathol 2024;19:43–49 doi:10.1186/s13000-024-01464-7 pmid:38414074

    CrossRefPubMed
65. 65.↵
    1. Hu M,
    2. Qian J,
    3. Pan S, et al

    . Advancing medical imaging with language models: featuring a spotlight on ChatGPT. Phys Med Biol 2024;69:10TR01 doi:10.1088/1361-6560/ad387d

    CrossRef
66. 66.↵
    1. Mirzadeh I,
    2. Alizadeh K,
    3. Shahrokhi H, et al

    . GSM-symbolic: understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:241005229 2024
67. 67.↵
    1. Bhayana R

    . Chatbots and large language models in radiology: a practical primer for clinical and research applications. Radiology 2024;310:e232756 doi:10.1148/radiol.232756 pmid:38226883

    CrossRefPubMed
68. 68.↵
    1. Bender EM,
    2. Gebru T,
    3. McMillan-Major A, et al

    . On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency; 2021:610–23
69. 69.↵
    1. Obermeyer Z,
    2. Powers B,
    3. Vogeli C, et al

    . Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019;366:447–53 doi:10.1126/science.aax2342 pmid:31649194

    Abstract/FREE Full Text
70. 70.↵
    1. Ong JCL,
    2. Chang SY-H,
    3. William W, et al

    . Medical ethics of large language models in medicine. NEJM AI 2024;1:AIra2400038 doi:10.1056/AIra2400038

    CrossRef
71. 71.↵
    1. Li H,
    2. Moon JT,
    3. Purkayastha S, et al

    . Ethics of large language models in medicine and medical research. Lancet Digit Health 2023;5:e333–35 doi:10.1016/S2589-7500(23)00083-3 pmid:37120418

    CrossRefPubMed
72. 72.↵
    1. Mannarswamy S,
    2. Chidambaram S

    . Opening the NLP Blackbox–analysis and evaluation of NLP models: methods, challenges and opportunities. Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD); 2021:447–448
73. 73.↵
    1. Ajwani R,
    2. Javaji SR,
    3. Rudzicz F, et al

    . LLM-generated black-box explanations can be adversarially helpful. arXiv preprint arXiv:240506800 2024
74. 74.↵
    1. Eaton P,
    2. Ince B,
    3. Iskovs A

    . The limitations of LLMs, or why are we doing RAG? Accessed May 21, 2025. https://www.enterprisedb.com/blog/limitations-llm-or-why-are-we-doing-rag
75. 75.↵
    1. Qiu S,
    2. Liu Q,
    3. Zhou S, et al

    . Review of artificial intelligence adversarial attack and defense technologies. App Sci (Basel) 2019;9:909 doi:10.3390/app9050909

    CrossRef
76. 76.↵
    1. Zhang S,
    2. Metaxas D

    . On the challenges and perspectives of foundation models for medical image analysis. Med Image Anal 2024;91:102996 doi:10.1016/j.media.2023.102996 pmid:37857067

    CrossRefPubMed
77. 77.↵
    1. Chen L,
    2. Zaharia M,
    3. Zou J

    . How is ChatGPT’s behavior changing over time? arXiv preprint arXiv:230709009 2023
78. 78.↵
    1. Delile J,
    2. Mukherjee S,
    3. Van Pamel A, et al

    . Graph-based retriever captures the long tail of biomedical knowledge. arXiv preprint arXiv:240212352 2024
79. 79.↵
    1. Stokel-Walker C

    . ChatGPT listed as author on research papers: many scientists disapprove. Nature 2023;613:620–21 doi:10.1038/d41586-023-00107-z pmid:36653617

    CrossRefPubMed
80. 80.↵
    1. Moy L

    . Guidelines for use of large language models by authors, reviewers, and editors: considerations for imaging journals. Radiology 2023;309:e239024 doi:10.1148/radiol.239024 pmid:37815449

    CrossRefPubMed
81. 81.↵
    1. Warraich HJ,
    2. Tazbaz T,
    3. Califf RM

    . FDA perspective on the regulation of artificial intelligence in health care and biomedicine. J Am Med Assoc 2024;333:241–47 doi:10.1001/jama.2024.21451 pmid:39405330

    CrossRefPubMed
82. 82.↵
    1. Wang G,
    2. Zhang S,
    3. Huang X, et al

    . Editorial for special issue on explainable and generalizable deep learning methods for medical image computing. Med Image Anal 2022;84:102727 doi:10.1016/j.media.2022.102727 pmid:36525844

    CrossRefPubMed
83. 83.↵
    1. Kim W

    . Seeing the unseen: advancing generative AI research in radiology. Radiology 2024;311:e240935 doi:10.1148/radiol.240935 pmid:38771182

    CrossRefPubMed
84. 84.↵
    1. Marks M,
    2. Haupt CE

    . AI chatbots, health privacy, and challenges to HIPAA compliance. J Am Med Assoc 2023;330:309–10 doi:10.1001/jama.2023.9458 pmid:37410450

    CrossRefPubMed
85. 85.↵
    1. Akinci D’Antonoli T,
    2. Bluethgen C

    . A new era of text mining in radiology with privacy-preserving LLMs. Radiol Artif Intell 2024;6:e240261 doi:10.1148/ryai.240261

    CrossRefPubMed
86. 86.↵
    1. Esmradi A,
    2. Yip DW,
    3. Chan CF

    . A comprehensive survey of attack techniques, implementation, and mitigation strategies in large language models. International Conference on Ubiquitous Security: Springer-Verlag; 2023:76–95
87. 87.↵
    1. Surla A,
    2. Bodhankar A,
    3. Varshney T

    . Generative AI: an easy introduction to multimodal retrieval-augmented generation. March 20, 2024. Accessed February 28, 2025. https://developer.nvidia.com/blog/an-easy-introduction-to-multimodal-retrieval-augmented-generation/
88. 88.↵
    1. Arasteh ST,
    2. Lotfinia M,
    3. Bressem K, et al

    . RadioRAG: factual large language models for enhanced diagnostics in radiology using dynamic retrieval augmented generation. arXiv preprint arXiv:240715621 2024
89. 89.↵
    Open AI. Introducing OpenAI o1-preview. A new series of reasoning models for solving hard problems. September 12, 2024. Updated September 24, 2024. Accessed February 24, 2025. https://openai.com/index/introducing-openai-o1-preview/
90. 90.↵
    1. Xexéo G,
    2. Braida F,
    3. Parreiras M, et al

    . The economic implications of large language model selection on earnings and return on investment: a decision theoretic model. arXiv preprint arXiv:240517637 2024
91. 91.↵
    1. Jabal MS,
    2. Warman P,
    3. Zhang J, et al

    . Language models and retrieval augmented generation for automated structured data extraction from diagnostic reports. arXiv preprint arXiv:240910576 2024