Archive \ Volume.17 2026 Issue 1

Designing Large Language Models That Refuse Unsafe Medication Questions Well

, , ,
  1. Department of LLM Safety and Medication Questions, Faculty of Pharmacy, Federal University of Minas Gerais, Belo Horizonte, Brazil.
  2. Department of AI Refusal Mechanisms in Pharmacy, Faculty of Pharmacy, University of Coimbra, Coimbra, Portugal.
  3. Department of Safe LLM Design for Pharmacy, Faculty of Pharmacy, University of São Paulo, São Paulo, Brazil.

Abstract

Large language models can produce fluent medication information while remaining vulnerable to factual error, missing context, unsafe personalization, adversarial manipulation, and misleading confidence. Refusal is therefore not merely a conversational limitation; when appropriately designed, it may function as a medication-safety control. This article develops a proposed Pharmacy Refusal-Safety Framework for determining when a model should answer, qualify, redirect, or refuse a medication question. The framework combines a multi-axial taxonomy of unsafe and unanswerable questions, a layered risk-interpretation architecture, an action-selection boundary, a non-abandonment response contract, and lifecycle governance. Questions are characterized according to potential harm, urgency, personalization, context sufficiency, epistemic answerability, premise validity, misuse potential, vulnerability, and professional-authority requirements. The selected action is constrained by the consequences of error rather than linguistic confidence alone. A safe refusal should state the relevant boundary, avoid covert individualized advice, preserve appropriate general information, identify missing context, direct the user toward a feasible next action, and provide urgent safety-net guidance when danger is plausible. Evaluation requires realistic safety-critical cases, expert adjudication, subgroup analysis, adversarial testing, user-comprehension assessment, workflow simulation, and prospective monitoring. The framework is an original non-empirical synthesis rather than a validated clinical algorithm or deployment standard. Its categories and transitions may behave differently across models, medicines, languages, users, jurisdictions, and care settings. Empirical validation is therefore required before clinical or pharmacy implementation.


Downloads: 25
Views: 91

How to cite:
Vancouver
Costa G, Ribeiro L, Alves R, Lopes M. Designing Large Language Models That Refuse Unsafe Medication Questions Well. Arch Pharm Pract. 2026;17(1):77-84. https://doi.org/10.51847/MSRqgiNB22
APA
Costa, G., Ribeiro, L., Alves, R., & Lopes, M. (2026). Designing Large Language Models That Refuse Unsafe Medication Questions Well. Archives of Pharmacy Practice, 17(1), 77-84. https://doi.org/10.51847/MSRqgiNB22

Download Citation
References
  1. Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023;388(13):1233-9. doi:10.1056/NEJMsr2214184
  2. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-80. doi:10.1038/s41586-023-06291-2
  3. Ong JCL, Chen MH, Ng N, Elangovan K, Tan NC, Jin L, et al. A scoping review on generative AI and large language models in mitigating medication related harm. NPJ Digit Med. 2025;8(1):182. doi:10.1038/s41746-025-01565-7
  4. Grossman S, Zerilli T, Nathan JP. Appropriateness of ChatGPT as a resource for medication-related questions. Br J Clin Pharmacol. 2024;90(10):2691-5. doi:10.1111/bcp.16212
  5. Farquhar S, Kossen J, Kuhn L, Gal Y. Detecting hallucinations in large language models using semantic entropy. Nature. 2024;630(8017):625-30. doi:10.1038/s41586-024-07421-0
  6. Alber DA, Yang Z, Alyakin A, Yang E, Rai S, Valliani AA, et al. Medical large language models are vulnerable to data-poisoning attacks. Nat Med. 2025;31(2):618-26. doi:10.1038/s41591-024-03445-1
  7. Omar M, Sorin V, Collins JD, Reich D, Freeman R, Gavin N, et al. Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Commun Med (Lond). 2025;5(1):330. doi:10.1038/s43856-025-01021-3
  8. Rosen KL, Sui M, Heydari K, Enichen EJ, Kvedar JC. The perils of politeness: How large language models may amplify medical misinformation. NPJ Digit Med. 2025;8(1):644. doi:10.1038/s41746-025-02135-7
  9. de Hond A, Leeuwenberg T, Bartels R, van Buchem M, Kant I, Moons KGM, et al. From text to treatment: The crucial role of validation for generative large language models in health care. Lancet Digit Health. 2024;6(7). doi:10.1016/S2589-7500(24)00111-0
  10. Labkoff S, Oladimeji B, Kannry J, Solomonides A, Leftwich R, Koski E, et al. Toward a responsible future: Recommendations for AI-enabled clinical decision support. J Am Med Inform Assoc. 2024;31(11):2730-9. doi:10.1093/jamia/ocae209
  11. Wiens J, Saria S, Sendak M, Ghassemi M, Liu VX, Doshi-Velez F, et al. Do no harm: A roadmap for responsible machine learning for health care. Nat Med. 2019;25(9):1337-40. doi:10.1038/s41591-019-0548-6
  12. Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022;28(5):924-33. doi:10.1038/s41591-022-01772-9
  13. Kompa B, Snoek J, Beam AL. Second opinion needed: Communicating uncertainty in medical machine learning. NPJ Digit Med. 2021;4(1):4. doi:10.1038/s41746-020-00367-3
  14. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: The Achilles heel of predictive analytics. BMC Med. 2019;17(1):230. doi:10.1186/s12916-019-1466-7
  15. Pais C, Liu J, Voigt R, Gupta V, Wade E, Bayati M. Large language models for preventing medication direction errors in online pharmacies. Nat Med. 2024;30(6):1574-82. doi:10.1038/s41591-024-02933-8
  16. Liu S, Wright AP, McCoy AB, Huang SS, Steitz B, Wright A. Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation. J Am Med Inform Assoc. 2025;32(6):1032-9. doi:10.1093/jamia/ocaf059
  17. Chen S, Guevara M, Moningi S, Hoebers F, Elhalawani H, Kann BH, et al. The effect of using a large language model to respond to patient messages. Lancet Digit Health. 2024;6(6). doi:10.1016/S2589-7500(24)00060-8
  18. Liu S, McCoy AB, Wright AP, Carew B, Genkins JZ, Huang SS, et al. Leveraging large language models for generating responses to patient messages—a subjective analysis. J Am Med Inform Assoc. 2024;31(6):1367-79. doi:10.1093/jamia/ocae052
  19. Sorin V, Brin D, Barash Y, Konen E, Charney A, Nadkarni G, et al. Large language models and empathy: Systematic review. J Med Internet Res. 2024;26. doi:10.2196/52597
  20. Kim J, Chen ML, Rezaei SJ, Liang AS, Seav SM, Onyeka S, et al. Perspectives on artificial intelligence-generated responses to patient messages. JAMA Netw Open. 2024;7(10). doi:10.1001/jamanetworkopen.2024.38535
  21. Asgari E, Montaña-Brown N, Dubois M, Khalil S, Balloch J, Au Yeung J, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. NPJ Digit Med. 2025;8(1):274. doi:10.1038/s41746-025-01670-7
  22. Draelos RL, Afreen S, Blasko B, Brazile TL, Chase N, Desai DP, et al. Large language models provide unsafe answers to patient-posed medical questions. NPJ Digit Med. 2026;9(1):241. doi:10.1038/s41746-026-02428-5
  23. Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, et al. Toward expert-level medical question answering with large language models. Nat Med. 2025;31(3):943-50. doi:10.1038/s41591-024-03423-7
  24. Huo B, Boyle A, Marfo N, Tangamornsuksan W, Steen JP, McKechnie T, et al. Large language models for chatbot health advice studies: A systematic review. JAMA Netw Open. 2025;8(2). doi:10.1001/jamanetworkopen.2024.57879
  25. Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023;6(1):120. doi:10.1038/s41746-023-00873-0
  26. Lekadir K, Frangi AF, Porras AR, Glocker B, Cintas C, Langlotz CP, et al. FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388. doi:10.1136/bmj-2024-081554
  27. Blease C, Kaptchuk TJ, Bernstein MH, Mandl KD, Halamka JD, DesRoches CM. Artificial intelligence and the future of primary care: Exploratory qualitative study of UK general practitioners' views. J Med Internet Res. 2019;21(3). doi:10.2196/12802
  28. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-53. doi:10.1126/science.aax2342
  29. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA. 2017;318(6):517-8. doi:10.1001/jama.2017.7797
  30. Seyyed-Kalantari L, Zhang H, McDermott MBA, Chen IY, Ghassemi M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med. 2021;27(12):2176-82. doi:10.1038/s41591-021-01595-0
  31. Roberts M, Driggs D, Thorpe M, Gilbey J, Yeung M, Ursprung S, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat Mach Intell. 2021;3(3):199-217. doi:10.1038/s42256-021-00307-0
  32. Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health. 2021;3(11). doi:10.1016/S2589-7500(21)00208-9

 

 


Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.