Making shiny objects illuminating: the promise and challenges of large language models in U.S. health systems

Abstract

[...]the regulatory landscape of AI in the U.S. has been chaotic in recent days, as reflected by the rescission of the Biden administration’s executive order on AI safety5, the uncertainty around many public agencies including U.S. Food and Drug Administration (FDA) and National Institute of Standards and Technology (NIST)6 which created the AI Risk Management Framework (RMF). Whereas traditional AI systems were often designed for batch processing or offline analysis with fewer real-time interaction demands, LLMs - depending on the use case - require infrastructure optimized for low-latency inference and seamless compatibility with interoperability standards such as Fast Healthcare Interoperability Resources (FHIR) to ensure fast response times in clinical workflows. Developing and deploying generative AI systems in healthcare involves significant investments in data collection, model development, computational resources, software tools, and ongoing maintenance. At a national level, the consideration of cost structure and resource planning should also include the gap between health systems that have the resources vs. those that do not, and the consequential impact faced by patients.