NYU Langone Health’s MCIT Department of Health Informatics, Institute for Innovation in Medical Education and Institute for Excellence in Health Equity held their first Generative AI Prompt-A-Thon in Health Care this past month.

During the event, teams of clinicians, educators and researchers worked together to find artificial intelligence-powered solutions to healthcare challenges using real-world, de-identified patient data.

The event addressed large language models (LLMs) that predict likely options for the next word in any sentence, paragraph or essay, based on how real people used words in context billions of times in documents on the internet.

Also called generative AI, LLM systems randomly fill in a mix of probable next words to give a feeling of variety and creativity. A side effect of this next-word prediction is the models are "skillful" at summarizing long texts, extracting key information from databases, and generating human-like conversations as chatbots.

Despite these advancements, such AI programs do not think, and can produce conclusions and references that do not exist, Prompt-A-Thon organizers said. Thus, they require close supervision by human users, especially in healthcare, where the technology has the potential to increase safety and improve care.

THE PROBLEM

"The problem we faced was how to engage members of our workforce that might have transformative uses for generative AI and might not be technologically savvy enough to participate in our other capacity-building initiatives like exploratory access or mentored projects," said Dr. Jonathan Austrian, associate chief medical information officer, inpatient informatics, at NYU Langone Health.

"These other initiatives worked well for highly motivated colleagues who solely needed our HIPAA-compliant, patient-secure NYU GPT to safely experiment with real-life clinical data or proprietary research ideas," he explained. "Our mentored projects were ideal for researchers, educators and clinicians who already had a more well-formed idea to leverage generative AI and required mentorship from our data scientist team to shepherd their ideas to the next level."

The gap was those frontline clinicians, researchers, educators and operational leaders who understand the problems the health system faces but need some concentrated time and in-person support to connect generative AI with those challenges. There was significant demand from the workforce to close this gap, and the fastest and most efficient means of meeting that demand was through an event coined a prompt-a-thon, Austrian said.

PROPOSAL

The Generative AI Prompt-A-Thon in Health Care was a mechanism to rapidly engage a large segment of the NYU Langone Health workforce in generative AI and, in parallel, publicize the health system's existing program of engagement initiatives to those who could not be accommodated by the prompt-a-thon.

The prompt-a-thon was intended to lower the barriers for the workforce to engage with generative AI. Staff highlighted the in-person nature of the event, at-elbow mentorship by generative AI experts and that no prior experience with generative AI was required.

"Sometimes, I felt like Captain Kirk speaking to our engineering team: 'Scotty, we need more compute!"

Dr. Jonathan Austrian, NYU Langone Health

"Beyond engaging a larger segment of our workforce in generative AI, we also felt such an in-person event mixing different specialties, roles and experiences could create those novel ideas and relationships that drive innovation: a true community of learning," Austrian said.

RESULTS

The health system will be using the results of the Generative AI Prompt-A-Thon in many ways.

"First, the 70 people who participated in person and the more than 500 people who watched the webinar remotely are incorporated into our community of learners that we will continue to engage with access to GPT, updates on available technologies and additional approaches to leveraging generative AI," Austrian explained.

"Second, we anticipate many of the ideas generated during Prompt-A-Thon will be further refined by our community and will ultimately evolve into applications operationalized at NYU and disseminated to the world," he continued.

Third, the direct observations by mentors and results of the survey that was conducted will inform how the health system continues to build its internal capacity to leverage generative AI.

"We invited our health science library staff to observe the workshops, as they will be collaborating with us on formalizing a curriculum in generative AI," Austrian said. "Based on the success of the event, we will be doing Prompt-A-Thons on the road with smaller groups of researchers, educators, clinicians and members of our corporate services.

"And finally, we learned much about the technological infrastructure needed to support scalable, intensive use of generative AI," he said. "Specifically, we had 70 people synchronously prompting NYU GPT. Concurrently, in our command center, we had our data scientist team observing those interactions in real time to understand any error messages or processing delays."

NYU Langone Health's partners at Microsoft were also on site to help ensure participants had a seamless technological experience and the health system can scale the experience as it expands usage.

The preliminary results of the survey speak to the impact of the prompt-a-thon on attendees. Of the 62 who responded, 90% believe the prompt-a-thon increased how efficiently they could perform their job with generative AI. Eighty-four percent said they were likely to submit a healthcare-related generative AI project.

ADVICE FOR OTHERS

Austrian definitely recommends other healthcare organizations consider an event similar to the Generative AI Prompt-A-Thon in Health Care.

"For the introductory talks, we spent significant time describing the capabilities of generative AI and the important ethical and trust issues to consider when using generative AI," he recalled. "Next time, we will spend a little more time on prompt engineering basics.

"We had to strike a balance between having our mentors be engaged in the prompting groups without stifling innovation or disrupting the group dynamics," he continued. "We settled on one mentor for every two teams of four people per team. Given how new generative AI is, I recommend staffing one mentor per team. The 'blank' prompting page was overwhelming for some of our groups in the beginning."

Finally, Austrian said he cannot overstate the importance of having a strong technological infrastructure to accommodate a prompt-a-thon.

"Our IT department spent significant time stress-testing NYU GPT and developing creative solutions to load balance all the users," he concluded. "Sometimes, I felt like Captain Kirk speaking to our engineering team: 'Scotty, we need more compute!'"

Follow Bill's HIT coverage on LinkedIn: Bill Siwicki
Email him: bsiwicki@himss.org
Healthcare IT News is a HIMSS Media publication.