Without devouring a single textbook in the field or even spending a day in medical school, the co-author of a study in prepress stage answered a series of practical questions. The amount of correct answers was enough to pass the exam that qualifies doctors in the United States.
But the examinee was not a member of Mensa (an organization that studies the gifted) or a health expert, but the ChatGPT artificial intelligence.
Created to answer user questions in conversational form, the tool has generated so much buzz that doctors and scientists are trying to determine what its limitations are – and what it can do for health and medicine.
What is ChatGPT – and what is not
ChatGPT, or Chat generative Pre-Treated Transformer, is a natural language processing tool powered by artificial intelligence (AI).
The technology, created by San Francisco-based OpenAI and launched in November, is not like a good-spoken search engine. She’s not even connected to the internet. Instead, a human programmer feeds a large amount of data online that is held on a server.
The AI can answer questions even if it has never seen a particular string of words before, because ChatGPT’s algorithm is trained to predict which word will appear in a sentence based on the context of what comes before it. That is, it relies on knowledge stored on its server to generate its response.
ChatGPT can also answer questions in sequence, admit mistakes and reject inappropriate questions, according to the company. It’s free while it’s in the testing phase.
There is great interest in the medical field
Artificial intelligence programs have been around for some time, but this one has generated so much interest that medical organizations, professional associations and journals in the field have created task forces to see how it can be useful and to understand what limitations and ethical concerns it may bring.
Dr. Victor Tseng’s clinic, Ansible Health, created a task force to address the issue. A pulmonologist, Dr. Tseng is the medical director of the California-based group and co-author of the study in which ChatGPT showed it could likely pass the medical entrance exam.
He said his colleagues started playing around with ChatGPT last year and were intrigued when the program accurately diagnosed fictional patients in what-if scenarios.
“We were so impressed and truly dismayed by the eloquence and kind of fluidity of his response that we decided to use him in our formal evaluation process and test him against the medical knowledge benchmark,” he recalled.
The test was applied to the three parts that US medical graduates must pass to be licensed to practice medicine. The exam is considered one of the toughest in any profession because it doesn’t ask simple questions with answers that can be easily found on the internet. In addition to testing basic science and medical knowledge and case management, it assesses clinical reasoning, ethics, critical thinking, and problem-solving skills.
The study team used 305 publicly available test questions in June 2022, when none of the answers or related context had been indexed in Google – and therefore were not part of the information ChatGPT was trained on. Study authors removed sample questions that had images and graphics and started a new chat session for each question they asked.
Students often spend hundreds of hours preparing and colleges often give them time away from classes just to study for the test. ChatGPT didn’t need any prep work.
The AI passed or nearly got all parts of the exam right without any specialized training, showing “a high level of agreement and insight in its explanations,” the study wrote.
doctor is impressed
“The exam usually contains a lot of red herrings,” he said. “It’s very difficult to do a good job or try to figure it out intuitively with an approach like that. It can take a person hours to answer a question this way. But ChatGPT managed to give an accurate answer about 60% of the time with convincing explanations within five seconds.”
Dr. Alex Mechaber, vice president of the US Medical Licensing Examination at the National Board of Medical Examiners, said the results of the ChatGPT pass did not surprise him.
“The content is fairly representative of medical knowledge and the test has the kind of multiple-choice questions that the AI is likely to be successful with,” he opined.
Doctor Mechaber said the board is also testing ChatGPT in the exam. Board members are especially interested in the answers the AI got wrong to understand why.
“It’s an exciting technology. We are also very aware of the risks that large language models bring in terms of the potential for misinformation, stereotypes and biases,” he said. “I think this technology is going to get better and better, and we’re excited to figure out how we’re going to embrace it and use it the right way,” he said.
Other medical possibilities
ChatGPT has already entered the discussion about research and publication.
The study results of the sick leave exam were even written with the help of ChatGPT. The technology was originally listed as a co-author on the research, but Dr Tseng says that when the study is published in the journal PLOS Digital Health later this year, ChatGPT will not be listed among the authors because it would be distracting.
Last month, the magazine “Naturecreated guidelines that said no such program could be credited as an author, because “any attribution of authorship carries with it responsibility for the work, which AI tools cannot assume.”
But an article recently published in the journal “Radiology” was written almost entirely by ChatGPT. The text questioned whether a human author could be replaced by the machine, and the program listed many of its possible uses, including writing study reports, creating documents for patients and translating medical information into a variety of languages.
Still, there are limitations.
“I think technology will help, but everything in AI needs a safety net, a handrail,” said Dr. Linda Moy, editor of Radiology and professor of radiology at NYU’s Grossman School of Medicine.
According to her, the ChatGPT article was quite accurate, but created some references.
Another concern of the doctor is that AI could manufacture data. AI is only as good as the information it is fed – and with so much inaccurate information available online on topics like Covid-19 vaccines, it could generate inaccurate results.
Artie Shen, a colleague of Dr Moy and a PhD candidate at the Data Science Center at New York University, is exploring the potential of ChatGPT as a kind of translator for other AI programs for analyzing medical images. For years, scientists have studied AI programs from startups and larger operations like Google that can recognize complex patterns in image data. The hope is that they can provide quantitative assessments to potentially discover disease even more effectively than the human eye.
“AI can give a very accurate diagnosis, but it will never say how it arrived at that diagnosis,” Shen opined. He believes that ChatGPT can work with other programs to capture your logic and observations.
“If they can talk, there’s a chance that these systems will transmit their knowledge in the same way as an experienced radiologist,” he said.
A tool to improve doctors?
Dr Tseng said that ultimately ChatGPT can improve medical practice in the same way that online medical information has empowered patients and forced doctors to communicate better because now they have to provide information about what patients read online.
ChatGPT is not going to replace doctors. Tseng’s group will continue to test it to learn why it creates certain errors and what other ethical parameters need to be put in place before using it in real life. But the doctor thinks the tool could make the medical profession more accessible. For example, a doctor could ask ChatGPT to simplify complicated medical jargon into language that someone who didn’t finish elementary school would understand.
“AI is here. The doors are open. My great hope is that it will really make us better doctors and service providers,” said Tseng.
Source: CNN Brasil
I am an experienced journalist and writer with a career in the news industry. My focus is on covering Top News stories for World Stock Market, where I provide comprehensive analysis and commentary on markets around the world. I have expertise in writing both long-form articles and shorter pieces that deliver timely, relevant updates to readers.