Can Chat GPT Diagnose my Patient?
ฝัง
- เผยแพร่เมื่อ 11 มิ.ย. 2023
- Artificial intelligence like Chat GPT is more available and more advanced than ever. But can it do the job of a doctor? And could it REPLACE a doctor? This is a difficult question but we'll start with an easier one- can Chat GPT diagnose one of my patients?
Details of the case have been changed for HIPAA purposes. Please comment below if you have a diagnosis to stump the AI!
Music is Circuit by Jeff Kaale
Subscribe to my fun weekly newsletter (for free!): eepurl.com/iaYycn
To check out a previous newsletter, click here: mailchi.mp/a9909f90caca/why-a...
For more Doc Schmidt content, check out my website: www.docschmidt.org/
Check out my children's book here: www.amazon.com/Night-Before-M...
Logo and graphics designed by iamlindaayoade.com and loigraphics.com (LOI Graphics Inc.)
Want me to make you a personalized video for you or your friend? Check me out on Cameo!
v.cameo.com/DFKBSe2HSib
Want to connect with me and watch more content?
Find me on TikTok!
vm.tiktok.com/ZMRFmqKts/
And Instagram!
docschmidtig?r=...
All content is intended as medical education or entertainment and is NOT intended to be medical advice. If you have any symptoms concerning you, please schedule an appointment with your doctor.
Join my channel to get access to perks! Click link below:
/ @doc_schmidt - บันเทิง
After practicing surgery for 13 years, I have never encountered a single case of Bile Acid Diarrhea. Therefore, it is truly impressive that ChatGPT was able to guess this correctly on its first try.
I think that's where AI diagnosis might shine the most. It's kinda like when newer docs only recently out of med school might pick up on a rare diagnosis/"zebra" because all the information is so fresh in their minds, whereas older doctors have seen plenty of common diagnoses, but memory may fade on knowledge of rarer things.
@@EverythingTheorist I think he was being facetious
You're not an MD
Neuro here. ChatGPT's ability to provide differential diagnoses is impressive, but its knowledge of neuroanatomy and its ability to localize lesions is at best at a medical student level. I once gave it a textbook left middle cerebral artery syndrome and could only localize to the "left hemisphere." It can correctly summarize the Circle of Willis, but started rambling like a resident during rounds when asked about the collateral systems it may form. I also gave it a vignette of expressive aphasia and could not localize to the inferior frontal gyrus. Considering ChatGPT's training data is only the internet, pretty much explains its limitations. This confirms that there's no escape from hitting the books! Not quite there yet, but a fun toy, though!
I'm a medical student and I agree. When given a classical MCQ prompt to provide the most likely differential it's very consistent even for rarer pathologies (assuming you use the right key words).
But when probed appropriately it becomes clear that it doesn't actually have a deep understanding of physiology and pathology, rather it's just very good at generating a response based off key information.
This is fine for anything that involves a spot diagnosis, but it's why it's not going to replace even radiologists or pathologists (in my opinion) because it can't handle the atypical and unusual cases like an experienced doctor can.
There's something called the mirror hypothesis with LLMs, you have to remember that it's more or less a simplified Broca's and Wernicke's area, trained on a huge amount of data, but without the rest of a brain. The user acts as the rest of the brain, so the way you break down and phrase the prompts makes a huge difference. We're working on some automated methods for this like AutoGPT, Tree of Thoughts, etc but for now without an API key you have to be very step-by-step and 'give them time to think' to get accurate results. A lot of LibGen is in there though, including neuroscience books and papers, but to get the model to consider them you might have to phrase the prompts in similar terminology to those sources, instead of how a medical student might ask the questions (that kind of abstraction between complexity levels is something that a different part of the brain handles, e.g. the user, until ToT is implemented)
How do you think it would do with spine and brain Cavernous Malformation/CCM2?
"but its knowledge of neuroanatomy and its ability to localize lesions is at best at a medical student level" Oh my god it's like I'm reading a Dr. Glaucomflecken skit 😄
I had to go back and check the commenter to be sure it wasn't Dr. G 😂
The really cool thing here is that its not specialised, it wasnt taught how to do this. Its just a general chat AI. A specialised diagnostic AI could do a LOT more.
You should DEFINITELY make this into a series, although you could go more in depth into the reason for your answers to its questions.
I would be curious if ChatGPT can get close if the patient history is more narrative, the way a doctor would typically hear it from a patient. Part of medical training is deciphering meaningful symptoms from long anecdotes.
In general ChatGPT is good at extracting information from long-ish paragraphs, so it might shine there. Would be interesting to see
Love your channel, Doc Schmidt. I was diagnosed with Celiac disease about 5 years ago and was still having “bathroom emergencies”, even though I know very well the dietary restrictions (my dad has had Celiac for decades!). I had a cholecystectomy 15 years ago and just was never quite right since. I did my research and landed on possible BAM. Called my GI, asked for a Chlorestyramine trial and…problem solved.
PS. I mix it with milk and it’s quite palatable that way.
It will get even better very fast, but I think the big thing it'll be is a interview tool for doctors. If you have it listening in the background of an interview it could prompt you with questions, differentials that you may not have considered, what investigations have the best evidence, and so on, and then automatically write up medical notes. It will never do the human thing of showing a patient you care and filling them with confidence and trust, but it can supplement all of the knowledge based skills for a doctor.
Make it diagnose an autoimmune condition, please. That would be interesting to see.
I don't think it would be great at autoimmune diseases based on the fact that autoimmune diseases often present as more common diseases and AI is asking "what is the response most likely to correspond with your prompt?" Using MS as an example, chatgpt would probably misdiagnose it as some sort of eye problem when given the most common early symptoms just like real doctors do
@@stephenchurch1784 That's why I would like to see it try! I want to know how badly it would do--sometimes I am just a bit of jerk.
As someone who has had this all my life, I would love to see Chat GPT try to guess Mitochondrial Disease
Med student here. IBS was my first thought. Dang it! So close. I would love to see more of this particular series.
Notice that the doc. was able to guide AI because he had elicited a complete history. That is the critical variable here. As every med. student knows, "Garbage in; garbage out".
I wonder what key information suggests that Bile-acid malabsorption was more likely. The fact that the gallbladder was removed?
Irritable bowel, leaky gut syndrome , histamine intolerance,DAO deficiency, and histamine intolerance please. Thank you, Dr Schmidt. Love your videos ❤
Love your channel, love your funny shorts (not pants) as a retired healthcare pro, this takes me back to the challenges of the old days. Thank you!
There are many lawyers who probably wish they hadn't used it...😂
Well, at least two. 🤣
context?
@@fulltimeslackerii8229they are the lawyers in law school who used chatgpt to finish assignments
@@fulltimeslackerii8229 Legal Eagle did a video about it recently. It's... A lot.
Please make this a whole series it’s fascinating!
I see this being used in like a waiting room or pre-going to the doctor on their recommendation to help with influx of patients.
Agreed!
I think this will be an excellent tool for experts..for an inexperienced person or even the patient themselves, they may mot know the right symptoms to input or right questions to ask.
Watched with interest. Remember, this is only the very beginning of AI. But in terms of its utility in clinical medicine, the art of medicine is in obtaining the pt's. history. It might be informative to ask a pt. to provide responses to the program's questions. I doubt it could then provide as precise a diagnosis as it did in the case you chose. As an aid to the physician who must have a complete differential for a constellation of symptoms, signs and historical data, it should be very useful.
Very interesting. My AI caveat is that after 13 years of being a doctor the one thing I have learned is that half the battle is knowing how to ask the question and knowing how to tease information from anwers that are not ckear. I don't think AI would cope well with that
That is the "art of medicine". Pts. will always tell you their diagnosis. All that you have to do is ask.
This is a great video, thank you for sharing!
I have been a physician for over 25 years. This is a tool that can help physicians, not replace them. My worry is when people without a medical background start to use this to self diagnosis ( I’m looking at you, GOOGLE) that it will lead to a lot of angst and people with incorrect information.
ChatGPT is very good for generating differentials based off prompts with key information (in some cases). But it struggles with limited information or with mimics.
For example every time I give ChatGPT a prompt for Wellen's syndrome (V2/3 T wave inversion+resolved chest pain) it generally fails to give this as the primary differential and usually resorts to saying myocardial infarct. Most clinicians would recognise that active MI is less likely in chest pain that resolved days ago, but when ChatGPT sees chest pain + T wave changes it assumes MI because those are the triggers for it to provide that answer.
I guess what I'm trying to say is that it gives the illusion of diagnostic accuracy because it's very good at generating answers based off key information but it lacks a deep understanding of physiology and disease that an experienced doctor has.
"Active" being the operative descriptive in your analysis. I wouldn't call AI's diagnosis wrong. Dangerous "stuttering" MIs are not uncommon and demand prompt intervention. The mistake would be to tell such a pt. who might have presented to an ER something to the effect that "you are pain-free, your enzymes are normal so, despite the appearance of your EKG, you can go home and call your physician in the morning for follow-up.
@@wholeNwon I don't think any experienced doctor would send a patient with Wellen's home before adequate work-up and treatment. But that's besides the point.
The whole point was to highlight that ChatGPT generates a response based off key information but lacks the understanding that doctors have. In this scenario, previous resolved chest pain should put a critical artery stenosis (ie Wellen's) as a higher differential than MI. A previous resolved MI would also be more appropriate.
Mainly for the simple fact that an evolving active MI doesn't spontaneously become symptom free for multiple days, it's a "common sense" approach that doctors with clinical intuition have but not an AI that relies on generating a response based off keywords.
It's a big reason why people who think AI will replace doctors are being overly optimistic. Medicine is more complex than giving an answer to a prompt with clear key information.
I tried this with ChatGPT 4 and this was the answer:
"Your patient's symptoms of resolved chest pain and an ECG with T-wave inversion in leads V2 and V3 may indicate several conditions. Here are the most likely differential diagnoses:
Ischemic Heart Disease: T-wave inversion can be a sign of ischemic heart disease, particularly if the patient was experiencing chest pain. This could be a recent myocardial infarction or unstable angina. Given that the pain resolved 2 days ago, it might indicate a completed non-ST elevation myocardial infarction (NSTEMI). Biomarkers such as troponin would help confirm this.
Wellens' Syndrome: This is a pattern of deeply inverted or biphasic T-waves in V2 and V3 that is highly suggestive of critical stenosis of the proximal left anterior descending (LAD) coronary artery. Patients typically present during a pain-free period, making it a relevant consideration for your patient."
Here's a link to the Convo: chat.openai.com/share/e41edc39-035b-4b6c-ae4c-fde571cea8e8
@@TheArikatz Notwithstanding the fact that this highly specific syndrome still wasn't the primary the differential, it was just an example to highlight the way this kind of generative AI functions.
Whether or not ChatGPT is able to correctly answer this particular basic question didn't change the fact that it doesn't actually understand how the body works. It recognises key information and patterns in the prompt given and then provides an answer based off that.
That means it can perform very well if there a spot diagnosis to be made, where there is limited key information relevant to making the diagnosis. But it also means it will always struggle when faced with cases that involve a mimic, or atypical signs, or which are complex, or all of the above.
@@m136dalieVery true; ChatGPT cannot replace a doctor. I'm curious what you would think about using a generative AI model like this (perhaps one trained more specifically on medical texts) as a *tool* to help doctors rather than as a complete solution on its own
Would love to see another episode of this
Womder if "IT" (Chewbacca/Robin) could diagnose lymphocytic colitis, celiac, or SI Joint Dysfunction, all of which took YEARS of seeing Drs and Specialists before I got diagnosed.
It would be interesting to see how it responds if the information wasn't neatly presented to it...
Great vid!
I mean, after cholecystectomy isn't that the most likely cause? Would love to see you give it rare diseases you've encountered.
got curious and asked the AI about a little bit of a curious medical thing i had as a kid. i had a fluid-filled cyst in my palm which ended up pulling on the tendons of my pinky finger, making it bent and unable to straighten. ended up having to get it surgically removed before it got to my other fingers and never gained back full function of the finger. the AI thought i had had a trigger finger at first, but it’s second guess was spot on! it seemed to be against the idea of surgical intervention though lol
i ran another one about my brother’s ruptured appendix after appendicitis (didn’t go to the doctor’s, appendix decided to check out, symptoms were better yay! and then way worse lmao) and it provided an ovarian cyst as one of the options. pretty sure i mentioned that the patient’s sex was male haha (brother is fine now btw!)
This was incredibly impressive! A generative AI similar to or based on ChatGPT definitely seems like a good diagnostic tool that doctors could use to help remind them of possible diagnoses.
I tried this with GPT-4, and it was able to diagnosis correctly guess that diagnosis (It didn't offer anything else, it was certain it was BAM/BAD. It also offered confirmation tests, a SeHCAT scan or treatment trial with a bile acid binder. I also asked if a colonoscopy was recommended, and GPT-4 said no, unless there was any red flags and suggested doing a trial treatment for bile acid malabsoption, and to only scope if her symptoms didn't improve.
I would also like to say that chatGPT (The green icon you used) is running the outdated GPT-3.5 which is much less capable than the GPT-4. However, you can only ask 25 questions in a 3 hour period with GPT-4.
Would be interesting if you tried this again with GPT-4.
Listed as "Ver Rare": Pyroderma Gangrenosum (I was care giver for nearly 2 years before it was correctly diagnosed, treated, and resolved). Sorry it's not GI, but it is rare, interesting, and as far as I know, still a medical mystery.
Yes, I've spent 20 years of misdiagnosis and being branded a drug seeker in some cases.
Its not gonna replace, its gonna help you! You are gonna be able to treat more patients and faster and save more lifes
I think a useful test would be to see if you give it symptoms that aren't necessarily indicative of any problems that need treatment, will it say that or will it come up with something just to answer the question of "provide a diagnosis."
Cavernous Malformation and possibly more specific CCM2 with spinal and brain cavernous. How’s that for rare? This is what I have and it’s Cavernous Malformation awareness month.
It would be interesting if you can get access to IBM Watson (the AI that won Jeopardy), which is actually designed for medical diagnosis (with different training data to the Jeopardy version!), to compare.
Try acute intermittent porphyria
It's funny how we're basically soing the same thing with ChatGPT that we did with Akinator half a decade ago--only now, instead of guessing fictional characters, it guesses life-or-death situations. 😨
When you said “Bile Acid Diarrhea”, I thought “oh, like what I get because I had my gallbladder removed last summer?”
Do I get an honorary doctorate now? 😂
It would be great another video like that but with direct answers from a patient although probably that's almost impossible. I'm not a medical professional but I don't think the average patient communicates that well medical history.
Fyi, it looks like you're using gpt 3.5. GPT 4 has been shown in studies to be a much better medical reasoner. It's generally far superior to 3.5 in all area of performance except for speed of text generation.
The SeHCAT test gets payed by insurance in some countries. You just have to find a doctor that knows it
Not a med student, but a hopeful pre-med. My first thought when you said she had her gallbladder removed was something to do with the inability to break down/absorb fats or other nutrients - kind of close, I suppose!
@Doc_Schmidt Is this using the free gpt-3.5 model or the subscription gpt-4 model in ChatGPT?
See if It can guess POTS, Gastroparesis and or intusseception
Id be truly impressed if it could diagnose a pheochromocytoma/paraganglioma or some type of secreting tumor.
Try Stauffer syndrome
Dx rosacea va lupus rash.
Huh! It just occurred to me to try out my car issues on ChatGPT... Really curious to see what it says!
How about Acalacia or Pure Autonomic Failure?
My second guess would have been EPI
I think chat GPT should be 1st line or as patient advocate to make sure they go to the right next place. Right now, there are so many hoops a promary care doc needs fill out on EMS, its not really seeing a doctor that is happpening anyways.
Long shot but could you do ornithinetranscarbamylase (OTC) deficiency? Its the reason i had to get a liver transplant.
Lots of GI related things involved of course
Are there actual medications that can help with the diarrhea after losing your gallbladder?
Hey Dr Schmidt! Is bile acid diarrhea also common in laparoscopic cholecystectomy? I had it three weeks ago and i was wondering! Thanks!
It can be - I am fine a long as I don't have chippy for lunch and chinese for dinner in the same day!
It can be - I am fine a long as I don't have chippy for lunch and chinese for dinner in the same day!
It can be - I am fine a long as I don't have chippy for lunch and chinese for dinner in the same day!
Dissociative identity disorder would be an interesting one
I’m curious if it would catch my proper diagnosis or come to the same erroneous conclusion as my prior hepatologist.
I was diagnosed with NASH based on bloodwork and imaging (MRI and FibroScan) as well as being significantly obese. I had elevated liver enzymes and abdominal tenderness. They were very surprised to find out that I had “cirrhosis” according to my scans, with low grade esophageal varices and portal hypertension. By the followup visit, my liver enzymes had returned to normal levels, and I lost a good 20lbs after starting Buspar. I had the lowest possible MELD score. No biopsy was taken. The usual diet and exercise lifestyle changes were recommended.
5-6 years later, I was finally diagnosed with hEDS, POTS, and most importantly, MCAS and chronic idiopathic urticaria. I also had some puzzling followup imaging that said I only had fibrosis.
I cross referenced my diagnoses to see if NASH was a common comorbidity as well. It wasn’t. I did a little more searching in medical journals and ran a specific search on mast cells and NASH. I found a paper cautioning other doctors that mast cell infiltration of the liver could give a false positive. At my next GI appt, I asked about this. I was now on multiple medications for my MCAS. The blood tests and FibroScan showed only mild fibrosis this time, and the supposed irreversible damage had absolutely vanished. My doctor thinks that this was indeed what happened to me. It’s been a life changing experience.
Do remember that the leading edge of this technology which we have at this present moment is the worst it will ever be going forward through time.
This will work for regular cases based on statistics of symptoms. But.....what if your patient is not in that group of statistics and their symptoms are getting worse and fever is higher . PT is middle aged over weight white woman that was healthy before onset of symptoms. Patient has a lesion on right jaw area that is much larger than last time she was seen. Dr treats for a spider bite and discharged. Patient didn't improve at home. Her fever increased and lesion is 10x bigger than day before, PT calls Dr.. Dr didn't do and culture of lesion on onset of being seen. PT did not fit the profile of statistics. Patient didn't return to hospital per Drs advice.
Next morning PT has fever 105 and highly tacticardia 170bpm. Bp 190/90. Patient is completely involved with lesion and very hot and red. ER Dr is sure she having heart issues and gives nitroglycerin meds. Patient sleeps. Nurse pleads for Patient to run culture test. Culture returns as raging infection and rushed to ICU and prep for surgery. PT suffers major heart issues with cardiditis. Angiography ordered.
So did the patient get the bile acid diarrhea after having her gall bladder out?
ChatGPT - it's Lupus...
😅
"If you want to guess along, just close your eyes for the next five seconds."
Proceeds to say the dx out loud
Welp lol
Well, he did say also don't listen
I have indolent systemic mastocytosis. It took 2 months, many thousands of dollars worth of tests and doctors from every specialty trying to figure out my symptoms before I got a diagnosis. I would love to see how AI does at guessing my diagnosis.
I don't know if you follow any of the LawTubers but the fact that ChatGPT just made up 6 legal precedents when asked to help write a motion is frightening! More frightening was the fact that the lawyers doubled down and said that of course the cases were real - ChatGPT assured them they were - when the court and opposing lawyers questioned their veracity! Personally, the concept of AI diagnosing me is frightening! It can only be as good as the programmer. - who may not be a doctor... I'll see a human being, thanks!!
This test was made with chat gpt 3? Or chatgpt 4?
I wonder how long it will be til health insurance companies refuse tests or treatment because "Chat GPT says..."
Please. The patient has to fill out forms that ask surgical history, so with symptoms starting back when gall bladder was removed, and symptoms triggered by food? My own first guess is straight from my nursing school med surg textbook that I havent looked at for thirty years - diarrhea due to bile acid. Too easy. Problem would be that a textbook explanation can still be incorrect.
Where can i find the power plug! Lol
If chat gpt says constipation then i think its right
ChatGPT is better then several doctors I met over the years.
Please ask about Hollow visceral myopathy.
Thank you.
Cyclic vomiting syndrome, please, and make sure to differentiate between it and cannabinoid hyperemesis syndrome.
If you're speaking from the experience of being a patient, then I'm really sorry you had to go through that. Misdiagnosis due to prejudice sucks.
Just out of curiosity, are you using gpt4 or gpt3.5? I'm just wondering since it seems like you're maybe using gpt3.5 (based on the way it kind of tapers off into becoming more and more general as the messages get longer), and you should maybe consider paying the extra 20usd they charge for access to gpt4 in order to truly get a real view of what AI can do these days... Gpt4 is RIDICULOUSLY more competent.
Within the next 10 years this debate will truely gain traction in every sense
More like 10 months. This is exploding all over the world.
Can it replace doctors? Hell no. Can it function as an *amazing* tool for doctors to use in order to help come up with weird "unicorn" illnesses when you have no idea after a slew of tests? Absolutely.
Ive had diarhea before which i learned was me expelling all the fat my intestines did not digest. Happens a lot. A lot of my weird gi issues. It is never a normal color fyi (tmi sorry)
I want you to test Chat GPT to see if It can guess the diagnosis of Tay-Sachs Disease.
Am I the only one who would love to shunt my job over to a bot and go live in some warm country for the rest of my life XD
Now AI has already trained 😅😅😅😅
You should test chatgpt on diagnosing things with vague symptoms. The stuff that doctors often misdiagnose. Like chronic fatigue syndrome, fybromyalgia, ehler-danlos syndrome, or even endometriosis. Let's see if ChatGPT is better at believing women.
Another point: try explain symptoms like a patient would rather than a doctor.
Try to stump it with Congenital Sucrase-Isomaltase Deficiency (CSID)
You should Ask it of it can diagnose Iktyosis. My brother have that
Could it diagnose achalasia?
ChatGPT is software and hardware. One can call it whatever one wants. Even Chewbacca is acceptable.
I’ll consider that
But consistently referring to ChatGPT as "he" is distasteful and sexist. Call ChatGPT Chewbacca if you wish, or give it some gender-neutral name like Robin. But, please Please PLEASE do not call it 'he'. At best, it is an IT.
@@Joy21090 Dude, people can call it whatever they want to call it. Don't like it? Tough!
Nobody has offended anyone by calling ChatGPT whatever they wish to call it, so there is no reason to complain.
@@Gustav.J you're mistaken on at least part of that screed
@@Joy21090 In English, "he" is the default for an entity of unknown gender. Doc Schmidt didn't do anything incorrect here.
One time, I gave it a simple biology, where the patient had hepatitis and was CMV+ and was vaccinated to hepB
It told me the patient has a hepB hepatitis lol
At least use GPT-4. It's the difference between a elementary school students and an undergrad.
ChatGPT will hardly even diagnose a hang nail for me. It's always "you should go see your expensive af doctor about it".
AI will be a very specialised and helpful tool....but we also need a human factor in medicine..it cant be all robotic.
For me,no way.I doubt it could replace humans.
Then you haven’t been paying attention.
You know its not chatgpt itself but another specialized AI that's gonna replace doctors right? Chatgpt is just the first proof that it is something tha CAN be done.
The problem would be getting a large enough database across several different languages and cultures to train the LLM better than a general AI.
Moreover, a doctor still curates the answer to lead it to a specific diagnosis in this video. A medical professional is still needed to translate the sociocultural nuance to something the AI would understand.
The real issue for doctors is whether MD/DO + AI would be significantly better than an NP/PA + AI for a majority of cases.
@@jonathanchan3087 Its just a matter of time, regulation and cooperation between institutions can speed up the proccess too. In my country diabetic retinopathy is aldeady been diagnosed using ML algorithms.
any doctor who pushed the mRNA injections deserves to be replaced.
So every doctor?
@@Doc_Schmidt no not the thinking doctors, just the sheep doctors. I hope someday those with myocarditis, tinnitus, joint issues, blood clots, strokes etc sue those doctors that they trusted and destroyed their otherwise healthy bodies.
It answered like an MD and not a ND. It didn't give you the recommended treatment of Ox Bile supplements which make up for bile deficiencies in the system which supports the liver, combats gut imbalances, and promotes gut health. I personally know someone with bile acid malabsorption who has success with them but also eats healthy with an ideal frequency for his condition. Meds are probably great for people who don't want to change their bad habits or don't respond well to ox bile supplements.
I would rather speak with a real human.