AI vs Doctors Competition (RESULTS)
ฝัง
- เผยแพร่เมื่อ 8 ก.ย. 2024
- Researchers just tested a bunch of AI models against a bunch of doctors. Here's who won...
There were 4 groups: Large language models, groups of expert eye doctors, eye doctors in training, and unspecialized junior doctors. Everyone had to answer 87 questions from a standard test that doctors in the UK need to pass before becoming an eye specialist. Here's how they each did.
The intention of this study is NOT to see if any AI tool can replace any doctors. This study just showed that in this one test, one model’s performance was almost as good as experts. We could imagine a world where more generalized doctors who don’t have easy access to a specialist could ask an AI for a second opinion or help make a referral or even help with a diagnosis. It’s one way researchers are exploring how to improve medical care for more people.
If you like optimistic science and tech stories, subscribe for more of our show Huge If True.
If you want to read the full study, search this: DOI:10.1371/journal.pdig.0000341
#science #sciencefacts #education #study
Asking a doctor for a second opinion from chat gpt would be wild
will be the future tho
I did the 3 days ago to check Chat's advice on reumatoïde artritis. It was superhigh glucoselevel 18+ mmol.
Drug interaction checkers are already a real thing. People use tools like medscape all the time... Once and a while you need to look at your literature again, cross check, and see if there's updated information available..
@@PeridotFacet-FLCut-XG-og1xxAlso, medical professionals already have their own search engines; it's not too far of a stretch to consider training their A.I. on the results too.
Akinator for medical problem. Answer a few questions and the AI will narrow it down to a few rare cases of cancer 😂
Google AI be like: "Yeah, you can safely apply sulphuric acid on your retina to see better 🤓"
Ai : I apologies for the mistake, you are right it's not necrosis!
After It decided I had to amputate my arm because I have a bruise 💀
@@jokrwx3nah those apologies is more annoying than anything and repeating the same mistakes over and over 😭
I glued 3 fingers back on thanks to A.I. only to learn now it was the chef A.I. and I should have given the fingers to 2 birds if I wanted them reattached to my hand.
Shine those eyes to see better in the dark. 😁
"I have a shoe shaped birthmark"
Google AI: "Stage 4 skin cancer"
As a med student, i can confirm that chat gpt 4 can answer about 90% of the standard questions I give to it correctly.
However, when i tried to test it on an actual case that presented to us in the hospital, or when a professor tweaks the question a bit to make it more difficult for us, the rate dropped to almost 40% - 50%.
I guess what I'm trying to say is that if you feel sick, dont try to figure out what's wrong by asking AI. Please go to a hospital (even though i know going to a hospital kinda sucks)
Well for a lot of people that is a pretty big financial decision. Having a completely free option that will hopefully be exponentially better with GPT-5 that's being trained right now will be an incredible resource for a lot of people.
A big thing is also about knowing what to ask. My cat throws up hairballs almost everyday, I thought it was normal for cats so I never really brought it up at the vet. Then she got a bald skin patch and the vet was confused since she did not have fleas, but the second I casually brought up the vomiting she connected the dots that it’s probably a food allergy.
@noone-ld7pt Yeah, that's the unfortunate truth tbh. I advise everyone to go see a doctor while if i get an illness myself, I'll check with Chat gpt to make sure it's serious enough, lol.
But a highly specific and trained language model would do better than any typical trained professional? Idk tho, I'm really toxic feeling that fullblown AI can replace any and every jobs that human will ever qualify for. Truly toxic mentally that I'm fixing.
@JuniorEN3213 you're probably right, though. 😂
Answering a multiple choice checklist is very different than looking at a real world situation and applying high judgement to come up with the best solution.
Too bad doctors at bad at judgement and are mostly just self-interested functionaries who specialize at knowing what insurance companies will reimburse the most $$ for. I can't wait for AI to replace doctors. We don't need these rich douchebags who mindlessly steer everyone they can towards precisely the medications and procedures that make them the most $$$$
Thats why AI is a tool gd damn, it will never replace humans, but it can make our jobs soooooooo much easier...
Then why are we giving the questions to humans 😂 looking forward to humans being replaced
Trained AI models have been better than doctors for a while now at detecting actual conditions from imaging/scans. Now they can also incorporate symptom information from textual content. If you look at what many of those algorithms do - minimizing error using mathematics, you'll see why it is only a matter of time before thousands of human jobs can be replaced by just a few human jobs. I suspect there may be a plateau until self-programming comes into being, and then it is an exponential explosion of intelligence.
@@bottomtext251 AI will replace humans, it already has, impossible to say for sure which jobs will be affected in the future, I do agree however that even as a tool it is still a huge advancement
Something else to consider, is that medicine is very hypothesis driven. If you don’t look for something, you won’t find it. Actually interviewing a patient and coming up with a differential diagnosis is much more difficult than answering a practice question.
Doctors are $$ driven; not hypothesis driven. The idea an AI with the same data as a doctor can't outperform their judgement is laughable. In 5-10 years, the idea of consulting a human doctor for a health issue will be exclusively a punchline -- like going to a witch doctor or a spiritual healer -- only something a deluded, superstitious moron would consider doing.
One of the first successful AI applications was with medicine. Doctors make decisions based on symptoms, but there are a load of rare diseases that often slipped through. AI was able to suggest the obscure diseases and suggest tests that should be performed to make a better diagnosis.
AI will probably cure all diseases because I don't see why we cannot have microscopic AI immune system able to handle any viruses. Just inject this AI fluid full of microscopic robots, and boom all cancers are fixed including brain tumors. Genetic engineering will also help cure diseases also, so the future seems bright but we all know someone is going to abuse it for money.
suggested tests = additional costs,
nah, we stick with the doctors
Yeah all of my doctors including specialists and 2 surgeons who did surgery on me were stumped on my issues. If you don't fit the top 80% of diagnoses and symptom combos, it feels like asking doctors from the 1800s, they're just flailing and hoping you either die or get better on your own.
@@bw1227 Maybe try not living in Murica
@@bw1227Doctors often run many unnecessary test for liability reasons.
Google AI be like yeah you can safely remove your eye to clean it from grains of sand under tap water and put it back on
😢
One of the most important skills good doctors have is interacting with patients to obtain an accurate picture of all the symptoms and other important information required for diagnosis. If this is not done correctly, no computer will ever be able to do an adequate diagnosis.
Absolutely true but that is not the point of video. It was not to say ai can replace doctors but doctors can still use it as a tool to a certain extent.
This is the same as many profession. There is so much unnecessary hate towards ai. While it is true that some jobs will be phased out, but those who work with ai, instead of against ai will have a better economic chance in the future.
Is the same when computer and smartphone came out in the everyday work world. Those who are against digital transformation are phased out, but those who able to utilise the tool continue to stay relevant and competitive in the job market
If you want to read the full study, search this! DOI:10.1371/journal.pdig.0000341
❤ thanks 🎉
Interesting! I'll read it, thanks 💚
You're my love you're my b forever, in this universe and forever ones as well!
@@FirstLast-ir3wg*beep beep* simp alert!🚨
Hi @CleoAbram, have you ever heard the story of Rolando Pellizza and Ettore Majorana's machine? You might be very interested in this story, which is incredible. The underlying theme would be very current and a video in which you bring the "mystery" to light would certainly be very popular. Thanks for the attention.
(Rino di Stefano is an excellent source from which to start researching)
"Doc are you sure removing my eye is the right way to go?" Doctor: "Yep, I even got a second opinion"
I'm not sure how exactly but if AI can take some of the easier or time consuming jobs away from experts, but still have critical results be reviewed, then it might help with doctor shortages.
I see it differently. Just the opposite. An AI will see what a doctor will miss because they do not get bored, tired, or in a rut after seeing 100s of people with the same problem. The AI will spot the one that is different.
Maybe the AIs drive down the salaries of human "doctors." No one will need to go to medical school to audition with Ultimate Corp. AI for the actor role of "human interface," or in oldspeak "doctor."
Computer Aided Diagnosis (CAD) not the graphic design one. I read a study of how CAD was developed in order to assist patients with endoscopy procedures. Unspecialized nurses were unable to effectively diagnos the differences between benign malignant verse harmful ones. Thus a specialist would need to be requested in order to figure out whether or not they needed extra procedures. This resulted in patients being charged heavy fees to be told that they are fine. CAD was designed so that if the potential malignant was above a certain confidence rating for harmful a specialist would then be requested. Instead of always having an expensive specialist. Saves patients money and hospitals time, specialists also aren't readily available across the world all the time.
🤣😆🤣😆😆" optimistic" science
@@danharold3087 AI is flawed tho, most don't understand not knowing something and will come up with something superficial instead of just saying "I don't know" in which case a real doctor would refer to a specialist
This tends to happen when you train on the entire internet. Data spoilage is a big problem in this case - those questions were definitely online.
Yes let's trust the random TH-cam content on this one, not the purple that actually did the study
@@clerothsun3933 The "purple that actually did the study"?
@@vk2igpeople
@@vk2ig Yes, purple as you know is the most dangerous color and even killed me once...
Just because the list of questions are not freely available online. Whos to say that eye specialist in training who strugled with the questions required for them to pass . Most likely took to reddit to ask for assistance which we know has been scrapped.
If my doctor starts asking an ai for advice I'm out
“These questions aren’t really available on the internet.” How do we know that?
Because they made new and novel questions for this test specifically. It's in the paper. Similar questions will obviously be out there, but if it was just regurgitating it should get just the linguistics right and the core answers wrong. The paper is a very good read if you have the time. I tend to be pretty pesimistic about the future so it's nice when something actually brings some hope.
@@noone-ld7ptLLMs cannot make up answers from scratch. They do NOT understand any subject, leave alone medical. They’re just large databases with voice querying. Thus, these or rephrased questions sure were in the training data
Do you have the link to the paper?
@@noone-ld7pt Do you happen to watch a lot of news?
@@noone-ld7pt but we can't inspect the training data, and if we're talking about models trained on internet-scale data, it's very difficult to synthesize novel questions or questions that are dissimilar enough to examples that are in the training data.
The prompting technique plays a huge role.
Yeah, and chatgpt has the highest likelihood of having polluted data due to popularity (and asam isn't the most trustworthy person either, so they may actively seek out this stuff)
Exactly. The AI was given 4 attempts to get each question right - the doctors were given 1.
I am doing research on something similiar right now! We are developing an AI to detect heart murmurs, in order to help primary care physicians to diagnose patients. I really hope to see a lot of this becoming available in the near future. Not to replace doctors, but to replace unnecessary referrals and concerns for patients.
ok, but were the doctors allowed to look up the answer if they were in doubt. Cause I would expect a doctor to know when and how to lookup things they do not feel sure about. I don't need them to know everything from on the op of there head.
I have very very rarely encountered a doctor who looked stuff up.
Almost all of them operate under severe time constraints when examining a patient.
@@mephisto8101 I have been lucky so see several doctors look up stuff and even ask other doctors opinion. Getting things right the first time can save you a lot of time down the line. But the world aren't perfect, but I would hope experts were better than 80% correct, in there day to day work.
@@mephisto8101Doctors almost always look things up when they don't know something. It's just they don't do it with the patient as often. I'm sure you've noticed how you will see a nurse first and give all of your info and vitals. That's when the doctor takes time to look things up if needed.
@@alistairgrey5089 The point is the AI does not have to look things up. It is not rushed for time. It will go as deep as it needs to every time. AI don't discriminate against people as doctors do.
@@danharold3087 AI can absolutely discriminate. It all depends on who writes the program, how it's written, what data sets it has access to, etc. You think an AI can't learn discrimination from the internet just as easily as it can learn anything else?
Lets play the Is this hallucinations or not 😂
You won't know until it's too late!
The hallucination problem will be solved with higher-quality training data and, most importantly, multi-shot response. Iterative refinement is very important.
@@youdontneedmyrealnamethey forgot the part where “these questions were not readily available on the internet”.
Once AI train on such information, hallucinations are going to be almost nonexistent.
@@iamtorrego hallucinations will never be non existent because the ChatGPT has no concept of right and wrong. It literally doesn't know if the words it is saying are true or not. While better (and more limited) training data will definitely improve it. It still requires plenty of human supervision. Even in coding where the input data is much higher quality, and it is useful, you still actually disregard many answers until you recognise one as what you need. You can just get through them quite quickly. Asking ChatGPT to assist you with something you're not an expert in is a recipe for disaster as you simply won't know how to even determine if its wrong.
@@youdontneedmyrealname ChatGPT and other AI modules are a literal joke, barely one step above a chat support bot.
The thing is, AI is theoretically capable of doing all of these things as a tool. It just isn’t as reliable as it would be to make it viable. Same way you probably wouldn’t feel comfortable sleeping in a self driving car
One major challenger is to have an accurate description about the medical condition that the patient has. The real test of AI is to have AI agent to interact with real patients.
And here, I am exhausting myself, studying for medical residency
I love AI for very specific tasks, like imagery identification. I don't want them to replace, but validate.
This is especially important for the people who have their moles check every year for skin cancer.
Using AI to help with diagnosis and to suggest various treatments is an amazing idea. How many people have died from a doctor prescribing something and being unaware of a drug interaction? An AI could help to avoid those mistakes while giving more detailed analysis for the doctors to look over.
AI can make the same mistake that a doctor can. The advantage is not that it's AI but that it can be executed algorithmically to check against every prescription. Doctor would get bored but AI never gets bored
@@mikhail_simin the AI will use all information available. That's all I meant. If the information is put into the system it cannot be overlooked by an AI the same way or can by a living doctor.
@@alistairgrey5089 it definitely can be overlooked!
@@alistairgrey5089 give any current AI a list of 500 words and ask it what the 325th word it. It will get it wrong
Those tools have existed for decades though as far I know as it's fairly unconplex. My doctor's, and nurse family member, use them. They punch in your meds and check the possible interactions as a double checker before prescribing anything new.
This highlights the need for improvement in testing methods today more than the advancement level of modern LLMs.
It’s crazy as well because it’s only GPT4 imagine how good it will be in just a few years
That's the point a lot of people denigrating this just don't get. It's like mobile devices - compare the capabilities first digital mobile phone with the current ones ... why people can't extend that comparison to other new technologies like AI, EVs, etc, I don't know ... maybe they're too scared of change?
@@vk2ig Yea people are terrified of change. And it's even more profound than the examples you gave because that is mainly hardware. These models, when trained, are software. Which means that when a new one is released they can literally change the entire world over night as long as there is internet.
honestly as someone who has had to suffer through dozens of doctors from multiple specialties for years and never getting anything out of it about a diagnosis for my debilitating chronic condition, let alone treatment, and being called a liar about my very real AND MEASURABLE symptoms to my face over and over again, i think most doctors should be replaced by robots. i would trust an unfeeling machine to care for me more than a human doctor with clear biases and no understanding of the patient experience.
Thas just your own bias, it's unfortunate tou had a bad experience but this is just a foolish way to see things
@@ThermsFriend Idk man, at some point you have to look around at all the people claiming to have experienced the exact same thing (aka everyone i've ever interacted with who had a chronic illness) and acknowledge that something is deeply wrong with the way the medical industry trains and staffs doctors and how they treat "difficult" patients that have been forced into poverty. I sincerely hope you never end up with a chronic condition and have to learn this the hard way like I did.
@@winterx2348 such a backwards mindset instead of wanting to fix the system you would rather turn to ai, do you think it will be better when the ai says you have this condition but you are not covered so too bad, it won't try to argue with insurance to get you treatment because it's just an emotionless robot but sure I guess as long as its not human
@@winterx2348 also you mention bias, but ai has been show to have bias and exhibit racist tendencies time and time again
@@ThermsFriend yes, I know, which tells you just how low I think the bar is for real doctors
Please state the nature of the medical emergency.
knowing and acting are different things. Knowledge alone isn't enough to *BE A DOCTOR*
We already do use AI to help with diagnoses, just not LLMs. AI is used to assist doctors with identifing tumors is things like brain scans, and can identify tumors that doctors would have otherwise missed.
The reason why GPT 4 was able to score so highly on the test is because it uses the most comprehensive training data. Also, they say these questions are not available on the internet but we all know that's not true, at least to some degree.
Someone in the comments said they created the questions for the test from scratch. Of course there will still be crossovers but still
Medical exams are all over the internet, so even if they made it from scratch, it would be easy for the LLM to guess the answers
My doctor has an AI powered prescreening test for macular degeneration, such that a patient can be screened by a nurse with no background in eye medicine because the machine does all the work. It's a fast and relatively cheap way to screen patients who may not otherwise seek out vision care.
I don't want an AI to be doctor.
Well, depending on AI. AI has been used extensively in medicine. But I don't want language learning models to be a doctor, lol.
I do doctors are shit at their jobs
Why not?
If an AI functioning as a doctor is more competent than human doctors, this should mean you will likely get overall better medical service on average from the AI doctor, than the human doctors.
If you have a medical ailment, especially one with potentially life threatening outcome, do you want the most competent doctor working on you, or the least competent doctor available?
Additionally, the AI doctor likely would require no appointments or waiting rooms for a consult, and it may not even charge anything for its service.
An AI doctor is not as likely to be subject to biases due to hubris and other human factors affecting its decision making process. An AI doctor is more likely to be logical and scientific in its assessment of a situation.
An AI doctor has the potential to learn from all of its patients and all historical medical case studies that exist. Human doctors are time and motivation restricted, such that they do not learn everything medical related in all medical fields, that exists. Additionally, human doctors are limited in the number of total patients that they will see in their career, and thus, they are limited in the amount of personal experience that they can acquire. An AI doctor has the potential to multitask and serve billions of patients in billions of cases, which ultimately could lead it to becoming a tremendously better informed expert in all possible human ailments and exceptional medical cases.
AI doctor services are the future of "medicine". Humans should stop paying to attend medical school. Accumulating massive student loans to attend medical school, only to major in a dead end profession, is not really a good use of personal time or economic resources.
@Fritz_Schlunder hallucinations.
Being a lawyer myself, what you mentioned is currently the most interesting option: Using AI not to replace you (too soon) but collecting a second opinion/reducing the likelihood of mistakes.
Until some bot tells you to stick the cheese in pizza with glue 😊
Yea google seems to be lagging significantly behind the state of the art. GPT 4o is pretty amazing tho, and the new frontier model has been anounced to be in training, so I'm pretty damn excited!
Google decided to take to answers from Reddit, The Onion, etc :D
Insurance company manager: "Hey boss! I just learned we can replace all diagnoses with A.I. bots!
“The intention of this was not see if any A.I. took can replace any doctor”
That’s exactly what this study was for. Crazy how even doctors are going to be replaced.
Exactly, people insisted that AI would be an aid to artists and not a replacement, yet some companies are already cutting their graphic designers. The purpose will always be to replace, because that's how they can save/make the most money.
Doctors won't be replaced.
Exam questions are already modified for text/image consumption, for a real patient the doctor needs to ask the questions and do the patient examination them selves.
No, it is not what it was for. There are a 1001 uses for this, and other medical AIs, that are not about replacing doctors.
Well, I guess if we are counting conditions being caught earlier so that they are more effectively and easily treated leading to fewer doctors being needed for more advanced cases and problems--like AI in insulin pumps making better decisions about administering insulin leading to fewer diabetics going to the hospital in DKA or hypoglycemic states, thus fewer doctors needed to treat those things--then maybe. But even then, I doubt it. Doctors won't be replaced by AI for a long, long time.
Doctors will be replaced when, and only when, we have ai in robots who are capable of knowing what an apple is. Give us a century or 5 and it might happen, but thats like, three more definitions of "artifical intelligence" away, at least
These AI models are trained for general uses. Imagine an AI model like gpt4 fine tuned specifically to be an eye doctor…it could potentially exceed the expert doctors.
Yeah you'll have AI and Google as a Dr. suggesting stage 10 double cancer for mild fever symptoms 💀
Actually an AI could be better than doctors that specialize.
@@danharold3087Not for as long as it requires active input from a human being.
The AI will hallucinate and remove your kidney without actually needing to.
People forget AI doesn't have critical thinking...to be fair, many people don't have it as well, maybe that's why they don't mind AI not having it
@@auricia201 People are confusing LLMs with what genuine artificial intelligence may be.
Recently, in the US, a doctor removed a liver when he was supposed to be removing a spleen.
I don't think AI would have mistaken these two organs.
"Oh no, the patient is going into shock! What do I do, GPT-5?"
"Sorry, but as an artificial intelligence I am unable to provide an answer to your question. If you have any other questions, feel free to ask! 😊"
Can't wait for doctor gpt to pull the "it's all in your head"
Its all fun and games till your "AI" doctor starts to hallucinate.
Real doctors can hallucinate too :D Better use a combo of both, the actual doc can decide if the AI is helpful or not
I think this says a lot more about how we test knowledge than anything else
Hey this is great, now insurance companies can replace doctors in training with AI, increasing unemployment, and making medical care even further out of reach from even more people! Super awesome news!!!
Or someone can make a medical AI available freely on the net, and everyone can just access it, describe their symptoms, and the medical AI can tell them whether or not they need to seek further treatment.
I swear some people just want to see negatives everywhere
@@ab51849When it comes to change, many people are quite conservative and reactionary. They're scared of the actual or perceived impact on themselves.
The same thing has happened with any significant technological change in the past (e.g. when the motor car came along and started replacing the horse as the preferred method of transport). We see it now with EVs replacing ICE vehicles, 5G telecommunications replacing 4G, "green" electricity replacing fossil-fuel generated electricity, etc.
Cleo, the thing is that those IA listed are language models, meaning that their cost function is optimizing "placing one word after another while keeping the phrase grammatically correct".
Thus GPT 4 performance can be caused by choosing the option which answers the question the most coherent way in a grammatical point of view, which is a great way to guess an answer.
That doesn't mean there will be no expert AIs helping professionals, but they are not those we praise today...
Would you rather have an ok, not great doctor helping you or an ok, not great doctor who uses AI as a tool helping you? AI doesn't have to replace or be better than all doctors to be useful.
I'd rather have a doctor who had to earn their job with authentic intelligence, and isn't dependent on the AI. The last thing I'd want to see, is a doctor blindly trusting an AI hallucination.
Oh yeah, for sure, “We’re not trying to replace any doctors…. Yet.”
Yes, I want see more optimistic science and tech stories.
There are a couple of catches with actually using this technology. AI does not 'think' on it's own so is not really intelligence. It does crunch data and give answers that seem human to us (mostly). When AI does make a mistake or do one it isn't just a little off, it is straight forwardly weird. Finally, If AI does actual doctor work, it will not be able to doubt itself, know there are limits, consult with others who know a procedure might have an uncertain outcome, not will if fear accountability because for it, doing harm can not have consequence.
I like it like I like google. I think AI is a good tool to supplement human education and abilities, not replace or supersede them.
When seeking a diagnosis based on symptoms, I don't need the medical professional to "think". I need it to match symptoms to possible causes and spit out a diagnosis. Why would an AI not be able to doubt itself or consult with others? There's nothing inherent about an AI that would prevent that.
The argument of whether or not it is intelligence is honestly quite silly to me. If it consistently gives intelligent answers why are people so adamant that it's not intelligence?
Also, with the right prompts LLMs are actually remarkably good at relflecting on their own answers, and listing options and suggesting improvements and resources. I don't care whether or not that is "doubt" per se, if it has the same effect. And I also don't see why it won't be able to consult with other experts givent that the mixture of experts architecture itself that they are built on does this inherently.
The accountability issue is a real one, but that is also a very human motivational factor. Accountability and consequences is inherently a variable, some people care more, some situations puts enormous preassure. Eliminating it could actually be beneficial to more consistent results.
I am a Doctor. We need these AI tools already. No they will not replace doctors. I understood this much more after I became a doctor
Actual ai tools are already being used 😭😭 LLMs are just gonna replace y'all bc they'll be cheaper, even if they suck
yk giving something all the information in the world would show it's rly smart
ngl with how negligent some doctors are, this is a better option for many people, especially in places that they have to wait months to even get a visit from a specialist
Aww she's so sweet. They will absolutely replace doctors. And not be influenced by the AMA and not be looking to make a quick buck off you either.
Cleo cool video, but I don't understand why everyone has this insistence on claiming AI is just a supporting tool. That's not optimism to me, just a misunderstand of computer scaling and society. Sure, right now GPT 4 is only as good as barely expert doctors. So of course we'll go to a human for treatment. But what happens when GPT 5 can beat every doctor on that list? What about when GPT 6 or 7 is has an error rate 10 times lower than the experts? Are we really going to pretend we're going to be ok with doctors just using that as an aide in their decision making? This, of course, happening while the LLMs evolve their interactive abilities and become capable of performing surgery, comforting patients, doing all paperwork in seconds, etc. I just don't understand the game everyone is playing in saying these AI will just sort of be a little tool in the (input industry expert) kit.
The trouble with that line of thought is that it assumes that the technology won't plateau in the near future. Most of the concerns about AI replacing workers stems not from present technology, but what the technology MAY become in the future if it keeps up its explosive present growth.
That is simply unlikely. There is only so much information that these Large language models can train on, and most of it has already been used up. So while future versions may be slightly better, I don't think we'll ever see as big an improvement as between gpt-3.5 and gpt-4 ever again.
@@navneetnair3314 Yes, motorcars can help get some load off horse-drawn carriages, but will never be as resilient and reliable as a good horse.
You can give it all the time you want, AI will still not have critical thinking. Its like picking a very good student, one of those who only studies for tests, and is able to get 90-100% by knowing bow to answer tests like questions, but then expect him to perform well when facing real life problema, even the ones related with the subject he was studying.
AI can only repeat what it was done before.
@@amos083 That was never my argument in the first place. I did not say that They CAN'T be as good as human professionals. They very much can if we had enough information to train them. The problem is that we really don't have that much more left.
Motorcars would never have been as reliable and resilient as a horse if there was some physical limitation on engine power and capacity and the cars of the 1890s were the best we could ever do!
In the end, these GPTs are just text prediction models. Feed them enormous quantities of data and they will begin to notice patterns and be able to give general replies to prompts. Run out of data and then that's the end of that.
@@amos083 Well said! :D
The problem is that people aren't looking at the history of motor vehicle development and seeing how the technology improved and bootstrapped itself to the point where barely anyone rides a horse to work, etc. (People are having the same problems with EVs - they claim they'll never buy one because of the CURRENT limitations ... I'd love to have an official register of such claims so that they can be played back to these people in 10 or 20 years time, LOL!)
With medical AI, all that will be needed is a demonstrated ability over time that people are being treated successfully and at lower cost than the US medical system can. People take their health seriously, e.g. they want to catch that cancer now instead of years later when it had metastasised and become terminal. Stem cell treatment will do likewise - people will see how their pets can be treated and say, "Why couldn't you do that for my kid when they broke their arm?" Of course, some conservative societies will ban such practices (e.g. certain US states), but it will succeed in other jurisdictions.
Nope. It’s being used to replace human doctors, nurses, etc. to make more money for the hospitals. Used AI at a major Fortune 500 company in the 1980’s. Should only be used in limited scientific research and applications not directly involving patients.
AI in the 1980 were mostly expert systems. The deep learning AI we have today are different.
@@danharold3087 And that much more dangerous.
@@corvette9675 Maybe. We can paint them as ugly as we like. SyFi has been doing that for several decades. No drama in ones that are saving mankind now is there.
1st, GPT 4 is available for free for anyone, I don't see how you make a conspiracy out of something anyone can use freely.
2. Wild comparison. The transformer model paper that these models are built on was written in 2017. Nothing you had in the 1980s is even remotely close to the same thing.
Yup. Why I won’t be subbing an ‘optimistic’ science channel
And now you’ve given AI that test the next time it will beat the doctors.
What happens when the actual DRs get lazy and depend on the Ai doc rather than their own experience?
Good question. This is exactly what is coming down the road. The AIs will eventually drive down human "doctors" salaries to nothing. And no human will want to be a doctor because there is no point in it. People will get dumber and dumber.
plus ai has the problem of not distinguishing normal variations in anatomy or physiology; asking it questions on a test is a lot different than asking it to read and interpret scans or tests
for example, i might know that my dehydrated patient's red blood cell count and potassium is low bc i just gave them iv fluids, so the anemia doesn't need to be treated; i actually just put this into chatgpt and got a whole workup which is wholly unnecessary
@@eretheah Please do not judge 'AI's by what LLMs are doing. They are not artificial intelligences.
Then hopefully that graph will already have far surpassed even the best doctors so it will actually be morally right for doctors to defer to it. With how far away this was last year and the scaling that we've yet to see the effects of I strongly believe this will happen within a few years.
@@danharold3087 Intelligence just means if you input data on an organism you get back coherent output from the organism. So yes LLMs are intelligent because they do give back coherent output.
This is a reflection on doctors. The standard of care and accuracy we get from them these days is on par with a mindless machine. No joke.
Most people don't understand computing power doubles every 18 months. That means in 5 years, AI will be 8x as powerful as today. In 10 years, AI will be approximately 128x as powerful as today. This means it could very well perform better than humans, and faster, and cheaper, and the Laws of Capitalism mean owners will replace as many humans as possible with AI. The problem being that unemployed humans do not buy goods or services, so the house of cards of Capitalism will collapse. This worries me.
What you don't understand is that computing power is not the restraining factor here.
Software itself needs to evolve.
You don't need supercomputers for lots of the AI models. The software needs to be more intelligent. Data needs to be available and correct to train models.
This is more or less independent of computing power right now.
I have a couple of issues with your statement:
What you're referring to is "Moore's law" and technically it doesn't state that computing power doubles, just that the amount of transistors on a chip doubles in said ~18 months time span (roughly). Meaning on the same amount of chip space as before, not that the chip is getting bigger, which results in the transistors becoming smaller. Of course this results in more compute power, but it does not necessarily mean that it scales linearly with transistors getting smaller and denser. Besides that, Moore's law has been stagnating for a few years now, because there are actual physical limits on how small a transistor can be (we're not there yet, but it's getting a lot harder). At such a low scale, you have to consider a lot more factors, than say on a rudimentary chip on a bread board (just for comparison's sake, but the idea is the same for small and really small chips). At the tiny scale, you'll suddenly have to think about the speed of light and how fast your signals can actually physically travel across the chip, whereas on a bread board scale, that's pretty much irrelevant. Anyway, my point is, computer chips are hard enough to manufacture as they are and making them smaller (and faster and more efficient and more powerful) has its limits and gets a lot harder at some point. Hence, the stagnation of Moore's law in recent years.
Then there is the point that twice the compute power, doesn't necessarily mean the thing will run twice as fast in real applications. If you take a problem and try to solve it with an inefficient algorithm, but throw a lot of compute at it, that might work. But you could also use a lot less compute and solve it with an efficient algorithm. Or solve much much larger problems in the same time, with the same compute.
So to recap:
Large language models (which you refer to as "AI" here, which isn't incorrect, but also nowhere near the whole of it, AI is much much more than LLMs) will almost certainly become more powerful in function in the coming years, because these models are currently able to solve problems that other algorithms/models couldn't solve before and experience a lot of breakthroughs at the moment, which in turn makes the research and development money fly their way. This next part is a bit of a personal opinion. I believe that at some point LLMs won't experience these massive breakthroughs anymore and then the focus will shift to some other technology. Maybe they will have replaced a lot of jobs by then, who knows. There will certainly be at least a few. But there are also going to be enough problems that LLMs won't be able to solve, because they weren't designed to solve them in the first place, no matter the supposed resources they run on. Try having GPT-4 drive a car autonomously, it won't work; completely different problem space (or maybe it can be made to work somehow! I highly doubt it, but I'll be happy to eat my words, if someone manages to prove me wrong). And I hope I made clear why "AI isn't going to be 128x more powerful in 10 years" and have not lost the point in all this. :D
Feel free to ask follow up questions, if I failed to do that and my point is unclear. Cheers :)
To clarify my point, it's not the technology that matters. What matters is that under Capitalism, humans are expendable resources, therefore Owners will replace humans with automated systems whenever possible. As pointed out in the video, certain software can already compete very well with humans, therefore, someone will say "Why do I need an expensive human, when a cheaper automated system can do the job?"
Look at your grocery store - how many humans have been replaced by some flavor of either self-service machine or automatic tally systems?
Sony and Disney execs are already talking about replacing thousands of humans with computerized systems for writing, animating and voicing.
Owners will displace humans.
As far as technology goes, remember that only 24 months ago (2022), "AI" (by any other name) wasn't taken seriously outside a few nerd groups. Now *everyone* can benefit from it. The tech curve may not match Moore's Law, but that's simply hair-splitting. It's evolving geometrically fast enough. Even if we say it "only" doubles every 36 months, in 10 years it will be 8x more powerful. Eight times a billion is Eight billion, which is seven billion more than "AI" needed to figure out how to draw fingers correctly for some hack typing strings into a keyboard. (I could as easily say "8 times a potato is 7 potatoes more than necessary, so don't get hung up on the numbers.)
But, to repeat, the main social disruptive factor is not technical, it's usage, and the flaws of our economic models.
@@PatricRogers I understand your point somewhat. But most of the people in my circle work in jobs which did not exist a hundred years ago. My wife and I work in jobs which did not exist 20 years ago. I can't count the times we tried to explain our respective parents what we do for a living.
It's not rocket science, but the previous generation has not a big grasp on it.
Yes, jobs will vanish in the future. Like they had in the past. How many people do you see service horses? How many people are operating elevators? You can nowaday operate large oil tankers with a dozen crew instead of a hundred. And no one is knocking on your window to wake you up. No one is bringing big ice blocks to your cellar.
Point is, the job market is always in motion, always has been. AI will take some jobs, others will be created.
It's not as simple as AI replacing all of us. In my job, my colleagues work closely with companies aiming to bring AI, LLM and neuronal networks into the shipping industry. It is not as easy as it sounds and lots of people need to train the model and check for errors.
What concerns me more is the shift of money from lower and middle class to the ultra rich for the past couple of decades. This has direct effects for most of us.
@@mephisto8101 good points well said
A lot of industries haved used AI and learning models for a long time to work on hyper specific tasks. Now since the AI bubble has been blown up we’ve put that focused lense on it and everything has to be labeled “AI”. 4 years ago it would’ve been business as usual.
I'd rather be a patient and get treated by jarvis than study medicine and have no job
"What is my diagnosis?"
"You are a plant"
"Got it"
I read an article about a lady who was ill with strange symptoms and she went from doctor to doctor and got no clear answer. Exasperated she put the symptoms into an AI tool and it came back that it was a very rare endocrine disorder. She then went to an endocrine specialist and lo and behold, the AI tool was right. I see this as something that will be used more and more for rare disease diagnoses.
Google AI: I’ve been telling you for years that you’re dying. 😅
Ai may not be able to accurately replace diagnostictians but their ability to congregate information quickly can make their application in the medical field more time efficient reducing strain on medical health care professionals
I also want to point out the level of training and expertise for UK eye doctors is different to that of America's (in terms of optometry). America has the most comprehensive, intensive, and medically based program in the world with only Canada having similar (but less medical in practice) jurisdiction after their training. So, using UK eye doctors (depending on how expert training was defined) is already skewing the results a bit.
Most people don’t seem to realize doctors look stuff up constantly about every aspect of treatment, conditions, etc. AI helping would just give them faster answers, not replace them.
It’s pretty much impossible for most people to not interpret this as AI being able to replace doctors… and I’m sure some news sources will take great advantage of this.
non language model AI's are also being used in procedures such as colonoscopies to spot potential cancerous areas early. It even spans to things like pancreatic cancer- the Mayo Clinic has an article about it but the AI was able to detect pancreatic cancer up to 5 years before it was even visible in traditional imaging. Pretty cool stuff, and to think we're only at the beginning stages of AI...
The questions not being available for training data means the ai knows without memorization.
Thanks for all the disclaimers, appreciate the journalistic rigor, integrity, and responsibility from you guys
Bruh, if doctors get help from AI, then I'm not paying for it.
you pay for doctor?
Congrats on a great conclusion here, most of the time is "AI should replace all humans", or "AI-phobia", but you sum it up perfectly
There is a big difference between a standardized test and a real-world scenario.
It's a great tool to help medical professionals but it will never replace the intuition of an actual human.
Growing up, my sister had a condition for well over a year, that our ENT could not figure out what was causing it- it was like a blocked saliva duct but nothing that should have helped was working. She was actually scheduled for surgery to have that duct removed, but he just so happened to bring her case up at a conference for help. One specific doctor knew exactly what it was, and it was something frequently seen among people who share her Alaskan native heritage. I think it will be so useful for helping in diagnostics and for sharing medical information, and so many people will get answers that otherwise might have taken years to get
I would note as a doctor that eye pathology is pretty straight forward mostly and that the imaging is very easy to interpret using algorithms. That's not what being a doctor is anyway.
I'd like to see an AI discuss why it doesn't feel resuscitating your Mother is viable or tell you that you have incurable cancer.
It's like Nuivelette using the O'ratrice Mechanique D'enalisé Cardinale
I love how they're like don't worry, we're not trying to replace you, we're just trying to see if AI is as good or better than you... 😂
Love seeing Froslass put in _work!_
I would have sacrificed Avalug at the end instead of Froslass, but it got the job done
The difference is that the language models basically have the entire internet at their fingertips at all times. If the doctors were able to take the test under the same conditions as the AI (with access to the internet) they'd do a whole lot better. The tests are also basic facts and textbook cases instead of the complex real world cases that real patients present, so the exams play to the language models' strengths (regurgitating well established facts) while the real world practice of medicine plays to the language models' weakness (actual critical reasoning skills).
Imagine a world where doctors give a shit and ACTUALLY do research to diagnose someone
We've used computerized diagnostic tools in psychology for years. They routinely out perform phds.
The part that is left out here is that the AI were allowed to answer each question 4 times, each attempted answer was reviewed until they either made 4 attempts or got the correct answer as reviewed by a an expert. The Doctors were not offered this kind of feedback. Then as far as I can understand the study, they just ignore how many attempts it took. So there isn't any data on how many attempts on average each question took to get right for example. They keep referring to the AIs reasoning ability, but I will remind everyone that ChatGPT cannot reason, it is not designed too. It can answer with statistically probabil words based on prompt input. So the test is absolutely wild on that front. I don't know why people keep forgetting that. At least if a real doctor gets it wrong, they're not likely to recommend something actually harmful or imagined. You have no gaurantee of that with an AI.
The problem with those types of tests is the limited amount of variable and how much it's based on information recall, which is basically what GPTs do best. Once you throw complex scenarios with unrestrained possibilities a type of abstract problem solving ability that the brain has and GPTs don't becomes necessary. The 3Blue1Brown video on GPTs explains very well how GPTs basically map words to probabilities, not concepts
I work at a vets and we have and AI that looks at every x-ray we take. It's not used for any diagnosis and is very sensitive to positioning and the quality of the x-ray but it can help to draw attention to things which may not be as obvious.
We only take multiple choice test to become doctors. Our joke is “well we’re grad there’s only ever possibly 5 things wrong with a patient.”
The hard part of medicine is knowing what questions to ask to form a diagnosis.
You can be sure American medical insurance companies are already prepping the "Chat GPT coverage" where you can't see an actual doctor until you've exhausted all the suggestions an AI offers first.
Radiologists in Denmark use AI to help look through breast cancer screenings, and flag scans that need a human to look at them.
Now imagine five years from now when the AI models outscore all of the human doctors ... and ten years from now when the AI models outscore all humans.
“They aren’t available on the internet” >> proceeds to show the questionnaire
Lol
“Not to replace doctors with AI” my butt! I’m sure that wasn’t the direct purpose of this study right now but there will be an ongoing long-term effort to replace doctors with AI unfortunately. The more advanced AI gets the more pressure there will be. The AMA will keep the pressure at bay for awhile, they still have a lot of power
The thing is we already have specialized medical algorithms for detecting spcific conditions in x-rays, mri scans and other data points. They're more useful than GLMs for this, so are what shouldve been tested in this study.
A Standard "Theoretical" Test-It should be highlighted in bold
The issue I know of with diagnostic AI is that they are a "black box" program-it's given starting data to train on as well as a list of possible diagnoses, then finds an algorithm that best matches the known data to the possible diagnoses. It isn't initially known how each algorithm was built, however, and they aren't necessarily logical-for instance, a medical AI designed to spot lung cancer might do so more accurately than human doctors, but it might be making an assumption based on what machine & facility the lung scans were taken from, which is not how you actually diagnose lung cancer and would exclude many patients with lung cancer who got scans from elsewhere.
Mathematically these things work, but in reality they are not able to make a proper diagnosis.
Idiocracy is becoming realer every day, we gonna become reliant on AI
I think the key word here is "specialized AI" which means that these new AIs that are to be used in this capacity are exclusively trained in their specific specialty. AI can be useful as tool, it is not meant to replace humans. If an AI can help overworked doctors with repetitive tasks/notes etc, so that doctors can be more efficient and spend more time with patients, then it would be incredibly useful.
There's more to medicine than coming up with a diagnosis. 90% of it is interpersonal skills. And NONE of it matters without the resources or funding to keep healthcare and social services running.
There’s a company doing this for veterinarians called VEA. The goal is to streamline routine cases and let the vets focus on more specialized cases
This is what AI should be used for. Not for stealing art
I would love to see AI be used as an early screening tool. The AI has time to interview the patient and ask a lot of questions, then the AI could give the doctor, who has very limited time with the patient, a prioritized list of symptoms and potential diagnosis/treatment. The doc can approve the results or recommend something different.
AI does best at very limited tasks. Like identifying a very specific tumor type in images. At that, it can be better than experts. Large language models are based on general internet information, not specialized tasks, they will be less accurate than expert systems.