This isn't the most relevant comment, but the fact that they continually raise the passing score on a criterion-referenced test just to maintain their fail rate is one of the most mind boggling things about the USMLE, and yet almost no one talks about it, so I appreciate you at least mentioning it. It's so transparently profit driven and totally undermines all their high falutin rhetoric about the stated purpose of the exam. I'm impressed with myself that I got that all out without using the phrase "fucked up" even once! I'm a professional, damn it.
27:35 I would even go further as USMLE has changed those types of questions from Increased, Decreased to actual values that students may need to cross check with pulling up the lab values tab and manually scroll thru values. Many student mitigate this by memorizing the reference values for labs to save time.
It is also important to note that internists licensed before 1990 have grandfathered lifetime certifications without any requirement to show understanding of current practices/literature for the subsequent 30 years. Some chose to recertify on their own accord, many did not. Some choose to continue MOC activities, many do not.
I’m astounded at how many times I’ve been told/reminded by faculty “med school isn’t just about passing your boards.” Like, wow, that’s so deep. thank you for your wisdom 🙏
17:34 This is a good point, but also have to say that medical schools do not cover third party resources ever, which also makes access to resources dependent on financial resources as well as access to recent mentorship to tell them what to do/buy EARLY in medical school.
Good video overall, definitely agree that the rise in scores can't just be singled out to one or two blanket excuses, and especially not explained by nonsense explanations like widespread cheating or step 1 going pass/fail. Certainly it's a multifactorial issue that does include better resources and dedicated study time, but also more efficient study strategies, more incentives to score higher, changes in teaching strategies, etc. It's also important to note that the score creep we see in step 2 isn't an isolated event, there are examples of score creep in a number of other academic realms outside of medicine as well. Considering only medical school-related factors likely doesn't cover the whole story either. Ultimately, it would be interesting to look at the underlying reasons that current physicians have these kinds of objections. Sure, there's a small component of pride involved. But anecdotally, my colleagues that I talk to overwhelmingly agree and admire that the newest crop of students have way better CV's than we did. However, if you asked many of us if these students turn out to be any better residents and physicians than we were, you'd get very mixed answers. And I think it really drives into the central question of: How effective are the scores, letters, publication numbers, etc for determining success as a doctor? I'm not sure myself, would be interesting to hear thoughts and if there's data to that. And it may be that mild to moderate swings in step 2 scores among comparative applicants just doesn't matter that much and certainly may raise questions of why this test isn't pass/fail too. Lastly, just a minor quibble about that USMLE question example regarding bulimia. It doesn't detract too from your point much I think because yes it's still a more difficult question than just a direct one liner without a vignette. But it does bear noting that large vignettes do offer more opportunities to get to the right answer, even without knowing the diagnosis or the significance of all the items presented. As in this example, the potassium and bicarbonate levels don't actually need you to know what the diagnosis is, just that there is significant loss from vomiting in the stem. There are some ways that students have learned to use these openings in a large vignette to get to the right answer. And maybe a stronger focus by students on learning the underlying pathogenesis rather than just memorizing a diagnosis have also helped them score better? Again, very very minor point I think and your point still stands for sure.
Thank you for such a high-quality comment. And of course, I agree. I do have a video on the Flynn Effect in IQ testing vs. USMLE Score Creep if you’re interested… it’s here: th-cam.com/video/FHOCuKW8new/w-d-xo.htmlsi=OwZ8x2YXgkvf39tN Agree also with your point about the bulimia question. I probably should have picked a more challenging question as an example - but I decided to go with the first question to avoid the ‘cherry picking’ concern.
There is use of recalls even inside us. US MD students buy recalls from overseas market thats how scores are rising since scores are what matters to match
I wouldn’t be surprised if some US students were using recalls. But I don’t think that’s behind the rise in USMLE scores. That trend has been longstanding and consistent over time. I discuss more about that here if you’re interested: An Update on USMLE Score Creep th-cam.com/video/1gKKAZ5aO8E/w-d-xo.htmlsi=-A4UdIRPgSKsPgqZ
This could not be more wrong. Recalls are not nearly as popular as you think they are in America. Certainly we generally don't need to rely on them to score high.
Do you have any specific thoughts on the increasing focus of ethics, communications, and quality improvement on USMLE? I took step 2 recently and scored very well because these topics happened to be my strengths. Same with the MCAT and psych/soc sections when I was an undergraduate playing to my strengths. Is this by design in order to prioritize/reward a new set of soft skills deemed valuable in future physicians altogether, or just a further push to increase the difficulty of step 2 in an era of score creep. Do you have any specific thoughts in this change in focus away from hard science in general? This has been another topic about standardized testing I was curious about.
I think this is more complicated than a general trend. Since STEP 1 went P/F, more people have arrived in their clinical years with a poorer knowledge base, yet the STEP 2 average still increased. To me, this means that students are studying harder than ever for STEP 2 as they’re managing to score better than ever despite coming in at a greater knowledge deficit. STEP 2 is no longer an afterthought and is quite literally everything, so I think we can expect an acceleration of this score increase in the coming years until people’s collective cognitive skills max out.
I know its the in thing to say its not because of step 1 going pass-fail. But making step 1 pass-fail forced medical schools to optimize their curriculum for high step 2 scores. These new curriculums have shortened preclinical time and have dedicated time to study for step 2. The average score went up for lots of other reasons (better qbanks, memory techniques, videos, etc) but the distribution tightened because everyone is optimizing medical school for one test.
Students game the system. Tons of ANKI, a small fortune spent on Qbanks. But then the game games the system, and what you have is an utterly toxic hamster wheel of unhealthy toxicity. The profession seems to think that a) you must be hazed with misery, pressure ulcers, and crushing debt and b) ignores the fact that memorizing piles of information is largely an absurd goal when everyone has the internet in their pocket.
For an office based practice I think this is a legitimate take. But anatomy and spatial memory is something that is very difficulty to lookup in the middle of a procedure. Online resources can help prep for a case. But once you are in the OR residents have their brain and hopefully a patient mentor.
This isn't the most relevant comment, but the fact that they continually raise the passing score on a criterion-referenced test just to maintain their fail rate is one of the most mind boggling things about the USMLE, and yet almost no one talks about it, so I appreciate you at least mentioning it. It's so transparently profit driven and totally undermines all their high falutin rhetoric about the stated purpose of the exam.
I'm impressed with myself that I got that all out without using the phrase "fucked up" even once! I'm a professional, damn it.
Thanks for watching - and for your outstanding decorum. I’m gonna do a video about the standard setting procedure later this winter.
27:35 I would even go further as USMLE has changed those types of questions from
Increased, Decreased to actual values that students may need to cross check with pulling up the lab values tab and manually scroll thru values. Many student mitigate this by memorizing the reference values for labs to save time.
Also HPI style q’s that take even longer to read
It is also important to note that internists licensed before 1990 have grandfathered lifetime certifications without any requirement to show understanding of current practices/literature for the subsequent 30 years. Some chose to recertify on their own accord, many did not. Some choose to continue MOC activities, many do not.
I’m astounded at how many times I’ve been told/reminded by faculty “med school isn’t just about passing your boards.”
Like, wow, that’s so deep. thank you for your wisdom 🙏
Thank you so much for talking about this! I think a video on how the passing score is set would be beneficial as well.
Thank you for watching! I’ll work on it. Stay tuned.
17:34 This is a good point, but also have to say that medical schools do not cover third party resources ever, which also makes access to resources dependent on financial resources as well as access to recent mentorship to tell them what to do/buy EARLY in medical school.
Good video overall, definitely agree that the rise in scores can't just be singled out to one or two blanket excuses, and especially not explained by nonsense explanations like widespread cheating or step 1 going pass/fail. Certainly it's a multifactorial issue that does include better resources and dedicated study time, but also more efficient study strategies, more incentives to score higher, changes in teaching strategies, etc. It's also important to note that the score creep we see in step 2 isn't an isolated event, there are examples of score creep in a number of other academic realms outside of medicine as well. Considering only medical school-related factors likely doesn't cover the whole story either.
Ultimately, it would be interesting to look at the underlying reasons that current physicians have these kinds of objections. Sure, there's a small component of pride involved. But anecdotally, my colleagues that I talk to overwhelmingly agree and admire that the newest crop of students have way better CV's than we did. However, if you asked many of us if these students turn out to be any better residents and physicians than we were, you'd get very mixed answers. And I think it really drives into the central question of: How effective are the scores, letters, publication numbers, etc for determining success as a doctor? I'm not sure myself, would be interesting to hear thoughts and if there's data to that. And it may be that mild to moderate swings in step 2 scores among comparative applicants just doesn't matter that much and certainly may raise questions of why this test isn't pass/fail too.
Lastly, just a minor quibble about that USMLE question example regarding bulimia. It doesn't detract too from your point much I think because yes it's still a more difficult question than just a direct one liner without a vignette. But it does bear noting that large vignettes do offer more opportunities to get to the right answer, even without knowing the diagnosis or the significance of all the items presented. As in this example, the potassium and bicarbonate levels don't actually need you to know what the diagnosis is, just that there is significant loss from vomiting in the stem. There are some ways that students have learned to use these openings in a large vignette to get to the right answer. And maybe a stronger focus by students on learning the underlying pathogenesis rather than just memorizing a diagnosis have also helped them score better? Again, very very minor point I think and your point still stands for sure.
Thank you for such a high-quality comment. And of course, I agree. I do have a video on the Flynn Effect in IQ testing vs. USMLE Score Creep if you’re interested… it’s here:
th-cam.com/video/FHOCuKW8new/w-d-xo.htmlsi=OwZ8x2YXgkvf39tN
Agree also with your point about the bulimia question. I probably should have picked a more challenging question as an example - but I decided to go with the first question to avoid the ‘cherry picking’ concern.
Thank you for the explanation! I notice that there will be an update on the question pool. Do you know when they update the question pool?
I think the MCAT will also eventually transition to short answer format. R.I.P. my grandchildren
I think you’re right.
If the mcat becomes short answer you’re cooked… that exam is already harder than any USMLE
There is use of recalls even inside us. US MD students buy recalls from overseas market thats how scores are rising since scores are what matters to match
I wouldn’t be surprised if some US students were using recalls. But I don’t think that’s behind the rise in USMLE scores. That trend has been longstanding and consistent over time. I discuss more about that here if you’re interested:
An Update on USMLE Score Creep
th-cam.com/video/1gKKAZ5aO8E/w-d-xo.htmlsi=-A4UdIRPgSKsPgqZ
This could not be more wrong. Recalls are not nearly as popular as you think they are in America. Certainly we generally don't need to rely on them to score high.
Do you have any specific thoughts on the increasing focus of ethics, communications, and quality improvement on USMLE? I took step 2 recently and scored very well because these topics happened to be my strengths. Same with the MCAT and psych/soc sections when I was an undergraduate playing to my strengths. Is this by design in order to prioritize/reward a new set of soft skills deemed valuable in future physicians altogether, or just a further push to increase the difficulty of step 2 in an era of score creep. Do you have any specific thoughts in this change in focus away from hard science in general? This has been another topic about standardized testing I was curious about.
I think this is more complicated than a general trend. Since STEP 1 went P/F, more people have arrived in their clinical years with a poorer knowledge base, yet the STEP 2 average still increased. To me, this means that students are studying harder than ever for STEP 2 as they’re managing to score better than ever despite coming in at a greater knowledge deficit. STEP 2 is no longer an afterthought and is quite literally everything, so I think we can expect an acceleration of this score increase in the coming years until people’s collective cognitive skills max out.
Thanks for the data
Does this apply to the MCAT as well?
I think its the rise in youtube videos. The information is easyer to get to
I know its the in thing to say its not because of step 1 going pass-fail. But making step 1 pass-fail forced medical schools to optimize their curriculum for high step 2 scores. These new curriculums have shortened preclinical time and have dedicated time to study for step 2. The average score went up for lots of other reasons (better qbanks, memory techniques, videos, etc) but the distribution tightened because everyone is optimizing medical school for one test.
3:21 Scoreflation
Don’t forget the recalls in overseas asian countries ! That plays a big role in increasing the mean of the score.
I love your work, BUT your intro and outro music is TOO loud.
Check out the more recent videos and see what you think.
was the answer A 😂
Students game the system. Tons of ANKI, a small fortune spent on Qbanks. But then the game games the system, and what you have is an utterly toxic hamster wheel of unhealthy toxicity. The profession seems to think that a) you must be hazed with misery, pressure ulcers, and crushing debt and b) ignores the fact that memorizing piles of information is largely an absurd goal when everyone has the internet in their pocket.
For an office based practice I think this is a legitimate take. But anatomy and spatial memory is something that is very difficulty to lookup in the middle of a procedure. Online resources can help prep for a case. But once you are in the OR residents have their brain and hopefully a patient mentor.