MORE Emergency Mailbag: The USMLE Cheating Scandal, Part II
ฝัง
- เผยแพร่เมื่อ 16 ก.ย. 2024
- Yesterday, I put out a video answering questions about the USMLE cheating scandal. And no sooner than I did, my inbox and DMs filled up again with more questions, necessitating a historically unprecedented second emergency mailbag.
If you missed Part 1, catch up here first:
EMERGENCY MAILBAG: The USMLE Cheating Scandal
• EMERGENCY MAILBAG: The...
…and be sure to also listen to Part 3:
Mailbag: Wrapping Up the USMLE Cheating Scandal
• Mailbag: Wrapping Up t...
_
NOTES:
Breaking the magic: how the USMLE three-digit score is calculated
• Breaking the magic: ho...
Why does the USMLE use experimental items?
thesheriffofso...
MUSIC:
‘Clutch,’ by Nicolas Major.
I find it commendable that cheaters are being identified. I firmly believe that individuals who resort to cheating in their exams should not be entrusted with the responsibility of treating patients. In the field of medicine, ethics play a pivotal role, and my stance is that those lacking ethical standards should not be granted a license to practice.
Thank you - and I agree. I’m really shocked that so many people - especially on X - seem to be missing that point.
100%
Security breach more likely at testing centers outside US
💯
Im a US DO graduate (now EM attending in Virginia). I got a 260 in Step 1 and 264 on step 2 genuinely, without any recall banks or cheating whatsoever. In the back of my mind I had a very slight worry that they could target me, but hearing this about how they identified the cheaters takes that away for me. Thanks for the great video.
I don’t think you have anything to worry about. If you haven’t seen it already, I broke down the methods the NBME acknowledged using (which came out in a lawsuit that one of the examinees filed after her score was invalidated):
Mailbag: Wrapping Up the USMLE Cheating Scandal
th-cam.com/video/MR1hzK-hu34/w-d-xo.html
I've seen PD's who run residencies with lots of IMG's just brush this whole issue off. You can clearly tell they're coping. It's good that you're covering all this!
Thank you for the kind words - and for sharing the story.
Numbers never lie. Honestly, I’m an IMG with low to average scores and I’ve been struggling because of them. And it makes me feel like crap whenever I see on Twitter someone sharing their 270+ scores. I know most of them are extremely smart and studied their ass for it. And even the people that cheated, if they got a 270 cheating, they might have gotten a 240-250 not cheating, which is still better than my scores. But to know that a person who thinks that they can get ahead like that in life are getting more opportunities than people like me, who might not be the brightest, but who work hard to compensate for just ok scores, leaves a bad taste in my mouth. And even worse, other IMGs with honest high scores might loose opportunities because of the cheaters. People really disappoint me, honestly.
As an IMG with low ck score who did not match I get it. More baffling thing is ppl who were invalidated have another chance to take the exam. But for someone with low score it almost seems like a inescapable crime u never commited and can never get rid off. Like being born into a lower caste or wrong ethnicity. There is no redeeming no matter wat and that's the saddest part in all this.
Your worth and your value is beyond your test scores. I understand that that might not be adequate to getting you to where you want to be but always remember you are always so much more than your test score. Good luck and I hope you don't give up and are able to practice still
Great video as always. I think sports are always a fitting analogy for medical education; lots of folks in their 20s-30s who are cutthroat and push themselves to unreasonable limits for prestige, at times pushing their peers down to climb over the top of them. One consideration I have with this USMLE crackdown is the contrast between obvious cheating and gradual floor-raising.
In pretty much every sport, the combination of advances in nutrition and fitness coupled with early specialization, strategic optimization by coaches and front offices, and rules changes have all led to records being pushed or broken across the board. In football, LBs and DEs are massive compared to when Staubach was in the pocket, yet they somehow run 4.7 forty times. In the Olympics, gold medalist track and field stars or weightlifters of even 3-4 decades ago wouldn't sniff the podium today, in a sport measuring purely objective times or lifts. The steroid era in baseball was a brow-furrowing and head-widening inflection point in what is otherwise a clear trend of performance increasing with time.
As some of your other videos and writings have pointed out, USMLE scores have risen every year because of several factors including better test prep resources and students focusing on STEP performance, sometimes to the detriment of their broader medical education. The situation in the Nepal is another inflection point, a too-steep improvement in performance relative to the gradual upward curve for the rest of the test takers. But like you mentioned in here, test prep resources incorporate recalls to some extent in their resources. I worked for a test prep company before medical school, and there were a small number of employees who were sent in to exams to scout: everything from recalling specific questions and passage details to counting the number of words in MCAT verbal/CARS passages.
As long as the field keeps improving every year and test prep resources become the de facto requirement for success, the biggest losers will always be the students who have to shoulder the financial burden of test prep through rising tuition or paying for outside resources, as well as the patients whose doctors spent the first three years of their medical education in "this is a waste of time if it's not on Boards" mode.
Great points. Thank you.
Hey man love the content! In many of your videos there is high pitched humming going on in the background (including this one). When I listen with headphones the problem is even more pronounced and it takes away from what you are doing. Just letting you know, keep up the good work!
"Math is real and numbers don't lie" Bryan carmody
If you’re interested in hearing the dissenting opinion on that statement, swing by my mentions on X and enjoy the strained logic and math nihilism. But like I said, regardless of whether you believe in it or not, math exists, and regardless of whether you understand it, some people do.
@@sheriffofsodium sir i just tweeted u the genius after watching this . This video will go down in history.
Appreciate both of your videos discussing this topic. Really nice to have some clear explanation especially when there's a lot of conspiracy floating around.
I thought my eyes were deceiving me! A part 2?!
Not how I’d planned to spend my Saturday night… but duty called.
@@sheriffofsodium it's much appreciated!
I am a doctor from Nepal and I haven't even applied for USMLE but I am getting second hand embarrassment. I know many of my friends who study day and night putting their health, social life on line for these exam but just because of some cheaters all those almost to waste. I just hope those who really prepared honestly for this exam doesn't get gobbled up in this.
It's easy. Take it in the US, and doubts of cheating goes away
@@TomatoesNut Wrong. Even STEP 3 was compromised and you can only take it in the U.S. These recalls are not as low quality as people think they are.
You know it better than anyone if they were honest period! Look in the mirror and ask your self were they honest?
R u sure they aren’t studying the question bank day and night
Yeah keep telling yourself... they are studying hard for cheating!!!
Powerful insights. Thanks Dr. Brian
Thank you for taking the time to watch.
Thank you so much for your videos on this matter. We are an IMG friendly program so this development affects us. My comment is not necessarily related to the topic, I agree with everything you said but wanted to give my 2 cents on your guess that Step 2 would likely become a Pass/Fail score. I believe the fact that the Step 1 score reporting has moved to that strategy is already rather hurting applicants. In the absence of valuable feedback, we focus on grades or scores. Unfortunately, more and more MSPE's and Dean's letters no longer provide a class ranking and some of the med schools still doing it base it (as well as the AOA status) mostly on pre-clinical performance. That leaves a lot of us involved in this process trying to guess if the people we bring in the program will end up being the solid physicians we hope to serve our communities (and ourselves) tomorrow.
This might sound weird, but can the USMLE also see time elapsed per question? So if a cheater is flying through compromised questions in just a few seconds each compared to spending 2-3 mins on a fresh new question, it becomes more obvious.
I also thought this would be a way to crack down on cheating
Great question - and yes, they can and do analyze those data. In the MCAT/spy glasses story I mentioned in Part 1, the fact that the examinee answered questions so quickly was one of the facts included in the lawsuit to suggest that he was only taking the test to pirate material. For examinees with previous exposure to exam material, you could imagine this coming into play, too. Just like before , you could break up the test into subsets of compromised vs. non-compromised items (or even just old/new items) and see if there’s a substantial difference in time to response. You’d expect that quantity to be similar across question subsets… unless some of the questions were familiar to you.
I love this question because it’s an example of the point I was making around 9:43 about how there are all kinds of statistical comparisons that can be made - in isolation or jointly with other observations - to identify someone who’s cheating.
@@sheriffofsodium it also gives them unfair advantage of extra time per question as they can finish the recalls in (lets say 30 seconds). 10 recalls per block will add to 10 mins of extra time for questions out of their recall pool. It's like they get 55 minutes to solve 30 questions while other get 45.
Good point.
Worth mentioning here that 270 is like…96th percentile. Maybe 95th. So those are the top 5% of performers compared to the standard curve. If you know, out of 20 people, that 5 of them got a 270+? That’s suspicious. Maybe they studied honestly-heck, maybe they encouraged and challenged each other to get higher scores-but it’s still at least suspicious.
I love math and I believe math never lies. Imagine how math helped us know everything about blackholes or any phenomenon going on in Universe which we can never see or experience at all.
IMG should respect US rules and regulations.
Being an IMG I believe US is fair and I am glad that they are doing their best to keep the integrity of the USMLE test.
Thank you for all the thoughtful breakdowns!
Thank you for watching.
It's just the tip of the iceberg. Recalls are not just restricted to one country. It happens worldwide.
I think we’re going to see other crackdowns in other parts of the world very soon.
@@sheriffofsodium i sincerely hope so.
Brilliant analysis doc. Suddenly some people have become math illiterate with their “what ifs” and “buts”. Cheating is not like any other mistake and it’s not something which can be redeemed! It’s an irredeemable blunder. The cheaters must be permanently banned. If I were ECFMG, I wouldn’t be giving this revalidation choice either.
I understand but think offering revalidation is more than fair as the evidence for cheating is strong, but not definitive. It's mathematical probability, but not 100% so offering redemption is fair on the very very small chance someone is falsely accused.
Why was Step 2 CS discontinued? Wasn't it a better metric than USCE/Clinical research to gauge the abilities of an individual?
The simple answer is that it was a casualty of the COVID-19 pandemic. Test centers shut down and it became an increasingly daunting financial proposition to reopen them amidst the public health crisis. But USMLE Step 2 CS was always an unpopular exam for U.S. medical students. I covered the history of the exam - and its cancellation - in a 5-part series on my website a couple of years ago:
thesheriffofsodium.com/usmle-step-2-cs/
I agree with your point, though - an in-person practical examination does provide some protection against ‘recall’-based cheating on written exams.
OET being an independently held exam, they don't have to worry about maintaining testing centers throughout the year. Without Step 2 CS, ECFMG and NBME/USMLE gets to save more/make more money.They don't have to hire a building at 5 centers in US throughout the year and recruit SPs (paid actors) to act as patients. Nor do they have to hire doctors to assess the notes. And with that, you have to pay almost 1000$ to the pathway to get certified by ECFMG and they get to make much more money through that than Step 2 CS. I don't believe the pandemic was the only issue with discontinuation of Step 2 CS. Having taken and passed both Step 2 CS and OET, I can honestly say OET is a joke of a metric to assess clinical skills of a student/doctor. @@sheriffofsodium
Thank you so much. I watched both videos, and I found them to be very insightful and objective. To be frank, I was shocked to learn that the USMLE was potentially compromised. I was unaware of the extent and severity of the problem, and how far medical students and physicians, whom I hold to the highest ethical standard, could go to obtain an unfair advantage & jeopardize patients by not being themselves. The videos were eye-opening, yet didn’t make me too cynical and I learned a lot from them.
Something I’ve been wondering after this news broke- the fact that the cheating was not discovered until someone likely tipped the USMLE off implies that the USMLE does not conduct regular internal audits of exam scores, or at least audits thorough enough to detect cheating backed by statistical power. Do you know if this is the case?
I think, until this point, you’re right. But in the wake off this, I believe we’ll see more routine monitoring. And I hope we do.
What are your thoughts on DO students taking and passing USMLE STEP 1 and STEP 2 CK and should that DO student be effectively treated as an US MD for matching? And, if not, why not? I feel that as a DO student with a comparable and competitive USMLE Step 2 CK score the stigma is still there...
My personal opinion is that we should get rid of COMLEX-USA and have all physicians/residency applicants take the USMLE. I’ve actually got a video that discusses some of that if you’re interested:
On USMLE for DOs - and the Future of the NBOME
th-cam.com/video/WfcZTF6cDiE/w-d-xo.htmlsi=pZjl4IrEF0Vw3-Lj
The Syrian usmle test center back in the day was closed for the same reason. They shared, documented questions, scores were > 250 in a country that English was not even used in medical education ..
Recalls are not remembered exactly. There must be leak in the real paper for them to get exactly score top 100 countries
will this decrease the average usmle step 2 ck score range?
No. I actually covered that question in Part 1:
EMERGENCY MAILBAG: The USMLE Cheating Scandal
th-cam.com/video/A3_HG5uc47I/w-d-xo.html
@@sheriffofsodiumI think the question was less asking about if this affects the curve or scale of scores and more will the average STEP2 score admitting to residencies change. Not sure, but I would guess, say, the average IM STEP2 would decrease if you take matched IMGs who cheated out of the "average score for those who matched into IM". Am I incorrect that IMGs affect that average?
12:50 is the definition of Uworld
Do you think the rampant use of recalls played a part in USMLE board changing the passing score from 208 to 214 ?
Good question - but no. I’ve written before about how the USMLE passing standard is set, and I’ll try to put out a video about that topic this sometime this spring.
I hope PDs get a more detailed emails about the methods used to ensure they don't flag any IMG score as dubious
24:42 could you elaborate more about the breadcrumbs? What did you mean in that bit?
Yes. I think they said “associated with Nepal” because if they didn’t give PDs some idea about who was affected, there would be panic. Specifying the country provided a way of letting programs know that not all of their applicants were impacted.
@@sheriffofsodium
Thank you for the response
I really hope PDs have a very different and more detailed email than us.
Unfortunately, unless the USMLE expands its question bank, I think recalls and cheating will continue to be a thing. All students will need to do is not score as high /get more questions wrong to fly under the USMLE radar--assuming you're correct that they used statistics to prove that cheating was occurring.
Anyways, thank you for your informative video.
Is it possible that in Napal professors were taking exam for medical graduates.
Dr. Hussain
I wonder if the best solution is using AI to change the wording of the questions as a very powerful tool to tackle this and having all exams in the USA. I wonder given 5 standard deviations above average if all the cheaters were actually caught out
I’m confident that NOT all the cheaters got caught. I recently highlighted one applicant’s score report on X. It showed USMLE Step 1 and Step 3 scores that were invalidated, but still listed a Step 2 CK score of 271. It’s pretty hard for me to believe that an examinee who cheated on Step 1 and Step 3 suddenly decided to play it straight on Step 2 CK… and still managed to score above the 95th percentile.
If a person used low-quality recalls and still got a score of 270+, is he also considered a cheater right?
95% of cheater’s scores fall in the confidence interval. As long as p
I knew a Jordanian who could not read the questions but scored 98% on all steps.
How?
I find it unconscionable that any of these implicated individuals, with their 1 in 100 million+ chance of being innocent, would be given a second chance. It is a PRIVILEGE to practice medicine in the U.S. as a non-U.S. citizen. These individuals, with their high likelihood of rampant cheating (and I’d be shocked if it was only Nepalis doing this), are too much of a liability to be allowed to practice in the U.S.
It’s possible that they won’t be. The USMLE could be aggressive in highlighting these applicants to programs and licensing authorities. Or they could just issue transcripts with “Score Not Available” (like they’ve done so far), and then dutifully report the retake scores, and let programs do with that what they will.
I agree that if they cheated they should reassess their life, they should not be practicing here. But to say it is a PRIVILEGE to practice medicine in the U.S. as a non-U.S. citizen, I'm not sure you understand how much work everyone puts through to survive this, and more so for the IMGs who have to be extra competitive. The hours are no joke, and the expectations are the same as a US grad despite not having that advantage of training here from the start. Do not get me wrong it is indeed a PRIVILEGE to take care of patients anywhere in the world, but don't act like we're doing IMGs a favor and throwing them a bone to come here to train. Most earned it through years of hard work, and when they get here they still have to worry about their visas and how long they can be here etc. all while being expected to carry the same load as everyone else. Not to mention the shortage US grads for certain specialties.
US citizens are also privileged because USA gets really hardworking cream candidates from all other nations. Its a give and take equation always !
The problem is usmle not students. Ifykyk. They need to release practice questions that help students not break them mentally and spiritually.
Cricket is a more brainy sport 😂😂. British invented the sport and made it so complex that Americans won't have interest in the sport
I’m afraid I don’t have any extended metaphors about cricket.
Thank you for watching!
Respectfully, If ECFMG performed statistic analysis using the compromised item they obtained from a third party, why was it done only to the Nepali candidates? ( If the argument is, it's because a high percentage of high scorers were Nepalese is without question singling out Nepal by disregarding the low percentages of high scores from other areas and disregarding the fact that high percentages of low or average scorers might be in other cohorts despite using those compromised items. And, if they did it by implying the compromised item was from some group of Nepalese candidates, it is very premature of ECFMG to imply it was from Nepalese candidates without showing concrete evidence, and if there is evidence of such, Where is it? How can it be proved? Why was the name of the country dishonored prematurely? Isn't it preposterous and discriminatory? Also, as ECFMG has already prematurely implied that the items were from Nepalese candidates, how can ECFMG be sure the materials were used only by Nepalese and not worldwide? So, on this basis, every candidate should be statistically analyzed using those compromised items, and until then, Why isn't the whole match process halted to maintain fairness?
If ECFMG obtained exposed exam materials from someone or somewhere, it is offensive to target only a few rather than questioning the credibility of the exam itself, nullifying every exam taken during the years or time frame that ECFMG conducted its investigation with those compromised items. And, in doing so, why not make every candidate around the globe during that period retake credible and uncompromised exams instead of making only a few take their so-called revalidating exam and tainting the qualifications and reputations of few for perpetuity despite being aware of their incompetency to conduct secure and fair exam for everyone?
How about cheaters who, for some special reasons (e.g., had difficulty memorizing answer keys, or intentionally answered multiple items incorrectly to get a less suspicious score), obtained scores around 260 and below? Don’t they have to be investigated and asked to take validation exams as well? How would the USMLE assure that low-scoring cheaters are also removed from the system?
The method I described in Part 1 - analyzing differences in performance on compromised vs. noncompromised test items - works *regardless* of the examinee’s test score. There’s really no explanation for a major difference in performance on these two sets of items… unless you had seen one set before the exam.
Thank you so much, sheriff! Do you think the USMLE will proactively investigate low-scoring cheaters as well, or do you think all efforts will only be focused on those with extremely high scores? I feel like all cheaters should be caught regardless of their scores to uphold the safety and credibility of the US healthcare system.
People cant afford uworld in Nepal, so no way they can afford recalls if they are really sold in markets. I have seen my seniors study for a whole year for one step, then another year for step 2. So , i am really shocked ! There must be something more to truth !!!
They can’t afford word but they can afford to apply for residency? That makes zero sense.
@DeepSeas.. yeah, people take loans for applying to the US or end up almost all their savings. People would buy uworld over recalls. My point is that recalls aren't sold in Nepal. But I never said recalls can't be available !!! Seniors guide juniors over the test. If something more sinister than just guidance is going on, i am unaware of it and i would like that to be figured out as well. Deserving and hardworking students must get justice. Usmle must be a fair exam, because fairness will always motivate the genuine hardworking ones. It's sad how whole country name is tarnished !
@@DeepSeas.. I got updated about new info. Damn, people were as said cheating years ahead than i had imagined. But gain, question were probably sold to other nationalities ? because it was readily available in telegram for nepali doc ? This culture of discussing past question was probably normalized as we practice past questions for Nepali exams and its never considered cheating here because question gets changed in Nepal unlike in usmle !!! People fail national exams here and struggle with the main entrance exams because questions arent repeated. Anyway, its depressing for a USMLE aspirant from Nepal. we will be judged and need to struggle a lot if we want to get an american degree of medicine now ! I am myself shocked everyday with new news !
Why us is not making pass or fail...like plab and mrcp ....and remove attempts ...what u sow that u reap....u made scores as filtering criteria and rat race started since then.....so make pass and fail to remove this ....and stop blaming to exam takers!
Can't tell if this is a bot... considering... no one... types ... like this...
Because if none of them are scored, there's no real objective criteria to rate applicants. Yes, it's flawed, but the alternative is based purely off nepotism and who you know for getting a residency spot. If your point is to advocate for IMGs, making it p/f would make it even worse.
Stop spreading lies. Exam center or pro-metric center had no role in this case. Nepal center is most strict nothing is allowed in the lockers plus there are 16 cameras and usmle has no valid evidence nepali candidates. They even audited the servers several time and found nothing or no security breach. Still nepalis were getting consistent high scores. So what they did was compared data of all candidates and checked there answers and sent blanket email to all those whose answers were similar. This is such bizarre way to prove that you are guilty. These allegations will be challenged in court and court will not accept this without any solid evidence. They have sent email saying unusual test scores, unusual response time and probability of this answer being right is one in a million what kind of stupid reason is that.
He is saying could have been security breach because there is no official notice saying there was any breach. USMLE used math to look who could have cheated based on similarities found among those examinee. Among the top scorers the similarities in incorrect answers and correct answer were found to be very strange which is 1 in a 100 million. So intrepretation made was they had access to questions before exam.
Math can give you knowledge what is in Blackhole or what happened billion years ago in Universe. So math dont lie
USMLE should also look into other countries too very soon
Like I said (11:35), math is real, whether you believe in it or not. And even if you don’t understand math, some people do. If you take a minute to actually think about what the USMLE did here, I’m not sure how you could believe nothing unfair occurred. The numbers don’t lie.
And even if, by some miracle of chance, a totally innocent applicant got swept up in the scandal just by winning the Bad Luck Lotte and getting that 1 in 100 million chance… they get to re-take the exam. So if they didn’t cheat, there’s no big deal - just do it again.
@@AjmatAnsari-rn1vn, you’ve got to remember that the purpose of the USMLE is not to socially stratify applicants into high-paying jobs or to reward people for hard work. It’s to *protect patients.*
State licensing authorities require all physicians - regardless of whether they go to medical school in the U.S. or elsewhere - to pass all three parts of a licensing exam. But if there are legitimate concerns that an examinee cheated on one or more of of those tests, then the licensing authority can’t be certain whether the physician is competent or not.
And in these cases, the concern that cheating occurred is absolutely legitimate. As you saw in the email to one examinee, the odds of such an unusual similarity in responses occurring by chance is less than 1 in 100 MILLION.
So yes, it’s *possible* that an innocent examinee will unfairly have to re-take an exam. And you’re right, that will put that person through some hardship. But that hardship to an individual has to be counterbalanced against the harm that could occur to patients if incompetent examinees are licensed.
And again, the chance that someone who didn’t have access to the exam ahead of time, but just so happened to provide the exact same responses as those who did - the chance of such an innocent person being caught up in this scandal are 100 million to 1. I’ll take those odds every time.
@@sheriffofsodium I agree, every doctors are aware of this , they have taken hippocrate oath during their medical school, patient care is their first priority. why this math calculation for the residents who was ecfmg certified 2 years back? Is not their math calculation working at that time? Its unfair for the innocent one who have not done anything and who give up everything to come this stage, who even one reached at stage to sacrifice own life to do residency here and all this go vain. I humbly request you, can you think for 1 minute staying in their shoes? please let me know . I adore ur every explanation but the step taken by ecfmg , usmle seems injustice because thats a very big step that burns everyone , including innocent one.
@@AjmatAnsari-rn1vn it isn’t a big deal, people don’t just accidentally get flagged as cheaters for no reason. It’s honestly a very forgiving system in that potential cheaters can have a second chance, especially given the fact that honest test takers don’t even get to have that grace. The cheaters should ideally not even be given a second chance since they’ve already made clear their intentions, and the implications from that which could negatively impact patient health outcomes should they be let through.