How To Catch A Cheater With Math
ฝัง
- เผยแพร่เมื่อ 24 มิ.ย. 2022
- Try catching cheaters yourself: primerlearning.org/
Support these videos on Patreon: / primerlearning
Plush blobs and other stuff: store.dftba.com/collections/p...
Binomial probability example (the whole section on Khan Academy may be helpful)
www.khanacademy.org/math/stat...
For discussion and updates
- Discord: / discord
- Twitter: @primerlearning
- Reddit: r/primerlearning
Made with Unity and Manim
github.com/Helpsypoo/PrimerUnity
www.manim.community/
Made possible by support through Patreon:
Anthony Eufemio
Jon Mundle
Zachariah Richard Fournier
Vladimir Duchenchuk
Spline
Christian Gruber
Kairui Wang
Luc Cedric R.
Erik Broeders
Anders Fjeldvær
Jeremy
Guguke
Shayn Osborn
Brian Cloutier
Eric Helps
Sean Barker
Stevie Hryciw
Ben Kamens
Tim Barber
Christy Serbus
Ghost Goat
Sam Shaw
Rikard Eide
Sam Van Cise
Kevin Holesh
Jason Prado
Jeff
Garrett
Daniel Kjellevold Steinsland
Alex Garber
Krum Valkov
Matthäus Pawelczyk
OuroCat
Jacob Luedecke
ketura
Gabriele Siino
Kevo
abledbody
Alba Caparros-Roissard
Jeff Linse
Timothy Furman
Andrew Lang
Flavio Kindler
John Shaffstall
Bez Noferesti
I love the idea that these blobs are dull enough to be entertained by flipping coins, but clever enough to cheat at it.
why would coin flips be dull?
@@NoNameAtAll2 It depends on how many repetions and individual tolerance.
Every time they get heads they eat 1 mango
You should meet the Australian blobs.
@@pablopereyra7126 Damn mangoers, inserting themselves into every conversation. I bet you like those simple, modern houses too, simpleton.
Imagine being the 3% of cheaters that had a weighted coin and still got only tails
skill issue
luck issue
NelsonMuntzhaha.jpg
get a better gaming chair
@@ozan____ *get (a) better (gaming c)hair
Given that the parameters are the game are: "The blob who flips heads feels happy," it might be possible that there are actually secret (deviant) tail-lovers amongst the blobs, who routinely cheat to "lose" more than expected.
blobdsm
Blobdsm
Blobdsm
BlobDSM: flip me harder
@@anastasiaklyuch2746 i regret so much
Flip 0 times and don’t mark anyone as a cheater. This takes the least coin flips, has 0 false positives, and only gives a false negative if someone is cheating.
The problem here is that the test needs to be the bottom line of whether or not someone is cheating in the game. If your test doesn't actually catch cheaters, ever, then they're gonna go back to cheating, go into the test, be declared not a cheater and they go back to being a cheater. Ad infinitum
@@callumbreton8930 well yea but it satisfies the guidelines so B)
@@ieatgarbage8771 Did you forget the guideline where you want to catch 80% of cheaters?
@@isaiahrosner3780 I haven't even heard it
@@isaiahrosner3780 catch 80% of witches!!! on sale now!!
the blobs have really evolved - from effectively wild creatures to gambling addicts
That’s the holy path for everyone who plays gacha games
xqc viewers be like
They are evolving, but backwards
Gambling with their feelings
Can't wait for the monkey/grape prostitution analysis.
Am I dreaming? A new Primer video? Can’t be.
Ikr
Literally I thought I looked at the notification wrong
in 1 year another one releases
I had to double take when I saw primer in my notifications
It’s true. All of it. The blobs. The doves. It’s all true.
I leverage significance testing pretty heavily in my day-to-day job, and really loved the way you broke it down here! Particularly appreciate framing statistical power (true positive rate) on the same level as significance (false negative rate), when in most cases the conversation/focus really only extends to significance/p-vals. Going to spend a few minutes scrolling through to purge some of these spambots clogging up the comments, then on to the next vid :)
Isnt there a way to have better results with less flips? Make it conditional?
Like after 10 flips observe only the ones with 6 or more heads and then accuse those who had another 7/10.
From rest of the group accuse those who had 9/10 in second round.
I was thinking some similar. As the number of flips increases, periodically remove the blobs that are statistically not cheating. The total number of flips (all flips for all blobs combined) would be lower but, since all blobs execute their flips concurrently, the time spent flipping would remain the same. In both methods, there would a group of blobs that flip the same maximum number of times. Using the method in the video, all 1000 blobs would flip that maximum number of times. Doing it in phases would only have a portion of the blobs flipping the maximum number of times. But the maximum number of flips would be the same in both methods and would take the same amount of time to accomplish. I’d be curious to know if the “phases” method would be more accurate, though.
@@robmoffett2700 This is typically how sussing out bad actors works in the real world! Usually to perform verification you need to do a pretty simple step, following that only in the case of likely fraud certain users are passed to a more in depth verification path while others are passed on directly to the service.
I think the main outcome is that it is very difficult to catch someone cheating using statistics. You should use statistics to raise suspicions, but you shouldn't call them a cheater without further evidence. 5% or even 1% is just way too high of a false positive rate for 1000 players.
My heart aches for those wrongly accused blobs
Chuckled
@LoveSweetDreams ok then let's hear your better idea.
@@zakir2815 what? They just asked if thats how the system works. They said no critique of it.
That one blob that got 23 heads out of 23 flips but wasn't a cheater
@LoveThemDames yeah basically
Love this video. I teach a graduate-level class on experimental design, but many of the students have forgotten a lot of the basic statistics by the time they take it. I'm bookmarking this as a potential reminder/class aid. Thanks!
I feel the pain. I'm starting a master in a few months so I'm trying to relearn all the forgotten info from the last 4 years.
I'm in love with idea how you approach the concept of frequentist hypothesis testing, which confuses so many. (from a statistics teacher). I will definitely recommend your video whenever we come to this part of the curriculum.
I dont even watch these to learn a subject better, i just watch and listen to this for fun! Its actually pretty entertaining!
I love how you take (sometimes boring) statistics concepts and explain them in a way that makes it simple and enjoyable for non-maths guys.
Good job, my dude.
This would have really helped in ap stats last year lol.
I find statistics pretty interesting on its own
p hacking and all that is pretty fascinating
@@NorroTaku Well, so do I since I'm a statistician myself. The thing about learning stats, tho, is that while it has some nice applications and ideas to explore, the more dense and theory-heavy subjects are really difficult to grasp and, as such, require a lot of effort to have a ok knowledge, let alone really understand.
These subjects, that are brought up in Primer's videos are really cool, but basic. When you go down the rabbit hole about learning a lot of probability, inference, stochastic processes and such, you can get kinda annoyed by the elitist nature of the maths behind stats.
@@matheusborges7944 by elitist do you mean that people keep stats intentionally convoluted in a gatekeeping way or something else?
@@aurumble Basically, yes. Stats, as any maths field, is difficult by itself, but it becomes even harder to understand when 99/100 books/papers are written in the most complex and incomprehensible ways possible.
To actually understand something about maths (by reading books or papers) you, most often than not, need to already have an understatement of the subject or to be a genius.
Unfortunately, maths is written not to be understood but to "wow" the reader about how unbelievably difficult the subject you studied is.
Let's say: if a cheater feels that someone is onto them, they will secretly switch to a fair coin.
Then, we're suddenly very deep in the domain of games theory.
A simple crossover that shows how quickly mathematical models become complex.
we could model this by a random chance that the cheater get this feeling, since it isnt based on anything that is contained in the game. Then it is again easy to model. But the mathematical model mentioned in the video IS already so complex that it defies the expectation of normal people who havent calculated this through. This is called the fallacy of small probability, or just the fallacy of small numbers.
what if the cheater has an army of fans that defend him or he admits to cheating
@@lilyliao9521 HAHAHA
May I propose an even simpler case that would complicated this; The weighted coins may change the odds of a heads anywhere within the range [0, 1].
@@activediamond7894 if this is a process that has a finite variance, it could be modelled just the same as the probability. If it doesnt, then theres no reason to even look for cheaters, since u cant get arbitrage from a process that is purely chaotic.
Great review of some of the content I learned in statistics class in college! Love it!
Your out-of-the-box thinking and unique perspective turned an otherwise mediocre presentation into a fantastic one *johnson spy* . You did a good job of catching the mistakes and keeping us from wasting time and by taking the wrong path. Your attention to detail really sets you apart from the crowd. Great work! Jack, Your great work has resulted in tangible, beneficial results to me. You’re a force to be reckoned
Don't go through your partner's phone. Not only is it a huge violation of your partner's autonomy but it's also just plain pathetic. Besides what will it get you? You'll either ruin your relationship by finding out they're cheating or ruin your relationship because they weren't but you still didn't trust them and went behind their back. If you feel the need to do this *JohnsonSpy* all this I saw it's just heart wrecking.
@@samuelbennett6981 are we still talking about blobs here?
No matter how hard you try for your relationship,people will still have their opinion.The above mentioned person got me undetected access into my spouse phone and texts
Wtf why are there so many of these bot comments on this video????
How tf does this have so many likes lmao
I was thinking of using stages where getting a certain percentage of heads puts you in a group that tests with significantly more flips. That brings the total number of flips needed down massively over having a whole group flip 50+ times.
I was thinking the same thing and saw this comment, it's like "random checks" they do at airports for suspicious individuals. Granted, the definition of our suspicious varies from the rather racist definition of "suspicious" that people who work security at an airport do.
In this case, maybe it would be good to ask them to flip their coin until they get 5 (or whatever number) tails. Then we check how many flips they needed. Blobs that got unlucky will quickly prove themselves innocent. We'll need to find the right numbers (how many tails do we ask for and which number of flips is our suspicion threshold) in a similar manner as described in the video.
We could even simulate which test requires fewer flips overall.
@@dragoncurveenthusiast If an unfair coin has a heads probability of 58%, it's still relatively likely for them to get an unlucky streak. I thought in the video in the last sample they would just make one of the groups small enough that there were too many false positives or not enough true positives, rather than make the cheating odds a lot tougher to catch.
Same
So maybe you wait for the bayesian testing video, which will teach you exactly that....
Primer: "How To Catch A Cheater With Math
"
Dream: sweating profusely.
Technoblade: sleeping like a baby.
@@bungeethehuman6292 💀you just did not-
@@bungeethehuman6292 Not funny
@@bungeethehuman6292 bruh moment
@BungeeTheHuman 🤡
Middle school me wishes there was a youtube channel or a teacher that made math as entertaining or as understandable as Primer does
Im in middle school and this help so much in my classes 🤣
I love how your the only TH-camr that makes the zoom to fit look good! Thank you for making my day!
2:08 right off the bat, there’s an extremely important perspective you’ve shared, which is to prioritize the experience of non cheaters over catching the cheaters. After all, why catch the cheaters if the game is no fun?
Some video game developers could take a page out of your book!
*cough*nintendo*cough*
@@rayquaza1053 **cough** any root-level anti-cheat **cough**
and governments..
I agree with the sentiment, but I think that even this 'low chance of accusing fair players of cheating" is not acceptable. The idea of having a false positive in a system that would most likely have very severe punishments is awful, especially because its very hard to prove that you were banned falsely. Sadly this is what a lot of systems do nowadays, and we've seen some of the consequences of people getting falsely banned and having it overturned, but most of them were big youtubers/streamers etc. Any other person without a big following is fucked if it happens to them.
@@johntravoltage959 “the idea of a false positive is awful”
Literally impossible to avoid
19:24 some poor blobs coin fell off the platform, look at bottom of screen
Lol
noo
*cues the climax scene from Jumanji*
Blob lost his "lucky" coin
Lamo
This is great. I would like to use this to introduce my kids to frequentist statistical tests and related notions. It also seems like it could be used to make a pretty good illustration of the scientific method if it were retooled just a bit at the end to explicitly state a hypothesis that you were testing and to explicitly discuss the steps (already implied) that you take to comply with the scientific method. Apologies if there is something in your catalog along those lines that I've missed. But I would love it if you could do this. Also, am I right to assume that there will someday be a Bayesian version of this video? Waiting with bated breath. These are great conceptual frameworks to use for education. Thanks so much for doing them.
If you force it on them they’ll hate it 😂
I first watched this video before taking a stats class. I now watch it halfway through a stats class and understand it so much more.(first learning about power this week!)
very cool work as always
Looking forward to the Bayesian version. Would be cool to be able to estimate exactly what the distribution of cheaters is. Like, what if you don't just get a single type of cheater, but a distribution of them? Perhaps distributed by some Beta-distribution, with most cheaters going close to 50% and very few going for much more audacious near-100%
Like setting your headshot percentage in a cheat for a shooter game
Yes
Thomas Bayes was a god amongst men, probably.
@@maracachucho8701 *slow claps*
Bayesians are like vegans of science: they are a minority, but boy are they a vocal one, and seems like all of them gathered in this comment section! Yours sincerely, a frequentist.
This is what I call excellent science communication. You clearly explained the topic and used understandable terms. You got yourself a subscriber, and I look forward to more videos.
Are you going to watch old videos?
I remember watching this when it came out and not really understanding anything (mostly because I knew nothing of statistics) and now that I'm taking a ap statistics class in hs, I get really excited recognizing a lot of these concepts
Great intro statistics video, very helpful cuz statistics is hard and often counterintuitive
I think it would have been interesting to show an example where there are very few cheaters. The result of this would be that even though we might accuse less than 5% of the non-cheaters of being cheaters and more than 80% of the cheaters of being cheaters, we’d accuse more non-cheaters than cheaters, which can make it feel like the test is bad.
Also i think philosophically if this were a real game in real world if you get 1 fair player as cheater you defeated completely the purpose of the "sport" labeling the real goat of the game as cheater for a communist greater goal.
"If we accuse one person of cheating who isn't it destroys the sport"
Thanks, but I'll take the communist greater goal here. The goal should be to give the legitimate players the best chance possible, not to make sure that no individual legitimate players are ever wrongly accused.
Calling thinking about the big picture objective rather than being obsessively concerned with liberal individual rights "communism" is an excellent endorsement of communism because there is literally never certainty and you cannot fully eliminate type 1 (or type 2) errors while also usefully detecting cheaters.
In other words this is a nonsense "philosophy" that would, for example, imply that courts would have no choice but to always assume innocence unless the probably of innocence was *exactly mathematically zero.*
So i.e. If we have HD video evidence of a murder, your logic says that we should assume that the evidence is random chance and entirely due to compression artifacts and cosmic rays aligning the pixels just so.
Your philosophy forces a standard of evidence which is best described as "beyond unreasonable doubt." There is a reason absolute certainty is not required to convict in any court anywhere, because there is ALWAYS uncertainty.
@@petersmythe6462 Thanks for making that all up. I was about to go to bed and I needed a fairy tail to relax me communist.
Communists always have the best fairy tails things like communism works.
I wish you'd talked about what happens when most of the population is fair and there's only a handful of cheaters. In that case you end up accusing way more innocent people than guilty ones even if you set the false positive % very low. This is a very poor outcome. Unless you follow up accusations with another, more powerful test examining the coin itself to know for sure if it's fair. The followup test is more disruptive and probably more expensive so the purpose of the first test is to minimize the number of followup tests needed. In such a situation the first priority of the first test should probably be minimizing false negatives. It's inconvenient for the false positives to have to get another test to prove their innocence, so you still want to try and minimize false positives too, but you want to catch as many of the cheaters as possible.
No? We make em drop a coin a coin and record the distribution of it until only few persistent deviations from the rate of normalization remain. Those are the cheaters. 2020 US election stats was exactly that.
This would be a good follow up video to this one.
i thought the specific problem with that situation was that the cheater effect size was NOT what the test assumed it was, not how many cheaters there were relative to the population size
@@brasilballs yeah that was the issue in the vid, imay is mentioning a variant issue assuming a low percentage of cheaters...
essentially it's the issue of reducing false-positives by as much as possible which is a pretty major real-world issue ^^ (false convictions for crimes or unneccessary medical treatments, unfair game bans etc etc)
That was exactly what I was thinking.. lol
I love watxhing this simple neat video i rly wish there was more of these like alot more i could listen to these for months
Can’t believe you uploaded what is essentially a really interesting intro to statistics lecture
I took AP Stats a couple years ago, and although I did well enough to get college credit, this is really well made and does a great job of explaining statistics. I feel like I learned a couple of things, and got to see things like p-values as less arbitrary numbers. Really cool, you’re a great teacher, thanks for the lesson!
Came down hear to say something similar, glad somebody did it for me. Took the actual class in college is really only difference lol
The title made me laugh, because it reminded me of that truly ridiculous (but very entertaining) speedrunning drama around that minecraft guy, dream. Catching cheaters with math is a useful and very fun skill!
There were people involved with catching speedrunning cheaters that said that he probably was not really cheating (on purpose) because his actions didn't make sense if he was.
Regardless, the game was modified (on purpose or by accident). The outcome was way way waaaaaaaayyyyyy too unlikely.
He also misbehaved pretty badly, trying to cast doubts on the statistical methods...
With how low the odds were, even if the methods had high uncertainty the game would still be definitely modified
@@didack1419 That's true, you can't really prove evil intent with math - like whether the game was modified on purpose. You can only show _that_ it has been modified.
@@baguettegott3409 The reason it's decently likely that the game was modified by accident was because he already modified the game to make other content, so it's reasonable that he could have left the modification without realising.
It's weird that he didn't say that from the beginning, because it was a reasonable excuse. I don't know. I guess he thought the mods were against him or something so they wrote a paper blaming him of cheating.
Or he really cheated and thought people would not buy it.
Or he was really sure he had the game unmodified, which is naive.
@@didack1419 he admitted later on that he did purposely cheat.
I love how you explained basic biostatistics so easily
Incredible video. it’s absolutely amazing for game developers and stat bans in games. I wish some games used this methodology
2:37 that is amazing
I think a good way to expand the simulation a bit would be to have different 'cheater' blobs that cheat at different percentages. Blatant cheaters at 100%, chronic cheaters at 75%, all the way down to situational cheaters at 25%. It's a big reason catching cheaters in games is so difficult; not everyone is always cheating in every scenario, and when they're choosing to cheat is often times super inconsistent. A really skilled player who also cheats is one of the most difficult things to detect, as well!
yeah for they have the skill, knowhow, and reputation to know: what cheating looks like/what legit play looks like, the skill to execute legit play, and the community goodwill to not immediately arise suspicion.
@Test to expand to that setting, it's a bit more complex because you'll need to explain Bayesian statistics - you'll have explain what prior distribution you're assuming for the cheaters is and derive/simulate a posterior distribution for the coin tosses and how exactly to calculate the probability that the player is a cheater.
It's an entire topic of its own and really not that straightforward of an extension from the frequentist case, but a framework exists to do that.
You could expand it even farther by adding "scammer", "blatant/subtle", "sore loser" etc blobs to the mix. Make it so that instead of getting a fixed amount of mood improvement or souring per flip, they can make bets. Have a variety of cheaters that use coins that vary from 51% heads all the way up to ones that just blatantly get head 100% of the time. Have blobs that start cheating if they get a streak of bad luck. Have blobs that switch out their coin for a weighted one whenever they bet big. Etc. Then you could also factor in the net loss/gain of their mood over time into the cheating algorithm.
And of course, blobs who are stupidly generous and choose to lose on purpose. Which actually does happen sometimes whilst cheating in video games
12:56 The cheeta graph 😅😂
It might be a cool idea to narrow down the blobs after a certain amount of heads, that being if their initial 5 or so coin flips is suspicious and what a cheater might have, place them into a group for further testing, which would make it not bother so many of the innocent blobs while at the same time providing more accurate results
Don't forget that (even at 0.75) only 24% of cheaters get 5 heads in a row... so if you use 5 heads then your method can NEVER catch more than 24%, even if subsequent tests are perfect. If you widen the net to include 3 out of 5 you get the 80%+, but also half the fair players. Ultimately you will need about as many iterations to get to your result... in fact without doing the math I'm guessing you cannot reduce the total number of flips in this way, but would be interesting to see for sure.
@@grenvthompson 13:00
@@fabio_fco Did you link this timestamp with no additional context as an arrogant way to "educate" the person you replied to? Or maybe you did this to showcase future reads, like me, the moment in the video which helps support Gren's post.
JJ's idea is great - what's if we cut the group after a certain threshold? We stop watching the people who are doing poorly because they couldn't possibly be cheating. This is a good idea because our next group of players will have more cheaters than fair players, and we can increase our "cheater catcher" threshold for this second group because we know for a fact there's more cheaters.
Gren replied with a confirmation that cutting the poorly performing participants will not INCREASE our True Negatives or True Positives - it will only help narrow those down. So if we were to focus on the 5 out of 5 table, we'll have a group consist of many cheaters and a few fair players, but we've already excused a few cheaters because they were performing "poorly" (enough).
I like both of these ideas, but feel like 3 out of 5 is too inclusive. In an even group of 1000 blobs (50 : 50 ratio), performing this split will leave us with 700 blobs in a 36:64 ratio. That's 36% of the 700 are fairplayers and 64% are cheaters. With this new group, we can perform a less forgiving threshold to the fair players that catch more cheaters, but that's still a lot of fair players being caught.
I think if we increase the initial threshold (instead of 3 out of 5, maybe 4 out of 7) and we create a less forgiving second threshold (11 out of 16, for example, I just randomly picked) - we can try to fit within our 5% and 80% rules. Notably, we if we combine the maximum number of flips for these two groups, we'll get 23 flips. I'll do the math later, but we could already pass our test with 23 flips - so, as Gren mentioned, this is only a valid strategy if we can use it to reduce the number of total flips.
@@grenvthompson or for the first round, 3/5 heads leads to further analasys. or use the first 10 flips, and if someone gets 7/10 heads, then investigate further.
I've played this every year for a year, brilliant game.
maybe he is saying that he only played once.
You meant to say "every year for a century"
every year for a day
it's been up for less than 3 months, so... what they said could be technically correct, but not "every day for a year" or "every year for X years" 🤓
With that sample size, the game MUST be brilliant. I'm off to go play it now. Just once.
Blob Maths and Blob Biology are cool, but I want to see Blob Society more.
they ask what is blob but they never ask how is blob
@@iamnotahuman2172 We saw happy blobs, sad blobs and dead blobs. I guess that is how blobs can be.
Me too
I want to see more blobs interacting
"We live in a society."
- Blob Joker
Really helped with developing my game! Thanks so much! Now there are way less cheaters!
I'm super looking forward to the Bayesian video!
Great video, but I want to point out one thing:
the true proportions are also very important. Let's say you have 1000 blobs, and only 10 cheaters. With a 5%/80% test you would get 50 false positive, and 8 true positive. So only ~14% of the accused cheaters are actually cheaters.
In the real world big factors for the limits of such tests are the consequences: What happens to a patient with a false negative test? Will he die soon (eg. cancer), or will he just stay two days longer sick at home?
And what happens with false positive test results? Will the person go to jail (eg. drug test), or undergo harsh medical treatment (eg. chemotherapy), or will he just take a day off?
In praxis these issues are often solved by confirming the results with additional, different tests (in case of the blobs the manipulated coins might have a different weight...), but should still be considered, because these tests again have the same problems, and every additional test costs money and time.
We could consider this a test of the police catching a criminal. If the odds of being caught are high it will act as a deterent so fewer cheaters will actually risk being sent to jail by cheating. But if too many innocent people are sent to jail, the people will rebel. New name of game "Politics"
They usually do a second different test. One is
@@David-sp7gc Thats why he can modify the rates which causes the p-value to be much lower and your confidence rating in catching a cheater.
The 80/5 is a generic prediction for the model, but by no means worthy of actual implementation. When they test for cancer, not 1/20 tests are a false positive.
Your numbers would be extremely improved if you went to catch 99% of cheaters and accuse less than 1% of fair players.
Then, 99% of your predictions will be correct. If you have 1000 fair players, a 14 fair players, the odds of you picking a fair player is extremely low.
I really thought the same as you did! And I first thought he had modified the number of cheater in his 60% exemple (but demonstrating assumption aren't fact was also educative)
That's why the assumed effect size and our acceptable p-values and a-values are really important and require careful thought.
For refereeing a game, we'd want to assume a fairly large effect size and we would want a pretty small p-value. The reasoning here is that the test itself serves as a deterrent - you only need to catch the most egregious cheaters and the rest will take warning.
You'd also want to assume that in a game setting, players are tested fairly frequently. They're probably 'tested' during every game they play. So when referreeing a game, it might me more effective to measure the false positive rate per tournament or per year (like the way we measure failure rates for contraceptives). Because what we're *really* interested in is "how likely is it for a fair player to be accused of cheating during the entire tournament."
In addition, you can risk not catching all the cheaters using only one test, because they'll be tested again and again every match. Even at a 20% true positive rate per match, that amounts to a huge risk throughout an entire tournament.
I love the application to real life sciences. Probabilities fill everything we do and it’s a good thing to have a process for determining results from a limited pool of testing.
Great video it gives a very intuitive way of understanding probability, and also demonstrates how unintuitive probabilities can be.
I dont like maths verym much but this mix out of games and math makes me love and be so interested in that entire topic
My head hurts so much after few flips with those False Positive and False Negative . My mind went blank. Have watch again, and again and again. . . To infinity
15:56 The froggy hat this blob has is absolutely adorable!
19:04 Keep an eye slightly below the stages, you'll see a few coins being flipped too far, falling off the stage entirely and into the void below.
lmao
Pretty solid introduction to p-values and hypothesis testing.
21:03 holy shit, love this way of learning. Teach by understanding mistakes if that makes sense.
okay so I have 3 points to make :
1) I didn't forget that the 75% chance of getting head for a trick coin was an assumption so I feel very proud about that.
2) I was studying binomial and bayesian distributions earlier this year and I would have LOVED to have this video available then because it's 10 times more clear than any lesson I had
3) That's a great video as expected from you, keep up the good work because I fricking love watching them
I literally have my statistics exam tomorrow and this is a beautiful video to sum up one of the chapters wish he posted this earlier
real life nerd emoji
1) You should feel proud! Good for you!
I was super proud of remembering that too :)
2) Same, only it was a couple years ago for me
3) Agreed
If this was a real game, where the players expect fifty percent win rate,this would majorly break the game. Getting accused of cheating is probably allot worse than losing, so every fair player would have to factor in that risk before they played. Depending on how bad it is to be accused, players might even decide to use coins that come up tails more often than heads.
The unit on Game Theory and maximal strategies comes later (but is also fascinating) :)
It would be solved by some kind of manual checking of the coin itself on the request. Say, manual checks are too tedious to be done at large scale but efficient for anyone who is too lucky but still not a cheater
And thus cheating aswell...
You could make a video with this, adding a sort of "catched based survival-method" to see what kind of test rules out the evolution of cheaters?
the coin falling off the map at 19:23 (awesome video btw)
I love this so much! As a game dev I'm desperate for anti-cheat. However gameplay time can prove dedicated people to get really lucky and having 1% false positives means that there's a much higher chance that actual dedicated players get lucky and falsely accused. That's why I believe it shouldn't be as brief as possible, but be constantly looked at. Like a history of average luck over all flips and over the last few flips, if the player has been playing for some time then the test becomes less brief assuming that they'll be more consistent with what they're doing. They won't randomly enable a cheat so theit test is more forgivìng and catches far less cheating scenarios, but if the history sees a sudden spike of luck then that's a sign to be investigated.
if the false positive rate is high then make the punishment mild like a 5 min ban or something
in addition to lowering the punishments for false positives, (imagine that content creators will not only be more likely to test positive(regardless of true or false), but also that they will have more runs at the test as they do it for a living) you may also find that you want to make a hidden strike system, and use that in there. some cheaters will cheat for a few runs and then play without them, which will mean that it starts to make it harder to catch them, so instead a trick you could use to catch them is to put strikes on a hidden list, if i get 10 heads by cheating a 100% chance, ill probably not go farther, and will return to normal flipping, to throw you off my scent, so you give me a strike for that, and even though its statistically insignificant for someone with thousands of flips, its still an oddity to be noted. as you perform larger oddities it can add larger sized strikes, until a certain number of points you just say "hey bud its pretty clear that youre cheating and then trying to hide it by playing badly afterwards, come in to our labs and do a game and do as well as you do in those good runs if you wanna not be banned" as an example; 10 wins in a row can be assumed to be about a 1/1024 chance regardless of the game, as long as the chance to win is 50%. if you win 10 times in a row, thats 1 point on your strike file, if you win, i dont know, 15 in a row(1/32768 chance), lets say its a 30 point strike (on top of the fact that 15 flips in a row as heads already will count as 6 points on their record, as it is 10 in a row, and 11, and 12, etc), and lets say that if you get 100 points on your record, then we give you the talk. make each incidence stick around for X flips where X is past the 1% false positive rate, and you will see that even so, the majority of cheaters that try to get around simple probability by playing with cheats very little of the time, they still will proc this method of detection.
You can create a model based on their ELO. Then use a standard distribution relative to the elo to preselect outliers. Then you can run a series of tests on these outliers.
@@nou5440 while that would work, the punishment would also have to be annoying
Something like changing the player's name to "I cheated" for an hour, and preventing them from changing the name back for the hour could work
@@lavantant2731 the only thing about that is that it's usually not a flat 50% chance to win or lose. Even with some kind of matchmaking. There are a lot of other factors at play.
I was just about to suggest a Bayesian approach, and then you said you are going to cover that in the next video. That is good.
I'm pretty sure that the best test method won't involve the same number of flips each time. It would look at the trend after each flip and accuse once the probabilities become strongly enough tilted in one direction or another. How far tilted depends on the cost of a wrong guess and the benefit of a right guess.
14:36 Once a player gets 16 Heads or 8 Tails, there is no need to continue to the full 23 flips. 16 Heads = Cheater, 8 Tails = Fair.
* I'm commenting as I watch = you might address this in a second...
you just made me watch an introductory lecture to nonparametrical statistics, nice. Would've been better if I saw this on the start of the semester tho, since i had just finished that class
One thing you can do to lower the effective number of coin flips is to have dynamic decision making. You can establish a threshold to sort cheaters and non cheaters at every level while allowing uncertain players to continue testing. For example, if someone has the first 6 flips come up tails, you can stop testing because they are most likely not a cheater.
What about a cheater who doesn't always use a cheat coin? Cheats need to balance profit against chance of being caught. Runs of tails would throw algorithms. Just asking
@@tim40gabby25 what you are talking about is effectively changing the cheating effect probability. We aren't testing for cheaters, we are testing for cheating, so as long as they aren't swapping coins in the middle of a test, it doesn't change your approach. If you wanted to monitor cheating in a game in the real world, you would need to adapt your detection strategies to account for subterfuge like the inconsistent use of cheats.
Yeah, you'd need a more sophisticated algorithm to catch players who only cheat occasionally, and eliminating folks due to runs of 'bad luck' often doesn't work in real life scenarios.
For a real world example, take Fall Guys, which has had its fair share of cheaters over it's lifespan. (Also using this example because I just lost a final round of Hex-a-Gone in which someone was using a flying cheat so it's fresh in my mind and I'm malding.) It's well known that many of the cheaters in that game often only use cheats if they make it to the final round, meaning there are less people left to report them, and they suffer enough pre-final losses to not look suspicious on paper.
Of course, in this scenario where the cheating is obvious to their opponents a reporting system *can* work, but only if it's well implemented and has some protections against false reporting. Like a player is only marked 'sus' once they receive enough reports matching the same parameters, or something like that.
@@LowPolyPigeon Interesting. Thanks
@@tylerhale8679 Interesting. Thanks.
19:24 Shoutout to this red dude at X=12, Y=2 throwing his coin off the platform.
yea lol thanks for giving out the coordinate of that blob
@@lephantriduc There is also another red blob at 19:11, X=12, Y=4 that yeets it off
2:29 - 2:40
Detective Blob, you've cracked the case!
This might be the only math video I've watched where I actually know what to do
Thanks AP stats 👍🏾
1:02 that blob with 0 heads be like me fr
When you're a mathematician but but yo man being suspicious:
yo
sus
"It's very easy to forget that assumptions is an assumptions and instead treat is as a fact", it's so true
It would be more computationally sane to falsely accuse a large chunk of players and then pass them through a more rigorous test to filter them out each time, maybe even loads and loads of a little more precise tests. It would be cool to see you making a video about it, on how many precise tests would need to be done, how many layers per se.
Great video as always.
We do this IRL for certain diseases: screening tests.
just commenting to say I have seen what you have done there with the cheetah at minute 12:50. Great video, enjoying it so far!
Looking forward to the Bayesian vid - that would be my approach. A nice way to contrast the Bayesian vs frequentist perspective is that.. frequentists condition on the “true” parameter and integrate probabilities over the random data. Bayesians, on the other hand, condition on the data and integrate probabilities over the (assumed) distribution of the unknown parameter. To me, this makes the bayesian approach more attractive, because in reality, we only know the data - not the parameter.
I want to see what this guy's studio looks like. He does this with just a laptop?! Unbelievable work.
In the interactiv version the strategy i found most consistent was flip 7 coins if 5 or more are heads than they are a cheater. Of course this strategy is not perfect because they limit the amount of coins you can flip but it will let you get pretty far in the game.
I would have love to have you as a teacher. You're just awesome at explaining, this is because you explain the basis and the "easy" stuff and I know how hard it can be for people who knows difficult stuff to forget how the basis can ben hard for novice. You don't, and I admire you for doing so
It's kind of fun to watch videos like this after hearing all sorts of similar (yet less entertaining) lectures from a friend that's soon going to have a PhD in Biostats. Can't wait to see what you can pull up for the Bayesian method!
What if the false positive/false negative rate was a ratio? Like for the given 5%/80% it would be 5:80, or 1:16. This makes a more flexible test - If you only falsely accused 1% of fair players, the test would be successful if you accused over 16% of cheaters. This seems pretty low, but what if there was a minimum % of cheaters accused, so the smallest ratio would be, say 4:64?
21:06 LOL, the moment you said "I'm not gonna tell you anything about this group" I was like "watch, they are using lower probability coins" :P I guess I was just thinking it just seemed the only thing to change at that point to make the test with that same criteria less successful
could you work with 2 thresholds, 1 where if you get below it you're assumed to not be a cheater, and 1 if you get above it you are assumed to be a cheater; if you fall between them, additional coins will be flipped.
This way, the amount of coins that need to be flipped can be lower for the very suspicious and very unsuspicious groups, and only the more uncertain group needs to flip additional coins.
Came here to say this, but thresholding would be more difficult as we have to figure out an "optimal" way of distributing the 80% true positive and 5% false positive
Absolutely, and this is indeed how we design tests in the real world. The simplest example would be, if we have someone who hits 16 heads in a row from the start, we don't need to continue with any more flips :)
Sequential analysis is a whole other can of worms, especially in the frequentist realm.
@@harrytsang1501 I agree that this is harder to optimize, but you don't have to go very far to make it better than the test mentioned in the video.
I struggle with statistics class and though this video does not directly tackle my questions, I really appreciate the time you take to explain them clearly, especially regarding the false positive ratio. I hope to see more blobs soon
Asparagus Omelet (No. 2)
Whisk 3 eggs with 1/4 cup chopped chervil, 1 teaspoon kosher salt and a few grinds of pepper; mix in 1 cup cooked thinly sliced asparagus. Cook in a buttered large ovenproof nonstick skillet over medium heat, undisturbed, until the edges are set, 3 to 5 minutes. Bake at 350 degrees F until cooked through, 3 more minutes. Fold in half and cut into strips. Butter 4 slices toasted pumpernickel bread; top with the omelet strips.
my idea of testing for this example would be to set the coin on its side and test to see how many times it lands heads. this makes it more obvious who will be a cheater because it removes the factor of randomness caused by flipping the coin.
Outstanding video! Assumptions, models, technical knowledge, evaluation, and beautiful communication - a mathematical masterpiece!
Omg I've watched people treat assumptions as facts and screw themselves.
Great video! Love all your content.
Still waiting for the Bayesian video!
I feel like you could make this more computationally efficient by determining one threshold to maximise true positives, discount all negatives, and then rescursively retest only those remaining. e.g. If
This guy takes the most simple of topics, makes them easy to understand, and makes learning fun!
This helped me understand P-value’s and statistics more than any other formal way I’ve been taught. Thank you.
At 19:23 the 12th from the left and 2nd from the bottom red blob dropped the coin off the map! Does this change the data at all or is that part of the video just visual while the real numbers are done elsewhere?
Is there any statistical significance to testing for runs (i.e. runs of heads or tails)? With some, admittedly, fuzzy thinking, it seems like you might be able to reduce the number of total coin flips needed to accurately guess cheaters by looking at runs instead of the total ratio of heads to tails. There would be more branches of the possibility graph that result in runs of some size, even if they don't end in drastically higher ratios.
Never seen a more clear explaination of hypothesis testing in my whole life. Thank you.
At the end of the video i was actually shocked that 22 minutes had aldeady passed. You made such a good job at explaining this topic in a very entertaining and instructional way. I learned a lot and you gained a new subscriber!
I'd love to see the next video in this series
Eagerly waiting for the Bayesian hypothesis testing video.
This is so well put. Can't explain how much I appreciate your ability to slow down and put this into a way anyone can understand.
Compare this to creators like Veritasium (as much as I love him) and they're fairly content to speed through complexities, leaving laymen in utter confusion.
"It's very easy to forget that assumptions are assumptions and instead just treat them as facts."
Truer words have never been spoken.
I originally was under the wrong impression that a player who has 100% accuracy in an hour long match with more than 5000 shots taken, is clearly cheating their caboose off, but now I know that likely isn’t the case from this video! Thanks!!!
You summed up my 3 month statistics class in 20 minutes… well done m8
I've literraly had my propability and statistics exam yesterday. It was sooo boring to learn. However i love your videos and how they spark my motivation so i can implement these topics to programming. Thanks :D
hah I finished that course last week! (5 days ago)
If this was a real world example, I'd expect the cheaters' coin's bias rate to vary from person to person. I wonder how much more complex that'd make the math.
Thanks for the video.
Another variable would be ratio of cheaters. What if it's only 10%? More false positives for one?
If there are alot less cheater (like less than 5%) you would end up catch more fair player than cheater even though the percentage is still the same.
@@koharaisevo3666 A (much) larger sample set is then required, but conceivable.
You add a second step retesting the coin after accusing a player of cheating to pull false positives lower at the 16-17 threshold. The chance of hitting 16+ more than once is so small that it would essentially be a 99% chance they were a cheater, and if it was just a lucky go there would be about a 2% chance you would falsely accuse someone.
What if there was another variable added analyzing the pattern and spread of consecutive heads, if we assume the cheat coin is 75% accurate and see a trend of 3 of every 4/ 6 of 8/ 9 of 12 could that help determine more cheaters and less false accusations especially if we see a trend of repeated success?