I Did a Study About the Dunning-Kruger Effect in Overwatch 2 and I'm Making That Your Problem Now
ฝัง
- เผยแพร่เมื่อ 19 มิ.ย. 2024
- No, this is not a joke, and no, this is not clickbait - I really did conduct an actual research study on the Dunning-Kruger effect in Overwatch 2, and today I’m going to share the results with you! Are OW2 players delusional? If so, how much so? Is there one role that’s more delusional than the others? And how many comments am I going to get saying that I am the actual manifestation of the Dunning-Kruger effect from people who can’t spell manifestation? No time like the present to find out!
Become a channel member! www.youtube.com/@TheViveros/join
Twitch: / theviveros (Tu/W/Th 7:00 PM - 10:00 PM MST)
Socials: linktr.ee/theviveros
Chapters:
00:00 (Intro)
01:51 (Part 1: The Background)
07:50 (Part 2: The Data)
25:41 (Part 3: The Analysis)
32:37 (Part 4: The Flaws)
43:08 (Conclusion) - เกม
one thing you need to take into account about how people feel about the impact of their role is the placement of each role in game. tank players play in the frontline so they don't see what their teammates are doing while supports play all the way back and can see every mistake their team is doing.
Unless it’s Moira lol
Yet as a tank player you notice heals from Healers and Picks from DPS. Picks means push and good heals means more aggressive play
*_Support Life:_* "Group Up!" "Group Up!" "Please stop going in solo!" "Group Up!" "Group Up!"
ALT+F4
@@erikhendrickson59 One reason that playing tank sometimes sucks everything. You say you're going in and where, no one behind you replies or says they don't like that idea. You walk forward and die. On your death, you look around and realize no one else was with you.
I look behind myself about 98% of the time now.
not exactly true with ball and doom. that'd go into hero-specific analysis
The comedy of a Sombra main collecting data to study and understand people
Ngl I kinda love the idea of Sombra sitting there and trying to set up a survey in Google Forms so she can see if her teammates are able to accurately assess their incompetence.
Just a thought. Is it possible support feels like it has less impact because the game's feedback system doesn't provide much "good" feedback to supports? As an example, the potg system is biased towards tank/dps because of how it rewards kills. Even if an Ana sets up a good play, she isn't the one who actually made that play so she rarely gets rewarded for it. Really the only feedback supports get is high healing numbers, which encourages them to inherently put the state of the game in the hands of dps/tanks.
That's a good point! I think it's exacerbated by the fact that supports aren't just unrewarded when they help things go right, but are also generally the ones blamed when things go wrong regardless of whose fault it was. If someone dies, the support tends to catch the flak for it even if it was literally impossible to save them because of the decisions that the other player made.
@@TheViveros It would be interesting to then take some of these details and carry it on to what the game itself does to help/hinder people's views of their own skill, or even to (unintentionally) promote some level of toxicity.
@@TheViveros this misconception has been driving me insane for years. I've been playing this game since 2017 all the way from gold to 4.4k (also as a support main after role Q) and I can assure you that, unless you are playing Moira or Brig, supports get flamed by far the least, on average. Maybe it's a bottom rank thing
@@giannis_tar i think supports get flamed the least when you have high healing numbers, but a lot more when you have a dps heavy support for example. I got baited into a healbot playstyle, because i got scared of my teammates, instead of making plays on my own
@@giannis_tar It's not a misconception. It very much does happen, in more than just overwatch. Supports have to play above and beyond the rest of the team in order to avoid getting flamed, and even that can't save them. If a player doesn't know what went wrong, or if they're the thing that went wrong and don't want to admit it, they resort to the classic "well the support wasn't healing me".
It’s so cool to see this type of analytical and statistical video on human behavior in overwatch, it would’ve been interesting to see if there is a correlation between gender or specific background information
I agree, I'd really wanted to be able to use the data on gender to see if there was some sort of connection that could be made but I would've needed a much larger sample size to be able to do something with that with any confidence, especially since any conclusion on that topic would've been really easy to take out of context. I did find it interesting that the gender breakdown was actually pretty similar across responses in all three roles, though; while the study is obviously really small and my audience isn't necessarily representative, I was surprised that gender didn't really seem to be a factor in which roles people tended to place in.
Spoiler alert, girls mostly will play supports, or mercy dva widow the most 🙃
@@Sigma_Mail where’d ya hear that lol
@@TheViveros out of curiosity, as someone who didn't take the survey, what were the gender options?
Edit: nevermind, I got to that section of the video. Commented like a minute early lol
@@scarymonsterss Hm, I remember seeing a study on reddit a while back saying girls were in fact more likely to play supports or female characters. It's interesting, but it's been too long to truly accurately remember and I don't know how relevant that is to much else.
I think something interesting to know would be what % of the supports were mercy players/mains- because mercy is unique in the fact that she doesn’t really get kill on her own that often and literally needs a teammate to hit a shot for blue beam to actually have an effect. I got to diamond with baptiste, who is literally just a dps with a healing right-click. Same things can be said for literally all the other supports. Perhaps playing mercy( a pocket hero) or just healbotting as Ana makes the player feel like their only contribution to the fight IS healing, and thus thinks that their rank is influenced heavily on other factors. I guess the only way to check this is to ask for their damage,healing,assists, and deaths per 10.
I definitely have been part of that group of people in past. ^-^; its strange how long it takes some players (myself included) to learn what exactly their overall goals and personal responsibilities are while playing. If you always play at a low level and never interact with higher level plays, most players never see characters used to their full potential, or might not even notice when its happening. Tbh i just woke up and this is lowkey a tangent sorry!!
Personally I think that you would HAVE to be a much higher rank to climb as support, because you’re so dependent on team synergy. If your team is picking good fights and positions, you can absolutely carry as support. If your team is all over the place and losing stupid fights, there really isn’t much you can do unless you can really pop off with DPS without dying or losing your own stupid fight.
@@deawinter Ok, I see where you're coming from, however- say if I'm a kiriko and I get a cheeky pick before the fight before tping out- that's an individualistic play that doesn't require your team (save the fact that they gotta be there for you to tp out). Essentially, what I'm saying is that it doesn't take a team for you, individually, to get a kill- it's not like 5 people are moving your mouse when you're shooting at someone. Unless you're specifically talking about mercy, which in then I have no idea how you're supposed to climb- or brig, where you really aren't looking for kills for the most part- for the majority of the support cast, if you can consistently kill at least ONE person while staying alive yourself, you've done your part- but I get it, sometimes it do be a team differential.
Yeah, i found that i won a lot more by hitting dmg shots as ana between heals, or not picking mercy if i had no good person to pocket. Also, W bap
A good support will recognize when their team isn't hitting their shots, and adjust their pick or playstyle appropriately. If you're mercy and your damage beam isn't getting much value, or you're getting jumped on by a sombra, swap picks to zen, moira, bap, ana or someone that can assist in damage while also still healing.
There is about a minute left in the video, and I'm only just NOW realizing that this video format could be the future of academic studies. This was accessible and entertaining, and I hope it becomes a larger trend
I’m 43 if that helps you feel less old. I’m also a mom. I highly encourage you kids to game with your mom!
Your video was REALLY thoughtful. It’s a beautiful example of how to do a really nice and interesting research paper. I hope this encourages people to look at STEAM differently, how it can intersect with hobbies, and different skills you can use to enter into an industry you like. I also hope this gives teachers, especially at a jr high/high school level, ideas for how to drive their students to topics outside the basic project topics like ‘plants: how do they grow’.
Thank you so much for this comment, it's very sweet!
It's interesting (if unsurprising) that support has by far the highest population of players who exclusively play one role. It would be interesting to see how their responses differ from people who also play other roles - I would expect them to be even more extreme. It sounds like that's no longer possible due to the method of anonymization though.
I found that interesting too, particularly because it didn't really correlate much with demographic data like age or gender in the ways that I think people might have assumed. I actually originally planned to discuss the difference between people who exclusively played one role, two roles, or three roles, but there just wasn't enough data for that - it would've been hard to make much of a sweeping statement about how total DPS players feel versus total tanks or total supports when the number of respondents who met that criteria was so small.
i have a lot of friends who say they only play support because the other roles are too stressful and im like "so you get flamed everyday... to avoid being stressed???"
@@Azarthes support players, especially on the casual side, do have a trend of going healbot. Which means they kind of stop interacting with the enemy, thus removing constant confrontation, thinking of controlled area, challenging enemies in duels, and so on..
@@IHateHandlesWayTooMuch Hmmm... it's true that the mechanics of support as a role might be uniquely autopilot-friendly. I play basically only tanks and I find myself autopiloting at times, but I can tell very precisely when I do it because I get twisted into a pretzel by the enemy team very handily when I autopilot. Supports might have less of a way of telling when the deaths of teammates are due to them healbotting or to the teammates' own incompetence.
@@Azarthes i guess it may depend on rank, but where i am (masters on all roles) support is by far the least harassed role. I prefer support for that reason, i dont gotta worry about getting flamed for missing a few shots, and when i do play well everyone is really nice about it.
Im impressed with how you've managed to take what you're doing in collage and make great yt videos with that. Talk about killing two birds with one stone. Keep up the great work :)
The statement you made, "You can definitely feel when your supports are doing well" is what feels like the exact thing I needed to hear to settle with my confirmation bias of thinking that support is lowkey the most impactful role. It is nuts watching gm supports heal their team instantly upon taking damage as apposed to watching gold supports taking 3 to 5 seconds solid to respond to their team taking hits on frequent occasion. Enabling your team to peak in performance is so powerful
As someone who works on the edge of academia, l wanted to congratulate you on your rigor. As you mentioned in my opinion by far the biggest issue with your pilot is the tiny sample size - not your fault and l would not be shocked for a larger sample to confirm - but the issue is there.
Now, it may be Dunning-Kruger effect talking - but as a low level support main the inability to control outcome (e.g. W or L) from my position seems very real. I have spent many games “doing my job” - pumping in heals, doing some damage to clean up stragglers, mines, traps, etc., minding the cart as needed - only to watch the team fall apart with no effective push or holds (tank issue) or to be hassled to the point of distraction with divers (Sombra, Genji, Tracer are common) (no peel issue). Of course, l make mistakes, too - l am low level after all - but it has been my personal experience (yes, N = 1) that low level support have less agency in the game than low level tanks or DPS. Which l could easily change by learning a new role.
Finally, congrats on the nice grade but do keep in mind the factors that surround grading (that are much less objective than even the Overwatch ranking system). I’m not going to comment on the academia system and spoil it - but l do urge an open eyes look at that for anyone considering it for themselves. Thanks again for the hard work and sharing it.
Holy shit good video I just want to say I really liked this video. The way you did it all academically was awesome because this is like the exact stuff I’m learning in my Phyc class right now (mostly the how to conduct a study part).
Anyways, I think the main takeaway is that bronze DPS players are gigachads
Thank you so much! And yeah honestly the bronze DPS players were the most surprising part of the whole study - I checked the data on them at least a dozen times while I was working on it because I was *certain* I must’ve made a mistake somewhere lol
Have you considered contacting Blizzard? I feel like this study might be useful for their team! Insane video, really fascinating and well made
Thank you! I don't know if they're the biggest fans of me (if they even know who I am) but if they did stumble on it and found it helpful that would be really cool!
Honestly they shuold do things like this themselves. I mean, they run the game, devs should put various surveys inside the game to get a better grasp of the community. That way, they get a lot more of responses, and they already have a lot of stats of the players.
@@TheViveros my dad owns Blizzard, he's Mr Blizzard. He says you're wrong.
they already now this obviously lol
i hope that more of these surveys can be spread to wider audiences and i feel like people like Flats would love to promote this to a larger more broad audience focused on both the entertaining side of OW (those who watch his gameplay and tiktok videos) and the analytical side of OW (those who watch his patch reviews and state of the game opinion pieces) rather than your mostly analytical audience. I hope that a similar study can be conducted in the future (possibly for season 3 since it seems this is the most enjoyable season) to measure topics like broad opinions on seasonal balance by rank/hero or generally how enjoyable each season has been. good shit and i hope we can get this to a wider audience
I just had that thought, there are bigger content creators who seemingly sincerely want to improve the state of the game (to benefit everyone). They could greatly benefit from hearing and spreading this information, in a way that may help create further beneficial research insights.
The only issue with using someone like Flats, especially for opinion responses, is that his community is a community--you're going to suffer from sampling bias. Flats' community is almost a monolith when it comes to opinions. That's also true for many large content creators. It's also true of most forum posters, although the average forum poster is sharply diverged from the average content creator consumer when it comes to opinions of balance. I would bet it might also be true of most Reddit community members, but I can't speak to that--I have no interactions with the Reddit OW community. Not to say that you wouldn't get better data by asking Flats' to promote it--more data is almost always better data--but it is something to think about.
You mentioned it towards the end but I do think its important to note that this survey did not factor in that had an outcome on my responses was that climbing ranks simply takes time.
I did not have the time to climb to the rank I think I would be able to achieve due to being busy with other things besides Overwatch, and this probably applies to other people as well. I'm aware I am not immune to the Dunning-Kruger effect but for future studies I think this is a factor worth considering since rank systems in competitive games prevent people from climbing/dropping rank too quickly.
I actually did mention that at one point in the video - it's definitely a relevant factor to consider because not everyone will have the same availability to grind and climb, or will be starting from the same place. If you're theoretically a Diamond 1 player but the game placed you at Silver 4 at the start of the season, you'd probably have to play close to a hundred hours over the season just to get to where you should be. Part of why the subject is interesting is because the typical application of the Dunning-Kruger effect doesn't really have a way to account for that - it can't distinguish between a valid criticism about an inaccurate evaluation and someone coping over their rank, and groups them both together. A few of the respondents are people I know personally who've mentioned that they just didn't have enough time to hit their perceived rank before the season ended but eventually climbed there in the second and third seasons, so if this were going to be done as a full study, that would be a factor to consider for sure.
@@TheViveros Just got to that part myself, should have figured you would bring it up whoops
I've also been able to climb to the rank i thought I deserved as of the time of answering the survey (Plat 4) in later seasons, and in fact even go above that (currently Plat 2)
@@TheViveros not just time, but also that we aren't matched against other people of the same visible rank, we are matched against MMR. So you don't just pub stomp your way up. I think the disparity between visible rank and MMR hurts your ability to do as good of a study as you want to do, but I do appreciate you taking the time to do it with what we have access to.
Blizzard really undermined everyone's ability to do data analysis about the game when they allowed people to set profiles to private, tbh. I understand their argument for it, but I think it was a mistake - it completely obliterated the community's ability to look at things like pick rates or average rankings. There's so much hidden data that it really does make it hard to know much of anything for sure - lin a study like this, it would've been really valuable to be able to check the pickrates of respondents against the average pickrates for the season to see if maybe the responses were skewed because my audience is disproportionately playing certain heroes or something, but there's no way to actually check that anymore. It's really unfortunate.
You’re my favorite OW youtuber. Can’t wait to sit down and watch this later.
Thank you so much! I think you’ll like it, this is already one of my favourites on the channel and the topic went to some really interesting places.
While it's pure conjecture, I did notice an interesting note; Previously, you covered the information vacuum in overwatch 2, and findings from this study could possibly interact with what information it *does* provide. In specifics - the medals. While not an actual indicator of proficiency in a role, they are something a lot of players hang off of for information.
In my experience, in bronze there are times in which damage players will not out-damage other roles. Everyone does damage, even if the amount you can dish out varies wildly between heroes. You also have the enemy damage heroes to compare yourself against. However, supports rarely have competition for healing and what constitutes a good amount of healing on one support might be completely unreliable on another (Zen v mercy in terms of what their goals as supports are)
This might have an influence on how bronze damage players see their skill - something that is for the most part quantifiable. You can directly see how much lower a number is to other numbers. Whereas what makes for a support playing well can be more nebulous, despite having a statistic reinforcing a part of their role. A mercy with gold healing might blame her team for dying in a teamfight rather than wonder if she could've played a better position to make it harder for the enemy to pick her.
This is an interesting point, especially because the game can't really measure something like how many players you've saved in the same way it measures eliminations - the game can't offer a lot of feedback about something that *might* have happened, and a lot of what supports do as far as healing goes can be simplified down as preventing deaths. A support who's playing out of their mind but whose teammates refuse to position themselves appropriately will look like they're failing because there are a lot of deaths, and the game really has no way of providing context for whether the damage and healing stats on a hero like Moira reflect a good or bad performance. It also doesn't help that a lot of supports have mechanics that can't be tracked easily in terms of impact or feedback - information about things like Ana hitting sleep darts, Mercy amplifying damage at the right moments, or Lucio doing a good job of boosting his teammates' speed is really, really hard to quantify in the way that damage dealt is. On top of all of that, supports are also the primary targets for the enemy team, which means that they also tend to face disproportionately high pressure and need their teammates to protect them - if they don't, then the support might look like they're underperforming even though it's just because the Cassidy won't turn around and throw his grenade at the Tracer who's killed his Ana five times already.
All in all, though, I think it's really curious how people have responded to the lack of medals in OW2. It's one of the few instances of Overwatch actually providing more data than it used to, but it seems like it's made people feel less rewarded. It might be a topic worth exploring in a future video, honestly.
@@TheViveros oh yeah, definitely! That mercy example would have a lot more nuance than I presented, especially in something like the DBD meta. I think, as far as saving an ally is concerned, you could theoretically check total health vs damage taken over a specific amount of time? Be it an entire fight or a set window, it would give *some* data to the player. That said, a statistic like that would inherently ignore abilities like Kiriko's Suzu due to how it interacts with sources of damage.
A statistic based on if Ana hit a sleep dart on an enemy that is currently ulting (such as Soldier, Sojurn, Winston, ect.) could theoretically work, but it'd have issues. I thiiiink that'd be the closest you could get to quantifying a "good" Ana sleep dart, if only because I can't think of an instance in which it would be an explicitly bad call. Even then, though, it'd be giving black and white data to an ability with a spectrum of value, which could cause players to avoid less quantifiably 'good' uses and hampering their growth. That sort of data in the negative of statistics tends to make things tricky.
If it does make a future video, I'll gladly watch it, because that sort of statistic player psychology is super interesting to me
Oh yeah, there are definitely ways to track whether someone was saved (Marblr just made a video about that actually iirc) but it definitely still struggles with grey areas and what actually counts as saving someone or not. Like you said, trying to turn a pretty context-heavy situation into black-and-white data is really challenging and risks sending the wrong signals to players about what is or isn't a good play, which could cause more harm than good.
The irritating thing is that there *are* ways to quantify the contribution certain supports bring to a team, and they're metrics that are *already* being tracked by the game. Teammates saved are one - either through healing or immortality fields and suzus, for example. Kills by the team on anti-healed enemies are another: if an enemy is killed and they were purple, *clearly* the Ana did something right. Hell, even Mercy-boosted kills are a good indicator of the Mercy's skill - as well as how few deaths they have. And for heroes like Lucio and Zen, damage and kills *are* an objectively good metric.
The wisened old bronze damage players somehow entirely dodging the problem is fascinating. I wish I knew what drove that, or if it were to be accurate if extrapolated to a larger portion of the playerbase.
Data analytics is so cool
Super cool analysis! This is really fun to watch lol. One thing I felt like mentioning is one of your questions that you view as positive (i.e. “agreement means you accept responsibility for your rank”) stuck out to me.
The question “do more difficult characters have proportionally higher impact” is a question that I think is kinda neutral. Since the community tends to have pretty solidly defined characters that they think of as easy or hard (“junkrat is easy, widow is hard” is not a hot take) this is more of a statement on game balance overall rather than a player’s specific relationship to their main.
Disagreement with this question could feasibly mean something like “season 2 genji being bad is the reason I’m in silver” but it could just as easily mean “I found more success trying to rank up playing rat than I did playing genji.” That first response sounds like a cope, but the second one just sounds like someone commenting on game balance and the current state of characters that the community thinks of as “hard.” In this second context I don’t think I would say this person’s disagreement was an indicator that they blame external factors for their rank.
Obviously this isn’t like discrediting to the rest of this study, I just think it’s an interesting point where categorizing a response on a 1-7 scale loses a bit of the context of the question.
Super cool video though, love to see this type of super in depth analysis! Good luck with any more school stuff you got on your plate as well!
Thank you, I'm glad you enjoyed it! And to explain that one a bit further, the reason I included it as a positive response was because disagreeing with it would still reflect a sentiment that the game doesn't proportionally reward skill, whereas agreeing with it would reflect a sentiment that the game generally does. The way the question was worded was meant to convey that this wasn't about which heroes an individual had the most success with, but the general trend they felt like existed within their role - if low-rank players routinely said that they felt like you got more bang for your buck from lower-skill heroes, then that might help identify whether it's a cope or just a general sentiment. In this instance, the data showed that even though lower-rank players did tend to disagree the most strongly, almost everyone else also disagreed to some extent.
However, it also does speak to the flaws with this kind of analysis that I touched on at the end - it's important to be careful with how we extrapolate data, and to be critical of studies when they try to do that. It's also kind of an indicator of why it can be so hard to actually make a definitive statement, just because even when you *do* have data, it can mean different things. If I'd had enough respondents to be able to use hero pick data, I would've probably tried to see if there was a correlation between people playing the perceived high-skill heroes and disagreeing with the statement as well, since that would probably be the most interesting way to expand upon the responses from that particular statement.
This is incredibly well done. I really appreciate how objective you tried to be in your interpretations, and your acknowledgment of some of the study's inherent flaws. Still some really interesting data though! It would be cool if Blizzard would publish more internal data so we could analyze some of this further. Overall I would just love to see a similar study done on a larger scale to the point of being able to draw some true conclusions.
I'm so glad to see this! I was really excited to see the results of the survey as you had initially stated, but this is a really interesting result!
This is awesome!! Please upload more, I really hope this video blows up!
I feel like the support Dunning Kruger can be explained by a few factors:
1. Low elo players think that they're healers, not supports, and so if they heal-bot they should win.
2. Support has a lot of low perceived skill ceiling heroes, like my mains brig and mercy (masters 3) which make it seem like you're doing everything right and getting value when you're not maximizing them, making players feel like they're better than they are.
3. It is harder to counter the enemies as support. If someone on the enemy team is killing you it's easy to blame the team, because "it's their job to defend you", when clever positioning or counter picking could save you, too
Great video!
I've not agreed with absolutely everything you've said in videos in the past, but the idea of this study was genuinely fascinating and I can't wait to see what you do next.
Speaking of which, I believe one of the most important factors for ranked is that it is pretty much as good as a system gets - but people need to play at least 50-100 games in a role to reach their accurate rank as that is likely to be slightly (or a lot) higher or lower than they placed.
Anyway, keep it up this is awesome content!
Thank you, I'm glad you enjoyed it! And for sure, that's a unique challenge to trying to apply the Dunning-Kruger effect to something like this - in a lot of contexts, the assessment can be standardized even if the external variables aren't. If I'm testing to see how good a thousand people are at math, I can ask them all to do the same test on the same day within the same time frame in the same location, at least in theory, and I can rely on that data knowing that everyone started from the same spot in terms of the actual assessment. But getting a comprehensive assessment of your OW rank isn't like that, because some people will start a season at an especially low rank and will have to invest a lot more time in reaching the same level as someone else who's equally-skilled but was placed higher, and the amount of time involved in getting there can be pretty massive. If I were going to do something like this again in the future, I would actually love to see if variables like income level or rural vs urban living or average time per week spent on other commitments like school or work have any relationships to a player's rank, just to see if some of the more traditionally-recognized variables that provide advantages hold true in this context.
I waited so much for you to upload this video! thank you it's a very interesting study I had fun watching it till the end
Loved this video and all the details. Very interesting to see!
Very interesting study, the fact you have an actual ranking to compare the answers too is quute unique, i would imagine in later seasons with a more accurate ranking you might get more polarizing results. My personal experience confirms this effect because there is a level of knowledge about something that gives you alot of confidence about doing it, but there is a level of wisdom that comes much later on about how many different types challenges and difficulties that can happen or be present in doing a particular job, simply not knowing all the nearly infinite ways something can go wrong is a very tangible reason for this effect.
I hope Blizzard pays you to be a lead researcher for them and to do this study on a much larger scale, we probably need at least 100 participants per role combination (dmg-supp, dps only, tank only, etc) across the three regions for more reflective results
What a great idea for a project, and very well executed. I especially loved your analysis of the study's limitations. It's very refreshing to see that kind of stuff on TH-cam these days.
One suggestion I would have given is to collect data on the number of games played (games won is possibly a good enough proxy since I think it's annoying to get games played by role from your career profile). It would have been interesting to see the interaction of this with perception of elo hell. You would think that over more games people would settle closer to their actual rank, but maybe they instead get more frustrated and convinced they are in elo hell. I'd be curious to see. Elo hell itself is a complicated phenomenon arising from probability, perception bias, and perhaps some things like personal play styles and adaptability. There's definitely a lot more to explore in that direction.
Also, even considering this as a study of the Dunning-Kruger effect has the inherent assumption that rank in overwatch is perfect and every response that doesn't line up with it is wrong. I don't even think your data is really about the Dunning-Kruger effect. The data itself is just about player perception, and the Dunning-Kruger effect is just a possible conclusion you can draw if you make certain assumptions. This makes me think you might have been able to avoid deceiving your participants and still gotten the same data.
Those are just some suggestions. They're just my opinion and there's always room for improvement when you do a study. As it stands, it's still very enlightening. Great work!
I like how you emphasized that it’s ultimately up to us to decide what we take away from this video and your research.
I appreciate it!
OW and data analysis, 2 of my favorite things. Great video!
I don’t even play overwatch, and you’ve got me hooked. This is so well-done and immediately got me fully invested.
Only partway through the vid, but am already subscribing! I love your video style!
This needs so many more views, considering the amount of work put in and detail given to each set of data…
I am super late, and I've said this before in a past video, but I absolutely love how you implement statistical findings in order to enlighten the community. An actual treat every time I see you upload.
Just wanna say this is the last OW related content I expected to see pop up on recommended, and the amount and quality of work that went into this one video alone is incredible. +1 sub :)
This was excellent to watch. One of the topics I'm curious about with games like Overwatch are what types of impacts voicelines--especially pre-match multi-hero conversations--have on player mood and gameplay for the match. There's multiple opening hero conversations where heroes will argue, be rude, or outright call each other warcriminals and I'd assume there'd be a non-zero effect on players psychologically just hearing arguments in the background before going into a match.
That's actually a really interesting point! I hadn't considered it before, but there probably is some truth to the idea that the personality of the character that somebody's playing as will affect how they perceive the game - it would make sense that if the characters are arguing with each other that that might prime them to be a little more hostile towards each other than they otherwise might have been.
As a Marketing postgrad student who is studying units on research methods and design, this video is amazing. I have wondered why weren't fully eduucationally capitalising on how the statistical significance of data obtained from gamers.
Finally, my interest in video essays and my interest in overwatch collide and can watch a 50 minute video about human behavior and overwatch. Thank you, Viveros.
I think it's interesting that many of these collective perceptions seem durable, like the idea that damage roles have higher impact at low rank, support has higher impact at top ranks, & that tanks are very susceptible to balancing changes.
It's an ecological perspective which could be observed in many team-based arena games, like League of Legends.
Basic individual execution like aim or positioning predominantly limits early ranks, while knowledge, tactics, & effortful practice mainly define the middle ranks.
At the top level, near-optimal, instrumental play becomes presumed - damage & aiming is statistically unlikely to remain a significant limitation; complex coordination, tactical maneuvers, & cooldown awareness are widespread if not unspoken.
At that level, there are fewer low-risk opportunities to capitalize on, & so play-making abilities + slight numerical adjustments start to have significantly more consistent impacts.
I waited passionately for this video since the study post.
Edit: This video is better than I could imagine.
Loved the video, wish there was more thoughtful analysis videos on ow just as it’s gets weirder the deeper you go
Really impressive video! I found it very digestible but also very thoughtful, definitely enjoyed this combination of schooling and the TH-cam channel
Honestly loved this video as someone that loves statistics and I'm so glad you tackled some of the major biases that I figured were going to pop up in this survey such as how the fact that they were looking back on the season, meaning you weren't getting the data in the moment and the main one of how you were gaining it from your own viewerbase which would lead to the bias of extremes. On top of that, you made sure to even take into account the fact that Overwatch is, by itself, inherently skewed in how it handles its roles and the queues within them. You made sure to cover all of that and address the many questions that can arise from the data gathered as well as acknowledge that there really isn't a great way to continue into further research and the factors that create that fate; be it from how the knowledge of what you're doing the study over would skew the results or how it is nearly impossible to get a good percentage of the player base to participate given you have those willing to troll or lie, those that have yet to even finish their placements, or those that lean towards either extreme due to those feelings being what inspired them to take the survey.
You did fantastic on this examination, and it was a real joy to watch through it all.
P.S. I am a complete Stats nerd lmao.
What an incredible video, both for the main subject and study, but also for the coda at the end talking about the hidden flaws of academic studies. Great food for thought, especially as someone who reads academic studies on occasion.
as someone who did their BA about video games and is currently making their MA about video games too, you're so valid about the lack of good literature to work with
It's *brutal* out there, I feel for you. In the literature review I did for the actual study, I ended up actually using both of the articles that I referenced in the video to make the case that the study should go forward because articles like them were proof that there's not enough research being done in the field lol
I am subscribing so I don’t miss another survey. I love knowing this kind of stuff, especially about myself. Nice video
I never graduated college, and can't really claim to be an expert on anything related to data science, but to answer your question on what the point of the video even was towards the end, what I took away from your explanation of the study is that video games are a field of study that not many people want to do honestly, and one that can just be hard to study honestly. You made a lot of good points along the way about the pitfalls of various studies in the past, as well as the pitfalls you faced, and it gave me something to think about. Thanks for the educational and entertaining video
Banger video. Really interesting to open this up and get jumpscared by an actual academic study including an ethics section
LOVED this video. As a newer player and a gold support main, I think it’s interesting because if my team is doing better than the other team, then yes, I don’t feel like I’m affecting the game as much as a good tank or DPS. However, I sometimes play QP with a GM player and I noticed in match ups where the teams are equal in skill, if one support dies, that team essentially loses the fight. I guess in comparison to the other two roles though, it makes sense to me why supports feel like they don’t have as much of an impact.
Love this!! I also did a study at uni regarding gaming and absolutely agree that there needs to be more done ❤
This was a very interesting video! Thank you for making it ^-^
As someone learning similar stuff in the realm of environmental policy, it was great to see it applied to Overwatch!
Time for an empirical test of which role has the most impact: measure how long it takes streamers to finish their Bronze to GMs as each role. XD
Would love to see more content like this
I love data and numbers!!!!!! ^-^!!!! I enjoy looking at graphs and you explained all the conclusions i wasnt catching really cleanly. It would be so interesting to see this against season 3 end data to see how the meta does or does not affect the gap. How much of it is morale or smthin. Lovely video as always !
Thank you! I wish I'd planned things out so that I could do a bit of a "Where are they now?" follow-up to see if these people had actually reached the rank they thought they deserved back in season 1.
I think the point at the end about bias and the need for a critical lens on all studies was very well put. Its something very impotant to note that most people don't take into proper consideration. Great video, excited to see the video on Elo-hell if you decide to make it/ any other videos you make in the future
That was really cool! I did consider the bias when you talked about sifting through the data at first but I definitely didn’t consider the possibility that you were making this just to sound smart. To me it sounds like the dunning Kruger effect simply because of its nature is makes it very hard to pick out any potential why(s) from the data. Maybe there has to be a new way to see the effect to work for Overwatch.
Alright, this is impressive. I've been casually watching your channel for a bit, but now I have to subscribe!
I put this on as background noise but got hooked in this was a get interesting watch. I’m suprised my cooked attention span made it all the way to the end
OMG this is incredible. I am a huge numbers nerd and freakin love this analysis. This is amazing. Keep up the great work!
Thats the kind of suggestion i want from youtube, stupidly long vids about random subjects mixed together !!
This is a very fascinating study that I wish could be ran again over the course of several seasons with similar (yet larger) data pools to be able to reach a more conclusive answer. It's an interesting psychological phenomena that would be fascinating to look further into. Unfortunately, without more data over a more extended period of time, while also maintaining the illusion that this isn't what was being looked into. Blizzard being more communicative is also an interesting X factor that could skew data either which way, and it's not something that could've been accounted for. Especially with retrospect, Blizzard all but confirming there was issues with the matchmaking back-end can feed into the belief that ELO Hell does exist, and Blizzard mentioning that they acknowledge how unfun support is, and that they want to do something to address the issue, can also lead into feeding more biases. There's also other uncontrollable factors like what content creator(s) those people primarily watch. So many different pressures could be leading people to feel more inherently vindicated in whatever their beliefs and biases might be.
This is a word vomit, but tl;dr: I really like this study, I wish it could be done many more times to help remove some of the incalculable variables, and with larger sample sizes. Having a more definitive answer would be such interesting information to have when it comes to tackling issues as a community, and approaching situations as a whole. And, in a more larger scope, how this data could help provide more proof that gaming in general is worth more investigation; particularly by people who are more on the pulse of that demographic.
that was a suprisingly good video, very informative
I'm 1/2 way through the video, but my theory on why Silver players are more likely to think ELO Hell is real is: since they're NOT Bronze, they "know" they're not 'bad' at the game, so it MUST be ELO Hell (and 'sh*t teammates') holding them back.
That's an interesting angle! I think there's definitely a grain of truth in that - if you're bronze, you're at the very bottom of the rankings, but if you're silver, you know that you're not the *worst* in the game so there's gotta be another reason why you can't climb.
Thanks for the thinking piece. :]
Love the Fusion Frenzy music in the background.
I feel like this could be so cool with an even bigger sample size! good video!
Also what a tumbnail
Thank you! And yeah, this definitely would've benefited from an even bigger sample size - part of the challenge with a study like this is that there's no way for me to identify people who might be prospective respondents and target them with the survey, since the game is played online and there's no way to identify an OW player out in the wild or anything. People had to come to me rather than me going to them, which put a bit of a ceiling on the sample population.
I really enjoyed this, thank you. The data makes a lot of sense in many ways.
Another thing to consider asking would be if it is that persons goal to climb in rank. I play for fun and to win, I am playing in comp after all and want to be a better player, but I don't necessarily want to climb in rank due to matches combining people across 12 (now 9 apparently) skill levels in a single match (to reduce wait times) and this would affect answers to these questions.
I really enjoyed this, and if you decided to do this study again I'd be interested in asking about who played Overwatch 1 and for how long, bc that could inform their perceived skill. For example, i was pretty solid plat for the years I played OW1, but got put in silver when starting up OW2. If I had filled out that survey, i likely would have put my OW1 rank for my perceived skill.
I actually did ask players if they had played OW1 and their total account play time, but since almost all the respondents said yes it didn't yield super useful data. Plus the game didn't allow you to access your old OW1 ranks at the time, so I didn't want to ask for those because it's possible people wouldn't remember if it had been a few seasons since they'd played comp and it might throw off the data. I only wanted stats from people where I could be relatively confident that they'd be able to give me the exact numbers.
Really liked the video, learned a lot from it and definitely changed my perspective about the Dunning Krueger effect. Even though this is still not a "professional investigation" I feel as though you did prove the presence of the Dunning Krueger effect in Overwatch 2. It's a shame that I can no longer participate in this type of study, especially because of how the meta has shifted. All supports currently are somewhat viable, and I'm sure the answers from supports would have been different. (I’m a support main so maybe there’s bias, idk)
Awesome video! Missed the time window to participate but an amazing video nonethless & I hope to get in on the study next time LOL. Also yes I knew the pokemon video was going to win. That thumbnail was something else LOL.
Love your voice, helped me fall asleep.
Love this video, truly. I get fairly salty when playing Overwatch, however I've noticed that most of my anger comes from the scoreboard of all things. If it shows a vast difference in kills or deaths, with my team on the lower end, I get mad. Or If we lose, but I have significantly better stats than my team, I get mad. Even though I know that kills and deaths aren't necessarily a great indicator of performance It still frustrates me, makes me claim poor matchmaking or what have you.
Nice! Wish i was playing during the study window. I hopped back on for the rammatra release
This video was so well made! We need more educated content like this in the overwatch community!
This is a fascinating video with a lot of wonderful analysis and clarity, so kudos to OP for putting in so much effort. but? I'm a little surprised that the bronze 5 bug wasn't mentioned in the limitations section. Even though the bug didn't impact many players, it wasn't fully patched out until Nov 17 (42 days into a season that lasted 61 days). The researcher acknowledged other limitations like selective sampling, grinding time, and language bias, so I think it's odd that the ranked bug wasn't included as at least a footnote. Great video tho-- I liked the in-depth methodology breakdown.
The main reason why is that it was only a placements bug. A bunch of people did initially get placed at the very bottom of the rankings when they shouldn't have been, but according to Blizzard, it only affected those initial placements - players who did randomly get placed way too low were still able to climb out of it pretty quickly by all accounts. It also wasn't as nearly as widespread of an issue as Reddit and Twitter hyped it up to be, and given that none of my respondents mentioned it as affecting them anywhere at any point, I don't think it was a big factor in their responses.
@@TheViveros That's understandable reasoning, but you're mistaken about it being a placements-exclusive bug. In October 2022, Blizzard claimed it was a placements bug (and, like you said, many players were able to climb out of the incorrect placements quickly). But in November 2022, Blizzard specified there was still a bug where "some players could be stuck in Bronze 5 even after several rank updates". Blizzard never confirmed how many users were affected by the bug, though it was likely a very small portion of the player base that got over-exaggerated on Reddit+Twitter copium. Yeah, it's unlikely that your participants had the Bronze 5 bug (and even if they DID experience the bug, there's no way to prove it), though I still think it could've been a relevant limitation to mention.
Sick video dude!
this video made me have a dunning kruger on dunning kruger (i thought i knew more about it)
I remember learning about the D-K effect from Drift0r vid on it a long time ago, love to see an expansion on this other than CoD
Man this video is AMAZING, gotta say didn't think anyone would do a actual study about overwatch, specially without it being just a "games are healthy/ good for you" or "games are bad/unhealthy for you" kinda thing, but this one was interesting, well done and actually brought quite some interesting things to the table.
I particulary thing that the sample being from twitter and youtube influenced how people voted on how each role performs in comparison with each other, as it is a common agreement, specially with content creators, that tanks have the most impact and that support are the hardest to actually get value from (at least from the content creators I watch, there could be some bias of course)
Gotta say tho, didn't expect bronze dps to be so self aware, as that tends to be the role most know for thinking everyone else in their team is bad (stereotype that doesn't match the data) but to be honest and not that surprised that support is the role to feel like their rank are wrong but they just can't win, as someone who plays all role on qp (not a competitive person) I found that support, my main role in ow1, became the one I played the least, because it felt so hard getting value. and being honest just not dying, while playing any of the characters I found fun. Now that I'm getting used to the 5v5 (and with hog dead to the ground) I feel like I can put out value while playing support, tho it feels like i have to try harder then when I'm playing Ana as to when I'm playing Ramattra, for example.
I think the basic Dunning-Kruger effect will always be there, and without a big change in the way the community feels about supports the role disparity too, but as the seasons go on I feel it will get weaker, seeing that adapting to 5 v 5 was a very hard hit on the community.
5:50 OH MY GOD HE WENT BACK TO TAXI THE MERCY. Praise.
DPS's low level of dissonance between skill level and rank doesnt surprise me very much. Damage has the most obvious and immediately actionable feedback when compared to the other two types. They arent perfect metrics or anything, but you immediately get to see how much damage, how many kills, and how many assists you have in a game. Because you have the easiest to understand feedback information I feel like DPS are gonna have understand the reasoning they are in their rank better.
very interesting, you should do a proportion z test to see whether or not the differences between the bars are significant. Perhaps assume they should be equal and calculate the p value, then put an asterisk next to the bars that deviate significantly. Also, you might want to randomize the question order if you collect more data in the future. good luck with your studies.
I love content like this, as a tank/ support player I absolutely agree with this data with regards to season 0 competitive
Awesome. This video checks the all boxes of my interests lol.
I love how many layers there can be to this. I found it was interesting to consider the impact the heroes played at each rank may have on the study results, even if there wasnt enough for the vid. It could be argued that a larger influence on the outcome of a match in lower elos is hero matchup rather than player skill. In most low rank matches youre going to see the healers fragging way more than higher ranks, which i know all too well comes from the thought “my dps arent doin shit here.” Granted, healers also see the most impact on an organized team, and considering most lower rank players dont use mic and tend to solofeed you can absolutely see why theres such a shift in their perception at the highest rank
This was super interesting! Like you say, I don't think we can draw any meaningful conclusions about the state of the game based on this data, but the act of raising additional questions is valuable too. And as far as the Dunning-Kruger effect itself: my interpretation is that, broadly speaking, confidence has practically zero correlation with ability. In fact my guess is that it's instead determined by general heuristics like familiarity, availability, and so on. In other words, our brains are actually wired to measure confidence based entirely on how a task makes us feel, regardless of the facts.
very cool vid! Ive done some overwatch stats analysis with overbuff, they have some tasty numbers
42:37 Wow, that widow snipe was pretty gross, we should nerf gengi to fix that.
everyone sleeping on the absolutely raw transition at 42:02
I used to think ELO hell was real, but i eventually realized that i wasn't thinking about my gameplay or considering what i could do better every match, i was just playing mindlessly and feeding
Man, I wish the sample size was larger, but I guess that's the life of surveying.
For season 3 this would be especially interesting both because of the meta changes (general sentiment seemingly being that the balance is better currently) and because of the matchmaking changes.
Personally speaking, my experience was not playing the game for 2 weeks at the end of season 2 because of sitting on my rank and not feeling too well, and then suddenly climbing, without being punished for losses already after the initial boost which seems to go away after 3/4 rank updates, while even going, say, 5-10.
I am now 2 tiers above what I consider my actual skill to be on a good day, and 3 tiers above what I consider myself on average, while I feel like I'm playing (at least mechanically) worse than I used to before the break (and I had accumulated over 200 games played in the previous season, while trying hard and attempting to be consistent - thus I consider that season's final and peak rank to be accurate)
At the same time there's talks of "fake GM' players who came to GM possibly because of playerbase size inflation, which I don't want to become (but am close to), and there is a good point that it should be logarithmically distributed at the top of the top instead of normal distribution (which hides the difference between "4200" and "4400" and "4600" players, which, according to constant T500 players, should be looked at as leagues of their own - but we no longer see that at all).
The influx of players both new and returning, which would fill the lower skill tiers, may also be inflating the mid-higher ranks, with, perhaps, the higher ranks feeling this impact later, as it takes time for the lower skill players to back down into the appropriate tiers, and thus impact the boundaries of each tier. This would, of course, differ based upon how the tiers are implemented.
And at the same time, the matchmaking has been changed to match people differently. This can boost some skillsets more than others indirectly, where a skillset is to, for example, punish the players on the other role who aren't as good, while your enemy might be good at dealing with players on the same role but not punishing other roles as much.
Hell, I don't even play overwatch, I'm just here for the data science.
I really hope someone with a larger platform will team up with you to potentially open the study up to a much wider audience to get a larger sample size because I LOVE this kind of thing and would be interested in seeing a more conclusive and comprehensive study done
Get Game Theory on board 😂 that would be a great way to continue this work without all of the responsibilities falling on you while you're still in college
i would love to see this study carried out at a larger scale. it’d be cool if you could partner with a bigger OW channel to get this survey out to a bigger population. also i feel like season 0 of overwatch 2 was unique in that they recalibrated and reset everyone’s rank. i was placed in silver at first after being in plat/diamond my entire career of ow1. so in this case i would definitely say that i was under ranked. by season 3 with more playtime i’ve worked my back to diamond in all roles.
Really interesting and well produced video. What is your academic backround? (What i missed is a detailed Methodology chapter in your video apart from the Backround Part 1)
Nice editing transition at 11:20 lmao
really great video. I'm impressed with your analysis and i'm inspired to do something similar :) also, isn't there a way to measure how heavily one person plays a role, measuring the amount people lean towards support/tank/dps?
Thank you! The problem is that the stats OW provides are your total games played in a season across all roles, your wins across all roles, and your wins in each role - without knowing the total games played in each role, you’d have to make a lot of assumptions about a person’s winrate in each role, which I couldn’t justify doing. If someone had a 50% winrate across all roles, and won 5 games in support, there’s no way to know if that 50% winrate applies; they could’ve won 100% of their support games and just lost 5 more games in DPS or tank to cover for it, or they could’ve played 50 games on support and only won 5, but then had an absolutely dominant run on DPS and/or tank. Even if you try to piece it together by looking at the total games played per hero in a role, you’ll still get an inaccurate number because playing Zarya, D.Va, and Winston in a single match will count as 1 match played for each of them on their respective pages.
Tldr is that unfortunately without people actively keeping track of their games played per role, there would be too much guesswork involved in trying to figure out a player’s distribution between the roles.