Does Clear Rate ACTUALLY Matter in Super Mario Maker?
ฝัง
- เผยแพร่เมื่อ 4 พ.ค. 2024
- WATCH ME LIVE ► / raysfire
follow me on twitter ► / raysfire
join my subreddit ► / raysfire
join my discord ► / discord
follow me on instagram ► / raysfire
In this video, I do a deep dive into the world of Mario Maker 2's level selection algorithm, with the hopes of figuring out exactly the game chooses which difficulty each level goes into. To do this, I did a series of experiments, as well as did a deep dive into some of the fringe levels in Super Mario Maker 2 that don't really make much sense on the surface.
This video would not be possible without the help of @DSteves, he helped immensely on this project, from giving me ideas for experiments to run, giving me the average clear rate and WR times, to giving me a list of the top 1,000 highest and lowest clear rate levels of each difficulty to help aid the research I did.
Please check him out here: / dsteves
versus ► bit.ly/raysfireVS
trolls ► bit.ly/raysfireTrolled
challenges ► bit.ly/raysfireSMM2Challenges
collabs ► bit.ly/raysfireCollabs
road to s+ ► bit.ly/raysfireSPlus
ALL super mario maker 2 ► bit.ly/raysfireSMM2
➡️ sub 4 more content ► bit.ly/Sub2Rays
👕 buy merch ► store.raysfi.re/
📹 edited by ►
me :)
#raysfire #Highlights #SuperMarioMaker2 #SMM2 - เกม
to see the full, unedited "clear rate" experiment, check it out on my vod channel here: th-cam.com/video/JTA0wkdl9M0/w-d-xo.html ✨
also, big shoutouts to dsteves for all of the help with this project! a lot of the data and experiments i used in this video wouldn't have been possible without his help!!
check out dsteves' channel here: www.twitch.tv/dsteves 😎
Nice video btw
Great vid
nice vid keep up the good work
It would have been cool to go back in time and track the average clear percent over time. I have a feeling that since the game released, less people are playing and the people that are still playing are really good at the game. So perhaps the levels got harder to accommodate that crowd thus the average clear percent would go down over time.
@@cak5171 Check around 8:59 to see that DSteves cleared K. Wonka's level, too. XD
So what we've learned is that we need to throw when clear checking automatic levels to help no skip runners.
Comment from one verified user with 9 likes placed over multiple comments of unverified users with so much more likes? Clearly, TH-cam is falling apart in terms of ratings of videos.
yo
And also throw triple shell jump, mashing, and near-pixel perfect precision levels
@@Fowlware fun fact: likes are not the only thing that TH-cam looks at to bump a comment.
comment placement is decided based on likes and how often the user has a interaction (and maybe subscribers)
I think the initial difficulty seeding being higher than expected makes a lot of sense. As the creator, you know all the tricks needed for a level, so it's easier for you than it would be for a new player. Having the initial seed be biased toward higher difficulty is actually pretty smart game design.
Yeah. I always wonder how one of my levels was initially placed after upload since it took me one attempt to clear check it, but found out later that most people who played it struggled and gave up.
That said, I still feel like you could get a better starting estimate by taking into accounts the # of failures before a successful attempt when clear checking it. It would make it so that harder to clear levels tend to be higher difficulty on average.
And yet I've made many levels that took me dozens of tries to clear check and yet they have a 5%-10% clear rate.
Many creators aren't the best or most skilled and yet make levels that test them to their limits, whereas there are significantly better players out there that are so much better and can beat the level easier.
Beyond that, in my experience most levels are straight forward. No tricks, tech, or lore needed. Just standard levels.
I think the point you brought up is very cogent in troll levels, little Timmy levels with random hidden blocks, and very hard levels by amazingly skilled players/creators. For the rest of levels, which might be the larger portion, it doesn't necessarily hold true.
Initial clear check just shouldn't affect difficulty at all, actually. Bad game design.
@@Syranovara If initial clear check didn't matter at all, it could be that most of the people would never get their levels in the random pool, because nobody could find them in the first place.
In SMM2 you need to get at least 1 play from somebody else before you get a new level into the random pool, and as a rookie level creator finding even that first player can be tough. I'm not exactly sure if the first play needs to finish the level.
Then again, when writing this it occurred to me that they could just use that first other player's clear rate to recalibrate the difficulty before putting the level into the correct random pool. They might actually do that, this video doesn't really confirm or deny anything related to that.
One of the things we also talked about briefly was if levels were reuploaded exactly the same way if a level would differ in initial difficulty, then after the video was in production I remembered about title screens and how I've seen the same level in easy and normal and sometimes even expert, the spike test now does show that this is the case. All in all the goal is to find some sort of algorithm that combines all the pieces together. While initial difficulty on clear check is one piece of the algorithm it's figuring out a way to combine that, possibly the clear rate, and the ratio of who flagged the level as cleared over who has not, as well as figuring out any other overlooked part of the equation that is the key here to getting a sure fire algorithm going.
Since the number of tries it takes the level maker comes into play, I wonder how that has affected the categorization of my own uploaded levels. I often quit a clear check and nerf my levels a little (because I don’t have the patience to beat them lol) until I end up beating them on the first or second try after failing to beat earlier versions of them. So I wonder if Nintendo factors in ALL attempts on a saved level, or if the counter starts over when the level maker quits a clear check and changes the level slightly. One of my levels like this has a 4% clear rate, but I ultimately beat its final form in one try (after failing to beat earlier versions of it). So I wonder which endless mode it ended up in initially because of that.
Thank-you! I’ve been on the lookout for resources that would allow me to see stats on my own levels. My curiosity can now be satiated.
Hey man! It's me, hamstercol from twitch, how are you?
I think that the algorithm might ignore any clear attempts of a person after the person beat the level for the first time.
Another theory I have is that clears of different persons count differently based on some internal skill level value of a player. Somewhat similar to ELO
Hi biggest maker
all songs used are from the kirby 64 ost:
0:00 - Mountain Stream
1:23 - Pop Star
2:51 - Rock Star
3:36 - Aqua Star
5:15 - Above the Clouds
5:57 - Boss Theme
6:45 - Neo Star
7:54 - Bumper-Crop Bump
9:47 - Ripple Star
10:49 - Shiver Star
12:29 - Factory Inspection
13:09 - Training
Thank you
based
Would’ve been a sin to leave Kirby 64 out of this 😝
Chad
The Kirby series has the best soundtrack, with the classic Sonic games being the second best.
I'm very curious if skips also factor into the Nintendo algorithm. For instance: take the puzzle level that was in Expert yet had a 17% clear rate, or the exceedingly long levels in Super Expert. Both would likely have very high skip rates, as many people simply do not like puzzle levels or lengthy levels. Someone may die once in a 2 minute long level and skip to the next one rather than attempt it again. This wouldn't impact the clear rate much, but it may have a bearing on the difficulty level.
Just a thought!
I had this same question but typically a lengthy level already indicates its difficulty. So if we take that into account I surmise they may have a separate yet similar algorithm to deaths that involve skips. Since every skip would be unique to a level, if a player didn't participate in the level there must be a reason why(which I would surmised levels in Expert and Super Expert get skipped over 5/10 to 9/10 times respectively)
If the player never clears the level like exit the software, it won't save like if you never played it, also The clear rate of 199,000 out of 400,000 is only showing every player's attempt to succeed, all the times the characters looses a life it counts as a failure point(meaning that is where the player's character fails there because of enemy, hitbox of icicle or liquid hazard),The shoe icon only shows how many people played it, and also the clear rate is everyone's attempt to reach the goal point there.
Never overestimate the clear rate if the player doesn't know what to do, give it a chance if you think you can complete it.
I had the same question while watching the twitch vod and I was really hoping someone would have brought it up live in twitch chat.
If I had to theorize, skips wouldn't affect difficulty as strongly as deaths but it would make sense to factor them in to a degree.
On one hand a skip could mean the player died once or twice and determined it was too difficult and skipped it or came across some difficult tech or some precise jump they knew they couldn't do so they skipped it before dying at all. In those cases the skip would represent the amount of difficulty the player WOULD have had if they had kept playing until they cleared the level. Either a great amount of time in the case of difficult tech without a way to die, or many many deaths in the case of precise jumps.
On the other had a skipped level may not be difficult but instead be too long, boring, or super annoying. The player could easily beat it, they just can't be bothered wasting their time on something they aren't enjoying. So these skips would be the opposite; they represent a hypothetical first try clear rather than an amount of deaths or struggle clearing the level.
So basically skips represent an unknown variable in the difficulty because we can't know the reason for every individual skip. In a hypothetical world where skips didn't exist and a level had to be played until cleared, a level's clear rate could potentially be drastically different than same real world level.
The more I think about it, the more curious I am on just how the algorithm DOES consider skips. Does have a way of predicting the reason why a specific player skipped a specific level? Does it gauge the player's own skill level based every other level they've attempted? Does it consider the amount of spent time in the level before skipping? Does it just give all skips a blanket weight and be done with it? Or did the programmers just say "hecc it" and leave skips out of the algorithim altogether?? 🤔🤔🤔
It's so interesting to think about and we will likely never 100% understand how the algorithim works. Unless someone at Nintendo leakes it out...
I think this would be grouped into the how many clear vs plays it has. Not the clear rate, but how many DIFFERENT players actually clear it. If the theory that levels that get cleared more often, even if the clear rate is low, are easier in difficulty, then skipping a level and never going back to clear it likely does increase the difficulty rating
What would be interesting is if they factored in player experience. An experienced player dying on a level could indicate harder level. An inexperienced player 1st trying a level could be strong evidence the level is easy
That was one of the things we were looking at also, but unfortunately there wasn't an easy way to test that - that's definitely a possibility in factoring in level difficulty though!
Probably wouldn't work as I am good at smm1 (so the skills carry over) but i have barely played smm2.
Agree with this. Someone who has beaten a lot of Super Expert levels should have a different weight in how a level gets classified from a newbie and how hard the level is for them. The algorithm should be looking at the reputation of WHO beats the level.
@Raysfire If the SMM2 scraped datasets contain the players who cleared each level, a correlational analysis could maybe be performed, but obviously this is well beyond the scope of a Twitch stream :)
@Bezray There would be outliers, but with enough data it would average out. And they could weight players less if they haven’t beaten many levels
I also think "skip" effects it too.
Lets say ur in normal, and the level LOOKs like a normal level, the player wont skip right away.... they die... but they dont skip... they still think its within their reach.
so many deaths, and a clear, or many deaths and a skip.
vs 0 to a few deaths, and a skip, which means the player might just be fishing for easier levels for their record.
so a low clear rate, but refusal to give up, might mean the level is considered to be in the right difficulty range. (along with likes and dislikes too)
The thing that the high clear rate super expert levels had in common is that none of them had any way of dying other than the time limit, so many people getting the level and skipping affects it (since skipping without dying doesn't count as an attempt)
Something to consider: tags could also potentially have an influence. Many of your outlier examples were puzzle levels.
Thats basically 0% chance. There are so many levels that some are going to have the same tags as other but be completely different difficulties. There have to be some easy levels with the puzzle tag
I think the issue with puzzle levels is that people just skip the level if they can't solve the puzzle, lowering its clear rate
my theory has always been initial clear rate determined by creator attempts, and then final clear rate determined by average attempts:clears per person. If 5 people have a 3:1 clear rate, and one has a 100:1 clear rate, then the average is closer to an easy level
I figure your deaths to set the difficultly are probably reflective of the deaths per completion in general. Easy levels are ones where people generally win on the first or second try, normal on the third or fourth, and expert more.
The reason the clear rates are different is because of people who skip after failing the first time or who retry for world record.
Only people who go on to clear the level or who die more than the expected attempts factor into the difficulty.
This would explain why some levels that force a death to respawn from a checkpoint appear in easy or normal despite having ~10% clear rates, the levels consistently get 1-2 deaths per successful attempt and almost no-one dies more often(because it'd need timeouts or similar), but a lot of people quit when they realise the level is a net negative to lives.
And now all I need is to get Mario maker 2!
YOU'RE GOING INTO SUPER EXPERT, SINGULAR SPIKE! This could actually be used to create break-levels for the harder difficulties. You know, give 'em a free win.
Hello, I am Mr. Fusion Man. There are no fusions in this video. I hope a versus challenge comes soon or I'm gonna be out of a job.
Lol
Idk the twister and galoomba at 0:15 - 0:19 seem pretty fused to me
@@rorycannon7295 sadly, the twister is closer than the galoomba, they are not fused.
@@EnderrBoi i mean they could be shadoingled along the z axis?
@@EnderrBoi but tru
I made the "Level 49*", thanks for playing and including this level in your experiment :)
I'm so glad you like the level.
Regards.
So that's why I could play endless easy and end up on a brand new level that's incredibly difficult, only to find a dev door several attempts later.
I got that first ballon level in SE endless too! Also I lost all my lives because I kept having one coin left and the clouds left. Also thanks for this video I’ve noticed too and have been wondering! Sometimes I would get a level with one person who played it, and I don’t know how it Nintendo knew for sure that it was hard. It’s very interesting.
12:00 you can also try to match the WR time for easy,medium,expert,super expert
Like wait 120s to finish the lvl for super expert.
Idk tho
Very insightful into the algorithm, thank you.
I would like to clarify the conclusions we’ve drawn though- the only thing we’ve proven from our own experiments is that creator clear check attempts noticeably affects the algorithm’s placement.
The data science studies were useful, noticing trends of lower clear rates and higher world records tending toward higher difficulty placements; however, we cannot say that the algorithm necessarily checks for these things.
In order to show causation we would have to do experiments similar to the spike test but with multiple other players/accounts, keeping the creator clear checks constant.
Through the power of SCIENCE!
The difficulty rating of chess puzzles can be calculated using a modified form of ELO from the players who succeed or fail at it. It would make sense for Mario Maker to use a similar system
This is great Rays, your content is so good. This was super interesting, and something I've always been curious about. It's great to see some of these questions answered!
other ideas i can think of for possible factors might be things like hazards in the level, level size, time spent creating the level, amounts of inputs on clear attemps (for example just holding right, holding right and jumping some, or having something more complex than that)
also maybe the level creator's previous levels have an effect?
I’m glad you overcame your pc issues to upload this masterpiece! Love all the content you’ve been putting out
Rays this is an amazing video that you clearly spent a lot of time making. I loved the creative style of it! Keep up the amazing work keeping everyone here entertained and informed on everything Mario maker. ❤
really great video! was interested throughout the whole thing, and loved the methodology used to discover things. keep up the great work :)
This was an amazing video, as you were explaining what your experiments were and what they ment I also found my self think "well what if..." "then that means..." and its as if you answered my thoughts! Kinda creepy XD but great minds think alike though.
The script was awesome and kept me intrigued and interested the whole way through. I also loved how you showed the stats, really helped me follow along.
Was worth the wait, I hope this video gets into the algorithm, it deserves it!
Hey Rays. Loved the content. It's really nice to see someone put so much effort into their content. Keep up the good work Mr Shud :)
Holy quality batman, this is so much different than what I remember from watching you a year or two ago. And I'm all for it!
The way that all these new videos are amazing. Keep up the good work!
I have always wanted someone to make a video just like this, thank you Raysfire!
raysfire's content is top notch. you should have way more subs! keep up the good work
Wow this was an amazing video!
I think I have to throw when doing clear checks on my kaizo's I build. I practice them so much in the editor, that I clear check them in a couple of tries. Then they get put into categories where people won't possibly like or be able to finish them. They're dead on arrival.
Thank goodness for the sub! I forgot about Quiz day! Awesome Vid Mr. Shud! It's really interesting that the creator in a sense can partially decide where their level will show up!
Spicy video rays! 👀 And awesome editing, loved the graph :D
Wait so does that mean it is possible to get a super expert difficulty level on easy if the creator beat it first try? My that sounds scuffed.
Anyway, great job on the video. I appreciate the amount of time you've put into research.
These new vids have been really great, you deserve way more attention man
Love the video. The whiteboard needs to be featured more. Possibly for the Mario Party CPU tourney?
I wont lie Rays, Ive been loving this new type of content on your channel. Keep it up 👍. Also no test for me baby WOOOOOO
I’ve always wondered this. Such a good video idea and well executed
Woah very cool experiment! Love these videos, always good content and high quality!!
My theory is that it starts out as the creator's clear rate, but after it's uploaded, it doesn't just determine the difficulty primarily off the level's clear rate, but instead the game might do 2 things:
1. Check for player outliers via their individual clear rate, and find the clear rate without those outliers.
and/or
2. Take the average time to clear into account.
This is actually really interesting how you can see super expert levels within the expert category - if the creator of the level of their level practices segements before attempting to upload it they will usually clear it relatively quickly possibly allowing the level to get misassigned
I love how Rays is truly just a lad living his life
such a high quality vid earned my sub good stuff
One of the coolest raysfire videos yet! And the use of the Kirby 64 soundtrack makes it all the better!
@raysfire This seems like a good machine learning task. Have you tried grabbing a programmer who knows a bit about data science and get them to run a regression algorithm on some of the data you collected? It could be a simple linear combination of some of those variables you identified.
If I were developing a game and wanted to check clear difficulty (and had lots of time) I think it'd be really smart to take a recording of the user's clear and try to mutate it by throwing in random delays / extra button presses / skipping button presses to get a sense for how "precise" the level was. I really doubt that's what they're doing though (it'd be expensive / take a long time to upload a level most likely).
this was really good! We need more big brain videos
Man, it’s so weird hearing you make documentary-style videos. I never thought they were your thing. But I think you do really well, I enjoy these videos!
My top 3 hardest levels:
1.Porcupuffer's Crate Clash
2.King Bully's Mask Mayhem
3.Yoshi's Treetop Troubles
It took me F.O.R.E.V.E.R to clear those 3 levels
Honorable Mentions:Chaos Chasm
Amazing video rays! Thanks for the science!
these videos just keep getting better and better
I always look forward to your videos! Keep up the awesome content :)
Thanks for this I shall use this knowledge to make an Easy level into an Expert level as to try to get on Panga's stream.
scanning for shudheads 📡
shudhead located 📍
All right, I study mathematics and the graph at 13:10 ringed a bell: a hyperbolic curve. If this graph is the way things work behind the scenes, then you can tell that the _product_ of the number of attemtps and the time per attempt is what counts - which is *the total time spent for the creator to beat the level.* I would love to have this theory tested.
Amazing video! I highly enjoyed it. More science, please!
This is really neat! Thanks for the science, rayfsfire!
I have once encountered a very hard course with a high clear rate. It was an SMB course which was very short (one screen) and contained a giant pit filled with lots of boo rings and falling goombas and I had to jump on the goombas to the flagpole while avoiding the millions of boos, which is extremely precise. I couldn't complete it.
epic, keep up the great content.
I really hope I can skip that test on thursday! I subbed, I promise!
Great vid!!! hope this one pops off just like the last one
These mario maker 2 vids r fire ngl. Idk why but i just started binging them 😩😩😩
this video is really good, and the kirby 64 ost just makes it better
Hype new rays video, hope it goes as well as the last couple
Outside of your mutiplayer with red and panga this type of content is some of your best. Keep it up!
I remember playing a level never beaten before or two in normal mode. Always wondered how that worked
i remember this level was used by someone who made a video about how the clear rate works, and yeah basically if you die a ton on your level while trying to beat it to upload it, it gets ranked harder
THE WHITEBOARD AND LAB COAT REVEAL! I don't know if imma pass the test on Thursday? Is there any retakes?
It may be half 11 in the Queensland of England but rays has priority over sleep
Always a good day went you got content from ya boy rays
Ty professor rays very insightful
Brilliant video!
Great video! Super interesting
Very good video! Keep it up💪
1:36 Spindel
I always have great respect for Dr. Shud
I got a "First Clear" in Super Expert once. The stage was literally so easy
2:15 I look in newest and see like 15 comments saying spindel but nobody said
beeBobble
There you go, AComputerMouse
Thank you for the lesson, Professor Shud
BANGER VIDEO RAYSFIRE ‼️🗣
Okay I have studied all night for the quiz on Tuesday, I think I'm ready
My theory is that the initial difficulty also depends on how many attempts on average it takes for the creator to beat any level.
Can't wait for the quiz on Thursday!
Worth the wait! Get this man a new chair!
Can you do this exact video but for how a level gets played in the Multiplayer versus mode?
I love videos like this
The clear rate is when the every player who attempts succeeds the level or if they fail/or character lose a life or fails to reach, also that is the total attempts of everyone even the same person can play 141/150 times and if another player plays and clears the level, that person will add to the numerator. The shoe marker is the amount of total players played the course currently before you/we do play the course.
First clear is the first person cleared the level, and the world record is anyone who did the quickest way to complete the level or the most efficient; and well played and honor for everyone who skillfully came close to anyone's World record, or matching theirs, even if its 00:005 around 00:03.45 seconds difference(or very close to WR)
I have a counterpoint to the clear rate for creator thought. In people's expert and super expert levels, some people put dev doors or dev exits to upload it because they don't want to do it. By this, the clear rate is 100% or close to it.
By default, these should be put into easy or normal. Therefore, without testing uploads, we can see that the creator clear rate does not have a "normal" impact.
When you see your tool being used in a video: 😇
i feel like this guy had a science assignment where he could do whatever topic and chose this
Kinda related, does anyone else notice a huge difficulty spike between normal and expert? Like a "hard" difficulty between normal and expett. I'm not the greatest player but to me endless normal is a fun challenge but it's getting less fun as I improve. But when I try expert, I'm lucky to beat 2 or 3 levels with skips. Sometimes I don't beat any. It just seems like the difficulty spike is too steep where maybe having a "hard" difficulty could possibly help.
Very cool video Rays
Thanks for validating me in my feeling that normal was way harder than easy.
A banger vid as always
1:35 spindel
Wonder if setting initial difficulty depends on your record as a player, aka fresh account vs somehow who played a ton of hard levels
amazing video, as always shud!
Here's EricVanWilderman's explanation: The creator could have overlooked something that could be massively cheesed. And almost all the players sees it, does it, raises the clear rate and it will move to a lower difficulty.
Here's Ryukahr's explanation: Players and the crator oneshot multiple times a level, raises the clear rate and it will move to a lower difficulty..
Ngl this is a really good video
Damn Professor Rays at it again with another banger. Spindel
you: a test on thursday
me with a uno reverse card : no u