to see the full, unedited "clear rate" experiment, check it out on my vod channel here: th-cam.com/video/JTA0wkdl9M0/w-d-xo.html ✨ also, big shoutouts to dsteves for all of the help with this project! a lot of the data and experiments i used in this video wouldn't have been possible without his help!! check out dsteves' channel here: www.twitch.tv/dsteves 😎
It would have been cool to go back in time and track the average clear percent over time. I have a feeling that since the game released, less people are playing and the people that are still playing are really good at the game. So perhaps the levels got harder to accommodate that crowd thus the average clear percent would go down over time.
I think the initial difficulty seeding being higher than expected makes a lot of sense. As the creator, you know all the tricks needed for a level, so it's easier for you than it would be for a new player. Having the initial seed be biased toward higher difficulty is actually pretty smart game design.
Yeah. I always wonder how one of my levels was initially placed after upload since it took me one attempt to clear check it, but found out later that most people who played it struggled and gave up.
That said, I still feel like you could get a better starting estimate by taking into accounts the # of failures before a successful attempt when clear checking it. It would make it so that harder to clear levels tend to be higher difficulty on average.
And yet I've made many levels that took me dozens of tries to clear check and yet they have a 5%-10% clear rate. Many creators aren't the best or most skilled and yet make levels that test them to their limits, whereas there are significantly better players out there that are so much better and can beat the level easier. Beyond that, in my experience most levels are straight forward. No tricks, tech, or lore needed. Just standard levels. I think the point you brought up is very cogent in troll levels, little Timmy levels with random hidden blocks, and very hard levels by amazingly skilled players/creators. For the rest of levels, which might be the larger portion, it doesn't necessarily hold true.
Initial clear check just shouldn't affect difficulty at all, actually. Bad game design.
2 ปีที่แล้ว +7
@@Syranovæ If initial clear check didn't matter at all, it could be that most of the people would never get their levels in the random pool, because nobody could find them in the first place. In SMM2 you need to get at least 1 play from somebody else before you get a new level into the random pool, and as a rookie level creator finding even that first player can be tough. I'm not exactly sure if the first play needs to finish the level. Then again, when writing this it occurred to me that they could just use that first other player's clear rate to recalibrate the difficulty before putting the level into the correct random pool. They might actually do that, this video doesn't really confirm or deny anything related to that.
Comment from one verified user with 9 likes placed over multiple comments of unverified users with so much more likes? Clearly, TH-cam is falling apart in terms of ratings of videos.
all songs used are from the kirby 64 ost: 0:00 - Mountain Stream 1:23 - Pop Star 2:51 - Rock Star 3:36 - Aqua Star 5:15 - Above the Clouds 5:57 - Boss Theme 6:45 - Neo Star 7:54 - Bumper-Crop Bump 9:47 - Ripple Star 10:49 - Shiver Star 12:29 - Factory Inspection 13:09 - Training
One of the things we also talked about briefly was if levels were reuploaded exactly the same way if a level would differ in initial difficulty, then after the video was in production I remembered about title screens and how I've seen the same level in easy and normal and sometimes even expert, the spike test now does show that this is the case. All in all the goal is to find some sort of algorithm that combines all the pieces together. While initial difficulty on clear check is one piece of the algorithm it's figuring out a way to combine that, possibly the clear rate, and the ratio of who flagged the level as cleared over who has not, as well as figuring out any other overlooked part of the equation that is the key here to getting a sure fire algorithm going.
Since the number of tries it takes the level maker comes into play, I wonder how that has affected the categorization of my own uploaded levels. I often quit a clear check and nerf my levels a little (because I don’t have the patience to beat them lol) until I end up beating them on the first or second try after failing to beat earlier versions of them. So I wonder if Nintendo factors in ALL attempts on a saved level, or if the counter starts over when the level maker quits a clear check and changes the level slightly. One of my levels like this has a 4% clear rate, but I ultimately beat its final form in one try (after failing to beat earlier versions of it). So I wonder which endless mode it ended up in initially because of that.
I think that the algorithm might ignore any clear attempts of a person after the person beat the level for the first time. Another theory I have is that clears of different persons count differently based on some internal skill level value of a player. Somewhat similar to ELO
I'm very curious if skips also factor into the Nintendo algorithm. For instance: take the puzzle level that was in Expert yet had a 17% clear rate, or the exceedingly long levels in Super Expert. Both would likely have very high skip rates, as many people simply do not like puzzle levels or lengthy levels. Someone may die once in a 2 minute long level and skip to the next one rather than attempt it again. This wouldn't impact the clear rate much, but it may have a bearing on the difficulty level. Just a thought!
I had this same question but typically a lengthy level already indicates its difficulty. So if we take that into account I surmise they may have a separate yet similar algorithm to deaths that involve skips. Since every skip would be unique to a level, if a player didn't participate in the level there must be a reason why(which I would surmised levels in Expert and Super Expert get skipped over 5/10 to 9/10 times respectively)
If the player never clears the level like exit the software, it won't save like if you never played it, also The clear rate of 199,000 out of 400,000 is only showing every player's attempt to succeed, all the times the characters looses a life it counts as a failure point(meaning that is where the player's character fails there because of enemy, hitbox of icicle or liquid hazard),The shoe icon only shows how many people played it, and also the clear rate is everyone's attempt to reach the goal point there. Never overestimate the clear rate if the player doesn't know what to do, give it a chance if you think you can complete it.
I had the same question while watching the twitch vod and I was really hoping someone would have brought it up live in twitch chat. If I had to theorize, skips wouldn't affect difficulty as strongly as deaths but it would make sense to factor them in to a degree. On one hand a skip could mean the player died once or twice and determined it was too difficult and skipped it or came across some difficult tech or some precise jump they knew they couldn't do so they skipped it before dying at all. In those cases the skip would represent the amount of difficulty the player WOULD have had if they had kept playing until they cleared the level. Either a great amount of time in the case of difficult tech without a way to die, or many many deaths in the case of precise jumps. On the other had a skipped level may not be difficult but instead be too long, boring, or super annoying. The player could easily beat it, they just can't be bothered wasting their time on something they aren't enjoying. So these skips would be the opposite; they represent a hypothetical first try clear rather than an amount of deaths or struggle clearing the level. So basically skips represent an unknown variable in the difficulty because we can't know the reason for every individual skip. In a hypothetical world where skips didn't exist and a level had to be played until cleared, a level's clear rate could potentially be drastically different than same real world level. The more I think about it, the more curious I am on just how the algorithm DOES consider skips. Does have a way of predicting the reason why a specific player skipped a specific level? Does it gauge the player's own skill level based every other level they've attempted? Does it consider the amount of spent time in the level before skipping? Does it just give all skips a blanket weight and be done with it? Or did the programmers just say "hecc it" and leave skips out of the algorithim altogether?? 🤔🤔🤔 It's so interesting to think about and we will likely never 100% understand how the algorithim works. Unless someone at Nintendo leakes it out...
I think this would be grouped into the how many clear vs plays it has. Not the clear rate, but how many DIFFERENT players actually clear it. If the theory that levels that get cleared more often, even if the clear rate is low, are easier in difficulty, then skipping a level and never going back to clear it likely does increase the difficulty rating
What would be interesting is if they factored in player experience. An experienced player dying on a level could indicate harder level. An inexperienced player 1st trying a level could be strong evidence the level is easy
That was one of the things we were looking at also, but unfortunately there wasn't an easy way to test that - that's definitely a possibility in factoring in level difficulty though!
Agree with this. Someone who has beaten a lot of Super Expert levels should have a different weight in how a level gets classified from a newbie and how hard the level is for them. The algorithm should be looking at the reputation of WHO beats the level.
@Raysfire If the SMM2 scraped datasets contain the players who cleared each level, a correlational analysis could maybe be performed, but obviously this is well beyond the scope of a Twitch stream :)
The thing that the high clear rate super expert levels had in common is that none of them had any way of dying other than the time limit, so many people getting the level and skipping affects it (since skipping without dying doesn't count as an attempt)
Thats basically 0% chance. There are so many levels that some are going to have the same tags as other but be completely different difficulties. There have to be some easy levels with the puzzle tag
I figure your deaths to set the difficultly are probably reflective of the deaths per completion in general. Easy levels are ones where people generally win on the first or second try, normal on the third or fourth, and expert more. The reason the clear rates are different is because of people who skip after failing the first time or who retry for world record. Only people who go on to clear the level or who die more than the expected attempts factor into the difficulty. This would explain why some levels that force a death to respawn from a checkpoint appear in easy or normal despite having ~10% clear rates, the levels consistently get 1-2 deaths per successful attempt and almost no-one dies more often(because it'd need timeouts or similar), but a lot of people quit when they realise the level is a net negative to lives.
The difficulty rating of chess puzzles can be calculated using a modified form of ELO from the players who succeed or fail at it. It would make sense for Mario Maker to use a similar system
Very insightful into the algorithm, thank you. I would like to clarify the conclusions we’ve drawn though- the only thing we’ve proven from our own experiments is that creator clear check attempts noticeably affects the algorithm’s placement. The data science studies were useful, noticing trends of lower clear rates and higher world records tending toward higher difficulty placements; however, we cannot say that the algorithm necessarily checks for these things. In order to show causation we would have to do experiments similar to the spike test but with multiple other players/accounts, keeping the creator clear checks constant.
What I learned: When uploading a Super Expert level that you want to trick Nintendo into thinking it's Easy: 1. Make a copy of the level *before* trying to upload it. 2. Try to beat the level. 3. If you win, great, go to step 5, because you just uploaded it! If you lose, then delete the level. 4. If you're here, then go back to step 1, but with the copy of the level. 5. Watch as your level quickly gets moved to Super Expert because Nintendo's algorithm isn't stupid, and will notice when people are struggling. Or just dev door it I dunno.
I got that first ballon level in SE endless too! Also I lost all my lives because I kept having one coin left and the clouds left. Also thanks for this video I’ve noticed too and have been wondering! Sometimes I would get a level with one person who played it, and I don’t know how it Nintendo knew for sure that it was hard. It’s very interesting.
Rays this is an amazing video that you clearly spent a lot of time making. I loved the creative style of it! Keep up the amazing work keeping everyone here entertained and informed on everything Mario maker. ❤
All right, I study mathematics and the graph at 13:10 ringed a bell: a hyperbolic curve. If this graph is the way things work behind the scenes, then you can tell that the _product_ of the number of attemtps and the time per attempt is what counts - which is *the total time spent for the creator to beat the level.* I would love to have this theory tested. Amazing video! I highly enjoyed it. More science, please!
other ideas i can think of for possible factors might be things like hazards in the level, level size, time spent creating the level, amounts of inputs on clear attemps (for example just holding right, holding right and jumping some, or having something more complex than that) also maybe the level creator's previous levels have an effect?
Wow this was an amazing video! I think I have to throw when doing clear checks on my kaizo's I build. I practice them so much in the editor, that I clear check them in a couple of tries. Then they get put into categories where people won't possibly like or be able to finish them. They're dead on arrival.
And now all I need is to get Mario maker 2! YOU'RE GOING INTO SUPER EXPERT, SINGULAR SPIKE! This could actually be used to create break-levels for the harder difficulties. You know, give 'em a free win.
Wait so does that mean it is possible to get a super expert difficulty level on easy if the creator beat it first try? My that sounds scuffed. Anyway, great job on the video. I appreciate the amount of time you've put into research.
@raysfire This seems like a good machine learning task. Have you tried grabbing a programmer who knows a bit about data science and get them to run a regression algorithm on some of the data you collected? It could be a simple linear combination of some of those variables you identified. If I were developing a game and wanted to check clear difficulty (and had lots of time) I think it'd be really smart to take a recording of the user's clear and try to mutate it by throwing in random delays / extra button presses / skipping button presses to get a sense for how "precise" the level was. I really doubt that's what they're doing though (it'd be expensive / take a long time to upload a level most likely).
I have once encountered a very hard course with a high clear rate. It was an SMB course which was very short (one screen) and contained a giant pit filled with lots of boo rings and falling goombas and I had to jump on the goombas to the flagpole while avoiding the millions of boos, which is extremely precise. I couldn't complete it.
This was an amazing video, as you were explaining what your experiments were and what they ment I also found my self think "well what if..." "then that means..." and its as if you answered my thoughts! Kinda creepy XD but great minds think alike though. The script was awesome and kept me intrigued and interested the whole way through. I also loved how you showed the stats, really helped me follow along. Was worth the wait, I hope this video gets into the algorithm, it deserves it!
This is actually really interesting how you can see super expert levels within the expert category - if the creator of the level of their level practices segements before attempting to upload it they will usually clear it relatively quickly possibly allowing the level to get misassigned
another potential theory: real ones remember the time alpharad deluxe did a co-op super world filled with a bunch of easy levels, but because of the ways they were set up (typically refreshing or auto levels) and the way co-op respawn works, it was near IMPOSSIBLE for them to beat the levels that way. they attempted an auto level for nearly THREE HOURS before giving up, switching back to single player, and clearing it normally. so it's possible that playing in co-op can also skew levels' difficulties, especially if co-op attempts count towards the clear rate. idk if it does, but it's just a theory. a g-
Thank goodness for the sub! I forgot about Quiz day! Awesome Vid Mr. Shud! It's really interesting that the creator in a sense can partially decide where their level will show up!
My theory is that it starts out as the creator's clear rate, but after it's uploaded, it doesn't just determine the difficulty primarily off the level's clear rate, but instead the game might do 2 things: 1. Check for player outliers via their individual clear rate, and find the clear rate without those outliers. and/or 2. Take the average time to clear into account.
Man, it’s so weird hearing you make documentary-style videos. I never thought they were your thing. But I think you do really well, I enjoy these videos!
i remember this level was used by someone who made a video about how the clear rate works, and yeah basically if you die a ton on your level while trying to beat it to upload it, it gets ranked harder
See I find normal significantly easier than expert. My normal record is currently well over 200 but my expert record is only 6. This is with skipping, which I use pretty liberally. I do agree though. Some normal levels could almost be super expert and I have played expert levels that are easier than your average normal level.
The clear rate is when the every player who attempts succeeds the level or if they fail/or character lose a life or fails to reach, also that is the total attempts of everyone even the same person can play 141/150 times and if another player plays and clears the level, that person will add to the numerator. The shoe marker is the amount of total players played the course currently before you/we do play the course.
First clear is the first person cleared the level, and the world record is anyone who did the quickest way to complete the level or the most efficient; and well played and honor for everyone who skillfully came close to anyone's World record, or matching theirs, even if its 00:005 around 00:03.45 seconds difference(or very close to WR)
It might be useful to note that who clears it might matter. If someone who clears other extreme difficulty ones with ease dies many times on the level- it's hard. If die a bunch in it, it still might be easy.
I have a counterpoint to the clear rate for creator thought. In people's expert and super expert levels, some people put dev doors or dev exits to upload it because they don't want to do it. By this, the clear rate is 100% or close to it. By default, these should be put into easy or normal. Therefore, without testing uploads, we can see that the creator clear rate does not have a "normal" impact.
If you click on a users profile, the game keeps track of how many clears they've had in each respective difficulty - ie it tells you how many easy levels theyve beaten, how many normal, etc. Is it possible that these stats are taken into consideration? Like maybe if a player has beaten a ton of super expert levels before, the game considers them skilled at the game and how much of an effect they have on a levels chosen difficulty is altered.
My top 3 hardest levels: 1.Porcupuffer's Crate Clash 2.King Bully's Mask Mayhem 3.Yoshi's Treetop Troubles It took me F.O.R.E.V.E.R to clear those 3 levels Honorable Mentions:Chaos Chasm
The levels I make that I rush through to make a record for people to try an beat gets dunked on in sheer amount of plays from my one level I took a leisurely stroll through and took my time to beat.
The level being placed into a higher difficulty than expected might reflect your skill as a player; if Nintendo thinks you're a skilled player because you previously cleared X tricky levels in Y tries, that may be taken into consideration.
to see the full, unedited "clear rate" experiment, check it out on my vod channel here: th-cam.com/video/JTA0wkdl9M0/w-d-xo.html ✨
also, big shoutouts to dsteves for all of the help with this project! a lot of the data and experiments i used in this video wouldn't have been possible without his help!!
check out dsteves' channel here: www.twitch.tv/dsteves 😎
Nice video btw
Great vid
nice vid keep up the good work
It would have been cool to go back in time and track the average clear percent over time. I have a feeling that since the game released, less people are playing and the people that are still playing are really good at the game. So perhaps the levels got harder to accommodate that crowd thus the average clear percent would go down over time.
@@cak5171 Check around 8:59 to see that DSteves cleared K. Wonka's level, too. XD
I think the initial difficulty seeding being higher than expected makes a lot of sense. As the creator, you know all the tricks needed for a level, so it's easier for you than it would be for a new player. Having the initial seed be biased toward higher difficulty is actually pretty smart game design.
Yeah. I always wonder how one of my levels was initially placed after upload since it took me one attempt to clear check it, but found out later that most people who played it struggled and gave up.
That said, I still feel like you could get a better starting estimate by taking into accounts the # of failures before a successful attempt when clear checking it. It would make it so that harder to clear levels tend to be higher difficulty on average.
And yet I've made many levels that took me dozens of tries to clear check and yet they have a 5%-10% clear rate.
Many creators aren't the best or most skilled and yet make levels that test them to their limits, whereas there are significantly better players out there that are so much better and can beat the level easier.
Beyond that, in my experience most levels are straight forward. No tricks, tech, or lore needed. Just standard levels.
I think the point you brought up is very cogent in troll levels, little Timmy levels with random hidden blocks, and very hard levels by amazingly skilled players/creators. For the rest of levels, which might be the larger portion, it doesn't necessarily hold true.
Initial clear check just shouldn't affect difficulty at all, actually. Bad game design.
@@Syranovæ If initial clear check didn't matter at all, it could be that most of the people would never get their levels in the random pool, because nobody could find them in the first place.
In SMM2 you need to get at least 1 play from somebody else before you get a new level into the random pool, and as a rookie level creator finding even that first player can be tough. I'm not exactly sure if the first play needs to finish the level.
Then again, when writing this it occurred to me that they could just use that first other player's clear rate to recalibrate the difficulty before putting the level into the correct random pool. They might actually do that, this video doesn't really confirm or deny anything related to that.
So what we've learned is that we need to throw when clear checking automatic levels to help no skip runners.
Comment from one verified user with 9 likes placed over multiple comments of unverified users with so much more likes? Clearly, TH-cam is falling apart in terms of ratings of videos.
yo
And also throw triple shell jump, mashing, and near-pixel perfect precision levels
@@Fowlware fun fact: likes are not the only thing that TH-cam looks at to bump a comment.
comment placement is decided based on likes and how often the user has a interaction (and maybe subscribers)
all songs used are from the kirby 64 ost:
0:00 - Mountain Stream
1:23 - Pop Star
2:51 - Rock Star
3:36 - Aqua Star
5:15 - Above the Clouds
5:57 - Boss Theme
6:45 - Neo Star
7:54 - Bumper-Crop Bump
9:47 - Ripple Star
10:49 - Shiver Star
12:29 - Factory Inspection
13:09 - Training
Thank you
based
Would’ve been a sin to leave Kirby 64 out of this 😝
Chad
The Kirby series has the best soundtrack, with the classic Sonic games being the second best.
One of the things we also talked about briefly was if levels were reuploaded exactly the same way if a level would differ in initial difficulty, then after the video was in production I remembered about title screens and how I've seen the same level in easy and normal and sometimes even expert, the spike test now does show that this is the case. All in all the goal is to find some sort of algorithm that combines all the pieces together. While initial difficulty on clear check is one piece of the algorithm it's figuring out a way to combine that, possibly the clear rate, and the ratio of who flagged the level as cleared over who has not, as well as figuring out any other overlooked part of the equation that is the key here to getting a sure fire algorithm going.
Since the number of tries it takes the level maker comes into play, I wonder how that has affected the categorization of my own uploaded levels. I often quit a clear check and nerf my levels a little (because I don’t have the patience to beat them lol) until I end up beating them on the first or second try after failing to beat earlier versions of them. So I wonder if Nintendo factors in ALL attempts on a saved level, or if the counter starts over when the level maker quits a clear check and changes the level slightly. One of my levels like this has a 4% clear rate, but I ultimately beat its final form in one try (after failing to beat earlier versions of it). So I wonder which endless mode it ended up in initially because of that.
Thank-you! I’ve been on the lookout for resources that would allow me to see stats on my own levels. My curiosity can now be satiated.
Hey man! It's me, hamstercol from twitch, how are you?
I think that the algorithm might ignore any clear attempts of a person after the person beat the level for the first time.
Another theory I have is that clears of different persons count differently based on some internal skill level value of a player. Somewhat similar to ELO
Hi biggest maker
I'm very curious if skips also factor into the Nintendo algorithm. For instance: take the puzzle level that was in Expert yet had a 17% clear rate, or the exceedingly long levels in Super Expert. Both would likely have very high skip rates, as many people simply do not like puzzle levels or lengthy levels. Someone may die once in a 2 minute long level and skip to the next one rather than attempt it again. This wouldn't impact the clear rate much, but it may have a bearing on the difficulty level.
Just a thought!
I had this same question but typically a lengthy level already indicates its difficulty. So if we take that into account I surmise they may have a separate yet similar algorithm to deaths that involve skips. Since every skip would be unique to a level, if a player didn't participate in the level there must be a reason why(which I would surmised levels in Expert and Super Expert get skipped over 5/10 to 9/10 times respectively)
If the player never clears the level like exit the software, it won't save like if you never played it, also The clear rate of 199,000 out of 400,000 is only showing every player's attempt to succeed, all the times the characters looses a life it counts as a failure point(meaning that is where the player's character fails there because of enemy, hitbox of icicle or liquid hazard),The shoe icon only shows how many people played it, and also the clear rate is everyone's attempt to reach the goal point there.
Never overestimate the clear rate if the player doesn't know what to do, give it a chance if you think you can complete it.
I had the same question while watching the twitch vod and I was really hoping someone would have brought it up live in twitch chat.
If I had to theorize, skips wouldn't affect difficulty as strongly as deaths but it would make sense to factor them in to a degree.
On one hand a skip could mean the player died once or twice and determined it was too difficult and skipped it or came across some difficult tech or some precise jump they knew they couldn't do so they skipped it before dying at all. In those cases the skip would represent the amount of difficulty the player WOULD have had if they had kept playing until they cleared the level. Either a great amount of time in the case of difficult tech without a way to die, or many many deaths in the case of precise jumps.
On the other had a skipped level may not be difficult but instead be too long, boring, or super annoying. The player could easily beat it, they just can't be bothered wasting their time on something they aren't enjoying. So these skips would be the opposite; they represent a hypothetical first try clear rather than an amount of deaths or struggle clearing the level.
So basically skips represent an unknown variable in the difficulty because we can't know the reason for every individual skip. In a hypothetical world where skips didn't exist and a level had to be played until cleared, a level's clear rate could potentially be drastically different than same real world level.
The more I think about it, the more curious I am on just how the algorithm DOES consider skips. Does have a way of predicting the reason why a specific player skipped a specific level? Does it gauge the player's own skill level based every other level they've attempted? Does it consider the amount of spent time in the level before skipping? Does it just give all skips a blanket weight and be done with it? Or did the programmers just say "hecc it" and leave skips out of the algorithim altogether?? 🤔🤔🤔
It's so interesting to think about and we will likely never 100% understand how the algorithim works. Unless someone at Nintendo leakes it out...
I think this would be grouped into the how many clear vs plays it has. Not the clear rate, but how many DIFFERENT players actually clear it. If the theory that levels that get cleared more often, even if the clear rate is low, are easier in difficulty, then skipping a level and never going back to clear it likely does increase the difficulty rating
What would be interesting is if they factored in player experience. An experienced player dying on a level could indicate harder level. An inexperienced player 1st trying a level could be strong evidence the level is easy
That was one of the things we were looking at also, but unfortunately there wasn't an easy way to test that - that's definitely a possibility in factoring in level difficulty though!
Probably wouldn't work as I am good at smm1 (so the skills carry over) but i have barely played smm2.
Agree with this. Someone who has beaten a lot of Super Expert levels should have a different weight in how a level gets classified from a newbie and how hard the level is for them. The algorithm should be looking at the reputation of WHO beats the level.
@Raysfire If the SMM2 scraped datasets contain the players who cleared each level, a correlational analysis could maybe be performed, but obviously this is well beyond the scope of a Twitch stream :)
@Bezray There would be outliers, but with enough data it would average out. And they could weight players less if they haven’t beaten many levels
The thing that the high clear rate super expert levels had in common is that none of them had any way of dying other than the time limit, so many people getting the level and skipping affects it (since skipping without dying doesn't count as an attempt)
Something to consider: tags could also potentially have an influence. Many of your outlier examples were puzzle levels.
Thats basically 0% chance. There are so many levels that some are going to have the same tags as other but be completely different difficulties. There have to be some easy levels with the puzzle tag
I think the issue with puzzle levels is that people just skip the level if they can't solve the puzzle, lowering its clear rate
I figure your deaths to set the difficultly are probably reflective of the deaths per completion in general. Easy levels are ones where people generally win on the first or second try, normal on the third or fourth, and expert more.
The reason the clear rates are different is because of people who skip after failing the first time or who retry for world record.
Only people who go on to clear the level or who die more than the expected attempts factor into the difficulty.
This would explain why some levels that force a death to respawn from a checkpoint appear in easy or normal despite having ~10% clear rates, the levels consistently get 1-2 deaths per successful attempt and almost no-one dies more often(because it'd need timeouts or similar), but a lot of people quit when they realise the level is a net negative to lives.
I made the "Level 49*", thanks for playing and including this level in your experiment :)
I'm so glad you like the level.
Regards.
The difficulty rating of chess puzzles can be calculated using a modified form of ELO from the players who succeed or fail at it. It would make sense for Mario Maker to use a similar system
Very insightful into the algorithm, thank you.
I would like to clarify the conclusions we’ve drawn though- the only thing we’ve proven from our own experiments is that creator clear check attempts noticeably affects the algorithm’s placement.
The data science studies were useful, noticing trends of lower clear rates and higher world records tending toward higher difficulty placements; however, we cannot say that the algorithm necessarily checks for these things.
In order to show causation we would have to do experiments similar to the spike test but with multiple other players/accounts, keeping the creator clear checks constant.
What I learned:
When uploading a Super Expert level that you want to trick Nintendo into thinking it's Easy:
1. Make a copy of the level *before* trying to upload it.
2. Try to beat the level.
3. If you win, great, go to step 5, because you just uploaded it! If you lose, then delete the level.
4. If you're here, then go back to step 1, but with the copy of the level.
5. Watch as your level quickly gets moved to Super Expert because Nintendo's algorithm isn't stupid, and will notice when people are struggling.
Or just dev door it I dunno.
I got that first ballon level in SE endless too! Also I lost all my lives because I kept having one coin left and the clouds left. Also thanks for this video I’ve noticed too and have been wondering! Sometimes I would get a level with one person who played it, and I don’t know how it Nintendo knew for sure that it was hard. It’s very interesting.
So that's why I could play endless easy and end up on a brand new level that's incredibly difficult, only to find a dev door several attempts later.
Hey Rays. Loved the content. It's really nice to see someone put so much effort into their content. Keep up the good work Mr Shud :)
Rays this is an amazing video that you clearly spent a lot of time making. I loved the creative style of it! Keep up the amazing work keeping everyone here entertained and informed on everything Mario maker. ❤
Lol I remember making that level, 0:20 didn’t think it would get on super expert
Fr?
I have always wanted someone to make a video just like this, thank you Raysfire!
Through the power of SCIENCE!
All right, I study mathematics and the graph at 13:10 ringed a bell: a hyperbolic curve. If this graph is the way things work behind the scenes, then you can tell that the _product_ of the number of attemtps and the time per attempt is what counts - which is *the total time spent for the creator to beat the level.* I would love to have this theory tested.
Amazing video! I highly enjoyed it. More science, please!
I’m glad you overcame your pc issues to upload this masterpiece! Love all the content you’ve been putting out
other ideas i can think of for possible factors might be things like hazards in the level, level size, time spent creating the level, amounts of inputs on clear attemps (for example just holding right, holding right and jumping some, or having something more complex than that)
also maybe the level creator's previous levels have an effect?
little timmies might leave their switch on at night and it counts as 1000-2000 of deaths in speedruns
Wow this was an amazing video!
I think I have to throw when doing clear checks on my kaizo's I build. I practice them so much in the editor, that I clear check them in a couple of tries. Then they get put into categories where people won't possibly like or be able to finish them. They're dead on arrival.
The way that all these new videos are amazing. Keep up the good work!
Hello, I am Mr. Fusion Man. There are no fusions in this video. I hope a versus challenge comes soon or I'm gonna be out of a job.
Lol
Idk the twister and galoomba at 0:15 - 0:19 seem pretty fused to me
@@rorycannon7295 sadly, the twister is closer than the galoomba, they are not fused.
@@EnderrBoi i mean they could be shadoingled along the z axis?
@@EnderrBoi but tru
really great video! was interested throughout the whole thing, and loved the methodology used to discover things. keep up the great work :)
And now all I need is to get Mario maker 2!
YOU'RE GOING INTO SUPER EXPERT, SINGULAR SPIKE! This could actually be used to create break-levels for the harder difficulties. You know, give 'em a free win.
raysfire's content is top notch. you should have way more subs! keep up the good work
12:00 you can also try to match the WR time for easy,medium,expert,super expert
Like wait 120s to finish the lvl for super expert.
Idk tho
Why does the Easy text disappear at 13:36
I love the nostalgic Kirby 64 themes in the background
spindel XD (someone said say spindel in the comments 1:33 )
Wait so does that mean it is possible to get a super expert difficulty level on easy if the creator beat it first try? My that sounds scuffed.
Anyway, great job on the video. I appreciate the amount of time you've put into research.
2:15 I look in newest and see like 15 comments saying spindel but nobody said
beeBobble
There you go, AComputerMouse
@raysfire This seems like a good machine learning task. Have you tried grabbing a programmer who knows a bit about data science and get them to run a regression algorithm on some of the data you collected? It could be a simple linear combination of some of those variables you identified.
If I were developing a game and wanted to check clear difficulty (and had lots of time) I think it'd be really smart to take a recording of the user's clear and try to mutate it by throwing in random delays / extra button presses / skipping button presses to get a sense for how "precise" the level was. I really doubt that's what they're doing though (it'd be expensive / take a long time to upload a level most likely).
I have once encountered a very hard course with a high clear rate. It was an SMB course which was very short (one screen) and contained a giant pit filled with lots of boo rings and falling goombas and I had to jump on the goombas to the flagpole while avoiding the millions of boos, which is extremely precise. I couldn't complete it.
Love the video. The whiteboard needs to be featured more. Possibly for the Mario Party CPU tourney?
This was an amazing video, as you were explaining what your experiments were and what they ment I also found my self think "well what if..." "then that means..." and its as if you answered my thoughts! Kinda creepy XD but great minds think alike though.
The script was awesome and kept me intrigued and interested the whole way through. I also loved how you showed the stats, really helped me follow along.
Was worth the wait, I hope this video gets into the algorithm, it deserves it!
scanning for shudheads 📡
shudhead located 📍
I love how Rays is truly just a lad living his life
Holy quality batman, this is so much different than what I remember from watching you a year or two ago. And I'm all for it!
It may be half 11 in the Queensland of England but rays has priority over sleep
i'm 11 months late... is the quiz still on thursday?
This is actually really interesting how you can see super expert levels within the expert category - if the creator of the level of their level practices segements before attempting to upload it they will usually clear it relatively quickly possibly allowing the level to get misassigned
Spicy video rays! 👀 And awesome editing, loved the graph :D
another potential theory: real ones remember the time alpharad deluxe did a co-op super world filled with a bunch of easy levels, but because of the ways they were set up (typically refreshing or auto levels) and the way co-op respawn works, it was near IMPOSSIBLE for them to beat the levels that way. they attempted an auto level for nearly THREE HOURS before giving up, switching back to single player, and clearing it normally. so it's possible that playing in co-op can also skew levels' difficulties, especially if co-op attempts count towards the clear rate. idk if it does, but it's just a theory. a g-
“Spindel”
Thank goodness for the sub! I forgot about Quiz day! Awesome Vid Mr. Shud! It's really interesting that the creator in a sense can partially decide where their level will show up!
I wont lie Rays, Ive been loving this new type of content on your channel. Keep it up 👍. Also no test for me baby WOOOOOO
THE WHITEBOARD AND LAB COAT REVEAL! I don't know if imma pass the test on Thursday? Is there any retakes?
My theory is that it starts out as the creator's clear rate, but after it's uploaded, it doesn't just determine the difficulty primarily off the level's clear rate, but instead the game might do 2 things:
1. Check for player outliers via their individual clear rate, and find the clear rate without those outliers.
and/or
2. Take the average time to clear into account.
I’ve always wondered this. Such a good video idea and well executed
Man, it’s so weird hearing you make documentary-style videos. I never thought they were your thing. But I think you do really well, I enjoy these videos!
I got a "First Clear" in Super Expert once. The stage was literally so easy
I remember playing a level never beaten before or two in normal mode. Always wondered how that worked
These new vids have been really great, you deserve way more attention man
i remember this level was used by someone who made a video about how the clear rate works, and yeah basically if you die a ton on your level while trying to beat it to upload it, it gets ranked harder
I love videos like this
I always have great respect for Dr. Shud
1:31
Spindel
WindWaver426 you are welcome
shudman scientist cosplay is juicing
I would consistently get levels I would call expert and super expert when playing normal. My normal record is exactly the same as my expert record.
Me too!
See I find normal significantly easier than expert. My normal record is currently well over 200 but my expert record is only 6. This is with skipping, which I use pretty liberally.
I do agree though. Some normal levels could almost be super expert and I have played expert levels that are easier than your average normal level.
Amazing video rays! Thanks for the science!
Chill out Rays nye the Science Guy
Thanks for validating me in my feeling that normal was way harder than easy.
such a high quality vid earned my sub good stuff
1:05 KITCHEN REVEAL!!!!1!!
One of the coolest raysfire videos yet! And the use of the Kirby 64 soundtrack makes it all the better!
these videos just keep getting better and better
Oh no! I was already subscribed with all notifications turned on and liked and commented on the video!
this video is really good, and the kirby 64 ost just makes it better
The clear rate is when the every player who attempts succeeds the level or if they fail/or character lose a life or fails to reach, also that is the total attempts of everyone even the same person can play 141/150 times and if another player plays and clears the level, that person will add to the numerator. The shoe marker is the amount of total players played the course currently before you/we do play the course.
First clear is the first person cleared the level, and the world record is anyone who did the quickest way to complete the level or the most efficient; and well played and honor for everyone who skillfully came close to anyone's World record, or matching theirs, even if its 00:005 around 00:03.45 seconds difference(or very close to WR)
Spindel! (look at 1:35 for context)
11:06 if you want to do *SCIENCE,* THEN PLEASE CALL IT A FUCKING HYPOTHESIS AND NOT A THEORY 😭😭😭😭😭😭
When you see your tool being used in a video: 😇
My theory is that the initial difficulty also depends on how many attempts on average it takes for the creator to beat any level.
i feel like this guy had a science assignment where he could do whatever topic and chose this
It might be useful to note that who clears it might matter. If someone who clears other extreme difficulty ones with ease dies many times on the level- it's hard. If die a bunch in it, it still might be easy.
Worth the wait! Get this man a new chair!
I have a counterpoint to the clear rate for creator thought. In people's expert and super expert levels, some people put dev doors or dev exits to upload it because they don't want to do it. By this, the clear rate is 100% or close to it.
By default, these should be put into easy or normal. Therefore, without testing uploads, we can see that the creator clear rate does not have a "normal" impact.
If you click on a users profile, the game keeps track of how many clears they've had in each respective difficulty - ie it tells you how many easy levels theyve beaten, how many normal, etc. Is it possible that these stats are taken into consideration? Like maybe if a player has beaten a ton of super expert levels before, the game considers them skilled at the game and how much of an effect they have on a levels chosen difficulty is altered.
I think the age ratio might also need to be factored into it. Some low clear rates of lower-difficulty levels may be because children are struggling.
This is really neat! Thanks for the science, rayfsfire!
Great vid!!! hope this one pops off just like the last one
Thanks for this I shall use this knowledge to make an Easy level into an Expert level as to try to get on Panga's stream.
Outside of your mutiplayer with red and panga this type of content is some of your best. Keep it up!
Okay I have studied all night for the quiz on Tuesday, I think I'm ready
raysclassic
Hype new rays video, hope it goes as well as the last couple
My top 3 hardest levels:
1.Porcupuffer's Crate Clash
2.King Bully's Mask Mayhem
3.Yoshi's Treetop Troubles
It took me F.O.R.E.V.E.R to clear those 3 levels
Honorable Mentions:Chaos Chasm
The levels I make that I rush through to make a record for people to try an beat gets dunked on in sheer amount of plays from my one level I took a leisurely stroll through and took my time to beat.
Can't wait for the day when we learn how to fill literally every difficulty with auto levels lol
These mario maker 2 vids r fire ngl. Idk why but i just started binging them 😩😩😩
2:07 🤨🤨🤨🤨🤨🤨🤨🤨raysfire?????🤨🤨🤨🤨🤨🤨🤨🤨
Always a good day went you got content from ya boy rays
The thumbnail is literally telling me that thespikeoverthere is an extreme demon
The level being placed into a higher difficulty than expected might reflect your skill as a player; if Nintendo thinks you're a skilled player because you previously cleared X tricky levels in Y tries, that may be taken into consideration.