Using Image Recognition to find Mario
ฝัง
- เผยแพร่เมื่อ 22 ต.ค. 2022
- Can a Python script find Mario using image recognition?
In this video I write a Python program to try to automatically find and click on Mario in the mini game Wanted. The program uses basic image recognition to locate textures, and it manages to be extremely effective. If you like the video, make sure to subscribe!
LINKS
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Support the channel: ko-fi.com/codenoodles
Itch.io: codenoodles.itch.io/
GitHub: github.com/OfficialCodeNoodles
█▀ █ █ █▄▄ █▀ █▀▀ █▀█ █ █▄▄ █▀▀
▄█ █▄█ █▄█ ▄█ █▄▄ █▀▄ █ █▄█ ██▄ - วิทยาศาสตร์และเทคโนโลยี
It seems like towards the end the problem wasn't with recognition but with the touching part. Moving icons seem to be hard for it. It clicks where they are rather than where they are going to be, which means that sometimes it can miss the icon.
I mean at least that's what it looks like to me. I have literally 0 programming knowledge so I could be completely wrong lol
U right
Nah, spot on here. This kind of image recognition, especially using small snippets on a big view, can take a good amount of time (in computer terms). So in the time between it took the screenshot and clicks, the image has changed without the program knowing.
Could theoretically solve that by having it perform the whole process twice, and extrapolating to where it should actually click, but that would be immensely difficult, especially once you account for the case of it going off screen (and thus coming out on the opposite side).
And even then, you might get false results when you have it not moving in a straight line, like all the levels where they move in a sinusoidal pattern. Which would in turn be even more obscenely hard to code around
@@genericfakename4372 and adding to that, even if it 100% predicted where the right one was gonna be, there is a huge chance that in that position it would be covered by another icon. I would have to keep following the path and predict a position where no other icon would clash... ouch ...
kinda awesome that our brains can just do that automatically and we just take it for granted
@@zewzit True
I think that also, it might be clicking on where the center of the icon would be, if it wasnt covered by other icons. Instead it should click on a random sub-tile that matches 100%, or the sub-tile that matches the closest.
It would be fun to see this same concept used in the bob-omb sorting minigame. Nice vid. Very good. 10/10 would subscribe again.
wouldbt they fail as its using scren shots to fund them so it only works wheb everythibg stays still
@Test Of Time Gaming bro, what you saying
@@maxwellthesillycat damn i guess four or five typos makes something incomprehensible
three of them are even the same type of typo
@@testoftimegaming5219 i mean this minigame also uses moving icons and it seems to be relatively fine with them here
@@testoftimegaming5219 well I don't know actual coding but I feel like a polished "if pink on X,Y coordinates (that don't include the goal area) is detected, click then drag to x,y coordinates (that correspond to the goal)" and then same with gray I guess(?)
Bro just casually hacking 💀
It is odd how nostalgic this is. I strangely remember this mini game extraordinarily vividly.
The NSMB minigames were probably majority of people's childhood.
I honestly want to see someone push this idea to the game's limits
And sm64DS
Nah for me it was SM64DS. I didn't knew that these Minigames are in another game too until I was 22 or so (A few years ago)
@@AlryFireBlade same!
SM64DS came out 1st and had a lot more minigames than NSMB.
@@ugonnankwonta6472 exactly! It deserves more recognition.
[Spoiler]
that AI had many close calls, the fact that it made it passed 100 is very impressive
Oh. You think 100 is a lot? On sm64 ds I got over 2000 once
@@hunterdunn2068 Man, we’re talking about a fuldging AI
@@Redinho1 Hunter is a computer speaking to us
@Redino I can count at least 100 times this ai messed up. I only messed up once in a group of 50 levels or so. The only reason I didn't keep going was I played so long my ds died. Also I was bored as hell
@@hunterdunn2068 That's what a computer would say
As a barely ok programmer, i just wanted to say your projects are exactly the type of problems i wanted to solve if i had more skill, you inspire me. Currently recovering from a prp shot on my arm, so this video helped me in this difficult time. Oh and your avatar is cool, cheers.
Best of luck with programming, and I hope your arm recovers quickly.
Take care!
@@BakaTaco Thank you! 😁
Seems like you could double the accuracy by programming it so that it looks for Red on Mario and Yellow on Wario. Since those colors are unique to that character, you could basically ensure that those 2 characters are never missed. It wouldn't have to match tiles at all. Once it detects the color, that's where the character is located. That would just leave Luigi and Yoshi to the rest of your code. Since virtually no time would ever be lost and only added for Mario and Wario, I can't imagine that the script would ever run out of time.
You would lose since a lot of the time their hats are completely covered and you can only see their face so you need to match other parts of their face.
Maybe one could add the colour purple for Wario's nose.
well also their chins (when cut off) and mustaches (when theyre covered)
dude this is already in the first 20 seconds a nostalgia punch in the face nose break for me, new super mario bros minigames were my life
They where also in mario 64 ds
Yeahhhh.
I remember just vibing in Luigis casino all damn evening
@@jimmyjohnjoejr.9020 he wasn’t asking for somebody to give him a way of playing the game. If he played it, I’m pretty sure he’s got the ds to play it again somewhere
@@beans3669 🤓
Daaaamn, I remember destroying my little brother in the bob-omb tennis minigame, it was so much fun!!
Ok but can we get an AI that can learn to manipulate RNG and beat Luigi at poker
Edit: yes i have watched the video about toad manipulating the cards its just that i posted this comment before that existed
Probably, doubt they used a hard RNG generator. Might just have to use the time converted by same RNG generator that they used, or something similar. (Totally not human like, but who is counting, when using super processing to play match the face)
You don't need to manipulate the RNG, the odds are in your favor as the player anyway.
Don't tell RTGames about that possibility.
I have 8000 coins on that minigame…
Someone ready did It without AI believe It or not, toad , YES, THAT TOAD WITH THE DRINKS is the culprit of It all, look for beating Luigi poker using toad or something like that
This video gave me much needed inspiration for my final year project in my Computer Science degree at university. I was already looking into doing something around image recognition but didn’t know if it was viable. Thank you
This is one of the easier games on NSMB/64DS, but this is still impressive! I've finally started trying to learn Python myself, and people like you are quite inspiring. Seeing how the AI performed here, have you considered other minigames? For example, "Which Wiggler?" should avoid some overlapping, but also requires more precision. Hide and Boo seek would be interesting as well, though the task is very different.
I love how the ai isn't perfect and makes mistakes instead of nailing it every time, it's soo much more entertaining that way
Edit: It isn't AI goddammit, read the replies before telling it to me for the millionth time please.
Edit 2: okay fine it might as well be AI, only not machine learning. Okay? So unless you have something else to say please stop.
Mainly because it is too slow to click it on time, not an issue of finding it
I personally just think it is an excuse to not have to work on it until it's good. or maybe I'm just a perfectionist
I didn't mean as in its better for it, only more entertaining which in the bots case is its main purpose. I don't think he's trying to build a bot that has purpose of playing the game efficiently for research, more make it do something novel to make a video about. Thus it works perfectly since the goal is not to be perfect but to work for the novelty of a bot that can play this game
Also when I refer to it making a mistake I don't mean not clicking instantly, I mean actual mistakes of clicking the incorrect icon.
That's the entire point of machine learning though.....
@@thechugg4372 he wasn't using machine learning. Tbh idk if using terms like AI are correct but in reality it doesn't matter that much
You could've just searched for red and yellow pixels for Mario and Wario respectively, since red and yellow only show up for those characters. This obviously couldn't work for Luigi and Yoshi, but it might speed up the process and ensure no time is lost on Mario or Wario. Great program tho!
Except for on ones like the example at 4:34 . If I remember this minigame right that is a static one so the hats never show. There are probably also some overlapping ones that don't move that will also completely obscure the hat.
@@justin66asdf I believe the idea would be to automatically click only if the appropriate red or yellow is present (hopefully within a single frame). If not, then you fall back to matching the image.
@@ZipplyZane I was just coming to comment the same thing
But sometimes their hats is completely obstructed.
@@Mabra51 And? A one-size-fits-all solution is generally unoptimized
I still play this game to this day.
Me and my brother take this game everywhere when we go on holidays with our DSIXLs.
We always played Minigames and Mario VS Luigi and just mastered every single thing about the game.
Wanted is one of our hated minigames because we played it so much that we would manage to find the faces in less than 1 second of the minigame clock ticking.
It's exhausting to play against that level of experience.
Really glad you made a video about NSMB, it means a lot to me that this game is still being played, I feel so alone and stuck in the past for still playing DS Lite games in 2022.
That is so cool!!! I love your tenacity very much. It makes me giddy to see you keep pushing through even when faced with a new problem! I love watching people show off what they do with coding, it really is technological magic to me! Thank you for sharing!!
I remember specifically that I managed to get to Level 256 on this game. It was when I was younger, so younger me thought, "yeah, time to stop here".
I still do not know how I managed to get that high of a score.
Younger me, you are an absolute gamer.
SM64DS had this same minigame. I managed to get to 105 before I decided to end it there.
I got 201 and thought I was good man u were insane
Once you get in a rhythm, you have to be the one to stop yourself lol I got to at least 1000 before making myself stop
I got to +1000 and just stopped because I was bored
@@dile3725 exactly Same with me
I noticed that most of, if not every time that the AI fails, it's on levels where the icon moves. I guess that's because the reaction time between getting the position of the icon and the moment the AI actually clicks on that position is long enough for the icon to be out of reach. I wonder if there is a way to fix this tiny problem.
I think part of the problem is, that the icon are partially covered.
I wonder if the game gives priority to the top most icon or the wanted Icon in case of overlap.
@@sarowie I played this game a lot when I was a kid and replayed it recently. I'm pretty sure that it prioritize the top most icon so you might be onto something here. Although, I didn't check the code of the game so I can't be sure.
@@SweetSaucer This program doesn't utilize any neural net, so since there is no ML being performed it doesn't qualify as an A.I
What I mean by "prediction" is just taking the difference between two integer points, really not that complex.
There may be an even simpler implementation but you saying I'm "overthinking" it is strange to me since my solution is really only 3 steps which can be converted to three python functions.
Extra screenshots are not needed for determining trajectory.
@@SweetSaucer ML/AI is all about prediction, that's why I gave you the pretext when I went on to misuse "prediction", a synonym could have been "determine" but I agree with you, prediction is just easier to use and common vernacular
Every time a screenshot is taken you can just compare the coordinates, that's what I mean. Pic at time 5 and pic at time 7 will have different the location of the face at different pixel coordinates, no extra screenshots are needed.
OpenCV can actually do a very good job at locating the faces despite the erratic movements since it's written in C/C++, it can definitely perform the task in the vid
@@SweetSaucer You can give your perspective, I don't understand what made you so angry with me
This game is a masterpiece in itself. I've been playing it again recently and It is SO ENTERTAINING. Im glad you made a video on it because not many people truly appreciate this amazing game.
im so glad this came in on my recommended page, such nostalgia combined with something thats created incredibly. I loved it
Very Impressive! I noticed that almost all of its misses at the end came from the fact that the icons were moving and the AI couldn't keep up. I wonder if there's a possible way to fix that?
Use a few frames to test for motion then wait for the loop to come back and match.
What I like is that the minigames are the same ones on Super Mario 64 DS and you can do crossplay multiplayer between the two.
Finally someone that mentions SM64DS!
oh crossplay multiplayer like that is a thing??? makes me wanna test it out for myself
I struggled watching this at night :(. The white background just blasts my eyes. But other than that, seeing my favorite minigame from the super mario DS game with programming, it brings me smiles on my face. Good job!
Actually such a great video! I loved playing those minigames as a kid.
I wonder if it'd work better if you had a program seperate sprites' color schemes and look for colors that occur only in one character (like red for Mario, Yellow for Wario and there are probably more colors than that) so that if there's a red or yellow it can find the icon faster
I mean, that is essentially what the 'sub icons' thing was doing. Except instead of a single pixel (which could be covered up), dividing it into groups of distinct characteristics that are less likely to be so.
Counterpoint:
Chins.
@@sirsins9598 then have the chins as a final resort, and the colour scheme take place between the 2 since it's more reliable
Colours only helps for 2/4 options though, so it isn't an efficient solution. And as mentioned you may not see that speck of red/yellow if the layout is unhelpful
sho pfp !!!
I imagine this must've inspired Wii Play's Find Mii seeing as it uses a similar timer-based system to determine how far you get.
This mini game was my childhood, i played it up to the thousands so often! Sometimes leaving it on and close my DS just to keep pushing my record for weeks.
HO boy that was intense, i was expecting it to not work at all until it was able to go infinitely but it was so fun seeing it have close calls. Great video!
I remember playing wanted with my mom for hours, i got so good at it that I can now spot the faces pretty fast and accumulate a lot of extra time
Wow, maybe one day we could get an AI that can play the actual new super mario bros ds in real time
Isn’t that just a TAS?
@@thebananaspeedruns9275 A TAS is a human playing the game, but putting in each and every one of their inputs frame-by-frame & via a program, rather than real-time & via a controller. That's why it's called a _Tool-ASsisted_ speedrun!
@@overthinkit_ There are TAS that have real time elements to them. Im willing to bet this AI could count as one if the minigame was timed and had a leaderboard.
Also it Tool-Assisted-Speedrun. The last S is for speedrun, not part of the word Assisted
@@overthinkit_ oh hey it’s the guy from quinbobin xD
For that you either need and hyper advanced ai able to recognize paterns types of colisions, enemies, etc... or just mess with the game code and somehow make it send specific signals to an external program
No joke. I played this game a few days ago. If not, exaclty the same day you uploaded this video.
Prior to that day you uploaded, it's been about 5 years ago since I last played.
I haven’t played these games in years, thanks for the trip back to memory lane.
I’ve played this game so much as a kid I find it funny how I can find mario faster than the AI sometimes
That is because you sre looking for red pixels for Mario. I would guess you can also find Wario faster, while performing similarly to the algorithm when it comes to Luigi and Yoshi.
@@andresmartinezramos7513 Mario and warios faces are differently shaped to the point of being notable
Thanks so much for making the code available! I’m working on a project that requires image recognition and this will help me a lot!
This completely blew me away... Awesome job!
NSMB was ahead of its time. Also, I can’t code Python except for print, so you’ve earned my sub
That was awesome!
Gonna try programming in the next Couple of months too!
I loved these mini games as a kid. Really fun video, I never got past 15 levels
You can tell even when it’s getting it wrong it’s cos it’s miss clicking on the ones that are moving, since the mouse just moves to the point where it’s at then the icon moves it misses. Very cool creation
Just found this channel while looking at code bullet. I immediately subbed after watching this.
Very entertaining video! Love your stuff :)
Someone was certainly inspired by Code Bullet haha. Great video though man, you've definitely beaten my high score.
Woah that is super cool! And I'm sure a lot of people would get a kick out of this even if they don't code, great work!
You just unlocked a memory for me, wow. I was addicted to these minigames. Especially during long car rides.
I was wondering why it seemed to hesitate on a couple of the moving levels? During the timelapse it seemed to wait a while and let the timer run down, or make a few misclicks before getting it right.
It only updates 2 times per second so if the character is behind another it doesn't locate it on that frame. It also misclicks occasionally because it isn't always fast enough.
Not only that, when the movement happens, the Wanted Character isn't always at the front. Any time you click on the wanted character, when an unwanted character is in the foreground on the exact space you clicked, it will think you clicked the unwanted character.
I lost a 60 odd run that way about 10 years ago.
@@CodeNoodles would it be possible to make it update faster?
@@McTuber42 More optimized code, use c++, or put everything in a c/c++ library that is performance intensive and call it from python
@@CodeNoodles Makes sense. I'm an extremely beginner programmer so I'm trying my best to learn
Wow this was actually a pretty interesting video. Definitely earned my subscription
I'm definetely impressed 😲
You code is really clean btw, love that.
This video gave me such nostalgia from my childhood. Just, that game, those mini games.
I remember playing Wanted during a power outage and I purposely let myself die because I just wanted to do something different. Wanted might be one of the easiest mini games ever
If you use glasses, try doing it without them. It's near impossible lmao
@@grylltheonion I don’t wear glasses
I love this AI. It’s how a very good Wanted! player would play.
BRO, THAT NOSTALGIA IS HITTING ME SO HARD.TY for reminding that I've played ALL OF THEM when i was a kid...
Luigi is the hardest one to find because his eyes looks almost like Mario and his green blends in with Yoshi. I have had tries where I ran out of time all because I couldn’t find Luigi until it was too late.
that level 107 was rigged, luigi was hiding for like 90% of it.
OMG BRO I USED TO LOVE THIS GAME!!! Idk lemme check what my highscore on it was
Knowing that Mario used to be Miniladd's gmod player model and the thumbnail has a wanted poster of him makes this a whole lot better
0:01 i just found your channel and i already love you
NSMB is love, NSMB is life
very good content, loving it so far, please continue ^^
Nice video. Good idea with the subicons, that wouldn't have ocurred to me
It's so neat seeing someone do this to a mini game you remember playing from your childhood.
These minigames were so much fun. I think I played these just as much as the main game as a kid
Cool video, keep up the great work!
This game is a certified hood classic 🔥
That was so cool! The smb mini games on the ds were such a big part of my childhood !
I've always wanted to see something like this made
Heh heh
These videos are so cool
This must be the perfect video for me!! I played that game for hours as a child, and I’ve coded in Python for thousands of hours! This is so cool!
This was really cool it just shows that programming doesn't have to be all serious,technical stuff and can just be fun stuff like this and could you release the entire unedited or un-sped up footage of the programs entire run and maybe a link to a document of the code you used?
Glad you liked it! If you want the code for this project it is on my Github.
This is so satisfying to watch, such a great idea!
Hi
This is extremely well made and nobody but ppl who code get that.
I loved this minigame as a kid, I’d play it until I got bored. I think my high score was like 500 if I’m remembering right. Cool to see it so well automated!
Wow, this was really cool! This was great to get in my recommended. I love NSMB. I played it so much as a kid!
god i love my 3ds
glad to still have one :D
This is awesome. It seems there is only one bug to fix. It seems you programmed the tap to be the center of the face(or around there) and when the other icons cover that spot the code doesn’t notice and ends up missing. This would be hard to fix without knowing have tight the hit box is though. If it’s only as big and the same shape as the the sprite, then you can tell it to tap exactly where the sun image is, if not, you may have to make it wait until its sure it can hit the target.
The new super mario brothers minigames have been a key and fundamental part of my childhood and has kept me alive all this time
this was a blast from the past thank you for this
Wouldn't it make for faster image processing to check the hex value for a specific pixel on the top screen a set amount of time after it taps a character on the bottom? (Sorry if this makes no sense lol)
I made no sense until I watched but now I understand what you are talking about. Like if it looks at the hat colour and it matches Mario, Luigi, warrior, wa Luigi or yoshis eyes then skip the top screen search and just look at the bottom screen for that image. Another improvement would be to wait for the bottom screen to not be completely black, if it does not already do that
@@memetech- yeah the completely black screen thing is a hurdle but after you tap the right character the timing should be perfectly consistent of when the character shows up i think
I think I see what your saying. If it looks for the hat color though (excluding Yoshi) what if the hat isn't visible on the bottom screen?
@@CodeNoodles it’s always visible on the top screen. You see the colour, match it to an image in the expansive database of a few Mario characters, and continue where this already does
@@CodeNoodles i'm talking exclusively about the image recognition of the top screen. it definitely isn't the brunt of the image recognition loading time though lol
You could make it so mario just checks for anything red, and wario for yellow, since I think those are the only ones that have those colors.
EDIT: Never mind, this is a flawed idea.
if the colours of their respective hats where blocked by another character, there would be no coloured distinction that the AI could detect. Your strategy might work for the first few stages, but once the screen is filled there is a higher probability that distinctive factors are lost.
@@omicra6874 Good point, I didn't think about that.
I honestly forgot about that minigame but now that I remember, that was my favorite minigame
Honestly I would love to see a video of you explaining your code and what it’s doing, I know that would take forever, but as a CS student, it’s so intimidating to watch, but I wanna learn it so bad
I remember playing until like level 40 and we had this group of friends that would come together to see who got the highest level lol
Who got first most of the time?
Amazing Video!
The minigames were the highlight of my childhood, I spent 10 days on the road with my parents going to Orlando and I played this game pretty much the entire time
You just reminded me of these minigames!
I played these a lot as a kid
I LOVE your a.i. videos
It's definitely an AI title, but it's not an AI video unfortunately. Still entertaining tho!
Nvm they changed the title, now it doesn't say AI in the title anymore.
Nice video! The title is definitely clickbait tho, since it doesn't actually learn anything and is just a pre-programmed algorithm. I wouldn't really call that an AI...
They didn't call it an ai tho
@@characookie241 Oh they changed the title. When I wrote my comment the title was something like "AI learns to find Mario"
That "buh-bye" at the end is the cherry on top:)
I totally forgot about these minigames, these were amazing!!
Have you tried doing this for the Bomb-omb sorting minigame yet? Since that minigame has two distinct colors it shouldnt be that hard to code
whats up code pasta
Code Pasta is a good nickname 😆
this man casually made a recaptcha completor and used it for mario
Yo, weirdly enough this unlocked a memory, this was my favorite mini-game. I don't remember what game it was from though.
what program are you using to write the python code?
Vscode
Spy TF2 catcher
Lol
I really love how your channel is blowing up out of nowhere
Dude I played these minigames so much and at one point I got this game to 1000 score. Never thought I would see it in my recomended.
This game owned me when I was young. Watching the bot beat the crap out of it was very relaxing.
I use to LOVE the trampoline and jumping on the shy guys mini games. Along with the one where u have make a path for the Mario head
I'd love to see a step up from this to 'Find Mii' in Wii Play. Awesome work, champ!
This find mario just unlocked a core memory I hadn’t remembered in a decade
Ironically the mini games here taught me blackjack and poker.