Automating Android Games with Python and OpenCV: Pong
ฝัง
- เผยแพร่เมื่อ 21 ต.ค. 2024
- Pong becomes pretty easy when OpenCV is involved.
Hope you enjoyed the video!
Check out this code here:
github.com/eng...
Join my Discord server to chat with me:
/ discord
Check out some code on my GitHub:
github.com/rea...
github.com/eng...
Come visit us on Reddit:
/ engineerman
Other Social:
/ _engineerman
/ engineermanyt
I love this series! Why stop?
I think the three videos explain 3 different ways to automate a game and the three ways can be done for all games.
Because coding shouldn't be JUST copy & paste lmao If he just serves you the solution to every single game on a platter it kinda kills the fun of trying to figure it out yourself.
I hope that he put translate cuz im bad at english but i understand what he said for like 85% maybe.🤷🏻♂️
I think just for entertainment value these are great. Very digestible and interesting to watch, good for a quick lunch break or something.
My guess is that this series of videos on automating android doesn't have much expressions on viewers, if it's a big hit there definitely will be a lot automation tutorial. Maybe Engineer Man just not very interested. I don't know~
Instead of if statements, use max() and min()...
to = max(ball_y, paddle_min_y)
to = min(ball_y, paddle_max_y)
I love opencv and it’s an awesome way to automate games. I once made a bot in an mmo using opencv for the fun of it. You should keep up the game automation series!
With regards to it missing the ball when it goes wild, there's probably something that can be done to improve it by leveraging the "deflections" off the walls. That way the paddle could go directly to the landing spot rather than follow the y-coord
the man writes code like its his mother tongue lol
i think PyVirtualDisplay would help a lot !
or just plain Xephyr or any alternative
you can pass the DISPLAY variable to mss to control that display directly
this way you won't have to switch between window just to play the game
it's a lot better this way
Thanks for the great content ! i love your channel man
Hi! Thank you for yet another great video.
You are an inspiration!
I like the approach. I would perhaps consider only moving the paddle if the ball is near your side. So, x must be greater/lower than a specific value. This way you won’t move the paddle if it’s not your turn, and also you don’t have to move it, if the ball is going to change direction due to hitting the roof. I think this could save you some time since you don’t move the paddle for no reason.
Best regards from Denmark.
Keep up the great work.
To get hands dirty and to have fun at the same time! I love your approach!
This a cool video, I think you could make do with the slow adb commands if you tried to predict the path that the ball traveled. You could run a linear fit on the first few screenshots and check if it would hit a wall or if it would go though the paddle area with a simple linear intersection calculation, that way you're only sending one swipe command instead of using all of the bandwidth that adb has to send a command for every y value. Collisions with the wall are pretty easy to model since they seem perfectly elastic, also even without doing that you could still use this approach but just start the fit over after a collision
Dude you're a beast man. Love your channel! 💪🏻
Ever since I saw someone using OpenCV to play the beach volleyball game in Mario Odyssey I really wanted to know how they did it, and I guess this explains the basics it pretty well. Thanks!
Excellent video. Recommend to improve performance WITHOUT rooting the phone to see how good you can get the performance. Will force best coding techniques - squeeze it as much as possible!
Thank you Sir so much for these inspirational videos. Your videos inspired me to pursue automation and I've completed my course in browser automation using Python.
I'm happy to say that I've done many small projects like this which I've always wanted to do but lacked the technical knowledge earlier, including a full fledged chess bot which plays online chess against AI and humans.
Thank you Sir, once again for all the inspiration and knowledge you have imparted through your channel. I'm a huge fan of yours and I pray that your channel receives much more subs, likes and shares because you truly deserve them. God bless you.
Hey! I'm an inspiring student who is pursuing coding and I was wondering if I possible that I could get your chess code or any code that you have done. I was wondering if I could get this so that I could break it down and get a better understanding of your code plus to experiment and test with. Would this be possible?
omg how did i not find your channel earlier?! good stuff you just earned my sub
Just ran across your video. Great Stuff! Really looking forward to the higher performance stuff because i'm currently battling with it myself. Specifically speed. I'm having a lot of trouble doing swipe gestures at fast speeds.
Great video! I think I know a way though you could possibly improve detection performance without rooting. KitKat and above includes a screen recording api. On my essential phone and nexus 6p i can reliably get about 60fps over USB-c.
Man! I love this channel! And I've been waiting for another Android android automation !
please do the high performance version
thank you for doing these videos
it helps us a lot
keep up the great work
Hey man, what about trying using a virtual android device? Like bluestacks or MemuPlayer. You can easilly root them and you can do much more with it. You do not need to use a real phone. BTW nice video. I have no experience with Adb, so I would rather create a program with pyautogui in python and control everything through that with a virtual andriod emulator. It is easier in my opinion. What do you thing about this option?
Since there is no android app anyway, a cheap way is pressing the pc monitor where the emulation is. Pythongui gives that feature. I currently use autohotkey but then I miss all benefits of python libraries.
You could wait for the ball to come back (just after the opponent has touched the ball, that mean as soon as x(t2)-x(t1)>0 ) and compute where the ball will be to already place your paddle.
thanks for the video. are you controlling a real android phone or an android emulator? if it's emulator, which one is it?
best series for python moderate programmer
I have learned a bunch of python things because of Engineer Man
This guy: talks about very technical stuff
what got into my brain: oooooooooooooooooo words with color. dem pretty
whats the name of the emulator / remote access programm?
oof. What does the colon in circles[0, : ] do, and how does it effect "pt"? I am somwhat familar with slicing, but a single colon is new to me. Thanks!
It gets every column in circles which is actually a matrix, search for slicing in Python/Numpy/Pandas. After that, pt is a slice of the matrix with the x, y, r values from the HoughCircles method for the first circle (row 0)
why not use the last two positions to extrapolate velocity and predict the ball's path?
I'd love to see the next one. I'm working on a similar thing like this, but a little more complicated... Like, automating Pokemon go xD
Detect if ball is moving from left to right, grab two points, draw a line, does it intersect with paddle_x?, no? repeat, yes? then move paddle to intersection.
To overcome the latency you might calculate the y value only when the x value is within a range sufficiently close to your paddle. In this way, you would also be seeing your paddle save the game at the very last moment, suspance! :)
actually letancy isn't the issue here, he should only need to move the paddle once. if he'd actually calculate the trajectory of the ball ;)
@@90hijacked that's very true, with trajectory calculation there wouldn't be any need to raise concerns about latency
@@francescotaurone6563 what's the formula to calculate that?
My math is not the best
@@dev.tirebit If you imagine a straight line going out from the surface the ball will strike, the ball will bounce away from the surface with the same angle that it entered, but on the other side of a line.
For calculating the trajectory, taking the ball's position in two frames is enough to calculate where it's going to strike next. Simply draw a line through the two points (and extend it towards infinity on both ends). That line will intersect the next surface the ball strikes. To put that in programmatic terms, you can take your two points and extract the difference in y and the difference in x between them. Then use the ball's current position coupled with the speed in the x and y direction that you just extracted to figure out which wall it will strike. Crucial here is the fact that the total speed of the ball does not matter, at all. Only the ratio between the x speed and the y speed.
Thank you very much for this tutorial!
Since the program is now running automatically, we do not have to actually see the GUI. Is there a way to start this from a console now and let it run on a server for example?
I havent looked at the limitations of adbshell but could it be getting flooded with too many commans, like if you put a sleep in could it perform better?
also, what about minkeyrunner
Direction of the ball could be calculated and thus the arrival position with only two images
It could, but you would have to take into consideration that the ball bounces off of the walls....it'll need some calculations, but certainly doable.
Love your videos! BTW what IDE do you use?
He's using Atom. In his setup video he talks about what plugins he uses (for example the minimap one)
Why final ! It's so cool keep going :)
Hello, I am practically new to opencv, and I would like, if you can, give me information on how to apply the haralik texture (entropy, contrast, correlation, energy, homogeneity) to an image; what I have so far is gray scale and histogram equalization, everything I use with OPENCV - JAVA - ANDROID STUDIO and the image is .JPG
Could someone please explain how the interface with android is developed?
using adb
instead of taking segmented screenshots of your monitor you should screenshot your phone. this may not work fast enough on adb though it will make the code robust against moving the window around
Donot stop this series..please
I am too bad at math to figure it out, but if you could start to calculate where the ball would end up, already when leaving the opposite wall (including when it bounces off the top and bottom), it would give a lot more time to process the image and move the pad.
"calculate the trajectory" was the words I was looking for....
I was thinking this too. A bit "mathy" but it would definitely be faster in calculating where the paddle should be placed :)
@@dvdseroney
A short brainfart (take it with a grain of salt):
Take to screenshots in a time intervall Δt, here it was something like 10 ms. With openCV you will get the Ball coordinates (x0,y0) and (x1,x2).
We define the left goal as x=0 and the right one as x = L .
First consider the case Δx = x1-x0
@@Ryuuuuuk Great (and very detailed!) explanation even if without diagrams. The part I didn't understand was how you reduced the equation to ==> (y1-y0)*(x-x0)/(x1-x0) + y0 after plugging in the line formula (my math days fail me)
Otherwise this should work I think. For looping through the formula, my guess is this should be done at any point y reaches zero or y reaches the maximum (assuming y=0 defines one boundary in the game area and max y defines the other boundary) Looping through the formula should be done while those conditions are met or until x reaches the pad's x coordinate.
All considering the ball is moving to the right of course :)
@@dvdseroney ah, I never wrote the complete statement for
m = Δy/ Δx = (y1-y0)/(x1-x0) , which I plugged in
f(x)= m *(x-x0) + y0 = Δy/ Δx*(x-x0)+y0 = (y1-y0)/(x1-x0)*(x-x0)+y0
Does anyone know if i can run bot like this on phone like in app or in any other way just to work without being plugged to pc?
That's quality content
Man ! i loove this channel
Look into AutomaterUI 2 and the Appium Driver. Might be able to run commands quicker since it runs a background app on the phone activating as a server.
Can you automate the Android game Smash hit? It is really difficult. At least drop some tips on how to do it.
how do you get the phone screen to show up and be able to control it?
I wanna make a bot for a online mobile game called dice yatzy. When I probably can do but it seems the game blocks adb commands and my thought process was use opencv to handle the main buttons like Play, next game, etc and let it play like that doesnt matter if it wins or loses the games. Unfortenly i have ZERO coding knowledge and the most ive done was try to make my own bot to handle ai clipping works with captions, edting cutting and descripting. I used chatgpt and copilot to do all that
What about memory data structure scanning on android with adb. U got some tuts on that too?
how you display oneplus device in pc screen? Is it emulator? By the way this game automation series is very good I think you should not stop it here.
scrcpy
@@mohameai5997 thanks
Dude what is that software you are using to screen cast? Can someone suggest me a good one for Mac?
Check out OBS studio.
Would love to see a video with the rooted performance. Trying to develop something similar of my own with OpenCV without using scrcpy but still need to feed openCV screen data.
Whats the software used in emulating android?
You should try AutoTouch jailbreak tweak, it has image recognition, text recognition, and a plethora of other features for automation and is built into the device at the root level. It allows for a lot of cool automation and would love videos of you checking it out
which android emulator you're using
maybe converting screenshots into binary images instead of grayscale and lowering the resolution will help in performance without creating issues
Neetly done. Respect😎
what can we use on a rooted phone to improve perfs?
I'm almost there with another game, and figured to calculate the latency, and with really bad lag, I managed to get close to a perfect shot every time..
why you dont use NOX android emulate ?
Very nicely done
Awesome as always 👏💪
How do you get the mouse coordinates to overlay over the window?
Under developer options. There's a toggle to turn on coordinates.
I'm sorry, developer options of what? scrcpy?
@@noxew On your phone, in settings. Same place as you enable USB debug mode.
You should watch his other video, where he shows you how to get this setup, using adb and scrcpy.
Hope this helps!
@@foxdk thanks! I'll try that. I thought I did, but I must have missed that part.
Could you point me to where I can learn how to write a script that auto responds to a forum users new thread? As in, a designer drops 20 items for sale and it's fcfs rules. I just want to be first for once!
how do you open the android screen
Thanks!
What are you using for the android emulator?
scrcpy and real android phone I guess
Does anyone know how to download ADB module for python?
Does anyone know how to install that pong game? I really want to try that .py code. 🙏
think threading would not have been necessary since python does not support real threading (global interpreter lock)
GIL prevents code from running in parallel. What you see here is code running concurrently, which is all I really needed.
@@EngineerMan okay but i don't understand why you need the concurrency, since you can only move the bar after determining the position anyway?
@@barneee5418 sir , can u help me with a game ,
there is a game called Ludo King, where you spin a dice and a no. will come and then you move that much forward, i want to change that and make it like so i can choose which no. to come , it is very important for me , if you can help then it will be great for me .
What if game is fullscreen
which linux distro?
Love the video, Thx
You should try Piano Tiles :)
Where is the high performance vid? You lied?
I love this , I subscribed , but it gets difficult for me to understand sometimes . Please help me out . I want to try to do the same for my game but how to set this up
I wonder when he going to do a part 2
can u do web browser pong games automation
Hello sir, I was trying get Live Chat from www.nonolive.com but failed to do so. I was wondering if u could help me. I also want to sign in to post chat... I admire ur work thank you...
This is a really great tutorial!. Imagine getting a heart from engineer man....
“Text here” in the description
Hey do one for fruit ninja... That will be awesome
can you make an aimbot for a shooting game?
HOW. DO IT ON ANDROID
print("Hello World!")
You are super cool
you are great
could you please include a walkthrough of how you are going to be rooting your phone?
still waiting!
👍👍👍👍👍🙂
You are awesome
Waiting for your rooted method
Instead of using adb commands to move the paddle, try using win32api commands or pyautogui commands to control the mouse cursor to move the paddle , its definitely faster than adb
Don't stopppppppppppppp
text here
text here