The most frightening thing about AI is that it's absolutely terrible in everything until it's not, and when it's not, it absolutely trashes every record ever. It's everything or nothing.
Does it ? Have no chance ? What would be a good benchmark for tracks it has never seen before ? Let's say it is able to finish 90 percent of maps that it has never seen, is that any good ? Does it need to do bronze, silver, gold or author medal ? Curious to have your expectations, you know, for a future video 😁
@linesight-rl what I meant was that it doesn't have much chance at performing as well as top players without any further training. This map is pretty uneventful. I'd say that AI having a top 100 time with a few hours of training would prove that Trackmania can be solved in the near future. Right now, I think it's highly overfitted to this particular track, and that's why it's so good.
Please try a trial map, even if only a small one. I would love to see what the AI can achieve on tricks that take hours for humans to perform. Maybe oachkatzlschwoaf on ice? Or only a segment?
I second this for sure, the reward function would be kinda hard to make and would likely prevent shortcuts and such but it would be extremely cool to see regardless.
problem with kacky is that its black and wight often, you either get it or you dont, so setting good rewards for getting close is SUPER hard, if they could do that i'd be super impressed.@@thomaslomba5181
I think the next step should be meta-learning, so that agent can start learning on the new map without weights resetting. RL^2 is the simplest meta-rl algorithm, give it a try!
Absolutely this. It might be really tough to integrate because it seems like a lot of the reward system that you use is based on learning the track, but if you can implement some form of meta learning it will cut down on your work per map by half at least. This would then free up more computing power to attempt higher frame rates while keeping the same amount of time to get results. Great work!
Yes maybe I didn't understand, if they tried to make the AI trained on the previous map play this map it wouldn't work ? Would the AI just try to follow the path of the previous map ? I'm confused
@@gregzerThe results from the inputs the AI is getting are tailored more specifically to this track. If you change the track, the AI would probably do "ok-ish" depending on what type of learning they used here, but it definitely wouldn't be optimal at all like we see in this video. Think of it loosely like learning to comprehend English, and then being told to read and comprehend Spanish. You've trained for so long to read English and comprehend that, but now you have a different problem you're trying to solve with (mostly) all the same types of inputs. Sure, you can "read" the Spanish, but you won't really know what to do with it to get any value out of it. It's not exactly the same type of thing, but the general idea is there.
@@gregzer The AI doesn't learn by doing random inputs until it finds ones that work. It instead looks a an image of the screen and tries to predict what it needs to do next, the longer it does this for, the better it gets at predicting
@@jjjoker5766 Sure but with a reward function running on 7 second, it's likely that the AI's strategy toward the end of the training is to find the closest picture from current one on the better previous runs and deviate very slightly on the 140 next inputs (20 fps x 7 second). Which means that if you give it a new map to learn, it will get confused cause it will try to replicate parts of the 1st map it got trained on, and the more maps you try to add the more confusing it will be, which means it will perform worse on the maps where it previously got close to WR while never reaching such times on the new maps. This is where meta-learning can provide good performance improvement, the IA would be less specialized and a bit more flexible, while (with a fair bit of luck) saving some data so they could improve fps or picture quality.
Well failing 53.xx on past streams I did not expect the AI to do so well to literally equal my time! Such a smart way of explaining and running this experiment too, mindblowing work!!!
Thats awsome! Well done. But my bet for the best further improvment is not the framerate, but the resolution. With higher resolution the AI should be able to take the corners more tightly
If performance is an issue with higher resolution, it would maby be an option to ignore parts of the frame? Like ignoring the top of the screen and what is located left and right of the car, to not bother about unnesesary pixels.
Higher resolution is already gained in a way by higher frame rate. In the same way you can see through very small vertical slots by walking past them, the AI can use pixel data from one frame to the next to determine very precisely where the walls are and drive close to them. Increasing resolution would be more costly as far as performance and likely would show less results than framerate increases.
frame rate is also very important, in the video the AI can only use a combination of 20 different key setups per second. With higher framerates, this number will increase so there is a lot more space for improvement
I love how your videos related to AI with trackmania is one of the main reason why I'm passionnate about AI now that I know how silly I could be if I could hold this much power into making my own little cars go vroom. Thank you for this video, I loved it !
I would love to see a more technical video about this AI, like what/how you got game information, which infomation you have used to feed the ai, and drawing GUI into the game window, awesome video btw!!!!
Absolutely phenomenal ! I will eat my hat if an AI ever finishes Deep Fear respawnless (or at least a couple checkpoints) with that same learning method, prove me wrong !
It is important to note that this model wasn't trained to play Trackmania, but was trained to play this one map. It has most likely just remembered what each turn looks like and the optimal way to take it, rather than figuring out how to take each turn as it sees them. A good way to train a neural network for this sort of role is to give it lots of pro and intermediate level replays of different maps and have it learn to predict the player's inputs. Then after a while of training for this, you take this model and make it do its own runs on lots of different maps and reward it based on speed for further training beyond what humans have achieved. This is how AlphaStar and AlphaGo were trained. Alphastar is a Starcraft II AI that is the best in the world and AlphaGo is a Go (a board game more complicated than chess in strategy) AI that is also the best in the world. They were both made by DeepMind, one of the biggest AI research companies in the world, owned by Google. One last interesting bit of information, it was DeepMind who made AlphaFold, the AI that solved the protein folding problem and will likely win a Nobel Prize.
The caveat is that it's making its improvements over people via being able to perfectly speedslide in a way a human physically can't. It's fundamentals are a lot weaker.
This is incredible, i love this. I am studying AI in uni and i hope, with more experience, to be able to make fun projects like this soon. I can't wait
I'm impressed how well you got it working with visual input! It's interesting to see how people either overestimate or underestimate how well AI will perform. A balanced take is quite rare. As someone who's trained racing AI before, I'd love to see this applied to other prestigious tracks :D. I expect it would train a lot faster if you give it data like speed, the dot product of velocity with the car's forward vector, and one with the sideways vector. The latter really helps it learn to control the drifts in my experience.
Thank you for your kind comment. I'm curious which kind of racing game you tried to apply RL to? As for the additional inputs you're suggesting, I did not take much time to explain this in the video but we are using this type of inputs. They are necessary along with the image to judge the car's dynamics, orientation, etc...
I love that one AI training's most primordial arenas is its use in driving. Automating travel is, ironically, one of those things that splits people down the middle in terms of modern issues. Some people see driving as something that is dangerous to automate because of potential injury or fatality, others would rather sleep on their hour-long commute to work if they're far from home. Then, on the opposite axis of the grid, you have Trackmania players; automating the same exact thing, but for COMPLETELY different reasons. Trackmania having AI drive its tracks could be useful for any number of reasons: it could eke out milliseconds and show players tricks or strategies they may not see, they can be used to make more definitive gold/silver/bronze times with a more reasonable baseline of difficulty by tuning its abilities based on what medals you want it to achieve. I don't know where I'm going with this, I just found it funny that even in the virtual world, some people still have arguments against having AI drive cars despite demonstrably less stakes at play. This was a fascinating video and gave me a lot of food for thought. Thank you for all your hard work.
Kindheitstrauma would be incredible to see how the AI handles extremely high speeds, wall rides, and loops. Additionally a kacky map would be absolutely insane, but honestly, from a layman's pov, Idk if it'd even be possible 😅
really impressive work! it is often i come across ai playing video games and then being disappointed because the performance wasn’t great. This is a different case. Absolutely amazing job at getting the ai to work so well. I think it might lie in the rewards system being so well made(i am not an expert so please tell me if there’s other reasons for the ai’s performance) Props to all the people who worked on this, amazing job.
Very nice video 👏👏 Your next challenge is to create a ‘general intelligent’ driver that could finish an arbitrary map. Maybe by changing the map every few training steps the model could build more general understanding of trackmania driving.
Can you train an AI on the random map challenge? That way you'd get a more general AI. It might not perform as well on individual tracks, but overall it might do pretty good.
I don't think that's how it works, the AI learns using a trial and error method, meaning the AI can only learn a map at a time. If you put it into a different map, it will be just as bad as it was in the start, and will have to learn everything again. At least that's what I understood from the video
@@vitorrodriguez3799 It could work. It would probably require an extra layer and more neurons per layer, so it would take longer to train. Also there's a chance that it won't learn the maps well enough because of the complexity, but you don't know until you try.
@@vitorrodriguez3799because it's trained on a single map. But if it's made to be trained on random maps, it will learn probably slower but it will eventually be good on all sort of map (as long as it doesn't requires tricks the AI doesn't know)
Incredible Work! How good does this agent perform on other maps? Did it learn to drive good in trackmania overall or did it overfit on Hockolicious? How important was the graphic input compared to the engine input?
a few things that could improve the ai: - increasing fps like you said in the video (i think 30 fps is good) - let it have analog steering input (if that’s too complicated limit it to 0, 20, 40, 60, 80, 100% values) - maybe give color screenshots?
This is insane. Amazing work! I'm interested to see it learn to do other tricks on its own like wall bangs, air brakes, speed drifts, ramm hit, etc, or maybe even start finding cuts that no one sees.
it would be nice to make an ai that is made to be good at generalization, eg, capable of doing new maps it hasn't yet seen fairly well/learn new maps quickly as it starts with good baseline "instincts", rather than it just potentially overfitting. But i understand that that would be much harder to do so its totally fair that it is the way it is. Incredibly impressive work! keep it up.
I wonder if it would become more capable if you rewarded it for its position throughout the run, instead of its own estimation of the final time (if I understand how it works anyway). Very interesting project, this deserves way more views
THE tastyy ?!!!!! Your time on this map is really impressive, I hope my computer will beat it some day because I personnally struggle to just get the gold medal!
God I love ml so much. Fantastic video. Curious on the optimization of the relationship between framerate and resolution. Read some other comments noting that an increase in framerate would essentially be an increase in both, although I would love to see a simple experiment using a simple drift track with framerate and resolution control values for testing. Can’t believe tmnf community is still going so strong after 2 decades, makes me so nostalgic
Incredible, it really is the perfect run, every drift endes exactly where it should, and I think the car accelerater a bit faster when “sliding forward”
This is very super impressive. Since you asked whats next: no idea how feasibly this is but would be cool to get gold medal on every track in the campaign/career in TMNF, driving on different surface types with the same model might be interesting
So, the objective would be to have one neural network that is trained on all maps at once, and able to reach gold medal on all these maps? It's definitely something we can try, I'm curious to see how it'll go.
This is really impressive. Great work ! I think you should experiment with a depth map input layer. Maybe by pluging in the DirectX's API. It might help the AI's better perceive it's position on the racing line and act accordingly.
In 16 and have been interested in ai for a few years now. My plan was to go to college with a focus in studying it, and I really wanted to eventually make a killer trackmania ai. I should've known with the current "ai revolution" (lol) happening that someone would definitely do it way sooner, but I'm not giving up. I'll make my mark on this game.
I know someone has probably suggested this, especially if youve been chatting with peeps in the TM community but, you can slow the game speed down rather than trying to increase your FPS. Theres been cheaters who've done that to set amazing times with human reflexes
You might find an improvement in time if you add a reward for least amount of time per checkpoint or overall time. I suspect that your max distance in 7 seconds is making the algorithm optimize for a slightly less than ideal line as the shortest route to the finish is likely to be the fastest. So if the current AI would be given an additional target of quickest time that weighs more than the distance traveled it might find more optimizations. I think that more FPS is going to help the AI mainly in busy sections where there is a lot of changes on the screen (repeated highspeed curves and so on) but I doubt it is going to do much in the way of increasing the overall time. One more reward that you might want to think of even without the time based reward is a reward for less direction changes, this should result in a smoother ride and less direction changes which in turn will result in a higher speed as cars are always fastest in a straight line and will slow down when changing direction. Besides that I'd say keep up the great work I love seeing how quickly the AI progresses from knowing nothing to reaching world record pace in just 70 hours is pretty impressive.
I would love to see a write up with more detail about the training. I know it wouldn't make a good TH-cam video but I'd be absolutely fascinated. Obviously the next step is generalization. You could use Cup of the Day Maps as a potential training pool, though perhaps it would be best to ask the mappers.
Putting the AI on an RPG track would be really interesting. Seeing it try to find the right track, and when it finds it balancing on beams or driving on blue walls would be interesting. Some sort of easy Mini RPG would be interesting to see.
I've seen many people suggest maps that are not available on TMNF... Do you have a specific map in mind? Let's start with an easy one if we're already changing map style though :)
@@linesight-rl I don’t really have a map ready, but it should be pretty easy to build. A couple blue walls that you’re supposed to drive on, maybe downhill ones to add a bit of challenge and then those double beams that you can barely drive on interspersed with checkpoints on platforms should be enough challenge. Maybe like the end of oach, so that it’s supposed to go fullspeed and jump into the finish.
At first I was sceptical of this approach, I thought having the AI deciding what to do next in each moment would make it gittery but it was very smooth in the end. I'm thinking tho, as humans we obviously don't decide what to do next for every frame, perhaps the AI would be more smooth out of the box if it had to commit to turns (like left 70% or full right and break) and then have a separate agent that considers the risks and if the risks breach a certain threshold, the AI aborts its turn and does something new. Another idea: Ik this is way easier said than done but would it be possible to instead of button presses, you give it certain actions, like turn left, turn right, bugslide, and so on, but obviously way more options, all the actions could be induvidually trained beforehand with the system you're using now using many different maps so that it's a pro at all the separate actions.
I have to wonder if a "two part" AI might be possible: A general "how to drive in Trackmania," portion and a "what do I need for this track?" portion. Training might involve cycling randomly through a couple dozen tracks, with the track specific portion being swapped in and out.
Keep in mind that 20fps is a frame every 50ms so the AI has a 50ms reaction time. Thats more than 4 times faster than most people. It doesnt really need more fps to play well. It isnt analyzing the input frames as a video like humans do, it just sees 20 pictures every second and decides which buttons to press for the next 50ms. It would be possible to slow the game down to give the AI more fps if processing power was an issue.
I'm a pretty big noob in terms of AI, but I have a few random thoughts. Having played TM a lot I'm concerned about 20 ticks. That could be a source of problems and the AI would probably benefit from being able to make commands asynchronously from the frame data. I don't have any experience with RNNs yet, but I'm assuming there's some kind of learning rate scheduling? It's performance is very erratic from run to run and I feel like it should be trying to refine more after about 40 hours. Also, is there a reason why there's no reward for new or matching PBs? I wouldn't apply it to sectors because sometimes the fastest way on a map is not by optimizing each sector but by setting up to snowball speed. In fact, I would try extending the rewards window of time, or changing the reward scheme because it might hold it back from finding the best solution. Very cool project though, I've been thinking about doing this exact idea for ages since the first TM AI videos came out! I was convinced an artificial vision approach was going to do great.
Can you PLEASE upload a tutorial video, maybe even on a different tutorial channel? Your voice is extremely understandable and your explanations are amazing and make sense. Instead of watching 10 hours of people repeating themselves over and over again I would like to actually learn. Pleaseeeeee consider. thank u
Some great work! Your code for a TH-cam video is cleaner than code submitted with conference papers… Interesting you’re using IQN 👀. What do you do for the reward shaping? I expect this video to do great, but just as a tip from one TH-camr to another, it would probably even do better with a different thumbnail.
😍 Love your videos, I've been subscribed for some time to your channel on my personal account ! Thanks for the compliments on the public code, our latest (private) version is both better and cleaner. I thought I had done a good job for the thumbnail, can I get in touch with you to discuss what could be better ? I'll use the email in the "about" section of your channel.
Happy to hear 😊 And sorry, I didn’t mean to be too blunt about the thumbnail comment 😅 it’s a decent thumbnail, I just think the video is top tier, and if you can get enough people to click on the video I’m guessing it will blow up And yeah that email works 👍
If we're going in this direction, the AI has also improved its time :o) I think our last run on map5 went down to 2:02:08, but we didn't update the video.
I think the key to a better and more consistant program would be for it to spend time learning the optimal acceleration, velocity, and position for each possible section of road in spans of 20, 30, 40, or more car lengths and also give it a full map to analyze so that it can plot a path between each section in the optimal way. I have seen a few videos were people allow programs to train on entire maps but this seems a slow way to train them because the program isn't focused on learning situations, it's focused on it's immediate situation. This is not how people get good at racing games, they learn how to drive sections of road and adjust their driving actions based on the section they are in and the next section coming up. This is made optimal for people when they already know the entire map, allowing them to know what to do on the entire map making accuracy of their inputs the new challange to become faster. The latter would not be a problem for the program and would mean that a programs ability to become faster on any map would be directly linked to its driving ability on individual sections of map not the entire map. Just a guess... I don't write programs so I can't be sure of this opinion.
Your idea is interesting, and it is actually what we are doing at the moment :) You may have noticed in the video when I say : "the AI predicts the reward that will be obtained in the next 7 seconds". This is how we define the sections, in blocks of 7 seconds. Each transition is interpreted as being part of a section chosen at random during training.
After some more consideration, another difference between pro and casual players is the implementation of certian actions such as power slides and other tactics in specific situations. Have you considered gathering a list of pro tips to save into the program for use in these situations? Or if you want pure self learning you could impliment a recognition of specific tactics in the reward system and the ability for the program to save them to memory and use them in the future when it comes across them on it's own . Maybe, even allowing the program to save things it selects as possible tactics to test... maybe it will come up with a new tactic. My suggestions are a reflection of how I see people learning and getting better over time. Most pro players take tips from the community, I don't see why allowing the program to do the same would be wrong. Unless you want a purely self taught program, but then I would definately suggest allowing it to save useful tactics for itself to test, perfect, and use.... such as power slides, the new gorilla grip, bug slides, ect.
I am not very good at Track Mania (in fact I have never played it), but I would love to see someone beat the time of that AI under its handicaps (grayscale, 160x120 pixels, and 20 FPS) It may not have beaten the world record, but there is no way the record holder could beat it under those conditions.
One thing I'm noticing, as a human: there is a lot of useful information contained in the color. Perhaps feeding the AI a color image might help it do even better, especially given the low resolution. (The luminance still seems to be the most important, but the Cb and Cr can always be lower resolution.)
FS must be pretty fitting for reinforcement learning since the reward can always be distance travelled/ higher velocity without having to adjust for long term reward.
how explorative did you make the AI? if you have more compute time on your hands it would be interesting to see how the AI would do with higher exploration (and FPS, as you mentioned). loved the video+presentation btw :)
You could try feeding it not just the current frame, but the last one, too. Maybe the AI can figure out how to derive motion from that and have it help the decision making.
Yeah I was surprised this wasn't done in the original project. Knowing not only position but velocity and acceleration are very important when modelling driving.
What I'd like to see (or try) is to have a selection of a N (e.g. 20) different, popular maps, and run learning in parallel and in batches, with random sequence of maps for different learning periods, and then test those against different maps and see if they can e.g. beat human record on 1st try :D
One thing that i suggest, for the "more than 20fps" question you posed is to use a TAS'ing software to judge, setting it to pause the game at every, say, 60th of a second or 120th
riolu AI i think its less about real-time processing time and more about total processing time, adding a one to any limitations will raise the complexity exponentially
i think having the AI run through the entire campaign might be fun, since it only needs gold to do that, i think this could be achieve in a reasonable itme, and i wonder if the AI could take lessons learned from previous maps with it. (i wonder if that speeds it up at all)
It absolutely can. The question is how well. Realistically, the more maps the AI trains on the better it should be at any new random map. And the less training time it would need to adapt to a new map. AITango has some videos on Mario Kart AIs where they trained it on 1 map over the course of a day or so, then trained another AI on I think 4 maps, which took significantly longer, but then when they placed that AI on a new track it only took an hour or two to learn the track just as well as it had the other 4. These types of AIs learn more similarly to humans than people realise (at least from a results-driven viewpoint), they just don't have the initial corpus of knowledge from a lived life that humans come into any new task with. LLMs do - you can explain to them the rules of a game they've never seen before, and they can play it, albeit with poor skills. But they're not just randomly guessing moves unlike AIs normally do at the start of training - they draw from their existing knowledge to form loose strategies in the game just as a new human player would.
Hello there, hope this message finds you well. I'm incredibly impressed with what you managed to achieve here, the AI capable of setting WR performances while relying on the visual clues rather than some exact data extracted from the game is something truly unbelievable and it left me speechless for several days. The impression is so strong, that I together with few other friends of mine decided to attempt to develop a similar (although way less capable) AI in our favourite environment, which is one of the hardcore racing simulations. But since we are beginners in the matter of developing complex neural networks, I'd like to ask you, if you would not mind to guide us in the development of our own racing AI. We believe that making this work reasonably well in the realms of complex simracing games would push the limits of what is possible even further and make the overall impression broader. But to get to the point of even partial capability we would need your precious expertise. Would you be please open to help us out? Sincerely. Michal
@@htgnef the environment we are trying to get it working in is called Live for Speed, it is a hardcore racing simulator, so it surely will be a much bigger challenge to make it work properly (let alone at decent speed), but we are determined to give it a try.
Hi. We're too busy to help you with coding this, but if you have discord and join the Trackmania interface discord (discord.com/channels/847108820479770686/1150816026028675133), we can talk about how it could be done and maybe answer whatever questions you might have.
I would love to see if your AI could match the winner of the midori map. Especially if it can learn to use the unique turning needed on it to get that extra speed.
i have a small question, what youre working on now? are you training the ai on a map or smt? i wanted to know, because i really like how the ai are getting soo good to the point that they can already do simple maps in trackmania, i would love to see more complicated maps tho, anyway, gl on future projects
Thanks for checking up on us :) We're still hard at work on this AI, we have enough results to make the next video. It's a lot of work, but I think you'll like the next video. Just need to be a bit more patient :)
Good Job :) Further improvements could be using a pretraind vision transformer (e.g. fastvit) or cnn (e.g. mobileone) from the pytorch-image-models git and maybe (pre) train the model with rl from observation (since you can extract a lot of wr-runs and their inputs from the exchange)
What GPU/hardware setup did you have? I didn't see this listed anywhere. I'm amazed by the 70 hour training time. I'm wondering if this result is mostly a testament to hardware improvements, OSS improvements, or your own ML implementation.
What I would find interesting would be, if using a model pre-trained on a different map, would lead to a faster learning speed. So one could estimate, whether the AI actually learns to drive, or just learns the layout of that one map.
I'd suggest adding previous frame as input for Ai as well or at the very least, its commands on the previous frame that way it won't need to guess its previous self to be "not indecisive"
WTF??? an AI drifts cleanly on hockolicious!??!??!
The most frightening thing about AI is that it's absolutely terrible in everything until it's not, and when it's not, it absolutely trashes every record ever. It's everything or nothing.
It's not really frightening when you consider that it has no chance on levels which it didn't see before.
@@bbbndddl until it does. And when it does, it'll probably beat every human driver ever no matter the track
Does it ? Have no chance ?
What would be a good benchmark for tracks it has never seen before ?
Let's say it is able to finish 90 percent of maps that it has never seen, is that any good ? Does it need to do bronze, silver, gold or author medal ?
Curious to have your expectations, you know, for a future video 😁
@linesight-rl what I meant was that it doesn't have much chance at performing as well as top players without any further training. This map is pretty uneventful. I'd say that AI having a top 100 time with a few hours of training would prove that Trackmania can be solved in the near future. Right now, I think it's highly overfitted to this particular track, and that's why it's so good.
Please try a trial map, even if only a small one. I would love to see what the AI can achieve on tricks that take hours for humans to perform. Maybe oachkatzlschwoaf on ice? Or only a segment?
it'll take months but i kinda wanna see that lmao, or some easy kacky xdd
I second this for sure, the reward function would be kinda hard to make and would likely prevent shortcuts and such but it would be extremely cool to see regardless.
this
problem with kacky is that its black and wight often, you either get it or you dont, so setting good rewards for getting close is SUPER hard, if they could do that i'd be super impressed.@@thomaslomba5181
this is a great idea, though an rpg seems easier for it.
I think the next step should be meta-learning, so that agent can start learning on the new map without weights resetting. RL^2 is the simplest meta-rl algorithm, give it a try!
Absolutely this. It might be really tough to integrate because it seems like a lot of the reward system that you use is based on learning the track, but if you can implement some form of meta learning it will cut down on your work per map by half at least.
This would then free up more computing power to attempt higher frame rates while keeping the same amount of time to get results.
Great work!
Yes maybe I didn't understand, if they tried to make the AI trained on the previous map play this map it wouldn't work ? Would the AI just try to follow the path of the previous map ? I'm confused
@@gregzerThe results from the inputs the AI is getting are tailored more specifically to this track. If you change the track, the AI would probably do "ok-ish" depending on what type of learning they used here, but it definitely wouldn't be optimal at all like we see in this video.
Think of it loosely like learning to comprehend English, and then being told to read and comprehend Spanish. You've trained for so long to read English and comprehend that, but now you have a different problem you're trying to solve with (mostly) all the same types of inputs. Sure, you can "read" the Spanish, but you won't really know what to do with it to get any value out of it.
It's not exactly the same type of thing, but the general idea is there.
@@gregzer The AI doesn't learn by doing random inputs until it finds ones that work. It instead looks a an image of the screen and tries to predict what it needs to do next, the longer it does this for, the better it gets at predicting
@@jjjoker5766 Sure but with a reward function running on 7 second, it's likely that the AI's strategy toward the end of the training is to find the closest picture from current one on the better previous runs and deviate very slightly on the 140 next inputs (20 fps x 7 second).
Which means that if you give it a new map to learn, it will get confused cause it will try to replicate parts of the 1st map it got trained on, and the more maps you try to add the more confusing it will be, which means it will perform worse on the maps where it previously got close to WR while never reaching such times on the new maps.
This is where meta-learning can provide good performance improvement, the IA would be less specialized and a bit more flexible, while (with a fair bit of luck) saving some data so they could improve fps or picture quality.
Well failing 53.xx on past streams I did not expect the AI to do so well to literally equal my time! Such a smart way of explaining and running this experiment too, mindblowing work!!!
Man you have to find a way to get yourself in the spotlight because this is amazing content and you deserve more than a couple 100 subs
Just send the video to wirtual on stream, i dont use twitch so im not gonna do it
Let time do its job, like the old days, it's good content after all...
This is incredible to watch. I also love how you intuitively explained how the learning system works and all, that's very cool
Thank you very much!
Thats awsome! Well done. But my bet for the best further improvment is not the framerate, but the resolution. With higher resolution the AI should be able to take the corners more tightly
If performance is an issue with higher resolution, it would maby be an option to ignore parts of the frame? Like ignoring the top of the screen and what is located left and right of the car, to not bother about unnesesary pixels.
@@nanoder7te maybe even the shape of an eye is enough data
Higher resolution is already gained in a way by higher frame rate. In the same way you can see through very small vertical slots by walking past them, the AI can use pixel data from one frame to the next to determine very precisely where the walls are and drive close to them. Increasing resolution would be more costly as far as performance and likely would show less results than framerate increases.
frame rate is also very important, in the video the AI can only use a combination of 20 different key setups per second. With higher framerates, this number will increase so there is a lot more space for improvement
@@vitorrodriguez4278 makes me think of riolu... that's basically what a higher framerate would mean for the AI.
I love how your videos related to AI with trackmania is one of the main reason why I'm passionnate about AI now that I know how silly I could be if I could hold this much power into making my own little cars go vroom. Thank you for this video, I loved it !
I would love to see a more technical video about this AI, like what/how you got game information, which infomation you have used to feed the ai, and drawing GUI into the game window, awesome video btw!!!!
this is amazing! i'm relieved to be part of the 23 who outclassed the AI x)
This is so cool! I always though ai driving in tm would never be this optimized. Well done! 🏁
Absolutely phenomenal ! I will eat my hat if an AI ever finishes Deep Fear respawnless (or at least a couple checkpoints) with that same learning method, prove me wrong !
Noted
Noted
Note to self: return to this comment section with a tasty hat once AI finishes "Deep Fear" respawn-less.
Should be easy for an AI if he can watch the records 🤣
AI wtf man just copy the wr..
It's brilliant! Well done, you're super good ! Good luck for the rest
That’s genuinely so amazingly cool that we can watch a computer program do things like this, and also things like TASes.
It is important to note that this model wasn't trained to play Trackmania, but was trained to play this one map. It has most likely just remembered what each turn looks like and the optimal way to take it, rather than figuring out how to take each turn as it sees them.
A good way to train a neural network for this sort of role is to give it lots of pro and intermediate level replays of different maps and have it learn to predict the player's inputs. Then after a while of training for this, you take this model and make it do its own runs on lots of different maps and reward it based on speed for further training beyond what humans have achieved.
This is how AlphaStar and AlphaGo were trained. Alphastar is a Starcraft II AI that is the best in the world and AlphaGo is a Go (a board game more complicated than chess in strategy) AI that is also the best in the world. They were both made by DeepMind, one of the biggest AI research companies in the world, owned by Google.
One last interesting bit of information, it was DeepMind who made AlphaFold, the AI that solved the protein folding problem and will likely win a Nobel Prize.
Never expected AI to get this far this fast. Awesome job! Makes me wonder what is to come. Loved your explanation too
The caveat is that it's making its improvements over people via being able to perfectly speedslide in a way a human physically can't. It's fundamentals are a lot weaker.
This is incredible, i love this. I am studying AI in uni and i hope, with more experience, to be able to make fun projects like this soon. I can't wait
Not only a super impressive time by the AI, but also a sick video
I'm impressed how well you got it working with visual input! It's interesting to see how people either overestimate or underestimate how well AI will perform. A balanced take is quite rare. As someone who's trained racing AI before, I'd love to see this applied to other prestigious tracks :D.
I expect it would train a lot faster if you give it data like speed, the dot product of velocity with the car's forward vector, and one with the sideways vector. The latter really helps it learn to control the drifts in my experience.
Thank you for your kind comment.
I'm curious which kind of racing game you tried to apply RL to?
As for the additional inputs you're suggesting, I did not take much time to explain this in the video but we are using this type of inputs. They are necessary along with the image to judge the car's dynamics, orientation, etc...
I love that one AI training's most primordial arenas is its use in driving. Automating travel is, ironically, one of those things that splits people down the middle in terms of modern issues. Some people see driving as something that is dangerous to automate because of potential injury or fatality, others would rather sleep on their hour-long commute to work if they're far from home.
Then, on the opposite axis of the grid, you have Trackmania players; automating the same exact thing, but for COMPLETELY different reasons. Trackmania having AI drive its tracks could be useful for any number of reasons: it could eke out milliseconds and show players tricks or strategies they may not see, they can be used to make more definitive gold/silver/bronze times with a more reasonable baseline of difficulty by tuning its abilities based on what medals you want it to achieve.
I don't know where I'm going with this, I just found it funny that even in the virtual world, some people still have arguments against having AI drive cars despite demonstrably less stakes at play.
This was a fascinating video and gave me a lot of food for thought. Thank you for all your hard work.
Kindheitstrauma would be incredible to see how the AI handles extremely high speeds, wall rides, and loops.
Additionally a kacky map would be absolutely insane, but honestly, from a layman's pov, Idk if it'd even be possible 😅
Anything is possible, you would just need to define the reward function differently to take into account the unique approach kacky requires.
I love how it went from „Struggles to get over a bump“ to „Hard for a veteran player to achieve“ real quick
really impressive work! it is often i come across ai playing video games and then being disappointed because the performance wasn’t great. This is a different case. Absolutely amazing job at getting the ai to work so well. I think it might lie in the rewards system being so well made(i am not an expert so please tell me if there’s other reasons for the ai’s performance) Props to all the people who worked on this, amazing job.
Very interesting to see the improvements. Well done and very well explained. Congratulations on this achievement!
Very nice video 👏👏
Your next challenge is to create a ‘general intelligent’ driver that could finish an arbitrary map. Maybe by changing the map every few training steps the model could build more general understanding of trackmania driving.
Can you train an AI on the random map challenge? That way you'd get a more general AI. It might not perform as well on individual tracks, but overall it might do pretty good.
agree on that, or just take way more maps. Else you end up overfitting super hard
the wirtual ai
I don't think that's how it works, the AI learns using a trial and error method, meaning the AI can only learn a map at a time. If you put it into a different map, it will be just as bad as it was in the start, and will have to learn everything again.
At least that's what I understood from the video
@@vitorrodriguez3799 It could work. It would probably require an extra layer and more neurons per layer, so it would take longer to train. Also there's a chance that it won't learn the maps well enough because of the complexity, but you don't know until you try.
@@vitorrodriguez3799because it's trained on a single map. But if it's made to be trained on random maps, it will learn probably slower but it will eventually be good on all sort of map (as long as it doesn't requires tricks the AI doesn't know)
Incredible Work! How good does this agent perform on other maps? Did it learn to drive good in trackmania overall or did it overfit on Hockolicious? How important was the graphic input compared to the engine input?
Wow! this is incredible!❤❤❤
I would love to see you teach the AI glitches, such as bugslides, or nosebugs. Maybe it could even learn to noseboost!
this channel will explode if you keep doing ai racing games
thats amazing work i cant believe it achieved those times. next up map suggestions ofc kacky
a few things that could improve the ai:
- increasing fps like you said in the video (i think 30 fps is good)
- let it have analog steering input (if that’s too complicated limit it to 0, 20, 40, 60, 80, 100% values)
- maybe give color screenshots?
very impressive indeed. I love Hockolicious and every insight is welcome
Well done Linesight, 400 subs overnight!
I would LOVE to see this AI become quick enough at learning new maps to compete in Cup of the Day. See if it can get division 1.
This is insane. Amazing work! I'm interested to see it learn to do other tricks on its own like wall bangs, air brakes, speed drifts, ramm hit, etc, or maybe even start finding cuts that no one sees.
6:01 I like this diagram. It shows the meaning of training. Not only virtually, but also reality.
this is so cool, definite new sub here :O
it would be nice to make an ai that is made to be good at generalization, eg, capable of doing new maps it hasn't yet seen fairly well/learn new maps quickly as it starts with good baseline "instincts", rather than it just potentially overfitting. But i understand that that would be much harder to do so its totally fair that it is the way it is. Incredibly impressive work! keep it up.
This is a solid success. Well done mate!
I wonder if it would become more capable if you rewarded it for its position throughout the run, instead of its own estimation of the final time (if I understand how it works anyway).
Very interesting project, this deserves way more views
THE tastyy ?!!!!! Your time on this map is really impressive, I hope my computer will beat it some day because I personnally struggle to just get the gold medal!
God I love ml so much. Fantastic video. Curious on the optimization of the relationship between framerate and resolution. Read some other comments noting that an increase in framerate would essentially be an increase in both, although I would love to see a simple experiment using a simple drift track with framerate and resolution control values for testing. Can’t believe tmnf community is still going so strong after 2 decades, makes me so nostalgic
Incredible, it really is the perfect run, every drift endes exactly where it should, and I think the car accelerater a bit faster when “sliding forward”
This is very super impressive. Since you asked whats next: no idea how feasibly this is but would be cool to get gold medal on every track in the campaign/career in TMNF, driving on different surface types with the same model might be interesting
So, the objective would be to have one neural network that is trained on all maps at once, and able to reach gold medal on all these maps?
It's definitely something we can try, I'm curious to see how it'll go.
@@linesight-rl man you gotta have a held-out dataset to show real performance
This is really impressive. Great work !
I think you should experiment with a depth map input layer. Maybe by pluging in the DirectX's API. It might help the AI's better perceive it's position on the racing line and act accordingly.
In 16 and have been interested in ai for a few years now. My plan was to go to college with a focus in studying it, and I really wanted to eventually make a killer trackmania ai. I should've known with the current "ai revolution" (lol) happening that someone would definitely do it way sooner, but I'm not giving up. I'll make my mark on this game.
We'll be waiting for your challenge!
Finding a simple cut would be an amazing AI achievement, because it has to divert from usual driving behavior using some complex information.
I know someone has probably suggested this, especially if youve been chatting with peeps in the TM community but, you can slow the game speed down rather than trying to increase your FPS. Theres been cheaters who've done that to set amazing times with human reflexes
Amazing work 👏 🙌
Wow, very impressive progress!
You might find an improvement in time if you add a reward for least amount of time per checkpoint or overall time. I suspect that your max distance in 7 seconds is making the algorithm optimize for a slightly less than ideal line as the shortest route to the finish is likely to be the fastest. So if the current AI would be given an additional target of quickest time that weighs more than the distance traveled it might find more optimizations.
I think that more FPS is going to help the AI mainly in busy sections where there is a lot of changes on the screen (repeated highspeed curves and so on) but I doubt it is going to do much in the way of increasing the overall time.
One more reward that you might want to think of even without the time based reward is a reward for less direction changes, this should result in a smoother ride and less direction changes which in turn will result in a higher speed as cars are always fastest in a straight line and will slow down when changing direction.
Besides that I'd say keep up the great work I love seeing how quickly the AI progresses from knowing nothing to reaching world record pace in just 70 hours is pretty impressive.
We are doomed. That's legit scary af and also impressive af. Damnnnn. WTF??? an AI drifts cleanly on hockolicious!??!??!.
As a guess, I think it could improve the result further if the ai can also steer values between 0 and full, or increase the inputs/time it can have
This is incredibly impressive!
this is super impressive!
I think this is great, somewhere between a TAS and a human run. Feels like it could be used another benchmark.
It basically is a TAS because he's performing on the same data that's being used for training. There's no generalization
I would love to see a write up with more detail about the training. I know it wouldn't make a good TH-cam video but I'd be absolutely fascinated. Obviously the next step is generalization. You could use Cup of the Day Maps as a potential training pool, though perhaps it would be best to ask the mappers.
This is incredible. I hope to see it beat all human players soon!
Wow congratulations!
Putting the AI on an RPG track would be really interesting. Seeing it try to find the right track, and when it finds it balancing on beams or driving on blue walls would be interesting. Some sort of easy Mini RPG would be interesting to see.
I've seen many people suggest maps that are not available on TMNF... Do you have a specific map in mind? Let's start with an easy one if we're already changing map style though :)
@@linesight-rl I don’t really have a map ready, but it should be pretty easy to build. A couple blue walls that you’re supposed to drive on, maybe downhill ones to add a bit of challenge and then those double beams that you can barely drive on interspersed with checkpoints on platforms should be enough challenge. Maybe like the end of oach, so that it’s supposed to go fullspeed and jump into the finish.
At first I was sceptical of this approach, I thought having the AI deciding what to do next in each moment would make it gittery but it was very smooth in the end.
I'm thinking tho, as humans we obviously don't decide what to do next for every frame, perhaps the AI would be more smooth out of the box if it had to commit to turns (like left 70% or full right and break) and then have a separate agent that considers the risks and if the risks breach a certain threshold, the AI aborts its turn and does something new.
Another idea: Ik this is way easier said than done but would it be possible to instead of button presses, you give it certain actions, like turn left, turn right, bugslide, and so on, but obviously way more options, all the actions could be induvidually trained beforehand with the system you're using now using many different maps so that it's a pro at all the separate actions.
I have to wonder if a "two part" AI might be possible: A general "how to drive in Trackmania," portion and a "what do I need for this track?" portion. Training might involve cycling randomly through a couple dozen tracks, with the track specific portion being swapped in and out.
is it processing power that make you chose 20 fps ? if the AI can have such a great time whit 20 fps i can't imagine whit 30 or even 40 fps
Keep in mind that 20fps is a frame every 50ms so the AI has a 50ms reaction time. Thats more than 4 times faster than most people. It doesnt really need more fps to play well. It isnt analyzing the input frames as a video like humans do, it just sees 20 pictures every second and decides which buttons to press for the next 50ms.
It would be possible to slow the game down to give the AI more fps if processing power was an issue.
Really good video !!
5:00 this AI is learning faster than I could 😮
I'm a pretty big noob in terms of AI, but I have a few random thoughts. Having played TM a lot I'm concerned about 20 ticks. That could be a source of problems and the AI would probably benefit from being able to make commands asynchronously from the frame data. I don't have any experience with RNNs yet, but I'm assuming there's some kind of learning rate scheduling? It's performance is very erratic from run to run and I feel like it should be trying to refine more after about 40 hours. Also, is there a reason why there's no reward for new or matching PBs? I wouldn't apply it to sectors because sometimes the fastest way on a map is not by optimizing each sector but by setting up to snowball speed. In fact, I would try extending the rewards window of time, or changing the reward scheme because it might hold it back from finding the best solution.
Very cool project though, I've been thinking about doing this exact idea for ages since the first TM AI videos came out! I was convinced an artificial vision approach was going to do great.
great vid and yeah 20 fps is definitely holding it back alot id imagine just a higher fps would allow you to beat the current world record
Can you PLEASE upload a tutorial video, maybe even on a different tutorial channel? Your voice is extremely understandable and your explanations are amazing and make sense. Instead of watching 10 hours of people repeating themselves over and over again I would like to actually learn. Pleaseeeeee consider. thank u
Some great work! Your code for a TH-cam video is cleaner than code submitted with conference papers… Interesting you’re using IQN 👀. What do you do for the reward shaping?
I expect this video to do great, but just as a tip from one TH-camr to another, it would probably even do better with a different thumbnail.
😍 Love your videos, I've been subscribed for some time to your channel on my personal account !
Thanks for the compliments on the public code, our latest (private) version is both better and cleaner.
I thought I had done a good job for the thumbnail, can I get in touch with you to discuss what could be better ?
I'll use the email in the "about" section of your channel.
Happy to hear 😊
And sorry, I didn’t mean to be too blunt about the thumbnail comment 😅 it’s a decent thumbnail, I just think the video is top tier, and if you can get enough people to click on the video I’m guessing it will blow up
And yeah that email works 👍
I would love to see an ai trained with a more general learning method, so that it retains basic skills and only has to adapt to environments.
this is so crazy, ggs
AI vs Kacky would be hilarious to see...
Current players already resemble AIs playing it. Bash car against wall and hope something happens.
I think it would be awesome to see it try to learn a track containing every different surface!
If we're going in this direction, the AI has also improved its time :o)
I think our last run on map5 went down to 2:02:08, but we didn't update the video.
this is amazing well done
I think the key to a better and more consistant program would be for it to spend time learning the optimal acceleration, velocity, and position for each possible section of road in spans of 20, 30, 40, or more car lengths and also give it a full map to analyze so that it can plot a path between each section in the optimal way.
I have seen a few videos were people allow programs to train on entire maps but this seems a slow way to train them because the program isn't focused on learning situations, it's focused on it's immediate situation. This is not how people get good at racing games, they learn how to drive sections of road and adjust their driving actions based on the section they are in and the next section coming up. This is made optimal for people when they already know the entire map, allowing them to know what to do on the entire map making accuracy of their inputs the new challange to become faster. The latter would not be a problem for the program and would mean that a programs ability to become faster on any map would be directly linked to its driving ability on individual sections of map not the entire map.
Just a guess... I don't write programs so I can't be sure of this opinion.
Your idea is interesting, and it is actually what we are doing at the moment :)
You may have noticed in the video when I say : "the AI predicts the reward that will be obtained in the next 7 seconds". This is how we define the sections, in blocks of 7 seconds. Each transition is interpreted as being part of a section chosen at random during training.
After some more consideration, another difference between pro and casual players is the implementation of certian actions such as power slides and other tactics in specific situations.
Have you considered gathering a list of pro tips to save into the program for use in these situations? Or if you want pure self learning you could impliment a recognition of specific tactics in the reward system and the ability for the program to save them to memory and use them in the future when it comes across them on it's own . Maybe, even allowing the program to save things it selects as possible tactics to test... maybe it will come up with a new tactic.
My suggestions are a reflection of how I see people learning and getting better over time. Most pro players take tips from the community, I don't see why allowing the program to do the same would be wrong. Unless you want a purely self taught program, but then I would definately suggest allowing it to save useful tactics for itself to test, perfect, and use.... such as power slides, the new gorilla grip, bug slides, ect.
Keep up the good work!
I am not very good at Track Mania (in fact I have never played it), but I would love to see someone beat the time of that AI under its handicaps (grayscale, 160x120 pixels, and 20 FPS) It may not have beaten the world record, but there is no way the record holder could beat it under those conditions.
One thing I'm noticing, as a human: there is a lot of useful information contained in the color. Perhaps feeding the AI a color image might help it do even better, especially given the low resolution. (The luminance still seems to be the most important, but the Cb and Cr can always be lower resolution.)
That's a good point, we do not need to store the RGB information with the same resolution for each channel. An interesting addition to the TODO list !
FS must be pretty fitting for reinforcement learning since the reward can always be distance travelled/ higher velocity without having to adjust for long term reward.
If the reward is speed, the car may not go where we need it to go :D
how explorative did you make the AI? if you have more compute time on your hands it would be interesting to see how the AI would do with higher exploration (and FPS, as you mentioned). loved the video+presentation btw :)
You could try feeding it not just the current frame, but the last one, too. Maybe the AI can figure out how to derive motion from that and have it help the decision making.
Yeah I was surprised this wasn't done in the original project. Knowing not only position but velocity and acceleration are very important when modelling driving.
What would be interesting is to see if the AI can beat Cup of the Day without getting banned....
Really impressive work.
What I'd like to see (or try) is to have a selection of a N (e.g. 20) different, popular maps, and run learning in parallel and in batches, with random sequence of maps for different learning periods, and then test those against different maps and see if they can e.g. beat human record on 1st try :D
One thing that i suggest, for the "more than 20fps" question you posed is to use a TAS'ing software to judge, setting it to pause the game at every, say, 60th of a second or 120th
riolu AI
i think its less about real-time processing time and more about total processing time, adding a one to any limitations will raise the complexity exponentially
Nice! Does the AI have to be trained from scratch on every new map?
Not really, but it will add another layer of complexity and might break more than it fixes especially on trial maps.
This is so cool !
i think having the AI run through the entire campaign might be fun, since it only needs gold to do that, i think this could be achieve in a reasonable itme, and i wonder if the AI could take lessons learned from previous maps with it. (i wonder if that speeds it up at all)
It absolutely can. The question is how well.
Realistically, the more maps the AI trains on the better it should be at any new random map. And the less training time it would need to adapt to a new map.
AITango has some videos on Mario Kart AIs where they trained it on 1 map over the course of a day or so, then trained another AI on I think 4 maps, which took significantly longer, but then when they placed that AI on a new track it only took an hour or two to learn the track just as well as it had the other 4.
These types of AIs learn more similarly to humans than people realise (at least from a results-driven viewpoint), they just don't have the initial corpus of knowledge from a lived life that humans come into any new task with. LLMs do - you can explain to them the rules of a game they've never seen before, and they can play it, albeit with poor skills. But they're not just randomly guessing moves unlike AIs normally do at the start of training - they draw from their existing knowledge to form loose strategies in the game just as a new human player would.
This is great. I'd love to see it on a fullspeed map :)
AI coming for our trackmania streamer jobs!!!
the AI is so good even though it's colour blind, has terrible eye sight and is lagging to boot
Hello there, hope this message finds you well.
I'm incredibly impressed with what you managed to achieve here, the AI capable of setting WR performances while relying on the visual clues rather than some exact data extracted from the game is something truly unbelievable and it left me speechless for several days.
The impression is so strong, that I together with few other friends of mine decided to attempt to develop a similar (although way less capable) AI in our favourite environment, which is one of the hardcore racing simulations. But since we are beginners in the matter of developing complex neural networks, I'd like to ask you, if you would not mind to guide us in the development of our own racing AI. We believe that making this work reasonably well in the realms of complex simracing games would push the limits of what is possible even further and make the overall impression broader. But to get to the point of even partial capability we would need your precious expertise.
Would you be please open to help us out?
Sincerely.
Michal
Which racing environment are you trying to learn in?
@@htgnef the environment we are trying to get it working in is called Live for Speed, it is a hardcore racing simulator, so it surely will be a much bigger challenge to make it work properly (let alone at decent speed), but we are determined to give it a try.
Hi. We're too busy to help you with coding this, but if you have discord and join the Trackmania interface discord (discord.com/channels/847108820479770686/1150816026028675133), we can talk about how it could be done and maybe answer whatever questions you might have.
@@linesight-rl Thank you very much for your response :)
I'll have a look there for sure.
I would love to see if your AI could match the winner of the midori map.
Especially if it can learn to use the unique turning needed on it to get that extra speed.
i have a small question, what youre working on now? are you training the ai on a map or smt? i wanted to know, because i really like how the ai are getting soo good to the point that they can already do simple maps in trackmania, i would love to see more complicated maps tho, anyway, gl on future projects
Thanks for checking up on us :)
We're still hard at work on this AI, we have enough results to make the next video.
It's a lot of work, but I think you'll like the next video. Just need to be a bit more patient :)
Good Job :)
Further improvements could be using a pretraind vision transformer (e.g. fastvit) or cnn (e.g. mobileone) from the pytorch-image-models
git and maybe (pre) train the model with rl from observation (since you can extract a lot of wr-runs and their inputs from the exchange)
Why did you make it take decisions on 20fps and not just as soon as it can? Like what if for instance because of compute issues you skip a frame?
They way you train AI reminds me a lot of my elementary days
What GPU/hardware setup did you have? I didn't see this listed anywhere. I'm amazed by the 70 hour training time. I'm wondering if this result is mostly a testament to hardware improvements, OSS improvements, or your own ML implementation.
For a long time : RTX 3060, Ryzen 5700G, 32GB ram.
I have very recently upgraded to a 4070 :)
@@linesight-rl God help us all, he doesn't even have a 4090. Imagine what you could do with an a100. We're doomed 🤣
Very cool, would like to see your improvements to the AI
A good watch.
What I would find interesting would be, if using a model pre-trained on a different map, would lead to a faster learning speed. So one could estimate, whether the AI actually learns to drive, or just learns the layout of that one map.
I'd suggest adding previous frame as input for Ai as well
or at the very least, its commands on the previous frame
that way it won't need to guess its previous self to be "not indecisive"