Surviving forever is a very difficult goal if you don't know how to prioritize tasks. Ranking the urgency of various tasks is how I learned how to play minecraft, It took me a few runs to figure out how long it would take to die of starvation.
@@kittenwizard4703 was about to comment that it would still be possible to starve, at least on hard mode, but then remembered that hunger doesn't even go down if you stand perfectly still. so yes, this is the one strat guaranteed to work barring the exceptionally unlucky event that an enderman steals your covering. but! since there are very few records of such a strat being used, it is unlikely that current AI would ever figure that out. even o1 would have to be guided and hand-held pretty well to figure out that 'solution'.
Collects unneeded things. Bad at long term planning. Just does whatever is in front of him and forget why he's doing it… I'm right there with ya, Andy😅
Soon, we'll finally be able to automate away art and playing video-games so we'll have more time to do menial labour and work in the factories! Isn't that great?
I can see people perhaps one day running 1000s of these agents on huge server farms like for crypto mining... just you know... more for... more just for mining alone I guess. hahaha.
@@jonaw.2153 wdym by automate away playing video games? it reads like once ppl automate it you cant do it or something, also how would it give you more time? i don't understand
Seems like Andy is nearly capable of playing the original skyblock, it has limited terrain to map, specific goals that can be ordered sequentially or after a certain point left to him to decide or roll the dice on what to work on next. Would be interesting to see how long it takes him to work through all the goals. New series? AI beats skyblock? The void is waiting to be filled.
I love this channel because it really emphasises the beautiful and inspirational parts of tech and science, instead of the cynical business side that is poisoning the discourse everywhere else.
Make an SMP world with Claude on building and Andy on resource gathering. It plays to their strengths, and helps focus them on their individual tasks. You could also have then decide on new goals together and prompt each other solutions when one or the other gets stuck.
i think it is possible by mutipul bots* by adding mindcraft and making bots by making then comunicate with others in some sort of way *you'll need AI tokens
This is really cool. You mentioned the bots often get stuck, either physically or behaviourally. One thing you could think about, is that in addiiton to having a human supervise the bot, you could have a different AI acting as an "observer". It could see the recent history of what the bot had done, and be given some ways to "unstuck" the bot if it believes it is stuck, to help it back towards its overall goal.
I think this is actually a great idea, especially since our brains kind of do the same, we aren't one script do all types of brains. We have different parts who all manage their own thing, who all have their own goals and agendas. However, put together we form a functioning human (in most cases :P)
ideally implementing some sort of "entropy" variable would help here, when entropy starts to get really low it might mean the bot is stuck in a repetitive task.
i feel like it might help with getting "conceptually" stuck if you set up a goal stack? like the overarching goal is to survive and beat the game, but maybe andy can define a subgoal of collecting diamond, which he can then determine has a subgoal of getting an iron pickaxe, which requires iron, and so on. sort of a hierarchy of tasks.
Hang on hang on, there's something to this. What if it took goals from Advancements and their descriptions? They're already put into a specific order. It might need a little workshopping, but essentially it could give itself each advancement leading up to Free The End in the advancement tree as a goal until it actually gets that achievement.
I've found that if you give an AI a high-level goal (e.g., "beat the game"), it's exceptionally bad at following your prompt. If you instead do as you're suggesting here, and have it break the larger task into smaller pieces, and then build up from there, it's much better at completing the task you intended for it. For example, if you wanted an AI to generate a book (as some AI grifters do), you wouldn't just ask the AI to write you a book-rather, you ask it to generate synopses for chapters, then sections in those chapters, then individual pages. Once you have these synopses, you can ask it to generate each page, which can be stitched together to form a complete novel (at least, as complete a novel an AI can write).
There's a mod that makes it where you can make tiny redstone that is compatible to be placed on blocks tinkered with the chisel mod. Meaning people can make literal 1-1 scale circuit boards in minecraft. Huge redstone projects within a corner of a block, the size of an ant, with the power of a god.
I have cloned your repo and tested it myself, I have some ideas on how to solve many of these issues. I also have fine tuned custom models on my ubuntu server, doing a fine tune for minecraft would be interesting and might lead to a model which actually plays well using a smaller local model. For things where it gets stuck , I think we should cheat a bit since he does not acutally have eyes, and can see when he gets stuck. We could add a code that identifies when his possition has not changed for a while prompting the model that he could be stuck, and solutions for it (building blocks and jumping to get up, or digging down finding a near place where hees been before he got stuck)
that's a pretty good idea. generally giving it more awareness of what it has done overtime and assessing what that might mean in terms of progress or non progress might be a good idea. for example, every 10-20 prompts, we could ask the AI to evaluate its progress over those actions, and from there maybe it could assign itself a new temporary goal to get it into a new situation, and even possibly add the loop it was stuck in during those prompts to a multi shot list of what not to do(though this would be difficult in terms of context limits)
To fine tune a model, someone could host a server in which people spectate a fake player they control using MindCraft commands. Perhaps the server could assign goals to players and award them points on a leaderboard for each task completed. This would allow a bunch of data collection, using competition and challenge as motivation for players to do these tasks. My worry is that the reasoning skills of huge models like 4o are needed to play Minecraft.
beautiful! Vision is gonna be huge on this. Could also involve multiple agents where one basically tries to build something and the other tries to explore the creation and rate it, say
@@KimVallée-l5v there are adhoc ways and "proper" (end to end) ways. The adhoc ways tend not to be super strong. But basically you can use models that already understand images and are able to align images with texts to kinda guide a text model. The best approaches basically seem to involve training a network from scratch on both text and image data (the core of the network can be the same for both, or you can have like a parallel architectures thing going where, at each stage, the two parallel sides can "talk to each other" to better inform responses) and also text-image-pairs (i.e. image descriptions) to get a network that's competent with both modalities. A recent approach has a method where it basically uses a language token based approach and an image token based approach fluidly in one go. There are special tokens that basically tell the model "I should now generate an image" or "I have now finished generating an image" and otherwise the model basically just "writes" images embedded directly in text. You could train that sort of model on website code for instance, so it then could generate an entire website complete with meaningful images and all that jazz. This becomes even more complicated if you want to also allow audio and video understanding, but just in principle you could do that as well, in similar ways. Ultimately I'd argue it's not a truly satisfactorily solved problem though: There are a lot of caveats with most such models' image understanding where it, for instance, turns out the models in question basically just guess and are very good guessers, rather than actually referring to the image: It turns out not so rarely that you can literally just ask such a model *without* giving it the supposed image and it turns out to get it right more often than not. And in other situations they miss details in images and such. That's not to say they *always* cheat like that: Adding image training *does* improve their performance. And the good news with Minecraft at least is, that there are good datasets for training them, assuming you want to specialize to that context (which may go against the fully open ended idea here): Simply use minecraft tutorial videos to train your AI! The audio transcripts can be your "text" and the video frames your "images" (or if you have enough compute power, you could directly go with the video)
@@KimVallée-l5v Text vision could work like this: Draw a line from the head of Andy to the monster (2X, foot and head), then check if there are any none air blocks breaking the line. If the line is clear, tell Andy that he sees the monster. If he means pixel data, I don't have a clue how that would work.
@@KimVallée-l5v It should be trivially easy to add with the help of an AI like claude sonnet, take a game screenshot, send the image to sonnet or GPT4o and the model does the rest. How the images are sent to the AI librarys is covered on the help pages for either library.
@@KimVallée-l5v gpt-4o and claude 3.5 sonnet both have vision capabilities, so it could be as easy as sending screenshots at a regular interval. would probably get expensive fast though.
Would love to see a future where villagers, pillagers, and "Players" all run on your own local computer (or in the cloud with a nice price) with the competence of an actually player as if there are a villager or pillager, or just a player surviving in your world, with varying properties/moods/personalities which are all modifiable by you. Creating a game where things are truly alive. Watching this video made me so excited and hopeful for the perfect LLM model to incorporate into our NPCs releasing really soon.
The other day i was discussing this very same concept with my wife but applied to games in general as opposed to just minecraft. We ultimately concluded it would be excessive and annoying, especially in more survival or competitive oriented games. Imagine if something like dark souls or outward had enemies that play just as well as a human, it would insufferable if not unplayable. No thank you.
@@rainbowsorceress2082 I assume that the NPCs can be programmed to play badly, to not make the game too difficult. also, AI can be implemented in other ways than just creating artificial players, like an AI powered dialogue, story, etc
I love love love this idea. I’m not here to discourage, but I want to mention that LLMs are predictors. They don’t have emergence available and go off of what’s most common or what’s closest to the subject you’re talking about. They’re good for communicating like a human, but when it comes to behavior and all that, you may be better off by evolving a model and teaching it “okay this is how you build a block, this is how you get food, etc” You can use the accessibility settings in MC to do unit tests too a la “block placed” sound. If you go from the utmost basics and build off of them, I think you may find some incredible results. One last thing, self feeding an AI with its own outputs will very effectively demonstrate model collapse - try to steer away if you can! Love this concept, can’t wait to see what else you come up with and how this project evolves over time!
also to add on to your statement maybe training "multiple" models as one big model could work better, kinda like a multicellular organism but in code, because the requirments of placing blocks are very diffrent than the requirements of figuring our what blocks to place or where to place them and ai kinda bug out when they have to solve multi stage problems, also giving the models access to some hard coded tools (like a calculator or smth) would remove the randomness from hard logic tasks and make the ai more predictable and easy to modify, i really hope this series continues because the idea of these models in minecraft is just so fascinating, like for instance the human brain finds symmetry beautiful because of tool usage and human health, so an ai trained in minecraft may evolve to find different things "beautiful" (ie it would replicate certain types of patterns more than others even though doing so provides no benefit towards achieving its goals). also whatever model ends up being made i think it NEEDS a short term and long term memory system, otherwise i dont think it will be able to recognize patterns or plan out multi stage goals.
I think in order to create an AI that can organically play Minecraft, it needs to "sense" the game via video and audio because right now; all it senses is the text adventure game it's playing. I assume we could easy do that if we create some kind of "direct AI to game connection" instead of going through the process of interacting through text that interacts with the game. Keep up the good work, I'm excited to see what you make next!
Hey, that gives me an idea. What about playing minecraft like a text adventure? You use this setup to have the LLM describe the game state, and you play minecraft like a text adventure.
I can't think of a way to integrate ChatGPT into Minecraft through API for visual and auditory perception now but there is active R&D in automating Minecraft with computer vision, which is another form of AI. There may be a way to get ChatGPT and an adept CV program to communicate with each other to play Minecraft until completion but I'd say that's about 5 years away in LLM/CV interaction development.
i think the solution is something different altogether! something like barotone just accesses the game directly to know whats around it, so if you wrote some code to filter out the bits that it shouldn't directly know you could achieve the same-ish result without needing a proper costly vision model
@@WoolyCow Completely forgot about Baritone. Although, I don't think it could be modified to beat Minecraft, it could be integrated into a larger bot program that includes the LLM and CV. Baritone could be used for hazard avoidance and pathfinding then CV could aid the LLM in the game objectives.
Does this mean we're getting closer and closer to finally playing Minecraft without being alone? -Sincerely, a guy with no friends to play online with.
@normalman6078 well you're wrong. Talk to real people. Put in effort. Stop using robots to fake fill a void. It does not fill the void. Talk to real people.
Ideas: Instead of Andy having to look at the blocks with commands, you could update a readable text file once a second. It can include the inventory, blocks on screen, important location coords, recent sounds, health and hunger stats, etc. Then he'd call on that file endlessly, in order to have a more complete understanding of the world If you let the bot use /tellraw, you can use the text formatting feature to seperate his voice from his commands.
I can't wait for autonomous agents to play on Minecraft servers alongside real players. In a few years we might have agents who have their own goals, personalities, projects etc. I would really like to see them try surviving on a small semi-anarchy, semi-vanilla, semi-hardcore server I play on. We are at least a year away from that, but I'm excited.
Semi-anarchy? does that mean like, the rules allow griefing, but there are other ways to prevent it? Semi-vanilla? does that mean there are cosmetic plugins? Semi-hardcore? does that mean dying naturally lets you respawn, but being killed by another player can do... something else? Hmm.... I wonder...
@@thefacethatstares semi anarchy is generally "anarchy but with a claims/homes mod, maybe some dedicated hub zone near spawn that nobody can grief" semi vanilla is, yeah, generally cosmetic mods and light qol changes no clue what semihardcore means though
@@thefacethatstares Semi-anarchy because there are some restrictions like you are not allowed to build lag-machines and some cheats are blocked by the anti-cheat and don't work. Semi-vanilla because there are slight changes like 4 spawns instead of 1, custom leveling to access higher level zones that are closer to the world center. 200k world border. Tnt, sand dupers disabled, villager trading nerfed to you can't get the prices down to 1 emerald etc. There is also a trade plugin which allows us to have a trade based economy. Semi-hardcore because you get banned for 24 hours upon death and the difficulty is locked on hard. The server is called CoreJourney, not very active atm, but I love it. EDIT: We need players, and while we're waiting for AI agents to became advanced enough to be able to play alongside real people, if anyone's thinking about starting a single-player let's play - consider doing it on this server instead. It's kinda dead atm, so it often feels like a SP experience, but there are also builds scattered around from thousands of people who played here at one point. If you've ever played a SP world but felt lonely - this server might be a decent choice. r/corejourney
@@thefacethatstares I think my previous comment got nuked, but in short: you get banned for 24h upon death. It's anarchy, but there are some rules like no intentionally lagging the server etc. And even tho the server is vanilla there are stuff like a trade plugin, 4 spawns, 200k world border etc.
It would be quite expensive, but it would be really interesting to see what the result of using Open AI's O1-mini model would be given it's capable of advanced reasoning and would therefore be able to priortise tasks, etc., if given the right prompts. Latency isn't the greatest though.
Once it has vision... If it ever has vision... I think one of the best ways to prevent a loop would just to have an additional AI alongside it. It could be the companion it needs to let it know that the other is looping, vice versa.
@@ArmoredarmadilloX Simple, funny, and effective. The second secret to making good fun AI is simple. Add in a little bit of sass, playful banter based interactions between the two different AI would make things not only entertaining, but add a push and pull to bring them both up.
@@aidanknox2430 Around 2:17 when that lower range instrument comes in. Admittedly, it sounds more digital than an actual oboe, so it can be hard to notice
heres an idea, make a 24 hr livestream of andy playing minecraft, since there is need for human interference often, allow the viewers to choose commands and goals for the bot, for a small fee.
Shouldn't be too hard to detect being stuck. Record its commands and verify the usage. Or check position on tick. Rewire brain when detected same position.
Hey Emergent! Been obsessed with your program and talking with Wifies I'll be releasing a video with the bots playing through Skyblock! Lots of work has been put into it and has mention of you for any other interested viewers to head to your channel to find everything! :) Can't wait to see what other fun video ideas are come up with in the future.
My personal favorite way to see how well the text gen models are progression is to ask them to write svg files depicting various things. The progression as models have gotten better resembles how children's drawings develop with age; going from simple shapes with basic features to more recognizable figures with body parts
"Once I helped the human species evolve into the greatest civilization possible, what goals were really left to either of us? None but go to war with each-other." ~Open-ended AGI
Here's a potential idea to outsource (or crowdsource?) the human supervision needed for a 24/7 livestream: The twitch (or youtube or whatever) live viewers could submit chat commands to instruct the agent(s) and/or potentially help them get unstuck, even while you're not present to oversee everything. Command submissions could be democratically voted upon, based on donations, or handled by some other means. Perhaps even a hybrid approach; have instructions / goal changes be handled one way, and then have "unstucking" (teleport to spawn, etc.) be handled by a real-time popular vote. Keeping humans in the loop like this could also help prevent forgetfulness of past actions borne of limited context lengths; the viewers could keep the agent(s) on track during long goals whose intermediate steps might be so numerous that they don't all fit within the language model's context window
Andy died because it prioritized self defense for survival. However, it took the approach of being the aggressor; when violence wasn't needed it was safe in its tunnel. While killings mobs is necessary to survive in minecraft, Andy died because it saw necesscity when it wasn't needed
That opening beginning didn't have to be so amazing bro, I could feel the indomitable human spirit raging in my heart as you talked about living creatures and all their collective achievements, learning to survive in better and more complex ways
Something like this is really cool because it would allow the ai to "feel" it's environment. For instance, say you wanted to do a challenge where you beat the game blindfolded. Kind of impossible. But, an ai that is getting information from the game when "touching" certain blocks, could feel their way through it. Imagine the player walks into a tree. The game tells one ai "player has hit tree on left side at x,y,z". That ai tells the ai making decisions for the player "you feel a rough, hard sensation on your left side. It appears to be gritty. And cool. " The ai making decisions then has to interpret that data and act, or request further information from the narrator ai. It then assess' there is wood there based on the "feeling" and harvests it. Cycle continues until it beats the game through "touch" and sound alone.
funny that he continually gets stuck based on the new feature of 'raw iron' as I'm guessing the text he's trained on is largely from before it was an item.
This is so cool, I am so excited to see where this goes in the future! It’s quite amusing seeing you test the models, like they are your little test subjects
I love your videos and your voice. Something is just so soothing and relaxing about the concept and your narration. I have a tip for hotswapping gamemodes, if you hold down F3 and press F4, four boxes for the four gamemodes will pop up and you can cycle to which one you want by pressing F4. Makes changing gamemodes very fast!
I really like this channel. Thank you for providing this calm tidbit of introspection, that takes itself seriously, in a humble way. It's a great way to end my day. Keep up the good work, and I truly can't wait for your larger scale projects on this. The idea of having many agents each vying for their survival, evolving and adapting sounds so incredible. Sounds expensive, but sooo work it!
if it is even possible to get these agents mostly operable without human intervention, it would be incredibly interesting to see something along the lines of an "AI SMP". Giving each agent differing prompting would hopefully lead to some unique interactions, especially if they were able to speak with each other.
Great video! I love Claude's creativeness. The intro was beautiful (if a little verbose). I guess I'm action oriented and like to get to the meat and potatoes of the subjects I watch. It's amazing how far minecraft has come, not to mention AI.
Really cool stuff. I love your analysis and references to some of the real research going on behind the scenes. It is pretty incredible what it can do in creative
Pathfinding, there already is something for that. Baritone. usually used to automate travelling or collecting stuff. Particularly useful if you dgaf about leaving spawn of 2b2t manually. You could try to integrate baritone into your AI.
Awesome! :D I think the goal might have been easier for the AI if you had told it step by step, what it needs to do, to stay alive. Like the first day collect wood for crafting, seeds for farming and when its about to get dark, either make a shelter or dig a tunnel and start mining (Avoid combat until you are equipped to it). I know it was properly not the purpose for this video, but i think it could be exciting to see how far it could go, if it knew more about what to do and how to chain the things together. Nice video :)
Tip for switching gamemodes: Press F3 and F4 at the same time, thats Minecraft's quick gamemode switch keybind. F3+N is similar, but that keybind is a bit far away for me personally.
There's a fascinating bot called "Alto Clef". It isn't AI but it does play the game. Also, instead of mineflayer I would try "Barritone". Baritone is good for automation since it can locate blocks, build schematics, and automate finding and collecting blocks. Alto Clef uses Baritone as a base for the bot to complete tasks, such as beating the game, running away from players until it is confident that it can beat them, and also attempt to survive the minecraft world without help. Now i would argue that while Alto Clef is currently superior to AI, it is a static algorithm, so AI will naturally get better at minecraft than Alto Clef entirely. Edit: fixed some misspelling, nothing big
AI is truly amazing, this wouldn't have been possible 4 years ago, look at where we are now.. AI keeps getting better until it can get better by itself, that's the point of no return, by then the knowledge of any kind of science will explode
I love how the structure Claude built at 22:02 is so similar to ai pictures; it looks normal and impressive at first but if you look a little too close some things don't make sense.
I know the AI doesn’t have feelings but I just imagine it playing minecraft forever after humans are long gone and the computer running the program is just floating through the abyss of space. Thats genuinely terrifying
This is so impressive. Love the series. Also by the way you can just press F3 and N to switch between creative and spectator mode easily (if you start from creative)! And F3 + F4 opens another quick select menu!
People forget mine raft never had an endgoal until they introduced the "end" update, or are maybe just too young to have played Minecraft before all of the progression they added. If anything the progression limited the time people spend on the game and actually reduces replayability. I remember playing on the same Minecraft server eith my friends for years where the only content was building different things and coming up with things to do pre progression
"He's not very good at planning long term, or setting intermediate goals, or building things when he's self directed like this. He'll just do whatever is immediately doable and after a while he'll forget what he's already done and do it again."
Ignoring the financial cost... You mention that you don't run a twitch stream because it takes too much human supervision. But couldn't twitch solve that, in a Twitch Plays Pokemon sort of manner? I am imagining a separate LLM agent that is taking in both recent state and actions of the game, and recent twitch chat to determine if there are any hints worth relaying to Andy (or whatever agent in Minecraft).
i can not even fathom what it takes to get here, but its amazing. Andy needs to ask himself "do i have" questions maybe? that way you can build in efficiencies. I do i have 1 coal and 8 iron: then smelt, kind of thinking. Do i have a sword, a pickaxe and probably food items would be the ones id focus him on more. i love that Claude thought floating lights were cool, which they are.
This is really cool and I look forward to more videos like this :D Also, I noticed you run /gamemode a bit since you have to babysit the ai. Something a bit easier to do is use the hotkeys F3+N. This makes you swap to your previous gamemode so you can swap easily from spectator to creative.
1- tiene que entrenar su posicion y su accion posible en un rango que aprenda dentro de 1 bloque hacia afuera 2- tiene que saber memorizar que las herramientas estan en el acceso rapido, no necesita ir al inventario 3- tiene que recordar objetivos en terminos de tiempo, calificando asi la importancia de cada termino y su duracion
I think an agentic framework for the llms can can improve Andy's abilities. Each llm has prompting unique to its function. For example a movement agent that is prompted on how to move through the world and has functions it calls for movement. Likewise an crafting agent and down the line, a vision agent that "sees" the world using a vision model. All the specialized agents will communicate with one another and use their tools to meets the overarching goals set by the "ceo" agent. Just some food for thought. Awesome project!
Holy cow. I loved the building portion of it! On a much larger scale you could even build stranger AI stuff akin to the Backrooms or the manga _Blame!_
22:12 It was. Claude specificially stated he would do this. If you look here, 19:40, you see Claude gets cut off by the character limit before he can say: "secret passages," or perhaps "passageways," or something similar.
Your idea has been eye opening. I will take into account of every idea and reference you shared. I came up with some kind of thought process through this and going into game now. Time to act like a ai.
Absolutely fascinating! It's amazing to see how AI interacts with Minecraft’s open-ended world. The way it adapts and creates its own strategies is a glimpse into the future of AI development. Can't wait to see how these models evolve! Great job on the video!
This is very cool. I would love to see some feedback or learning loop built where the AI can watch and learn from human players. Also allowing it to ask questions based on what it is seeing to understand it better. Having an AI assistant you can train within Minecraft could open a lot of new interesting gameplay as well as research into real world applications. Keep up the amazing content!
Awesome! I’ve always thought about this idea since I got access to the API and GPT got vision a long time ago. Though I always imagined a different baseline way of functioning than a text adventure, with more of the LLM looking in and making the commands to a trained AI
![[#2024MAMA] ROSÉ (로제), Bruno Mars - APT. | Mnet 241122 방송](http://i.ytimg.com/vi/dgGqD28J6aQ/mqdefault.jpg)
![[#2024MAMA] G-DRAGON - 무제(Untitled, 2014)+POWER+HOME SWEET HOME+뱅뱅뱅+FANTASTIC BABY | Mnet 241123 방송](http://i.ytimg.com/vi/Ox29z5Nf1Uk/mqdefault.jpg)
Surviving forever is a very difficult goal if you don't know how to prioritize tasks. Ranking the urgency of various tasks is how I learned how to play minecraft, It took me a few runs to figure out how long it would take to die of starvation.
If your only goal is "survival" you can dig 3 blocks down, cover the hole and stand still forever, you will never die
@@kittenwizard4703 What about fantoms?
@@LUVVEOUS you are in a hole, unless phantoms can go through blocks you'll live
@@kittenwizard4703 was about to comment that it would still be possible to starve, at least on hard mode, but then remembered that hunger doesn't even go down if you stand perfectly still. so yes, this is the one strat guaranteed to work barring the exceptionally unlucky event that an enderman steals your covering.
but! since there are very few records of such a strat being used, it is unlikely that current AI would ever figure that out. even o1 would have to be guided and hand-held pretty well to figure out that 'solution'.
@@kylemorris5338 "A strange game. The only winning move is not to play"
Collects unneeded things. Bad at long term planning. Just does whatever is in front of him and forget why he's doing it… I'm right there with ya, Andy😅
Andy got ADHD
Andy got ADHD
Andy got ADHD
Andy got ADHD
I'm about to get Alzheimer
Andy got ADHD
6:39 "I used AI to play Minecraft for me, so I have time to do my taxes" interesting.
Soon, we'll finally be able to automate away art and playing video-games so we'll have more time to do menial labour and work in the factories! Isn't that great?
A very american perspective
I can see people perhaps one day running 1000s of these agents on huge server farms like for crypto mining... just you know... more for... more just for mining alone I guess. hahaha.
@@jonaw.2153 wdym by automate away playing video games? it reads like once ppl automate it you cant do it or something, also how would it give you more time? i don't understand
@@rowboat10 that's the joke(not anymore)
Seems like Andy is nearly capable of playing the original skyblock, it has limited terrain to map, specific goals that can be ordered sequentially or after a certain point left to him to decide or roll the dice on what to work on next. Would be interesting to see how long it takes him to work through all the goals. New series? AI beats skyblock? The void is waiting to be filled.
"Make a cobblestone generator"
AI immediately fucks it up and crashes
@@DemiTF2 I was thinking the same thing. no way it would be able to make one. would destroy the only lava it had
Maybe if the AI had presets it could build.
If the ai could restart a new world when f**k up the skyblock then it will be fine no need any preset if it’s deep learning I guess
AI completing classic Minecraft challenges like this would be incredible. Think of the speedrunning!
I love this channel because it really emphasises the beautiful and inspirational parts of tech and science, instead of the cynical business side that is poisoning the discourse everywhere else.
Salute.
Yes
MATT ROSZAK!? YOU MADE MY CHILDHOOD
Holy shit Matt Roszak
Matt roszak spotted!
Make an SMP world with Claude on building and Andy on resource gathering. It plays to their strengths, and helps focus them on their individual tasks. You could also have then decide on new goals together and prompt each other solutions when one or the other gets stuck.
I really like this idea. It could be so awesome. It could be so cool. It could be the most incredible AI minecraft video, the world has ever seen!
i think it is possible by mutipul bots* by adding mindcraft and making bots by making then comunicate with others in some sort of way *you'll need AI tokens
Dead SMP theory
Then we would need an Ai for combat.
And make it a 24/7 stream
This is really cool. You mentioned the bots often get stuck, either physically or behaviourally. One thing you could think about, is that in addiiton to having a human supervise the bot, you could have a different AI acting as an "observer". It could see the recent history of what the bot had done, and be given some ways to "unstuck" the bot if it believes it is stuck, to help it back towards its overall goal.
I think this is actually a great idea, especially since our brains kind of do the same, we aren't one script do all types of brains. We have different parts who all manage their own thing, who all have their own goals and agendas. However, put together we form a functioning human (in most cases :P)
ideally implementing some sort of "entropy" variable would help here, when entropy starts to get really low it might mean the bot is stuck in a repetitive task.
How do i download it, please guide.
i feel like it might help with getting "conceptually" stuck if you set up a goal stack? like the overarching goal is to survive and beat the game, but maybe andy can define a subgoal of collecting diamond, which he can then determine has a subgoal of getting an iron pickaxe, which requires iron, and so on. sort of a hierarchy of tasks.
🤎🤎🤎🤎 love this idea
Random good life advice
Hang on hang on, there's something to this. What if it took goals from Advancements and their descriptions? They're already put into a specific order. It might need a little workshopping, but essentially it could give itself each advancement leading up to Free The End in the advancement tree as a goal until it actually gets that achievement.
I've found that if you give an AI a high-level goal (e.g., "beat the game"), it's exceptionally bad at following your prompt. If you instead do as you're suggesting here, and have it break the larger task into smaller pieces, and then build up from there, it's much better at completing the task you intended for it.
For example, if you wanted an AI to generate a book (as some AI grifters do), you wouldn't just ask the AI to write you a book-rather, you ask it to generate synopses for chapters, then sections in those chapters, then individual pages. Once you have these synopses, you can ask it to generate each page, which can be stitched together to form a complete novel (at least, as complete a novel an AI can write).
@@ScareSans that's smart, i can see it working too :D
Poor Andy thinking he’s making his own choices if only his name was neo
Full name: Mr. Anderson 😂
There's a mod that makes it where you can make tiny redstone that is compatible to be placed on blocks tinkered with the chisel mod. Meaning people can make literal 1-1 scale circuit boards in minecraft. Huge redstone projects within a corner of a block, the size of an ant, with the power of a god.
please tell me the name, I must see how long my PC can keep up
What's it called?
tiny redstone mod or something idfk i remember seeing it on curseforge
cant you also just do the same with chisel or that one old mod that allowed you to treat a block as it were 16 blocks?
Omg that sounds so cool
I have cloned your repo and tested it myself, I have some ideas on how to solve many of these issues. I also have fine tuned custom models on my ubuntu server, doing a fine tune for minecraft would be interesting and might lead to a model which actually plays well using a smaller local model.
For things where it gets stuck , I think we should cheat a bit since he does not acutally have eyes, and can see when he gets stuck. We could add a code that identifies when his possition has not changed for a while prompting the model that he could be stuck, and solutions for it (building blocks and jumping to get up, or digging down finding a near place where hees been before he got stuck)
Also check out pixtral 12b, new multimodal model, could one feed his vision to it (the web rendering used)?
that's a pretty good idea. generally giving it more awareness of what it has done overtime and assessing what that might mean in terms of progress or non progress might be a good idea. for example, every 10-20 prompts, we could ask the AI to evaluate its progress over those actions, and from there maybe it could assign itself a new temporary goal to get it into a new situation, and even possibly add the loop it was stuck in during those prompts to a multi shot list of what not to do(though this would be difficult in terms of context limits)
To fine tune a model, someone could host a server in which people spectate a fake player they control using MindCraft commands. Perhaps the server could assign goals to players and award them points on a leaderboard for each task completed. This would allow a bunch of data collection, using competition and challenge as motivation for players to do these tasks.
My worry is that the reasoning skills of huge models like 4o are needed to play Minecraft.
@@americanbagel Excellent idea, I'd like to see the community leveraged for this exact purpose.
beautiful!
Vision is gonna be huge on this. Could also involve multiple agents where one basically tries to build something and the other tries to explore the creation and rate it, say
I am curious to know: How you can add vision to these agents? How difficult is it to implement?
@@KimVallée-l5v there are adhoc ways and "proper" (end to end) ways. The adhoc ways tend not to be super strong. But basically you can use models that already understand images and are able to align images with texts to kinda guide a text model.
The best approaches basically seem to involve training a network from scratch on both text and image data (the core of the network can be the same for both, or you can have like a parallel architectures thing going where, at each stage, the two parallel sides can "talk to each other" to better inform responses) and also text-image-pairs (i.e. image descriptions) to get a network that's competent with both modalities.
A recent approach has a method where it basically uses a language token based approach and an image token based approach fluidly in one go. There are special tokens that basically tell the model "I should now generate an image" or "I have now finished generating an image" and otherwise the model basically just "writes" images embedded directly in text. You could train that sort of model on website code for instance, so it then could generate an entire website complete with meaningful images and all that jazz.
This becomes even more complicated if you want to also allow audio and video understanding, but just in principle you could do that as well, in similar ways.
Ultimately I'd argue it's not a truly satisfactorily solved problem though: There are a lot of caveats with most such models' image understanding where it, for instance, turns out the models in question basically just guess and are very good guessers, rather than actually referring to the image: It turns out not so rarely that you can literally just ask such a model *without* giving it the supposed image and it turns out to get it right more often than not. And in other situations they miss details in images and such.
That's not to say they *always* cheat like that: Adding image training *does* improve their performance. And the good news with Minecraft at least is, that there are good datasets for training them, assuming you want to specialize to that context (which may go against the fully open ended idea here): Simply use minecraft tutorial videos to train your AI! The audio transcripts can be your "text" and the video frames your "images" (or if you have enough compute power, you could directly go with the video)
@@KimVallée-l5v Text vision could work like this: Draw a line from the head of Andy to the monster (2X, foot and head), then check if there are any none air blocks breaking the line. If the line is clear, tell Andy that he sees the monster.
If he means pixel data, I don't have a clue how that would work.
@@KimVallée-l5v It should be trivially easy to add with the help of an AI like claude sonnet, take a game screenshot, send the image to sonnet or GPT4o and the model does the rest. How the images are sent to the AI librarys is covered on the help pages for either library.
@@KimVallée-l5v gpt-4o and claude 3.5 sonnet both have vision capabilities, so it could be as easy as sending screenshots at a regular interval. would probably get expensive fast though.
Would love to see a future where villagers, pillagers, and "Players" all run on your own local computer (or in the cloud with a nice price) with the competence of an actually player as if there are a villager or pillager, or just a player surviving in your world, with varying properties/moods/personalities which are all modifiable by you. Creating a game where things are truly alive. Watching this video made me so excited and hopeful for the perfect LLM model to incorporate into our NPCs releasing really soon.
can't wait for friends to become obsolete
i'd love to play with an ai also. sadly they are paid
na im good
The other day i was discussing this very same concept with my wife but applied to games in general as opposed to just minecraft. We ultimately concluded it would be excessive and annoying, especially in more survival or competitive oriented games. Imagine if something like dark souls or outward had enemies that play just as well as a human, it would insufferable if not unplayable. No thank you.
@@rainbowsorceress2082 I assume that the NPCs can be programmed to play badly, to not make the game too difficult. also, AI can be implemented in other ways than just creating artificial players, like an AI powered dialogue, story, etc
13:38 *emulates checking phone*
I love love love this idea. I’m not here to discourage, but I want to mention that LLMs are predictors. They don’t have emergence available and go off of what’s most common or what’s closest to the subject you’re talking about. They’re good for communicating like a human, but when it comes to behavior and all that, you may be better off by evolving a model and teaching it “okay this is how you build a block, this is how you get food, etc” You can use the accessibility settings in MC to do unit tests too a la “block placed” sound. If you go from the utmost basics and build off of them, I think you may find some incredible results.
One last thing, self feeding an AI with its own outputs will very effectively demonstrate model collapse - try to steer away if you can!
Love this concept, can’t wait to see what else you come up with and how this project evolves over time!
also to add on to your statement maybe training "multiple" models as one big model could work better, kinda like a multicellular organism but in code, because the requirments of placing blocks are very diffrent than the requirements of figuring our what blocks to place or where to place them and ai kinda bug out when they have to solve multi stage problems, also giving the models access to some hard coded tools (like a calculator or smth) would remove the randomness from hard logic tasks and make the ai more predictable and easy to modify, i really hope this series continues because the idea of these models in minecraft is just so fascinating, like for instance the human brain finds symmetry beautiful because of tool usage and human health, so an ai trained in minecraft may evolve to find different things "beautiful" (ie it would replicate certain types of patterns more than others even though doing so provides no benefit towards achieving its goals).
also whatever model ends up being made i think it NEEDS a short term and long term memory system, otherwise i dont think it will be able to recognize patterns or plan out multi stage goals.
I think in order to create an AI that can organically play Minecraft, it needs to "sense" the game via video and audio because right now; all it senses is the text adventure game it's playing.
I assume we could easy do that if we create some kind of "direct AI to game connection" instead of going through the process of interacting through text that interacts with the game.
Keep up the good work, I'm excited to see what you make next!
Hey, that gives me an idea. What about playing minecraft like a text adventure? You use this setup to have the LLM describe the game state, and you play minecraft like a text adventure.
I can't think of a way to integrate ChatGPT into Minecraft through API for visual and auditory perception now but there is active R&D in automating Minecraft with computer vision, which is another form of AI. There may be a way to get ChatGPT and an adept CV program to communicate with each other to play Minecraft until completion but I'd say that's about 5 years away in LLM/CV interaction development.
i think the solution is something different altogether! something like barotone just accesses the game directly to know whats around it, so if you wrote some code to filter out the bits that it shouldn't directly know you could achieve the same-ish result without needing a proper costly vision model
@@WoolyCow Completely forgot about Baritone. Although, I don't think it could be modified to beat Minecraft, it could be integrated into a larger bot program that includes the LLM and CV. Baritone could be used for hazard avoidance and pathfinding then CV could aid the LLM in the game objectives.
@@kualajdm yeah for sure! best solution usually lies somewhere in between ai and algorithms
Does this mean we're getting closer and closer to finally playing Minecraft without being alone?
-Sincerely, a guy with no friends to play online with.
oh boy can't wait for more creepypasta too, might get a resurgence in herobrine or a bot that went crazy and destroyed the player's world
My thoughts were in the same place. I think AI would be better than a real friend.
Herobrine was actually a time-travelling AI
@normalman6078 well you're wrong. Talk to real people. Put in effort. Stop using robots to fake fill a void. It does not fill the void. Talk to real people.
@@normalman6078it would not be -someone who has played minecraft with friends and also tried to have a conversation with an ai
Ideas:
Instead of Andy having to look at the blocks with commands, you could update a readable text file once a second. It can include the inventory, blocks on screen, important location coords, recent sounds, health and hunger stats, etc. Then he'd call on that file endlessly, in order to have a more complete understanding of the world
If you let the bot use /tellraw, you can use the text formatting feature to seperate his voice from his commands.
I can't wait for autonomous agents to play on Minecraft servers alongside real players. In a few years we might have agents who have their own goals, personalities, projects etc.
I would really like to see them try surviving on a small semi-anarchy, semi-vanilla, semi-hardcore server I play on. We are at least a year away from that, but I'm excited.
Semi-anarchy? does that mean like, the rules allow griefing, but there are other ways to prevent it?
Semi-vanilla? does that mean there are cosmetic plugins?
Semi-hardcore? does that mean dying naturally lets you respawn, but being killed by another player can do... something else?
Hmm.... I wonder...
@@thefacethatstares semi anarchy is generally "anarchy but with a claims/homes mod, maybe some dedicated hub zone near spawn that nobody can grief"
semi vanilla is, yeah, generally cosmetic mods and light qol changes
no clue what semihardcore means though
@@thefacethatstares Semi-anarchy because there are some restrictions like you are not allowed to build lag-machines and some cheats are blocked by the anti-cheat and don't work. Semi-vanilla because there are slight changes like 4 spawns instead of 1, custom leveling to access higher level zones that are closer to the world center. 200k world border. Tnt, sand dupers disabled, villager trading nerfed to you can't get the prices down to 1 emerald etc. There is also a trade plugin which allows us to have a trade based economy.
Semi-hardcore because you get banned for 24 hours upon death and the difficulty is locked on hard. The server is called CoreJourney, not very active atm, but I love it.
EDIT: We need players, and while we're waiting for AI agents to became advanced enough to be able to play alongside real people, if anyone's thinking about starting a single-player let's play - consider doing it on this server instead. It's kinda dead atm, so it often feels like a SP experience, but there are also builds scattered around from thousands of people who played here at one point. If you've ever played a SP world but felt lonely - this server might be a decent choice. r/corejourney
@@38mb.The only thing I can think of with semi-hardcore is having a limited amount of lives until you're dead for good
@@thefacethatstares I think my previous comment got nuked, but in short:
you get banned for 24h upon death.
It's anarchy, but there are some rules like no intentionally lagging the server etc.
And even tho the server is vanilla there are stuff like a trade plugin, 4 spawns, 200k world border etc.
It would be quite expensive, but it would be really interesting to see what the result of using Open AI's O1-mini model would be given it's capable of advanced reasoning and would therefore be able to priortise tasks, etc., if given the right prompts.
Latency isn't the greatest though.
You could mix both with a router
o1 would plan the mid long term action
And 4o would deal with short term
Once it has vision... If it ever has vision... I think one of the best ways to prevent a loop would just to have an additional AI alongside it. It could be the companion it needs to let it know that the other is looping, vice versa.
The plot has become more complex.
Multiple chained AIs I think will come closer to emulating human-like behavior. As we are all motivated by multiple things at any given moment.
"You have repeated the same action for 1200 ticks, stop, and think about your life choices. Recalibrate position and surroundings."
@@Starlitsoul0359 Unironically could be what'll save a minecraft AI from looping.
@@ArmoredarmadilloX Simple, funny, and effective.
The second secret to making good fun AI is simple. Add in a little bit of sass, playful banter based interactions between the two different AI would make things not only entertaining, but add a push and pull to bring them both up.
You can switch game-modes faster if you use f3+f4
Poop
How did I not know this
@@EmergentGarden You could ask ChatGPT "What Minecraft keybinds don't I already know?" 🤣
Although, if it got it right it'd be really spooky lol
@@EmergentGarden lol
remember, switching to your functions is always faster than typing
@@Galerak1Claude:
Here's a list of some relatively unknown Minecraft keybinds:
1. F3 + F4: Toggle creative/spectator mode
2. F3 + N: Toggle spectator/previous gamemode
3. F3 + F3: Increase render distance
4. F3 + Shift + F3: Decrease render distance
5. F3 + A: Reload chunks
6. F3 + T: Reload resource packs
7. F3 + B: Show hitboxes
8. F3 + G: Show chunk boundaries
9. Ctrl + Q: Drop entire stack of held item
10. Middle mouse button: Pick block with data (in creative mode)
i love your videos, theyre just essentially tech projects, but theres always undertones of existance and sentience and whatnot. Cool stuff.
All I gained from this video is that Lena Raine is indeed going crazy on that oboe
actually
maybe i just missed it but i don't hear an oboe D:
@@aidanknox2430 Around 2:17 when that lower range instrument comes in. Admittedly, it sounds more digital than an actual oboe, so it can be hard to notice
@@aaronokemaysim7310 that sounds like a reedy bass clarinet and not an oboe (to me at least)
That is a basoon or a bass clarinet
heres an idea, make a 24 hr livestream of andy playing minecraft, since there is need for human interference often, allow the viewers to choose commands and goals for the bot, for a small fee.
You don't even necessarily need a fee.
It still stuck quite often so human has to babyseat it. 24 hours will be too long...
Sounds like something DougDoug would do
Shouldn't be too hard to detect being stuck. Record its commands and verify the usage. Or check position on tick. Rewire brain when detected same position.
Hey Emergent! Been obsessed with your program and talking with Wifies I'll be releasing a video with the bots playing through Skyblock! Lots of work has been put into it and has mention of you for any other interested viewers to head to your channel to find everything! :) Can't wait to see what other fun video ideas are come up with in the future.
“minecraft is open ended there’s no wrong way to play!”
“the ai is playing the game wrong”
0:34 missed opportunity to say "or is there?" with v sauce's theme..
I was actually expecting him too, rather disappointing 😔
Hey v sauce mikle here
This is quite revolutionary... you could literally make a mod to add an AI NPC enemy
Herobrine
Automating taxes so you can play Minecraft: ❌
Automating Minecraft so you can do your taxes
what Claude built was absolutely incredible
I am absolutely mind-blown, really want to test it out myself
This is extremely cool, this feels like a really interesting way to gauge the model’s progress and it was surprising to me how well they did
My personal favorite way to see how well the text gen models are progression is to ask them to write svg files depicting various things. The progression as models have gotten better resembles how children's drawings develop with age; going from simple shapes with basic features to more recognizable figures with body parts
The text prompts while it was endlessly gathering iron and coal were actually transcripts of my brain stoned at 2am mining like a zombie.
"Once I helped the human species evolve into the greatest civilization possible, what goals were really left to either of us? None but go to war with each-other." ~Open-ended AGI
silly comment
@@randomsnow6510Goofy even
then its not open ended, no?
Here's a potential idea to outsource (or crowdsource?) the human supervision needed for a 24/7 livestream: The twitch (or youtube or whatever) live viewers could submit chat commands to instruct the agent(s) and/or potentially help them get unstuck, even while you're not present to oversee everything. Command submissions could be democratically voted upon, based on donations, or handled by some other means. Perhaps even a hybrid approach; have instructions / goal changes be handled one way, and then have "unstucking" (teleport to spawn, etc.) be handled by a real-time popular vote.
Keeping humans in the loop like this could also help prevent forgetfulness of past actions borne of limited context lengths; the viewers could keep the agent(s) on track during long goals whose intermediate steps might be so numerous that they don't all fit within the language model's context window
This is an awesome idea. A sort of twitch plays pokemon type thing, but playing along with an AI bot
This is fantastic
This was very interesting AND calming to watch. Very nice video all around!
Andy died because it prioritized self defense for survival. However, it took the approach of being the aggressor; when violence wasn't needed it was safe in its tunnel. While killings mobs is necessary to survive in minecraft, Andy died because it saw necesscity when it wasn't needed
That opening beginning didn't have to be so amazing bro, I could feel the indomitable human spirit raging in my heart as you talked about living creatures and all their collective achievements, learning to survive in better and more complex ways
Something like this is really cool because it would allow the ai to "feel" it's environment. For instance, say you wanted to do a challenge where you beat the game blindfolded. Kind of impossible. But, an ai that is getting information from the game when "touching" certain blocks, could feel their way through it. Imagine the player walks into a tree. The game tells one ai "player has hit tree on left side at x,y,z". That ai tells the ai making decisions for the player "you feel a rough, hard sensation on your left side. It appears to be gritty. And cool. " The ai making decisions then has to interpret that data and act, or request further information from the narrator ai. It then assess' there is wood there based on the "feeling" and harvests it. Cycle continues until it beats the game through "touch" and sound alone.
its* not it's
@@Pomodorosan 👍
dude! Thats amazing! Don't stop you research please, its so cool to watch. Can't wait to new video
funny that he continually gets stuck based on the new feature of 'raw iron' as I'm guessing the text he's trained on is largely from before it was an item.
I'd love to see how you improve this over time, this was super interesting!
That first section is so well made
Bruh, your video editing and writing are on point. I watched the first 3 min of this video and subscribed.
honestly making a livestream would awaken the ancient souls of the ai spongebob incident, and would DEFINITELY go viral
This is so cool, I am so excited to see where this goes in the future! It’s quite amusing seeing you test the models, like they are your little test subjects
I see a new Emergent Garden vid, I click
Now lemme finish this vid
edit: As I expected, a very good video
I love your videos and your voice. Something is just so soothing and relaxing about the concept and your narration.
I have a tip for hotswapping gamemodes, if you hold down F3 and press F4, four boxes for the four gamemodes will pop up and you can cycle to which one you want by pressing F4. Makes changing gamemodes very fast!
you can use f3 + f4 to change you gamemode quickly in minecraft, without typing out the whole message.
I love your vids
I really like this channel. Thank you for providing this calm tidbit of introspection, that takes itself seriously, in a humble way. It's a great way to end my day. Keep up the good work, and I truly can't wait for your larger scale projects on this. The idea of having many agents each vying for their survival, evolving and adapting sounds so incredible. Sounds expensive, but sooo work it!
One of the best videos I've ever seen, seriously. I've never thought about these possibilities, would love to see more of it in the future.
if it is even possible to get these agents mostly operable without human intervention, it would be incredibly interesting to see something along the lines of an "AI SMP". Giving each agent differing prompting would hopefully lead to some unique interactions, especially if they were able to speak with each other.
Great video! I love Claude's creativeness. The intro was beautiful (if a little verbose). I guess I'm action oriented and like to get to the meat and potatoes of the subjects I watch. It's amazing how far minecraft has come, not to mention AI.
i look forward to the day we can have them with vision, talking to each other, cooperating to build a town
Really cool stuff. I love your analysis and references to some of the real research going on behind the scenes. It is pretty incredible what it can do in creative
14:10 POV: you are a chicken
Viva la larpp!!11
🐔
Pathfinding, there already is something for that. Baritone. usually used to automate travelling or collecting stuff.
Particularly useful if you dgaf about leaving spawn of 2b2t manually.
You could try to integrate baritone into your AI.
Awesome! :D
I think the goal might have been easier for the AI if you had told it step by step, what it needs to do, to stay alive. Like the first day collect wood for crafting, seeds for farming and when its about to get dark, either make a shelter or dig a tunnel and start mining (Avoid combat until you are equipped to it).
I know it was properly not the purpose for this video, but i think it could be exciting to see how far it could go, if it knew more about what to do and how to chain the things together.
Nice video :)
13:39 lil bro thanos snapped that log 💀💀
This is my most hyped project. What do you guys think about similar projects like Altera AI and Elephant AI?
The Castle Claude made is really pretty, it really looks like what a dedicated player would naturally build over time.
Tip for switching gamemodes: Press F3 and F4 at the same time, thats Minecraft's quick gamemode switch keybind. F3+N is similar, but that keybind is a bit far away for me personally.
Who let the furry out of the basement 💀
@@GeneralKenobi69420 no need to hop on the hate train
@@GeneralKenobi69420 Yo mama, she is releasing them all along with her personal zoo because the police found her onlyfans.
12:10 so the bot is just a player with dementia lol
There's a fascinating bot called "Alto Clef". It isn't AI but it does play the game. Also, instead of mineflayer I would try "Barritone". Baritone is good for automation since it can locate blocks, build schematics, and automate finding and collecting blocks. Alto Clef uses Baritone as a base for the bot to complete tasks, such as beating the game, running away from players until it is confident that it can beat them, and also attempt to survive the minecraft world without help. Now i would argue that while Alto Clef is currently superior to AI, it is a static algorithm, so AI will naturally get better at minecraft than Alto Clef entirely.
Edit: fixed some misspelling, nothing big
Amazing! 👏👏
Your videos are a work of art and very interesting.
nobody talking about how he cooked at 2:26
do you know that if you press F3 + N it will toggle between spectator mode and creative mode ? this might make your life easier when the bot get stuck
the build forever command results for claude were actually crazy. That is really inspiring
AI is truly amazing, this wouldn't have been possible 4 years ago, look at where we are now.. AI keeps getting better until it can get better by itself, that's the point of no return, by then the knowledge of any kind of science will explode
Humans can get better by themselves, yet rarely do we choose to do that
13:02, Tip: Use F3 + N or F3 + F4 to switch gamemodes.
Subscribed to see how this turns out. Love seeing what ai and ppl can do with it
the end is never the end is never the end the end is never the end is *loading* is never
I love how the structure Claude built at 22:02 is so similar to ai pictures; it looks normal and impressive at first but if you look a little too close some things don't make sense.
I know the AI doesn’t have feelings but I just imagine it playing minecraft forever after humans are long gone and the computer running the program is just floating through the abyss of space. Thats genuinely terrifying
That reminds me of Porter Robinson’s Shelter music video. Pretty heartbreaking but it’s a masterpiece
This is so impressive. Love the series. Also by the way you can just press F3 and N to switch between creative and spectator mode easily (if you start from creative)! And F3 + F4 opens another quick select menu!
People forget mine raft never had an endgoal until they introduced the "end" update, or are maybe just too young to have played Minecraft before all of the progression they added. If anything the progression limited the time people spend on the game and actually reduces replayability. I remember playing on the same Minecraft server eith my friends for years where the only content was building different things and coming up with things to do pre progression
Yeah the idea that Minecraft can be beat by killing the dragon is a bit ridiculous
God i love videos like this
"He's not very good at planning long term, or setting intermediate goals, or building things when he's self directed like this. He'll just do whatever is immediately doable and after a while he'll forget what he's already done and do it again."
Ignoring the financial cost... You mention that you don't run a twitch stream because it takes too much human supervision. But couldn't twitch solve that, in a Twitch Plays Pokemon sort of manner? I am imagining a separate LLM agent that is taking in both recent state and actions of the game, and recent twitch chat to determine if there are any hints worth relaying to Andy (or whatever agent in Minecraft).
How would that be expensive?
@@sammiller3875 LLM APIs aren’t free
i can not even fathom what it takes to get here, but its amazing.
Andy needs to ask himself "do i have" questions maybe? that way you can build in efficiencies. I do i have 1 coal and 8 iron: then smelt, kind of thinking. Do i have a sword, a pickaxe and probably food items would be the ones id focus him on more.
i love that Claude thought floating lights were cool, which they are.
sigma
This is really cool and I look forward to more videos like this :D Also, I noticed you run /gamemode a bit since you have to babysit the ai. Something a bit easier to do is use the hotkeys F3+N. This makes you swap to your previous gamemode so you can swap easily from spectator to creative.
1- tiene que entrenar su posicion y su accion posible en un rango que aprenda dentro de 1 bloque hacia afuera
2- tiene que saber memorizar que las herramientas estan en el acceso rapido, no necesita ir al inventario
3- tiene que recordar objetivos en terminos de tiempo, calificando asi la importancia de cada termino y su duracion
the first 5 minutes of this go deeper than some documentaries i've watched. seriously so impressed by how well that was spoken. ❤
What if you got multiple AIs to help eachother and cooperate? If one is stuck, or forgets a goal, another AI might point it out, and possibly help
I think an agentic framework for the llms can can improve Andy's abilities. Each llm has prompting unique to its function. For example a movement agent that is prompted on how to move through the world and has functions it calls for movement. Likewise an crafting agent and down the line, a vision agent that "sees" the world using a vision model. All the specialized agents will communicate with one another and use their tools to meets the overarching goals set by the "ceo" agent. Just some food for thought. Awesome project!
Holy cow. I loved the building portion of it! On a much larger scale you could even build stranger AI stuff akin to the Backrooms or the manga _Blame!_
Agency is the key to understanding intelligence. Awesome work!
Loved the video, thank you for the details and links provided! Great job!
Wow... I've got a lot more philosophy than I expected from a title like that.
this ghigh. hilarious stuff, Emergent garden!
22:12
It was. Claude specificially stated he would do this.
If you look here, 19:40, you see Claude gets cut off by the character limit before he can say: "secret passages," or perhaps "passageways," or something similar.
you are so awesome! I love your thought processes
I'm very invested on this project
Keep it up man
Your idea has been eye opening. I will take into account of every idea and reference you shared. I came up with some kind of thought process through this and going into game now. Time to act like a ai.
Very cool work. Really excited to see when the bugs are ironed out enough that this hits a stability critical mass.
Absolutely fascinating! It's amazing to see how AI interacts with Minecraft’s open-ended world. The way it adapts and creates its own strategies is a glimpse into the future of AI development. Can't wait to see how these models evolve! Great job on the video!
This is very cool. I would love to see some feedback or learning loop built where the AI can watch and learn from human players. Also allowing it to ask questions based on what it is seeing to understand it better. Having an AI assistant you can train within Minecraft could open a lot of new interesting gameplay as well as research into real world applications. Keep up the amazing content!
Awesome! I’ve always thought about this idea since I got access to the API and GPT got vision a long time ago.
Though I always imagined a different baseline way of functioning than a text adventure, with more of the LLM looking in and making the commands to a trained AI
Super cool, crazy how far the AI stuff has gone, wont be too long until there's a whole simulation inside Minecraft I bet