You gotta admire how versatile and full of possibilities Minecraft is. The game has been out for about 15 years, and people are still doing unique and interesting things with it.
With the exception of not supporting multithreading, Minecraft is almost a perfect environment for so many things. Just simple enough to not make everything a daunting task, but just complicated enough that there are many different angles to experiment with or ways to add things yourself.
The only other game with as much lasting prevalence imo is tf2 since even now every single meme and event or situation gets turned into a tf2 parody video with millions of views in 2024 still
It's kind of cute how the AIs always place some "special" block as the top of their pyramids instead of just making the whole thing out of one material
@@zyansheeptrue, the rich in Egypt I believe had gold blocks at the tip of their pyramid, a specific guy who was important for some reason I don't remember, probably was a pharaoh- wanted a solid gold pyramid but because that's unreasonable he eventually settled for a gold cap for the pyramid.
@@Nezha_Main That's right, one of my favorite things about the pyramids is that when they were built they didn't look like just rough stacks of sandstone, they were smooth and shiny white because of the decorative layer of limestone on the outside, and the gold cap on top just added to the image. Over time these materials were reused or stolen and all that was left was what we see now. Anyone interested should definitely look up some images of what we think they look like at the time
@@PinkeySuavo The tinted shadow on the ground with his shader pack is creating the geometric gradient pattern because the sun was at the exact angle parallel to the layered glass sphere the AI built, which is diagonal both laterally and longitudinally.
The AI doesn’t even know what that tower looks like IRL, you should get it to describe its own design first so it realises what it’s even building. I’d also recommend something like snippets or get it to make its own codeable rulesets to building before it builds.
Absolutely, I really feel like a secondary model layer whose job is to describe what the building might look like if it was built in minecraft(block types, specific geometric patterns, etc) could go a long way here.
this is a great suggestion! personally i usually use a multi-step process for my tasks that involve AI as well, rather than trying to get them to solve it for me all at once. break the problem down into many separate chunks, go through them one by one to get some solid foundational blocks before combining all the results into a coherent set of solutions. such is the way of programming.
yeah, like if it had a command that took a 2d array of blocks and rotated it around an axis, that would allow it to make a coliseum by drawing a cross section.
@@EmergentGardenI could see pros and cons of this. The reduction of the chat spam seems nice, but in the event that it starts placing blocks somewhere it shouldn’t (like what happened in this video), it might help to see the coordinates aren’t right and easily find where it was. It’s really just up to you which you prefer.
Uhhh maybe also sendcommandfeedback bc like that's for command blocks. I don't think they use command blocks (also ik there is probably something else there but I use bedrock)
I’m really curious to see where AI inside of minecraft goes. I imagine in a few years we’ll have a way to train them on good and bad builds, give them a better concept of what constitutes a “building”, but what I’d REALLY love to see is some kind of simulation between them of an SMP, giving them access to baritone and allowing them to work out how to get things done in an open environment. Let their decision making guide them to the best path. Have them identify problems and come up with contextual solutions. E.G. I’ve been attacked by monsters outside my home at night, the best course of action would be to light up the surrounding area and build a wall to keep monsters out. Have them break that down into simply steps that don’t have to be ai powered but can just be automatic, craft torches, go strip mining to get resources for the wall, etc. I saw a paper about a similar project a while ago where they put down LLM agents in a simulated town and the “people” in the town had a small memory where they remembered previous experiences and had some agency on what they wanted to do, one of them planned a Valentine’s Day party and invited guests to it entirely on its own with no input. Very interesting stuff.
The paper you're referring to was done to advance research in multi-agent cooperative llm systems but a better examples of this would be the "Voyager" minecraft-llm paper. I've just completed a masters in this area, and i'll say we could do this today, it's just it would be costly to train and we have better ways to use our resources than advancing minecraft ai. When making a minecraft AI for research purposes Minecraft is used only as a proxy for the real world, with the idea that if we can make a solution that is super general and solves minecraft it will solve real world useful tasks. If we start coding minecraft-specific tasks for it to do, we're distracting ourselves from the proxy and just making something to play minecraft.
Are the models integrated with vision, or is it literally just "blindly write code and hope for the best" ? I know it'll be expensive, but imagine if you supplied the model with periodic screenshots.
It has vision but not in way we have. It basically has coordinates of blocks nearby not like it can actually see anything it created. All of these things are just numbers to them
@@braineaterzombie3981 I'm just saying, most of these models (not sure about Llama, but it's open source so I'm sure people have made Llama Vision or whatever) have vision capabilities. As in, you can send them pics and they'll understand. Not EXACTLY the same as how we humans 'understand' images, but good enough for use.
@@braineaterzombie3981they do have vision, like in "image data input". 4o even also has audio input (and output even). Passing stream of images still hasn't been released yet, but it's possible with 4o For now we can pass occasional screenshots to gpt4o, gpt4vision and Gemini 1.5
as I've been watching this series I've been thinking about how cool it would be to have a mod that adds a ton of alien-like AI generated structures, it really feels like aliens looked at human architecture and tried to re-create it. I can just imagine stumbling upon those pillars at 4:35 and following them all the way just to find an empty, strange looking building with seemingly no purpose. gameplay wise I imagine players would be disappointed but personally I'm just here for the vibes, and the vibes these buildings give off are immaculate.
I avoided using this word in my original comment because it gets a bad rep, but the whole vibe is very "liminal", it just feels like you're not supposed to be there almost. Everything feels unfinished, and unnatural. The pyramids were pretty good but the smaller structures like the colosseums really give off the vibes I'm talking about.
This might happen actually i hope a modder do it, i would also like to have a AI npcs in the world, imagine wandering finding a npc and try to talk to him to either help u or do something for u, and imagine if they have random personalities, like some of them are agressive, some are not, some are lonely and act weird, i feel it would be insane
I love videos like this that just explore what AI can currently do and treat it as a novelty to play with, rather than making any grandiose promises or predictions. The societal implications of this tech are huge of course, but we can also just slow down, take it in, and play some video games.
There’s something really amazing and unique the even “incomplete” buildings are. Very symmetrically interesting patterns! Keep on making more of these tests and videos!
11:08 lmao. When I was 9 in 2011 I got operator to a small server that had world edit. So I wanted to build a huge hollow glass sphere in the sky... but I set it wrong and the server crashed. When I re-logged, parts of the world for hundreds of blocks were made of glass shapes that weren't quite fully rendered into the giant sphere I had tried to create.
A while ago I tried to get this working, but they were blissfully unaware of their code not working, or that they had previously written code. I might be able to get it working now, but I"m 100% demotivated. I'll just watch your videos instead. Also, you thought about experimenting with instructing the agents to build a fully functional vanilla neural network?
I wonder if it would help to use a multimodal model and automatically take screenshots of the building for feedback so the model can iterate until it judges it to look good.
These kinda remind me of the feeling of the original backrooms with these structures that are almost human but are very clearly not human at the same time
If anyone sees this, I need an explanation. I still don’t rly understand how the AI is kinda just building in Minecraft. It’s just a language model. Is it just coding and that code is put into “mindcraft” then it translates that code into the characters doing something? And how does it kinda just automatically build something? I don’t get how AI is using cheats. And how is the code that AI is creating to build different from mojang’s code that is used to create terrain and villages and stuff?
"Is it just coding and that code is put into “mindcraft” then it translates that code into the characters doing something?" Basically, yes. They write code that gets executed in the game, "mindcraft" being a framework to facilitate that. "And how does it kinda just automatically build something? I don’t get how AI is using cheats." The agents build internal representations of the world, use those to plan buildings, and write code to implement the plans. That's an oversimplification, but yeah. The cheats are just code in the games codebase, and are called from the command line like other functions. They need to be enabled by setting allow cheats to true, IIRC. "And how is the code that AI is creating to build different from mojang’s code that is used to create terrain and villages and stuff?" The agents build only what they're told to, but they can create new designs. The game code has static designs for everything. These are all just my observations.
I believe for the /setblock command you can use keep and it'll only change the block if it's an empty air block - then, if something goes wrong while the bots are building, you could tell them to just replace everything they just built with air as a sort of "undo" feature
I still see a huge gap with AI; it cannot SEE what it is building as it's making it. there isn't even a workaround (that I can think of) for this, which means that they will continuously be unable to notice and fix their own bugs in their code
If you watched one of the previous videos, Claude realized it made a mistake in the middle of building and tried to correct it, only it ended up building the second house into the first house
You could use a greedy-meshing-like approach (but with volumes instead of faces) to post-process the AI's output into chunks of the same block type. Then you can use fill commands to place multiple blocks at once, which would make it much faster. Btw I believe you can also disable chat messages for commands, so the chat is not spammed with setblock messages.
Hello, @EmergentGarden. How it would be setting an AI Minecraft bot in wild there and making it learn by itself from its mistakes via machine learning? This idea fascinates me. Great videos, by the way.
This is a much bigger deal than the narrow domain of minecraft. You’re showing how we can augment the capabilities of the models with better design choices for how they navigate their task. This is a multiplier on algorithmic / scaling improvements for future models.
Are you able to get it to add on to a build it already created like with the panthenon build ask it to add a floor to the bottom of it or is that not possible? I’m so excited to see how advanced the AI builds can get.
Imagine the possibilities. An interesting idea would be to see how he can change the world, ask him to delete everything and create something else to see his capabilities... it would be incredible
This video makes me feel very uncomfortable in a strange way. Seeing these megastructures knowing that no human being was involved in the block to block generation of them. It's like finding ancient structures that have no history or meaning. The feeling of alone-ness evoked by bleak, programmatic structures with none of the chaos or intent that marks a place as human. Each one of these structures feels like they could be the last building at the end of the world, or maybe I'm just off my meds. In a game already as lonely and melancholy as minecraft, these structures almost feel more desolate to me than if the desert had been empty. None of this is a mark against the video or project, mind you, I think this is extremely interesting and the video is well presented. I just wanted to share how the structures made me feel.
Is this even trained on minecraft builds? if not i think thats the problem. Using a LLM/Langauge model for a building ai is unconventional. When you are able to make your own ai and train it with actual minecraft builds and maps for data it would be 100x better
can't wait for ai to sample every map in existence adding to that if it ever gets implemented I'm interested in seeing if I can fine tune the ai to a specific style or type of build by creating some kind of Lora with my own builds
I like that some of these look a little alien, that one listing pyramid with all the gaps feels so strange. That Parthenon, colosseum, all have a strange, nearly otherworldly aesthetic, vaguely recognizable but still strange. I like it.
I have an idea 💡. To enhance AI builders, you should integrate versioning capabilities. This will enable them to receive feedback on their performance and analyze real-world data effectively.
hi sir i'm trying to use the code but i really dont know how to do it so can you maybe make a little tutorial or just reply to this comment so i can do it by myself? sorry for my bad enghlish, have a nice day
i've watched all of your mindcraft videos, and they are very interesting. I have been interested in AI for a while now, so its interesting to see it implemented in a game as complex as minecraft, and seeing the way that AI creates code in a more visual way. I would like to see if there was a way you could get the AI to build something complex, such as a mansion, by having specific parameters about where the AI can build and build each room separately, until you get a final product. And also potentially having different AI models work on the same more complex build, so we would be able to see a more obvious difference in the way they generate code. Overall, these mindcraft videos have been great :)
Amazing work. How do the models react to more abstract requests, such as, "Build a gloomy house"? Or how about "Build something that demonstrates the Banakh-Tarski paradox" (or, more simply, "Build something that proves the Pythagorean Theorem")?
This is one of my favorite AI projects to follow, particularly because of the rapid advancement of AI and the growing competition in the LLM market. Comparing the capabilities of different AI agents in Minecraft is such a novel and awesome idea, like something straight out of the Matrix. Just imagine how advanced these bots would be if they utilized Multimodal models such as GPT-4o!
When using commands, using a ~ Infront of a coordinate will make it relative to the command source (player) So /setblock ~ ~10 ~ sandstone would place sandstone 10m above the AI's feet. This might be useful in preventing the AI from building at (0, 0) - but if the bot physically moves mid build this may not be suitable
I'd love to see them try something with commands/command blocks, like "hey gemini, drop me a pair of unbreaking 2 boots that also apply a haste affect". Could be useful for SMP servers
I'm gonna blow your mind right quick... You can use /fill with two sets of coordinates to possibly speed up building even more, add some more precision, and significantly reduce the amount of commands the bot has to run. This may help with your crashing problem with the "dumb" models. :P I would recommend that you let the models use /fill for larger walls and floors and /setblock for finer details like windows and doors. Let me know what you think!
The Parthenon* the AI built has a great vibe. An Abstract psuedo roman structure, and broken up columns floating in neat rows away from it in a flat plane. I can see something like this being used in a story about something otherworldly. Precursors, an AI ran cyberspace being broken, a depiction of heaven, etc. Accidental AI art.
this makes me think of future AI "archaeology". imagine advanced AIs hundreds of years from now tracing back the evolution of AI designed architecture to these Minecraft structures...
I hope that one day when computers are powerful enough someone would create a mod that incorporates one off ai generated buildings into minecraft survival gameplay, it would make minecraft feel more mysterious and fun kinda like the feeling we get back when we just started playing minecraft without knowing much about gameplay mechanics
As level designer myself i find this videos quite interesting. Wonder if they could work with .map format which is used to compile BSP maps in Quake, Half-life, etc.?
Cool video! I remember once having an idea, it would be cool if there was a server where only AI players would play, who would self-learn thanks to live players, after which it would be difficult to distinguish them from real people, they could communicate with each other, build houses and grief each other
While “bypassing all that crap” makes for bigger and Better™️ builds, I think I personally find the AI needing to navigate the Minecraft world and its inventory more interesting. For me, the first video showed AI and Minecraft as it’s played as a game; using commands is just language model -> set block at position to state which could be achieved in any 3D software. To be honest I’d be more interested revisiting the concepts of having the AI as players that need to interact with the world, not just bots that stand there and run commands.
giving them the capability to use /fill would allow for more efficiency and maybe versatility than individual /setblocks, (and with it, more destructive power, of course)
Have you considered trying to allow the agents to improve upon their earlier attempts at generating new actions? Typically when I’m trying to get boilerplate code out of LLMs they are pretty good at fixing their mistakes when they are pointed out. You may even be able to get them to start abstracting the code they’ve written and then addressing certain parts of the structure. Another thought would be to give them access to world edit commands to try to give them better tools to work with.
gaming in 2024 looks crazy. loved this. also gemini being one of the dumber, but not quite the dumbest AI is so accurate that it's funny. i really want to love gemini... but now i just default to chat gpt for anything that requires any level of reliability.
Imagine the possibilities this would have if this could analyze real buildings from google maps. Build the earth project could be completed so much quicker. You could have not only cities but every home (on google maps) be built
I think i understand why Llamas skyscraper was so short. It is ten blocks (floors) tall so theorethically it did the assigned task right, it just so happened that what the AI considered a floor and what you considered a floor differed. It just thought that floors don't also need some hollow space in between them.
This is really cool, we can't lie about that. Have you discovered anything new with the AI's since this video? When is your next one? I'm really enjoying your videos, your showcasing of their abilities are fun to watch.
I swear, a company needs you to create a AI to make something like this, i bet that this company will be good friends with OpenAI and you will be the chief (im Brazilian, and i must say, these videos are so cool! Hello from brazil!)
"Don't worry I'm not gonna crash into them" 💀💀
Dark!!
👨✈️✈️🏬🏬
This dudes humor is something else
@@BadChess56It's so subtle, but when you hear it, it hits different
mr president, a second minecrafter has hit the towers
You gotta admire how versatile and full of possibilities Minecraft is. The game has been out for about 15 years, and people are still doing unique and interesting things with it.
With the exception of not supporting multithreading, Minecraft is almost a perfect environment for so many things. Just simple enough to not make everything a daunting task, but just complicated enough that there are many different angles to experiment with or ways to add things yourself.
@@Gandhi_Physique As long as you have enough runtime to deal with it being one of the worst optimised games I've ever played
The only other game with as much lasting prevalence imo is tf2 since even now every single meme and event or situation gets turned into a tf2 parody video with millions of views in 2024 still
It's kind of cute how the AIs always place some "special" block as the top of their pyramids instead of just making the whole thing out of one material
Sentient love to build tall
He tells bots to do this to check their capabilities
Humans do this too!
@@zyansheeptrue, the rich in Egypt I believe had gold blocks at the tip of their pyramid, a specific guy who was important for some reason I don't remember, probably was a pharaoh- wanted a solid gold pyramid but because that's unreasonable he eventually settled for a gold cap for the pyramid.
@@Nezha_Main That's right, one of my favorite things about the pyramids is that when they were built they didn't look like just rough stacks of sandstone, they were smooth and shiny white because of the decorative layer of limestone on the outside, and the gold cap on top just added to the image. Over time these materials were reused or stolen and all that was left was what we see now. Anyone interested should definitely look up some images of what we think they look like at the time
That tower joke caught me off guard 💀💀
ikr
same
"See those towers over there? Let's fly towards them"
"Don't worry I'm not going to crash into them" was so out of pocket
me when september
:)))))))))))))))))))))))))))))))))))))))
@@solarisNT-v4j eleventh
Some unlicensed 13 years ago
0:18 I’ll give that joke a 9 out of 11
Underrated comment
6:05 The sun being at the exact parallel angle making the geometric shadow was a fantastic coincidence
Also, you should install the Distant Horizons mod so we can see all the builds in the world at once!
If you are on Nvidia, Nvidium does an even better job than distant horizons!
what do you mean?
@@PinkeySuavo The tinted shadow on the ground with his shader pack is creating the geometric gradient pattern because the sun was at the exact angle parallel to the layered glass sphere the AI built, which is diagonal both laterally and longitudinally.
@@michaelproeber1953 wouldn't the effect be the same even if sun was higher/lower etc? It looks like a normal shadow to me :P
The AI doesn’t even know what that tower looks like IRL, you should get it to describe its own design first so it realises what it’s even building.
I’d also recommend something like snippets or get it to make its own codeable rulesets to building before it builds.
Absolutely, I really feel like a secondary model layer whose job is to describe what the building might look like if it was built in minecraft(block types, specific geometric patterns, etc) could go a long way here.
this is a great suggestion!
personally i usually use a multi-step process for my tasks that involve AI as well, rather than trying to get them to solve it for me all at once. break the problem down into many separate chunks, go through them one by one to get some solid foundational blocks before combining all the results into a coherent set of solutions. such is the way of programming.
yeah, like if it had a command that took a 2d array of blocks and rotated it around an axis, that would allow it to make a coliseum by drawing a cross section.
the best pyramid turning out to be surrounding previous failed pyramids feels like a metaphor for something, but i'm not sure what.
Every failure is a crucial step towards progress
it's almost liek sum king of... forshadow wing...
9:17 that would be sick having a ancient city inside of a pyramid
Wise word to remember about A.I. "The more creative powers you give them, the more destructive powers you give them. There is no way around it."
tbh if humanity can't find a way to coexist with or survive against something we created ourselves that's just a massive skill issue on our end
yeah that quote stuck out to me as well... it goes hard tbh
Killswitches??
Real
6:00 omg that actually caught me off-guard holy moly
I dont get it
@@HappyPlaysWasTaken the perfectly made rainbow sphere is really impressive
yeah same, maybe i am just easily impressed but when gpt built that i really thought "wow thats so cool!"
You can do "/gamerule commandblock_output false" to make it so that it won't display every single block that they place in the chat
Oh cool, will use that. Thanks!
@@EmergentGarden small mistake it's "commandBlockOutput" not "commandblock_output"
@@EmergentGardenI could see pros and cons of this. The reduction of the chat spam seems nice, but in the event that it starts placing blocks somewhere it shouldn’t (like what happened in this video), it might help to see the coordinates aren’t right and easily find where it was. It’s really just up to you which you prefer.
@@Sniper1.1 interesting. Hadn't considered that
Uhhh maybe also sendcommandfeedback bc like that's for command blocks. I don't think they use command blocks (also ik there is probably something else there but I use bedrock)
I’m really curious to see where AI inside of minecraft goes. I imagine in a few years we’ll have a way to train them on good and bad builds, give them a better concept of what constitutes a “building”, but what I’d REALLY love to see is some kind of simulation between them of an SMP, giving them access to baritone and allowing them to work out how to get things done in an open environment. Let their decision making guide them to the best path. Have them identify problems and come up with contextual solutions. E.G. I’ve been attacked by monsters outside my home at night, the best course of action would be to light up the surrounding area and build a wall to keep monsters out.
Have them break that down into simply steps that don’t have to be ai powered but can just be automatic, craft torches, go strip mining to get resources for the wall, etc.
I saw a paper about a similar project a while ago where they put down LLM agents in a simulated town and the “people” in the town had a small memory where they remembered previous experiences and had some agency on what they wanted to do, one of them planned a Valentine’s Day party and invited guests to it entirely on its own with no input. Very interesting stuff.
The paper you're referring to was done to advance research in multi-agent cooperative llm systems but a better examples of this would be the "Voyager" minecraft-llm paper. I've just completed a masters in this area, and i'll say we could do this today, it's just it would be costly to train and we have better ways to use our resources than advancing minecraft ai. When making a minecraft AI for research purposes Minecraft is used only as a proxy for the real world, with the idea that if we can make a solution that is super general and solves minecraft it will solve real world useful tasks. If we start coding minecraft-specific tasks for it to do, we're distracting ourselves from the proxy and just making something to play minecraft.
What happens if you ask the AI to build esoteric concepts? Like "build the meaning of life" or something bizzare and conceptual like that.
It will probably make something shit.I is really hard for them to work even if proper prompt were given
Dirt blocj
it builds the library of babel
Are the models integrated with vision, or is it literally just "blindly write code and hope for the best" ? I know it'll be expensive, but imagine if you supplied the model with periodic screenshots.
It has vision but not in way we have. It basically has coordinates of blocks nearby not like it can actually see anything it created. All of these things are just numbers to them
@@braineaterzombie3981 I'm just saying, most of these models (not sure about Llama, but it's open source so I'm sure people have made Llama Vision or whatever) have vision capabilities. As in, you can send them pics and they'll understand.
Not EXACTLY the same as how we humans 'understand' images, but good enough for use.
@@braineaterzombie3981they do have vision, like in "image data input". 4o even also has audio input (and output even). Passing stream of images still hasn't been released yet, but it's possible with 4o
For now we can pass occasional screenshots to gpt4o, gpt4vision and Gemini 1.5
@@braineaterzombie3981it knows where it is because it knows where it's not
I'm pretty certain in this case it's the latter
I wonder how it would handle a prompt asking it to make a maze or labyrinth
this guy hit a gold mine with these videos, endless ideas. Id like them to try to build cities or towns next.
as I've been watching this series I've been thinking about how cool it would be to have a mod that adds a ton of alien-like AI generated structures, it really feels like aliens looked at human architecture and tried to re-create it.
I can just imagine stumbling upon those pillars at 4:35 and following them all the way just to find an empty, strange looking building with seemingly no purpose. gameplay wise I imagine players would be disappointed but personally I'm just here for the vibes, and the vibes these buildings give off are immaculate.
I avoided using this word in my original comment because it gets a bad rep, but the whole vibe is very "liminal", it just feels like you're not supposed to be there almost. Everything feels unfinished, and unnatural. The pyramids were pretty good but the smaller structures like the colosseums really give off the vibes I'm talking about.
I agree a vast desert with these ai generated structure is so ominous and like you said liminal I hope someone makes something like that someday
This might happen actually i hope a modder do it, i would also like to have a AI npcs in the world, imagine wandering finding a npc and try to talk to him to either help u or do something for u, and imagine if they have random personalities, like some of them are agressive, some are not, some are lonely and act weird, i feel it would be insane
@@iish4d_w360 theres actually a minecraft mod trying to do that now, you need credits for it tho
tbh it would make a great base
I love videos like this that just explore what AI can currently do and treat it as a novelty to play with, rather than making any grandiose promises or predictions.
The societal implications of this tech are huge of course, but we can also just slow down, take it in, and play some video games.
I would love these AI bots building liminal spaces in minecraft. It would totally feel disorienting.
That would be epic
100% agree, AI is very creative and I can imagine them doing that in the future for sure
A procedurally generated liminar dream/nightmarescape
OMG you are genius
[･･]
@@FFA-fg2bl "AI is very creative"
It was a good choice giving the bots cheats.
There’s something really amazing and unique the even “incomplete” buildings are. Very symmetrically interesting patterns! Keep on making more of these tests and videos!
Survival test! It would be the best thing to just have a survival and have these slaves to mine and get wood. Would be so funny.
You should see if this can be used to help with the build the earth project. This might actually make it possible with further developments.
Genius idea
@@Nitekom Thank you
11:08 lmao. When I was 9 in 2011 I got operator to a small server that had world edit. So I wanted to build a huge hollow glass sphere in the sky... but I set it wrong and the server crashed. When I re-logged, parts of the world for hundreds of blocks were made of glass shapes that weren't quite fully rendered into the giant sphere I had tried to create.
A while ago I tried to get this working, but they were blissfully unaware of their code not working, or that they had previously written code. I might be able to get it working now, but I"m 100% demotivated. I'll just watch your videos instead. Also, you thought about experimenting with instructing the agents to build a fully functional vanilla neural network?
I wonder if it would help to use a multimodal model and automatically take screenshots of the building for feedback so the model can iterate until it judges it to look good.
it has the data of the area around it
You can also use /fill command to make it even faster if you are doing cuboid etc or use worldedit for more complex stuff like spheres.
Worldedit also has a handy undo function. I have to use it a lot.
These kinda remind me of the feeling of the original backrooms with these structures that are almost human but are very clearly not human at the same time
As someone who is tinkering with writing AI and loves Minecraft I thank you for making these videos.
"The more creative power you give them, the more destructive power you give them." - Foreshadowing our future AI overlords.
im watching this while rendering a 12:03 minute long video lol
Are u able to use gpt-4o aalready and is it better than gpt-4? Also gpt-4o might be even better with the vision aspect.
If anyone sees this, I need an explanation.
I still don’t rly understand how the AI is kinda just building in Minecraft. It’s just a language model. Is it just coding and that code is put into “mindcraft” then it translates that code into the characters doing something?
And how does it kinda just automatically build something? I don’t get how AI is using cheats.
And how is the code that AI is creating to build different from mojang’s code that is used to create terrain and villages and stuff?
"Is it just coding and that code is put into “mindcraft” then it translates that code into the characters doing something?"
Basically, yes. They write code that gets executed in the game, "mindcraft" being a framework to facilitate that.
"And how does it kinda just automatically build something? I don’t get how AI is using cheats."
The agents build internal representations of the world, use those to plan buildings, and write code to implement the plans. That's an oversimplification, but yeah. The cheats are just code in the games codebase, and are called from the command line like other functions. They need to be enabled by setting allow cheats to true, IIRC.
"And how is the code that AI is creating to build different from mojang’s code that is used to create terrain and villages and stuff?"
The agents build only what they're told to, but they can create new designs. The game code has static designs for everything.
These are all just my observations.
Can you ask it to build a multi floor maze inside the pyramid?
9/11 joke detected
I just started to watch the video and immediately he says “See those buildings, I’m gonna fly into them” lmao
I believe for the /setblock command you can use keep and it'll only change the block if it's an empty air block - then, if something goes wrong while the bots are building, you could tell them to just replace everything they just built with air as a sort of "undo" feature
I still see a huge gap with AI; it cannot SEE what it is building as it's making it. there isn't even a workaround (that I can think of) for this, which means that they will continuously be unable to notice and fix their own bugs in their code
If you watched one of the previous videos, Claude realized it made a mistake in the middle of building and tried to correct it, only it ended up building the second house into the first house
You could use a greedy-meshing-like approach (but with volumes instead of faces) to post-process the AI's output into chunks of the same block type. Then you can use fill commands to place multiple blocks at once, which would make it much faster.
Btw I believe you can also disable chat messages for commands, so the chat is not spammed with setblock messages.
Hello, @EmergentGarden. How it would be setting an AI Minecraft bot in wild there and making it learn by itself from its mistakes via machine learning? This idea fascinates me. Great videos, by the way.
This is a much bigger deal than the narrow domain of minecraft.
You’re showing how we can augment the capabilities of the models with better design choices for how they navigate their task.
This is a multiplier on algorithmic / scaling improvements for future models.
You: hmm interesting its made out of concrete, glass and glowstone
Also you: please build it out of concrete, glass and glowstone
This guy is a legend, in less than 30 seconds in he already made a two tower joke that includes planes. (You know what i mean)
Are you able to get it to add on to a build it already created like with the panthenon build ask it to add a floor to the bottom of it or is that not possible? I’m so excited to see how advanced the AI builds can get.
Im hoping one day you can fill minecraft with 100 bots and watch the chaos unfold
Genuinely one of the best things I've seen done with AI, this has actually given me build ideas
this is awesome dude, im so invested in this project
If you want to fill a place with the same block just use /fill command instant of placing every block with /setblock .
Imagine the possibilities. An interesting idea would be to see how he can change the world, ask him to delete everything and create something else to see his capabilities... it would be incredible
This video makes me feel very uncomfortable in a strange way. Seeing these megastructures knowing that no human being was involved in the block to block generation of them. It's like finding ancient structures that have no history or meaning. The feeling of alone-ness evoked by bleak, programmatic structures with none of the chaos or intent that marks a place as human.
Each one of these structures feels like they could be the last building at the end of the world, or maybe I'm just off my meds.
In a game already as lonely and melancholy as minecraft, these structures almost feel more desolate to me than if the desert had been empty.
None of this is a mark against the video or project, mind you, I think this is extremely interesting and the video is well presented. I just wanted to share how the structures made me feel.
I have a question, how do I get them to build all this in my Minecraft world is there like a tutorial?
0:05 september time
I do not know how you do this, but I have another option: use AHK
9:38 the music combined with "built it on top of failed pyramids" is so emotional.. makes me want to cry. Stellar video !!
The failed pyramids actually worked out in their favor too, like a supporting platform that made it look better I love it
Is this even trained on minecraft builds? if not i think thats the problem. Using a LLM/Langauge model for a building ai is unconventional. When you are able to make your own ai and train it with actual minecraft builds and maps for data it would be 100x better
can't wait for ai to sample every map in existence
adding to that if it ever gets implemented I'm interested in seeing if I can fine tune the ai to a specific style or type of build by creating some kind of Lora with my own builds
Could you use claude 3.5 which is the new best model?
Would be great if they could make some map arts!
Make different ai’s go into hardcore mode and see which one survives
How do you install the chatbots onto your game?
Blind Spots is the best minecraft beta song
when they build something frozen on the inside
I wonder if the Ai could use world edit to do stuff?
I like that some of these look a little alien, that one listing pyramid with all the gaps feels so strange. That Parthenon, colosseum, all have a strange, nearly otherworldly aesthetic, vaguely recognizable but still strange. I like it.
bro im only 12 seconds into the video 💀
This is amazing. I must try it!
Can't wait when you test Sonnet 3.5
Looking forward to the setup tutorial!
minecraft music......
this is by far my most anticipated AI projects. I canot wait to try it in the future when it's a bit more stable and easy to use
I love these Ai building stuff in minecraft videos
that first face got me good
have two but ask one to the the right side of the brain and the other the left when building togeather
I have an idea 💡. To enhance AI builders, you should integrate versioning capabilities. This will enable them to receive feedback on their performance and analyze real-world data effectively.
hi sir i'm trying to use the code but i really dont know how to do it so can you maybe make a little tutorial or just reply to this comment so i can do it by myself? sorry for my bad enghlish, have a nice day
That giant pyramid looks like it might be as big as the one cubfan built in Hermitcraft SMP a couple seasons ago.
i've watched all of your mindcraft videos, and they are very interesting. I have been interested in AI for a while now, so its interesting to see it implemented in a game as complex as minecraft, and seeing the way that AI creates code in a more visual way. I would like to see if there was a way you could get the AI to build something complex, such as a mansion, by having specific parameters about where the AI can build and build each room separately, until you get a final product. And also potentially having different AI models work on the same more complex build, so we would be able to see a more obvious difference in the way they generate code. Overall, these mindcraft videos have been great :)
Amazing work. How do the models react to more abstract requests, such as, "Build a gloomy house"? Or how about "Build something that demonstrates the Banakh-Tarski paradox" (or, more simply, "Build something that proves the Pythagorean Theorem")?
whats the shaders ?
This is one of my favorite AI projects to follow, particularly because of the rapid advancement of AI and the growing competition in the LLM market. Comparing the capabilities of different AI agents in Minecraft is such a novel and awesome idea, like something straight out of the Matrix. Just imagine how advanced these bots would be if they utilized Multimodal models such as GPT-4o!
When using commands, using a ~ Infront of a coordinate will make it relative to the command source (player)
So /setblock ~ ~10 ~ sandstone would place sandstone 10m above the AI's feet.
This might be useful in preventing the AI from building at (0, 0) - but if the bot physically moves mid build this may not be suitable
Are you going to do any testing with Gemma, Gemini 1.5 Flash, Claude 3.5 Sonnet, Llama 3.1, or Gpt4o- mini?
I'd love to see them try something with commands/command blocks, like "hey gemini, drop me a pair of unbreaking 2 boots that also apply a haste affect". Could be useful for SMP servers
I wonder if you could just have an AI buddy. A friend. A modded AI pal to survive & build with you. Like an NPC+. That'd be amazing.
This creations will only get better and more advanced as the LLM's get more advanced. Really looking forward to what these are capable of. Subbed!
I'm gonna blow your mind right quick...
You can use /fill with two sets of coordinates to possibly speed up building even more, add some more precision, and significantly reduce the amount of commands the bot has to run. This may help with your crashing problem with the "dumb" models. :P
I would recommend that you let the models use /fill for larger walls and floors and /setblock for finer details like windows and doors. Let me know what you think!
The Parthenon* the AI built has a great vibe. An Abstract psuedo roman structure, and broken up columns floating in neat rows away from it in a flat plane. I can see something like this being used in a story about something otherworldly. Precursors, an AI ran cyberspace being broken, a depiction of heaven, etc.
Accidental AI art.
this makes me think of future AI "archaeology". imagine advanced AIs hundreds of years from now tracing back the evolution of AI designed architecture to these Minecraft structures...
I hope that one day when computers are powerful enough someone would create a mod that incorporates one off ai generated buildings into minecraft survival gameplay, it would make minecraft feel more mysterious and fun kinda like the feeling we get back when we just started playing minecraft without knowing much about gameplay mechanics
As level designer myself i find this videos quite interesting. Wonder if they could work with .map format which is used to compile BSP maps in Quake, Half-life, etc.?
Cool video! I remember once having an idea, it would be cool if there was a server where only AI players would play, who would self-learn thanks to live players, after which it would be difficult to distinguish them from real people, they could communicate with each other, build houses and grief each other
While “bypassing all that crap” makes for bigger and Better™️ builds, I think I personally find the AI needing to navigate the Minecraft world and its inventory more interesting. For me, the first video showed AI and Minecraft as it’s played as a game; using commands is just language model -> set block at position to state which could be achieved in any 3D software. To be honest I’d be more interested revisiting the concepts of having the AI as players that need to interact with the world, not just bots that stand there and run commands.
giving them the capability to use /fill would allow for more efficiency and maybe versatility than individual /setblocks, (and with it, more destructive power, of course)
So satisfying how quick and dirty a.i. this can be used to develop new worlds for Minecraft. Definately next gen gaming for minecraft.
Have you considered trying to allow the agents to improve upon their earlier attempts at generating new actions? Typically when I’m trying to get boilerplate code out of LLMs they are pretty good at fixing their mistakes when they are pointed out. You may even be able to get them to start abstracting the code they’ve written and then addressing certain parts of the structure. Another thought would be to give them access to world edit commands to try to give them better tools to work with.
gaming in 2024 looks crazy. loved this.
also gemini being one of the dumber, but not quite the dumbest AI is so accurate that it's funny. i really want to love gemini... but now i just default to chat gpt for anything that requires any level of reliability.
Imagine the possibilities this would have if this could analyze real buildings from google maps. Build the earth project could be completed so much quicker. You could have not only cities but every home (on google maps) be built
I think i understand why Llamas skyscraper was so short. It is ten blocks (floors) tall so theorethically it did the assigned task right, it just so happened that what the AI considered a floor and what you considered a floor differed. It just thought that floors don't also need some hollow space in between them.
This is really cool, we can't lie about that. Have you discovered anything new with the AI's since this video? When is your next one? I'm really enjoying your videos, your showcasing of their abilities are fun to watch.
I swear, a company needs you to create a AI to make something like this, i bet that this company will be good friends with OpenAI and you will be the chief (im Brazilian, and i must say, these videos are so cool! Hello from brazil!)