ฝัง
- เผยแพร่เมื่อ 16 มิ.ย. 2024
- I'm making a voxel engine with RUST! (the programming language...)
I originally was making my voxel game with Unity, but my love for the Rust programming language caused me to see if it's viable to swap over to that.
This episode is all about optimizing the code for the world generation.
Multithreading, benchmarking, profiling and data compression, are all things I've tackled with lately.
My discord group:
/ discord
Want to support me?
Patreon: / tantandev
XMR:
43Ktj1Bd4Nkaj4fdx6nPvBZkJewcPjxPB9nafnepM7SdGtcU6rhpxyLiV9w3k92rE1UqHTr4BNqe2ScsK1eEENvZDC3W1ur
timestamps:
0:00 hi
0:43 Tantans mixtape: gotta go fast
0:55 video layout
1:33 world generation
2:58 what we need to optimize
4:04 queue
4:37 multithreading
7:27 benchmarking & profiling
9:39 Generation settings for my pc
9:58 problems
10:26 Unity VS Rust
11:09 pain
11:22 the winner is
12:11 data compression
Resources:
The tech I'm using!
Graphics rendering api: wgpu.rs/
multithreading: github.com/rayon-rs/rayon
benchmarking: github.com/bheisler/criterion.rs
profiling: crates.io/crates/profiling, using Optick feature
The tutorial I followed to learn wgpu:
sotrh.github.io/learn-wgpu/
Relevant resources that may help when learning how to make a voxel engine:
sites.google.com/site/letsmak...
Voxel Game Mesh Optimizations by @Hopson
• Voxel Game Mesh Optimi...
I Made Minecraft in 24 Hours by @Sam Hogan
• I Made Minecraft in 24...
Making Minecraft from scratch in 48 hours (NO GAME ENGINE) by @jdh
• Making Minecraft from ... - บันเทิง
I can't thank you guys enough for the support!
my discord group:
discord.gg/9P8QSYf
Maplestory brought the familiar system back time to go back on your Perma beginner account plzz :(
Practical optimization is two parts memory allocation strategy, one part algorithmic complexity with maybe a little knowledge of CPU caching thrown in. Ideally you minimize the number of memory allocations and adopt algorithmic approaches that bring you as close as possible to O(1). Got a loop that does 1 million allocations, first start by doing one allocation of a million instances and your program will gain performance increase in orders of magnitude. On the low end once you've addressed allocation strategy and algorithmic complexity you can eek a little bit out by removing or simplifying calculations, or finding opportunities to precalculate, etc. Throwing more threads at your code is not optimization. Keep in mind that a modern CPU-core executes billions of instructions per second, and the reason your code runs so slowly is because the CPU is effectively idle waiting for other tasks. Throw more threads at the problem without fixing the underlying cause is not optimization and will only compound the problem. Fix the underlying problem first, then once you have a sleek fast implementation, make it parallel and you'll unlock the true potential of the hardware.
I feel bad, your devlogs being so underrated
slow and steady wins the race :)
Tantan really dipped for a month a wrote his own engine lmao
Now, let me get back to my cave...
lmao
Your devlogs have to be one of the most underrated on this platform. Awesome job and love the song!
Dude thanks a lot!
Rust is love, rust is life
You use Rust?
I use Rust. Poorly, mind you, but I do use it.
@@openroomxyz I do
yes baby!
Your videos are as addicting as coding Rust. Thanks for those super high quality content and format.
Thank you dude!
Using any kind of println!(...) or similar console output for debugging will always slow down your program. Also, did you test your Rust code in a --release build or a debug build (default when you run cargo build/run) against whatever Unity is doing by default 🙂? You didn't mention, so I thought to ask.
Our favorite rapping/didgeridoo playing Swedish coder is BACK.
haha that has to be the funniest description I've ever heard!
Just found your videos recently and I love how you show your code with the highlighting, different angles, and smooth transitions! Appreciate the work you put into making these high quality devlogs!
I appreciate it!
This stuff is so interesting, I love your video style. Didn't even realize the video was 16 minutes long.
Ah good to hear, thanks!
Now that I am making a voxel engine of my own, these videos are so helpful! Thank you!
nice video! i rly liked that you talked about bitwise operations, it's such an interesting topic, this video is amazing as always, can't wait for the next one
Thanks! It was super interessting learning about bitwise operations, zeroes and ones everywhere :)
Great video! Really like the indepth dive into stuff and even I not a programmer understood :)
Impressive that you manage to not only do all that work on the game and engine but also create this high-quality video content!
Keep up the good work! For a one-person-army these are some really impressive results!
These are a joy to watch. I want that profile & optimize song on spotify :D
Just stumbled on your channel! ABSOLUTELY LOVE YOUR CONTENT. Great work 😊
Thank you means a lot!
Watching your videos gives me energy, thank you.
these comments gives me energy, thank you!
dope, gl with improving the speed!
Thanks! Got lots to learn still
Looks like you jumped into the optimisations hole and have been having a good time! It's excellent to hear how you've been making this faster and in your own entertaining way. 😂
man, these videos motivate me so much. bless you, keep this up!
Thank you for the fuel! These comments motivate me
well produced, informative and funny!
I started learning rust thanks to you, mr tantan! I am loving it!
keep up with the videos man, you're a inspiration to keep up with rust.
Woah, what an on spot recommendatjon youtube! I was always curious to see how would a custom engine work
Don't think it was a waste of time!, you got alot of experience and since youtube recommended me this video it means this video is doing great, keep up
2x slower than the most famous engine on the market is awesome, you're a genius, looking forward to subscribing (sorry, i'm REALLY picky)
Great work! Really sells the difference between the two engine approaches. One pain point is that Perlin is a very visibly square-biased function for noise. There's a sort of vicious cycle in the PCG community where it's taught far more often than the alternatives which address its issues, and far too often not in the context of techniques which mitigate its issues. As resource creators we should strive to teach the best possible information, and this is an area that really needs it. Best of luck with your channel!
Indeed. Perlin noise should be banished forever. Hopefully more devs will start using Simplex noise.
Oh wow I just cant get enough of tantan!
this is pure tantantainment!!
As for now here are all of the libraries I'm using:
image = "0.23"
winit = "0.24"
cgmath = "0.18"
env_logger = "0.7"
log = "0.4"
futures = "0.3"
bytemuck = {version = "1.4", features = ["derive"]}
anyhow = "1.0"
tobj = "3.0.0"
rand = "0.8.3"
lazy_static = "1.4.0"
noise = "0.7.0"
bracket-noise = "0.8.2"
profiling = { version = "1.0", features = ["profile-with-optick"]}
lagoon = { version = "0.1.2", features = ["recv", "scope"]}
hashbrown = "0.11.2" # fast hashmap
num_cpus = { version = "^1", optional = false }
wgpu = "0.8.1"
# pooling library
lifeguard = "0.6.1"
generational-arena = "0.2.8"
Having optimized code just feels so SATISFYING 😩
the minecraft chunk size is going to get bigger soon. Really makes me wonder how they were able to handle such without lagging. I really wish GMS2 (the game engine I use) can handle multithreading without having to add external code. Maybe I should move to Rust? If ever I do, you know who to blame :) Great video as always!
Here I am, googling how Minecraft chunk generation and world saving (anvil file format) work in the morning, and in the evening Tantan releases the most comprehensible take on it ever. :D
haha perfect timing!
I wrote something similar in Java and later C++ years ago, but went farther, with ambient occlusion, textures, some variability on lighting, proper shadows. I then started to convert it to Vulkan and lost progress as life got in the way.
Compression of those vertices is critical, but I recommend you skip the unnecessary bit fiddling and just use bytes for relative x,y,z, and another byte for your "face" (normal).
Another key is to reduce geometry by making things like texture and brightness (from MC-style falling light, or ambient occlusion) dynamic so that you can ALWAYS mesh large flat areas into simpler shapes (with simpler geometry than it would be if you used one square per cube). I found moving brightness to textures and using a binning system was optimal.
Good luck! I had a TON of fun working on my system!
Oh, and be sure to use ambient occlusion, render front-to-back, and if you can, render down the primary, secondary and finally terteriary axis. That was my last improvement, and it made a huge difference in eliminating invisible geometry.
He did Rust because it was too slow
dude... that mixtape... do you have more of it? 🙏 i can't get enough
Further compression could be achieved by storing colors as references so you don't need to replicate the rgb val for every item.
I would love too see maybe some rust tutorials
I personally wanna see more rust tutorials also!
I don't think I could at the moment handle doing that and keeping up with making devlogs unfortunately.
As polyglot dev, I feel like there's more "it magically does this" in Rust than any other language I work with.
First off gratz on the knowledge gain and engine :D. Secondly I have been wondering and unable to find an answer to if the order of the triangles of a mesh matter in terms of speed? Like imagine a flat mesh terrain/grid, and removing a quad/two triangles and adding in say a tree that fits the hole left behind. Would adding the triangles to the end of the array effect speed in some way? Or imagine the order of all the triangles was random. I know the vertex order determines the face/normal but what about the triangle order?
Back info on why I am wonder is simply put I am thinking about creating a map using tiles but combining or adding the meshes together and so the order of the times might not line up cleanly.
Even if optimizing triangle order helps the gpu I don't want to spend time fixing my mesh if the gpu already optimizes incoming meshes since im sure its likly faster lol
You should run the multithreaded stuff as tasks so they can run across frames rather than running them in parallel in-place like that. Then your per-frame cost is just the cost of kicking off the tasks and just check for task completion to finish things up. A parallel-for for stuff like chunk loading or mesh creation isn't ideal. Maybe even have task dependencies for tasks so you could kick off a chunk load task and then a mesh creation task with a dependency on the chunk load task.
To add some additional compression, you could find repeated blocks and have a repeat type block that indicates the previous block is repeated N times before the next block is placed. this would drastically reduce the size of your chunks depending on how random the block distribution is. for chunks with lots of contiguous blocks you could save a ton of space. even if the distribution is highly variable, it can only have smaller or equal storage size than storing every block.
For runtime he could probably omit the position. Currently he uses 6 bits for each axis, if he stored all the blocks in a chunk and retrieved position using indexing, he would have to store even empty space, but each block would have 18 bits less (good for cache). Chunks that are all empty could be ommited.
Not sure why the normal thing, I would just check if there are neighbors directly. Maybe it's some smart trick I am not understanding, well...
Color can be palettized into just few bits, who needs all the 16 million of it anyway.
For storage on disk standard compression algorithms should do the trick.
wow that is a really good tip!
Man everyone is giving me so many good tips, so much to do ;P
Wow, nice!
veru nice!
Rayon is such a sick library
it's so cool!
New to graphics dev to. Not familiar with wgpu-rs. Tinkered with gl and ash crate with rust. Maybe a solution could be adding more render data in a single draw call using batch data calls as much as possible? My understanding is that each cpu call to gpu has performance hit on the cpu side, so focusing cpu multitasking solely on gathering data for a single batch gpu call might do the trick. Still learning. Not sure how multitasking works on the gpu side yet.
Also not sure if you have backface culling enabled. Significantly reduces the amount of drawing the gpu has to process. Interested in how well this works for ya!
Nevermind, just saw your following video.
Hell yeah he's back
I'm back baby!
Found my favourite youtuber. Can you give tutorial on rust and games?
Aye, thanks :)
I have some Rust learning resources linked in my discord group, here are some relevant:
[Articles/books]
Everyone should probably read this:
doc.rust-lang.org/book/
Many examples to learn from:
doc.rust-lang.org/stable/rust-by-example/
Full ascii rougelike game tutorial:
tomassedovic.github.io/roguelike-tutorial/
[TH-cam channels]
YouCodeThings: Fun but also informative:
th-cam.com/users/YouCodeThingsvideos
Doug Milford, has some good short tutorials on getting started with Rust:
th-cam.com/video/2KTG3OQPPJ4/w-d-xo.html
@@Tantandev alright man thanks 😀. Youre rlly underrated
Great vid, Tantan! You probably know this but writing out to the console will eat your frame rate. Switch off any debugging output when you frame test. But you probably knew that. Take care, sir!
Good point, I needed a way to track how many chunks where loaded whilst comparing to the Unity version.
I guess I could've printed out the moment it reached 2k chunks and compare to the Unity version now that I think about it haha.
Not the most scientific way to compare :)
@@Tantandev yeah. In unity you can use .net's stopwatch. I'm sure there's something similar in Rust.
Your videos make me want to pick up Rust so badly! It's a wonderful language. I just don't have the time!
Nice video! Is it possible to use mesh instancing on the GPU? Ie. upload the mesh for a single cube, then upload the (position, colour, ...) for in all the voxels you want in a buffer to the GPU, and let it do its thing in parallel? Not sure if that would be faster - it might decrease all the CPU work you need to do for generating the meshes though? I think this is what WebRender does for rendering web pages: you just have a single box that is instanced all around and modified using shaders.
This is the approach I took to rendering something similar in unity and it worked like a charm
I didn't even think of doing that, it sounds on theory that this method would be faster in some aspects.
But I'm exploring a lot of things that are new to me, multithreading for example. So some things I'll naturally learn over time what works best.
One thing I'm speculating on is, when I get physics into the engine, maybe I actually need mesh data,
not sure but that is one reason I'll stick with this method for now, many unknowns.
@@Tantandev if you haven't already I'd check out Bevy. If you really want to write your own engine that's cool, but you're also in a unique position to do some interesting publicity/possible contributions to the community as well
You might be able to compress the colour data by having presets much like the normal options. Then you just need to save an id for the preset and not the rgb value
for Unity, you could try unity dots or the jobs system with burst
Yay ur back!
I am alive and well :P took an uncommon time to edit the video
Epic
this might be a stupid quesiton but, have you tried just.. particle instancing all of the voxels on the gpu?
epic
hi, man. Can you consider how to store the huge scene voxel datas? not use noise to generate height.
You may want to use Simplex noise (Open Simplex noise since Simplex noise is patented until jan 2022) it have better performance for noise (especialy at higger dimension) and have less artifact than perlin noise
@15:24 you can make the code easier to read by writing (pos >> 18) & 63 instead of (pos & 16515072) >> 18
and also storing colors as floats? do you really have a 10 bit display where that would be of any use? use 8 bits per channel, at max
yay! not dead
Hurrah!
Neat.
Every time he says voxel I hear Vauxhall.
now I can't stop hearing it man lol
You can represent numbers like 0b11_1111u32 instead of 63u32 in Rust which might make it a *bit* more readable
I think using an enum to represent the various sections of the float and the normal directions would help make things readable as well.
@@Ian-sm9uv oooh good idea!
Man, finding tricks like this in Rust is so neat, I'm learning a lot just by the nature of the language and also you commenting ;P
Nice.
Not sure if you're already doing this or not, but you should only generate meshes for voxels that can be seen. You might even be able to precompute this data at generation / modification time.
Hello Tantan who is back
yaay!
senpai noticed me :D
:D I don't think I would've looked into data compression if it wasn't for your comment!
awesome
u awesome
@@Tantandev
Does it make to start learning Rust in 2021 ? I am thinking to give it a try. Can i use the code I write in Rust inside Unity game engine? Can you use Rust on web ( wasm? ).
You can use rust on the web but not on the unity game engine, you can use it in Godot tho
You technically can use Rust in Unity, but no one would probably recommend that, because you need to write so many bindings :P
WASM support for rust is growing a lot, I really like the game framework Macroquad, runs surprisingly well on the web.
Doesn't hurt to try it if you have some time, it just has a bumpy initial learning curve but after that it gets progressivly easier
Eyoo it's been a long time
Indeed!
Can we get the full version of the song?? It's littt 🔥🔥🔥
You should use domain rotated noise, its weird and has a directional bias without
This was probably an obvious step, but I just want to quickly ask if you compiled with "--release". Rust doesn't turn on optimizations by default and the layers of abstraction that it uses can slow down the code pretty significantly if not told to get inlined.
yes I did use --release :D
@@Tantandev 👍 I saw in the next episode that you were compiling in release mode. Just making sure since it can be an easy mistake to make when trying to speed things up.
I've gone done this road, multithreading, using a compute shader for the noise in Unity to get decent speed and the lowest level of graphics API available, literally setting the vertex attributes and whatnot. It was still about half the speed of Minecraft, which was disappointing. I explored using Vulkan and C++ (I'm good with C++) but I got bored and lost hope after getting stuck on how to do compute shaders with documentation that either assumes you are doing the most minimal project ever or assumes you are a rocket scientist. I'm just gonna wait until technology advances haha
7:36 There's no link to the Discord in the description :'(
Woops forgot to link it, here it is: discord.gg/9P8QSYf
Minecraft also has 16x16x16 chunks, they are just not inifinte on the Y-axis
cool
you became my favorit person ever in earth
MPSC - lol 😂
There is nothing, except synchronization via mutex yet discovered for multithreaded processing. The only question is: how often locks are acquired. This might be optimized via “optimistic locking”, but in case of rendering it might be a waste of time, since concurrently will be almost in 100% of cases and you’ll be forced to acquire a lock after additional checks anyway.
whitch program do you use to edit your videos? Davince?
it's called Sony Vegas pro 14, it costs money but I managed to pick it up from a cheap humble bundle some year ago
@4:18 my first though was: WTF, why is the block data stored in a hashmap? You loose all data locality advantages by using that.
Hi tan, great video. Have you tried compiling your rust projects to wasm? i wonder how they perform on web, if it's as hype as everyone talks about
@2:04 Please _don't_ use perlin noise. It is has many problems compared to the significantly faster and better simplex noise which is why Ken Perlin developed the replacement to his older perlin noise.
What ide do you use for rust?
Visual studio code
I also use vim plugin and rust analyzer
Here, wanting to see if switching from Unity to Rust just for performance, I really like C #, but come on, Rust is beautiful.
For now I think I'll stick with Unity
Using Rust definetly requires a lot more work. I wanna use it though, because I'm learning so much!
are you generating every side even if they're not visible?
I do not, my meshing algorithm checks for each face we are about to generate if an adjacent voxel is solid.
If it's solid, don't generate the face, otherwise do.
Here's the code that does this:
From line 72, process_voxel function.
github.com/TanTanDev/first_voxel_engine/blob/main/src/voxel_tools/mesh_builder.rs
2x faster seems bad, but you have to realise that there are expert mathematicians and programmers dedicated for optimising such a popular game engine with perhaps even inline assembly at some places! I expected you to be around 5-6x slower, so this is impressive, especially considering that Rust compiler isn't very smart right now and leaves some optimisation opportunities at the table..
I expected the same. Only half the speed is very impressive. Granted, it seems like the Unity version also does more (Looks like smarter generation), but for a first time? Very impressive
What if the real treasure was the game engines we make along the way?
I can't even code 1 Single Game lol
btw, do you know the guy "John Lin" which makes a incredible Voxel Game Engine, which is prob better than any Voxel Game Engine out there.
The Better GPU's get, the better Voxels games can get, lets see how they will look like in 10 Years xD
9:54: Okay, i have a beffy Computer compared to yours (GTX 1080, AMD Ryzen 7 3700x 8-Core 3,60GHZ)
My Computer isn't beffy compared to the 3000 Series Users tho lmao
Still cool thoo
Shank data )
I dont think there is ever a reason to sleep in any game/render loop. Try to remove sleeps and do anything else :D
Need new video every day :(
I can't seem to find your discord
woops, forgot to link it, it's here: discord.gg/9P8QSYf
how are you going to do networking now? XD
I'm not to worried about it, many libraries are available, and data serialization looks to be really awesome with Rust
@@Tantandev Oh! :D I'm looking forward to that episode! :D
You should start thinking more about how many assembly instructions are executing per line of your code and figure out how to reduce that as small as possible, sometimes is storage size, sometimes it is how you store, sometimes its the logic implemented, sometimes ita how the logic is implemented, naive implementations are always the very exact specific mentally conceptual implementation and they are always slow
Whattt my pc has an rtx 3050 and ryzen 5 5600x
stronk!
The normal looks like the coronavirus
haha woops, I was going to use arrows, but the Unity version I made the representation with doesn't have a cone 3d model :P
compute shaders on gpu
one representation of the voxel, all point to the same model
you then have only on-off data for each voxel
music videos please
everyone has been saying a lot of good things about rust, maybe i should try it?
Hola hablas español..?
You don't need to store normals: in fragment shader, use derivatives to calculate it from positions
Cool and, uh... you tried.
haha, gradually success shall be recieved!
@@Tantandev I don't doubt it!