ฝัง
- เผยแพร่เมื่อ 6 มี.ค. 2024
- What is CUDA? And how does parallel computing on the GPU enable developers to unlock the full potential of AI? Learn the basics of Nvidia CUDA programming in this quick tutorial.
Sponsor Disclaimer: I was not paid to make this video, but Nvidia did hook me up with an RTX4090
#programming #gpu #100secondsofcode
💬 Chat with Me on Discord
/ discord
🔗 Resources
CUDA nvda.ws/3SF2OCU
GTC nvda.ws/3uDuKzj
CPU vs GPU • CPU vs GPU vs TPU vs D...
🔖 Topics Covered
- How does CUDA work?
- CUDA basics tutorial in C++
- Who invented CUDA?
- Difference between CPU and GPU
- CUDA quickstart
- How deep neural networks compute in parallel
- AI programming concepts
- How does a GPU work? - วิทยาศาสตร์และเทคโนโลยี
Shoutout to Nvidia for hooking me up with an RTX4090 to run the code in this video, get the CUDA toolkit here nvda.ws/3SF2OCU
🥇
ZLUDA be like:
yes mom, I need a 4090 to run CUDA.
Damn you really put that rtx4090 through hell
So this is sponsored?
Little know fact, CUDA is actually so fast, that it can bend spacetime and make 100 seconds last 3 minutes and 12 seconds, truly revolutionary.
Underrated comment
He ran the seconds in parallel with Cuda.
Serious question, why are these videos never 100 seconds?
Because it's just the name of the series. A catchy title, really. I don't think anyone cares if they're exactly 100s.
To be fair, he explained it in 90 seconds, the rest is building an app.
I've done a bit of CUDA in uni for a class in parallelism. Let me tell you, writting truly parallel code is a pain in the ass. Ain't no way all those scientists are writing CUDA code, probably some Python abstraction that uses C++ and CUDA underneath.
Like PyTorch and Tensorflow
"model.to("cuda:0") is the only cuda you need to know unless you're developing new algorithms or doing something truly wacky
some (x) mostly (o)
yeh thats why pytorch and tensorflow exist, i have parallelism and HPC both this sem, writing openmp and MOI codes, truly a pita
There are a few geniuses who write libraries and then there are thousands of devs who build products out of them....
The #1 computing platform for vendor lock-in
And so is Apple.
Dell in the server space too
Cisco as well
yall forgetting about aws? 😂
No, Nvidia is an open computing platform dedicated to the development of democratized development and open standa--- Pfff 🤣🤣🤣 hahdahha!!
0:36 this just taught me matrix multiplication, thanks
fr, this 3 seconds animation was better in explaining it than most other explanaitions, and he didnt even spoke about it really.
The best thing that had ever happened to me was figuring our what matrices actually represent (a linear transformation) and I've been able to do matrix multiplication without any memorizing simply because its just intuitive now. Try this also because schooling has failed us
I think that is taken from @3blue1brown, @Fireship ??
@@alvinbontuyan8083 can you give a quick example on what you mean with this? I'm not that smart, thanks!
lmao fr, those 3 seconds are extremally helpful
Seeing "Hi Mom!" continue to be in your videos is such a beautiful thing. Hope you're holding up well
Yes, my eyes got wet when I saw that
Mom be like: I am proud of you, my son
Wait, where?
Where? What did you watch in this video then, lol. @@kamikaze9271
Here: 1:45, 2:53
@@kamikaze9271 2:52
0:45 IEEE 754 moment
When you use TFLOPs, is it single precision or double precision? Because I see double precision here.
Gives me PTSD from my master's thesis. Had to modify 4 flags in clang to get acceptable results. Took me a while to figure out.
@@cloudytheconqueror6180Single precision. Double precision is often much slower, though the rtx 4090 is just able to get into the teraflop range for f64
I did some CUDA programming assignments for my college Parallel Computing class.
That course was the second hardest CS course I've ever taken (The hardest one is Compilers but that's in its own league). Human brains really weren't designed to think in parallel.
Which college and course?
The teacher probably sucked like most academic teachers. If you had fireship it would be a hundred times easier
I hope that was graduate level, cause otherwise that is horrific
I would argue that people were not really "designed" to think in any specific way... neuroplasticity for the win... same way that most programmers can think of code. Practise makes perfect.
@@duckbuster1572 It's common for it to be a course in your last year of undergrad... I dont see why it would be horrific.
1:09 still day zero of not mentioning AI
AI is definitely worth mentioning.
@@2099EKPlease, can we just don't? Physics models (for example) are much more interesting (in my opinion) than curve fitting on steroids. (Just a matter of avoiding a cliche and showing a greater range of GPU computing applications)
Why, fitting so much complex curves that reflect reality is indeed worth mentioning @@upolpi3171
It’s more like zero minutes 😂
@@anon8510You're literally on a technology channel, you Twitter drone.
Bruh, are you my FBI agent? I just looked CUDA up a few hours ago.
Yeah man, he monitored your web traffic, saw that you wanted to learn about cuda, and then made this video as fast as he could since he knew you would watch it.
Now I'm scared about tomorrow's video
I was thinking to learn about CUDA. He is a mind reader
That's classified.
literally doing an homeword in cuda rn
ngl, I'm really loving how often these videos are being uploaded. It's often, but not so often that I feel overwhelmed and just spaced out enough that I feel a little excited when a new one comes out!
wait until he drops some existential crisis type content lol
Wrote Cuda at university .. getting the indices, blocks etc right ... that was fun (also since thread count depends on the actual GPU model). For the final project, we were allowed to use libraries such as thrust which made my life a ton easier by abstracting away most of the fun stuff.
thread count is not depended on GPU model (max 1024 threads per block), total block size and number of cores are depended on number of SMs and cuda computability.
Sounds like the "fun" was actually "fun boilerplate but it's still just boilerplate". Correct? Or... are you being _purely_ sarcastic?
@@BrahvimBoth actually. It was fun in the beginning, but with more complex projects/tasks it became harder to understand how to use it correctly (espeically kernel launch configs with the dimensions, etc). Mabye, with more experience, it would be easier for me today than it was at that time.
But don't get me wrong, they also showed how to do the same thing with OpenCl and the amount of boilerplate code for this to run was way more than with Cuda.
And when they allowed using thrust for the final project, most of the boilerplate code was gone because thrust abstracts that away. It was more fun to work with an API that offers host and device vectors and a standard library for common tasks. But, thrust also abstracts away the launch configurations for kernels etc, so you loose control (which was fine for me because I struggelded with the more advanced concepts). But I guess you will loose some speed/memeory effeciency like with all abstractions.
@@KoaIa200you are right. I am sorry. The more advanced kernel launch configs with block size etc was quite hard for me and I haven't used Cuda in years now. But I remeber struggeling with the concepts after the initial easy tasks
@@BrahvimNo, it actually was fun, but it is also hard. And if you compare to OpenCL it is actually much much less boilerplate code.
In the beginning, exercise were quite easy but with more complex tasks, it became much harder. For the final project we were allowed to just thrust which is a library that makes things much easier. E.g. it provides host and device vectors and it also handles all boilerplate stuff. However, you will loose control because it is a abstraction and probably some speed. But today, if I would need to do Cuda again it would be with thrust (at least in the beginning)
Thanks! I was having this discussing with my coworkers the other day about what separates a gpu from a cpu and this was an excellent explanation!
1:39 Complier :D
no, complier
Gotcha moment😀
Marcomplier
Whoa, my universes are operating in parallel. I just learned about CUDA this morning for the first time, and here's a new fireship video about it.
Man, you are a genius. I wrote my masters thesis on CUDA and there's no way how I would be able to explain this in 100 seconds.
Respect! 🎉
Can I read your master's thesis?
same , LMK when you get it@@klekaelly
Could you do it in 192 seconds??
Really, I thought Opencl will do this just fine. Funny thing is
ALL GPU's are designed to be parallel computers and
AMD in actually more massively parallel than Ngreedia.
He didn't describe anything that is just cuda specific, did you really not get that when writing your thesis?
@klekaelly thank you, but it was on cuda version 1.0, which is really outdated from both software and hardware perspectives. Furthermore it is not in English. But I really appreciate your interest!
Not using or planning to use CUDA but man did this just help me make sense of some terms I see being thrown around! Awesome!
Best 100 seconds I've had in a bunch of seconds. Thanks!
Who asked you?
@@etrestre9403 me?
@@etrestre9403 Not allowed to speak their mind?
@@Mkrabs yeah I was just wondering who asked them
@@etrestre9403 sorry for your mental illness
I'll admit, I tear up a little every time I see the "Hi Mom" in your vids.
Nice to see a video touching C++'s ecosystem for a change. Now make one about SYCL, so even people who don't find free RTX 4090 cards in their mailbox can get into high performance parallel computing using modern ISO C++ instead of custom CUDA syntax.
yeah, Nvidia dominates in parallel computing because software engineers only know CUDA.
@@vladislavakm386 You got that backwards, but ok.
Funny, I had to install NVIDIA CUDA for a thing I'm doing and forgot what CUDA does, searched it, and found this video that was just posted an hour ago! WHAT TIMING!!!
You just explained parallel computing in 100s better than my lecturer did in more than 100 days🔥
Yet misses the fact this is NOT cuda specific.
Or your lecturer set you up well to follow this very basic, high speed summary. Like a reader of the LOtR series can see meaning in the film series' long, dreary shots.
Yo I just wanted to say thank you for making this kind of stuff so interesting and digestible. You make these extremely complex, time intensive languages, apis, tools, etc., and make them incredibly approachable.
Love your content. Cheers.
Aside from this very informative video ... Heartwarming that you put in that "Hi mom"-message.
Probably one of the most concise videos on this topic.
Surprised that it took this long to get a CUDA in 100 seconds. 😆
I did not expect this...
I'm calling Miguel.
Great presentation of the topic of CUDA architecture and Nvidia GPUs in such a compact and fast form. As always, brilliant video!
Many thanks for every video on your channel, you doing very big and cool work
Bro, Can you do more Hardware videos, just like this
Hardware videos 💀
Next please do OpenCL in 100 Seconds, seriously
He didn't get paid for that.
id love to see that
Savage comment 😁@@noanyobiseniss7462
OpenCL for the win! Same performance as CUDA, yet runs on literally every GPU from Nvidia, AMD and Intel.
Awesome video! Thank you for the heads up in the conference!
As a programmer I absolutely love your series on programming languages and tools ! Cannot be more clear, and full of knowledge. Thank you. This also refresh common knowledge such as the C video!
The SDK has already gotten alot more convenient in the last 5-6 years. Memory used to require the SDK to manually copy back and forth. From what I remember the manual copying is still available, but in my DLI course when I was trying it out, having it be auto managed is slower than manually moving it all into memory first and running the operation. Using it in managed improves the developer experience signficantly but on each access if the memory block hasn't been copied I believe the managed system will still need to move it over on demand. To pass my CUDA DLI exam to meet the passing criteria, one of the steps I opted to manually copy. One can only dream of the day we have unified memory architectures then we don't have to deal with the copies.
Yeah, you can probably keep on dreaming about that. Memory management is the primary contradiction that you must solve if you want your CUDA program to go fast. Either you need to get all of the data in the register file / shared memory or you have Too Much Data and have to do horrible things and maybe even have some of that data out of core and it will go much slower than it could. There's no cache coherence protocol so if you need it you have to move things around manually and do some synchronization. Fun stuff.
You didn't explain what CUDA does you explained what a GPU does...
CUDA just has special optimizations over normal GPU parallels.
Your example will work fine on every GPU and doesn't require CUDA to be parallel.
All GPUs calculate the pixels using multi threading and multiple cores.
I mean he explained how to get started with it and clarified how it's different to programming on the CPU. Also I'm pretty sure the > syntax is specific to CUDA so you wouldn't be able to just run this anywhere. And GPUs in graphics are usually just dealing with essentially a 2D array of pixels rather than 3D like here.
@@Aoredon AMD's ROCm also uses the > syntax and I kinda agree with OP, this would've been good if it was titled "GPUs in 100 seconds" but as things stand it's hardly anything CUDA-specific
This is a summary channel, not overly detailed.
Correct and well said!
The extension of the file was .cu tho
That was a great summary! Thank you!!!
Thanks for the video! Easy to understand and that helped me a lot to get a basic understanding of CUDA
Sometimes I regret my career choices
Im using cuda for fluid simulation, it’s a real game changer in terms of speed
I loved the animations and thr explanation..i just finished a cuda course for my masters so it was minx blowing to see a whole weeks worth of lectures effortlessly compressed in ... 100 seconss
Can I see the course?
YES! I was waiting for this one
Hey, that was nice! I use both CUDA and OpenACC EXTENSIVELY to build CFD applications, and the performance on gpus is really fantastic... when done well xD strongly recommend against managed memory for complex production codes, if only for the fact that it seems to disable device/device DMA comms when using MPI. For anyone thinking about porting to GPUs, recommend to not half-arse it, and just make all data available to devices. Host/device exchanges can be brutally costly, and will likely eat up all your gains. Finally, it works with C and Fortran as well, for anyone curious about it :) Fireship, be nice to see a beyond 100 seconds of this, covering OpenACC and offloaded OpenMP as well😊
Which CFD software has CUDA acceleration? Just Ansys Fluent right now right?
@adialwaysup8184 not really, we performed some testing on A100s and H100s and offloaded omp was WAY slower. Sure it's portable, but acc is still getting love. It's also syntatically easier and cleaner in my opinion.
@jaiveersingh5538 take a look at research code. Nek5000 uses CUDA, and as well as NekRS if I remember well. Our own code started as CUDA Fortran but we eventually moved to OpenACC. Easier to use and explain to other users. Quite a few libraries behind research soft also uses CUDA, or even OpenCL. For matrix free SEM methods, CUDA might be a bit hard to implement, but it's as fast as it gets.
@@lucasgasparino6141 For us, omp was performing 2% slower than acc and 6-8% slower than cuda. Though, the performance was much worse on clang than nvhpc
@@lucasgasparino6141 In my experience, currently, there's a major discrepancy in how well a compiler optimizes code for accelerators. The is doubly important when it comes to nvidia, since the nvptx backend is far from perfect. But if the same tests are done on nvidia say with nvhpc. I found an overall 2-3% gap between openmp and openacc. I do agree with your second point, openacc is much cleaner to write and integrates well, but at that point you're backing up in a corner with nvidia's hardware. Openacc might be an open standard, but no one except nvidia gives it a serious consideration. If you're going all in with nvidia anyway, why bother with openacc and just move to cuda.
But can CUDA center a div?
💀💀💀
Yes when you center a div in CSS, the browser uses your GPU for rendering the pages on your browser
center div
exit vim
I use arch btw
hmm yes, very original "I've been programming for two weeks" joke
Impressive explanation of how we can harness the power of our GPU using Nvidia's CUDA for more than just gaming. The practical demonstration expounded the potential of parallel computing considerably.
Cuda is Awesome! I did one of my thesis on parallel processing in 2016 using CUDA for a super fast blood cells segmentation. Then used CUDA for mining crypto on the GPU.
Use me as the button "I understood NOTHING"
The title: "Nvidia CUDA in 100 Seconds"
The duration: 3:12
You must be new here
Just as I needed. Simple and quick introduction for it.
I would love to see Elm in 100 seconds soon! It definitely deserves more love.
Interesting little factoid: if you are doing parallel cuda programming, and have to compute on a subset of a large block of memory, often it's faster to operate on the whole block and simply ignore the additional data, without checking for actual boundaries. If conditions kill performance in cuda kernels, at the point that often it pays off to just compute garbage and discard it at the end, rather than prevent it from computing it.
If conditions are usually translated to compute discard.
But they give false appearances, and also if the if condition is difficult to compute that adds to the runtime cost.
warp divergence does not matter if the other threads are doing nothing in the first place... just dont have if else and you are fine.
Better add those bounds checks, don't want to crash with access violations...
0:46 truly a masterpiece from our beloved GPU
@@starsandnightvision not a native speaker but ty for pointing it out
Having used the CUDA Toolkit for implementing LSTMs and CNNs for Computer Vision and Sentiment Analysis projects using Tensorflow GPU and ScikitLearn libraries of Python which utilized my laptop's NVIDIA GPU, the process of writing raw CUDA Kernels in C++ is somewhat new for me and seems fascinating.
Great video, Fireship. However, it's worth noting that writing performant and optimized raw CUDA code is very difficult and not practical. Usually, you aren't writing your own CUDA code but rather using NVIDIA's highly optimized CUDA libraries, such as cuBLAS, cuFFT, and cuDNN. These libraries implement common primitives such as matrix multiplication, neural net operations, etc
Yes, but where is the fun in that
@@yogsothoth00 If you think that is fun you would probably get hired by Nvidia to write more libraries for them
He did a 100 seconds video on PyTorch. So, he probably expand on this too. This video is specifically about CUDA.
Yes , if you can install them and find the right version
So basically: "when writing code one uses libraries." Thank you Capt. Obvious.
Can't wait to install ZLUDA on my linux pc!
Great Video! A ROCM video would awesome too. Could help me explain my suffering to friends on using CUDA native apps in a crappy docker container for less performance vs native Nvidia.
Hello sir,
Today is my High school IT exam.
I thank you for giving so much knowledge in these years.
Thank you sir
Data Scientists don’t use CUDA, they use Python abstractions like Tensorflow or Torch which parallelize their work using CUDA assuming an NVIDIA GPU is available.
"Data scientists don't use CUDA, they use CUDA" :D
The guy above you doesnt knows what the word abstraction means lmao@@el_teodoro
@@el_teodoroor rocm? or vulkan? or metal?
A: how complex the CUDA is ?
B: Even the Fireship doesnt make sense
Honestly, it's a rather low-level API, so it CAN get excessively complicated. That being said, you'd mostly use the basics of CUDA, and complexity would come from making the algorithm you're trying to implement parallel itself. Of course, the real magic is that you can optimize the SHIT out of it, I.e. overengineer the kernel 😅 but yeah, trust me when I say he covers only the intro bits about CUDA, this thing is a rabbit hole.
This is great! Thank you 👏
Love how this video came out 20 minutes after I did intensive google search about CUDA :D
Just recently seen the news abour Nvidia banning the use of translation layers on CUDA software like ZLUDA for AMD. That video's right on time.
Which is what he should be making a video on but you don't get free 4090's for that content.
@@noanyobiseniss7462 NVIDIA doesn't wanna let go that sweet sweet monopoly type proprietary stuff.
@@demonfedor3748 Pretty anti competitive company that bleeds users dry. I have no clue why its userbase is so filled with gaslit fanbois. I guess it comes down to the misery likes company mantra.
@@noanyobiseniss7462 Every big company wants to get as much profit as the next guy. NVIDIA does it through proprietary stuff, AMD does it by open standarts to claim the moral high ground. Pros and cons to each approach but the goal remains the same. NVIDIA has a lot of fans because they innovate a lot and are trailbrazers in multiple areas. Real time hardware ray tracing, DLSS, G-SYNC, frame generation, GPGPU aka CUDA, OPtiX, just to name a few. I know most of this stuff is proprietary and/or hardware locked but it's still innovation. I don't mean that AMD doesn't innovate. Mantle that subsequently led to Vulkan was a big deal, chiplet GPU and CPU design, 3D-Vcache on CPUs and GPUs, SAM. There's no clear winner, however NVIDIA is currently performance king. Intel wants in the game for over 15 years but they got big shoes to fill. Was a big blow when Larrabee failed.
@@noanyobiseniss7462 Every big company wants to get as much profit as the next guy. NVIDIA does it through proprietary stuff, AMD does it by open standarts to claim the moral high ground. Pros and cons to each approach but the goal remains the same. NVIDIA has a lot of fans because they innovate a lot and are trailbrazers in multiple areas. Real time hardware ray tracing, DLSS, G-SYNC, frame generation, GPGPU aka CUDA, OPtiX, just to name a few. I know most of this stuff is proprietary and/or hardware locked but it's still innovation. I don't mean that AMD doesn't innovate. Mantle that subsequently led to Vulkan was a big deal, chiplet GPU and CPU design, 3D-Vcache on CPUs and GPUs, SAM. There's no clear winner, however NVIDIA is currently performance king. Intel wants in the game for over 15 years but they got big shoes to fill. Was a big blow when Larrabee failed.
Opencl next!
I doubt AMD will pay him a 7900XTX to do it.
Valuable video!
awesome animations on the video man
Getting CUDA to run on your Windows machine is one of the greatest problems of modern computer science.
Edit: "getting CUDA-related libraries in a Python environment to correctly run neural networks"
lol, holy wow this really is a noob channel
Getting it to run the "official" way, from Visual Studio, is not much of a problem. Now, getting CUDA-related libraries in a Python environment to correctly run neural networks - THAT's a challenge. Especially with how much of a bother Conda is.
Lots of ML stuff doesn't have good support on windows. Probably good idea just to run an Ubuntu VM if you plan to do much locally.
I can finally build my own LLM now!
Nice! I use Thrust to abstract a bit on those cuda and apply generic programming. Maybe do a video on openCL? 😊
Thanks for the video!
hey, that's more than 100 seconds
I literally just finished an exam on cuda wtf
What course do you offer
@@acestandard6315 where do u study?
@@SalomDunyoIT Nunya University
I didn't understand MOST of it, but still loved it , thanks!
Bro, your way to teach, much faster than my mind..
0:25 what is the game name
Leaving a dot here for a captain to show up.
I also would like to know this. Anyone?
Nothing worse than buying an AMD card and being locked out of anything AI (and these days it's a LOT of things). Never again.
Your not too bright are you.
Google ZLUDA my friend ...
This channel should go down the history is the greatest work done by humanity. Absolutely legendary introductions & quality level
Thank you for the video!
Cuda is closed source and therefor a non starter for anyone that believes in freedom standards.
I wouldn't recommend nvidia to anyone, their CEO is crazy!!
And the alternative is what?
Hospitals, the garbage collection, fire departments, etc aren't open source either, but you're kinda forced to use them.
Nvidia has got us all by the balls.
Your balls are firmly placed in Nvidia's hands.
God speed your efforts to come up with a freedom alternative.
@@MrCmon113 the alternatives exist! In case of CUDA, OpenCL is the alternative that works on all GPUs. And in case of gaming, AMD cards preform very well (and their drivers are open source)
Finally 🎉🎉🎉
I challenge you to do CUDA matrix multiplication using C
dam bro i have my linear algebra exam next week and you just taught me how to matrix multiply at 0:36 (teacher took 3 classes to explain)
This is just mind-blowing 😮
Would love to see some more videos on parallel computing, with more explanation of this kind of code. Maybe a more in-depth video on Beyond Fireship?
This is a very slick advert for Nvidia 😅 didn't realize it was an ad until the end.
0:36 this matrix multiplication animation is really REALLY good!!!!!
Great timing! Just got a new green GPU to mess around with and this'll help.
I worked on a porject using CUDA enviornment, this brought some memory like the copying from host to device and vice versa. I'm sure I'll be working on it again in the future.
Can I mention this video as part of my channel intro? I use NVIDIA CUDA to re-render and upscale all my video clips for TH-cam nowadays!! You give a really good explanation of how it all works.
The video editing must take hours for each upload
Well done brother
Wow what great timing to mention ZLUDA
That was a pretty entertaining ad.
You should do a video on SHMT (simultaneous and heterogeneous multithreading)
Just subscribed for more "in 100 seconds" videos 👍🏻
Holy crap, I didn't realize it was that simple.
1:43 I love how Fireship knows his mother is watching, and threw in a quick shoutout 😂
Game Developers Conference (GDC) is also that week.
Thanks, Jeff!
Very informational on CUDA and NVDIA ,👌👌👌Do you own research but dont' miss out on AI & NVIDIA its touching all companies & all sectors.
I would really enjoy a follow-up to this, maybe on the other channel to discuss ROCM.
great visualization and videos indeed!
Dude found a way to expense his Nvidia GPU, by writing ten lines of code. Well done ❤
Thanks, sir! 🙏
500K views in 1 day. thats some serious growth right there
I honestly recommend the GTC if you're into graphics or just interesting curiosities