For a newbie in the field of parallel computing and gpu accelerated computing like myself, thanks for taking the time to lay a strong foundation on how parallel computing works. It helped me correct a wrong assumption I had about GPUs.
you probably dont care but does anybody know a way to log back into an Instagram account..? I stupidly forgot my password. I would love any tricks you can give me.
@Trenton Lian thanks for your reply. I got to the site through google and im waiting for the hacking stuff now. Takes quite some time so I will reply here later with my results.
This was a great video to me, I have very limited C++ experience and was looking for an explanation of CUDA. Another video like this could easily have been 70-80% over my head. This one was only about 15% whoosh. And now I actually find C++ interesting again!
Yes. I wrote an articel about it. towardsdatascience.com/fibonacci-linear-recurrences-and-eigendecomposition-in-python-54a909990c7c?source=friends_link&sk=6dbc6bcaf2b9551e108da037963aea33
Nice talk, but your history is wrong -- the first graphics-on-a-chip was done by Silicon Graphics, Inc in about 1985, with the "geometry engine" chip. SGI dominated the industry in the late '80s and the '90s, much like Apple and Google do now. NVidia was a later spin-off in the late '90s formed mostly by people from SGI.
Evans & Sutherland was doing hardware-accelerated, rasterized graphics in the '60s and '70s. SGI did pioneer the bringing of the technology to the general workstation markets in the '80s, though.
@@NUCLEARARMAMENT "Raster graphics" on a chip may have been done earlier than SGI, but not fully 3D polygonal rasterization, which is a the basis of modern 3D computer graphics.
@@paulhansen5053 Also, the CT5 simulator from 1981 may not count as being from the '70s or '60s, but from what I understand, the CT5 was capable of realtime, rasterized, 3D polygonal rendering and was $20 million at the time. It used gouraud shading, if memory serves. There were several other CT (continuous tone) simulators developed by E&S in the '70s that did something similar or of much lower capability than the CT5 of '81. There was also the Digistar planeteriums that date back to the early '80s, and the Picture System goes back to at least the early '80s. Might be vector or raster, not entirely sure myself, though.
I was just starting to learn OpenGL (with the Superbible) to get a feel for how gpus work. So far it's been very fun. I had always assumed it was waaay more complicated to make anything work in a gpu, but it doesn't seem to be the case.
Lmao, I thought I had to write some sort of gpu assembly. But they don't even publish it, in practice, the gpu driver is what exposes what you can do with it... by what I read with a quick google search.
Siraj, thanks for taking time to create these videos. It is unfortunate that people view your videos and then feel inspired to complain about a free gift. Folks could just keep it moving or add helpful insights.
You can also skip the middle man and create directly a python extension which uses CUDA. In my submission you can find a very minimal implementation of it: github.com/tterava/Mandelbrot It has all the necessary functions to allow your your C (or CUDA) code to be called from python code directly. All you need to do is to use the nvcc compiler to build the extension and python knows how to use it automatically.
In reference to 8:00, what happens if you specify a number of GPU threads/cores that is greater than that of the GPU itself? Is there a GET function that retrieves these values so as to prevent exceptions?
There must be some : min(no_of_cores, input_cores) inbuilt , so if you pass the input cores more than the no of cores , it'll take the no of cores else it'll take your input
Thread and blocks size they are not infinite. and of which of which I know it is more profitable to perform in a loop than to create large thread and blocks meshes.
Hey Siraj, I love watching your videos because of the way you tell the story. Great graphics mate. Love the reference to rocket man too... lol keep up the good work.
Here's an idea. You can speed up Fibonacci number calculation by using the closed form formula. You distribute the multiplication of the powers over multiple threads. To assure accuracy, instead of using floats, you can write a class to represent numbers of the form a + b * sqrt(5), where a and b are rational numbers.
You can calculate the nth fibonacci number without calculating the previous fibonacci numbers with its closed form. ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-042j-mathematics-for-computer-science-fall-2005/readings/ln11.pdf (page 9)
Do we need the learn Cuda if we plan on using Tensorflow? I know Cuda allows us to process on many cores, but I heard something along the lines that Tensorflow already has this idea built in. Is this true, or do we need to pair Cuda and Tensorflow in our deep learning programs?
Hi siraj, can you guide me what to install inorder to execute CUDA code? I'm scratching my head since a long time for the successful execution of the program on windows using Visual Studio.
I spend several hours yesterday trying to install tensorflow-GPU in Anaconda so I could use my GPU to speed up training of our neural network. It's an incredible pain in the ass as there are 7565 dependencies that all need to be installed in specific way and you're forced to register for the Nvidia Developer Spam Program. I didn't manage to do it in the end. Anyone have any tip for painless installing of tensorflow-GPU? I was really suprised and annoyed by how difficult it was.
I understand that this video is meant to be an introduction but can you make a video or two where you go into more detail and do some coding of this? I like your format of short videos with memes as introduction and longer videos for the coding but I can’t find the longer coding video for using CUDA or GPUs
I need a help please so urgent , I want to have all the prediction of each class my softmax gives me only one result of one class , is there an instruction softmax multilabel
What machine do you use for your GPU programming? Can you tell us how to port what you are doing from your on-premises computer to a Cloud provider? And which Cloud providers support this? That would be interesting to know.
Easier way to start is a cloud instance, eg. on AWS p2.xlarge (aws.amazon.com/ec2/instance-types/p2/). If you do a lot of computation and want to save expenses for cloud or train faster, you can build you own computer (eg. timdettmers.com/2015/03/09/deep-learning-hardware-guide/).
hi everybody. I need a help I have a python code which include CUDA but I'm beginner on python use, when I run it I get error, (Torch not compiled with CUDA enabled) may someone help me to solve this issue? I'm doing my final research. I look forward to hearing from you
you could also use faster asymptotic algorithm variant, if you want only to compute a single term of the sequence. It suffices to use a square and multiply on a certain matrix. It doesn’t make it parallel though :(
Could you help me, I need the best GPU for science purposes not more expensive than 2500$ (it should work good with double): could you tell me on what characteristics should I look and what is the best variant?
Mac doesn't support CUDA I think?, Maybe if you use windows 10 (Or even better Ununtu) on the Macbook through bootcamp with a 3.0 minimum level GPU card through thunderbolt might work
Amund Lindberg i have intel gpu but some are saying that cuda will only work on nividia gpu’s ..gpu is the reason of that error? plzz reply thanks in advance🙂
For those who have not tried it out yet and have an AMD chip don't bother...it gives segmentation fault and doesn't run at all and a/c to otoy company they claim that they can do it it's not released yet so sadly go back and do it for open cl and Thanks Siraj, you have great videos. EDIT: Don't even install that nvidia-cuda-toolkit that it will recommend you if you have amd gpu... it will ruin your graphic drivers and you will be unable to boot into the system
There are some performance trade-offs when using Unified Virtual Memory (UVM). Memory pages are migrated between the host and the GPU on-demand (plus some nifty prefetching heuristics) but frequent page faults often stall the system. I suggest using explicit memcpy semantics between the host and the GPU.
Thank you Daniel, that was the kind of trade-off explanation I was looking for. Kevin, I'd like to read more about that. New GPUs seem to be _meant_ for that kind of stuff, don't they? I'd be surprised if it would really kill your GPU.
I hate you for not mentioning 3Dfx! Voodoo graphics. And Voodoo 2. And around Voodoo 3 Nvidia bought the company. BUT! 3Dfx started everything years before Nvidia. 1997! How could you?! This video is way too useful to thumb it down though, good stuff.
I dislike people making video and not do all research. Forgetting about 3Dfx and Voodoo... I know name GPU is from GeForce 2 (not 1, there are still some thing calculated on CPU on GeForce 1), but in 3D graphics first card was Voodoo, and open Pandora box. Is sad they was one innovation company, and not move forward, if they do we have 3 major GPU company not 2.
@@existentialchild698 yeah know, but is a shame having to buy one when you already have one in your sistem. anyways i just make it work. compiled tensorflow with it and now i have enable gpu in my mid 2012 macbook pro
@@MygenteTV Okay. There's also a service made by Google called "Google Colab" that allows you to run your python code on an Nvidia Tesla K80. (That's a really good GPU that would otherwise cost you over $2000 depending on where you buy it).
Typing 1
can you explain please? I'm not too familiar with c++
@@1kounter '
people who do this have 99% chance of using the ternary operator whenever possible
@@jeffreylebowski4927 So it's not exactly a million elements? This is misleading.
@@pcrizz Yeah i agree, there was no need to make things more complicated and then not even be precise about it. - Its bad to do this as a teacher.
That was very well explained. I have only have taken one course, and you made it clearer than my professor or fellow students ever did.
Thanks for actually showing the speed increase with more threads, haven't seen anyone do this.
LOL. Loved the graphic at 6:23! Brought tears to my eyes.
For a newbie in the field of parallel computing and gpu accelerated computing like myself, thanks for taking the time to lay a strong foundation on how parallel computing works. It helped me correct a wrong assumption I had about GPUs.
you probably dont care but does anybody know a way to log back into an Instagram account..?
I stupidly forgot my password. I would love any tricks you can give me.
@Trenton Lian thanks for your reply. I got to the site through google and im waiting for the hacking stuff now.
Takes quite some time so I will reply here later with my results.
@Trenton Lian It did the trick and I now got access to my account again. Im so happy:D
Thanks so much you saved my account :D
@Liam Christian You are welcome =)
This was by far one of the most enlightening videos you have put up on your channel. Thanks and keep up the good work!!
"cuda" in polish means miracles (we pronounce it little diffrent though) so it's kind of funny because gpu seems like little miracle
"Tsoo-dah", no doubt!
This was a great video to me, I have very limited C++ experience and was looking for an explanation of CUDA. Another video like this could easily have been 70-80% over my head. This one was only about 15% whoosh. And now I actually find C++ interesting again!
OK those animations + your relevant narration is by far the best combination for learning. Loved it!
3:08 actually the n'th fibonacci number can be found in O(log n) using matrix exponentiation.
Yes. I wrote an articel about it.
towardsdatascience.com/fibonacci-linear-recurrences-and-eigendecomposition-in-python-54a909990c7c?source=friends_link&sk=6dbc6bcaf2b9551e108da037963aea33
And you actually do not need matrices, only the diagonal values from eigendecomposition.
It can be found in O(1). As far as I remember the formula is derived using LDU decomposition or Diagonalising a matrix, for matrix exponentiation.
Thank you for this video
The Knowledge of Siraj knows no bounds.
Nice talk, but your history is wrong -- the first graphics-on-a-chip was done by Silicon Graphics, Inc in about 1985, with the "geometry engine" chip. SGI dominated the industry in the late '80s and the '90s, much like Apple and Google do now. NVidia was a later spin-off in the late '90s formed mostly by people from SGI.
Evans & Sutherland was doing hardware-accelerated, rasterized graphics in the '60s and '70s. SGI did pioneer the bringing of the technology to the general workstation markets in the '80s, though.
@@NUCLEARARMAMENT I kinda don't think so, actually don't have time to check it out. Pretty sure that E&S in '60s and '70s was all vector graphics
@@NUCLEARARMAMENT "Raster graphics" on a chip may have been done earlier than SGI, but not fully 3D polygonal rasterization, which is a the basis of modern 3D computer graphics.
@@paulhansen5053 Also, the CT5 simulator from 1981 may not count as being from the '70s or '60s, but from what I understand, the CT5 was capable of realtime, rasterized, 3D polygonal rendering and was $20 million at the time. It used gouraud shading, if memory serves. There were several other CT (continuous tone) simulators developed by E&S in the '70s that did something similar or of much lower capability than the CT5 of '81. There was also the Digistar planeteriums that date back to the early '80s, and the Picture System goes back to at least the early '80s. Might be vector or raster, not entirely sure myself, though.
Excellent presentation! Thanks for sharing this vital knowledge!
Video quality is awesome ! Great job
Love your videos. Please don't stop!
I was just starting to learn OpenGL (with the Superbible) to get a feel for how gpus work. So far it's been very fun. I had always assumed it was waaay more complicated to make anything work in a gpu, but it doesn't seem to be the case.
Lmao, I thought I had to write some sort of gpu assembly. But they don't even publish it, in practice, the gpu driver is what exposes what you can do with it... by what I read with a quick google search.
CUDAs to you, for covering... CUDA!
I have no idea what kind of videos i am watching ... but i sure will learn
Is 'Introduction to CUDA GPU Programming' a new series that is going to come up ?
Swanand Kulkarni I hope so
i wasn't planning on it but this vid seems to be doing well so may consider
Oh Siraj, your tutorials are incredible and inspiring....
Thank you so much for this video. It has helped me massively to prepare for my computer science exam.
Good stuff Siraj!
What's the link for the GitHub at the end of the video?
OK, found it: github.com/alberduris/SirajsCodingChallenges/tree/master/Stock%20Market%20Prediction
Thanks! ^^
thanks bro!
Awesome!
just updated, can't believe i forgot it, won't happen again
these memes increase production quality by 10x
Did you check the results of the addition? I have run the code and the values of 'y' before and after the add function remained ... what is wrong?
Siraj, thanks for taking time to create these videos. It is unfortunate that people view your videos and then feel inspired to complain about a free gift. Folks could just keep it moving or add helpful insights.
Can this be doe with Python ?
Victor Gallagher yes you can usw pygpu or my recommendation numba
Cool! Thanks for replying, I only starting learning Python 3 months ago and just discovered numba this week. Regards
yes mathema.tician.de/software/pycuda/
You can also skip the middle man and create directly a python extension which uses CUDA.
In my submission you can find a very minimal implementation of it:
github.com/tterava/Mandelbrot
It has all the necessary functions to allow your your C (or CUDA) code to be called from python code directly. All you need to do is to use the nvcc compiler to build the extension and python knows how to use it automatically.
as stated you can use numba. just decorate your function with "vectorize"
Love your style... Steel the same Siraj, subject very interesting!!
In reference to 8:00, what happens if you specify a number of GPU threads/cores that is greater than that of the GPU itself? Is there a GET function that retrieves these values so as to prevent exceptions?
There must be some : min(no_of_cores, input_cores) inbuilt , so if you pass the input cores more than the no of cores , it'll take the no of cores else it'll take your input
Another Awesome Video, Loved it
thanks!
Very helpful video. Thank you :) !
You don't need a "for loop". Each thread computes sum of a[blockIdx.x * blockDim.x + threadIdx.x] + b[blockIdx.x * blockDim.x + threadIdx.x].
Thread and blocks size they are not infinite. and of which of which I know it is more profitable to perform in a loop than to create large thread and blocks meshes.
Freaking
Awesome.
I regret I have but one up-thumb to give.
Say what you want about Siraj's bad choices recently, the dude is a master teacher.
Hey Siraj, I love watching your videos because of the way you tell the story. Great graphics mate. Love the reference to rocket man too... lol keep up the good work.
Your videos are awesome , thanks a lot for this quality content :)
your my new favorite youtuber. LOLs.
thanks for the clear explaination
Love your videos bro! Time to put down that redbull though lol just kidding happy holidays!
This was really, really helpful to me. Thank you.
I saw this in my recommended. The most I knew about gpu was that u need that for the cool games. Now imagine me watching this video.
I didn't know TES V was available for the NES platform. I wonder if it has mod support.
Just got a Jetson Nano - looking forward to learning more!!
Very useful video, thanks a lot, Go ahead
Here's an idea. You can speed up Fibonacci number calculation by using the closed form formula. You distribute the multiplication of the powers over multiple threads. To assure accuracy, instead of using floats, you can write a class to represent numbers of the form a + b * sqrt(5), where a and b are rational numbers.
can someone tell me how to execute a python code using gpu , through cmd wihtout using any virtual environment
So If I was to make my own gpu would I have to learn C++ or something else?
You can calculate the nth fibonacci number without calculating the previous fibonacci numbers with its closed form.
ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-042j-mathematics-for-computer-science-fall-2005/readings/ln11.pdf (page 9)
Nice read, thanks.
siraj is noob
You beat me to it...
Fib(n) = ( 1.6180339..^n - (-0.6180339..)^n ) / 2.236067977..
it's only an approximation
see the top answer here codereview.stackexchange.com/questions/163354/using-2-threads-to-compute-the-nth-fibonacci-number-with-memoization
OpenCL please
ill put that in the queue
That would be awesome ! Keep it open guys ! (Even is CUDA is pretty awesome...)
THIS!!
We should not be promoting proprietary tech
@@SirajRaval What do I need to know to create my own deep learning framework? What are the books and courses to get knowledge for this?
At the end how is kernel parsing entire array if 'Stride' variable will be size of block. Some elements of array will be skipped?
Do we need the learn Cuda if we plan on using Tensorflow? I know Cuda allows us to process on many cores, but I heard something along the lines that Tensorflow already has this idea built in. Is this true, or do we need to pair Cuda and Tensorflow in our deep learning programs?
I have a CS degree but this shit entire level above my understanding
Hi guys. PLease i ran the code on google colab and all it outputted was Match error: 0.
Please can anyone help me?
Hi siraj, can you guide me what to install inorder to execute CUDA code? I'm scratching my head since a long time for the successful execution of the program on windows using Visual Studio.
Absolutely lovely visuals!!!
I spend several hours yesterday trying to install tensorflow-GPU in Anaconda so I could use my GPU to speed up training of our neural network. It's an incredible pain in the ass as there are 7565 dependencies that all need to be installed in specific way and you're forced to register for the Nvidia Developer Spam Program. I didn't manage to do it in the end.
Anyone have any tip for painless installing of tensorflow-GPU? I was really suprised and annoyed by how difficult it was.
Awesome video, as usual. You should do a video on what makes a good feature and how to select good features (feature engineering). Have a good one!
I understand that this video is meant to be an introduction but can you make a video or two where you go into more detail and do some coding of this? I like your format of short videos with memes as introduction and longer videos for the coding but I can’t find the longer coding video for using CUDA or GPUs
Yeah I need to do a long cuda video good idea
What do I need to know to create my own deep learning framework like PyTorch? What are the books and courses to get knowledge for this?
I need a help please so urgent , I want to have all the prediction of each class my softmax gives me only one result of one class , is there an instruction softmax multilabel
the cpu is the powerhouse of the pc
I hope khrono group goes through with its plan of making vulkan a replacement for open cl
Vulkan is the successor of OpenGL.
Dave Axiom but as it got rid of most of that api's( open gl ) baggage it also makes sense to use it for computation
Dave Axiom www.phoronix.com/scan.php?page=news_item&px=IWOCL-2017-OpenCL
Dave Axiom www.phoronix.com/scan.php?page=news_item&px=clspv-OpenCL-To-Vulkan
What machine do you use for your GPU programming? Can you tell us how to port what you are doing from your on-premises computer to a Cloud provider? And which Cloud providers support this? That would be interesting to know.
Easier way to start is a cloud instance, eg. on AWS p2.xlarge (aws.amazon.com/ec2/instance-types/p2/). If you do a lot of computation and want to save expenses for cloud or train faster, you can build you own computer (eg. timdettmers.com/2015/03/09/deep-learning-hardware-guide/).
th-cam.com/video/Bgwujw-yom8/w-d-xo.html
Thanks.
Great video, thank you.
Can you show me Parallel Programming inside scripting C# Unity programmes please?
Great tutorial man!
thanks for this siraj
hi everybody. I need a help I have a python code which include CUDA but I'm beginner on python use, when I run it I get error, (Torch not compiled with CUDA enabled) may someone help me to solve this issue? I'm doing my final research. I look forward to hearing from you
what a passionate tutorial! I wish you were my professor for my parallel programming course. Well done!
Since you are adding the same numbers in the array, won't path prediction affect the speed of addition?
Does anyone know if tensorflow 1.12, CUDA 10 and CUDNN 7.4 are all compatible?
Damn you are such a great teacher dude.
For fibonacci you can also just use a memoizer and store the previous values calculated on a cache to make your algorithm faster xd just a trick 👍
you could also use faster asymptotic algorithm variant, if you want only to compute a single term of the sequence. It suffices to use a square and multiply on a certain matrix. It doesn’t make it parallel though :(
2:22-wasnt 3Dfx around before nvidia?
Could you help me, I need the best GPU for science purposes not more expensive than 2500$ (it should work good with double): could you tell me on what characteristics should I look and what is the best variant?
Awesome tutorial, thanks.
will i be able to use CUDA if i use an external nvidia gpu with my macbookpro? is this recommended?
Andrew what gpu
Mac doesn't support CUDA I think?, Maybe if you use windows 10 (Or even better Ununtu) on the Macbook through bootcamp with a 3.0 minimum level GPU card through thunderbolt might work
macOS supports cuda.
metal 2 shading language from Apple is Best for Apple hardware.
I havent bought the external gpu yet, but looking at the 1080. just wanted to know if its feasible before getting it
Will CUDA work on my 3DFX Voodoo 3000?
finally u'r coding in C, :) :) :) :) good video man
hey Siraj, do you write your own scripts? who funds all this?
Siraj I ll be waiting for your reply...
Can you please make a video on how to use Google collab notebook to train our machine learning project
You don't need decive sync after kernel executions.
can anyone explain this error?
"CUDA driver version is insufficient for CUDA runtime version"
how to resolve this error..
when you install cuda, dont pick custom install... pick express will override your old drivers. This makes sure everything will work.
Amund Lindberg i have intel gpu but some are saying that cuda will only work on nividia gpu’s ..gpu is the reason of that error? plzz reply thanks in advance🙂
What's the practical usage of this?
Anyone has ideas about where the cafe room is at time 1:45?
how parallel computing can be done with python??
What about AMD graphics ? Do we have any ML package to process on AMD graphics? Thank you in advance.
good work, helped me a lot
For those who have not tried it out yet and have an AMD chip don't bother...it gives segmentation fault and doesn't run at all
and a/c to otoy company they claim that they can do it it's not released yet so sadly go back and do it for open cl
and Thanks Siraj, you have great videos.
EDIT: Don't even install that nvidia-cuda-toolkit that it will recommend you if you have amd gpu... it will ruin your graphic drivers and you will be unable to boot into the system
You just opened my eyes to parallel programming. Thanks for the quick overview.
this might be a noob question, but what do 1
its bit shifting to the left on the binary representation of the number, for example 1
Seems like CUDA tries to make it as easy as possible to access the memory from both GPU and CPU, but what's the tradeoff?
Can be very wearing on a GPU reducing its lifespan
Source?
There are some performance trade-offs when using Unified Virtual Memory (UVM). Memory pages are migrated between the host and the GPU on-demand (plus some nifty prefetching heuristics) but frequent page faults often stall the system. I suggest using explicit memcpy semantics between the host and the GPU.
Thank you Daniel, that was the kind of trade-off explanation I was looking for.
Kevin, I'd like to read more about that. New GPUs seem to be _meant_ for that kind of stuff, don't they? I'd be surprised if it would really kill your GPU.
oh and nvidia didn't release the first graphics processing unit
It's like missiles of knowledge coming through his brain.😂😂😂😂
i dont understand except that there are cuda cores in GPUs and some graphics pipelines.
I hate you for not mentioning 3Dfx! Voodoo graphics. And Voodoo 2. And around Voodoo 3 Nvidia bought the company. BUT! 3Dfx started everything years before Nvidia. 1997! How could you?!
This video is way too useful to thumb it down though, good stuff.
good point
I dislike people making video and not do all research. Forgetting about 3Dfx and Voodoo... I know name GPU is from GeForce 2 (not 1, there are still some thing calculated on CPU on GeForce 1), but in 3D graphics first card was Voodoo, and open Pandora box. Is sad they was one innovation company, and not move forward, if they do we have 3 major GPU company not 2.
i was gonna say that i paid $400 for my 3dfx in like 98
@Artem You are right 3Dfx did start it all.
i like how you pointed at the vertex shader image when you said "cpu"
the real stuff here is how are you using GPU in a mac? can you make a tutorial on how to make it work? as many know nvidia and apple are at war.
There are external GPUs that are compatible with MacBooks. Favorite of mine, the Razer Core X. Almost every eGPU is compatible with Nvidia.
@@existentialchild698 yeah know, but is a shame having to buy one when you already have one in your sistem. anyways i just make it work. compiled tensorflow with it and now i have enable gpu in my mid 2012 macbook pro
@@MygenteTV Okay. There's also a service made by Google called "Google Colab" that allows you to run your python code on an Nvidia Tesla K80. (That's a really good GPU that would otherwise cost you over $2000 depending on where you buy it).
@@existentialchild698 i see thank you.
@@MygenteTV no problem, thankful to help! 😊
The benchmark for using 256 threads completed 171 times faster. Is that typical of parallelized operations in cuda?
Hip can now be used for cuda. Break the monopoly.