3:04 Note that 77 TOPS isn't the entire NPU. The NPU alone is going to deliver 3x the performance over current gen as said by Lisa Su, which is around 45 TOPS NPU, similar to that of Qualcomm and Intel. Update: From what it seems Strix seems to have an NPU of 50 TOPS, which is the most powerful single NPU out of all upcoming processors. However, by complete raw AI performance, Lunar Lake is said to have the highest at 120 TOPS - 48 from NPU, 67 from GPU and 5 from CPU. That's compared to AMD's 80 Total TOPS, where the vast majority of tops are coming from the NPU alone.
Not exactly, why do you think to do something with just 1-2W power consumption is worth less like to do that with 300-400W power consumption..? NPU only can do AI things, but it's efficient in that, only reason why it's not worth to stack it infinitely, because it's can't do anything else, meanwhile the GPU can do graphical calculations when it's not doing AI thigs..
Very nice summary. I think I saw an Intel CPU slide that said "100 TOPS" (Arrow Lake?). I think the NPU part of that CPU is 45+ TOPS but using the whole processor it reaches 100 TOPS. Or something like that. I may have misunderstood the slide. Regardless, it is exciting times for CPU tech.
Don't be fooled by the number 'tops'. It means very little in the realm of AI and inferencing agains local models, when the bottleneck is the memory bandwidth. GPUs can have bandwidths of 1000+ GB/s. The M Chips from Apple can range from 100 to 800 GB/s. The snapdragon X Elite was told to have about 135 GB/s according to their official specifications. Also: NPUs currently are lots of times proprietary, meaning only limited kinds of applications can make use of them. It is a step to the right direction in general - being able to do AI tasks locally, rather than being bound to some server and service. But currently their existence means little. There's a good chance that currently release NPUs will become obsolete very quickly, because of the restrictions they impose on models that can be run.
And what if I tell you, that the NPU engineers can put a SRAM/DRAM inside the chip and with the proper NN compiler software, they can reduce the usage of external memory? Also, the main reasons to develop NPU-s are the cases when you have to deal with Power-Performance questions. Think about self-driving solutions: every manufacturer is going to choose an NPU over any GPU, because as long as most of Nvidia GPUs does not care about the power consumption (and the latest cards using 200-300 W), a well-built NPU could require only 20-30 W for the same result. Some explanation about same result: most modern neural networks use a lot of complex math operations with large GMAC requirement inside of them. If you build a hardware that can help the parallelisation of the exact operation, you will need less TOPS to execute it as you run it on a GPU. (GPUs can solve matrix multiplications efficiently, so it’s compiler tries to convert everything to it and optimize it with various kinds of algorithms. But in the end you replaced an operation with a more “expensive” operation. )
You praise unreleased /unavailable Intel and Snapdragon products too much and ignore Ryzens that are fully available for almost a year now. At least in my nearly year old laptop with 7840HS there is NPU - but there is not enough information of how to actually use it!
we got a new awesome tech youtuber , nice ,short ,informative .
Thanks! I'm trying my best!
Let's see the software support for this kind of hardware.
Keep up with the good work, Multzinity. Cheers from Brazil!
3:04 Note that 77 TOPS isn't the entire NPU. The NPU alone is going to deliver 3x the performance over current gen as said by Lisa Su, which is around 45 TOPS NPU, similar to that of Qualcomm and Intel.
Update: From what it seems Strix seems to have an NPU of 50 TOPS, which is the most powerful single NPU out of all upcoming processors. However, by complete raw AI performance, Lunar Lake is said to have the highest at 120 TOPS - 48 from NPU, 67 from GPU and 5 from CPU. That's compared to AMD's 80 Total TOPS, where the vast majority of tops are coming from the NPU alone.
Put into a sentence, an NPU is what a GPU does, but worse.
Not exactly, why do you think to do something with just 1-2W power consumption is worth less like to do that with 300-400W power consumption..? NPU only can do AI things, but it's efficient in that, only reason why it's not worth to stack it infinitely, because it's can't do anything else, meanwhile the GPU can do graphical calculations when it's not doing AI thigs..
@@TamasKiss-yk4st A GPU uses 300-400W if it's specifically the 4090, and doing heavy operations. I can sense the "Pulled out of my ass" already.
Very nice summary.
I think I saw an Intel CPU slide that said "100 TOPS" (Arrow Lake?). I think the NPU part of that CPU is 45+ TOPS but using the whole processor it reaches 100 TOPS.
Or something like that. I may have misunderstood the slide.
Regardless, it is exciting times for CPU tech.
Yep, you're right! I'm just using NPU TOPS since this video is only about Neural Processing Units.
Don't be fooled by the number 'tops'. It means very little in the realm of AI and inferencing agains local models, when the bottleneck is the memory bandwidth.
GPUs can have bandwidths of 1000+ GB/s. The M Chips from Apple can range from 100 to 800 GB/s.
The snapdragon X Elite was told to have about 135 GB/s according to their official specifications.
Also: NPUs currently are lots of times proprietary, meaning only limited kinds of applications can make use of them.
It is a step to the right direction in general - being able to do AI tasks locally, rather than being bound to some server and service. But currently their existence means little. There's a good chance that currently release NPUs will become obsolete very quickly, because of the restrictions they impose on models that can be run.
And what if I tell you, that the NPU engineers can put a SRAM/DRAM inside the chip and with the proper NN compiler software, they can reduce the usage of external memory? Also, the main reasons to develop NPU-s are the cases when you have to deal with Power-Performance questions. Think about self-driving solutions: every manufacturer is going to choose an NPU over any GPU, because as long as most of Nvidia GPUs does not care about the power consumption (and the latest cards using 200-300 W), a well-built NPU could require only 20-30 W for the same result.
Some explanation about same result: most modern neural networks use a lot of complex math operations with large GMAC requirement inside of them. If you build a hardware that can help the parallelisation of the exact operation, you will need less TOPS to execute it as you run it on a GPU. (GPUs can solve matrix multiplications efficiently, so it’s compiler tries to convert everything to it and optimize it with various kinds of algorithms. But in the end you replaced an operation with a more “expensive” operation. )
Good job
How many TOPS does a mid-range and a high-end GPU have?
Intel Ultra lunar lake was actually said to have a total of 100 tops.
Yeah, but the NPU itself is delivering 45+ TOPS.
You praise unreleased /unavailable Intel and Snapdragon products too much and ignore Ryzens that are fully available for almost a year now. At least in my nearly year old laptop with 7840HS there is NPU - but there is not enough information of how to actually use it!
Noob Processing Unit ?
Microsoft is just good, let them do what thay think. Ryzen AI will demolish intel!
Probably the most useless feature on your pc
Only being used for ai