ไม่สามารถเล่นวิดีโอนี้
ขออภัยในความไม่สะดวก
Do we really need NPUs now?
ฝัง
- เผยแพร่เมื่อ 4 ก.ย. 2024
- The first 500 people to use my link will get a 1 month free trial of Skillshare (sponsored): skl.sh/techalt...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
►►► This video ◄◄◄
Microsoft with Copilot+ PCs, Google on the Pixel, Apple with iPhone and Macs, Qualcomm with Snapdragon X, Intel and AMD, all the tech companies want you to care about the NPU, Neural Processing unit or Neural engine in their new machines to run AI workloads, neural networks, etc. But are these chips any good?
The Story Behind
This video on Nebula: nebula.tv/vide...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
►►► TechAltar links ◄◄◄
Social media:
mas.to/@techaltar
threads.net/te...
bsky.app/profi...
/ techaltar
/ discord
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
►►► Attributions & Sources ◄◄◄
Channel Music by Edemski: / edemski
Other music by Epidemic Sound: nebula.tv/epidemic
The first 500 people to use my link will get a 1 month free trial of Skillshare (sponsored): skl.sh/techaltar06241
useless
love the shirt :)
the real question is, do we really need to know this ??
or you are just making video for the sponsors or.... ?
"silicone valley snake oil such as blockchains"
I will take any unwanted snake oil crypo you are willing to throw away
the evil sponsorship has consumed you
Engineer: AI Needs a ton of ram.
Tim Cook: Great, let’s ship our MacBook *PRO* with 8Gb.
"And when people complain we'll say Apple's 8gb equals to 16gb of MS ram because magic ✨✨"
That's Apple's pricing structure, they want to look competitive with the pricing with the "base" model while charging some insane premium for the usable ones people would actually buy.
@@Artanis5 There's some technical truth to that, at least on iOS and iPadOS devices - they do transparent compression on memory pages not in active use. It's nowhere near 2x though, but using 3GB on an iPhone where 4GB would be necessary on Android is tenable. Apple is largely a marketing company, but this is one of the few examples of good engineering, partially enabled by the fact they've got a higher degree of integration from hw to sw. With that said, if I were to buy an iDevice for a premium price, I'd want the full 4GB or 8GB or whatever and for this tech to be used to increase multi-tasking performance. LPDDR is sufficiently low power that the battery gains from cut down memory are not as useful and it's not as if they're passing the relatively minuscule cost savings on to the customer.
that is decoy marketing strategy!
Very bad situation.
People: we want better battery life in laptops.
Microsoft: we will use NPU to do tons of work in the background.
People: disable it to get better battery life.
it's called over improving something... Windows 10 is good enough, so they made Windows 11, for microsoft, not it's users.
@@stachowiwindows was good enough 10 years ago. They should have stopped then
Much more vocal is the sort of person who solves one side of a Rubik's Cube and gives up.
LTT just showcased the first OLED TV, it was not preceding a world where every monitor is now travel-sized. After sufficient adoption of NPUs they would eventually be many times more efficeint at tasks and save battery more than conventional CPUs ever could.
But it's a question which is better, integrated NPU in the CPU when we look at the comparison with GPUs.
@@sboinkthelegday3892 I just want a well supported dedicated card for the NPU that I can plug in via something like PCIE. Doesn't need to be on the CPU or GPU imo, though a basic one on at least one of those may make some sense for lightweight tasks, similar to graphics modules on CPUs today I think.
@@stachowiso a 7700k is good enough, but people still buy and use 13900ks because it's faster
- Computer, what big NPU you have!
- I need it to better spy on you.
That's not what a NPU is for. Just because Microsoft requires one for CoPilot Plus's REcall Feature does not mean the NPU is spying on you, the software is.
@@thelaughingmanofficial Strange logic. You could just as well say "Your Nvidia GPU is not made for showing graphics, the drivers for it are". Duh!
@@sem_skywalkerblaming hardware.
For faulty software usage is clown behaviour.
🤡 : The camera was made to spy on you.
@@sem_skywalkeryour Nvidia GPU is made to do parallel processing, which just happens to be very efficient for graphics which is why the drivers render graphics through it
So yeah, the drivers tell the processor to make graphics
@@sem_skywalkeralso an npu is just very efficient at some operations that most ai models use a lot during inference
It's doing everything except for spying on you, especially since the npu being in your laptop means the processing can happen locally without sending your screen to Microsoft
I already saw laptops with NPUs but 8gb of soldered ram 😂
Meanwhile a used old Thinkpad can be upgraded with 32 gigs and accomplish 90% of an every day users needs WITHOUT an NPU.
Looked at an Apple machine?
@@bhume7535 Correction: Marketed as 32gb max but can actually be stuffed with 64gb with high-quality so-dimms and still works! ;)
@@othername2428 well mine only has one sodimm. So 32 for me.
solder RAM is way faster !
WIN WIN WIN
Just when they faced limits with CPU and GPU suddenly out of nowhere appears new NPU thing that you definitely need in every device.
Cap
you only need it if you plan to use AI stuff, the problem is that probally many apps is going to use AI stuff even if you don't want.
@@gabrielandy9272 you don't need anything other than CPU and GPU
@@gabrielandy9272 Are going to? Are going to? Have been doing for about a decade (what did you think RTX was?). NPUs are basically tesla cores that run on an apu instead of a massive gpu. Microsoft and apple pushed for integrated npus so laptop and mobile users could have access to the tech that made Nvidia trillions.
"No questions. Just consoom products and get excited for the next products."
guess that is why Apple was saying 8G is enough in the recent past so they would have a solid up-sale later
Oh that was so stupid. And still is. Seriously really expensive laptops and 8Gbyte standard for only $200 you get 8Gbyte more????
Apple is insane
Unfortunately this is exactly what happens when there is no adequate competition. Let's remember Intel's monopoly when AMD were one step from bankruptcy. Every single new generation of i-series processes had +5% improved performance at best. Yet still, Intel thought that it is necessary to have a new chipset for almost every single new CPU generation. And in the the end they even started using regular thermal paste INSIDE of their CPUs.
If AMD were able to return from dead and become a new leader in CPU segment, Microsoft has all chances to do the same, if they'll change their management which only wants to spy more on their customers and include as many ads as possible into already non-free OS. Corporations, gaming and familiarity - three main reasons why people still use Windows.
@@MasticinaAkicta no. Apple isn’t insane, customers who buy them are.
Apple is very clever. They know how to milk their sheep.
@@MasticinaAkicta you spelled 'insanely greedy' wrong.
To be fair, as someone who hates apple's BS, Apple's software is very efficient, so it can be argued that the base non-pro models can get away much more effectively than on windows (esp after manufacturer crapware is factored in), even though I don't know what swapfile/pagefile on MacOS is.
the realization that 2017 is seven years ago 😳
what ??? really ..... o no time fly too fast ....
when you realize 2015 was five months ago: MIND BLOWN
why did you have to go do the math 😭
💀
we living in simulation ever since 2020 someone please wake up 😭
We do not want AI. We want Performance and Battery Life. PERIOD
Exactly, that is why NPUs are a thing, they increase both performance and battery life when running AI tasks.
@@Kikikan no one was saying anything about wanting running AI tasks ;)
Amen. The tech companies are delusional. Either that or they’ve got an ulterior motive for pushing this crap on us.
@@IvoPavlik Then I also hope you do not take pictures with your phone, or use auto-generated subtitles at TH-cam, because those are also AI tasks ;)
@@Kikikan then put the npu on the camera module itself. YT subtitles work don't take much compute, and could be calculated when the video is uploaded to YT itself instead of recalculating over and over again. NPUs just don't help average people much right now, they are too narrow in scope with too few uses. GPUs got made in response to demand, NPUs were made to try to create demand.
Looking forward to the AI bubble bursting. There might still be some AI that's worthwhile but it's way overhyped and underdeveloped right now
Going to have to agree. I don’t think Nvidia is just going to be the biggest company in the world indefinitely.
Specifically the LLM bubble. AI like AlphaFold is way underhyped.
It's not AI. It's algorithmic transformers. They are not intelligent.
@@ahmedal-hijazi3618If it falls hard enough, the GPU market could open up massively for new competitors. Really unfortunate that for professional work its Nvidia or nothing.
fuck yes I'm so tired of hearing about AI every time I open any tech media! This has happened with the Big Data fad, VR, and other tech trends. But so far this is the most obnoxious one because it is literally everywehre even your grandma is talking about AI for some reason! They are playing us like a damn fiddle
Yea recall is a huge security and privacy nightmare
I agree but man would it be nice to roll back when a coworker messed up a Excel formula 😂.
Only real positive in my world lol
@@Theausomecaleb use git or subversion some kind of version control on that file and then any version you want to revert to can be done. Though there is a bit more technical know how that goes into that and getting all your coworkers to learn that is a uphill battle. Plus source code version control is really good at text files and if you are working with images or binary file types you will probably going to need an extensions for your chosen version control solution that works with the file formats. Otherwise your version control will get slow and bloated keeping every binary version in it's historical archive.
By never sending your screen anywhere over the internet and instead doing all processing locally? Sure
@@PFnoveit will require modyfing registry to not send knowing Microsoft and how they love to make spyware
@@itsmenatika nope, if they send it back to Microsoft with no straight forward way to disable it they'll get a multi billion dollar lawsuit probably
The eu won't be happy for sure since they fined over a billion over USB-c vs lightning
I've transitioned from 'wow, that's kind of cool. Wonder what they'll come up with next?" to "Fuck off with all the AI, please," in about six months.
Wow, that's kind of cool. I wonder how many ways they'll find to abuse it.
"ai" (really just glorified autocomplete) got so much worse the last 8 months its actually hillarious. but its still apparently needed in every device they could possibly stuff it in because... money
i can't believe you're not ok with having something screenshooting everything you do on your PC, and let's face reality, upload it or some summary version of it, to be sold to some company in a "do you agree with the new terms of service" without a opt out option. You're being unreasonable man /s
@@Jeffcrocodile yeeah.. when i read about that windows update with this new feature, that was the day i switched to linux. i still have windows on a seperate harddrive but.. i find myself using it less and less. im sick and tired of this (to quote luis rossman here:) "rapist mentality" of corporations becoming more and more apparent each day... i want my dumb things back. 99% of things dont need to connect to the internet imho.
@@h0125t There's your problem.. using Win 11.
We need a video explaining the clean shave, brutal betrayal 😂
Certainly, quite shocked when I see him 😅
dude looks great clean shaven
No wonder I feels something is off..
Lost password, had to create egg account and it takes 2000 posts to change profile pic.
Except this is in digital twin.
clean shaven martin jumpscare (very scary)
As long as Linux doesn't, no
I get where you are coming from, but that's a brain-dead take. Computer hardware is driven by software which can utilize it, the os platform is generally irrelevant (excluding 1st-party software which are tied to the os). Just as an example, lots of games use DLSS upscaling which uses Nvidia Tensor cores (basically an NPU), got nothing to do with Mac OS, Linux or even Windows for that matter.
Welcome to the 1990s. Enjoy your stay.
Linux user try not to constantly tell everyone that they use Linux challenge: Difficulty: Impossible
@@beckergreyso apparently, you don't need it until a lot of software uses it. And until something in Linux actually needs it (not gonna happen in next 5-10 years), tech can be ignored.
U actually can't name 5 programs that actually require this thingie, and can't be stupidly emulated by OpenCL or Cuda
@@isadora-6th Did you even watch the video? The name of the game is 'efficiency in low power hardware'.
This is the best explanation of NPU so far! Other people that have talked about it are either only talking from political/emotional standpoint or from the money aspect. No one has ever discussed it in a practical application angle like you did. Great work, man!
That’s TH-cam commenters nowadays!
I'm sure Apple's 8GB laptops will be totally 'equivalent' to 16GB for AI use...
Apple fan boy 🍼
According to themselves, yes. When it comes to actually fitting the whole model in memory, no.
@@abhijith6919 Hardly. I was being sarcastic.
@@LeonardTavast That was my point. Apple's stinginess with RAM was all fun and games until they wanted to catch up with AI, and now looks very short-sighted.
@@axi0matic catch up? i seem to remember the neural processor being added to iphones way before there was a real use case for them. oh wait, the measure app! cant think of a more useful use of AI than the amazing measure app.
NPU = not probably used.
Should be called UFB = Used for bloatware
Not Phor U
My first thought was "can I save $10 on the CPU by getting one with the NPU disabled?"
Voice dictation is pretty useful.
@@demolition3612 typing is more useful ngl
Curious note: in the early days of NPU in phones, they were less powerful and power efficient than GPUs. Huawei, Samsung, and Qualcomm relied a lot on their GPU to do AI work. Google with Visual Core started to deviate and then with Tensor, few months after Tensor Snapdragon 8 Gen 1 was the first Qualcomm chip with a really powerful NPU.
They were used mostly for that "ok, google" voice recognition back in the day, not this "LLM" bullcrap, it was supposed to be less powerful, but it was actually more power efficient because it wasn't to be used for generative AI which runs it for a longer time.
To give some clarity and correct some misunderstandings:
NPUs are way cheaper than GPUs.
NPUs use a lot less power and are more efficient than GPUs.
The total of parameters are cross all layers.
To do a neural network (NN) calculation you’ll only need the previous output and the current layer. This is why NPUs don’t need excessive RAM.
Specifics:
Face recognition on an image only needs a model that is less than 500MB in size and a NPU of about 1TOP.
Small Language Models (SLMs) need about 2-6TOPs to have a reasonable response time.
Real-time object detection needs about 10TOPs.
This is all running under 2W.
A 13TOP NPU is about $70 USD.
All depends on precision. What quality and to what precision is the facial recognition? The rest of your comment, you could talk in terms of units/stats/specs, instead of arbitrary terms such as large/small language models. What's large?
Im frequently seeing TOP mentioning data type(?) (Like int8, fp16, etc..) what does it affect?
@@gilangwahyu4450 Precision of your models. The more decimal points allowed, the more specific your parameters can be, allowing for higher precision on your results.
int8, only allows whole 8-bit numbers. That is -128 to 127, since the first bit is reserved to state whether it's positive or negative. fp16 allows 16-bit, floating point numbers. So you can have decimals, hence more precision.
@@Potato_Quality7 Idk about in practice, but in my masters, theoretical NPUs don't have precision the same way GPU chips have. It doesn't use binary or transistors. It is extremely basic multiplication/addition gates, not on/off logic gates. It isn't digital.
Because it is the most simple signal additions, it uses the minimum amount of power and has extremely high performance. The trick is to convert the digital signals to analogue for the chip and back to digital.
@@HeyJD123 Yeah, that makes sense. The point there is to spend as little energy as possible while maintaining feasible results. Moreso for a hobbyist. I still am not too informed about the whole thing, but for now, it seems to be more of a gimmick, since what it can do is very specific.
Frankly I don’t want it if it means I can avoid AI features. Unfortunately it looks like the options are “run the AI locally” or “give us your data and run it in the cloud”. I don’t want either.
Absolutely hate how AI is being forced into everything now.
I don't know if it's been forced as much as it's supply and demand .. also imagine making an non NPU device and an NPU device of same product - why would anyone keep inventory of non NPU products which almost no one will buy
It's not forced. You are given the chance to use it. It's not like they make you log in to your devices via a chat bot and send email and IMs and everything else. They're making it easy for people who do want it to use it.
now? OCR is AI, google pixels cameras have always been terrible but theyve taken the best photos because of their AI systems, weve got image identification, facial recognition, fingerprint recognition, machine learning, etc. its been coming for a very long time. generative AI is simply the most recent step.
If I can run NPU with Ollima then I am happy camper
It's been there all along, the only difference is that the it got enough public attention that the focus of marketing shifted towards those features. You probably use a ton of ANN based features without knowing, and have been for years...
One way I could be more interested is if the NPU had a little more data types, and would be good for other DSP-style calculations, like impulse responses for audio.
From reading the docs, you *could* use it for convolution and maybe some more operations, but I don't know how better they are than a GPU
@@shadamethyst1258 An NPU would be way faster than a GPU for convolution while drawing a lot less power. Still, audio is CPU bottlenecked because the processing is serial, not parallel. GPUs are nice for 3D rendering graphics and vector calculations, not designed for AI/machine learning.
At least an NPU would be extremely helpful for stem separation, my i7-12700K alone takes like 20 mins to process stuff, i watched the latest Apple M chip doing the same thing and it takes seconds to finish the processing because of the NPU. CPU wise, my 12700K is equivalent to an M2 Pro and slightly more powerful.
Qualcomm hexagon npu was previously a DSP with matrix extensions
AMD's AI engines are direct descendants of the DSP engines in Xilinx FPGAs!
Dude i must say this is the best video that i have come across that explains the need and differences of various components (such as CPU and GPU) in simple terms.
As a programmer, i can think of a handful of non-AI/DL/ML workflows that could see significantly improved performance esp. in 3D processing and post-processing
Simply put, anything that processes matrixes in both parallel and branching patterns.
Some of this can be simplified to vector SIMD operations, but that's not a 1:1 replacement for parallel matrix operations.
So-called Tensor cores are physically designed specifically for matrix operations.
Isn't that basically what GPU's are for?
I'm impressed that you explained a simple neural network correctly. Not a lot of people actually understand it. There's of course a lot more to it like back propagation and quantisation.
I would be interested if the NPU can run larger open LLMs utilising RAM which is cheap compared to trying to increase VRAM. Unless you're willing to give Nvida the price of a car and your kidney and maybe your first born child then 24gb and maybe 28 if they bump the 5090 a little. Ultimately I'd love to see expandable VRAM like we had back in the 90's.
The problem with this is that, with time things became more and more reliant on memory placement rather than memory space. That is, nowadays, it's not just whether or not you have the memory somewhere you can find it to process it, it's also whether or not it is in shared memory, local memory, managed memory, mapped memory, etc. Those things all change performance by several orders of magnitude, and so while you could technically run a model using normal RAM, it would still be extremely slow due to bandwidth issues because you'd have to be copying memory all over the place.
NPU
. Normal processing unit
IGPUS - Integrated Graphic that Sucks
GPU - Grossly Priced Unit
I was wondering what it stood for. For some silly reason my brain said "nothing processing unit" 😊 thanks 👍
@@vevisa3287 Goat Packaged Unit
Neural processing unit (not normal)
Wow , you explained it in such an easy way. Amazing video
Glad you liked it
Whats concerns me is that at some point in the future you will have massive for profit corporations having easy access to billions of NPU's.
I fear that our devices will become a grid computer for manufacturers against the will of the customer.
Sure, it would be efficient and beneficial, but FOR WHOM?
A capitalist would just love to abuse our private processors and electricity to benefit their own businesses: what an insideous way is it to disguise corporate carbon emissions by pushing them silently onto the users of their proprietary hardware and software.
Thank you sooo much for explaining this more in detail instead of just saying "neuronal network" as if everybody knows how they work.
The resson you want a dedicated NPU instead of just a GPU.
Is because it leaves the GPU free to do its own well optimized Specialized functions.
It's the same logic that got us to move on from just having a single Processing Unite
That, and that CPUs were and still are enormously bad at extremely parallelized tasks.
12:53 the use of the word "was" in this sentence made me giggle and rub my hands together all evil like
As a computer scientist that works with this stuff, yes it is very much needed for AI. The GPU is like a more generalized version of this, with the exception of nvidia Gpus that are built with this extra task. It works, and yes a powerful GPU can do the same… but you could scale a NPU to the scale of a GPU and get the same if not more performance with 10x less power. It would also free up the GPU development to only focus on graphics, and offload neural stuff to the NPU, similar to when GPUs became popular. Meaning GPUs would also get a significant speed up for graphical rendering in the future when paired with a NPU
Thank you for giving this beardless man the chance to upload a guest video on your channel - looking forward to a video by the real Martin next time 👍
They're designed to monitor everything you say, read, and type. There's no way this ends up going badly.. It will likely only be niche manufacturers who will offer laptops without a neural engine for privacy reasons. After Microsoft's debacle with Copilot, there's no reason to think they won't be pumping this in with the usage data sent back to them.
facts, from their perspective the ROI on data has no real cap so any extra bit they can get (however ethical) can be monetized in some way with near infinite return potential
in lack of NPU, they could just use the GPU instead. Not having one is very most likely not going to be an effective protection.
One in a million people care about privacy, if that. Think how many people use Android and iPhone, almost all of them. I only know one person who cares enough about privacy not to use them. And he still uses Windows... Recall will happen, it has such huge advantages for consumers that the privacy loss won't even factor in to their decision. Soon it won't be on-PC, we will accept that everything gets sent to the cloud because that will make it even more powerful. Eventually we will even get fully homomorphic encryption for NPUs too.
@@user-ub6pz4up8r One in a million?! A much bigger percentage than that is actually willing to take steps to protect their privacy, such as 17% of people using tracking-free search engines, 31% of people having used VPNs, 15% using encrypted email services, etc.
To me this suggests that at least about 15% of poeple care enough about their privacy to be willing to compromise some amount of convenience to protect it, about 1 in 7.
Additionally, more than half of people voice concerns about how their data is used instead of simply not giving a shit, but most do not take any steps to prevent this.
Every time i see a die-shot where there is an NPU i cry for the cores/gpu/cache that could have taken that space.
Yesssss 1000x yes, same thing I think, specially since gpu can run ai if that was the case... Such a waste of silicon and everyone's time
Exactly. They could be giving us CPUs that are 2 - 3x as powerful, instead they jam in extra spyware that you can't opt out of
Spoken like a person who doesn't know Jack about how pcs work@@chrismurphy2769
Cores are literally the same size lad. What are you talking about
@@user-pp5cm1sc7x NPU takes up space that can otherwise be "the type of chip we actually want and pay for in a cpu", such as more cores/cache.
And if you watch the video, some cpus have even larger npus.
My computers always have powerful GPU so I have no need for NPU and I don't want one, and I don't want a cpu designed with drawbacks necessary to accommodate one.
What's more is I don't trust what companies will program NPUs to be used for.
Working with Embedded Devices in building automation equipped with (micro) NPUs. Targeting Video and Audio based person localization with a low heat and power footprint at low costs :)
The video breaks down perfectly the most important parts of NPU computation with well created graphics. Thumbs up 👍🏽
Yes, NPU are useful, and it doesn't have to be for "LLMs" or image generators only. It's for stuff like upscaling, OCR, voice-to-text, image object recognization, etc.
8:13 TLDR - Models that can run on your phone are small enough to be trained on a capable home computer, not a requirement that it's trained in a server center
So this video is a great introduction, but 8:13 - to give a high-level gist of it, training could be done on any device, but several multiples of the size of your model needs to fit into the device's RAM, and training speed is limited by that and your processing power. The same is for inference / run time, but you only need basically a few multiples of the size of your model in RAM. So large models that needed to be trained in server centers are being run in server centers (you send your input over the internet, then it's computed on their end, then sent back to you), or medium sized ones may have been trained in a server center can run on your computer. Models that can run on your phone are small enough to be trained on a capable home computer
EDIT: holy shit I continued watching and this is more that a great intro, this is a great overview of the current space
Hopefully this means that GPUs wont cost an arm and a leg because everyone will be buying NPUs instead for AI/cryptomining
So, chip makers rushed to make a worse version of a GPU, which most people already have, just to say it's an "AI laptop/PC" now?
This is absolutely happening because computer sales slowed down after the pandemic. There is zero consumer demand for anything going on here.
Bro, this is better than Nvidia having unquestioned Monopoly on ai. At least with these NPUs you are able to use ddr system RAM which is much cheaper than VRAM specially if it's from Nvidia who intentionally increases the cost of higher VRAM models in unproportionate to VRAM prices and also cutting the VRAM amounts in gaming GPUs so it won't self-compete.
But ai is definitely underdeveloped for mainstream consumption and overpushed by tech companies when there isn't proper value to NPUs yet.
Would be useful if NPUs can act as DSP, 2D GPU maybe through image generation then mainstream consumers start getting use from those npus
@@jdargui1537 Integrated graphics could already do that, except integrated graphics is far more useful than an NPU.
Good job but I'd add two more very important factors to the AI chip requirements: 1) it needs to use low power and 2) it needs to be low-cost.
Its great to see that we are seeing developments in machine learning realtime beard removal models!
your brain is now online?
Npu can only run small ai models in a PC. There isn't enough ram to run llm
It depends, if the NPU uses system RAM on a X86 system it would be practical. The issue would be memory bandwith, so we will get PCIe NPU cards in the future with dedicated memory (NRAM?).
Ironically on the mobile devices there's more RAM because the NPU/GPU/CPU has a unified memory model. So if you have a mobile with 16GB of RAM, it technically has more RAM for the NPU than most of the mid-level GPUs on the PC, Nvidia is profiting a lot from just selling memory chips overpriced, because you can't just put more RAM in the GPU like you can on system RAM.
But when it comes to the HPC GPUs for Datacenters, then they have removable memory.
And before anyone says GDDR is faster, it is not, GDDR4 has the same performance than DDR5, the only difference is that the CPU access the memory at 512 bits (yes, a 64bit CPU access memory at 512 bits, what do you think "DDR" means, nowadays it transfer 4 bits per clock, and with dual channel, that gives you 512 bits ) and the GPU access it as 4096 bits. DDR is a bit faster in random access, and GDDR is a bit faster on sequential bursts, but you can get system memory that has 12-12-12-12 and it'll be faster than GDDR, GDDR is mostly marketing.
I think this is a good thing tbh. Regardless of whether AI is a bubble or not, having hardware that can massively parallelise workloads like the ones demanded by AI is never a downside. It's the same way how GPUs pivoted from graphics to massively parallelised computations. Increasing the chips' raw compute power can only take us so far, at some point we have to bite the bullet and come up with smarter designed chips
Did you consider that perhaps the NPU size (currently) has a pretty much fixed absolute size so that is why the proportion of NPU size / total processor size looks bigger on smaller chips. If that is the case this means that there is not really a priority for NPUs on smaller chips but rather that is just the required size for it to be functional.
To my knowledge there is no reason why the size would be fixed. Also, NPUs being in mobile devices since 2017 while they are only now making it to laptops and still being nowhere near PCs shows that they are very much tied to size
@@TechAltar Couldn't it be one of those cases where one doesn't want to design other versions because of the R&D cost that comes with it? Similar to how CPU cores vary in quantity, but (usually) not in design?
whe-whe-where is your beard Tech Altar sensei!!??
This is, hands-down, the best video on this topic on youtube. Great job! Hope you get ta TON of shares!
"More TOPS, more TOPS"
Also said the bottoms.
The NPU for most people is to help companies spy on you using your own hardware to assist.
I think you missed the main point of NPUs by saying they are a "minor part" of the die. Thats the appeal! The fact that you can have substantial ai workloads using only a few watts via a tiny component on the cpu is huge.
No need for the extra price and power of a dedicated graphics card which pulls 100x more power and also for dedicated gpus require y extra hardware attached to the computer
I think this equation will change as they get larger, though. I wouldn't want to shell out $1000 for a phone or $2000 for a laptop with the puny NPU's they currently have if I could avoid it.
Until you remember that memory capacity and bandwith are a thing, meaning that the NPU
is only useful for small things like voice assitants and useless for anything interesting.
we don't need NPU. the GPU is not used during most operation, while not as effective as purpose build NPU, it is powerful enough to deal with most of the workload. so what's the point? only mobile devices really want them because of energy reason which most people don't care about on their workstation.
You dont*
@@user-pp5cm1sc7x So you are going around parroting a contrarian take here? What happened with siri, amazon alexa and google assistant? its been practically a decade and nobody cares about them even though they keep getting forced onto users. AI is literally the same thing and NOBODY wants it. Look around and stop fooling yourself.
Gpu can't be forgotten as they are useful for games and rendering
No beard you feel so different xD is this a different set?
AI removed the beard.
Just wanted to chime in on the amazing clarity of your explanations in this video. I wish even 1% of the videos I watch were as brilliantly explained.
3:25 It's wild CPUS used to be ART made of wood and marble and gold and now it's just glass and plastic how tech has fallen.
If it needs its own large memory cache, they might as well use the NPU's die area as extra cache for the GPU and run it there. A discrete NPU only really makes sense for very low-power devices and devices where it has to run constantly in the background. I wonder why Microsoft would want one in every PC? 👁👁
Tesla cars have had a NPU since 2017, which has been finally utilized in 2023 for FSD version 12. The software is only starting to use the NPUs
For at least 2 decades, NPU stood for Network Processor Unit, it was used to translate all the URL handles into the actual adress, it was a bit sloppy that Network gave way to Neural since they still exist. Tidbit, Network PUs also use an inteeresting type of DRAM called RLDRAM for Reduced Latency, these DRAMs have about 20* the throughput of regular DRAMs for hash computations. Its a real pity this type of DRAM never found a place in general computing, it could be the basis of outer most cache off chip and soldered RL DRAM with regular slower DRAM DIMMS.
You are amazing! I work in deep learning and really liked the way you explained the math.
Don't think of it as failure. Think of it as time-released success.
I want an efficient laptop. I dont want windows to push their ai crap into my throat
Still one of the best tech channels on TH-cam. This video is extremely well done
that's a hell of a neat informative explanation. nicely done.
battery>storage>performance.
Thats what I want, in order. Performance is more than good empugh for amything im going to use (until new games assume I have newer hardware and refuse to work on somethong a year old) so im completely happy to stagnate it to improve battery, and bter storage, denser and faster access would be great
I still use my first computer...purchased as an adult in 1982. It's no longer a daily driver (close), but it demonstrates my pre-connected tech headspace (300-1200 baud modems are just not comparable to 'connected' in the 21st Century)
I don't think my generation is ever going to be acceptant of the surveillance functionality that has been an early use of this tech. (Microsoft's 'Recall' saw me move to Linux as my primary OS as I anticipated the direction Microsoft was heading and wanted no part. Also refuse to land-fill my older, but still absolutely functional hardware, but that's another tale - I've held onto 40 year old hardware after all.) It has been my observation that the under 40 crowd have fully accepted trading privacy for convenience, and this tech is truly for them. It just gives me the heebie-jeebies.
From my OS watching my activities, AI narration in TH-cam content, to my questioning my own eyes at every turn, I was dragged kicking and screaming into the 21st Century. I was raised in the Golden Era of Science Fiction. Feels like neural networks are a recipe for dystopia.
Thank you for placing your sponsorship at the end of your video, I make it a point to always play videos to the end when this is done (I tend to skip 'in video' sponsorships). I don't speak for everyone, but I am far more willing to 'pay' when I've watched a video that I've enjoyed, rather than being interrupted. This is especially true when the content creator creates the ad. It has a more natural flow after the fact. Anyway, it's appreciated enough to be noted here.
Inference with low latency is the future model. Paper like LLMRoute explain how prompts can distribute the complexity between the SLM (local) and the foundation LLM (cloud) so you can have a conversation with your device.
AI doesn't provide anything that enriches my experience using a computer.
Yes it does. Voice recognition, autocorrect, autofill, recomendations, camera autofocus, camera upscaling and quality improvements and more.
@@----GOD----- But those are all things that the computer could process without an NPU.
Consider a possible on-demand application for an NPU would be to review a collection of photos, make associations of those photos which are related, provide image enhancements for sub-optimal pictures. For instance, if you took several pictures of a scene; but one of the pictures turned out to be blurry, by referencing the other pictures of the scene, it should be able to clean up the initially blurry picture. So long as it only does the operation on the files given and only outputs to the same storage device, I doubt that anyone would have any privacy concerns over its use.
I certainly don't need one right now and I can't see myself needing one any time soon.
I'm also not getting anything with a new CPU any time soon because my devices still do everything I need them to do
ngl NPU will increase efficiencies but there will be a workaround for disabling the telemetry/data collection without affecting the program or make them less intrusive. But the NPU may remain idle if there's nothing using it
The simple answer to "disable data collection" would be just to not use windows.
Depending on what you need the computer for, Linux would be the answer .
@7:20 how is -0.5 + -0.5 = 0 ?
and at 7:29 "0" is annoted to white box whereas it should be a black box and vice versa
-.5 + -.5 is of course -1, but it is common practice to apply a non-linearity like ReLU to increase the expressiveness of the model. ReLU is defined as max(0,x), which means that all negative values are changed into a zero. As it is a toy model for educational purposes and he didn't explain ReLU, I guess he just simplified the explanation and a '0' can be easier to discriminate from '1' than '-1'
Thanks, this is incredibly helpful 😄
@@wilcohermens3492 Oh, so he didn't made a typo? He should at least mention that "-1" is zeroed or something, I absolutely thought that he made a mistake!!
Yes you will, at lot more apps and programs will be integrating AI
Very good video, common TechAltar W.
Glad you liked it!
The NPUs will be the main unit of a device in a few years. This is one of these things that seems useless at first, but nobody can live without it after a time. Like the PC, it was said "who could want a computer in his home?".
I have a question. If you turn off all the stupid AI shit, will the processor help performance or is it just completely wasted bloat?
Follow up, how do we make sure they make non NPU versions in the future. I don't want to buy in at all. I will keep my current phone or downgrade before I buy into this bs.
@@janedoe3043as mentioned in the video, pretty much any phones since 2017 have had NPUs. Chances are the phone you use now has one. As explained in the video, they’re primarily used to speed up things like processing the image you take with your camera, applying blur effects, removing wind sounds from your voice during a call, etc.
There isn't currently any real use for an NPU on PC but using a phone without one would be quite miserable.
NPU is merely an accelerator. Just like how there have been dedicated 3D accelerators in the mid-late 90s. Nowadays all that 3D acceleration is integrated into almost all graphics chips including iGPUs and almost nobody thinks twice. All those AI/LLM software can and probably will run on CPU, so I don't think there is a point in avoiding NPUs like plague.
@@koghs if you have an M1 or newer Mac you’ve already got NPUs and have been using them for years
10:24 I need that too
Gpus with things like Tensor cores are much better npus anyways. Npus are only more powerefficient. (Oh he said exactly that at the end of the video, I wrote it before I finished the video.)
"Gpus with things like Tensor cores are much better npus anyways"
Because of the dedicated memory and the speeds. If we have dedicated NPU cards, GPUs would return to their roots (rendering 3D graphics as an example).
@@saricubra2867 Pretty much they already exist, just not for us normal people. Just look at how much shit tons of money Nvidia makes by selling their server AI accelerator cards.
@@geiers6013 Exactly, that stuff isn't available to consumers, definetly not fair.
Maybe it will take time until they shrink those sillicon dies for a practical use.
There are available versions, not from Nvidia but other companies. These NPUs just don't have the power of industrial NPUs. On the other hand, we couldn't afford industrial NPUs even if Nvidia would sell them to us.
A Coral USB Accelerator using a Google Edge TPU coprocessor is much more affordable. Costs just 60$.
@@geiers6013That's not true. Any Nvidia GeForce card has a processor for neural networks. Even my laptop has this included.
But you can also buy dedicated processors designed for AI, like the Coral USB Accelerator which uses a Google Edge TPU coprocessor and costs just 60$. They just are less powerful. An industrial NPU is just out of range for private people because of the extreme costs of such devices .
We have had assistants for a long time now that monitor video and audio for trigger events without recording and labeling everything, and the smarter they are the better they will serve us. Some have already fully adopted them and others have not, but as they become more useful, more will make use of them. NPU's have a place in personal computing, but recording and labeling everything they see and hear won't be it for everyone.
A dedicated NPU that i can plug into my PC through a PCIe slot?
Definetly.
My 12th i7 takes like 20 mins for stem separation of music tracks, with a dedicated NPU would be nothing, machine learning stuff would be extremely easy.
the npu most likely will be worse than your cpu, even for ai tasks. its only selling point is power efficiency.
@@destiny_02Why is this? Genuinely curious.
@@destiny_02mhhh some truth here I tested GPU, NPU and CPU from my phone A55 and CPU performed much better
@@neroetal im curious how are you testing them
Man, I studied Bionics and Neural Networks back in 1988, at the TU Berlin (Prof. Ingo Rechenberg). That had been much too early. I never have been able to monetize that fundamental knowledge for me. My Diploma Thesis was dealing with engines of actual electrical cars back in 1993, running for about 300km with a high temperature NaS battery. I can say, those Neural Networks do work exactly in that specific way.
I have to say, apple has a good approach here with their unified memory.
On a desktop pc you need high amounts of vram..
But nvidia only lets you buy 24gb on "normal" gpus.
Meanwhile apple can put as much ram next to their soc's as they want (for 200$ per 4gb lmao), allowing them to load massive models.
Them: "we need 50 TOPS"
GPUs: I got 200-1200 TOPS
Them: "push out the 50 TOPS NPUs!"
People here making fun of your shaved beard but I say you look very young. More like you hit puberty. Very cute.
Exceptional video! I’ve never seen neural nets explained so brilliantly!
0:40 nope - physically, like AppleNeural engine, they are just co-processor on steroids. Application-Specific-Integrated-Circut. Problem is business model - software and applications.
Thanks to new research you do not need to use matrix multiplication to run ML models.
UGH! Tired of the scam-talk about AI.
Just like Cloud, cloud Cloud a few years back.
and they going to force you to use it just how your are probably using the cloud.
Yeah Affinity Photo rocks! Got tired of Adobe's cost and bloat. Affinity does everything I used Photoslop for!
If I can't buy a motherboard without an NPU I must remember to pry it off. AI forced down my throat makes me feel like a goose.
Seriously? Do you have any idea what the heck you're talking about?
It's built into the CPU dude.
The NPU will be a tiny module somewhere between the CPU and the iGPU underneath a processor’s integrated heat spreader, entirely separate from the motherboard until you place it into the socket and close the latch.
It will be in the cpu, but you should be able to disable it in bios
Hating something and showing ignorance at the same time. You have passed the human test.
I prefer to be true to myself, even at the hazard of incurring the ridicule of others, rather than to be false, and to incur my own abhorrence.
Being able to run AI locally can become a really, really big deal on the long term.
With a local processing unit, we may be able to choose which AI model we run; We're not dependent on the whims of big IT companies; We can observe how google search isn't as good now as it was in the past.
When cloud served AI start to loose it's neutrality because of financial incentives we'll see how much damage it can cause. It may sound silly now, I believe strongly that in the long term, personal NPUs is actually a very positive and important thing for democracy.
TBH, I personally think that image processing for webcams on laptops is reason enough to have an NPU. These new Snapdragon windows laptops have some of the best looking webcams in any laptop. People have a ton of calls with their laptops on battery, and being able to significantly improve image quality with an insignificant impact on battery life is worth a lot.
AI hype is like 5G hype, just another marketing gimmick
What? 5g isn’t a marketing gimmick. It’s just faster mobile internet.
I think it's more analogous to something like when the PC industry starting putting "VR Ready!" on every single PC part in 2017. No, you don't need a label on your power supply saying it's VR compatible lol
An NPU isn't a higher priority in a smaller vs. bigger chip regardless of the percentage of die space it takes up. The priority for the operation of the devices they go in is the same, which is can they do the AI ops fast enough. Considering the size of the device doesn't matter since they're solving the same types of problems the NPU doesn't HAVE to get bigger for bigger die.
Now, give AI functionality in different software to hit a point to where users WANT to use different features then the NPU die space may need to get bigger, but it makes no sense to have a big NPU when most people don't want to use AI functionality..
So it's not really a tradeoff being made about die space for the NPU. It's a tradeoff being made between the typical person who right now may not use any AI functionality in software and others who do.
Mozilla(Firefox) have just released LLM that runs on General CPU's/ LLM Files so no special GPU or NPU required
Any LLM can run in the CPU. It isn't any special. Using GPU/NPU just makes them faster/efficient
@@swarnavasamanta2628I'm very hungry now.
@@alicethegrinsecatz6011 So am I, deal with it.
Just a hunch, but I think they'll become essential for resource management and security. There's a lot of use for something that can have a "feeling" about misbehaving software, hardware, and security risks.
*N O .*
*We need CPUs that dont eat the battery like Chrome eats ram and don't crash like Windows Me.* * _coughs_ * Intel * _coughs_ *
Intel - degradation inside!^^
i9-13900HX has twice the battery life of a Ryzen 9 7000 series mobile while watching youtube videos (Jarrod's Tech). Some of those chips can look very competitive vs Apple's stuff, AMD also has nice chips.
No I am not sick of hearing about onboard on-device AI and NPUs. That's why I am watching your video, LOL.
Same😂
So done with AI
By accepting yourself and being fully what you are, your presence can make others happy.