Never seen anyone run it using the NPU or GPU at a reasonable speed. Especially a 13 billion model. I wonder how it would do on a modern Phi-3 mini model.
I don't think there's an AI problem with Linux. I run several different ones on my GPU. However, my outlook is that the NPU should be there if you want to use it; it shouldn't have to be used simply because it's there.
Precisely. And it should be optional. I may (at some point) elect to install a LLM on my linux distro. But that's the crux of it - it would be MY CHOICE
The problem is NOT AI. The problem is that a closed source AI or LLM has potential for bias (ask a LLM if it can give you a fully unbiased result - it's "interesting"). Any opensource model has higher potential for scrutiny.
I'm reminded of AOLs search engine, which produced very different search results for the Republican and Democrat party. Don't remember which one was comically obviously favoured.
All the AIs are biased from origin, because all the datasets are biased and always will be. But work is done to reduce that bias (or at least stir it towards the ideas we support). Anyone complaining about censorship and bias in LLMs is just angry because the properly fine-tuned LLM they use doesn't output the racist or illegal stuff they want.
The problem is AI is not a thing that will ever be real and is more of a marketing gimmick being used to violate privacy even more, just like Crypto/web3/blockchain bs.
@@ronilevarez901 "ideas we support" There's the issue right there. Who's we, what ideas? etc. Thinking you are 100% correct and your ideas are deserving of support is incredibly arrogant.
@@pylotlight "The one with The Truth deserves to be arrogant". C'mon, let's face it. World is made of sheeps. They need a shepherd. Along History, the few with a vision have had to take the reins and lead the many towards whatever he thinks is the right way for the world to be, correct or not. This is not a democracy. It's a bunch of companies building products. You want to control them because this product has the potential to reshape your entire world in a way you might not like it? Talk to your politicians. That's the established method. But complaining about someone's views and ideas just because you don't like them and you use a product made by those people? Nah. (Btw , I'm looking for funding to create an AI world leader. Donations welcome).
I think the 'conundrum' Linux faces is what makes Linux great, in a 'free as in freedom' perspective. Microsoft will likely force users to use their AI 'tools,' so that they harvest information from users to ultimately profit from that data (and who knows what else). Linux will allow users to choose if and how much AI they want to use... you install the tools you want. There may be a bit of a learning curve installing these tools (or even getting hardware setup to use these tools), at least at first, but people will still be able to choose AI tooling that is useful to them. I think this will be especially useful and beneficial for creatives who will find Microsoft's tooling invasive and Apple's cost of entry too high for what they get (A Mac Pro without the ability to add a discrete GPU?). The big silicon companies (Nvidia, AMD, Intel, etc.) are all working on support for their NPU's (it would seem like a good idea for Nvidia to jump into this market, now that I think about it). Phoronix keeps pretty close tabs on AMD, Intel and Qualcomm's progress in supporting their NPUs on Linux. As far as Nvidia is concerned, I don't know if they will add ai support to future kernel drivers, considering their big competitive advantage is their proprietary CUDA platform, which many AI tools are tuned for out of the box.
I have not yet seen any use that I have for AI. I learned how to organize a file directory in the MS-DOS days, and I know where I save things. I do my own graphics and write my own words. The only time I talk to my computer is when I'm cussing at it. AI is developing into a powerful tool for some applications, and much good may it do people in those fields, but just as my refrigerator doesn't need an Internet connection, my computer doesn't need AI.
gentoo don't want llm generated code in their repo because the source the llm got it's code from is of dubious origin, you're conflating a lot of stuff
Linux has a bigger problem in the Wayland Age - accessibility tools. I’d want to see these tools updated or rewritten before considering new features. But I also know these are completely different realms of development that aren’t all that related unless we use the NPU to help fuel these tools…?
AI is like Cortana no body ask for it, no body wanted it, but microsoft forced it on everyone. NPU can be used for a lot of thing all AMD NPU's are on open drivers. on Linux it will be more on the app side to the system side as it possess a security risk.
I'll waste the NPU until the regulations and court cases are done. I have a lot of reasons for not wanting over hyped bots running on my system. We'll see but for now, nope. I don't care if a chip is dormant and unused.
@@Dave-rd6sp Good for you. As I said, I'll wait until some things shake out. I won't be using Microsoft spyware and if you want to put glue on your pizza, go for it. Your computers are yours, mine are mine. Not into having 20 chatbots installed and I don't find them useful yet. Enjoy your computing.
@@robinirie98 How you read me is entirely up to you. I wasn't angry or defensive. The person I responded to decided not to read what I wrote and tried to school me on how things are. My original comment wasn't intended for anything except to say how **I** feel. So, yeah, I said they could do what they want to do with their hardware/software and I would do what I wished. The glue part was from a response Gemini gave a user, I found it funny. It may have been to obscure for you to get the joke ... again, not my issue. And how I sound has nothing to do with my comments or the video. If you don't like forthrightness ... it is what it is. :)
AI is great for tasks where the goal can be tightly constrained-I use AI image upscaling a lot and I'd love it to be more convenient-but I think AI just sucks for open ended tasks. I was really impressed with generative and linguistic AI at first, but the more I've seen the less useful any of it looks. The output is just unreliable and low quality. It's usually just good enough to make you waste a bunch of time trying tweak it into something actually good. In the end I've decided it's easier just to do it myself. It's much like my experience the first time I drove an automatic transmission car. I'd become so used to manual shift that I found the automatic's help entirely unhelpful. Even after over twenty years of driving, I still find it takes more mental effort to drive an automatic because I can't stop myself from trying to manipulate the stupid machine into doing what I want it to do. Maybe if I'd never learned manual shift and I'd started out with the idea of letting the machine do the thinking it would be different. That's about how I feel about using AI to help me do things. I find the help unhelpful. Anyway, my computer is several years old and I'm still using Windows-and still hating it-because I want to keep using all my weird software. If stuff like AI upscaling gets incorporated conveniently into more software, I can see the benefit of having built in AI hardware but right now that's still in the category of not missing what I've never had.
1. Linux will have good support for Snapdragon Elite X, they are already working on it, so Linux will be able to use it. 2. AI integrated to the desktop the way Microsoft does it is a disaster and should be avoided in Linux at all cost.
Creators have been on Linux for ages - I switched full time in 2010 as a professional illustrator. In terms of extant AI - if you can barely write and you don't have a creative bone in your body, AI can help you suck less. If you're already competent at these things, it enshittifies your work.
well I think that is why so many people like ai. as the companies are sending it out. they suck at writting, art stuff. they either dont want to spend the time to get good or "cant" spend the time getting good at the writing/art (cause you know, adulting, busy lives with family, work etcc).
yes it is absolutely going to gather dust. On someone else's PC, because I'll still be using my old PC from 20 years ago when I moved over to Linux. That's more important than the spyware concern: people want out of the never-ending upgrades that bring no new benefits. And, with these videos, the concerns of people who think this way: buying a shiny new laptop and worrying the AI chip inside it is going to gather dust - are the concerns of slaves. They don't matter; their products don't matter; and their money doesn't matter.
Agreed. The option to avoid NPU-laced products won't last very long, before you're forced to use older hardware. The fact the average consumer is still tech-illiterate just means this AI tech is going to spread that much easier, with very little push back.
Using AI locally on my PC without issues, if I get a PC with an NPU I'm pretty sure I'll be able to use that too. The main difference been, I have full control, not Microsoft or Apple. Here is my AI (locally installed) apps. 1. Tabby in VS-Code, no need to pay for Microsoft's co-pilot. Using 4060 GPU. 2. Frigate DVR, AI powered object detection, using a Google Coral edge TPU, then linked to Home Assistant. 3. Krita with AI generative Fill, Create etc. No need for for some expensive paid subscription like Adobe etc. Using 4060 GPU. I could of course add more AI controlled stuff if I wanted, so really not sure why you think Linux has an AI Problem, I would say Windows / Apple have the AI problem as they control it, not you.
Know what? For over 20 years all my systems have an integrated FPU! And in my Linux deskop environments "this FPU is collecting dust" too! Do I care? - NO!
I'd love to hear your ideas on how it can be used. I suggested graphics applications and active real time scanning of the system for security issues, malware, etc. When GPUs started to come out, no one thought it would be used for crypto mining or machine learning. How could we use it to benefit desktop Linux?
@@JeremysTechChannel neither of those things are done on the NPU on windows, that's not what it is for. and it is in fact very bad at either of those tasks It is for training Large Language Models, or even machine learning. Both fields which are used extensively on So there is really not much use for it outside of that.........and maybe analysing and recalling everything a user does on a computer, which i'm sure is not what you have in mind. I would suggest you look into what an NPU is made for and what it does before claiming linux has a "AI Problem", it can't have a problem when there is no use case for it (for regular users)
@@JeremysTechChannel I've currently been building my own solution using function/tool calling for being able to run commands on my server or control my smart light setup with basic english. It's all my code so I know exactly how it all works. I do currently consume external AI services for part of this as none of the local ones are good enough yet, having NPU to assist in this would actually be super helpful as a customised automation agent.
@@fuseteam @JeremysTechChannel NPUs on end user PCs are mostly used for running not training. Malwarebytes has an option for using AI for scanning. And with NPUs being common every big dev that can train AI is going want to use it in some way. Search for recently uploaded videos on AI and Copilot+ PCs. AI Text Transcription & Translation is available on Windows with NPUs. Accessibility using Speech2Text and head gestures. Qualcom has shown this. Noise removal for audio, esp. live conferences or Youtubing. Audacity also supports this using NPUs now. AI Image & Video Upscaling using AI. Paid Video Editing Software are using it. They've shown demos of this. Image editing software are going to be using it. For eg. for Text2Img, Image modification using prompt, Background Removal, Changing the color of an object with AI, AI Retouching and so on. Davinci Resolve is already using it. Video & Image editing without AI is going be tedious like factory production vs manual crafting. Gaming: They have shown AI chatbots which give you details of what is going on in the game when you ask about it. And give you tips and more info and instructions if you want. AI Game upscaling with DLSS etc. AI Video upscaling for videos you are watching. No 8k content, no problem! The applications of AI are as endless as a CPU. The ones I mentioned are just some applications and this is just the beginning. On launch they have this & more, imagine after some years. And NPUs have on advantage. Low battery usage. So, it's perfect for Laptops.
What about using the NPU for surround sound applications? Or upscaling audio quality for really old music recordings? A lot of audiophiles might enjoy this if it's done right.
It's fixed function hardware intended to do exactly this, but more efficiently than brute forcing through regular CPU instructions. It's like the media accelerators in ARM SOCs and GPUs, that all it's intended to do. One of the elements that makes ARM so efficient is that core compute ability is super limited, so accelerators are added to do specific things that would otherwise be very intensive with general compute instructions. All the major ARM options already have NPUs.
The NPU is the answer. It should be used for beginners who have just came to Linux to help them with the CLI/terminal. Essentially, you press the penguin button on your keyboard and it helps you install programs or solve driver issues for your specific hardware. The hope is that it would help advance the adoption of Linux, instead of hinder it.
VI and Neural Networks have great potential in optimization type task (like designing the most optimal mechanical shape for something). And I've long wanted to get my engineering design off Windows - and I'm sure I'm not alone, so tools will be forthcoming.
I'm a Windows user and... what the hell is the problem ? As a gamer, I know that 80% of my CPU is dedicated to video tasks and is, therefore, gathering dust. Does it upset me ? No. In the end, it's a sort of a spiritual problem. Sometime ago, in America's history, a group of Christians said no to electricity : these were the Amish and are quite good at what they do. Nowadays, a new group will say no to upcoming AI technologies. Would there be local use of AI ? I heard about a 4 TB excel spreadsheet with an AI. And of course, if Nvidia is gonna have AI on video boards means textures are gonna teach some generative models to the AI embedded in the card... and these are going to be offline uses of the AI that could be getting on Linux as well. In the end, one does not have to pick between Windows and Linux. As I see things now, as a concerned Christian conservative, I'm starting to move towards a hypervisor and create specific machines based on what I want to do with a certain machine. There is no Linux vs Windows. There is a complementarity there. It's also a spiritual problem: you may simply "give in" to the AI if you start using it too much. You may get to like the Recall feature. And before you know it, the AI will become a part of your life, like that CEO in Westworld whose AI was dictating what he should do. And of course if you become like that, Linux will simply look antique to you afterwards.
I wonder how Apple's Neural Engine and the NPUs on present and future AI PCs compare, as I wonder whether development targeting the latter will also benefit Asahi Linux on the former.Ditto the integrated GPU architectures, I suppose.
I don't get the problem. "I want an NPU but I don't want to use it for NPU stuff." We don't even know what NPU stuff might even be. And I don't understand why you would fear for your privacy when running your own local AI. I think in the future, not using "AI" features will be like not using autocorrect (though only a moron would equate the two).
If AI comes to Linux operating systems i would rather be dead. This would mean that Microsoft, Open AI, Nividia, Apple and Google has poisoned everything in the computer world, and i do not wish to be part of that fiasco. It would also mean the evil ones have won.
It's a fair question. I have to say that right now at this point, I don't really envision people who buy for instance a premium Microsoft AI computer are the kinds of people who would suddenly become concerned about privacy and delete Windows and install Linux on the machine instead. I'm sure someone at some point might try, there's probably someone who has installed Linux on an old Surface Pro or something like that, just to show that it can be done, but I don't think that's going to be a common situation. My question is, in the case of premium NPUs though, is it something that a Linux user even could theoretically take advantage of? Are we assuming that someone can just write a Linux program that could take advantage of the NPU's capabilities and it would just work? If that's the case then all someone needs to do is just do it; but on a certain level I tend to expect there's going to be some proprietary locks or protections on these NPUs. Until the day arrives that you can just buy the new Ryzen NPU off the shelf and stick it in your own Gigabyte AM7 motherboard or whatnot.
I wont come back to windows. Nextcloud implemented AI, Ollama can replace Chat-GPT. Nvidia published Chat-RTX. Google and Apple also will get into AI. I Rather use macOS or have a little less options in opensource LLMA than using windows again.
I don't think the problem is AI. I've no problem with AI helper. I don't trust Microsoft, and their operating systems are like watching paint dry. I have a couple of systems that I've run Windows on ; a converted 10 year old HP chromebook, and an 8 year old AMD PC. Both run like absolute dogs under Windows, but are useable under Linux. The Chromebook is a bit slow, but not so you sit there and drum your fingers.
Funny argument here, are MS hardware THAT GOOD that you'd buy it and then switch it to Linux??? Unlikely, secondly if you live in say the Philippines are you distraught that your car has a heater that will never be used and go ape shit crazy about it? Think again, NO, I don't use all the features of my hardware either, but I'm not losing sleep over it. Seriously there are BETTER hardware options out there than MS and they are growing bigger each year, even machines that run Linux as Base with new and better architecture and reasonably priced. Dump Gates and his J's and company, forget NPU's and move on, don't need that heater, it's warm always here ;) and AI is growing so fast it'll be smart enough to run on ANY machines AND save resources while at it.
Speech to Text for example? It uses "AI" and requires lot of hardware resources. Translating documents? I just wrote a book about Linux Mint. More than a hundred pages, more than hundred photos (even animated GIF-s). And is Hungarian. A lot of screenshots are localized, and the notations on pictures (graphically embedded) are also in Hungarian. You may recognize only the arrows, and numbers from there. A few people asked me for translated version, but is too hard to me (it was a few month just to writing the original). I recommended them to translate it by AI solutions. Supposedly the AI can manipulate the photos in fantastic way. Guess can he translate the text on the photos to another language? Nope. Anyway, an AI model for specific task will be welcome in Linux but not for generally everything, not using by OS but by certain software strictly for a certain task. Like Speech to Text does.
I'm 1000% opposed to corporate AI, for privacy reasons, for ethical reasons, and for quality reasons. But I'd be happy having some of a self-hosted voice recognition home assistant that I run firewalled in on my own network, trained with my own data or some kind of open standard shared model. Take a memo, play a song, where are my keys, that sort of thing. Seeing as a I code and host my code exclusively on Linux, I'd be happy being able to harness the hardware if it's already there, and I dare say it would be a lot more efficient than running on a GPU. I'm fine with that level of "dumb" AI. For what it's worth, I'm also a creative, I paint, design and do a lot of 3d modelling in Blender, and I think generative AI is a tragic joke for desperate losers with low standards and no talent.
The biggest obstacle for doing anything AI in general is building an ethically sourced model. Everyone can argue until the cows come home about the merits of AI, but it's all moot if the databases that these models are being built upon are full of scraped webpages, images, and even extremely personal information (IDs medical information, etc) obtained without the sources' knowledge or consent :/ I personally couldn't give a damn whether or not I have an NPU gathering dust in my system. Never asked for it, and anyone who *would* want to use such hardware should be better served having the option to obtain it separately in the form of something like a PCIE card, rather than packaged into their CPU anyway :/
The npu is going to suffer the same thing as the GPU, certain applications will use them, and most of the time they will be idle. It would be nice if open source models could be run at OS level to have a Truely airgapped "cortana" on your system (perhaps Jarvis would be a better name cuz microsoft) that could correct code, and just offer suggestions in general application use. Having Idle components of silicon also isnt the worst thing in the world. But it WOULD be nice to make use of it.
AI tech alone is not the problem, AI model may start to be a problem but as long as you choose it I guess this is ok. In my sense there is at least 2 problems, first is the integration question, do we should integrate an AI (even a free one) into an operating system ? To be effective an AI will necessary have to keep some of your info (a minimal history that will let the AI keep the subject of the interaction and all the answers it could already told) or it will just be another not clever assistant, but in that case where the data will be saved ? Will it be uploaded to a server or in your own computer (using some disk space, how much ?), will it be encrypted ? With a common key for each system that use that AI on this particular distro or will it be encrypted locally with a passphrase you choose on your own, maybe based on hardware signature ? The second one is Why would you need an AI help you use a PC ? For the people that can't use a keyboard or any input device, as long as they can talk we can use AI to execute orders, it could be a cheapest and safest solution than implanting a chip in the patient brain. In this particular case I see a great benefit, but for general purposes if you can use a keyboard/mouse or controller it would just be way faster. AI is already usable in linux, my first experience with AI was chatGPT, my second experience is when I tried to use an AI chatbot on my own hardware with a free model, running on a pi4, was quite long but it worked.
We kind of already have a variety of type of AI's on our system already. Most people don't realize your virus scan is like a very, very crude AI. Your TTS is a type of AI and definitely your speech dictation. These virus scans and then we'll actually need this as viruses get super powerful. We'll probably use this chip for things like that.
Well, as someone who has worked with AI, i must say Linus is right. Current AI is overhyped and constantly falls short of expectations. It's almost like the compiler hype, when people though, we would only need to input a rough description of the Code for it to work, or fuzzy logics "We will never have to develop control systems again". Those were completely empty promises. Look, AI has it's use in some niche cases, but most of it will go away when the hype is gone.
I have an AI assistant in my Linux system since a half year or so ago. I took a relatively small model at the time a 7B parameters one (now you can use 1B ones with the same 'cleverness', I am using a 3B one) and gave it some tools using mostly the ones langchain provides and I few I wrote myself, so it can execute commands in my system and so. The current iteration runs in my CPU at very acceptable speed, using llama.cpp. The good thing is it does only run (and use battery) when I ask something to it. Over time, it has gained also access to a few websites I use for work, so it also searches for me, extending my searches and summarizing the results for me. It's actually quite a silly and direct thing with the current available tools.
What does not allowing AI bots has to do with utilizing or not utilizng NPU? Neural network has 100s of far cooler use than LLM. And NPUs can be similarly used as Linux users use their GPU.
NPUs are new. Linux will adapt to it like any new hardware in time. Developers just need to get their hands on it i give it 6 months tops before we start seeing distros that can utilize them in ways not even windows can. Its just like a GPU. GPUs are processors designed to do certain things. We now use them for all kinds of crazy stuff from crypto mining to hyper realistic 3d simulations. NPUs are just another type of processor. Yes theyre designed to work with AI models but i garuntee thats not the only thing theyre good for. The issue with the stuff Microsoft is doing isn't the hardware its the spyware software and the fact its a evil company holding the reigns. Once open source devs start tinkering with NPUs we will get all sorts of cool uses for them. My concern with it is security. Requiring all new devices to have these things is kinda insane all it takes is 1 hardware exploit (which in something new like this is bound to show up) and every single system has a security issue to deal with. I think theyre gonna regret not taking their time with this and doing it right, but if a 0 day shows up in the hardware of these things we can probably implement a software fix like we did with that one that showed up in CPUs a few years ago.
Buying a laptop that forces you to pay for Windows as part of the package is funding the enemy. Microsoft and a number of big tech companies care more about serving themselves than serving their customers. An independent open source AI that answers Linux questions would be the worst nightmare of proprietary Big Tech. And most likely a great boon to Linux.
we dont need "features" we just need the instruction set exposed. an kernel dont need to use it, just need to let the applications that do need it, use it. and linux already does, the entire research and development of AI was done on top of linux (just look at windows ai studio for example, you NEED to install an ubuntu VM to run it!) unlike windows that include a bunch of things, linux is just an kernel, and the OS vendor chose what to include with it, that is why linux torvalds dont have to worry about AI features, the ones who may care about it or not are the distro vendors. if companies like canonical think their users want those features they will develop, if they think the reason why people are flocking to linux is because they are afraid of AI, then they wont, linux is in a big advantage considering we will have options for both publics. if i understand that correctly, those npus are like asics for the current algorithms of AI, just like we have hardware decoders for video.
The problem for me is still GIGO. In my hobbies and interests, AI can't even correctly write and pronounce names unless they are English. I guess AI doesn't know because its programmers don't know and don't want to learn.
I would like to think that with linux and an NPU we can use it to effectivdly manage the other resources in the computer in a more complex manner. We can eventually write an algorythm for an NPU alowing for highly complex functions in a flexible manner. At worst at least the NPU will free up your graphics card.
The issue is, probably, just probably it makes you desktop experience a little easier, but at what cost. Privacy and Storage. And for Pro, who have been using the system the old, this is not even a feature. Just buy a mac.
I'm sure the npu will be explored by some utilities for better functionality. There will be a lot of wild concepts, after a while when the dust will settle, we'll have some great use cases for npu in linux. They'll be integrated in the mainstream distributions shortly thereafter. I don't think it'll take 10 years though. I hope to see some good utilities in next 1-3 years
AI itself is just a tool. It's the implementation of it that concerns me. Previous track records of Microsoft mean I wont trust it on Windows. However so long as I know exactly what it is doing on my machine, I would be happy to use it so long as I know I can disable it totally when I am doing things like banking etc.
No point in fretting over unused NPUs. AI is not worthy the worry. Linux and its users should just go along doing what they've been doing all along. The AI emperor has no clothes. The problem is nobody is looking past the corporate hype to see it yet.
If AI and Linux have a problem, then AI has to go. Don't they have a billion "good reasons" to violate your privacy, and never give you a way to completely turn it off.
I want a local AI Assistent in my Desktops which is able to see what is installed, what is dependent on what, has a API for apps to make data available for the AI (best would be if you have a Setting in each app on what to share with the AI Assistent) and a lot more. Problem is, who is gonna train that AI? And, yea, this is how appleAI works
I'm GLAD there's issues with AI and Linux!...I'm against this nonsensical push....this constant insistence that every thing that has electricity running through it need A.I. ENOUGH!....I will continue to "revert" backwards?..and use Laptops that have the ability to turn off UEFI / Secure Boot....I will continue to use Tor...and Linux distributions that help to return some bits of privacy to your computer usage. And to the devs who can't "see" it?....remember the movie "Jurasic Park".....you're all so busy trying to see if you COULD do it?....that NONE of you are questioning if you SHOULD!. Are you really desperate to see a car drive itself?...to see a robotic android device administer needles in the arms of Humans?...are you really so foolish that you cannot see your folly? Its ok....Icarus ALSO didn't see his folly?....until he flew too close to the sun and the wax holding his wings together MELTED!
People use their computer efficient before so called Ai or rather simulated intelligence. Though gpts and search engine have different internals, Isn't gpt just a glorified search engine, that's more expensive to manage
Why not use the LLM locally and train it to favor Linux and open source over other options, ask it to develop local firewalls and give algorithms for personal defense defense against external threats?
There is no AI Problem on Linux, Linux is used for AI. If AI is used on the desktop depends on the user. I guess many Linux users do not want it period.
AMD Ryzen AI also have NPU in their AMD XDNA which appears to have been the first dedicated AI processing silicon on a Windows x86 processor. Although it looks like you have to install an xdna plugin/driver to use the npu with linux or the npu sits there dormant/unused?
Leaving AI behind is the reason why I'm thinking about switching to Linux, is the thing. I don't care about a part the OEM forced on me "gathering dust".
Nice video, why don't they make a plugin as part of the operating system that a person can enable and choose what resources to give to ie that way some of your system is still private. A bit like a container, Linux container or docker container. The Linux community needs to develop along with AI but still keep privacy one of the fundamental foundations of linux
the true freedom is having the choice to use or not use private software on your OS. when Linux turn into the main OS thats will be the PRIME, all people will be FREE able to chose what tools to use or not use.
Linux still is a "personal" computer OS. Now you did say desktop Linux, not quantum science for the masses. A npu sidecar unit, like a server unit to add on if you want, DONE, not for me.
The question is if the tooling will be competitive with Windows or not and if anyone cares to utilize GAI/LLMs for normal workflows. If you think GAI is a bad thing the lack of integrations isn't a problem, but there is reasonable questions as to if there are enough developers that will find actually useful privacy-first open-source integrations that people will find actually useful or not. Right now a ton of GAI/LLMs are very effectively ran under Linux in general, but the integrations are still aimed towards super technical users. If LLMs settles down as just being an over-glorified chatbot as many think there would be a lot of wasted effort that could have been spent on a more refined desktop, but if things actually get to a point where AI is a "real" advantage, it wouldn't be good to ignore it and just let Microsoft/Apple be the only ones with decent well fleshed out solutions for normal end-users.
"what linux is going to do with npu" sounds like a nonsense question. Linux doesn't do anything with your computer, it allows you to do anything you want with it, that's what operating systems are for
It would be nice if certain programs took advantage of the NPU. Programs for graphics and audio and office suites to perform enhancements. All locally, of course, with occasional updates. Imagine having a local AI take a song idea, create lyrics, have the audio AI developer a music suggestion for it, then have it develop an album cover... maybe even a music video suggestion that can be further modified.
I generally look at it as 'where this capability is useful to my workload'. A lot of what's in proprietary OS'es is where its useful to corporations and by extension useful enough for us to tolerate it. I don't think it's a problem. It's really a matter of intentions. Note: A LOT of 'capabilities' in desktop computer hardware often take a while to get used or even 'enabled'. Every OS kernel arch doesn't necessarily support the full spectrum of underlying capabilities (and some come and go). When GPU's were 'new', hardly anything took advantage of it. I expect NPU's to follow the same track (specialized use cases, over hype, more practical general use cases over time until it's 'normalized'.... and corporate malfeasance along the way we won't want to 'normalize' on linux. aka: not a 'new' problem but just the typical 'uptake' until a technology is 'normalized').
I'm already using AI on a per need basis on my Linux system and driving it through nvidia cuda. The applications I use already support CUDA and i'm sure they will also support NPU as well, so I don't see a problem here for Linux. So on Linux you can already run a language model like chat-gpt locally as well as image generation and other applications. As time moves on especially with the introduction of the new Qualcomm snapdragon which already supports Linux, we'll see even more support for AI on Linux. Point is, if the application supports NPU or CUDA then you have no problem running AI regardless of the operating system ( Windows / Mac / Linux ). Also another point is that if you want to run AI locally or manually for privacy reasons, you'll have to set it up yourself anyway regardless of the operating system. People have been running local AI on Windows, Mac and Linux already for the past year, take the most popular ones like Ollama and Stable Diffusion for example. So, I don't see any problem here at all. If you want privacy you take responsibility and you take care of yourself, period.
There is no such thing as AI, therefore Linux cannot possibly have a problem with it. What we have is machine learning that is being hyped up by many, mostly to push their own approaches which all are just more or less advanced versions of Joseph Weizenbaum's Eliza that he wrote back in the 1960s. By now, most of what can be found on the internet has been feed to those machine learning systems by thousands of underpaid humans and without that input, not one of them would be able to even do the silly things they do now and for which they consume more and more energy in times of a massive climate catastrophy where we should focus on saving energy, not wasting even more. Therefore, any chip dedicated to machine learning that just collects dust is a better one that wasted more energy. Even better would be if no resources had been wasted to produce it in the first place. Although almost all was fed to these systems, they did not develop anything even remotely intelligent but rather show more and more how useless they are because garbage in produces garbage out and the more fake news (garbage) they produce that is then fed to the other systems, the more it will lead to a downward spiral. Two very simple examples why nothing in machine learning has anything to do with intelligence: - Any chess robot will most likely beat any human at playing chess but should s fire break out near it, it would keep playing chess until it is burnt down since it has no concept of fire or what it could do. - Any robot that was taught to pull something using a rope has no idea that it cannot push the same thing using that same rope because it has no ideas at all in the first place but it also has no concept of what a rope actually is.
I respectfully disagree. Where code can re-write itself beyond the interpretation and understanding of its programmers (and yes this has happened with stock trading algorithms) it has potential to become "self-aware" and therefore meet the definition of an artifical intelligence. And no, I'm not suggesting a stock trading algorithm isn an AI- I'm using it as an example of how software can grow beyond the programer's ability to predict and interpret it
@@maxanderson9187 We are being told that actual AI is just around the corner, yes, now really! since Eliza was written and it remains as far away as it always was. Code changing in ways that programmers do not understand has nothing to do with self-awareness at all, it merely shows that the code added more code that it found somewhere without any kind of understanding what that code might do. Then humans have to check what the new code does and to tell the machine learning system if this was an improvement or rather garbage. Of course, only the human-approved improved code is then used to do the next random changes. Otherwise, it would quickly degenerate again. Remember, garbage in - garbage out and no, the models cannot decide themselves if what they produced is garbage or genius.
NPU just dosent seem like it would be powerful enough to do anything. Im rocking an RTX3060 12gb and it can just barely manage stable diffusion and llama3 8b. My thoughts are that running inference on a onboard NPU that shares system memory will make it very slow for the limited number of applications that it can run... So my conclusion is to just pick up a K80 or something like that if you want to run inference as youll get much more bang for your buck.
What is the AI problem? This was so frustrating to listen to because there was a lot of dancing around but no statement of what the problem actually is.
Speed up processing for what specifically? If you're meaning general processing to help the CPU, then I don't think it wouldn't help much due to the latency and inefficiency. Just like we don't use graphics cards for general processing.
@@pigeonprotagonist6357 lol i know nothing about the npu lol i just know how since i started using linux, it seems like everything can get repurposed lol
if your concern is about npu support under linux, it will land fine
the snapdragon elite x is already being upstreamed
The NPU is nothing new. I have used the NPU in the ARM Rockchip RK3588 with Linux to run Llama2:13b locally. And that chip is almost 3 years old.
same Linux Desktop users already using it, I am using AI in gimp using openvino-ai-plugins-gimp
Never seen anyone run it using the NPU or GPU at a reasonable speed. Especially a 13 billion model. I wonder how it would do on a modern Phi-3 mini model.
I don't think there's an AI problem with Linux. I run several different ones on my GPU. However, my outlook is that the NPU should be there if you want to use it; it shouldn't have to be used simply because it's there.
Precisely. And it should be optional. I may (at some point) elect to install a LLM on my linux distro. But that's the crux of it - it would be MY CHOICE
The problem is NOT AI. The problem is that a closed source AI or LLM has potential for bias (ask a LLM if it can give you a fully unbiased result - it's "interesting"). Any opensource model has higher potential for scrutiny.
I'm reminded of AOLs search engine, which produced very different search results for the Republican and Democrat party. Don't remember which one was comically obviously favoured.
All the AIs are biased from origin, because all the datasets are biased and always will be. But work is done to reduce that bias (or at least stir it towards the ideas we support).
Anyone complaining about censorship and bias in LLMs is just angry because the properly fine-tuned LLM they use doesn't output the racist or illegal stuff they want.
The problem is AI is not a thing that will ever be real and is more of a marketing gimmick being used to violate privacy even more, just like Crypto/web3/blockchain bs.
@@ronilevarez901 "ideas we support" There's the issue right there. Who's we, what ideas? etc. Thinking you are 100% correct and your ideas are deserving of support is incredibly arrogant.
@@pylotlight "The one with The Truth deserves to be arrogant".
C'mon, let's face it. World is made of sheeps. They need a shepherd.
Along History, the few with a vision have had to take the reins and lead the many towards whatever he thinks is the right way for the world to be, correct or not. This is not a democracy. It's a bunch of companies building products. You want to control them because this product has the potential to reshape your entire world in a way you might not like it? Talk to your politicians. That's the established method. But complaining about someone's views and ideas just because you don't like them and you use a product made by those people? Nah.
(Btw , I'm looking for funding to create an AI world leader. Donations welcome).
I think the 'conundrum' Linux faces is what makes Linux great, in a 'free as in freedom' perspective. Microsoft will likely force users to use their AI 'tools,' so that they harvest information from users to ultimately profit from that data (and who knows what else). Linux will allow users to choose if and how much AI they want to use... you install the tools you want. There may be a bit of a learning curve installing these tools (or even getting hardware setup to use these tools), at least at first, but people will still be able to choose AI tooling that is useful to them. I think this will be especially useful and beneficial for creatives who will find Microsoft's tooling invasive and Apple's cost of entry too high for what they get (A Mac Pro without the ability to add a discrete GPU?).
The big silicon companies (Nvidia, AMD, Intel, etc.) are all working on support for their NPU's (it would seem like a good idea for Nvidia to jump into this market, now that I think about it). Phoronix keeps pretty close tabs on AMD, Intel and Qualcomm's progress in supporting their NPUs on Linux.
As far as Nvidia is concerned, I don't know if they will add ai support to future kernel drivers, considering their big competitive advantage is their proprietary CUDA platform, which many AI tools are tuned for out of the box.
I have not yet seen any use that I have for AI. I learned how to organize a file directory in the MS-DOS days, and I know where I save things. I do my own graphics and write my own words. The only time I talk to my computer is when I'm cussing at it.
AI is developing into a powerful tool for some applications, and much good may it do people in those fields, but just as my refrigerator doesn't need an Internet connection, my computer doesn't need AI.
gentoo don't want llm generated code in their repo because the source the llm got it's code from is of dubious origin, you're conflating a lot of stuff
Linux has a bigger problem in the Wayland Age - accessibility tools. I’d want to see these tools updated or rewritten before considering new features.
But I also know these are completely different realms of development that aren’t all that related unless we use the NPU to help fuel these tools…?
AI is like Cortana no body ask for it, no body wanted it, but microsoft forced it on everyone. NPU can be used for a lot of thing all AMD NPU's are on open drivers. on Linux it will be more on the app side to the system side as it possess a security risk.
I'll waste the NPU until the regulations and court cases are done. I have a lot of reasons for not wanting over hyped bots running on my system. We'll see but for now, nope. I don't care if a chip is dormant and unused.
AI has already become >50% of my PC time. Far from overhyped and still accelerating.
@@Dave-rd6sp Good for you. As I said, I'll wait until some things shake out. I won't be using Microsoft spyware and if you want to put glue on your pizza, go for it. Your computers are yours, mine are mine. Not into having 20 chatbots installed and I don't find them useful yet. Enjoy your computing.
@@bitterseedsyou sound angry and defensive
@@robinirie98 How you read me is entirely up to you. I wasn't angry or defensive. The person I responded to decided not to read what I wrote and tried to school me on how things are. My original comment wasn't intended for anything except to say how **I** feel. So, yeah, I said they could do what they want to do with their hardware/software and I would do what I wished. The glue part was from a response Gemini gave a user, I found it funny. It may have been to obscure for you to get the joke ... again, not my issue. And how I sound has nothing to do with my comments or the video. If you don't like forthrightness ... it is what it is. :)
@@bitterseeds Apologies if I got you wrong
AI is great for tasks where the goal can be tightly constrained-I use AI image upscaling a lot and I'd love it to be more convenient-but I think AI just sucks for open ended tasks. I was really impressed with generative and linguistic AI at first, but the more I've seen the less useful any of it looks. The output is just unreliable and low quality. It's usually just good enough to make you waste a bunch of time trying tweak it into something actually good. In the end I've decided it's easier just to do it myself.
It's much like my experience the first time I drove an automatic transmission car. I'd become so used to manual shift that I found the automatic's help entirely unhelpful. Even after over twenty years of driving, I still find it takes more mental effort to drive an automatic because I can't stop myself from trying to manipulate the stupid machine into doing what I want it to do. Maybe if I'd never learned manual shift and I'd started out with the idea of letting the machine do the thinking it would be different. That's about how I feel about using AI to help me do things. I find the help unhelpful.
Anyway, my computer is several years old and I'm still using Windows-and still hating it-because I want to keep using all my weird software. If stuff like AI upscaling gets incorporated conveniently into more software, I can see the benefit of having built in AI hardware but right now that's still in the category of not missing what I've never had.
NPU support is already there, I am using serveral turing rk1 and coral devices to accelerate lot of scientific research in Desktop Linux
This is exciting and interesting. Thanks for sharing.
Cool man. Can you guide me to proper Chip?
Please provide some links
1. Linux will have good support for Snapdragon Elite X, they are already working on it, so Linux will be able to use it.
2. AI integrated to the desktop the way Microsoft does it is a disaster and should be avoided in Linux at all cost.
Creators have been on Linux for ages - I switched full time in 2010 as a professional illustrator. In terms of extant AI - if you can barely write and you don't have a creative bone in your body, AI can help you suck less. If you're already competent at these things, it enshittifies your work.
well I think that is why so many people like ai. as the companies are sending it out. they suck at writting, art stuff. they either dont want to spend the time to get good or "cant" spend the time getting good at the writing/art (cause you know, adulting, busy lives with family, work etcc).
Dude you come across as a little crazy. No one wants AI in their Linux. Only you. Please no one listen to this guy...
I bet games will start leaning on npus for offloading npc ai eventually
Agreed, I want my own personal AI model on my Linux machine!
I run my own AI locally and I control it. atm use a ChatGPT like interface to interact with it would be nice if I could integrate it with my desktop.
yes it is absolutely going to gather dust. On someone else's PC, because I'll still be using my old PC from 20 years ago when I moved over to Linux. That's more important than the spyware concern: people want out of the never-ending upgrades that bring no new benefits. And, with these videos, the concerns of people who think this way: buying a shiny new laptop and worrying the AI chip inside it is going to gather dust - are the concerns of slaves. They don't matter; their products don't matter; and their money doesn't matter.
Agreed. The option to avoid NPU-laced products won't last very long, before you're forced to use older hardware. The fact the average consumer is still tech-illiterate just means this AI tech is going to spread that much easier, with very little push back.
the "new" benefits seem to be just slightly better versions of things past or looks a little different.
I don't really care for the idea of ai. I hope that NPU could be used as storage or for better graphics.
Using AI locally on my PC without issues, if I get a PC with an NPU I'm pretty sure I'll be able to use that too. The main difference been, I have full control, not Microsoft or Apple.
Here is my AI (locally installed) apps.
1. Tabby in VS-Code, no need to pay for Microsoft's co-pilot. Using 4060 GPU.
2. Frigate DVR, AI powered object detection, using a Google Coral edge TPU, then linked to Home Assistant.
3. Krita with AI generative Fill, Create etc. No need for for some expensive paid subscription like Adobe etc. Using 4060 GPU.
I could of course add more AI controlled stuff if I wanted, so really not sure why you think Linux has an AI Problem, I would say Windows / Apple have the AI problem as they control it, not you.
Know what? For over 20 years all my systems have an integrated FPU! And in my Linux deskop environments "this FPU is collecting dust" too! Do I care? - NO!
you say "i don't want the NPU to be unused" but what do want to use it for?
train a llm? probably already work under linux
.......what else?
I'd love to hear your ideas on how it can be used. I suggested graphics applications and active real time scanning of the system for security issues, malware, etc. When GPUs started to come out, no one thought it would be used for crypto mining or machine learning. How could we use it to benefit desktop Linux?
@@JeremysTechChannel neither of those things are done on the NPU on windows, that's not what it is for. and it is in fact very bad at either of those tasks
It is for training Large Language Models, or even machine learning. Both fields which are used extensively on So there is really not much use for it outside of that.........and maybe analysing and recalling everything a user does on a computer, which i'm sure is not what you have in mind.
I would suggest you look into what an NPU is made for and what it does before claiming linux has a "AI Problem", it can't have a problem when there is no use case for it (for regular users)
@@JeremysTechChannel I've currently been building my own solution using function/tool calling for being able to run commands on my server or control my smart light setup with basic english. It's all my code so I know exactly how it all works. I do currently consume external AI services for part of this as none of the local ones are good enough yet, having NPU to assist in this would actually be super helpful as a customised automation agent.
@@fuseteam @JeremysTechChannel NPUs on end user PCs are mostly used for running not training.
Malwarebytes has an option for using AI for scanning. And with NPUs being common every big dev that can train AI is going want to use it in some way.
Search for recently uploaded videos on AI and Copilot+ PCs.
AI Text Transcription & Translation is available on Windows with NPUs.
Accessibility using Speech2Text and head gestures. Qualcom has shown this.
Noise removal for audio, esp. live conferences or Youtubing. Audacity also supports this using NPUs now.
AI Image & Video Upscaling using AI.
Paid Video Editing Software are using it. They've shown demos of this. Image editing software are going to be using it. For eg. for Text2Img, Image modification using prompt, Background Removal, Changing the color of an object with AI, AI Retouching and so on. Davinci Resolve is already using it. Video & Image editing without AI is going be tedious like factory production vs manual crafting.
Gaming: They have shown AI chatbots which give you details of what is going on in the game when you ask about it. And give you tips and more info and instructions if you want.
AI Game upscaling with DLSS etc.
AI Video upscaling for videos you are watching. No 8k content, no problem!
The applications of AI are as endless as a CPU. The ones I mentioned are just some applications and this is just the beginning. On launch they have this & more, imagine after some years.
And NPUs have on advantage. Low battery usage. So, it's perfect for Laptops.
What about using the NPU for surround sound applications? Or upscaling audio quality for really old music recordings? A lot of audiophiles might enjoy this if it's done right.
The NPU doesn't have to explicitly be used for AI. Assuming so is a gross oversimplification. Those units can be put to (arguably better) other uses.
Thank you, yes this.
NPU is for AI and nothing more. But AI can almost do anything now.
It's fixed function hardware intended to do exactly this, but more efficiently than brute forcing through regular CPU instructions. It's like the media accelerators in ARM SOCs and GPUs, that all it's intended to do. One of the elements that makes ARM so efficient is that core compute ability is super limited, so accelerators are added to do specific things that would otherwise be very intensive with general compute instructions. All the major ARM options already have NPUs.
The NPU is the answer. It should be used for beginners who have just came to Linux to help them with the CLI/terminal. Essentially, you press the penguin button on your keyboard and it helps you install programs or solve driver issues for your specific hardware. The hope is that it would help advance the adoption of Linux, instead of hinder it.
Oh that's a cool idea. Even help recall commands and how to use them for more experience users.
And run scripts on vocal command
more like Linux doesn't hype AI support 😅, even Adobe demoed their AI features in Ubuntu last year, lot of SBC also has NPU and linux only
VI and Neural Networks have great potential in optimization type task (like designing the most optimal mechanical shape for something). And I've long wanted to get my engineering design off Windows - and I'm sure I'm not alone, so tools will be forthcoming.
I'm a Windows user and... what the hell is the problem ? As a gamer, I know that 80% of my CPU is dedicated to video tasks and is, therefore, gathering dust. Does it upset me ? No. In the end, it's a sort of a spiritual problem. Sometime ago, in America's history, a group of Christians said no to electricity : these were the Amish and are quite good at what they do. Nowadays, a new group will say no to upcoming AI technologies. Would there be local use of AI ? I heard about a 4 TB excel spreadsheet with an AI. And of course, if Nvidia is gonna have AI on video boards means textures are gonna teach some generative models to the AI embedded in the card... and these are going to be offline uses of the AI that could be getting on Linux as well. In the end, one does not have to pick between Windows and Linux. As I see things now, as a concerned Christian conservative, I'm starting to move towards a hypervisor and create specific machines based on what I want to do with a certain machine. There is no Linux vs Windows. There is a complementarity there. It's also a spiritual problem: you may simply "give in" to the AI if you start using it too much. You may get to like the Recall feature. And before you know it, the AI will become a part of your life, like that CEO in Westworld whose AI was dictating what he should do. And of course if you become like that, Linux will simply look antique to you afterwards.
I wonder how Apple's Neural Engine and the NPUs on present and future AI PCs compare, as I wonder whether development targeting the latter will also benefit Asahi Linux on the former.Ditto the integrated GPU architectures, I suppose.
I don't get the problem. "I want an NPU but I don't want to use it for NPU stuff." We don't even know what NPU stuff might even be. And I don't understand why you would fear for your privacy when running your own local AI. I think in the future, not using "AI" features will be like not using autocorrect (though only a moron would equate the two).
If AI comes to Linux operating systems i would rather be dead. This would mean that Microsoft, Open AI, Nividia, Apple and Google has poisoned everything in the computer world, and i do not wish to be part of that fiasco. It would also mean the evil ones have won.
And then people thought Windows 11 did add stuff people didn’t ask for. Maybe then MacOS should be the final answer if Linux won’t be it.
It's a fair question. I have to say that right now at this point, I don't really envision people who buy for instance a premium Microsoft AI computer are the kinds of people who would suddenly become concerned about privacy and delete Windows and install Linux on the machine instead. I'm sure someone at some point might try, there's probably someone who has installed Linux on an old Surface Pro or something like that, just to show that it can be done, but I don't think that's going to be a common situation.
My question is, in the case of premium NPUs though, is it something that a Linux user even could theoretically take advantage of? Are we assuming that someone can just write a Linux program that could take advantage of the NPU's capabilities and it would just work? If that's the case then all someone needs to do is just do it; but on a certain level I tend to expect there's going to be some proprietary locks or protections on these NPUs. Until the day arrives that you can just buy the new Ryzen NPU off the shelf and stick it in your own Gigabyte AM7 motherboard or whatnot.
I wont come back to windows. Nextcloud implemented AI, Ollama can replace Chat-GPT. Nvidia published Chat-RTX. Google and Apple also will get into AI. I Rather use macOS or have a little less options in opensource LLMA than using windows again.
I don't think the problem is AI. I've no problem with AI helper. I don't trust Microsoft, and their operating systems are like watching paint dry. I have a couple of systems that I've run Windows on ; a converted 10 year old HP chromebook, and an 8 year old AMD PC. Both run like absolute dogs under Windows, but are useable under Linux. The Chromebook is a bit slow, but not so you sit there and drum your fingers.
Funny argument here, are MS hardware THAT GOOD that you'd buy it and then switch it to Linux??? Unlikely, secondly if you live in say the Philippines are you distraught that your car has a heater that will never be used and go ape shit crazy about it? Think again, NO, I don't use all the features of my hardware either, but I'm not losing sleep over it. Seriously there are BETTER hardware options out there than MS and they are growing bigger each year, even machines that run Linux as Base with new and better architecture and reasonably priced. Dump Gates and his J's and company, forget NPU's and move on, don't need that heater, it's warm always here ;) and AI is growing so fast it'll be smart enough to run on ANY machines AND save resources while at it.
Speech to Text for example? It uses "AI" and requires lot of hardware resources. Translating documents? I just wrote a book about Linux Mint. More than a hundred pages, more than hundred photos (even animated GIF-s). And is Hungarian. A lot of screenshots are localized, and the notations on pictures (graphically embedded) are also in Hungarian. You may recognize only the arrows, and numbers from there. A few people asked me for translated version, but is too hard to me (it was a few month just to writing the original). I recommended them to translate it by AI solutions. Supposedly the AI can manipulate the photos in fantastic way. Guess can he translate the text on the photos to another language? Nope.
Anyway, an AI model for specific task will be welcome in Linux but not for generally everything, not using by OS but by certain software strictly for a certain task. Like Speech to Text does.
I'm 1000% opposed to corporate AI, for privacy reasons, for ethical reasons, and for quality reasons. But I'd be happy having some of a self-hosted voice recognition home assistant that I run firewalled in on my own network, trained with my own data or some kind of open standard shared model. Take a memo, play a song, where are my keys, that sort of thing. Seeing as a I code and host my code exclusively on Linux, I'd be happy being able to harness the hardware if it's already there, and I dare say it would be a lot more efficient than running on a GPU. I'm fine with that level of "dumb" AI.
For what it's worth, I'm also a creative, I paint, design and do a lot of 3d modelling in Blender, and I think generative AI is a tragic joke for desperate losers with low standards and no talent.
The biggest obstacle for doing anything AI in general is building an ethically sourced model. Everyone can argue until the cows come home about the merits of AI, but it's all moot if the databases that these models are being built upon are full of scraped webpages, images, and even extremely personal information (IDs medical information, etc) obtained without the sources' knowledge or consent :/
I personally couldn't give a damn whether or not I have an NPU gathering dust in my system. Never asked for it, and anyone who *would* want to use such hardware should be better served having the option to obtain it separately in the form of something like a PCIE card, rather than packaged into their CPU anyway :/
The npu is going to suffer the same thing as the GPU, certain applications will use them, and most of the time they will be idle. It would be nice if open source models could be run at OS level to have a Truely airgapped "cortana" on your system (perhaps Jarvis would be a better name cuz microsoft) that could correct code, and just offer suggestions in general application use. Having Idle components of silicon also isnt the worst thing in the world. But it WOULD be nice to make use of it.
AI tech alone is not the problem, AI model may start to be a problem but as long as you choose it I guess this is ok.
In my sense there is at least 2 problems, first is the integration question, do we should integrate an AI (even a free one) into an operating system ? To be effective an AI will necessary have to keep some of your info (a minimal history that will let the AI keep the subject of the interaction and all the answers it could already told) or it will just be another not clever assistant, but in that case where the data will be saved ? Will it be uploaded to a server or in your own computer (using some disk space, how much ?), will it be encrypted ? With a common key for each system that use that AI on this particular distro or will it be encrypted locally with a passphrase you choose on your own, maybe based on hardware signature ?
The second one is Why would you need an AI help you use a PC ? For the people that can't use a keyboard or any input device, as long as they can talk we can use AI to execute orders, it could be a cheapest and safest solution than implanting a chip in the patient brain. In this particular case I see a great benefit, but for general purposes if you can use a keyboard/mouse or controller it would just be way faster.
AI is already usable in linux, my first experience with AI was chatGPT, my second experience is when I tried to use an AI chatbot on my own hardware with a free model, running on a pi4, was quite long but it worked.
We kind of already have a variety of type of AI's on our system already. Most people don't realize your virus scan is like a very, very crude AI. Your TTS is a type of AI and definitely your speech dictation. These virus scans and then we'll actually need this as viruses get super powerful. We'll probably use this chip for things like that.
Well, as someone who has worked with AI, i must say Linus is right. Current AI is overhyped and constantly falls short of expectations. It's almost like the compiler hype, when people though, we would only need to input a rough description of the Code for it to work, or fuzzy logics "We will never have to develop control systems again".
Those were completely empty promises.
Look, AI has it's use in some niche cases, but most of it will go away when the hype is gone.
I have an AI assistant in my Linux system since a half year or so ago. I took a relatively small model at the time a 7B parameters one (now you can use 1B ones with the same 'cleverness', I am using a 3B one) and gave it some tools using mostly the ones langchain provides and I few I wrote myself, so it can execute commands in my system and so. The current iteration runs in my CPU at very acceptable speed, using llama.cpp. The good thing is it does only run (and use battery) when I ask something to it. Over time, it has gained also access to a few websites I use for work, so it also searches for me, extending my searches and summarizing the results for me. It's actually quite a silly and direct thing with the current available tools.
What does not allowing AI bots has to do with utilizing or not utilizng NPU? Neural network has 100s of far cooler use than LLM. And NPUs can be similarly used as Linux users use their GPU.
NPUs are new. Linux will adapt to it like any new hardware in time. Developers just need to get their hands on it i give it 6 months tops before we start seeing distros that can utilize them in ways not even windows can. Its just like a GPU. GPUs are processors designed to do certain things. We now use them for all kinds of crazy stuff from crypto mining to hyper realistic 3d simulations. NPUs are just another type of processor. Yes theyre designed to work with AI models but i garuntee thats not the only thing theyre good for. The issue with the stuff Microsoft is doing isn't the hardware its the spyware software and the fact its a evil company holding the reigns. Once open source devs start tinkering with NPUs we will get all sorts of cool uses for them.
My concern with it is security. Requiring all new devices to have these things is kinda insane all it takes is 1 hardware exploit (which in something new like this is bound to show up) and every single system has a security issue to deal with. I think theyre gonna regret not taking their time with this and doing it right, but if a 0 day shows up in the hardware of these things we can probably implement a software fix like we did with that one that showed up in CPUs a few years ago.
Buying a laptop that forces you to pay for Windows as part of the package is funding the enemy. Microsoft and a number of big tech companies care more about serving themselves than serving their customers. An independent open source AI that answers Linux questions would be the worst nightmare of proprietary Big Tech. And most likely a great boon to Linux.
Is NPU capable of doing something like FSR or DLSS stuff? Then let be it! I want more "free power" of my GPU to make better graphics in my games :)
we dont need "features" we just need the instruction set exposed.
an kernel dont need to use it, just need to let the applications that do need it, use it.
and linux already does, the entire research and development of AI was done on top of linux (just look at windows ai studio for example, you NEED to install an ubuntu VM to run it!)
unlike windows that include a bunch of things, linux is just an kernel, and the OS vendor chose what to include with it, that is why linux torvalds dont have to worry about AI features, the ones who may care about it or not are the distro vendors.
if companies like canonical think their users want those features they will develop, if they think the reason why people are flocking to linux is because they are afraid of AI, then they wont, linux is in a big advantage considering we will have options for both publics.
if i understand that correctly, those npus are like asics for the current algorithms of AI, just like we have hardware decoders for video.
The problem for me is still GIGO. In my hobbies and interests, AI can't even correctly write and pronounce names unless they are English. I guess AI doesn't know because its programmers don't know and don't want to learn.
I would like to think that with linux and an NPU we can use it to effectivdly manage the other resources in the computer in a more complex manner. We can eventually write an algorythm for an NPU alowing for highly complex functions in a flexible manner.
At worst at least the NPU will free up your graphics card.
Real AI is fake and crap. WHY would anyone want a bot (or underpaid person) guessing your thoughts (after breaking much of your privacy, etc??
The issue is, probably, just probably it makes you desktop experience a little easier, but at what cost. Privacy and Storage.
And for Pro, who have been using the system the old, this is not even a feature.
Just buy a mac.
I'm sure the npu will be explored by some utilities for better functionality. There will be a lot of wild concepts, after a while when the dust will settle, we'll have some great use cases for npu in linux. They'll be integrated in the mainstream distributions shortly thereafter. I don't think it'll take 10 years though. I hope to see some good utilities in next 1-3 years
AI itself is just a tool. It's the implementation of it that concerns me. Previous track records of Microsoft mean I wont trust it on Windows. However so long as I know exactly what it is doing on my machine, I would be happy to use it so long as I know I can disable it totally when I am doing things like banking etc.
No point in fretting over unused NPUs. AI is not worthy the worry. Linux and its users should just go along doing what they've been doing all along. The AI emperor has no clothes. The problem is nobody is looking past the corporate hype to see it yet.
If AI and Linux have a problem, then AI has to go. Don't they have a billion "good reasons" to violate your privacy, and never give you a way to completely turn it off.
I want a local AI Assistent in my Desktops which is able to see what is installed, what is dependent on what, has a API for apps to make data available for the AI (best would be if you have a Setting in each app on what to share with the AI Assistent) and a lot more.
Problem is, who is gonna train that AI? And, yea, this is how appleAI works
I'm GLAD there's issues with AI and Linux!...I'm against this nonsensical push....this constant insistence that every thing that has electricity running through it need A.I.
ENOUGH!....I will continue to "revert" backwards?..and use Laptops that have the ability to turn off UEFI / Secure Boot....I will continue to use Tor...and Linux distributions that help to return some bits of privacy to your computer usage. And to the devs who can't "see" it?....remember the movie "Jurasic Park".....you're all so busy trying to see if you COULD do it?....that NONE of you are questioning if you SHOULD!. Are you really desperate to see a car drive itself?...to see a robotic android device administer needles in the arms of Humans?...are you really so foolish that you cannot see your folly?
Its ok....Icarus ALSO didn't see his folly?....until he flew too close to the sun and the wax holding his wings together MELTED!
The data stacks or piramids, also need to be open data, all the ai software open sourcemto prevent governs and gafam feudals lord monopolize it
People use their computer efficient before so called Ai or rather simulated intelligence.
Though gpts and search engine have different internals,
Isn't gpt just a glorified search engine, that's more expensive to manage
Why not use the LLM locally and train it to favor Linux and open source over other options, ask it to develop local firewalls and give algorithms for personal defense defense against external threats?
There is no AI Problem on Linux, Linux is used for AI. If AI is used on the desktop depends on the user. I guess many Linux users do not want it period.
Maybe a local LLM could script for you and you can just make small edits to the script if needed. Could be a big time saver.
AMD Ryzen AI also have NPU in their AMD XDNA which appears to have been the first dedicated AI processing silicon on a Windows x86 processor. Although it looks like you have to install an xdna plugin/driver to use the npu with linux or the npu sits there dormant/unused?
A local open source AI (Neuronal Network) would be great for me, just a bit privacy concerns
simple fix. just have two different downloads . one for regular linux and another for npc or whatever you said users.
AI doesn't belong in desktops if you want something predictable.
in the grand scheme of things, "what am I going to do with this extra processing power?!" is not a bad problem to have
I don’t see it as a problem. AI is for people too lazy/stupid to think for themselves.
Honestly i think an NPU is pretty useless when a GPU is way way faster
Leaving AI behind is the reason why I'm thinking about switching to Linux, is the thing. I don't care about a part the OEM forced on me "gathering dust".
We need an localized open source LLM model integrated into a Linux desktop distro.
I'll wait for a non-deterministic true-random matrix multiplier unit.
the question is, do you actually need a dedicated chip for an Alexa on steroids?
No IS NOT AI,Sorry,Stop Misinfo please
Nice video, why don't they make a plugin as part of the operating system that a person can enable and choose what resources to give to ie that way some of your system is still private. A bit like a container, Linux container or docker container. The Linux community needs to develop along with AI but still keep privacy one of the fundamental foundations of linux
Give me an open source local model. I like AI and privacy
I think if Qualcomm release driver for their new snapdragon CPU/GPU then maybe this will unlock NPU i for devs to use whatever they want.
the true freedom is having the choice to use or not use private software on your OS. when Linux turn into the main OS thats will be the PRIME, all people will be FREE able to chose what tools to use or not use.
can you add timestamps to your vids
Games and Audio drivers.
Linux still is a "personal" computer OS. Now you did say desktop Linux, not quantum science for the masses. A npu sidecar unit, like a server unit to add on if you want, DONE, not for me.
The question is if the tooling will be competitive with Windows or not and if anyone cares to utilize GAI/LLMs for normal workflows. If you think GAI is a bad thing the lack of integrations isn't a problem, but there is reasonable questions as to if there are enough developers that will find actually useful privacy-first open-source integrations that people will find actually useful or not.
Right now a ton of GAI/LLMs are very effectively ran under Linux in general, but the integrations are still aimed towards super technical users. If LLMs settles down as just being an over-glorified chatbot as many think there would be a lot of wasted effort that could have been spent on a more refined desktop, but if things actually get to a point where AI is a "real" advantage, it wouldn't be good to ignore it and just let Microsoft/Apple be the only ones with decent well fleshed out solutions for normal end-users.
My prediction:
Some 13 year old will figure out how to mine crypto on the NPU.
Pinokio might help you out.
"what linux is going to do with npu" sounds like a nonsense question. Linux doesn't do anything with your computer, it allows you to do anything you want with it, that's what operating systems are for
Get off my lawn!!!
It would be nice if certain programs took advantage of the NPU. Programs for graphics and audio and office suites to perform enhancements. All locally, of course, with occasional updates.
Imagine having a local AI take a song idea, create lyrics, have the audio AI developer a music suggestion for it, then have it develop an album cover... maybe even a music video suggestion that can be further modified.
Not sure if I want it or not?
I generally look at it as 'where this capability is useful to my workload'. A lot of what's in proprietary OS'es is where its useful to corporations and by extension useful enough for us to tolerate it. I don't think it's a problem. It's really a matter of intentions.
Note: A LOT of 'capabilities' in desktop computer hardware often take a while to get used or even 'enabled'. Every OS kernel arch doesn't necessarily support the full spectrum of underlying capabilities (and some come and go). When GPU's were 'new', hardly anything took advantage of it. I expect NPU's to follow the same track (specialized use cases, over hype, more practical general use cases over time until it's 'normalized'.... and corporate malfeasance along the way we won't want to 'normalize' on linux. aka: not a 'new' problem but just the typical 'uptake' until a technology is 'normalized').
I'm already using AI on a per need basis on my Linux system and driving it through nvidia cuda. The applications I use already support CUDA and i'm sure they will also support NPU as well, so I don't see a problem here for Linux. So on Linux you can already run a language model like chat-gpt locally as well as image generation and other applications. As time moves on especially with the introduction of the new Qualcomm snapdragon which already supports Linux, we'll see even more support for AI on Linux.
Point is, if the application supports NPU or CUDA then you have no problem running AI regardless of the operating system ( Windows / Mac / Linux ). Also another point is that if you want to run AI locally or manually for privacy reasons, you'll have to set it up yourself anyway regardless of the operating system. People have been running local AI on Windows, Mac and Linux already for the past year, take the most popular ones like Ollama and Stable Diffusion for example.
So, I don't see any problem here at all. If you want privacy you take responsibility and you take care of yourself, period.
There is no such thing as AI, therefore Linux cannot possibly have a problem with it.
What we have is machine learning that is being hyped up by many, mostly to push their own approaches which all are just more or less advanced versions of Joseph Weizenbaum's Eliza that he wrote back in the 1960s.
By now, most of what can be found on the internet has been feed to those machine learning systems by thousands of underpaid humans and without that input, not one of them would be able to even do the silly things they do now and for which they consume more and more energy in times of a massive climate catastrophy where we should focus on saving energy, not wasting even more. Therefore, any chip dedicated to machine learning that just collects dust is a better one that wasted more energy. Even better would be if no resources had been wasted to produce it in the first place.
Although almost all was fed to these systems, they did not develop anything even remotely intelligent but rather show more and more how useless they are because garbage in produces garbage out and the more fake news (garbage) they produce that is then fed to the other systems, the more it will lead to a downward spiral.
Two very simple examples why nothing in machine learning has anything to do with intelligence:
- Any chess robot will most likely beat any human at playing chess but should s fire break out near it, it would keep playing chess until it is burnt down since it has no concept of fire or what it could do.
- Any robot that was taught to pull something using a rope has no idea that it cannot push the same thing using that same rope because it has no ideas at all in the first place but it also has no concept of what a rope actually is.
I respectfully disagree. Where code can re-write itself beyond the interpretation and understanding of its programmers (and yes this has happened with stock trading algorithms) it has potential to become "self-aware" and therefore meet the definition of an artifical intelligence. And no, I'm not suggesting a stock trading algorithm isn an AI- I'm using it as an example of how software can grow beyond the programer's ability to predict and interpret it
@@maxanderson9187 We are being told that actual AI is just around the corner, yes, now really! since Eliza was written and it remains as far away as it always was. Code changing in ways that programmers do not understand has nothing to do with self-awareness at all, it merely shows that the code added more code that it found somewhere without any kind of understanding what that code might do. Then humans have to check what the new code does and to tell the machine learning system if this was an improvement or rather garbage. Of course, only the human-approved improved code is then used to do the next random changes. Otherwise, it would quickly degenerate again. Remember, garbage in - garbage out and no, the models cannot decide themselves if what they produced is garbage or genius.
NPU just dosent seem like it would be powerful enough to do anything. Im rocking an RTX3060 12gb and it can just barely manage stable diffusion and llama3 8b. My thoughts are that running inference on a onboard NPU that shares system memory will make it very slow for the limited number of applications that it can run... So my conclusion is to just pick up a K80 or something like that if you want to run inference as youll get much more bang for your buck.
Um, I know NPUs are kind of like a co processor. Can it be used for puposes outside of AI? Does it have a shader like language I can play with?
What is the AI problem? This was so frustrating to listen to because there was a lot of dancing around but no statement of what the problem actually is.
I think using it for Hardware Acceleration would be awesome!
ok so how can we not have ai and use the npu to speed up processing lol
Speed up processing for what specifically? If you're meaning general processing to help the CPU, then I don't think it wouldn't help much due to the latency and inefficiency. Just like we don't use graphics cards for general processing.
@@pigeonprotagonist6357 lol i know nothing about the npu lol i just know how since i started using linux, it seems like everything can get repurposed lol
never trust a grown man wearing a Star Wars shirt.