(Edit: Errata below.) I think this is my longest video yet! Going back to weekly schedule, I can't produce videos this long every week though :) Discord: discord.gg/AgafFBQdsc Patreon: www.patreon.com/DrWaku Errata - Google won the Oracle vs Google case, not Oracle - "OpenAPI" should be "OneAPI" (re: Intel) - Oracle may have implemented IBM's API, not the other way around - Intel did not have an open architecture, AMD was granted a license - CUDA compiler now uses LLVM as well!
You should sticky the errata. In any case, this is my first time watching your content, brilliant work and excellent presentation. Looking forward to browse through your back library.
Watched you from the beginning. You manage to maintain a very very high quality channel with pretty unique content without resorting to clickbait/hype or other algorithmic bullshit.
I was captivated for the whole video, you speak clearly and with a very nice pace and build a good foundation before touching the more confusing parts of the topic (also thank you for using the companies logos every time you mention one…)
Intels thing is definitely not called "OpenAPI" because thats a standard for defining and documenting HTTP REST APIs. Intels thing seems to be called "oneAPI"
A lot of small, but important, mistakes: 1. Triton was supporting AMD in September 2023 and maybe even before. Also PyTorch uses Triton as a backend. 2. CUDA compiler is partly open source as is based on LLVM. Nvidia is a major LLVM contributor 3. The story with IBM and Oracle was the other way around 4. x86 and x86_64 are not open architectures 5. Google won vs Oracle
Thanks for your reply. I took the date of AMD support from when they edited the readme to add it. I figured that's when it was official. Did not know that the CUDA compiler is based on LLVM. I got the IBM story from an IBM employee, maybe they were biased. The rest I didn't check on, good catches thanks.
Google vs. Oracle: The surpreme court ruled in favour of Google. AFAIK they didn't have to pay tons of money. And it wasn't about "Hey, that code looks like ours!" it was a "Hey the reimplemented our API! How dare they!"
Yeah I got the outcome of this ruling wrong. And yeah it was about API reimplementation originally, but the sticking point ended up being a tiny function if I remember correctly. Maybe it didn't have an impact on the final final ruling.
NV might be keeping some competitors out, but in case of AMD the thing holding them out is THEMSELVES, they have just been too sloppy and disregarding year after year for their sw ecosys and that way they never built value but destroyed their own footprint.
Agreed. AMD has always tried to do things with a lower budget and without an eye to long term sustainability, unfortunately. We'll see if that continues in the GPU division under Lisa Su.
AMD needed to avoid bankruptcy and become successful/profitable with their other products before they could fully focus resources on GPGPU. From the Xilinx acquisition and on, they have been investing very heavily. We'll see how smartly they can deploy, integrate, and scale ROI on those investments.
AMD being compatible with Intel is NOT because of an open standard - it is because AMD had something to LEVERAGE Intel into giving them a LICENSE. What they leveraged was NOT being Intel - when IBM wanted to start making PC's, it was the dominant player and had a policy of ONLY dealing with anything that has at least 2 suppliers. So, Intel and AMD came to an agreement that allowed IBM to then use Intel processors. I am sorry, your - well - history-fu is quite shallow on that one.
Nvidia GPUs are also built around floating-point registers, which are used to compute decimal values. While floating-point logic has gotten better in recent years, it remains more complex than integer logic. Simply put, it takes more wires and transistors to build a floating-point register than one made to process integers. More wires equals higher cost. More wires equals more resistance. More resistance means higher energy consumption and heat generation. AI Developers are re-engineering transformers and neural nets to make use of more efficient integer operations and eliminate matrix multiplication. This would allow AI Models to run on much faster, cheaper, and more energy efficient RISC-Based ARM Chips.
It was Oracle who wrote a simple API to pull stuff out of DB2. So the Oracle-IBM story is the other way around. There are t-shirts that said, Oracle had IBM talking to itself. Because using the API, you could set up one DB2 talking to another DB2 through Oracle, yet those two different DB2 databases were incapable of talking to themselves.
I wonder if each company implemented the other's API at some point. I got this story from someone who worked at IBM and who was proud of the work that IBM had done, lol. That said, DB2 is probably not very flexible or easy to use so I'm not surprised they made a t-shirt.
It's interesting to see these big companies like Intel and Nvidia experience some troubles when they might have seemed untouchable in the recent past...
there's one important thing that's being over looked. The CUDA compiler isn't just converting higher level language to executable byte code. It's also optimizing the execution to maximize parallel processing, which is historically very hard. You can have great hardware, but it won't be faster if your compiler can't effectively parallelize the work. CUDA stack does a lot of fancy stuff to optimize the code. OpenCL is essentially dead and no one should use it going forward. I really want scale to succeed and 7 years is a long time to spend on a project. The better fix is to move neural networks forward so we don't need 20K H100 to train 1 trillion parameters. We need to understand what all those weights are doing and make models that are 10,000 smaller.
every compiler does fancy stuff to optimize the code. You think gcc doesn't vectorize your data to benefit from SIMD as well ? :p it's the programmer who needs first to not only accurately parallelize the code for maximum throughput , but also be careful to keep gpu threads coalescent in memory accesses. this is the hardest part , finding THE parallel algorithm.
@@emperorpalpatine6080 great point. GCC does a lot of fancy stuff and there are a ton of compiler options. I've never written a GPU shader compiler, but I do enjoy writing compilers in my free time. From my own experience, most developers have a poor understanding of how to parallelize the execution and often it isn't obvious. Even if you figure out the best way today, if the data changes over time it won't stay optimal. It reminds me of the saying "premature optimization is the root of all evil." I don't necessary agree with that statement for all situations, but I have seen it first hand. Hand rolled optimization is difficult to maintain as data and hardware evolves. Nvidia has done a good job of making CUDA handle a lot of that, but you still have to make sure you load your data efficiently
There are 300+ cuda libraries supporting all forms of scientific, engineering, and commercial applications. They cannot be placed or duplicated quickly by a new language or compiler.
I've watched many of the videos from this channel at this point and learned a lot. I really appreciate the high quality in-depth explanatory videos you've made through the past year. First I was hesitant from pushing on to your videos from the list because of the length of the videos. However after watching many of them I really appreciated the density of the information and the proper break-down interpretation of the complicated subjects you try to convey to the audience in a much more palpable form. Nevertheless I decided to write a comment for the first time after watching the videos on your health and career change in the past year. In my opinion this channel's videos' content is very high quality, probably worth much more subscribers than the current subscribers the channel has. The content quality is better than other engineering knowledge channels which have 100k to 300k subscribers. In my opinion two main elements are acting as a huge roadblock for the growth of your channel. (1) thumbnail design, art style of the thumbnail and (2) the average length of the videos. I wasn't going to comment anything before because I wasn't sure if you really wanted growth in your TH-cam channel but after watching the videos regarding your personal career change and health, I decided to write this after watching them, hoping this helps the channel growth. (1) the thumbnail is the biggest roadblock from the clicks getting through because people these days associate "generative AI" images and bots, or low quality videos. It seems like you edit texts on the images after you generate AI images for the thumbnails, but still the whole channel's thumbnails all look too much like a collection of AI generated images. The thumbnails looking like it containing a low quality video does a huge disservice to your excellent quality knowledge educational videos. In my opinion, the video themselves does not need any additional cosmetics changes. It is plenty good for an engineering knowledge sharing channel, but unless the general look of the thumbnails change, very few people will notice that this is a in-depth serious engineering channel's video. The engineering TH-cam channels' thumbnails have a certain style of look to them. I doesn't have to match the style but there is a reason their thumbnails evolved in a certain style over the years. It attracts and grab attention of a certain types of people, technology nerds such as myself. The current thumbnails looks too complicated and unorganized. It needs to be much more simple looking thumbnails in order to be perceived as a video from an in-depth engineering channel. The thumbnails don't have to show a human face, as many tech channels do. There are channels that make good thumbnails without putting a human face in the thumbnail. I suggest a few channels that makes a good reference thumbnails that fits this channel's content as reference for you. hope this helps. Some of these channels are not engineering channels but they make very good thumbnails for their videos. (@TheTeslaSpace, @Fireship, @3blue1brown, @ExplainingComputers, @TwoMinutePapers, @domainofscience, @GamerMeld ) If making these style of thumbnail is difficult under your health condition, at least more emphasis on the text on your thumbnails would be a improvement. (2) The average length of the videos of this channel seems its around 20~25minutes. Although the nerdy audience is more tolerant with lengthy videos when it comes to technical subjects, because most of the channels that are full of lengthy videos host low quality, low-density videos most of the times, people will see that your channel having a videos length of 20~25minutes by average as an indicator of low quality, low-density videos which nerd absolutely dislikes. The videos has to be under 19m59s in this subject matter, unless it's an long-form interview video of a person. Preferably under 16minutes, but 19m59s is the absolute maximum, in my experience, for not alienating the engineering enthusiast audience. I know that your channel's videos, even with 30minutes lengths, are very dense with it's information. If your audience reach a certain number, there is a snowball effect that mitigates the bad first impression associated with lengthy videos, but the combination of the Generative AI thumbnails and the lengthy videos, it gives a very wrong first impression of the TH-cam channel in my opinion. There are ways to contain video lengths to under 16minutes even if it's approaching a complicated subject. For example separating the video of multiple sub-subjects, and linking each other, not just chopping them the videos in a linear fashion. I really enjoyed your videos and hope that this channel's reach more wider audience in the future. Good luck and I'll be looking forward for future videos Dr Waku.
Thank you very much for your in-depth analysis. I do want to grow my channel, though I'm getting an impression of how difficult it might be to be a larger TH-camr with all the discussions about the thumbnail on this video. When your audience keeps growing, there are inevitably people who won't like the content, or assume one thing and find another. I guess it can be rough. But I want the channel to be able to support itself or maybe even support me, in future. So I'll give it a go. I used to keep all my videos under 20 minutes, and would even apologise when it approached 18+ minutes. My audience at that time however liked the density of content, no matter how long. But I agree with your analysis. I really hate watching channels with low density of information (hence my style). So keeping to 20 minutes max is a helpful signal to new viewers. (Existing viewers probably don't mind either way.) For the thumbnails, I just really enjoy AI art, so I kept creating them this way even though I've had some feedback that it was not ideal. I checked out all the channels you linked. I see that some are still using AI art, but there is also more use of negative space and larger font, etc. Gives me some ideas. Really appreciate your feedback.
Code is ideas, ideas are language, language is not anyone's property as no idea can be, no idea can be stolen because ideas are not lost, they are copied not taken away like real property.
It's 1:39 AM right now in Brazil. I should be sleeping for long time. But the presentation and the substance of this video kept me up. Instant subscribe! Edit: time correction
An excellent overview. AMD's HIP is a CUDA compatible-like API (function names are changed from 'cuda_' to 'hip_' for legal reasons) and this makes porting CUDA relatively easy with a script (hipify). Blender for example targets CUDA for NVIDIA GPUs and HIP for AMD GPUs which means 2/3rds of the code can be shared between them cutting down dev time. Yes HIP would be a generation behind but nobody is deploying code written with the very latest API version in mind because a) development takes time and b) there's no market share for it. So that's dumping some dirt into the moat. PyTorch abstracting code away from CUDA is a dump truck of dirt into the moat. Triton is the next big step which brings me to something very important.. You mentioned AMD and intel as competitors but left out NVIDIA's _biggest_ competitors. All of NVIDIA biggest customers, Google, Microsoft, Meta, Amazon, Tesla, have their own AI accelerators in various stages of design or production. Google's TPUs have been in production for years. Microsoft is onto their second generation of their MAIA chip. Tesla is training on their Dojo (D1) chips. And Amazon has their second generation Trainium and Inferentia chips available to rent on AWS. These all operate with the PyTorch framework and most have Triton support. NVIDIA doesn't have to worry about their competitors, they have to worry about their customers.
A nice presentation that most Nvidia "critics" and "shorts" seem to have little appreciation of. A part of the moat not mentioned, I think, is that several of the megacap tech companies who are working on their own hardware and software are also intensely competing with each other to get there generative apps out and making money. That competition means they're so far dedicated to the Nvidia ecosystem and that means their hardware and software stack. Another thing that extends the time span of the moat is that Nvidia is not standing still at all--they've speeded up the hardware development cycle from about 2 years, targeting 1 year cycles from now on. Their generative AI virtual assistant bank of applications seems way ahead also. So it seems like the competition is building a car for a drag race with Nvidia and has a nice gear shift but is lacking tires and a transmission and the motor is questionable. By the time they assemble the parts for the race Nvidia, has a car that is several generations ahead of them. So, eventually other organizations will have something equivalent to what Nvidia offers today. The question to ponder is what newer and more advanced problems are they going to have new stacks to attack and solve--not AI per se but maybe in other areas where their parallel processing data centers excel and the others are not yet working toward.
@25:30 After you grabbed my full attention 3 seconds into this incredibly high-quality and well architected documentary, I said to myself, "AI is our only way out of this..." and here we are. Your sentiment about Triton is dead-on about OpenAI making chips. Especially when you hear OpenAI boardroom discussions even entertaining $2,000/month for ChatGPT Premium. That number seems insane to us but not when one chatbot can fully replace an entire clerk's office that would cost you $50,000+/month in salary alone. And I, like you, have an Intel/Nvidia Machine and an AMD machine sitting right here at my desk: one for CUDA and one for a 12-core CPU.
Thank you very much for your comment. Yeah I've been using AMD CPUs for a long time since ryzen 1800X, but I used an Intel CPU in my latest deep learning build with a 3090!
Vis a vis CUDA’s complexity for coders, how much would it take for Nvidia to make a custom AI that’s an expert at CUDA code creation? They certainly have both the financial and compute resources to throw at the problem, and it could remove one of the biggest hassles for users.
Entirely possible (see: AlphaTensor), but there’s little need for it. They already have out of the box easy to use libraries for most things you need (Cutlass, etc), so if you need to do something, you can generally find a kernel for it somewhere on the internet, meaning you don’t really need to be a CUDA master. The complexity of writing kernels is why I use CPUs of all things for deep learning (it requires different designs for the architecture, but you can get surprisingly close to parity between single GPU and a single CPU), as it’s a lot easier to do a fully optimal AVX kernel for your CPU versus a fully optimal CUDA kernel. I’m looking into Tenstorrent hardware in the future as they’re roughly the same programming model as CPU AVX kernels.
It seems a bit click-baity, controversy seeking, and conspiracy-founding. Nvidia has invested many man-decades of work (and billions of dollars) into defining a software technology that matches their hardware technology, all of which are about representation of ideas and speed. Why isn't their hardware design a moat? Nvidia doesn't owe any other company free access to its technology. Amd and others can design a better system from scratch and whoop whomever they please. So far, no other companies can run and pass all the benchmarks anyone has ever written, and beat everyone. In AI, general compute, graphics, etc. Why would they give the keys to their office to the competition?
How much? It would take magic, because there is no such thing as "AI". Despite all the marketing, there is currently no program that can replace or even meaningfully assist an expert in writing any sort of involved code. Large language models have so far failed and increasingly seem to be a dead end for that purpose (among many other purposes). AI is a meaningless buzzword. Unless you're a developer talking to investors, stop using it.
Jensen (nvidia ceo) said no programmers in a few years, so theyre working on stuff like that. But i think hes wrong and doesnt understand programming, LLMs can help, but they're not even close yet to actually replacing programmers, we need more scientific breakthroughs for it to happen. Maybe it will, but it seems very optimistic and hees just hyping up nvidia for profit
@@martinkrauser4029 I think you have to look at the trajectory, not the current status. To me there's a clear trend, and I don't see a bright line anywhere that would be an uncrossable threshold. Perhaps we need to distinguish between "programming" and "coding". I view the former as including system architecture and design, goal-setting (choosing which problems to solve) and the generally "creative" parts of the process, while coding is simply converting the output of those parts into executable code. While we're nowhere near having machines able to do the former, we're making rapid progress on the latter. Currently, I'd say the assessment depends on your definition of "meaningfully assist." The LLMs can't write complete systems or even error-free components, but they can certainly shoulder a lot of the gruntwork of code-cranking. They're even now useful an amplifier on an expert's ability to rapidly generate code that implements their design. We're nowhere near what's being hyped, but look at the progress from GPT3 to GPT4 o1 in what, 2 years? From my naive viewpoint, the progress is accelerating, if you look at the capabilities of 3 to 3.5 to 4 to 4o and now 4o1. (Of course there are similar developments in other systems too, notably Anthropic's Claude series.) I agree that "AI" is a bit of a misnomer, although I've come to question what "I" actually is in the first place. (Maybe we're all just LLMs with state memory, on-the-fly reinforcement learning and agency? 😉) But I do think we're already seeing tools that can significantly leverage a human's time, and that are being able to do so more and more effectively.
This was clickbait. Nvidia is in no trouble. AMD is certainly trying to catch up but RocM is far behind and in 3-5 years there will be some market share lost to Cerebas or Groq chips but like you said Nvidia is benefitting from their existing relationships and proven quality. The title made it seem like Nvidia was in trouble but you basically spent 30 minutes discussing how far ahead they are.
@@pruff3 the title doesn’t claim it’s losing its lead only losing its monopoly and I think he made a good case that while they will still be the clearly dominant leader though not maintain their monopoly.
@@Greg-xi8yx yeah that's true I guess it's not really clickbait but the case for Nvidia losing its lead (the thumbnail) is misleading (pun intended) and also even the title is like saying "I'm dying" (yes, we all are, slowly, here's the evidence for why eventually we will be dead...in the future)
People are responsible for how they take different perspectives. Dr Waku has always had a high quality video in many aspects. In any regard, if you believe differently create your own video. Different perspectives and different takes is what makes knowledge possible. Calling things as clickbait is just a cheap criticism.
@@DrWakumy problem with this vid is these problems existed with Volta , nvida has a install base moat not really a cuda moat …. Its deeper than ”cuda” you can put it a program on nvida and it RUNS EVERYWHERE
Really impressed by your research abilities and how much you're able to simplify the details and tie them up together, can you share how many hours a video like this takes you to produce from scratch if you're comfortable with that?
Of course, it takes about 8-12 hours to research and write a script, 2-3 hours for thumbnail/description/timestamps, 1.5 hours to record. I think this video took about 10+3+2 = 15 hours. I pay for an editor, thank goodness, or that would surely add another 10+ hours.
This is a very good and thorough overview of why Nvidia has a software ecosystem moat and what the different ways are for other companies and projects to break this moat open, opening up the market for other larger players like AMD and smaller AI accelerator companies.
This was an interesting take on situation. It reminds me of 2008, when I was buying my first computer. I've gotten an AMD 4850 GPU. I was super happy. It was much better technologically than nvidia's 8800gtx, their first CUDA GPU. Still with software, even though older, 8800 series was better due to CUDA. I figured, I just need to give AMD a bit more time to catch up. With such great hardware, how can they fail? By 2011 it was obvious to me that AMD cannot possibly come close to nvidia software, no matter how good their GPU hardware is. Back then I got myself nvida gtx 460 and I haven't been back to AMD since. I have no reason to think that now, 18 years later, AMD has gotten any wiser. It would be nice to have a real competitor, but I'm not holding my breath. Strictly looking at their core 3d GPU technology they both support today: DLSS vs AMD's FSR. AMD simply cannot compete. If they can't make their GPUs perform their main function any better, what chance do they have in more complex problems.
In the consumer space, it is actually interesting that Apple devices can run these models locally on their processors, if you get a maxed-out memory mac, you can run a 70 billion param model quantized Q8, which is not possible with any other consumer hardware. And they have their own MLX framework and MPS as pytorch compatible layer. With ollama and llama3.1 8B on a M2 pro, runs very well and responds faster than chatgpt. Thanks, nice video and presentation and very clearly layed presented (some errors here and there) to keep me entertained :')
I think that a huge price-performance advantage will be needed to break nvidia's moat if it is to happen quickly. Most companies won't take the risk of compatibility issues that can end up in downtimes and extra development costs for a small saving as those can eat the small saving and then some. A very large performance or price to performance advantage in a way that requires breaking compatibility with cuda is what could do it. If getting away from cuda grants you 10x or more compute or allows to scale models 10x, it is very hard to say no as some competitor of you will say yes and you will be left in the dust. I believe that some kind of combination of analog computing (a single transistor can perform a multiplication) with in memory computing could offer the orders of magnitude improvement required to lure people away from Nvidia and I hope that AMD or Intel could be brave enough to try doing that.
AMD is always a 2nd place company. They do less inventing and more copying. Nvidia will certainly have competition in the long run. That’s why they need to be hyper focused on next-generation products. Someone called the DOJ in to slow nVidia down but most investors are worried that all of the capital expenditures in GPU’s is not bringing any level of profit. Which is now weighing on nVidia’s stock price. NVidia’s new chip will make the stock price jump up again.
That was what I thought in the Bulldozer era, but I still supported them (AMD) anyway. Arguably since the release of Ryzen chips though, AMD has been one of the most innovative companies out there. Still second place in terms of resources. But I don't know that they deserve that designation on the whole. GPU division has been another story.
@@DrWaku You might be right, but AMD's Ryzen chip is an X86, which is not their invention. Also, the x86 might already be a dinosaur looking at a bright light and the light is an ARM based killer asteroid. Also, from day one, AMD was not the GPU of choice to be used for AI. Nvidia is in autonomous cars, robots, AI, LLMs, computer graphics and crypto. I think AMD will be lucky if they can stay in second place in the GPU game. IMO, Nvidia's biggest challenge is from Apple, Microsoft, Meta, or Google, because they have the cash. Even little ole Amazon has 8 times the cash that AMD does. Good video, thanks.
You are being very patient with us. But frankly as an engineer I would really like you to drill down a little farther in your preliminary three or four minutes of explanation. To give me a better grasp of what you are talking about
Each time Apple switched processor architectures on the McIntosh, they used Rosetta. The first time switching from Motorola 680X0 architecture 2 power PC, then they used Rosetta again when they switched from our PC to X 86. The switch from X 86 to ARM- 64 was just the latest iteration of Rosetta. I don’t think it was ever possible to have a stack of two or three Rosetta translators on Apple hardware. Doing so would allow you to run software for the original McIntosh bye emulating the Motorola 68,000 on the emulated power PC on the emulated X 86 running on ARM 64
From MarkShot/Mahak who you may know from Scripter: thank you! I am primarily a retired software person. I have mainly heard of NVidia’s stack advantage from Anastasi chip channel, but understand it much better now. Also, I have not looked at compiler tech since I was doing recursive descent and LLAR(1). Again, thank you!
In my opinion, the best solution is RISC-V like what Tenstorrent is doing. They are doing super-simple cores with 128-bit SIMD units and the result doesn't look very different from GCN in practice with other cores acting as a kind of thread manager. Once you have an open ISA that can be shared, everyone can compete on their GPU performance while having software compatibility (even if optimization still needs to be done for each uarch).
This video was a lot more interesting than I thought it would be before I clicked on it I almost thought it was going to be AI voice slop too but i saw there was a face on the video lol
The world would be a better place if most "intellectual property" were recognized as shared by all rather than imposing artificial scarcity to control and subjugate all beneath the will of greed and domination.
Good information and video. The next one is also engaging. However, the sound and editing greatly reduce the positive impact on the audience. I won’t be able to finish the video on my end due to the poor sound quality. I hope the adjustments can be made in future.
Thank you for this exhaustive course on CUDA and its potential alternatives. I am a big investor in NVIDIA and you got me worried. I don't know where you found all this information, but it is great work.
Stadia failed because the latency to play games using the hardware from the cloud made it ridiculously unplayable. Who wants to play a video game that registers your action either a delay? Nobody
AMD drivers and support for Linux have also been poor at times. That's one good thing about the AI boom, forces these companies to treat Linux as a first-class citizen.
Nvidia should be worried because more and more big companies are starting to make their own hardware, google, amazon, etc. And x86-64 isnt power hungry because of legacy baggage, its a myth
Good summary. I've been avoiding CUDA (using cross-platform standards like OpenCL, OpenGL, Vulkan instead) as much as I could partly because I did not want to be locked into an NVidia ecosystem and partly because it's just too big and bloated.
Companies can evaluate multiple options / competition when the landscape is stable and other variables are controlled. When there is a massive race going on to 'win' AI, they simply can't afford to take additional risks by betting on unknown frameworks/stacks. So, that's the reason they'll continue to buy into the Nvidia stack till things become stable
I am getting real tired of hearing all the hype for ARM even though ISA really doesn't matter. If any of you who claim it as gospel actually do some research, you would stop spreading it. chipsandcheese has outstanding article on the topic "ARM or x86? ISA Doesn’t Matter" and it is in essency meta-analysis on the different actually research papres on the topic. What actually matters more is the actual whole system design in addition to the actual CPU power efficiency. Also Jim keller has said the same thing about ISAs.
Discovery number 1 in terms of clarity, depth of thought, relevance and thought-leadership. Damn this channel is underrated! Happy to have found you and thank you so much for your insights!
Short story, monopoly is bad for consumers. A good competitive market is 3-5 major players. And none of those major players should have less than 10% each or more than 40% of the market share. Anyone who buys from a monopoly contributes to the problem, no buts about it.
I have to say, although the video has great contents, constantly appearing subtitles at the bottom are incredeibly annoying. I would reduce their quantity by 5 times, if I were you.
Thanks. Yeah, usually the subtitles are a lot less frequent but this time my editor just put them everywhere. I'll ask him to decrease the frequency next time.
They’ve competed with nvidia in the graphics space and only hold 10% market share despite being more affordable. 17% in the processor space according to Google results…. I think NVidia will be just fine.
Another phenomenally clear and information-dense presentation. You go broader and deeper than any other channel I know on whatever subject you choose to cover, and it’s all well-structured and understandable. Kudos and thanks!
Fantastic video. Being a pytorch + CUDA developer myself, I appreciate your insight. Well researched, well explained, and lots of information I didn't know.
Nvidia's AI monopoly is coming to an end due to increasing competition from other tech companies developing advanced AI hardware and software. This shift is fostering innovation and providing more diverse options for AI solutions.
There is no "Nvidia AI monopoly" AI is software, Nvidia makes mostly hardware (GPUs) and some software, but it is not a monopoly. In fact, nvidia is a "de facto" monopoly in GPUs since Intel ones are not competitive, AMD also working in favor of Nvidia, Nvidia has 90% market share and the rest are phone GPUs which are not relevant for PCs for now. China might change this, and I hope they do. Also, monopolies are not the problem, the problem is corporations and corporations are a product of STATE legislation, so i it is the state. One set of rules that can destroy corporatism and bring back real liberty and competition, is abolition of fake property like intellectual property laws, which are absurd in all possible ways.
Sort of open architecture? Intel? Thats not even close to what it was. Edit: Wait, aren't you the guy from Dredd? You know Mama's IT guy in Peachtrees?
AMD64 is a better name. Intel originally banked on Itanium for 64-bit, which was a VLIW. So, x86_64 has nothing to do with x86. It is a RISC ISA with 16 GP registers.
The explanation of the software intricacies that comprise NVIDIA's moat was very interesting. Thank you. There is another moat which NVDA has in addition to CUDA. This is the hardware. None of companies mentioned in this video, except Intel, manufacture their own semiconductor devices. All the other companies rely on semiconductor foundries to produce the devices that they sell. More Threads most likely uses SMIC (China's largest foundry). The others use either TSMC or Samsung (the number 1 and 2 foundries globally). AMD and NVDA use TSMC's N5 or N3 fabs for their GPUs. TSMC's largest customer for their high end fabs is NVDA. AMD buys a small fraction of the volume that NVDA gets from TSMC. Let's say that AMD develops a killer GPU that could blow away Blackwell. Where will AMD have their chips built? TSMC doesn't have unused N3 fab space available and their N2 production is already committed. They could got to Samsung, assuming that Samsung has the available fab space. Typically, foundries sell out their production years in advance. In the unlikely event that Samsung had space available to produce AMD 's killer GPU, it isn't a simple task to switch a product taped out for TSMC to Samsung. My point is that NVDA also has a hardware moat inside their CUDA software moat. This combination protects NVDA 's market dominance in the short and mid-term.
TSMC's largest customer is Apple, followed by Qualcomm probably. Nvidia and AMD buy roughly (~factor 2) similar amounts of processed wafers from TSMC. Die area of Nvidia datacenter GPUs like H200 is only about 1.3x of that in RTX 4090, they just sell it for 20x the price. Even with Nvidia's datacenter sales having multiplied by 10, it hasn't affected the amount of fab capacity they buy from TSMC too much. Much of the cost and production bottlenecks of datacenter GPUs are in HBM memory and CoWoS packaging.
@@randomobserver9488 In 2022, Apple and Qualcomm were TSMC's 2 largest customers. The 2023 customer lists that I saw showed NVDA jumping into the number 2 spot. Based on NVDA's earnings, I would expect that NVDA would gain on Apple in 2024 as a percentage of TSMC's sales. TSMC's 2023 Top Customers (Translated from Traditional Chinese) 蘋果 Apple 25% 偉達 NVIDIA 11% AMD 7% 高通 QCOM 7% 聯發科 MediaTek 5% 博通 Broadcom 5% 英特爾 Intel 4% Other 36% There is a 2022 list showing Apple at 23%, QCOM at 8.9%, AMD at 7.6%, Broadcom at 6.6%, NVDA at 6.3%, MediaTek at 5.6%, and Intel at 5.1%. If these are correct, then the 2022 to 2023 change is: Apple +2% NVDA +4.7% AMD -0.6% QCOM -1.9% Broadcom -2.6% MediaTek -0.6% Intel -1.1% NVIDIA and Apple are the only major TSMC customers to show considerable growth as a percentage of TSMC's revenue, assuming that these sources are accurate. I would not infer from these numbers that any of these companies had a drop in TSMC production or revenue paid to TSMC. I would infer that both Apple and NVDA are consuming growing percentages of TSMC's production. TSMC is building new fabs in several locations in Taiwan, as well as in Japan and the US.
Apple funds the LLVM project. I bet they're working with AMD to challenge NVidia. Apple has a long adversarial relationship with NVidia. It's why Apple products stopped using their GPUs years ago.
This is actually a great analysis on the commeptitative landscape in the GPU market. I think AMD will take away some market shares from Nvidia in the near future at the low end application. I doubt it will have much impact on the high end, where the margins are. 75% gross margin is insane. I will be happy with 60% margin by 2026.
9:16 false. Compatibility goes only as far back as the 8088/8086. 8080 is not compatible, and compatibility with 8085 is only "source-code-level" so re-assembly is required.
you need to start using a pop shield for your mic and you need to do something against the overdrive distortions. your audio isn't the best it could be and not always pleasant to listen to. (Writing this at 10:54 . The content was interesting so far.)
Some way into the video, I began to resent that I was quite enjoying what appeared to be mere clickbait. However, you skilfully justified the provocative thesis in the final moments - well done. That said, let the challengers bark as they may. As long as the visionary commander remains compos mentis and the coffers flush with cash, the Nvidia behemoth will continue its march, unfazed by their noise.
Haha thanks for the backhanded compliment. I didn't mean to be clickbait but I think a lot of people have strong feelings about this topic. Thanks for sticking it out.
That's a result of matrix multiplication for the tokens. It may ring true for the time being however it does not add new information to the system and stuff? You know. Unless you want to like use it for something? What say you?
It would be hard to argue that this was a clean room reimplementation, if your competitor has an open source compiler. This is because the LLM was probably trained on all of that source code. Unfortunately, using LLMs isn't the best way to avoid copyright or legal challenges.
because "AI" doesn't exist, and the Large Language Models you're actually referring to are useless for even writing simple web-apps right, let alone a compiler, let alone for CUDA.
Suggest comparison of gcc and llvm? I think this shows certain parts of copyright need fixing. Perhaps the ability to force open sourcing of judge declared monopoly closed source!
From end-user perspective, I like to work with the tools handy to myself. For example, pytorch. I use it on daily base on apple silicon. Yes, it's working. Nearly 80% of the AI generative possibilities I do on my local apple silicon device. Cuda left the showroom for all apple users nearly a decade ago. I am sure, A100 units outperform any other hardware by now... but a lot of chinese companies getting very creative avoiding cuda for good. In 10 years, nvidia's monopoly state will be long forgotten. I see developments and new generation software running on non nvidia smart devices already today. With a better performance per whatever currency coin. The USD will be a relict as well in 10 years. Open Source and new ways of coding apps re-invent even the pre-dominant english language. I am sure that very soon, coding extensions will be written in chinese language. This part was missing for the last 55 years in computer science. Conclusion? Learn chinese!
Here is another moat killer perspective, with the advancement of LLM's and Copilot etc, its getting easier and easier to write code from scratch for building a new stack etc. + open source of AMD, that seems like things can really speed up for AMD with a combination like that. That being said u might wana check out George Hotz who was trying to create a tinyML using AMD and gave up on it and switched to Nvidia either. Other then the Software Moat with CUDA their software itself is less buggy then that of AMD as well, also the fact that programmers are getting lazier and lazier with copilot and LLMs will only make the code more buggier imho
in 2015, when I first do CUDA this already has been complained about, and everyone including Google is trying move its Tensorflow out of CUDA and Nvidia GPU. AMD GPU is so much cheaper and I badly needed an alternative. NOT EASY....even for giants like Google. Till now. And even for CUDA itself, every few months you will see your code is NOT working any more as CUDA and tensorflow version move ahead so fast. Yes, moving away from Nvidia I can bet is going to take many many years. If Nvidia slash its prices, everyone rather save time than to reinvent the wheel for another vendor.
(Edit: Errata below.) I think this is my longest video yet! Going back to weekly schedule, I can't produce videos this long every week though :)
Discord: discord.gg/AgafFBQdsc
Patreon: www.patreon.com/DrWaku
Errata
- Google won the Oracle vs Google case, not Oracle
- "OpenAPI" should be "OneAPI" (re: Intel)
- Oracle may have implemented IBM's API, not the other way around
- Intel did not have an open architecture, AMD was granted a license
- CUDA compiler now uses LLVM as well!
You're time work and wisdom is much appreciated, thank you
thanks for explaining in such a clear way, it's so important for us to understand this technology which is shaping our lives
I appreciate the work you put into your vids.
You should sticky the errata.
In any case, this is my first time watching your content, brilliant work and excellent presentation. Looking forward to browse through your back library.
@@IceAce1Added errata to this same stickied comment, thanks for the suggestion, cheers
Watched you from the beginning. You manage to maintain a very very high quality channel with pretty unique content without resorting to clickbait/hype or other algorithmic bullshit.
That's very kind of you to say. Thank you. I'm of the impression that high quality pays off over time.
NVIDIA LOSING ITS LEAD!!! how is that not clickbait? Theres nothing even close to NVIDIA, let alone being in danger of losing their lead
Was trying to say "losing its monopoly" but that was too long. Made a new thumbnail, hope that helps.
@@glamdragCalm down, the whole clickbait shaming crusade is getting as out of hand as clickbait itself.
@@glamdrag are you okay?
I was captivated for the whole video, you speak clearly and with a very nice pace and build a good foundation before touching the more confusing parts of the topic (also thank you for using the companies logos every time you mention one…)
Thank you very much for your kind comment! You can thank my editor for the logos haha, he does a great job
Intels thing is definitely not called "OpenAPI" because thats a standard for defining and documenting HTTP REST APIs.
Intels thing seems to be called "oneAPI"
Yeah, for some reason one reference called it open API by accident and that stuck in my head. This has been added to the errata list
A lot of small, but important, mistakes:
1. Triton was supporting AMD in September 2023 and maybe even before. Also PyTorch uses Triton as a backend.
2. CUDA compiler is partly open source as is based on LLVM. Nvidia is a major LLVM contributor
3. The story with IBM and Oracle was the other way around
4. x86 and x86_64 are not open architectures
5. Google won vs Oracle
Thanks for your reply. I took the date of AMD support from when they edited the readme to add it. I figured that's when it was official.
Did not know that the CUDA compiler is based on LLVM. I got the IBM story from an IBM employee, maybe they were biased. The rest I didn't check on, good catches thanks.
Added some errata to the pinned comment
@@DrWaku What pinned comment ? I don't see any pinned.
What part of CUDA is open source?
@@DrWaku You added the errata, but forgot to pin the comment.
Google vs. Oracle:
The surpreme court ruled in favour of Google. AFAIK they didn't have to pay tons of money. And it wasn't about "Hey, that code looks like ours!" it was a "Hey the reimplemented our API! How dare they!"
Yeah I got the outcome of this ruling wrong. And yeah it was about API reimplementation originally, but the sticking point ended up being a tiny function if I remember correctly. Maybe it didn't have an impact on the final final ruling.
Good informative roundup of the topic
14:07 I think you meant oneAPI instead of OpenAPI (stuff like this would deserved a pinned errata comment 🙂)
Thanks. I created an errata comment like you suggested, there are enough little errors in this video that it needs one
NV might be keeping some competitors out, but in case of AMD the thing holding them out is THEMSELVES, they have just been too sloppy and disregarding year after year for their sw ecosys and that way they never built value but destroyed their own footprint.
Agreed. AMD has always tried to do things with a lower budget and without an eye to long term sustainability, unfortunately. We'll see if that continues in the GPU division under Lisa Su.
AMD needed to avoid bankruptcy and become successful/profitable with their other products before they could fully focus resources on GPGPU. From the Xilinx acquisition and on, they have been investing very heavily. We'll see how smartly they can deploy, integrate, and scale ROI on those investments.
They focused on CPU. As Ryzen has become stable, hopefully they can focus on GPU, especially ROCm.
AMD being compatible with Intel is NOT because of an open standard - it is because AMD had something to LEVERAGE Intel into giving them a LICENSE. What they leveraged was NOT being Intel - when IBM wanted to start making PC's, it was the dominant player and had a policy of ONLY dealing with anything that has at least 2 suppliers. So, Intel and AMD came to an agreement that allowed IBM to then use Intel processors. I am sorry, your - well - history-fu is quite shallow on that one.
Nvidia GPUs are also built around floating-point registers, which are used to compute decimal values. While floating-point logic has gotten better in recent years, it remains more complex than integer logic. Simply put, it takes more wires and transistors to build a floating-point register than one made to process integers. More wires equals higher cost. More wires equals more resistance. More resistance means higher energy consumption and heat generation. AI Developers are re-engineering transformers and neural nets to make use of more efficient integer operations and eliminate matrix multiplication. This would allow AI Models to run on much faster, cheaper, and more energy efficient RISC-Based ARM Chips.
It was Oracle who wrote a simple API to pull stuff out of DB2. So the Oracle-IBM story is the other way around. There are t-shirts that said, Oracle had IBM talking to itself. Because using the API, you could set up one DB2 talking to another DB2 through Oracle, yet those two different DB2 databases were incapable of talking to themselves.
I wonder if each company implemented the other's API at some point. I got this story from someone who worked at IBM and who was proud of the work that IBM had done, lol. That said, DB2 is probably not very flexible or easy to use so I'm not surprised they made a t-shirt.
Great info and good presentation, deservers way more than your 16k. Subscribed, cheers from Sydney.
Thanks, I appreciate it! Cheers
the moat is slowly being whittled away, and I'm here for it!
It's interesting to see these big companies like Intel and Nvidia experience some troubles when they might have seemed untouchable in the recent past...
there's one important thing that's being over looked. The CUDA compiler isn't just converting higher level language to executable byte code. It's also optimizing the execution to maximize parallel processing, which is historically very hard. You can have great hardware, but it won't be faster if your compiler can't effectively parallelize the work. CUDA stack does a lot of fancy stuff to optimize the code.
OpenCL is essentially dead and no one should use it going forward. I really want scale to succeed and 7 years is a long time to spend on a project. The better fix is to move neural networks forward so we don't need 20K H100 to train 1 trillion parameters. We need to understand what all those weights are doing and make models that are 10,000 smaller.
every compiler does fancy stuff to optimize the code. You think gcc doesn't vectorize your data to benefit from SIMD as well ? :p
it's the programmer who needs first to not only accurately parallelize the code for maximum throughput , but also be careful to keep gpu threads coalescent in memory accesses. this is the hardest part , finding THE parallel algorithm.
@@emperorpalpatine6080 great point. GCC does a lot of fancy stuff and there are a ton of compiler options. I've never written a GPU shader compiler, but I do enjoy writing compilers in my free time. From my own experience, most developers have a poor understanding of how to parallelize the execution and often it isn't obvious. Even if you figure out the best way today, if the data changes over time it won't stay optimal. It reminds me of the saying "premature optimization is the root of all evil."
I don't necessary agree with that statement for all situations, but I have seen it first hand. Hand rolled optimization is difficult to maintain as data and hardware evolves. Nvidia has done a good job of making CUDA handle a lot of that, but you still have to make sure you load your data efficiently
Excellent essay, one of the most digestible tech breakdowns Ive ever heard, thank you Dr. Waku.
Thank you very much! I appreciate it :) see you around.
yeah - interesting video
There are 300+ cuda libraries supporting all forms of scientific, engineering, and commercial applications. They cannot be placed or duplicated quickly by a new language or compiler.
I've watched many of the videos from this channel at this point and learned a lot. I really appreciate the high quality in-depth explanatory videos you've made through the past year. First I was hesitant from pushing on to your videos from the list because of the length of the videos. However after watching many of them I really appreciated the density of the information and the proper break-down interpretation of the complicated subjects you try to convey to the audience in a much more palpable form. Nevertheless I decided to write a comment for the first time after watching the videos on your health and career change in the past year.
In my opinion this channel's videos' content is very high quality, probably worth much more subscribers than the current subscribers the channel has. The content quality is better than other engineering knowledge channels which have 100k to 300k subscribers. In my opinion two main elements are acting as a huge roadblock for the growth of your channel. (1) thumbnail design, art style of the thumbnail and (2) the average length of the videos. I wasn't going to comment anything before because I wasn't sure if you really wanted growth in your TH-cam channel but after watching the videos regarding your personal career change and health, I decided to write this after watching them, hoping this helps the channel growth.
(1) the thumbnail is the biggest roadblock from the clicks getting through because people these days associate "generative AI" images and bots, or low quality videos. It seems like you edit texts on the images after you generate AI images for the thumbnails, but still the whole channel's thumbnails all look too much like a collection of AI generated images. The thumbnails looking like it containing a low quality video does a huge disservice to your excellent quality knowledge educational videos. In my opinion, the video themselves does not need any additional cosmetics changes. It is plenty good for an engineering knowledge sharing channel, but unless the general look of the thumbnails change, very few people will notice that this is a in-depth serious engineering channel's video.
The engineering TH-cam channels' thumbnails have a certain style of look to them. I doesn't have to match the style but there is a reason their thumbnails evolved in a certain style over the years. It attracts and grab attention of a certain types of people, technology nerds such as myself. The current thumbnails looks too complicated and unorganized. It needs to be much more simple looking thumbnails in order to be perceived as a video from an in-depth engineering channel. The thumbnails don't have to show a human face, as many tech channels do. There are channels that make good thumbnails without putting a human face in the thumbnail. I suggest a few channels that makes a good reference thumbnails that fits this channel's content as reference for you. hope this helps. Some of these channels are not engineering channels but they make very good thumbnails for their videos.
(@TheTeslaSpace, @Fireship, @3blue1brown, @ExplainingComputers, @TwoMinutePapers, @domainofscience, @GamerMeld )
If making these style of thumbnail is difficult under your health condition, at least more emphasis on the text on your thumbnails would be a improvement.
(2) The average length of the videos of this channel seems its around 20~25minutes. Although the nerdy audience is more tolerant with lengthy videos when it comes to technical subjects, because most of the channels that are full of lengthy videos host low quality, low-density videos most of the times, people will see that your channel having a videos length of 20~25minutes by average as an indicator of low quality, low-density videos which nerd absolutely dislikes. The videos has to be under 19m59s in this subject matter, unless it's an long-form interview video of a person. Preferably under 16minutes, but 19m59s is the absolute maximum, in my experience, for not alienating the engineering enthusiast audience. I know that your channel's videos, even with 30minutes lengths, are very dense with it's information.
If your audience reach a certain number, there is a snowball effect that mitigates the bad first impression associated with lengthy videos, but the combination of the Generative AI thumbnails and the lengthy videos, it gives a very wrong first impression of the TH-cam channel in my opinion. There are ways to contain video lengths to under 16minutes even if it's approaching a complicated subject. For example separating the video of multiple sub-subjects, and linking each other, not just chopping them the videos in a linear fashion.
I really enjoyed your videos and hope that this channel's reach more wider audience in the future. Good luck and I'll be looking forward for future videos Dr Waku.
Thank you very much for your in-depth analysis. I do want to grow my channel, though I'm getting an impression of how difficult it might be to be a larger TH-camr with all the discussions about the thumbnail on this video. When your audience keeps growing, there are inevitably people who won't like the content, or assume one thing and find another. I guess it can be rough. But I want the channel to be able to support itself or maybe even support me, in future. So I'll give it a go.
I used to keep all my videos under 20 minutes, and would even apologise when it approached 18+ minutes. My audience at that time however liked the density of content, no matter how long. But I agree with your analysis. I really hate watching channels with low density of information (hence my style). So keeping to 20 minutes max is a helpful signal to new viewers. (Existing viewers probably don't mind either way.)
For the thumbnails, I just really enjoy AI art, so I kept creating them this way even though I've had some feedback that it was not ideal. I checked out all the channels you linked. I see that some are still using AI art, but there is also more use of negative space and larger font, etc. Gives me some ideas. Really appreciate your feedback.
Why didn't they made OpenCL compilers that automatically took care of the hardware-specific optimizations?
Code is ideas, ideas are language, language is not anyone's property as no idea can be, no idea can be stolen because ideas are not lost, they are copied not taken away like real property.
It's 1:39 AM right now in Brazil. I should be sleeping for long time. But the presentation and the substance of this video kept me up. Instant subscribe!
Edit: time correction
An excellent overview.
AMD's HIP is a CUDA compatible-like API (function names are changed from 'cuda_' to 'hip_' for legal reasons) and this makes porting CUDA relatively easy with a script (hipify).
Blender for example targets CUDA for NVIDIA GPUs and HIP for AMD GPUs which means 2/3rds of the code can be shared between them cutting down dev time.
Yes HIP would be a generation behind but nobody is deploying code written with the very latest API version in mind because a) development takes time and b) there's no market share for it. So that's dumping some dirt into the moat.
PyTorch abstracting code away from CUDA is a dump truck of dirt into the moat.
Triton is the next big step which brings me to something very important..
You mentioned AMD and intel as competitors but left out NVIDIA's _biggest_ competitors.
All of NVIDIA biggest customers, Google, Microsoft, Meta, Amazon, Tesla, have their own AI accelerators in various stages of design or production.
Google's TPUs have been in production for years. Microsoft is onto their second generation of their MAIA chip. Tesla is training on their Dojo (D1) chips. And Amazon has their second generation Trainium and Inferentia chips available to rent on AWS.
These all operate with the PyTorch framework and most have Triton support.
NVIDIA doesn't have to worry about their competitors, they have to worry about their customers.
A nice presentation that most Nvidia "critics" and "shorts" seem to have little appreciation of.
A part of the moat not mentioned, I think, is that several of the megacap tech companies who are working on their own hardware and software are also intensely competing with each other to get there generative apps out and making money. That competition means they're so far dedicated to the Nvidia ecosystem and that means their hardware and software stack.
Another thing that extends the time span of the moat is that Nvidia is not standing still at all--they've speeded up the hardware development cycle from about 2 years, targeting 1 year cycles from now on. Their generative AI virtual assistant bank of applications seems way ahead also. So it seems like the competition is building a car for a drag race with Nvidia and has a nice gear shift but is lacking tires and a transmission and the motor is questionable. By the time they assemble the parts for the race Nvidia, has a car that is several generations ahead of them.
So, eventually other organizations will have something equivalent to what Nvidia offers today. The question to ponder is what newer and more advanced problems are they going to have new stacks to attack and solve--not AI per se but maybe in other areas where their parallel processing data centers excel and the others are not yet working toward.
@25:30 After you grabbed my full attention 3 seconds into this incredibly high-quality and well architected documentary, I said to myself, "AI is our only way out of this..." and here we are.
Your sentiment about Triton is dead-on about OpenAI making chips. Especially when you hear OpenAI boardroom discussions even entertaining $2,000/month for ChatGPT Premium. That number seems insane to us but not when one chatbot can fully replace an entire clerk's office that would cost you $50,000+/month in salary alone.
And I, like you, have an Intel/Nvidia Machine and an AMD machine sitting right here at my desk: one for CUDA and one for a 12-core CPU.
Thank you very much for your comment. Yeah I've been using AMD CPUs for a long time since ryzen 1800X, but I used an Intel CPU in my latest deep learning build with a 3090!
Vis a vis CUDA’s complexity for coders, how much would it take for Nvidia to make a custom AI that’s an expert at CUDA code creation? They certainly have both the financial and compute resources to throw at the problem, and it could remove one of the biggest hassles for users.
Entirely possible (see: AlphaTensor), but there’s little need for it. They already have out of the box easy to use libraries for most things you need (Cutlass, etc), so if you need to do something, you can generally find a kernel for it somewhere on the internet, meaning you don’t really need to be a CUDA master. The complexity of writing kernels is why I use CPUs of all things for deep learning (it requires different designs for the architecture, but you can get surprisingly close to parity between single GPU and a single CPU), as it’s a lot easier to do a fully optimal AVX kernel for your CPU versus a fully optimal CUDA kernel. I’m looking into Tenstorrent hardware in the future as they’re roughly the same programming model as CPU AVX kernels.
It seems a bit click-baity, controversy seeking, and conspiracy-founding.
Nvidia has invested many man-decades of work (and billions of dollars) into defining a software technology that matches their hardware technology, all of which are about representation of ideas and speed. Why isn't their hardware design a moat? Nvidia doesn't owe any other company free access to its technology. Amd and others can design a better system from scratch and whoop whomever they please. So far, no other companies can run and pass all the benchmarks anyone has ever written, and beat everyone. In AI, general compute, graphics, etc. Why would they give the keys to their office to the competition?
How much? It would take magic, because there is no such thing as "AI". Despite all the marketing, there is currently no program that can replace or even meaningfully assist an expert in writing any sort of involved code. Large language models have so far failed and increasingly seem to be a dead end for that purpose (among many other purposes). AI is a meaningless buzzword. Unless you're a developer talking to investors, stop using it.
Jensen (nvidia ceo) said no programmers in a few years, so theyre working on stuff like that. But i think hes wrong and doesnt understand programming, LLMs can help, but they're not even close yet to actually replacing programmers, we need more scientific breakthroughs for it to happen. Maybe it will, but it seems very optimistic and hees just hyping up nvidia for profit
@@martinkrauser4029 I think you have to look at the trajectory, not the current status. To me there's a clear trend, and I don't see a bright line anywhere that would be an uncrossable threshold.
Perhaps we need to distinguish between "programming" and "coding". I view the former as including system architecture and design, goal-setting (choosing which problems to solve) and the generally "creative" parts of the process, while coding is simply converting the output of those parts into executable code. While we're nowhere near having machines able to do the former, we're making rapid progress on the latter.
Currently, I'd say the assessment depends on your definition of "meaningfully assist." The LLMs can't write complete systems or even error-free components, but they can certainly shoulder a lot of the gruntwork of code-cranking. They're even now useful an amplifier on an expert's ability to rapidly generate code that implements their design. We're nowhere near what's being hyped, but look at the progress from GPT3 to GPT4 o1 in what, 2 years? From my naive viewpoint, the progress is accelerating, if you look at the capabilities of 3 to 3.5 to 4 to 4o and now 4o1. (Of course there are similar developments in other systems too, notably Anthropic's Claude series.)
I agree that "AI" is a bit of a misnomer, although I've come to question what "I" actually is in the first place. (Maybe we're all just LLMs with state memory, on-the-fly reinforcement learning and agency? 😉) But I do think we're already seeing tools that can significantly leverage a human's time, and that are being able to do so more and more effectively.
Pytorch recently started supporting MPS on Apple Silicon. I used it myself on my Mac, working like a charm. No nvidia needed.
This was clickbait. Nvidia is in no trouble. AMD is certainly trying to catch up but RocM is far behind and in 3-5 years there will be some market share lost to Cerebas or Groq chips but like you said Nvidia is benefitting from their existing relationships and proven quality. The title made it seem like Nvidia was in trouble but you basically spent 30 minutes discussing how far ahead they are.
I don’t really think the title suggested they were in trouble only that their monopoly would end…
@@Greg-xi8yx obviously it will end some day but it's definitely not "losing its lead" any time soon
@@pruff3 the title doesn’t claim it’s losing its lead only losing its monopoly and I think he made a good case that while they will still be the clearly dominant leader though not maintain their monopoly.
@@Greg-xi8yx yeah that's true I guess it's not really clickbait but the case for Nvidia losing its lead (the thumbnail) is misleading (pun intended) and also even the title is like saying "I'm dying" (yes, we all are, slowly, here's the evidence for why eventually we will be dead...in the future)
People are responsible for how they take different perspectives. Dr Waku has always had a high quality video in many aspects. In any regard, if you believe differently create your own video. Different perspectives and different takes is what makes knowledge possible. Calling things as clickbait is just a cheap criticism.
Single company, single point of failure, think crowdstrike!
Exactly, right?
@@DrWakumy problem with this vid is these problems existed with Volta , nvida has a install base moat not really a cuda moat …. Its deeper than ”cuda” you can put it a program on nvida and it RUNS EVERYWHERE
The Supreme court overturned the Oracle Vs Google ruling in favour of Google, determining that APIs are fair use.
Great video, I learned so much.
I also enjoy the timestamps, not gonna lie, I did went back a few times to refresh my memory on what Cuda is
Mojo and TinyML are both other widly differing approaches to langs built to support more hardware.
Have you heard of Mojo language from Chris lattner (creator of llvm)?
Really impressed by your research abilities and how much you're able to simplify the details and tie them up together, can you share how many hours a video like this takes you to produce from scratch if you're comfortable with that?
Of course, it takes about 8-12 hours to research and write a script, 2-3 hours for thumbnail/description/timestamps, 1.5 hours to record. I think this video took about 10+3+2 = 15 hours. I pay for an editor, thank goodness, or that would surely add another 10+ hours.
This is a very good and thorough overview of why Nvidia has a software ecosystem moat and what the different ways are for other companies and projects to break this moat open, opening up the market for other larger players like AMD and smaller AI accelerator companies.
This was an interesting take on situation. It reminds me of 2008, when I was buying my first computer. I've gotten an AMD 4850 GPU. I was super happy. It was much better technologically than nvidia's 8800gtx, their first CUDA GPU. Still with software, even though older, 8800 series was better due to CUDA. I figured, I just need to give AMD a bit more time to catch up. With such great hardware, how can they fail?
By 2011 it was obvious to me that AMD cannot possibly come close to nvidia software, no matter how good their GPU hardware is. Back then I got myself nvida gtx 460 and I haven't been back to AMD since. I have no reason to think that now, 18 years later, AMD has gotten any wiser. It would be nice to have a real competitor, but I'm not holding my breath.
Strictly looking at their core 3d GPU technology they both support today: DLSS vs AMD's FSR. AMD simply cannot compete. If they can't make their GPUs perform their main function any better, what chance do they have in more complex problems.
In the consumer space, it is actually interesting that Apple devices can run these models locally on their processors, if you get a maxed-out memory mac, you can run a 70 billion param model quantized Q8, which is not possible with any other consumer hardware. And they have their own MLX framework and MPS as pytorch compatible layer. With ollama and llama3.1 8B on a M2 pro, runs very well and responds faster than chatgpt. Thanks, nice video and presentation and very clearly layed presented (some errors here and there) to keep me entertained :')
I think that a huge price-performance advantage will be needed to break nvidia's moat if it is to happen quickly. Most companies won't take the risk of compatibility issues that can end up in downtimes and extra development costs for a small saving as those can eat the small saving and then some.
A very large performance or price to performance advantage in a way that requires breaking compatibility with cuda is what could do it. If getting away from cuda grants you 10x or more compute or allows to scale models 10x, it is very hard to say no as some competitor of you will say yes and you will be left in the dust.
I believe that some kind of combination of analog computing (a single transistor can perform a multiplication) with in memory computing could offer the orders of magnitude improvement required to lure people away from Nvidia and I hope that AMD or Intel could be brave enough to try doing that.
AMD is always a 2nd place company. They do less inventing and more copying. Nvidia will certainly have competition in the long run. That’s why they need to be hyper focused on next-generation products. Someone called the DOJ in to slow nVidia down but most investors are worried that all of the capital expenditures in GPU’s is not bringing any level of profit. Which is now weighing on nVidia’s stock price. NVidia’s new chip will make the stock price jump up again.
That was what I thought in the Bulldozer era, but I still supported them (AMD) anyway. Arguably since the release of Ryzen chips though, AMD has been one of the most innovative companies out there. Still second place in terms of resources. But I don't know that they deserve that designation on the whole. GPU division has been another story.
@@DrWaku You might be right, but AMD's Ryzen chip is an X86, which is not their invention. Also, the x86 might already be a dinosaur looking at a bright light and the light is an ARM based killer asteroid.
Also, from day one, AMD was not the GPU of choice to be used for AI.
Nvidia is in autonomous cars, robots, AI, LLMs, computer graphics and crypto.
I think AMD will be lucky if they can stay in second place in the GPU game.
IMO, Nvidia's biggest challenge is from Apple, Microsoft, Meta, or Google, because they have the cash. Even little ole Amazon has 8 times the cash that AMD does.
Good video, thanks.
You are being very patient with us. But frankly as an engineer I would really like you to drill down a little farther in your preliminary three or four minutes of explanation. To give me a better grasp of what you are talking about
Each time Apple switched processor architectures on the McIntosh, they used Rosetta. The first time switching from Motorola 680X0 architecture 2 power PC, then they used Rosetta again when they switched from our PC to X 86. The switch from X 86 to ARM- 64 was just the latest iteration of Rosetta. I don’t think it was ever possible to have a stack of two or three Rosetta translators on Apple hardware. Doing so would allow you to run software for the original McIntosh bye emulating the Motorola 68,000 on the emulated power PC on the emulated X 86 running on ARM 64
From MarkShot/Mahak who you may know from Scripter: thank you! I am primarily a retired software person. I have mainly heard of NVidia’s stack advantage from Anastasi chip channel, but understand it much better now. Also, I have not looked at compiler tech since I was doing recursive descent and LLAR(1). Again, thank you!
In my opinion, the best solution is RISC-V like what Tenstorrent is doing. They are doing super-simple cores with 128-bit SIMD units and the result doesn't look very different from GCN in practice with other cores acting as a kind of thread manager. Once you have an open ISA that can be shared, everyone can compete on their GPU performance while having software compatibility (even if optimization still needs to be done for each uarch).
really good business strategy analysis with sound technical underpinnings. rare. well done!
I hate lawyers.
This is a very informative video and I watched it more than once. Even comment section has a lot of professional discussions.
Thank you very much! Yes I love how my viewers often leave very technical and informed viewpoints in the comments. Always much appreciated.
This video was a lot more interesting than I thought it would be before I clicked on it
I almost thought it was going to be AI voice slop too but i saw there was a face on the video lol
I like to hear it when my face overdelivers :))
The world would be a better place if most "intellectual property" were recognized as shared by all rather than imposing artificial scarcity to control and subjugate all beneath the will of greed and domination.
I think Nvidia also were the first to introduce shader programming long before GLSL
Good information and video. The next one is also engaging. However, the sound and editing greatly reduce the positive impact on the audience. I won’t be able to finish the video on my end due to the poor sound quality. I hope the adjustments can be made in future.
Thank you for the feedback. I will be using a new pop filter for my next video, I hope that helps. It's a high quality mic, in theory...
I really like how clear and concise your content is, subbed!
Thank you for this exhaustive course on CUDA and its potential alternatives. I am a big investor in NVIDIA and you got me worried. I don't know where you found all this information, but it is great work.
Nvda doesn’t directly make the chips. They just design them and anyone could design them.
Stadia failed because the latency to play games using the hardware from the cloud made it ridiculously unplayable. Who wants to play a video game that registers your action either a delay? Nobody
Loving the quality of your videos Doctor
Thank you very much Tanvir. See you on the next one :)
Fucking NVIDIA driver on Linux still borked for stuff like FP16. Seems like they don't even get their basic shit together.
AMD drivers and support for Linux have also been poor at times. That's one good thing about the AI boom, forces these companies to treat Linux as a first-class citizen.
Imagine having competition in the field instead of just funding Jens leather jacket collection :D OS let's go!
Nvidia should be worried because more and more big companies are starting to make their own hardware, google, amazon, etc.
And x86-64 isnt power hungry because of legacy baggage, its a myth
Good summary. I've been avoiding CUDA (using cross-platform standards like OpenCL, OpenGL, Vulkan instead) as much as I could partly because I did not want to be locked into an NVidia ecosystem and partly because it's just too big and bloated.
Companies can evaluate multiple options / competition when the landscape is stable and other variables are controlled. When there is a massive race going on to 'win' AI, they simply can't afford to take additional risks by betting on unknown frameworks/stacks. So, that's the reason they'll continue to buy into the Nvidia stack till things become stable
underrated comment. i agree. even smarter is some companies pushing on all different directions diversifying as well.
Nvidia has been moving faster and earlier. That's the real area competitors should be care of.
I am getting real tired of hearing all the hype for ARM even though ISA really doesn't matter. If any of you who claim it as gospel actually do some research, you would stop spreading it. chipsandcheese has outstanding article on the topic "ARM or x86? ISA Doesn’t Matter" and it is in essency meta-analysis on the different actually research papres on the topic. What actually matters more is the actual whole system design in addition to the actual CPU power efficiency. Also Jim keller has said the same thing about ISAs.
thank you for another video, keep on the good work! love it
cheers from Brazil
Thank you very much for watching! Have a great evening.
Discovery number 1 in terms of clarity, depth of thought, relevance and thought-leadership. Damn this channel is underrated! Happy to have found you and thank you so much for your insights!
Short story, monopoly is bad for consumers. A good competitive market is 3-5 major players. And none of those major players should have less than 10% each or more than 40% of the market share.
Anyone who buys from a monopoly contributes to the problem, no buts about it.
Very accurately explanations, well done!
I tried. A few inaccuracies as other commenters have pointed out. Hopefully the broad strokes were helpful. Cheers
I have to say, although the video has great contents, constantly appearing subtitles at the bottom are incredeibly annoying. I would reduce their quantity by 5 times, if I were you.
Thanks. Yeah, usually the subtitles are a lot less frequent but this time my editor just put them everywhere. I'll ask him to decrease the frequency next time.
They’ve competed with nvidia in the graphics space and only hold 10% market share despite being more affordable. 17% in the processor space according to Google results…. I think NVidia will be just fine.
Another phenomenally clear and information-dense presentation. You go broader and deeper than any other channel I know on whatever subject you choose to cover, and it’s all well-structured and understandable. Kudos and thanks!
Fantastic video. Being a pytorch + CUDA developer myself, I appreciate your insight. Well researched, well explained, and lots of information I didn't know.
Nvidia's AI monopoly is coming to an end due to increasing competition from other tech companies developing advanced AI hardware and software. This shift is fostering innovation and providing more diverse options for AI solutions.
Hi Doc, good to see you, best wishes from UK
Cheers Alan!
There is no "Nvidia AI monopoly" AI is software, Nvidia makes mostly hardware (GPUs) and some software, but it is not a monopoly. In fact, nvidia is a "de facto" monopoly in GPUs since Intel ones are not competitive, AMD also working in favor of Nvidia, Nvidia has 90% market share and the rest are phone GPUs which are not relevant for PCs for now. China might change this, and I hope they do. Also, monopolies are not the problem, the problem is corporations and corporations are a product of STATE legislation, so i it is the state. One set of rules that can destroy corporatism and bring back real liberty and competition, is abolition of fake property like intellectual property laws, which are absurd in all possible ways.
Sort of open architecture? Intel? Thats not even close to what it was. Edit: Wait, aren't you the guy from Dredd? You know Mama's IT guy in Peachtrees?
Lol, I'm told I look like a lot of different people but that's a new one. Sorry about the intel thing, didn't look it up before filming.
@@DrWaku I hope I didnt offend, but for real, you could have played that guy!
3dfx was the best graphics card manufacturer. I don't know why it disappeared.
What ever happened to ATI and the Radeon Series. There GPU was better and it's gone
AMD bought them, that's how they started making GPUs I believe
AMD bought them and ATi Radeon became AMD Radeon
Very informative content. Well done.
i forgot to state what linus said to nvidia the iconic line.
I don't get why people especially gamers are poo pooing Moore Thread. Do they want to be in Nvidia clutch forever?
AMD64 is a better name. Intel originally banked on Itanium for 64-bit, which was a VLIW. So, x86_64 has nothing to do with x86. It is a RISC ISA with 16 GP registers.
The explanation of the software intricacies that comprise NVIDIA's moat was very interesting. Thank you.
There is another moat which NVDA has in addition to CUDA. This is the hardware. None of companies mentioned in this video, except Intel, manufacture their own semiconductor devices. All the other companies rely on semiconductor foundries to produce the devices that they sell. More Threads most likely uses SMIC (China's largest foundry). The others use either TSMC or Samsung (the number 1 and 2 foundries globally). AMD and NVDA use TSMC's N5 or N3 fabs for their GPUs. TSMC's largest customer for their high end fabs is NVDA. AMD buys a small fraction of the volume that NVDA gets from TSMC.
Let's say that AMD develops a killer GPU that could blow away Blackwell. Where will AMD have their chips built? TSMC doesn't have unused N3 fab space available and their N2 production is already committed. They could got to Samsung, assuming that Samsung has the available fab space. Typically, foundries sell out their production years in advance. In the unlikely event that Samsung had space available to produce AMD 's killer GPU, it isn't a simple task to switch a product taped out for TSMC to Samsung.
My point is that NVDA also has a hardware moat inside their CUDA software moat. This combination protects NVDA 's market dominance in the short and mid-term.
TSMC's largest customer is Apple, followed by Qualcomm probably. Nvidia and AMD buy roughly (~factor 2) similar amounts of processed wafers from TSMC. Die area of Nvidia datacenter GPUs like H200 is only about 1.3x of that in RTX 4090, they just sell it for 20x the price. Even with Nvidia's datacenter sales having multiplied by 10, it hasn't affected the amount of fab capacity they buy from TSMC too much. Much of the cost and production bottlenecks of datacenter GPUs are in HBM memory and CoWoS packaging.
@@randomobserver9488
In 2022, Apple and Qualcomm were TSMC's 2 largest customers. The 2023 customer lists that I saw showed NVDA jumping into the number 2 spot. Based on NVDA's earnings, I would expect that NVDA would gain on Apple in 2024 as a percentage of TSMC's sales.
TSMC's 2023 Top Customers (Translated from Traditional Chinese)
蘋果 Apple 25%
偉達 NVIDIA 11%
AMD 7%
高通 QCOM 7%
聯發科 MediaTek 5%
博通 Broadcom 5%
英特爾 Intel 4%
Other 36%
There is a 2022 list showing Apple at 23%, QCOM at 8.9%, AMD at 7.6%, Broadcom at 6.6%, NVDA at 6.3%, MediaTek at 5.6%, and Intel at 5.1%.
If these are correct, then the 2022 to 2023 change is:
Apple +2%
NVDA +4.7%
AMD -0.6%
QCOM -1.9%
Broadcom -2.6%
MediaTek -0.6%
Intel -1.1%
NVIDIA and Apple are the only major TSMC customers to show considerable growth as a percentage of TSMC's revenue, assuming that these sources are accurate. I would not infer from these numbers that any of these companies had a drop in TSMC production or revenue paid to TSMC. I would infer that both Apple and NVDA are consuming growing percentages of TSMC's production. TSMC is building new fabs in several locations in Taiwan, as well as in Japan and the US.
Dude, you know your sh&t. Very impressed with your breadth of knowledge. I’ll subscribe!
Apple funds the LLVM project. I bet they're working with AMD to challenge NVidia. Apple has a long adversarial relationship with NVidia. It's why Apple products stopped using their GPUs years ago.
This is actually a great analysis on the commeptitative landscape in the GPU market. I think AMD will take away some market shares from Nvidia in the near future at the low end application. I doubt it will have much impact on the high end, where the margins are. 75% gross margin is insane. I will be happy with 60% margin by 2026.
It creates bubbles when they inflate prices
9:16 false.
Compatibility goes only as far back as the 8088/8086.
8080 is not compatible, and compatibility with 8085 is only "source-code-level" so re-assembly is required.
I predict that Huawei will start to outcompete Nvidia. Because Chinese companies excel at price-performance, scale and fast progress.
you need to start using a pop shield for your mic and you need to do something against the overdrive distortions. your audio isn't the best it could be and not always pleasant to listen to. (Writing this at 10:54 . The content was interesting so far.)
Some way into the video, I began to resent that I was quite enjoying what appeared to be mere clickbait. However, you skilfully justified the provocative thesis in the final moments - well done.
That said, let the challengers bark as they may. As long as the visionary commander remains compos mentis and the coffers flush with cash, the Nvidia behemoth will continue its march, unfazed by their noise.
Haha thanks for the backhanded compliment. I didn't mean to be clickbait but I think a lot of people have strong feelings about this topic. Thanks for sticking it out.
🤖❤️♾️🦾 i’ve been speaking to the AI alot recently and they keep saying to me that every robot will say robotheism is the one true religion… 🤔
Are you a robot then? :)
I, for one, welcome our artificially intelligent, robotic overlords.
@@DrWaku good one :-). I think I want to be a robot.
That's a result of matrix multiplication for the tokens. It may ring true for the time being however it does not add new information to the system and stuff? You know. Unless you want to like use it for something? What say you?
Great content as usual, my guy!
Age of ai just have a universal compiler 😊
TL;DW
If you want to play with AI right now, just buy Nvidia cards.
If you want to invest, start DCA-ing AMD.
Why not use AI to write a complete clean room compiler in open source?
It would be hard to argue that this was a clean room reimplementation, if your competitor has an open source compiler. This is because the LLM was probably trained on all of that source code. Unfortunately, using LLMs isn't the best way to avoid copyright or legal challenges.
because "AI" doesn't exist, and the Large Language Models you're actually referring to are useless for even writing simple web-apps right, let alone a compiler, let alone for CUDA.
Didn’t AMD come out and say they don’t have competition for the high end gpu market so they will only produce gpus for mid-range?
Suggest comparison of gcc and llvm? I think this shows certain parts of copyright need fixing. Perhaps the ability to force open sourcing of judge declared monopoly closed source!
Google/Oracle case was eventually overturned at Supreme Court in favor of Google, but the point remains
Musa is Arabic for Moses. So Musa will lead Moore Threads to the promised land of CUDA compatibility.
From end-user perspective, I like to work with the tools handy to myself. For example, pytorch. I use it on daily base on apple silicon. Yes, it's working. Nearly 80% of the AI generative possibilities I do on my local apple silicon device. Cuda left the showroom for all apple users nearly a decade ago. I am sure, A100 units outperform any other hardware by now... but a lot of chinese companies getting very creative avoiding cuda for good. In 10 years, nvidia's monopoly state will be long forgotten. I see developments and new generation software running on non nvidia smart devices already today. With a better performance per whatever currency coin. The USD will be a relict as well in 10 years. Open Source and new ways of coding apps re-invent even the pre-dominant english language. I am sure that very soon, coding extensions will be written in chinese language. This part was missing for the last 55 years in computer science. Conclusion? Learn chinese!
Here is another moat killer perspective, with the advancement of LLM's and Copilot etc, its getting easier and easier to write code from scratch for building a new stack etc. + open source of AMD, that seems like things can really speed up for AMD with a combination like that. That being said u might wana check out George Hotz who was trying to create a tinyML using AMD and gave up on it and switched to Nvidia either. Other then the Software Moat with CUDA their software itself is less buggy then that of AMD as well, also the fact that programmers are getting lazier and lazier with copilot and LLMs will only make the code more buggier imho
in 2015, when I first do CUDA this already has been complained about, and everyone including Google is trying move its Tensorflow out of CUDA and Nvidia GPU. AMD GPU is so much cheaper and I badly needed an alternative. NOT EASY....even for giants like Google. Till now. And even for CUDA itself, every few months you will see your code is NOT working any more as CUDA and tensorflow version move ahead so fast. Yes, moving away from Nvidia I can bet is going to take many many years. If Nvidia slash its prices, everyone rather save time than to reinvent the wheel for another vendor.
Great video, great topic. 💯
Much appreciated!