Being hard on code and neutral on LLMs is not a contradiction. If someone submits terrible code it doesnt matter if they used AI or not, if they submit good code it also doesnt matter. The point is that you judge code on the merit of the code, not the source. Its honestly really strange to try so hard to apply a moral or even value judgement to something which is just an inanimate tool that can be used or misused.
Yeah it just a tool, but knowing how much Linus hated C++ it's kinda surprising that he is so neutrals towards AI when there is so much garbage code produced by it.
Its cause if its shit you dont really improve anything and have no idea where to even look to make it not shitty. Its fine if your end game is immediate satisfaction. But if your aiming to exceed the average. Then relying on a "tool" that averages isnt the way to go
You have to understand the context of the situation. This was a live interview. Linus couldn't afford to be as much a keyboard warrior as he naturally is. I'm completely convinced that would blow a fuse if someone submitted a patch to the kernel containing LLM code.
I think Linus is ambivalent or neutral about LLM coding because he doesn't direct his anger towards unconscious, inanimate agents. What he gets upset about is when a human, who should know better, tries to merge garbage code generated by an LLM without understanding what they are attempting to merge.
I think that you are correct but Id also like to point out that getting worked up over llms is fruitless at its core. Its an exercise for people who dont understand the world.
Yeah. There's a lot of money to be made in capturing attention by selling outrage stories of LLMs and just AI in general, when it's really just repackaging people problems into a new shiny exterior. That's not to say skepticism is unwarranted, but you're gonna have a more focused discussion once you isolate human decision making elements.
AI for coding is (currently) basically a replacement for Stack Overflow and Google If you just plug in AI generated code into your system, you're gonna have problems, just like you would if you copy code from SO as-is If you consult the AI, learn from it, and review what it produces and how it stacks up to your needs, then it becomes a net positive force that can both help your with trivial boring tasks, and also teach you things
Re: LLMs - you should read/review the recent paper on LLMs and learning outcomes for students. They basically found that although LLMs helped students improve *while* they had access to them the overall learning outcomes were poorer when access was taken away vs when access was never provided. Basically students who learned with LLMs became poorer at learning things in general or at least didn't improve at learning things compared to their peers who didn't use LLMs.
@@-weedle that's the weird bit. Just saw it reported on yesterday and now I can't find it. Actually have the video paused at ~7:30 because I'm off in another tab looking for the darned thing 🤣 Will update as soon as I find it.
I am objectively a great programmer (as judged by my peers over the years during my carer), and I like Copilot very much. I don't think it made me better, quality wise, but it made me faster on the boring tasks.
The reason he's so chill about LLMs is because he trusts his review process. He nitpicks everything and takes it seriously. Therefore it doesn't matter if the code came from LLM or from a human, it would still need to go through him or his trusted review body.
@@atiedebee1020 Block them. If you submit bad code and don't correct I am very confident they just block them, or through them in spam, or whatever. and maybe, just maybe if AI leads to more new contributors who feel confident to submit code for the first time because of AI, create a learning channel for them if they need additional guidance on how to review LLM output before submitting it. You know assuming ignorance but good faith.
And in 3 years time, we have neural networks in our compilers warning you about your novel solution because it deviates from the average quality of all the code it was trained on.
Why would data scientists make average quality code datasets? You have to assume that data scientists are complete imbeciles for them to purposely train LLM's to make dumb suggestions. If it's overly suggestive, then the dataset will be changed to make it stop suggesting so much. probably through RLHF.
I'm not a great or experienced coder, but one issue I already see with LLMs is that it breaks an important aspect of coding, which is the dissection of idea implementation. A huge benefit I get from coding is that it forces me to really think about what it is that I am trying to do.
Kernighan's Law suggests that debugging is twice as hard as writing code. Letting the LLM write the code and then debugging the result is a direction with subtle issues. It probably means that you can crank out your low-end work even faster than before, but you may not be able to improve the quality of your high-end work at all. And Amdahl's Law would suggest that making your low-end work easier to do may not free much if any time up to put more hours into your hard jobs. The problem in that case isn't in having time to do the actual hard work, it's that your job involves grinding through boilerplate.
My general rule of thumb is to use LLMs to learn how to *approach* a problem, then go figure out the details myself. If I am ever asking it about specific numbers in a problem, I have strayed too far from its purpose (in my opinion). Something like: "I want to make ____ kind of project, how might I start that?", or even: "I am stuck on ____ step, what might be a few good things to try?" are both fine. But nowadays, as soon as I search anything like: "will this loop go out of bounds of this array?", I start a new chat because I shifted its focus too far in the original. Once the numbers are wrong, I don't think I've ever seen them correct themselves. In rare cases, I'll ask it to explain what some code will do, but that's only if the documentation is truly abysmal, which to be fair, sometimes it is. I just see it as a way to sift through all the more niche or hidden code discussions online.
I agree, it's the best rubber duck short of an actual human subject matter expert, which often times you may not have access to depending on what kind of problem you are working on. Bouncing ideas off your significant other for instance, is probably not going to be useful if you are trying to write something like an inverse fast Fourier transform, but the LLM will have the context needed to plan an approach. Using it to actually write code is iffy, it can get you like 50%-70% there often but you may end up spending more time fixing the output than it would take you to just write it.
yeah this is how I've used it. I wanted to make a music app and all the code it gave me didn't work because the library had been updated but it suggested patterns I could choose to adopt or dismiss
@@gljames24 Basically, though it's a great rubber duck that LIES. And if someone doesn't know enough they are literally incapable of spotting the lies. And way too many treat these LLMS as sources of truth. In part maybe because a subconscious misconception views everything the LLM generates as based 100% on exact-words a person has written in that sequence like it's a search engine and not an adlibs slot machine.
@@TheSulross As it should be, CRUD already been done for almost three decades, will be weird if AI can't learn it with that amount of dataset. But last time I check it kinda shit at doing visual to code based task (like asking to generate HTML+CSS with certain visual specification), they will do the bare minimum and then unable to expand it to something that you want.
@@TheSulross there are already alot of projects for this, it's super optimistic to even refer to the output of such toolchains as "prototypes" considering the amount of unmaintainable garbage, nonsense code, and refactoring they need.
@@ch3nz3n It never will, the whole technology is a step in the wrong direction as far as real AI goes. All LLMs by their very nature can only rearrange and regurgitate that which already exists, it literally can't come up with something new because the underlying algorithms just take what it's been trained on (stuff that already exists) and tries to rearrange it in ways that best fits the prompt. That's a massive oversimplification but at it's core that's what it's doing
AI is going to be the corporate equivalent to buying a $3000 Gibson, Les Paul guitar thinking it’s gonna make them a better player without learning how to actually play.
Even though I believe in the potential of AI, I am against corporations having a stranglehold on access to it. The future should be a place where we can all develop AI the same way we develop applications. Corporations apply the classic tactic of turning people into helpless consumers so they keep paying for whatever services that are being peddled. Independently assembled AI should be the direction to move towards.
Actually it is more like buying a $3000 collectors edition of a Harley Davidson bike. In 1:24 scale. With "real working engine", that is literally just a translucent engine block that gyrates the pistons if you turn the wheel.
I have been programming in C++ for 30+ years. I use LLM's in all their forms for coding. Using an LLM for coding successfully involves breaking off chunks of functionality that it can handle... and it usually involves defining function signatures for it. You'll only know what an LLM can handle by using it a lot. More complicated uses can only be tackled by providing it with extensive guidance in the form of pseudocode. Also, I never "trust" an LLM. I have to maintain the code so I have MUST understand it. Yes, they do make mistakes... but given the size of functions I'm asking it to write those mistakes are usually easily spotted.
@@censoredeveryday3320 I pay for and use ChatGPT & Claude mostly for technical discussions and exploring ideas (though I sometimes use them for code generation as well). I use github copilot is vscode... and as of last night I use Cursor pro.
It's ok for self isolated functions, usually, but it falls apart when it needs to interface with multiple systems already designed. And I would NOT use it in memory unsafe languages.
I used gemini for a short while while writing my thesis work, but after it didn't help but instead i had to give them the answer, i stopped using it. i dont have much experience with them but i still think it can be useful for simple tasks or to have a starting point for figuring out something, but as a replacement or trusting its output without confirmation, i dont think it's perfect.
that just means you don't know how to use them or are bad at natural language. It's like being given freshman college interns and giving them tasks too hard for them
If tell the LLM to write something it's usually bad (even GPT4o struggles with regular expressions). But something like codeium in vscode auto completing a line I've started is almost always correct, saving on keystrokes.
Totally agreed. I tried a LLM stress test where I asked Mixtral 8x7b to make a console Hello World but with WinMain entrypoint in C++ without using any function from the standard library, only functions from the Windows API. In my requirements, the code had to work properly whether the UNICODE macro was designed or not and there had to be no #ifdef UNICODE in the LLM answer. Let's say that it was an abject failure. There are exemplars on how to do each of the specific tasks I asked for on GitHub and on Stackoverflow but they are few and far between. To code it, you're better off just with the MSDN doc 😂
Then dont write prompts that produce mistakes... Its not hard to guestimate the ability of an LLM and decide to only ask it questions within its range of ability.
As someone who is a novice with code, I concur with your opinion. It's vital to have a strong foundation of understanding, and an LLM should supplement this, not replace it. If you always take shortcuts you will never build up the knowledge and skills to do anything well, and I think this is true for everything, not just coding
@@Karurosagulinking the entire documentation and then asking your specific query is faster as the "getting started" might not answer the thing you need.
@@HRRRRRDRRRRR Most of them? IDK, I think the quality depends on a lot of factors And by the way, English is not my main language and I've read many docs without problems wether they are poorly written or not. Most problems i've had have been with: very new libraries and frameworks, very specific topics within existing docs that haven't been updated after a new release, misinterpreted features that turned out to be hotfixes and then got removed, and so on TL;DR Skill issues
I am with Prime here, in the sense that making gpt write parts of the code is no problem is fast is great, but eventually u don't know what is going on in the codebase. Specially when you return to a project 1 month later.
nah, I use copilot all the time to create little scripts for me. They are often wrong at some places, so I have to either modify the code or tell copilot to correct it. But it still saves me a lot of time, as I just have to tell it which language, which input I have and which output I expect and it writes code that mostly works.. the rest is some discussions. Or the 10% bug fixing. It also helps a lot if you are not really familiar in that language. e.g. scripting for powershell, while you are more a Linux guy ;) Still the code is more readible than most developer code I deal with most of the time, Maybe a little too much comments in the code :P but you can remove that as well
For me personally, as a low/mid level IT support, LLM's help me a lot, because i was always a quieter person and somewhat shy to ask questions to the seniors at work... With the LLM's there is no such issues and i excel at the tasks quicker ^_^
@12:00 This is the crux of the problem. Using the 'always blow on the pie' quote. Always create and run the tests even on LLM created/guided work. This isn't just field specific.
LLM turns all bugs into subtle bugs. LLM turns compilation errors into syntax correct bugs with logical flaws that take ages to discover what when wrong.
@@alexandrecolautoneto7374 users asks "make this" > LLM outputs > user copy and pastes > compiler fails > user gives LLM negative feedback > LLM model evolves to avoid negative feedback > stealth code. Currently reminds me of so many web performance "services" that just insert javaScript to trick auditing tools to spit out higher points. Mission accomplished for everyone that doesn't understand the actual code.
This is a skill issue. In 10 years, the mark of a good programmer will be their ability to debug LLM code. Prime & others are coping because the introduction of LLMs caused a total paradigm shift in regards to writing good code quickly. NVIM and all this other DX shit they obsessed over is a brick compared Cursor AI + Claude. These guys jerked each other off over their WPMs, but are trapped in their old ways when something better comes out.
@@exec.producer2566 I loved the theory, but the reality is that LLM are just the not right model for coding. No matter how we improve it will always hallucinate, it's just how they work.
His view is probably not negative because the code that he reviews is from developers that know how to use AI. Meaning they don't just tell a LLM to "make a driver in rust" they just use it for tedious repetitive code tasks.
LLM's are great replacement for searching for obscure methods in API documents. When string matching doesn't cut it, I always resort to LLMs. And they find stuff that I couldn't find in a few minutes with 70%~ accuracy
My problem with LLMs are currently: - For my hobby projects they are a useful help to bootstrap and do boilerplate. For example: Create me an ORM for this database schema. - At work: Large codebase in Typescript, Dart, Go and Rust - Copilot expecially is useless.With recent gemini and Claude Sonnet i had slightly better experiences but it's still awful. My longterm concern is: Is the production -> training data loop without much feedback mechanism in between. It has already been shown that the quality of LLMs declined the more you feed them LLM produced input data. So i currently won't rely on them much. At least not for code where it matters.
Linus talked about LLMs in future tense. He never said anything about using them for programming here and now. I think he's just optimistic about their potential in the future.
5:45 I interpret Linus's opinion here as "LLM can be a great code linter but you should assume its output as opinion about the code and then decide by yourself if you want to actually change the code". Though this obviously assumes that the developer skill issues are more about the accuracy of the implementation instead of overall algorithm or mis-understanding data structures or thread locking.
I have been using LLMS since quite some time. I am not a full time developer but have to write some python code from time to time. It really helps me when I start a script. You just tell it - I need a class named XYZ with methods a,b,c which have the following parameter and which return that. The script has the following command line parameters … etc. And it will perfectly do that. Then you have to kick in and write what you need your script to do. From time to time you ask how I can get this and that done. At the end you can let it help you to optimize your code in a short time if you for instance have a little bit too much if then else stuff in your code. But at the end you have to understand each line and judge what is a good recommendation and what not.
Best use I have for LLMs is as a user integrating very basic features in websites through grease monkey. Doesn't take long to change a website that requires hover on an icon to show a picture to one that shows them by default. It's not hard, it's not code that will see reuse. It's just fiddly normally. And with chatgpt you can actually just roughly ask it with the right info and receive a good enough result.
It’s interesting what you said about software development and understanding the problem domain, and bugs due to edge cases or things you’d not quite fully understood. Because that’s sounds like you’re coming up with a solution to some kind of problem within a set of constraints (or trying to understand what the problem actually is about and what the constraints are.) It’s like a level higher than a particular programming language. It’s more like designing and being able to understand certain types of problems. So say if you were using a language that just didn’t allow certain classes of bug (like memory errors) so it was high level and the LLM didn’t need to generate that kind of code, and it became more about expressing solutions to known problems (I want to say applying patterns but I don’t mean design patterns - something probably more high level) and if an LLM was working at this level then I think they’d be really useful. “I can see you’re trying to do this kind of software, have you thought about applying technique / approach / algorthim X.” If you could somehow turn that knowledge about problems and their solutions into some abstract model that LLM could use to spot patterns and suggest techniques, to help you understand the problem space, then that’d be good. I genuinely don’t know if LLMs work like that at the moment. 🤷♂️
A lot of coding is repetitive work, gpt is really good at repetitive stuff, that's where it helps. If you manage to build your code good enough where it is composable and reusable, the LLM will see the pattern and suggest you ways to compose it correctly
@@specy_ But that isn't making my code 10x better, it's just getting it done faster. If all it's doing is recognizing the pattern of what I'm coding and completing it, then the AI isn't making me do my job better, it's just letting me do the same job but slightly faster.
@@kaijuultimax9407 yeah ofc it won't help u make better code, but it helps u make faster code. If u can save 50% of your time when writing one feature, you can use that time to make it better yourself.
I've found LLMs mixed. The line-based auto-complete is 80-90% useful, especially writing similar repeated code. The other times it has got in the way, but on balance I generally prefer to have this functionality. Using LLMs to ask questions, I found it helpful when trying to identify a Bootstrap class -- my Google searching didn't find the class, but asking an LLM helped me find the class name that I then looked up in the docs. Some other approaches I've asked for I've ended up adapting the code to the way I wanted to write it, using the LLM code as a basis. In some other instances it didn't help me solve the problem so I used different approaches.
LLM-assisted development development is worse than StackOverflow-assisted development. I am not worried. There will be more work to fix the LLM fallout for the professionals who know what they are doing.
As a freelancer with 30+ years of experience, mostly working on projects for a few consulting companies, I’ve found LLMs incredibly useful. Switching between languages like Python, React, C++, and C# means I often forget specific details, especially with new updates. Recently, I was assigned to a next.js project. Yet another js-framework, 😞 , and LLMs have been a lifesaver for quickly getting up to speed with syntax. They also help generate solid code with good comments on common code stuff, including unit tests, which keeps my workflow pretty efficient if I can say so.
AI is great for remembering syntax with context. You don't ask it to build a house which its terrible at; you ask it to build a wall 4 times, a floor, a roof... Etc which it's actually pretty good at
I really like whatever Microsoft has baked in to the new visual studio where when you're refactoring a project and you make a similar change 2 or 3 times, vs will give you the red "press tab to update" the next time you move to a similar line. It sure beats trying to come up with a regular expression to search and replace. Simetimes.
16:32 not negative, but he has very high standards. He's like Gordon Ramsay, who can be an asshole in the kitchen, but is a sweetheart when you see him outside the kitchen or interacting with children
I think LLMs for coding are great, but turn off the 'auto-suggest' feature. That stuff messes up with your thinking process because you have an idea of what the function is supposed to do, but at that exact moment a whole bunch of code appears and you've got to read through an lose your train of thought, and maybe sometimes it's what you want, and sometimes it's not. Instead I think having that bound to a key is super nice, so you write your function, and code as you normally would. Then you stop, and realise it's OBVIOUS what your about to do next, you know it, the LLM knows it. so save everyone time by hitting that 'manual' button and have the code pop up when YOU ask it to. That I think is just 'better auto-complete', and it's amazing. It's the perfect combination of automation and still having enough control to get to where you need to go without ending up with a bunch of code you don't understand and can't debug.
15:50 maybe not dumber, but out of practice. I'm a full stack independent developer that employed a frontend dev for a few years. I find that using LLMs is the same if you don't "take in" all the code that is produced
it might also be because even though you are using an LLM for help, the PR you produce will always depend on the person doing it. Whether they rely fully on it, or not.
Handwritten code or AI tools code are the same, if you have the patience to test/debug the code Don't blame AI tools for your own limitation, for not knowing how to properly test a code or just being plain lazy, if you don't know how the code works it is your responsibilty to learn it, otherwise it is like a dagger above your head once it is deployed in production
From an automation standpoint, I'm hoping/expecting AI to find a useful home in property-based testing. Helping to more quickly zero in on corner cases that the implementation missed.
LLMs have come a long way for coding even in the last few months. Sometimes it's about identifying the use cases its good at. One of the best use cases I use it for is dumping in an entire project, often times with lots of spaghetti code into context (Which now get as big as 2 million tokens and can be cached on repeated calls to save money) and asking it to locate the parts of the code that do X. It will surface the code, I can do a quick find and I'm on my way. It probably grabs what I need 80-90% of the time on the first shot with modern models. It seems kind of like common sense but it's a good idea to use LLMs for the aspects of coding they are good at and probably a bad idea to use them for coding tasks they underperform on. Unfortunately, things emerge and change so fast what an LLM is good or bad at coding wise is shifting quite a bit and not obvious out of the box.
The argument that I feel you miss a little is that there are different ways you can use it. You are absolutely correct that knowing the code and being a good coder is a must but the bits you can bounce off or syntax you can gather quickly is invaluable. Often times, I am going back and forth between docs and llms and I will know when the LLM is bullshitting and I need to go to docs. In a way your right but also there are other ways you can learn to use it that is more helpful. if any of that makes any sense.
LLM's writing code is my dream come true , i remember writing code on basic as a child on a commodore 64 now i can just ask the LLMfor a program at it just produce awsome results and evolving constanly . (i usaly use chat gpt , i remember the chat gpt's first steps in writing code from the codex playground sand box , it was so awsome for me)
You should do the recent François Chollet ARC Prize talk at some point. Getting takes on LLMs from engineers like Linus is more of a personality test than anything else at this point. You should listen to what actual AGI researchers think about LLMs.
I've pasted in code that doesn't work into an LLM that was non-trivial and it spotted a bug for me, where I got the memory ordering wrong on some atomic operations. I feel like this is the sort of thing where the right answer exists out there a multitude of times and the LLM can pull together all these resources and explain why your code is broken. It's super useful for that stuff, and is way better than just trying to only absorb this from the docs. Also it's a great rubber duck.
I have been arguing since GPT3 that AI will be amazing static analysis tools one day, awesome to see linus agree. It makes perfect sense as a good 30% of the bugs we catch in code reviews at work probably could have been caught by an AI (although maybe not a large language model with the current design)
It seems like he sees it like a logical consequence of C -> compiler magic -> assembly. Now it is AI -> some black box magic -> C -> compiler magic -> assembly
100% it still takes SMEs to be effective. The LLM still depends on the inputs being perfect to be right. Copilot depends upon inputs that are not always right, or not always relevant, or not always applicable. Code that is older version, etc. Still takes insights and SMEs to be useful
For me it's pretty simply. I meet developer, software engineer, sysadmin, network admin, cloud admin, qa tester, data analyst, data engineer, devops engineer or cybersecurity professional who lost his/her job because a random person who is not IT/CS professional can work their job using LLMs, and I'll be like "God exists, and it's AI."
I think it'll be either domain professionals using LLMs to do their job, or it will just be entirely automated. If you have a random person who knows nothing about the domain using an LLM to do the job, then you can likely just automate the job at that point.
There is a line where LLMs are helpful on one side and problematic on the other side and today they exist on both sides. You can use an LLM as a tool to help you work, some might like copilot others may like using LLMs to scope out a problem, and in other cases they may not be useful at all. I don't think that they will be replacing devs anytime soon but they will be alleviating us from some tasks. Using a LLM is like being part of a team an having to read the code of some other developer, you can read it for structure or you can read it for solving the problem it is trying to address.
The LLM's are very useful to people who know enough to check their work. They're a productivity multiplier. Additionally, they do serve as a decent second set of eyes. They do catch bugs.
The current paradigm just fundamentally doesn't work for identifying bugs. The code available to train on likely got published after the big bugs are fixed. The small remaining ones are not identified in the training material as bugs.
The difference is this: He cares about the results that land in his inbox. What tools people use is kind of besides the point. He will rake you over the coals if the patch you submit sucks regardless of how you got there.
They are not. They produce text like we do, due to emergent abilities. And we hallucinate all the time too. Wr just catch ourselfs often, many people do not
Whats bad is LLMs help me remember things ive forgotten. So the really rare things i use once every few months. And it speeds that up. WHICH MEANS I never remember those little things. An example is building the initial code for a program. run through all the logic a 1000 times and loops and so on. but when you want to do something like open a file, or grab a different library, if you only do that once every few weeks it's harder to memorize.
Your brain is a very efficient cache. If it doesn't retain something, it's because you don't need to retain it. Something you do once in a blue moon is not something worth remembering off the top of your head, since the extra time taken to re-familiarize with the concept isn't that big compared to everything else.
@@ruukinen Except I've found that it also means that you get less ideas that involve the things you rarely do. And some of my best ideas combine my deep understanding of the current system with smaller rarer things I encountered in the past. But when I use LLMs, my brain got rusty at doing that. So I backed off, and it feels a lot like rebuilding cardiovascular endurance after not exercising for a few months. Maybe that's just me...
Ideally it should be used for internal pre-checks for a PR, using for automated bug bounties is obviously bad. But if I can run against my PR to check if I missed anything before submission, sounds good to me.
Where I think I've gotten to is, when I get it by an LLM, and if it doesn't get it in 1 attempt, I move on to figure it out myself. As the amount of time I would previously spend on some error that was because of a single character being wrong, and wasn't picked up by the likes of a linter/lsp was frustrating. But beyond that use case I feel like I've felt it frustrating longer term, because of its lack of quality mostly. That and learning Neovim, I think of something that I haven't encountered before that I wanna do, and it tells me if there is a hotkey I don't know about. But again, that's a ask it something, it's got 1 chance, but because it's essentially just answering from a manually, it's been like 99% accurate for the basic learnings anyway, as I would have just been googling them anyway.
I think an llm wich would spot sth that could be a bug, setup an env to test the bug, run the program in order to trigger the bug, then check if it worked as wanted and then give this to the user, then it would be really interesting
I think the issue is people expect the LLM to just do everything, however, they’re most effective if you know the problem and understand your codebase and then just use it for writing functions where you know what goes in and what should come out. That speeds me up, anyway. It’s a basic take but I don’t ask an LLM to just solve everything for me.
I think Linus is more focused on the result than the method. If you can use an LLM and produce good code, that's great. Personally, I like to use an LLM. I use the one built into IntelliJ for $10 a month, I think it's GPT. I think it saves me time. I just ask it the stuff I would have to lookup or dig through documentation for. Does it produce bad code, or code with errors? yeah, all the time, but I never accept its code without going through it myself. It saves me the boilerplate, reading documentation, and gets me to the solution faster. I think the most productive way to use it is to ask it how to do what you want to do with the thing you're not familiar with, then look at the documentation and understand what it proposed. I wanted to move some TypeScript batch jobs to Lambda, so I needed a bundler. It took some trial and error, but knowing almost nothing about Rollup, I was able to get a working prototype in days instead of weeks. It took some trial and error to get it where I wanted it to be, but I think I got it done twice as fast as I would have otherwise.
2:17 I'm not quite sure which incident that you're referring to, but, if it's about the Bcachefs author, that didn't have anything to do with his code, but rather not following the rules of development and RC cycles where he was submitting 1000+ line patches during RC cycles. That got Torvalds pretty upset.
i like claud because the UI is better than gpt-4o also the older just gpt-4 seems like it has more domain knowledge and makes more thoughtful responses especially for financial stuff. Like in my back testing of a trading strategy i added commissions and it was off by like 99% and saw that and knew that something was wrong where as claud is just like "welp the 1$ commission is why you're down 99%"
The question is how good multimodal LLMs or VLMs will be in the future like 10 years from now...They are somewhat decent today but can get magically good as time goes by
That's what I've been saying!! Even if you had a "perfect" AI that was always right, If you just use cheat codes for everything you never learn. that's like half the fun in life is learning
In the early days of linux I downloaded slack, IIRC. But I got a kernel panic because I was supposed to change over from the boot floppy to the root floppy (something like that, maybe someone will remember how this worked better -- basically this was the first problem anyone trying it for the first time would likely have, and surely well documented). So, I e-mailed Linus. He explained via private e-mail that you had to change the floppy or whatever. Yeah, problem solved, I was "in!". Can't remember if I thought to thank him.
My experience with co-pilot and with cursor is that they are distracting and not particularly useful and getting your way when you already know exactly what code you want to write
The biggest thing I worry about with LLMs has nothing to do with some competent developers submitting code that was built within the help of AI. It revolves around these "services" which are being guided by people who don't care one way or the other, services that won't care about wasting a significant amount of time. I'm currently working on a browser with a team of people and after we went viral on hacker news, this AI company approached us and started submitting "code reviews" to our PRs without our consent. The stuff that it pointed out was just ridiculous, it was all stuff on par with the curl nonsense. We had to tell this company that we wanted nothing to do with their experiment because it was just wasting our time.
I use CodiumAI in PR reviewies for a year now. What are the results? Tthe review generated by it is a bunch of suggestions that is based on a very large good practice list. Its a bit like linter but a bit higher level. 80-90% of those suggestion are like "hey u did A make sure that it won't cause B" and usually its worthless because it won't cause B. So its more of a show of that yes it can read your code. 10% is kind of useful and yes u can write a better text for user or slightly imporve readability of your code. Its usually its not worth re-commiting. It never finds an issue that it would 100% be sure that code change is required. No, its only "maybe, perhaps, probably". It does not produce suggestion at order or magnitute higher than good practices. The scope of suggestion is usually single line.
the croudstrike bug was not because of a bad test. it was a release bug, i saw somebody said some random file in update had all 0 inside, so this is why even we can't trust the code that review our code. "being safe rather than sorry" stuffs i guss!
There is a Chinese term called: 盲人摸象 ( Blind men and an elephant). This is what I feel like it I ask LLM to generate the code I don't have experience to work on before
CGPT Helps me every day I don't have time to fix thing and deploy things as a 2 main team. I need all the help I can get to output work. But none of the things I work on is Mission Critical for Performance.. I just need it work and I need fast solutions.
The I in LLM stands for intelligence.
The I in Linus stands for Intelligence.
I mean if you use a new library and search a specific function it can safe a few minutes
I too can copy and paste
The G in Tom stands for genius.
And the S in LLM stands for security.
Being hard on code and neutral on LLMs is not a contradiction. If someone submits terrible code it doesnt matter if they used AI or not, if they submit good code it also doesnt matter. The point is that you judge code on the merit of the code, not the source. Its honestly really strange to try so hard to apply a moral or even value judgement to something which is just an inanimate tool that can be used or misused.
Yeah it just a tool, but knowing how much Linus hated C++ it's kinda surprising that he is so neutrals towards AI when there is so much garbage code produced by it.
doesn't*
It's*
Its cause if its shit you dont really improve anything and have no idea where to even look to make it not shitty.
Its fine if your end game is immediate satisfaction. But if your aiming to exceed the average. Then relying on a "tool" that averages isnt the way to go
You have to understand the context of the situation. This was a live interview. Linus couldn't afford to be as much a keyboard warrior as he naturally is. I'm completely convinced that would blow a fuse if someone submitted a patch to the kernel containing LLM code.
this
So basically "AI is fine because you all suck anyway"
Which is a Linus take as a Linus take can be :D
🤣🤣
TRUUUUU
Bro why do you have to hit so hard.
A slap in da face
Linus goes off on developers cutting corners and breaking rules. Not respecting how important the kernel is.
I think Linus is ambivalent or neutral about LLM coding because he doesn't direct his anger towards unconscious, inanimate agents. What he gets upset about is when a human, who should know better, tries to merge garbage code generated by an LLM without understanding what they are attempting to merge.
I think that you are correct but Id also like to point out that getting worked up over llms is fruitless at its core. Its an exercise for people who dont understand the world.
Yeah. There's a lot of money to be made in capturing attention by selling outrage stories of LLMs and just AI in general, when it's really just repackaging people problems into a new shiny exterior. That's not to say skepticism is unwarranted, but you're gonna have a more focused discussion once you isolate human decision making elements.
AI for coding is (currently) basically a replacement for Stack Overflow and Google
If you just plug in AI generated code into your system, you're gonna have problems, just like you would if you copy code from SO as-is
If you consult the AI, learn from it, and review what it produces and how it stacks up to your needs, then it becomes a net positive force that can both help your with trivial boring tasks, and also teach you things
I use it all the time for scripts and jumping off points or single functions. But it's just a start that speeds things up.
This is the way.
Yeah i am using it currently learn sfml game dev cpp.
Re: LLMs - you should read/review the recent paper on LLMs and learning outcomes for students. They basically found that although LLMs helped students improve *while* they had access to them the overall learning outcomes were poorer when access was taken away vs when access was never provided. Basically students who learned with LLMs became poorer at learning things in general or at least didn't improve at learning things compared to their peers who didn't use LLMs.
The google/stackoverflow/Wikipedia effect. 😂
I mean it's not surprising.
You are learning to use a tool, and when tool is taken away you are less productive. Simple as that.
What's the name of the paper? nvm got it
@@-weedle that's the weird bit. Just saw it reported on yesterday and now I can't find it. Actually have the video paused at ~7:30 because I'm off in another tab looking for the darned thing 🤣
Will update as soon as I find it.
Correct, now write a React to-do app, pen and paper only.
I am objectively a great programmer (as judged by my peers over the years during my carer), and I like Copilot very much. I don't think it made me better, quality wise, but it made me faster on the boring tasks.
This. So much this. All of this.
link ur github great programmer
me too
I'm objectively a genius 100x programmer and i think copilot stinks.
Just wait, most people think like that and after a few months they realize they aren't faster
The reason he's so chill about LLMs is because he trusts his review process. He nitpicks everything and takes it seriously. Therefore it doesn't matter if the code came from LLM or from a human, it would still need to go through him or his trusted review body.
He’s chill cuz he already made millions and about to retire.
Right. If you aggressively fight regression than every attempt at chance either improves things or changes nothing.
But if more people start submitting AI crap, its going to be a lot more work to find the parches that actually matter
@@atiedebee1020 Block them. If you submit bad code and don't correct I am very confident they just block them, or through them in spam, or whatever.
and maybe, just maybe if AI leads to more new contributors who feel confident to submit code for the first time because of AI, create a learning channel for them if they need additional guidance on how to review LLM output before submitting it. You know assuming ignorance but good faith.
@@ryanlee2091that would make sense for most people but he is not most people and still gets fired up about code pretty often
And in 3 years time, we have neural networks in our compilers warning you about your novel solution because it deviates from the average quality of all the code it was trained on.
Yeah. What we will ultimately get is not AI, but artificial stupidity... at the best, artificial superficiality.
Will need a new category of attribute to sprinkle in code and disable AI analysis around actual innovative code.
Hybrid Ai Linter: " I'm sorry Dave, I cannot let you commit that"
Why would data scientists make average quality code datasets? You have to assume that data scientists are complete imbeciles for them to purposely train LLM's to make dumb suggestions. If it's overly suggestive, then the dataset will be changed to make it stop suggesting so much. probably through RLHF.
@@TheSulross Copilot told me `# $NO-COPILOT$` would work, it did not... just to stop it from stomping on my code currently.
Dude! The thumbnail looked like you interviewed him! I was so exicted!
Baited (like me)
Same 😅
I got baited too
imma park here with yall
Skill issues
I'm not a great or experienced coder, but one issue I already see with LLMs is that it breaks an important aspect of coding, which is the dissection of idea implementation.
A huge benefit I get from coding is that it forces me to really think about what it is that I am trying to do.
Kernighan's Law suggests that debugging is twice as hard as writing code. Letting the LLM write the code and then debugging the result is a direction with subtle issues. It probably means that you can crank out your low-end work even faster than before, but you may not be able to improve the quality of your high-end work at all.
And Amdahl's Law would suggest that making your low-end work easier to do may not free much if any time up to put more hours into your hard jobs. The problem in that case isn't in having time to do the actual hard work, it's that your job involves grinding through boilerplate.
My general rule of thumb is to use LLMs to learn how to *approach* a problem, then go figure out the details myself. If I am ever asking it about specific numbers in a problem, I have strayed too far from its purpose (in my opinion).
Something like: "I want to make ____ kind of project, how might I start that?",
or even: "I am stuck on ____ step, what might be a few good things to try?" are both fine.
But nowadays, as soon as I search anything like: "will this loop go out of bounds of this array?", I start a new chat because I shifted its focus too far in the original. Once the numbers are wrong, I don't think I've ever seen them correct themselves.
In rare cases, I'll ask it to explain what some code will do, but that's only if the documentation is truly abysmal, which to be fair, sometimes it is. I just see it as a way to sift through all the more niche or hidden code discussions online.
It's a great rubber duck.
I agree, it's the best rubber duck short of an actual human subject matter expert, which often times you may not have access to depending on what kind of problem you are working on. Bouncing ideas off your significant other for instance, is probably not going to be useful if you are trying to write something like an inverse fast Fourier transform, but the LLM will have the context needed to plan an approach. Using it to actually write code is iffy, it can get you like 50%-70% there often but you may end up spending more time fixing the output than it would take you to just write it.
yeah this is how I've used it. I wanted to make a music app and all the code it gave me didn't work because the library had been updated but it suggested patterns I could choose to adopt or dismiss
@@gljames24 Basically, though it's a great rubber duck that LIES.
And if someone doesn't know enough they are literally incapable of spotting the lies.
And way too many treat these LLMS as sources of truth.
In part maybe because a subconscious misconception views everything the LLM generates as based 100% on exact-words a person has written in that sequence like it's a search engine and not an adlibs slot machine.
LLMs are great for Rubber ducking or small snippets. They can't replace a human programmer.
Yet
Am pretty sure could get AI to generate an entire web interface CRUD application - in my programming language and tech stack of choice.
@@TheSulross As it should be, CRUD already been done for almost three decades, will be weird if AI can't learn it with that amount of dataset. But last time I check it kinda shit at doing visual to code based task (like asking to generate HTML+CSS with certain visual specification), they will do the bare minimum and then unable to expand it to something that you want.
@@TheSulross there are already alot of projects for this, it's super optimistic to even refer to the output of such toolchains as "prototypes" considering the amount of unmaintainable garbage, nonsense code, and refactoring they need.
@@ch3nz3n It never will, the whole technology is a step in the wrong direction as far as real AI goes. All LLMs by their very nature can only rearrange and regurgitate that which already exists, it literally can't come up with something new because the underlying algorithms just take what it's been trained on (stuff that already exists) and tries to rearrange it in ways that best fits the prompt. That's a massive oversimplification but at it's core that's what it's doing
AI is going to be the corporate equivalent to buying a $3000 Gibson, Les Paul guitar thinking it’s gonna make them a better player without learning how to actually play.
Even though I believe in the potential of AI, I am against corporations having a stranglehold on access to it.
The future should be a place where we can all develop AI the same way we develop applications.
Corporations apply the classic tactic of turning people into helpless consumers so they keep paying for whatever services that are being peddled.
Independently assembled AI should be the direction to move towards.
Actually it is more like buying a $3000 collectors edition of a Harley Davidson bike. In 1:24 scale. With "real working engine", that is literally just a translucent engine block that gyrates the pistons if you turn the wheel.
This analogy is even better when you consider Gibson's quality control has been trash for over a decade
At least with guitar this expensive the sunk cost fallacy would force you at least try to improve to justify this investment.
Remember the Gibson auto-tuning guitar? Lol.
He is outcome focused, he doesn't care how people get there.
I get that exact feeling as well. Linus just seem to treat LLM's as just another tool, which it currently is.
I have been programming in C++ for 30+ years. I use LLM's in all their forms for coding. Using an LLM for coding successfully involves breaking off chunks of functionality that it can handle... and it usually involves defining function signatures for it. You'll only know what an LLM can handle by using it a lot. More complicated uses can only be tackled by providing it with extensive guidance in the form of pseudocode.
Also, I never "trust" an LLM. I have to maintain the code so I have MUST understand it. Yes, they do make mistakes... but given the size of functions I'm asking it to write those mistakes are usually easily spotted.
Which LLMs do you use for C++ ?
@@censoredeveryday3320 I pay for and use ChatGPT & Claude mostly for technical discussions and exploring ideas (though I sometimes use them for code generation as well). I use github copilot is vscode... and as of last night I use Cursor pro.
It's ok for self isolated functions, usually, but it falls apart when it needs to interface with multiple systems already designed. And I would NOT use it in memory unsafe languages.
I used gemini for a short while while writing my thesis work, but after it didn't help but instead i had to give them the answer, i stopped using it.
i dont have much experience with them but i still think it can be useful for simple tasks or to have a starting point for figuring out something, but as a replacement or trusting its output without confirmation, i dont think it's perfect.
I can't stand working with LLMs for coding. I spend more time correcting their mistakes than it would take me coding the things from scratch.
that just means you don't know how to use them or are bad at natural language. It's like being given freshman college interns and giving them tasks too hard for them
I think it's good for pointing vaguely towards something you can try to use
If tell the LLM to write something it's usually bad (even GPT4o struggles with regular expressions). But something like codeium in vscode auto completing a line I've started is almost always correct, saving on keystrokes.
Totally agreed. I tried a LLM stress test where I asked Mixtral 8x7b to make a console Hello World but with WinMain entrypoint in C++ without using any function from the standard library, only functions from the Windows API. In my requirements, the code had to work properly whether the UNICODE macro was designed or not and there had to be no #ifdef UNICODE in the LLM answer. Let's say that it was an abject failure. There are exemplars on how to do each of the specific tasks I asked for on GitHub and on Stackoverflow but they are few and far between. To code it, you're better off just with the MSDN doc 😂
Then dont write prompts that produce mistakes... Its not hard to guestimate the ability of an LLM and decide to only ask it questions within its range of ability.
There are 10 reasons LLM's spit out crap code and both of them are hard to fix
Watching this with subtitles on is trippy
As someone who is a novice with code, I concur with your opinion. It's vital to have a strong foundation of understanding, and an LLM should supplement this, not replace it. If you always take shortcuts you will never build up the knowledge and skills to do anything well, and I think this is true for everything, not just coding
I love LLMs to read documentation for me to quickly get started with new libraries.
Why not read the documentation itself? Every doc out there for a library or a framework has a "Getting started" section in its first pages
@@Karurosagu Because most of them are poorly written, and I'm not autistic enough to understand the author.
@@Karurosagulinking the entire documentation and then asking your specific query is faster as the "getting started" might not answer the thing you need.
@@HRRRRRDRRRRR Most of them? IDK, I think the quality depends on a lot of factors
And by the way, English is not my main language and I've read many docs without problems wether they are poorly written or not. Most problems i've had have been with: very new libraries and frameworks, very specific topics within existing docs that haven't been updated after a new release, misinterpreted features that turned out to be hotfixes and then got removed, and so on
TL;DR
Skill issues
@@Trahloc "Getting started" is not a specific feature
holy heck I don't think I've ever seen you go more than a minute without stopping the video
KEKW, youtuebers always does that
I am with Prime here, in the sense that making gpt write parts of the code is no problem is fast is great, but eventually u don't know what is going on in the codebase. Specially when you return to a project 1 month later.
nah, I use copilot all the time to create little scripts for me. They are often wrong at some places, so I have to either modify the code or tell copilot to correct it. But it still saves me a lot of time, as I just have to tell it which language, which input I have and which output I expect and it writes code that mostly works.. the rest is some discussions. Or the 10% bug fixing. It also helps a lot if you are not really familiar in that language. e.g. scripting for powershell, while you are more a Linux guy ;)
Still the code is more readible than most developer code I deal with most of the time, Maybe a little too much comments in the code :P but you can remove that as well
For me personally, as a low/mid level IT support, LLM's help me a lot, because i was always a quieter person and somewhat shy to ask questions to the seniors at work...
With the LLM's there is no such issues and i excel at the tasks quicker ^_^
@12:00 This is the crux of the problem. Using the 'always blow on the pie' quote. Always create and run the tests even on LLM created/guided work. This isn't just field specific.
LLM turns all bugs into subtle bugs. LLM turns compilation errors into syntax correct bugs with logical flaws that take ages to discover what when wrong.
Heavily depends on the language. With Rust, they hallucinate shit that gets instantly caught by the compiler.
@@LtdJorge But they are evolving to hallucinate as coherent as possible. The future model will just be better at tricking te compiler.
@@alexandrecolautoneto7374 users asks "make this" > LLM outputs > user copy and pastes > compiler fails > user gives LLM negative feedback > LLM model evolves to avoid negative feedback > stealth code.
Currently reminds me of so many web performance "services" that just insert javaScript to trick auditing tools to spit out higher points. Mission accomplished for everyone that doesn't understand the actual code.
This is a skill issue. In 10 years, the mark of a good programmer will be their ability to debug LLM code. Prime & others are coping because the introduction of LLMs caused a total paradigm shift in regards to writing good code quickly. NVIM and all this other DX shit they obsessed over is a brick compared Cursor AI + Claude. These guys jerked each other off over their WPMs, but are trapped in their old ways when something better comes out.
@@exec.producer2566 I loved the theory, but the reality is that LLM are just the not right model for coding. No matter how we improve it will always hallucinate, it's just how they work.
His view is probably not negative because the code that he reviews is from developers that know how to use AI. Meaning they don't just tell a LLM to "make a driver in rust" they just use it for tedious repetitive code tasks.
LLM's are great replacement for searching for obscure methods in API documents. When string matching doesn't cut it, I always resort to LLMs. And they find stuff that I couldn't find in a few minutes with 70%~ accuracy
My problem with LLMs are currently:
- For my hobby projects they are a useful help to bootstrap and do boilerplate. For example: Create me an ORM for this database schema.
- At work: Large codebase in Typescript, Dart, Go and Rust - Copilot expecially is useless.With recent gemini and Claude Sonnet i had slightly better experiences but it's still awful.
My longterm concern is: Is the production -> training data loop without much feedback mechanism in between. It has already been shown that the quality of LLMs declined the more you feed them LLM produced input data. So i currently won't rely on them much. At least not for code where it matters.
Linus response sounds natural to me. He's basically saying: "Hey, if this code gets to match kernel standards, I don't care where it's coming from.".
Linus talked about LLMs in future tense. He never said anything about using them for programming here and now. I think he's just optimistic about their potential in the future.
5:45 I interpret Linus's opinion here as "LLM can be a great code linter but you should assume its output as opinion about the code and then decide by yourself if you want to actually change the code".
Though this obviously assumes that the developer skill issues are more about the accuracy of the implementation instead of overall algorithm or mis-understanding data structures or thread locking.
I have been using LLMS since quite some time. I am not a full time developer but have to write some python code from time to time. It really helps me when I start a script. You just tell it - I need a class named XYZ with methods a,b,c which have the following parameter and which return that. The script has the following command line parameters … etc.
And it will perfectly do that. Then you have to kick in and write what you need your script to do. From time to time you ask how I can get this and that done. At the end you can let it help you to optimize your code in a short time if you for instance have a little bit too much if then else stuff in your code. But at the end you have to understand each line and judge what is a good recommendation and what not.
Best use I have for LLMs is as a user integrating very basic features in websites through grease monkey.
Doesn't take long to change a website that requires hover on an icon to show a picture to one that shows them by default.
It's not hard, it's not code that will see reuse. It's just fiddly normally.
And with chatgpt you can actually just roughly ask it with the right info and receive a good enough result.
It’s interesting what you said about software development and understanding the problem domain, and bugs due to edge cases or things you’d not quite fully understood.
Because that’s sounds like you’re coming up with a solution to some kind of problem within a set of constraints (or trying to understand what the problem actually is about and what the constraints are.) It’s like a level higher than a particular programming language. It’s more like designing and being able to understand certain types of problems.
So say if you were using a language that just didn’t allow certain classes of bug (like memory errors) so it was high level and the LLM didn’t need to generate that kind of code, and it became more about expressing solutions to known problems (I want to say applying patterns but I don’t mean design patterns - something probably more high level) and if an LLM was working at this level then I think they’d be really useful.
“I can see you’re trying to do this kind of software, have you thought about applying technique / approach / algorthim X.”
If you could somehow turn that knowledge about problems and their solutions into some abstract model that LLM could use to spot patterns and suggest techniques, to help you understand the problem space, then that’d be good.
I genuinely don’t know if LLMs work like that at the moment. 🤷♂️
7:10 so if we get LLMs to review and respond to these requests, we'll have LLMs arguing with each other.
Saying you're 10x better with AI is like writing terrible code just to cite a massive performance improvement
or the person just makes spelling mistakes and the bot just corrects the spelling
A lot of coding is repetitive work, gpt is really good at repetitive stuff, that's where it helps. If you manage to build your code good enough where it is composable and reusable, the LLM will see the pattern and suggest you ways to compose it correctly
@@specy_ But that isn't making my code 10x better, it's just getting it done faster. If all it's doing is recognizing the pattern of what I'm coding and completing it, then the AI isn't making me do my job better, it's just letting me do the same job but slightly faster.
@@specy_ "Resuable code" is in general a fallacy that will bite you in the tuchus sooner or later.
@@kaijuultimax9407 yeah ofc it won't help u make better code, but it helps u make faster code. If u can save 50% of your time when writing one feature, you can use that time to make it better yourself.
I was fully expecting LLMs get reckt
I've found LLMs mixed. The line-based auto-complete is 80-90% useful, especially writing similar repeated code. The other times it has got in the way, but on balance I generally prefer to have this functionality.
Using LLMs to ask questions, I found it helpful when trying to identify a Bootstrap class -- my Google searching didn't find the class, but asking an LLM helped me find the class name that I then looked up in the docs. Some other approaches I've asked for I've ended up adapting the code to the way I wanted to write it, using the LLM code as a basis. In some other instances it didn't help me solve the problem so I used different approaches.
Linus had a valium this morning
LLM-assisted development development is worse than StackOverflow-assisted development. I am not worried. There will be more work to fix the LLM fallout for the professionals who know what they are doing.
growth pains, it will be the opposite in short term.
Nice copium
@@Boschx How is it copium, its already led to worse code that needs revision, I doubt that will magically get better...
As a freelancer with 30+ years of experience, mostly working on projects for a few consulting companies, I’ve found LLMs incredibly useful. Switching between languages like Python, React, C++, and C# means I often forget specific details, especially with new updates.
Recently, I was assigned to a next.js project. Yet another js-framework, 😞 , and LLMs have been a lifesaver for quickly getting up to speed with syntax. They also help generate solid code with good comments on common code stuff, including unit tests, which keeps my workflow pretty efficient if I can say so.
8:30 I'm weirdly annoyed that Prime didn't react at the "humble" joke :(
AI is great for remembering syntax with context. You don't ask it to build a house which its terrible at; you ask it to build a wall 4 times, a floor, a roof... Etc which it's actually pretty good at
"openAI.... f*ck you!"
- Linus probably
There is an identical comment below you, I think you are a bot.
There is an identical comment below you, I think you are a bot.
There is an identical comment below you, I think you are a bot.
There is an identical comment below you, I think you are a bot.
There is an identical comment below you, I think you are a bot.
I really like whatever Microsoft has baked in to the new visual studio where when you're refactoring a project and you make a similar change 2 or 3 times, vs will give you the red "press tab to update" the next time you move to a similar line.
It sure beats trying to come up with a regular expression to search and replace. Simetimes.
I think that last bit is key. If your in it to learn, turn off autocomplete at least.
If your just trying to ship your GPT wrapper, then go for it.
16:32 not negative, but he has very high standards. He's like Gordon Ramsay, who can be an asshole in the kitchen, but is a sweetheart when you see him outside the kitchen or interacting with children
I think LLMs for coding are great, but turn off the 'auto-suggest' feature. That stuff messes up with your thinking process because you have an idea of what the function is supposed to do, but at that exact moment a whole bunch of code appears and you've got to read through an lose your train of thought, and maybe sometimes it's what you want, and sometimes it's not.
Instead I think having that bound to a key is super nice, so you write your function, and code as you normally would. Then you stop, and realise it's OBVIOUS what your about to do next, you know it, the LLM knows it. so save everyone time by hitting that 'manual' button and have the code pop up when YOU ask it to.
That I think is just 'better auto-complete', and it's amazing. It's the perfect combination of automation and still having enough control to get to where you need to go without ending up with a bunch of code you don't understand and can't debug.
I use llms to help me understand the problem I'm trying to solve. You can bounce ideas off of it to help you learn.
15:50 maybe not dumber, but out of practice. I'm a full stack independent developer that employed a frontend dev for a few years. I find that using LLMs is the same if you don't "take in" all the code that is produced
it might also be because even though you are using an LLM for help, the PR you produce will always depend on the person doing it. Whether they rely fully on it, or not.
Handwritten code or AI tools code are the same, if you have the patience to test/debug the code
Don't blame AI tools for your own limitation, for not knowing how to properly test a code or just being plain lazy, if you don't know how the code works it is your responsibilty to learn it, otherwise it is like a dagger above your head once it is deployed in production
From an automation standpoint, I'm hoping/expecting AI to find a useful home in property-based testing. Helping to more quickly zero in on corner cases that the implementation missed.
Off topic, before watching the video, just to say ThePrimeagen is a beautiful soul
When an LLM creates something is a guaranteed mess but other than that its op.
Especially for learning (real) languages.
LLMs have come a long way for coding even in the last few months. Sometimes it's about identifying the use cases its good at. One of the best use cases I use it for is dumping in an entire project, often times with lots of spaghetti code into context (Which now get as big as 2 million tokens and can be cached on repeated calls to save money) and asking it to locate the parts of the code that do X. It will surface the code, I can do a quick find and I'm on my way. It probably grabs what I need 80-90% of the time on the first shot with modern models.
It seems kind of like common sense but it's a good idea to use LLMs for the aspects of coding they are good at and probably a bad idea to use them for coding tasks they underperform on. Unfortunately, things emerge and change so fast what an LLM is good or bad at coding wise is shifting quite a bit and not obvious out of the box.
Linus is a simple creature who understands complex problems.
The argument that I feel you miss a little is that there are different ways you can use it. You are absolutely correct that knowing the code and being a good coder is a must but the bits you can bounce off or syntax you can gather quickly is invaluable. Often times, I am going back and forth between docs and llms and I will know when the LLM is bullshitting and I need to go to docs. In a way your right but also there are other ways you can learn to use it that is more helpful. if any of that makes any sense.
LLM's writing code is my dream come true , i remember writing code on basic as a child on a commodore 64
now i can just ask the LLMfor a program at it just produce awsome results and evolving constanly . (i usaly use chat gpt , i remember the chat gpt's first steps in writing code from the codex playground sand box , it was so awsome for me)
You should do the recent François Chollet ARC Prize talk at some point.
Getting takes on LLMs from engineers like Linus is more of a personality test than anything else at this point. You should listen to what actual AGI researchers think about LLMs.
I've pasted in code that doesn't work into an LLM that was non-trivial and it spotted a bug for me, where I got the memory ordering wrong on some atomic operations.
I feel like this is the sort of thing where the right answer exists out there a multitude of times and the LLM can pull together all these resources and explain why your code is broken. It's super useful for that stuff, and is way better than just trying to only absorb this from the docs. Also it's a great rubber duck.
I have been arguing since GPT3 that AI will be amazing static analysis tools one day, awesome to see linus agree. It makes perfect sense as a good 30% of the bugs we catch in code reviews at work probably could have been caught by an AI (although maybe not a large language model with the current design)
It seems like he sees it like a logical consequence of C -> compiler magic -> assembly. Now it is AI -> some black box magic -> C -> compiler magic -> assembly
100% it still takes SMEs to be effective. The LLM still depends on the inputs being perfect to be right.
Copilot depends upon inputs that are not always right, or not always relevant, or not always applicable. Code that is older version, etc.
Still takes insights and SMEs to be useful
For me it's pretty simply. I meet developer, software engineer, sysadmin, network admin, cloud admin, qa tester, data analyst, data engineer, devops engineer or cybersecurity professional who lost his/her job because a random person who is not IT/CS professional can work their job using LLMs, and I'll be like "God exists, and it's AI."
If random person can do this using llm - special person can do it too.
@dmitriyrasskazov8858 A random person would be perfectly happy with a significant lower salary then special one.
I don't quite understand. Are you saying you already seen this happen, or is it more of a "if/when it happens in the future..."?
@@ottowesterlundThe later option.
I think it'll be either domain professionals using LLMs to do their job, or it will just be entirely automated.
If you have a random person who knows nothing about the domain using an LLM to do the job, then you can likely just automate the job at that point.
There is a line where LLMs are helpful on one side and problematic on the other side and today they exist on both sides. You can use an LLM as a tool to help you work, some might like copilot others may like using LLMs to scope out a problem, and in other cases they may not be useful at all. I don't think that they will be replacing devs anytime soon but they will be alleviating us from some tasks.
Using a LLM is like being part of a team an having to read the code of some other developer, you can read it for structure or you can read it for solving the problem it is trying to address.
The LLM's are very useful to people who know enough to check their work. They're a productivity multiplier. Additionally, they do serve as a decent second set of eyes. They do catch bugs.
The current paradigm just fundamentally doesn't work for identifying bugs. The code available to train on likely got published after the big bugs are fixed.
The small remaining ones are not identified in the training material as bugs.
The difference is this: He cares about the results that land in his inbox. What tools people use is kind of besides the point. He will rake you over the coals if the patch you submit sucks regardless of how you got there.
how do you not hallucinate when you're playing statistical probability?
They are not. They produce text like we do, due to emergent abilities. And we hallucinate all the time too. Wr just catch ourselfs often, many people do not
LLMs are useful, but they are not the end all be all. I find them most useful to get acquainted with a library quickly.
Whats bad is LLMs help me remember things ive forgotten. So the really rare things i use once every few months. And it speeds that up. WHICH MEANS I never remember those little things. An example is building the initial code for a program. run through all the logic a 1000 times and loops and so on. but when you want to do something like open a file, or grab a different library, if you only do that once every few weeks it's harder to memorize.
Your brain is a very efficient cache. If it doesn't retain something, it's because you don't need to retain it. Something you do once in a blue moon is not something worth remembering off the top of your head, since the extra time taken to re-familiarize with the concept isn't that big compared to everything else.
@@ruukinen Except I've found that it also means that you get less ideas that involve the things you rarely do. And some of my best ideas combine my deep understanding of the current system with smaller rarer things I encountered in the past. But when I use LLMs, my brain got rusty at doing that. So I backed off, and it feels a lot like rebuilding cardiovascular endurance after not exercising for a few months. Maybe that's just me...
@@Muskar2 yeah like maths without calc and maps without google, makes you feel like a lost 5 yr old separated from parents in a crowd
Ideally it should be used for internal pre-checks for a PR, using for automated bug bounties is obviously bad. But if I can run against my PR to check if I missed anything before submission, sounds good to me.
Where I think I've gotten to is, when I get it by an LLM, and if it doesn't get it in 1 attempt, I move on to figure it out myself. As the amount of time I would previously spend on some error that was because of a single character being wrong, and wasn't picked up by the likes of a linter/lsp was frustrating. But beyond that use case I feel like I've felt it frustrating longer term, because of its lack of quality mostly. That and learning Neovim, I think of something that I haven't encountered before that I wanna do, and it tells me if there is a hotkey I don't know about. But again, that's a ask it something, it's got 1 chance, but because it's essentially just answering from a manually, it's been like 99% accurate for the basic learnings anyway, as I would have just been googling them anyway.
I think an llm wich would spot sth that could be a bug, setup an env to test the bug, run the program in order to trigger the bug, then check if it worked as wanted and then give this to the user, then it would be really interesting
Oh an LLM augmented compiler/linter could be quite cool actually
I think the issue is people expect the LLM to just do everything, however, they’re most effective if you know the problem and understand your codebase and then just use it for writing functions where you know what goes in and what should come out. That speeds me up, anyway. It’s a basic take but I don’t ask an LLM to just solve everything for me.
7:19 "he" 🥶
LLMs have been trained on all of his rants, and the entire existing linux kernel. Hard to argue with that.
I think Linus is more focused on the result than the method. If you can use an LLM and produce good code, that's great.
Personally, I like to use an LLM. I use the one built into IntelliJ for $10 a month, I think it's GPT. I think it saves me time. I just ask it the stuff I would have to lookup or dig through documentation for. Does it produce bad code, or code with errors? yeah, all the time, but I never accept its code without going through it myself. It saves me the boilerplate, reading documentation, and gets me to the solution faster.
I think the most productive way to use it is to ask it how to do what you want to do with the thing you're not familiar with, then look at the documentation and understand what it proposed. I wanted to move some TypeScript batch jobs to Lambda, so I needed a bundler. It took some trial and error, but knowing almost nothing about Rollup, I was able to get a working prototype in days instead of weeks. It took some trial and error to get it where I wanted it to be, but I think I got it done twice as fast as I would have otherwise.
2:17 I'm not quite sure which incident that you're referring to, but, if it's about the Bcachefs author, that didn't have anything to do with his code, but rather not following the rules of development and RC cycles where he was submitting 1000+ line patches during RC cycles. That got Torvalds pretty upset.
no we are not moving to rust
i like claud because the UI is better than gpt-4o also the older just gpt-4 seems like it has more domain knowledge and makes more thoughtful responses especially for financial stuff. Like in my back testing of a trading strategy i added commissions and it was off by like 99% and saw that and knew that something was wrong where as claud is just like "welp the 1$ commission is why you're down 99%"
you do need to be open to the possibility of things in order to truly understand what they are.
The question is how good multimodal LLMs or VLMs will be in the future like 10 years from now...They are somewhat decent today but can get magically good as time goes by
Somewhat ironically, the latest models do actually pick up that strlen check before a strcpy far better
That's what I've been saying!! Even if you had a "perfect" AI that was always right, If you just use cheat codes for everything you never learn. that's like half the fun in life is learning
In the early days of linux I downloaded slack, IIRC. But I got a kernel panic because I was supposed to change over from the boot floppy to the root floppy (something like that, maybe someone will remember how this worked better -- basically this was the first problem anyone trying it for the first time would likely have, and surely well documented). So, I e-mailed Linus. He explained via private e-mail that you had to change the floppy or whatever. Yeah, problem solved, I was "in!". Can't remember if I thought to thank him.
My experience with co-pilot and with cursor is that they are distracting and not particularly useful and getting your way when you already know exactly what code you want to write
I really like your attitude as a dev.
The biggest thing I worry about with LLMs has nothing to do with some competent developers submitting code that was built within the help of AI. It revolves around these "services" which are being guided by people who don't care one way or the other, services that won't care about wasting a significant amount of time. I'm currently working on a browser with a team of people and after we went viral on hacker news, this AI company approached us and started submitting "code reviews" to our PRs without our consent. The stuff that it pointed out was just ridiculous, it was all stuff on par with the curl nonsense. We had to tell this company that we wanted nothing to do with their experiment because it was just wasting our time.
I use CodiumAI in PR reviewies for a year now. What are the results? Tthe review generated by it is a bunch of suggestions that is based on a very large good practice list. Its a bit like linter but a bit higher level.
80-90% of those suggestion are like "hey u did A make sure that it won't cause B" and usually its worthless because it won't cause B. So its more of a show of that yes it can read your code.
10% is kind of useful and yes u can write a better text for user or slightly imporve readability of your code. Its usually its not worth re-commiting.
It never finds an issue that it would 100% be sure that code change is required. No, its only "maybe, perhaps, probably".
It does not produce suggestion at order or magnitute higher than good practices.
The scope of suggestion is usually single line.
the croudstrike bug was not because of a bad test. it was a release bug, i saw somebody said some random file in update had all 0 inside,
so this is why even we can't trust the code that review our code.
"being safe rather than sorry" stuffs i guss!
Maybe I am missing the point, but strlen will cause buffer overrun if there is no terminating null char..
There is a Chinese term called: 盲人摸象 ( Blind men and an elephant). This is what I feel like it I ask LLM to generate the code I don't have experience to work on before
CGPT Helps me every day I don't have time to fix thing and deploy things as a 2 main team. I need all the help I can get to output work. But none of the things I work on is Mission Critical for Performance.. I just need it work and I need fast solutions.