btw, someone could have also pasted in a projectId=example.com# and the # would have ignored everything past the domain they provided. They could have also doing projectId=example.com/ and it would have treated the convex .site as a sub path on their own service. Basically, sanitize your inputs. dumb mistake. shout out to the guy who emailed me the info 🙌
huge props for being transparent about this. this is a super easy (and unfortunately common) type of bug, and will only be more common as AI is leaned on more and more heavily
Classic client side validation error. Error here: assuming the client is always sending valid input. Happens to the best of us. Especially when not paying attention thinking it's not a "serious" service. Source: Nasty cascading privilege escalation exploit from said client side validation. The type of mistake you only make once.
This is one of the reasons I prefer using a URL object builder for injecting query params into an endpoint. I also don't have to worry about encoding the query param values.
I think the proper way to write code with AI is pretty similar to how we write code normally. Copy changes incrementally, test the code changes, then make changes as necessary. I have to admit tho it's pretty tempting to just make AI do all of the unfun stuff where you're in a hurry, fell into that trap as well
Amazing video as always. This is why i think only mid to senior engineers should use these ai tools. When interns/juniors blindly use these they will never learn
Normally TDD is super annoying -- but its actually great for putting the robots on rails, and I feel like Cursor would do well to have their UX try and nudge you in this direction, especially when you're generating/applying lots of or large edits. Forces you to read it a few times to catch little stuff like this before it turns green & helps claude not randomly change a variable name without you noticing (hes a lil heckin prankster sometimes). Would recommend, esp if you're newer to a lang/framework or just leaning AI-heavy for what you have on your todos that day. Oh, also, todos are great too tbh, lol. Ideal world, source code is the last thing left to generate before you move on.
Very good. Another idea is to add some static code analysis tool into a git hook or at the pipeline of the application, so it can check for vulnerabilities before code is shipped.
You could encrypt the token when passing it between the proxy and the server, re-encrypting it for each request. This ensures that even if intercepted, the token remains secure, as unauthorized actors won’t have access to the decryption key used by the server.
Hi Cody i dont know what it is but i love your videos talking about AWS and cloud, if its possible for you to make more of them, you explain concepts very well!
the funny thing is i can't even use chat gpt at work because it's not even possible to blindly generate rust code. the majority of it doesn't work. you can brute force it but that takes more time than if you had just solved it on your own.
What is the purpose of a proxy ? is it just another service that handles authentication? between the client and your main services? I want to understand what benefit it provides, versus just having the client call each service
I will say this seems more like a code review and checking issue and doesn’t really have to do with AI too much. If you had learned this GO from googling, would you have caught that it had a security issue? I’m not sure that it coming from an LLM is the major issue. All really good points and steps to follow though!
I must be missing something... How does your proxy send requests to a service that someone else is running, just by giving a query string? Isn't it only sending stuff internally within the server? Or is this because of some PaaS that you're using?
Considering I made a custom byte code virtual machine and domain specific assembly language for authorization, I would probably have complexified a proxy server system quite a lot. The simplest scheme I would endorse would be to use an HMAC or Ed25519 signature instead of a secret key in the Authorization header.
@@k98killer I understand very well how asymmetric cryptography works, but signing entire requests is the wrong tool for the job and introduces more things that can go wrong. You are overcomplicating everything, when all you need is your own CA and you don't need to fuck with anything in your app code. Are you big on web 3 or something?
@@sarabwt "Web 3.0" is a marketing scheme to separate naive investors from their money. TLS literally uses asymmetric encryption to encapsulate entire requests and responses. An additional layer of asymmetric cryptography encapsulation adds security at the cost of complexity and CPU cycles, but it is not universally "the wrong tool for the job". In this instance, it would have mitigated the vulnerability. Edit: I don't get why you want to die on this hill. It's okay to just admit you wouldn't feel comfortable doing this yourself.
To be honest, in order to gather every security issues + thinking of it what can go wrong, its much better if writte… If i know common exploits, better than ai. And its not a hard thing… than you can do better. I understand its fast to generate. But only in the short term. After you realise the mistakes it will take much more time. I like ai, i use for simple staff, validate idea, translation. But currently the code which they can output is low… and you have to be specific and you have to be a good developer.
Looks like SQL injection. Probably they don't even needed to use convex for this. They could insert their domain and add "/" in the end and the convex domain would be ignored.
Same thing applies as before. This is equivalent to "do not blindly copy code from Stack Overflow without understanding it". We all do ti anyway, let's be real.
@@WebDevCody If you encrypted the request and response rather than simply shipping the secret across the wire as a header, there would be no way for anyone who intercepts the traffic to figure out what the secret key is. Use a random 16 byte IV/nonce for each encryption and check an HMAC before decrypting, and it will be unbreakable. If the service receives a bad HMAC, respond with 400 and plaintext error. With a 32 byte key, it'll take an attacker until the heat death of the universe to guess correctly.
nah... cursor DESIGNED theyr AI in that way to do so, which also automatically sends notification to them and then they reach out like heroes. yes I'm mad... but still....
In a world where web development is over saturated, it's good that it's open the doors for unethical hackers to exploit their mistakes since they don't have a clue how to prevent this.
NOOO, Please don't ask AI to review the code generated by AI. If you are saving your time writing the code, take your time to review the code, and do it carefully. Don't take all the AI shortcuts, the code will be crap!
Hate to say it, but student-teacher multi-agent is actually a great way to generate higher quality code if that's what you're aiming to do. That said, please still review your own code after the 2nd model pass, for now they're just little drunk interns that need lots of babysitting.
Don't worry, the next generation of LLM will make this bugs even harder to find. LLM's will generate the most clean and beautiful crap code that you will ever seen.
@@WebDevCody but they will not suffer from the same hallucination problem? Do you really wanna have a security assessment based on probability? As you said in your video: the first time AI didn't find any issues, the second time found. Idk, probability-based-coding still sounds inefficient to me.
@@alexandrecolautoneto7374 you act like human's are perfect. Human developer suck at their jobs just as much, and most teams do NOT have a designated security expert reviewing every line of code written. All if takes if a human to scroll a little too fast on accident and he could miss a critical security bug.
@@WebDevCody yeah, maybe I'm too much on the defensive. It's just from experience, every bug that AI introduces to my code takes me more time than everything else and I turned from a passionate enthusiastic to a hater. I think AI is far from what they are selling and most of people are being scammed. Sorry for being too intense, great work btw, keep up with the videos.
This came as a surprise. To noone. LLMs don't know what the code they print out does. They don't think logically, they don't think about security. They don't think. I think LLMs have a place in coding, but you need to know code if you're going to use it. Great for hashing out ideas and looking up documentation on things, not great to rely on for coding everything for you. That was glaringly obvious from the first time I tried to use an LLM to code me something with a bit of a complexity. It just couldn't do it. Even when I told it exactly what to do, why specific parts would not work, it kept falling back on faulty flawed code. Good as a tool, but you can't let it take control over your code. It's simply not capable of it.
they are called assistants for a reason, and I agree, it's a great tool to learn specific stuff once at a time, and it may still give out flat wrong information in some areas, great for not having to write agonizing code, or as shortcuts to writing code, I find it's actually quite accurate as long as you know what to ask it for, and if it's flat out wrong then it's easier to write your own iteration then feed it to the chat, and ask it logical questions - then it'll do OKAY enough to finish the implementation for you.
also the worst type of code it generates is always the dynamically typed languages/scripting languages, in 1 iteration of a simple router method it will pass arguments in a certain way, in the next one it'll use a totally different convention or syntax, or even response/error checking, totally chaotic.
btw, someone could have also pasted in a projectId=example.com# and the # would have ignored everything past the domain they provided. They could have also doing projectId=example.com/ and it would have treated the convex .site as a sub path on their own service. Basically, sanitize your inputs. dumb mistake. shout out to the guy who emailed me the info 🙌
I am really starting to appreciate this Cody guy. He is not afraid of being honest and going back on the hyped stuff he tries. I love that.
I appreciate you
Wow, AI is a massive security risk; who'd have thought…
What does Cursor do to ensure secrets aren't leaked to the AI?
@@seancooper5007I think they're not using user data to retrain the ai, I think that should be enough
people make similarly dumb mistakes though
@@yassinesafraoui once data hits third party server anything can happen to it. There's a good chance it'll end up in readable logs.
@@kvinkn588 this was human error, the human happened to be aided by AI that generated more code than they could digest and scrutinise thoroughly.
The CEO of Nvidia told me I should though.
what a good time to do bug bounty hunting
AI is generating all new industries
good thing you found out about it quickly
huge props for being transparent about this. this is a super easy (and unfortunately common) type of bug, and will only be more common as AI is leaned on more and more heavily
Imagine hackers intentionally create huge amount of obscure codes with security hole with some seo, and AI model trained with those codes.
Classic client side validation error. Error here: assuming the client is always sending valid input.
Happens to the best of us. Especially when not paying attention thinking it's not a "serious" service.
Source: Nasty cascading privilege escalation exploit from said client side validation. The type of mistake you only make once.
These kind of videos are so valuable. Please make more of them!
This is one of the reasons I prefer using a URL object builder for injecting query params into an endpoint. I also don't have to worry about encoding the query param values.
Need a code security reviewer AI to double check our coder AI
Sounds like a good idea Steven, get making one! :D
Finally someone is talking about it. All those people going bonkers on how they created "SaaS" with AI, never bought that crap.
I mean, I don't doubt they haven't, but it is probably filled with bugs they don't even know
Thank you for sharing this bit, I feel like a lot of content creators don't show mistakes they've made, so I appreciate it!
I think the proper way to write code with AI is pretty similar to how we write code normally.
Copy changes incrementally, test the code changes, then make changes as necessary.
I have to admit tho it's pretty tempting to just make AI do all of the unfun stuff where you're in a hurry, fell into that trap as well
Vulnerability or not, props for showing it and talking about it in a TH-cam video.
Amazing video as always. This is why i think only mid to senior engineers should use these ai tools. When interns/juniors blindly use these they will never learn
3:35 Wait, it's a side project yet you're just trying to ship stuff? How does that work
Normally TDD is super annoying -- but its actually great for putting the robots on rails, and I feel like Cursor would do well to have their UX try and nudge you in this direction, especially when you're generating/applying lots of or large edits. Forces you to read it a few times to catch little stuff like this before it turns green & helps claude not randomly change a variable name without you noticing (hes a lil heckin prankster sometimes). Would recommend, esp if you're newer to a lang/framework or just leaning AI-heavy for what you have on your todos that day. Oh, also, todos are great too tbh, lol. Ideal world, source code is the last thing left to generate before you move on.
Very good. Another idea is to add some static code analysis tool into a git hook or at the pipeline of the application, so it can check for vulnerabilities before code is shipped.
AI is going to keep actual engineers in work for years to come, just cleaning up the mess.
Great video buddy
Do you think it can train itself over our .env variable values? I have been thinking about this now-a-days
You could encrypt the token when passing it between the proxy and the server, re-encrypting it for each request. This ensures that even if intercepted, the token remains secure, as unauthorized actors won’t have access to the decryption key used by the server.
love the content and the passion!!!!
Hi Cody i dont know what it is but i love your videos talking about AWS and cloud, if its possible for you to make more of them, you explain concepts very well!
Yeah may e
the funny thing is i can't even use chat gpt at work because it's not even possible to blindly generate rust code. the majority of it doesn't work. you can brute force it but that takes more time than if you had just solved it on your own.
I recommended using snyk for security to find vulnerabilities in your code
hi cody, what is the name of your keyboard? I really like the sound of it.
Have you done any modifications to it?
thanks
Im pretty sure it’s a utility that creates fake keyboard sounds; it’s not real mechanical keyboard sounds.
klack app
What is the purpose of a proxy ? is it just another service that handles authentication? between the client and your main services? I want to understand what benefit it provides, versus just having the client call each service
I will say this seems more like a code review and checking issue and doesn’t really have to do with AI too much. If you had learned this GO from googling, would you have caught that it had a security issue? I’m not sure that it coming from an LLM is the major issue. All really good points and steps to follow though!
Cody, may I ask you: what AI are you using to generate images and speaker in your scarystorygenerator?
replicate for the image and openai for the text to speech
You probably want to use the "net/url" package to construct the URL safely instead of concatenating strings.
Thanks I’ll check that out
This is why i don't use ai in my editor. I only use the chatgpt prompt. And even then I only use it for planning projects and stuff.
I find the same shiz happens in my code when I lean on AI too much for a project without checking every line.. Testing sucks but a must 😮💨
I've just become addicted to your videos! Great one as usual. 🔥
Curious to see the prompt that generated the code.
I must be missing something... How does your proxy send requests to a service that someone else is running, just by giving a query string? Isn't it only sending stuff internally within the server? Or is this because of some PaaS that you're using?
8:00 | the 1gb txt of domain names
That's why I thought it would be better to have some sqlite db containing projects he has
Considering I made a custom byte code virtual machine and domain specific assembly language for authorization, I would probably have complexified a proxy server system quite a lot. The simplest scheme I would endorse would be to use an HMAC or Ed25519 signature instead of a secret key in the Authorization header.
32 character secret yolo
it doesn't matter, the attacker would still be able to steal the key
@@sarabwt No, because the key would not be transmitted across the wire. I feel like you don't understand the concept of asymmetric cryptography.
@@k98killer I understand very well how asymmetric cryptography works, but signing entire requests is the wrong tool for the job and introduces more things that can go wrong. You are overcomplicating everything, when all you need is your own CA and you don't need to fuck with anything in your app code. Are you big on web 3 or something?
@@sarabwt "Web 3.0" is a marketing scheme to separate naive investors from their money. TLS literally uses asymmetric encryption to encapsulate entire requests and responses. An additional layer of asymmetric cryptography encapsulation adds security at the cost of complexity and CPU cycles, but it is not universally "the wrong tool for the job". In this instance, it would have mitigated the vulnerability.
Edit: I don't get why you want to die on this hill. It's okay to just admit you wouldn't feel comfortable doing this yourself.
To be honest, in order to gather every security issues + thinking of it what can go wrong, its much better if writte…
If i know common exploits, better than ai. And its not a hard thing… than you can do better.
I understand its fast to generate. But only in the short term. After you realise the mistakes it will take much more time.
I like ai, i use for simple staff, validate idea, translation. But currently the code which they can output is low… and you have to be specific and you have to be a good developer.
Looks like SQL injection. Probably they don't even needed to use convex for this. They could insert their domain and add "/" in the end and the convex domain would be ignored.
Reminded me of CORS and allowed origins
Same thing applies as before. This is equivalent to "do not blindly copy code from Stack Overflow without understanding it". We all do ti anyway, let's be real.
We don't all do it though :(
What about secret key encryption ?
not sure how that would help? they'd still have the encrypted key they could send over after they capture it.
@@WebDevCody Indeed. My mistake :)
@@WebDevCody If you encrypted the request and response rather than simply shipping the secret across the wire as a header, there would be no way for anyone who intercepts the traffic to figure out what the secret key is. Use a random 16 byte IV/nonce for each encryption and check an HMAC before decrypting, and it will be unbreakable. If the service receives a bad HMAC, respond with 400 and plaintext error. With a 32 byte key, it'll take an attacker until the heat death of the universe to guess correctly.
@@k98killer No need to reinvent TLS.
@@k98killer it’s over https, how would they get the key?
I just found out watching this video i shipped to prod something with a simmilar vulnerability dang at my job.
It happens, I doubt anyone exploited it
nah... cursor DESIGNED theyr AI in that way to do so, which also automatically sends notification to them and then they reach out like heroes. yes I'm mad... but still....
Classic injection problem
am I missing something or can you not have allowed origins fix this globally… edit nevermind I just saw the part for alloweddomains
Bro just leaked his IP too.. Comcast..
Please don’t ddos me, I want to be able to watch Netflix.
@@WebDevCody didn’t plan on it :)
Don’t use get requests for authenticated services. Duh.
In a world where web development is over saturated, it's good that it's open the doors for unethical hackers to exploit their mistakes since they don't have a clue how to prevent this.
NOOO, Please don't ask AI to review the code generated by AI. If you are saving your time writing the code, take your time to review the code, and do it carefully. Don't take all the AI shortcuts, the code will be crap!
Hate to say it, but student-teacher multi-agent is actually a great way to generate higher quality code if that's what you're aiming to do. That said, please still review your own code after the 2nd model pass, for now they're just little drunk interns that need lots of babysitting.
but wait man.. that architecture is highly sketchy tho. With or without ai
I have my reasons
Don't worry, the next generation of LLM will make this bugs even harder to find.
LLM's will generate the most clean and beautiful crap code that you will ever seen.
I disagree. We’ll have specialized ai agents in the coming years trained specifically to help pinpoint security issues.
@@WebDevCody but they will not suffer from the same hallucination problem? Do you really wanna have a security assessment based on probability? As you said in your video: the first time AI didn't find any issues, the second time found. Idk, probability-based-coding still sounds inefficient to me.
@@alexandrecolautoneto7374 you act like human's are perfect. Human developer suck at their jobs just as much, and most teams do NOT have a designated security expert reviewing every line of code written. All if takes if a human to scroll a little too fast on accident and he could miss a critical security bug.
@@WebDevCody yeah, maybe I'm too much on the defensive. It's just from experience, every bug that AI introduces to my code takes me more time than everything else and I turned from a passionate enthusiastic to a hater. I think AI is far from what they are selling and most of people are being scammed.
Sorry for being too intense, great work btw, keep up with the videos.
NEVER TRUST THE USER - EVER
Input validation+sanitation would have stopped this - but yea ... always validate/sanitize user input...
This is sometimes overlooked when you think you are the only user and forget that other people can use it
lol if only there was say a four year program to teach you about these exploits ….
so security issues are the only limitation of ai-generated code?
is that what I said?
First!
This came as a surprise. To noone. LLMs don't know what the code they print out does. They don't think logically, they don't think about security. They don't think. I think LLMs have a place in coding, but you need to know code if you're going to use it. Great for hashing out ideas and looking up documentation on things, not great to rely on for coding everything for you. That was glaringly obvious from the first time I tried to use an LLM to code me something with a bit of a complexity. It just couldn't do it. Even when I told it exactly what to do, why specific parts would not work, it kept falling back on faulty flawed code. Good as a tool, but you can't let it take control over your code. It's simply not capable of it.
they are called assistants for a reason, and I agree, it's a great tool to learn specific stuff once at a time, and it may still give out flat wrong information in some areas, great for not having to write agonizing code, or as shortcuts to writing code, I find it's actually quite accurate as long as you know what to ask it for, and if it's flat out wrong then it's easier to write your own iteration then feed it to the chat, and ask it logical questions - then it'll do OKAY enough to finish the implementation for you.
also the worst type of code it generates is always the dynamically typed languages/scripting languages, in 1 iteration of a simple router method it will pass arguments in a certain way, in the next one it'll use a totally different convention or syntax, or even response/error checking, totally chaotic.
lgtm