David Tolnay. He's the hero Rust deserves, but not the one it needs right now. So we'll hunt him. Because he can take it. Because he's not our hero. He's a silent guardian, a watchful protector. A dark knight.
@@jamesnewman9547 Welcome to the Rust "community". Then again, it was just like this since the early days. It's just been blowing up lately, and so everyone's been made more aware.
@@jamesnewman9547 To be fair, after a certain point of popularity and usage, you do have a moral obligation to treat your own crate as a public commodity that must be maintained with the public in mind. You're not required to ofcourse, it's your own crate, but you morally should.
@@jamesnewman9547 Well that's the thing with morals - they're subjective. But I'd say it's become an expectation because most FOSS programmers do treat their projects like this, and the morality in this situation is simply derived from trying to look after the greater public's good.
When I was younger and something at work upset me, I'd write up an absolutely scathing email. I'd spend a lot of time getting the absolutely perfect wording. I'd work to make it as terse as possible, while still including every detail of what made me upset. And then I'd delete the email without sending it. Writing that first email would drain me of so much energy, that I'd then write a second email which would be more like "Eh, I don't understand why this was done. Seems to me we could have done something better than this".
@@funkdefied1 not sure but on my keyboard I just roll the ball of my pinky on ctrl and pinky finger tip onto tab. it's basically one quick motion. my index finger doesn't even leave f.
i get the concern but calling this terroristic, as well as the point of dtolnay no longer being a FOSS maintainer, is just absolutely ridiculous. Accusing someone of terrorism because they shipped a bad change with security issues is absolutely overblown, and serde is still free and open source which makes it's maintainer a FOSS maintainer
This is just how the Rust "community" at large operates. A bunch of people who just think the language is neat, and an oversized representation of needy loud drama queens.
@@chrishoppner150 The only ones that are loud are the tech-influencer crowd and its following. There are always bad things happening in any community. It's just that you now hear it all the time because of dudes like prime who make money off of it.
12:13 actually it can run at runtime. Since the original purpose of these sort of libraries is to generate code, a malicious version of it could add a malware payload to every build as well.
@@anonymousalexander6005 it could be as simple as injecting some malicious code into every derived serializer and deserializer. That could could really do anything. As for "immediately raising suspicions" perhaps, because it would make the serializers and deserializers very slow if they were to, for example, make network requests or execute shell commands. However, that would be wasteful. They could probably use a static variable with a once cell or similar to ensure it would only execute once for each type of serializer/deserializer. The malicious code could also easily start another thread to do its work, so some slowdown could be avoided that way too.
@@curly35 Yes, technically you'd need to vet every library release (diffs) and build them in-house to truly avoid having any malicious code entering your build cycle or released code. Huge companies do that, but if you're not Google or Pentagon you likely don't have the resources to be that safe. And it's usually done for other reasons, not for security.
16:40 Rust is also pretty authoritarian. You /could/ use a fork of Serde, but then YOUR serde types are not compatible with the serde types of the other library. You really have to decide. Do you want your own system, or do you want to use the system everyone else uses. You could also fork every lib you use and make them use your serde fork. It's mostly because of the orphan rules. I love them, but sometimes I hate them. I guess, that's the main reason why people coordinate to use the same crates. In most languages you would just implement serde for all the types you import, if they don't already implement them. In Rust, it's pretty annoying, so you'd rather demand everyone to use the serialization library you use for your crate. That forces everyone to use the same library. If you don't use it, you are excluded by parts of the ecosystem. And I consider that some kind of authoritarianism. It might be effective to coordinate on some specific systems, but it takes away some of your freedom. You either have to obey, or you have to do your own thing almost completely. You can't just implement your serde fork for each type form other libraries. You /could/ do it if you derive every library you use inside your serde fork itself, but that's far from scalable. You're never done.
This shows the problem with having a deficient standard library. All these third party packages are providing core functionality, and end up depended on by the vast majority of projects. But the third party maintainer gets very little in return for all of this, and too often ends up overworked and unable to effectively maintain the critical library.
I agree. The fact that the Regex crate isn't in the standard library is insane. Although I think it just got promoted to being under the Rust namespace at least. Not sure if I remember right.
serde, syn, and so on are not really needed for everyone, they are not a core functionality its just that many many people choose to use them (but you can use other json crates and writing custom proc macros with 0 deps) your argument sounds like react should be a part of js
@@CYXXYC If you think parsing JSON in this day and age are not a core functionality, then you're insane. Go for all the mockery that it has so little feature compared to Rust already have very good JSON parsing capabilities in the standard library since forever.
@@muhwyndham java doesnt have json in std, c++ doesnt have json in std, js has json in std because duh its js-on, go is just weird also i didnt call you insane, and it just shows that youre coping hard
I'm sure I'm missing something, but can someone explain to me how downloading and running a precompiled binary is so much of a risk factor but downloading source from the same place, compiling it in your machine and then running it isn't? It's not like you're going to check the download code every single time, someone could still push a malicious version
As I understand, binaries are unreadable by humas, therefore they will cause too much damage before the malicious feature is even found out. Sure, anyone can still push a malicious version, but it's much faster to detect because it is open source. So risks are mitigated because they are easier to detect. Like you said, it's not impossible to cause damage, just harder
Theoretically, if there is some malicious code, it's easy to spot. Of course, not every user gonna check, but since the lib is so widely used, someone is gonna notice "add dangereous code" release. With binary, there is no way to check it until it's to late. But it all in theory. In practice open-source is non standart, and you probably still gonna use some closed binaries as lib/software anyway
answer : it's not. People, particularly developers, absolutely seethe when you say that but it's just flat out true. You aren't checking the code before you compile it, you're trusting that other people have checked it as a statistical certainty. If you ARE ever checking the code, you aren't going to find the local file on you're machine, your going to go to the repository and read the code. This IS a security concern, in the same way wearing a tinfoil hat is a security concern - sure, maaaaabye it'll do something, it's possible, but realistically it's just not. More security is better yes, but the reality is this is a massive convenience boost that saves absurd amounts of time and doesn't actually compromise security. Make the main crate precompiled and then make a secondary crate that's non-compiled. That way people who just really really really really really want to compile it themselves can, but everyone else can get the actual quality of life bumps.
20:21 This resembles the commit message I wrote, after begrudgingly unpinning chrono from 0.4.22 because 0.4.23 deprecated almost all panicking APIs (which I only used with literals anyway).
As a potential compromise, couldn't you make the binary first build (sure first one is going to be slow), then cache it and reuse it in the future, and a way to have sections built in release mode, hell if we are running rust code, couldn't we spawn a second copy of the rustc to do that already? Why isn't that a potential solution to this issue here? In fact isn't this is the idea behind ccache for c compilation, maybe partial binary caching is something rust should have generally?
@@EwanMarshall you edited your message my question before you added stuff about release was "doesnt cargo already cache built things" and yes someone in comments here already told me that you can turn on release for some deps
I don't use rust (sadly), so I think I am misunderstanding something here. I understand people's concerns that using cargo to build a project using serde-derive download a binary instead of building the binary themselves from the source. And that building your project with out having to also build the serde binary is faster... But when you compile a rust project, is it downloading each referenced crate's source(or locally reading the pre-dowloaded source code) and building the binary? Why can't you download the source code of the crates you want to use in a project, build their binaries from the source once, cache those resulting binaries until you update the crate version or rust compiler, and then whenever you compile your project, it just uses those binaries instead of building them on the fly? (This is how I assumed it would work and don't know why it wouldn't do it that way. But if it does work this way, I don't see how dowloading the binary from the crate would be any faster except on the very first project compilation?)
Cargo already does cache dependencies but it's based on compiler configuration, which will require re-building if those change. Yes, in normal scenarios it downloads the source code and builds on your system and caches them. As for serde_derive, it relies heavily on dependencies like proc_macros2, syn, etc. by the same creator. These modules provide APIs for generating Rust code at build time-they also themselves rely heavily on macros. Macros needs to be run at the pre-processing stage, so cannot start compiling serde_derive and other dependencies until those dependencies have already been resolved and built resulting in a waterfall effect. And chances are that serde_derive is not the only dependency within your own project either. This change was a way for the author to provide faster build times to users, as well as a way to push Rust/Cargo team to support pre-compiled dependencies natively, rather than relying on unsafe tricks like these to get around it. It has been removed as of 1.0.184 and no longer ships with prebuilt binaries due to the debacle around it and other use-cases (like Fedora not being able to use it as they cannot redistribute binaries they have not built themselves)
@dealloc Thanks, I think your second paragraph is describing the main issue for me. But I'm still a little fuzzy... 1. I'm not sure why macros needing to be run at the pre-processing stage before crate compilation would effect this. Or how that would be different from non-macro dependencies. 2. Are you saying that there are cyclical dependencies that are causing a problem? I'm not sure how Rust could even work if this wasn't already handled. Like you say, waterfall down the dependency compilation, and used the cached binaries for a crate if its dependencies don't change. I understand how the second one can become an issue if you're constantly updating crates and adding new ones, but if you are targetting specific crate version and have scheduled times to update/add new crates or new compiler changes to the project every week or so, then worst case, wouldn't it only have to compile all those binaries on a developer machine a single time between updates?
@@ttuurrttlle By pre-processing I mean proc macro crates must be compiled before they can be used and when they are used they are applied during build-time of whatever crate uses them, in order to inline the code that it produces before the rest of the source can be compiled as a whole. To prevent any recursive compilation, proc macros must be distributed as a separate crate. For example, serde_derive is the crate that provides proc macros for when you use derive feature with the serde crate. When you enable the derive feature from serde, it will depend on serde_derive which must go through the steps described above, before serde crate itself can be compiled. All crates are compiled in parallel by default, as long as any shared dependencies within graph are resolved as well. For example if crate A and B relies on C, C's dependency graph must be resolved first, before A and B can compile.
The thing I don't understand: why is the macro being compiled in debug mode in the first place? Like, it obviously just doesn't bindly copy all the build settings. Cause other wise cross-arch builds wouldn't even work. Like, Bazel has a similar paradigm: compilers/build tools can be compiled. It compiles the host tools in optimized mode (by default, that can be changed), regardless of whether the target is being compile in debug mode or not.
Is the risk factor really any higher than the pre-compiled DLLs/.so/whatever a lot of other languages use for third party libraries? I understand there will be edge cases but for most of us, is it really any worse than (N)Hibernate, bass.DLL, etc?
Python almost exclusively uses pre-compiled DLLS/SOs for EVERYTHING since the core language is so slow and everything has to be written as pre-compiled C libraries instead.
It gives you the impression that people are constantly evaluating the source code for security issues, the reality is they download it and compile themselves to their own binary and are in the same position. There is an unwarranted trust that security issues cant be embedded in source code distribution.
The answer is... Kinda, yeah? Because it's native, not IL, so it's much harder to analyze, it's running as a compiler plugin, so it would perform evil in a way that would be much harder to detect with virus scanners, and it would diffuse blame to the actual binary built, increasing the time to finding and stopping the attack. (I'm assuming a supply chain attack here, not much interesting in just sending emails from a dev box.)
What I'm not understanding is why does it matter? Like I totally understand the concept of a precompiled dll being an "attack vector", but at the same time how many people are downloading crates and manually going through and reviewing the code for it? So to me it just seems like a facade of security because nobody is actually going to be going through and reviewing the code themselves of the crates they use in their project so what difference does it make if you download it precompiled? So if it's established that this pre-compiled binary isn't malicious at all, but simply has the ability to be malicious then calling the guy a terrorist is completely uncalled for. I think the comment about having a separate crate for a pre-compiled version at 13:12 is the most reasonable suggestion lol. Maybe if someone could explain to me what the real issue is here, but to me if the repo is public, you can view the actions for it to see it gets compiled then uploaded as a crate I just don't see how that is different from downloading it uncompiled if no one is actually going through every single crate they use and reviewing its code.
Why not just have the option for precompiled wheels/build from source. And your project can decide which one to use. Or even stage your own compiled variant so it doesn't get built all the time - but you still cache a build you did once every few weeks? Crates can be yanked - but that's quite the process
i dont get why cargo cant just compile proc crates in release mode and save them somewhere i mean there's gotta be a reason to require a separate crate for proc macros right?
my immediate thought, even if each project recompiles them instead of "saving somewhere", rust definitely needs to allow users to compile libs at different optimization levels
@@CYXXYC rust does allow different optimisation levels for specific dependancies my question is if youre already requiring a separate crate for proc macros, why not compile them in release and cache it
@@kebbil i think i found that config thing. wonder if it affects proc macros, because then the problem is pretty much solved does cargo recompile proc macros themselves every build? if it does, it sounds not like a feature request, it sounds like a bug, since rust doesnt recompile most other deps if you didnt change version. if it doesnt, then a single long compile on first compile sounds good enough and all next compiles should be quick
All this was such an asinine situation. I honestly don't understand why the rust community isn't trying to make their compiler much faster. Linking with cargo is a freaking mess and especially in the case of macros, build times just explode if you are not careful. There's no excuse as to why the compiler needs to be this slow. It's also really sad to me that a handful of people control most of the popular crates in the ecosystem and yet their contributions are not heavily documented.
Honestly, while I get why it'd make people _mad,_ I think DTolney is in the right here, and certainly not being malicious. It's his crate, he can do what he wants with it. Others can also fork it. It could've been handled better, sure. But I wouldn't say he's trying to "force cargo's hand", that's ridiculous.
Exactly - or force a subscription. Until these guys get a truly free and open source community that is controlled by normal programmers, the business owner guys will always be weary of rust. Its like ~10 guys that write everything important and nobody has locked those guys into long-term contracts or figured out if somebody gets hit by a bus then who steps up.
then just compile it yourself. You know, that thing that you could always do and still can? Or hell, even just make an automated test script that runs everytime a new update is published, download the binary, download the source, build the source into a binary, compare downloaded and compiled binaries, if they don't match send out a bulk-email to the maintainers of the top 100 crates or something. You realize the linux kernel has countless binary blobs in it, right? And those aren't even open source, those are straight up closed source proprietary blobs that you can't verify. If we want to ACTUALLY make this comparison fair, you'd point out that every distro besides gentoo ships a precompiled kernel. If you aren't using this exact argument against Fedora, Arch, Ubuntu, KDE Neon, and every single other distro that ships a precompiled kernel, then shut the fuck up because I don't listen to hypocrites.
5:25 “What’s the solve?” Cringe. Why do people in tech insist on using verbs in place of nouns (nominalization) all the time? Informative video though. 😊
CIs are a thing, too. So unless you have thousands to burn a week/month on CI, because either compile times were slow, making release cycles slower, or you ended up upgrading the hardware, I'd wager that making compile times faster is a good candidate to focus on.
@@dealloc release cycles are indeed slower on rust, because 1. you take too long to write rust 2. you dont write usual many bugs that cause you to spam fixes and recompile over and over 3. once you write something in rust, in most cases its already quite good so it gets barely updated and ready to be left rusting
You just dont hear about other language drama I mean take js drama Shits insane In rust people get very passionate because it is such a pure and based language that aspires to perfection in a way tbh
@@kartonradrust is decent enough. people will notice the flaws and stop having the honeymoon reaction to it eventually. just takes several years, and in the meantime the hype compounds
those dramas don't affect people using Rust at all. the rust honeymoon ends in your first fight with the compiler. learning rust is like climbing out of very deep sand pit. but rust is so declarative, that it makes you a better programmer in the end. as all the shit you fighting the compile against are things that would be problems in other languages. so even if rust just cease to exist tomorrow... you are fit to use whatever other languages. and while it exists. it does deliver on its promises... painfully so
Because you need to implement it first, and it's what dev proposed to add to cargo. As far as I get it. It could have been solved with million other ways tbh
Also, people who commented with a thumb down emoji on the serde changes got blocked by the serde organization (see the "serde-blocked" repository). The root cause is the fact that Rust has a poor standard library. Writing or using a serializer requires 3rd party dependencies, which is nuts. That should be part of the language or the standard library. Go and Zig never had such issues.
Dtolnay's reaction (14:40) of (Im paraphrasing) " If MY CHANGE does not work for you, SOMEONE ELSE should make it work" comes off as unfair. He made the change, and now everyone else has to run in circles and react to it. In addition, the "(as I have done for ... and ... ... which I contribute SIGNIFICANTLY to)" that directly follows, comes off as arrogant and self-congratulatory. We show who we are not by our actions but by our reactions. IMO While I agree that this switch to precompiled binaries is a 💩y thing to do, calling it "terroristic" goes a bit too far. What this whole situation shows beautifully however is a problem that all package-based programming languages have. (Node, PHP, and Go have it too, to a certain degree). It is the fact that you now depend on 3rd parties and these 3rd parties get control over your projects. In Return, you get to use the results of knowledge you don't have. All so, that you can keep up with the breakneck speed that everything moves with.
How is it unfair? It's his codebase. What's unfair is every random drama llama coming in and shouting "I demand you make changes to YOUR code because I'm to lazy and/or dumb to do it myself" The community accuses him of trying to force his agenda on them, yet they don't seem all that bothered that they are doing the exact same thing to him in reverse. His response is the only correct one. If you don't like it, fork it. I'm willing to bet that a large majority of the people crying "binary blob bad" downloaded a precompiled OS, precompiled IDE, precompiled rustup, cargo, rustc, etc. Why is that entire supply chain perfectly safe but serde is such a massive security hole?
@@_mr_andersson I appreciate your thoughts. They just confirmed to me, that not learning Rust was the right decision. Just because you don't literally compile everything from a source does not make the criticism less valid. How is his action unfair? Let's look at an example outside of the coding space. How would you like it if your gas station sold you the wrong kind of fuel and then told you to "Fix it yourself. It's not their fault. They just changed it to something they like more. If you don't like it, run your own damn Gas Station". For the people who have trouble running the precompiled versions, this is exactly, what happened. He changed the thing they depend on and now is telling everyone having problems to fix it themselves. Being the good guy or gal you are, you would probably accept that and just say "Thank you, I'll visit again". But some people just don't like being forcibly bent over like that, you know.
@@AScribblingTurtle I see you made an attempt at an analogy. Not exactly valid though since no one is paying for serde. Try something like this instead.. AverageRustUser's local gas station has been giving away free fuel for the past few years. Suddenly they change the fuel type and AverageRustUser's car doesn't run anymore. They demand that the station owner switch back to the old fuel. The owner says "No I like this one better, but here are all the materials you need to build and run your own gas station. Completely 100% free." AverageRustUser replies with "Reeeeeeeee!" There are some things dtolnay could have done better though. He should have increased the major version, and he should have written a better changelog.
@@_mr_andersson At least we agree on what he could have done better at minimum. As I said in my post, this problem of people changing the rules on their packages and breaking other projects depending on them is not exclusive to Rust. However other package providers are aware, that they carry a certain responsibility if people start depending on their work. Changing your "free product", then changing the rules, and then completely ignoring the people who can't use the new changed version, because "Nobody pays you" would be an even bigger dick move. IMO. Just because you provide something for free does not mean you are not responsible for what you provide.
What does this means ??? Didn't Rust having already a standard library,something like on c or c++? pls explain me,bcz i don't understand the entire situation. Thanks.
probably when it stops being maintained or developed by anyone handling Rust. When you're directors straight up publicly say shit like "kill all men" "fuck you not sorry" and your organization covers their ass, people tend not to want to trust you with, well, frankly fucking anything.
@@gregandark8571 Similar to C and C++, Rust strives to keep its standard library feature-minimal. This is contrary to most other modern languages starting even with ones like Java, C# and Python, which attempt to ship "batteries included", so there is a standard and accepted way to do most things.
@@gregandark8571 It has a standard library but it covers relatively narrow set of features, for example: you need to install a crate to get regular expressions, serialization etc. Some say it's a good thing, but I disagree. I've been in this industry for 17 years and I don't recall too many cases when I though "damn, I wish I had to search for external dependency" ;)
![[MV] A leap of faith...ความเชื่อใจครั้งสุดท้าย (From "Petrichor the series")](http://i.ytimg.com/vi/U1piZH2CNXA/mqdefault.jpg)
This channel has turned into the authoritative source on all rust drama. It is like TMZ but for nerd shaming....
we all out here for the nerd shaming pastime
and im here for it, i want more of this content.
nailed it
David Tolnay. He's the hero Rust deserves, but not the one it needs right now. So we'll hunt him. Because he can take it. Because he's not our hero. He's a silent guardian, a watchful protector. A dark knight.
This seems like one developer was trying to fix a major flaw in the Rust ecosystem and due to another flaw, there was only one way of doing it.
@@jamesnewman9547 Welcome to the Rust "community". Then again, it was just like this since the early days. It's just been blowing up lately, and so everyone's been made more aware.
@@jamesnewman9547 To be fair, after a certain point of popularity and usage, you do have a moral obligation to treat your own crate as a public commodity that must be maintained with the public in mind. You're not required to ofcourse, it's your own crate, but you morally should.
@@VivekYadav-ds8oz More like if the public doesn't like it, the public can fork it.
@@jamesnewman9547 Well that's the thing with morals - they're subjective. But I'd say it's become an expectation because most FOSS programmers do treat their projects like this, and the morality in this situation is simply derived from trying to look after the greater public's good.
@@boomshakalaka8567 at the cost of fracturing the ecosystem, yes. Thank God forking is an option, but I don't think it should be the first resort.
When I was younger and something at work upset me, I'd write up an absolutely scathing email. I'd spend a lot of time getting the absolutely perfect wording. I'd work to make it as terse as possible, while still including every detail of what made me upset. And then I'd delete the email without sending it. Writing that first email would drain me of so much energy, that I'd then write a second email which would be more like
"Eh, I don't understand why this was done. Seems to me we could have done something better than this".
This is how I write TH-cam comments 😂
@@SimonBuchanNz - 😄
This is good advice
Familiar....
Based advice
I love how prime has a million shortcuts and custom key bindings, but clicks tabs in his browser when trying to find something :D
Tbf, ctrl+tab is awkward. Not sure what it’s like on his keyboard
@@funkdefied1 not sure but on my keyboard I just roll the ball of my pinky on ctrl and pinky finger tip onto tab. it's basically one quick motion. my index finger doesn't even leave f.
I'm surprised he doesn't use Vim keybindings for browser add-on .... its actually decent
Yeah surprised he's not using Vimium (Firefox and Chromium) or maybe Quite browser.
@@Definesleepalt Yup. Shift+t to find tab by name
i get the concern but calling this terroristic, as well as the point of dtolnay no longer being a FOSS maintainer, is just absolutely ridiculous. Accusing someone of terrorism because they shipped a bad change with security issues is absolutely overblown, and serde is still free and open source which makes it's maintainer a FOSS maintainer
FOSS thing was basically just "uhmmm, I don't let you call yourself a FOSS maintainer" which is silly.
This is just how the Rust "community" at large operates. A bunch of people who just think the language is neat, and an oversized representation of needy loud drama queens.
@@chrishoppner150 The only ones that are loud are the tech-influencer crowd and its following. There are always bad things happening in any community. It's just that you now hear it all the time because of dudes like prime who make money off of it.
I mean this is something that needs a bit of drama to get things moving in a better direction.
12:13 actually it can run at runtime. Since the original purpose of these sort of libraries is to generate code, a malicious version of it could add a malware payload to every build as well.
Once you peruse the Vault 7 docs, you see how insidious the reach can be -- and not just by the US. Those people have a lot of creativity.
@@anonymousalexander6005 "extremely hard" is very subjective. Not to mention that as long as there is an incentive, people will find a way.
How is that different than any ruby gem for instance? These ppl are crazy, every third party library has this risk does it not?
@@anonymousalexander6005 it could be as simple as injecting some malicious code into every derived serializer and deserializer. That could could really do anything. As for "immediately raising suspicions" perhaps, because it would make the serializers and deserializers very slow if they were to, for example, make network requests or execute shell commands. However, that would be wasteful. They could probably use a static variable with a once cell or similar to ensure it would only execute once for each type of serializer/deserializer. The malicious code could also easily start another thread to do its work, so some slowdown could be avoided that way too.
@@curly35 Yes, technically you'd need to vet every library release (diffs) and build them in-house to truly avoid having any malicious code entering your build cycle or released code. Huge companies do that, but if you're not Google or Pentagon you likely don't have the resources to be that safe. And it's usually done for other reasons, not for security.
The twitch comment, “TJ see you again tomorrow on 0 days since rust drama” has me dying 😂😂😂😂😂😂
I swear this is the 4th time I have seen this title.
Tomorrow will be the 5th
16:40 Rust is also pretty authoritarian. You /could/ use a fork of Serde, but then YOUR serde types are not compatible with the serde types of the other library.
You really have to decide. Do you want your own system, or do you want to use the system everyone else uses.
You could also fork every lib you use and make them use your serde fork.
It's mostly because of the orphan rules. I love them, but sometimes I hate them.
I guess, that's the main reason why people coordinate to use the same crates.
In most languages you would just implement serde for all the types you import, if they don't already implement them.
In Rust, it's pretty annoying, so you'd rather demand everyone to use the serialization library you use for your crate.
That forces everyone to use the same library. If you don't use it, you are excluded by parts of the ecosystem. And I consider that some kind of authoritarianism.
It might be effective to coordinate on some specific systems, but it takes away some of your freedom.
You either have to obey, or you have to do your own thing almost completely.
You can't just implement your serde fork for each type form other libraries.
You /could/ do it if you derive every library you use inside your serde fork itself, but that's far from scalable. You're never done.
This shows the problem with having a deficient standard library. All these third party packages are providing core functionality, and end up depended on by the vast majority of projects. But the third party maintainer gets very little in return for all of this, and too often ends up overworked and unable to effectively maintain the critical library.
I agree. The fact that the Regex crate isn't in the standard library is insane. Although I think it just got promoted to being under the Rust namespace at least. Not sure if I remember right.
serde, syn, and so on are not really needed for everyone, they are not a core functionality
its just that many many people choose to use them (but you can use other json crates and writing custom proc macros with 0 deps)
your argument sounds like react should be a part of js
@@CYXXYC If you think parsing JSON in this day and age are not a core functionality, then you're insane. Go for all the mockery that it has so little feature compared to Rust already have very good JSON parsing capabilities in the standard library since forever.
@@muhwyndham java doesnt have json in std, c++ doesnt have json in std, js has json in std because duh its js-on, go is just weird
also i didnt call you insane, and it just shows that youre coping hard
@@muhwyndham Hell, part of why Java became a thing was the standard Java library was so accessible, have we forgotten 30 year old lessons?
you guys are great together--technical synergy as well as comedic riffing
12:20 It could derive something different. For example it could send all your program data to some server, whenever it's serialized.
I'm sure I'm missing something, but can someone explain to me how downloading and running a precompiled binary is so much of a risk factor but downloading source from the same place, compiling it in your machine and then running it isn't? It's not like you're going to check the download code every single time, someone could still push a malicious version
As I understand, binaries are unreadable by humas, therefore they will cause too much damage before the malicious feature is even found out. Sure, anyone can still push a malicious version, but it's much faster to detect because it is open source. So risks are mitigated because they are easier to detect. Like you said, it's not impossible to cause damage, just harder
Theoretically, if there is some malicious code, it's easy to spot. Of course, not every user gonna check, but since the lib is so widely used, someone is gonna notice "add dangereous code" release. With binary, there is no way to check it until it's to late.
But it all in theory. In practice open-source is non standart, and you probably still gonna use some closed binaries as lib/software anyway
answer : it's not.
People, particularly developers, absolutely seethe when you say that but it's just flat out true. You aren't checking the code before you compile it, you're trusting that other people have checked it as a statistical certainty. If you ARE ever checking the code, you aren't going to find the local file on you're machine, your going to go to the repository and read the code.
This IS a security concern, in the same way wearing a tinfoil hat is a security concern - sure, maaaaabye it'll do something, it's possible, but realistically it's just not. More security is better yes, but the reality is this is a massive convenience boost that saves absurd amounts of time and doesn't actually compromise security. Make the main crate precompiled and then make a secondary crate that's non-compiled. That way people who just really really really really really want to compile it themselves can, but everyone else can get the actual quality of life bumps.
@@robonator2945or like a precompiled feature flag
FIrst, versions are locked. Second, the community will more easily detect malware in source code than in a binary.
rust's inclusivity invites all the twitter users
Savage ❤
There's nothing better than collaboration videos between these two gentlemen.
20:21 This resembles the commit message I wrote, after begrudgingly unpinning chrono from 0.4.22 because 0.4.23 deprecated almost all panicking APIs (which I only used with literals anyway).
As a potential compromise, couldn't you make the binary first build (sure first one is going to be slow), then cache it and reuse it in the future, and a way to have sections built in release mode, hell if we are running rust code, couldn't we spawn a second copy of the rustc to do that already? Why isn't that a potential solution to this issue here? In fact isn't this is the idea behind ccache for c compilation, maybe partial binary caching is something rust should have generally?
doesnt cargo already do that?
@@CYXXYC it certainly can't with this use of release build in debug, so not sure, but one could override that in this way.
@@EwanMarshall you edited your message
my question before you added stuff about release was "doesnt cargo already cache built things"
and yes someone in comments here already told me that you can turn on release for some deps
I don't use rust (sadly), so I think I am misunderstanding something here.
I understand people's concerns that using cargo to build a project using serde-derive download a binary instead of building the binary themselves from the source. And that building your project with out having to also build the serde binary is faster...
But when you compile a rust project, is it downloading each referenced crate's source(or locally reading the pre-dowloaded source code) and building the binary?
Why can't you download the source code of the crates you want to use in a project, build their binaries from the source once, cache those resulting binaries until you update the crate version or rust compiler, and then whenever you compile your project, it just uses those binaries instead of building them on the fly?
(This is how I assumed it would work and don't know why it wouldn't do it that way. But if it does work this way, I don't see how dowloading the binary from the crate would be any faster except on the very first project compilation?)
Considering the build code itself is rust code, one could do this anyway, we can for a compiler if needed to do it.
Cargo already does cache dependencies but it's based on compiler configuration, which will require re-building if those change. Yes, in normal scenarios it downloads the source code and builds on your system and caches them.
As for serde_derive, it relies heavily on dependencies like proc_macros2, syn, etc. by the same creator. These modules provide APIs for generating Rust code at build time-they also themselves rely heavily on macros. Macros needs to be run at the pre-processing stage, so cannot start compiling serde_derive and other dependencies until those dependencies have already been resolved and built resulting in a waterfall effect. And chances are that serde_derive is not the only dependency within your own project either.
This change was a way for the author to provide faster build times to users, as well as a way to push Rust/Cargo team to support pre-compiled dependencies natively, rather than relying on unsafe tricks like these to get around it.
It has been removed as of 1.0.184 and no longer ships with prebuilt binaries due to the debacle around it and other use-cases (like Fedora not being able to use it as they cannot redistribute binaries they have not built themselves)
@dealloc Thanks, I think your second paragraph is describing the main issue for me. But I'm still a little fuzzy...
1. I'm not sure why macros needing to be run at the pre-processing stage before crate compilation would effect this. Or how that would be different from non-macro dependencies.
2. Are you saying that there are cyclical dependencies that are causing a problem? I'm not sure how Rust could even work if this wasn't already handled. Like you say, waterfall down the dependency compilation, and used the cached binaries for a crate if its dependencies don't change.
I understand how the second one can become an issue if you're constantly updating crates and adding new ones, but if you are targetting specific crate version and have scheduled times to update/add new crates or new compiler changes to the project every week or so, then worst case, wouldn't it only have to compile all those binaries on a developer machine a single time between updates?
@@ttuurrttlle By pre-processing I mean proc macro crates must be compiled before they can be used and when they are used they are applied during build-time of whatever crate uses them, in order to inline the code that it produces before the rest of the source can be compiled as a whole.
To prevent any recursive compilation, proc macros must be distributed as a separate crate. For example, serde_derive is the crate that provides proc macros for when you use derive feature with the serde crate.
When you enable the derive feature from serde, it will depend on serde_derive which must go through the steps described above, before serde crate itself can be compiled.
All crates are compiled in parallel by default, as long as any shared dependencies within graph are resolved as well. For example if crate A and B relies on C, C's dependency graph must be resolved first, before A and B can compile.
The thing I don't understand: why is the macro being compiled in debug mode in the first place? Like, it obviously just doesn't bindly copy all the build settings. Cause other wise cross-arch builds wouldn't even work.
Like, Bazel has a similar paradigm: compilers/build tools can be compiled. It compiles the host tools in optimized mode (by default, that can be changed), regardless of whether the target is being compile in debug mode or not.
Is the risk factor really any higher than the pre-compiled DLLs/.so/whatever a lot of other languages use for third party libraries? I understand there will be edge cases but for most of us, is it really any worse than (N)Hibernate, bass.DLL, etc?
Python almost exclusively uses pre-compiled DLLS/SOs for EVERYTHING since the core language is so slow and everything has to be written as pre-compiled C libraries instead.
I also question your operating system kernal, firmware and the compiler at some point if we are going the full supply chain attack vector.
"is it worse than nhibernate" - no. Nothing is worse than nhibernate.
It gives you the impression that people are constantly evaluating the source code for security issues, the reality is they download it and compile themselves to their own binary and are in the same position. There is an unwarranted trust that security issues cant be embedded in source code distribution.
The answer is... Kinda, yeah? Because it's native, not IL, so it's much harder to analyze, it's running as a compiler plugin, so it would perform evil in a way that would be much harder to detect with virus scanners, and it would diffuse blame to the actual binary built, increasing the time to finding and stopping the attack. (I'm assuming a supply chain attack here, not much interesting in just sending emails from a dev box.)
you should get a vertical tabs extension
What I'm not understanding is why does it matter? Like I totally understand the concept of a precompiled dll being an "attack vector", but at the same time how many people are downloading crates and manually going through and reviewing the code for it? So to me it just seems like a facade of security because nobody is actually going to be going through and reviewing the code themselves of the crates they use in their project so what difference does it make if you download it precompiled?
So if it's established that this pre-compiled binary isn't malicious at all, but simply has the ability to be malicious then calling the guy a terrorist is completely uncalled for. I think the comment about having a separate crate for a pre-compiled version at 13:12 is the most reasonable suggestion lol. Maybe if someone could explain to me what the real issue is here, but to me if the repo is public, you can view the actions for it to see it gets compiled then uploaded as a crate I just don't see how that is different from downloading it uncompiled if no one is actually going through every single crate they use and reviewing its code.
Why not just have the option for precompiled wheels/build from source. And your project can decide which one to use. Or even stage your own compiled variant so it doesn't get built all the time - but you still cache a build you did once every few weeks?
Crates can be yanked - but that's quite the process
Dramas are needed to solve problems in Rust. Because Rust is the Drama Queen, RDQ for short.
i dont get why cargo cant just compile proc crates in release mode and save them somewhere
i mean there's gotta be a reason to require a separate crate for proc macros right?
my immediate thought, even if each project recompiles them instead of "saving somewhere", rust definitely needs to allow users to compile libs at different optimization levels
@@CYXXYC rust does allow different optimisation levels for specific dependancies
my question is if youre already requiring a separate crate for proc macros, why not compile them in release and cache it
@@kebbil i think i found that config thing. wonder if it affects proc macros, because then the problem is pretty much solved
does cargo recompile proc macros themselves every build? if it does, it sounds not like a feature request, it sounds like a bug, since rust doesnt recompile most other deps if you didnt change version. if it doesnt, then a single long compile on first compile sounds good enough and all next compiles should be quick
Isn't the solution to add an option to build proc_macros in release mode all the time?
building is slower in release mode
It takes more time to build them in release than it speeds up by a bug margin.
That was a whole lot of reading for a video.
dtolnay is David Tolnay. Look it up
Did something happened to rust again
All this was such an asinine situation. I honestly don't understand why the rust community isn't trying to make their compiler much faster. Linking with cargo is a freaking mess and especially in the case of macros, build times just explode if you are not careful. There's no excuse as to why the compiler needs to be this slow. It's also really sad to me that a handful of people control most of the popular crates in the ecosystem and yet their contributions are not heavily documented.
So Python compiled interpreter is terroristic…..
so is Linux, basically all distros ship pre-compiled kernels. If you aren't using gentoo, you're a victim of terrorism.
I have a question.
When did verbs become nouns?
2:30 They start explaining what happened
EDIT: Actually they waffle for another while, try 3:20 instead. My bad.
Honestly, while I get why it'd make people _mad,_ I think DTolney is in the right here, and certainly not being malicious. It's his crate, he can do what he wants with it. Others can also fork it. It could've been handled better, sure. But I wouldn't say he's trying to "force cargo's hand", that's ridiculous.
What's a long compilation at a big company?
Honestly it feels legal had the last laugh in this debacle.
always does
Lispchads stay winning.
For those who are looking for it, the issue is resolved in 1.0.185, so versions 1.0.172-1.0.184 should be banned in security scanners.
Pre-compiled is a great way to slip in nation state backdoors.
Exactly - or force a subscription. Until these guys get a truly free and open source community that is controlled by normal programmers, the business owner guys will always be weary of rust. Its like ~10 guys that write everything important and nobody has locked those guys into long-term contracts or figured out if somebody gets hit by a bus then who steps up.
Almost as good as buying computer hardware instead of fabbing it yourself
then just compile it yourself. You know, that thing that you could always do and still can? Or hell, even just make an automated test script that runs everytime a new update is published, download the binary, download the source, build the source into a binary, compare downloaded and compiled binaries, if they don't match send out a bulk-email to the maintainers of the top 100 crates or something.
You realize the linux kernel has countless binary blobs in it, right? And those aren't even open source, those are straight up closed source proprietary blobs that you can't verify. If we want to ACTUALLY make this comparison fair, you'd point out that every distro besides gentoo ships a precompiled kernel. If you aren't using this exact argument against Fedora, Arch, Ubuntu, KDE Neon, and every single other distro that ships a precompiled kernel, then shut the fuck up because I don't listen to hypocrites.
Well he's not going to be in proc-macro3
424 people blocked on github and twitter now..
jhe-san format
5:25 “What’s the solve?” Cringe. Why do people in tech insist on using verbs in place of nouns (nominalization) all the time? Informative video though. 😊
I swear I never had a compile time issue. Just get an extreme Workstation. You have to live with that in Rust.
meanwhile, people who live in countries where just the CPU alone costs a year worth of income even if you have a very high income
@@kuhluhOG rust is a privilege /j
CIs are a thing, too. So unless you have thousands to burn a week/month on CI, because either compile times were slow, making release cycles slower, or you ended up upgrading the hardware, I'd wager that making compile times faster is a good candidate to focus on.
@@kuhluhOG Then use Go, you can compile it even on a rotten Tomato 🍅
@@dealloc release cycles are indeed slower on rust, because
1. you take too long to write rust
2. you dont write usual many bugs that cause you to spam fixes and recompile over and over
3. once you write something in rust, in most cases its already quite good so it gets barely updated and ready to be left rusting
If I were dtolnay, I would probably yank every version of all my crates then peace out
nah what about forcing things into cargo? gotta keep that authority
Nix helps with precompiled binaries while preserving trust well, in my opinion.
who?
@@CYXXYC Nix is a Linux distro
@@hughesd.mungus9819 *NixOS is a linux distro based on Nix, a package manager and the name of the functional configuration language it uses
@@CYXXYC Nix, the package manager & build tool.
@@hughesd.mungus9819 a Linux distro/a package manager/a language. Pretty confusing if you ask me.
Rust seems cool, but there's enough drama in the world, I don't need a programming language with one.
this particular one isnt about rust itself really, its just this clickbait title at it *again*
You just dont hear about other language drama
I mean take js drama
Shits insane
In rust people get very passionate because it is such a pure and based language that aspires to perfection in a way tbh
@@kartonradrust is decent enough. people will notice the flaws and stop having the honeymoon reaction to it eventually. just takes several years, and in the meantime the hype compounds
those dramas don't affect people using Rust at all.
the rust honeymoon ends in your first fight with the compiler. learning rust is like climbing out of very deep sand pit.
but rust is so declarative, that it makes you a better programmer in the end. as all the shit you fighting the compile against are things that would be problems in other languages.
so even if rust just cease to exist tomorrow... you are fit to use whatever other languages.
and while it exists. it does deliver on its promises... painfully so
@@kartonrad
Use var!
No use let!
NO!!! Const!!!!!!
..
..
..
..
..
Hey, you don't need to use semi-colons, ya-know.
🤯🤯🤯😡😡🤬👹
im a react dev, i going to bet on golang instead of rust
Just write your own JSON parser, how hard could it be 🤷♂️
seriously, i really really fail to see how this problem wouldn't be solved with better compile cache.
does anyone care to explain why it doesn't?
Because you need to implement it first, and it's what dev proposed to add to cargo. As far as I get it. It could have been solved with million other ways tbh
Is TJ short for Tom Jenius
Also, people who commented with a thumb down emoji on the serde changes got blocked by the serde organization (see the "serde-blocked" repository).
The root cause is the fact that Rust has a poor standard library. Writing or using a serializer requires 3rd party dependencies, which is nuts. That should be part of the language or the standard library. Go and Zig never had such issues.
What's the meaning of "I am sorry or thank you"?
Rust isn't going to make it. It's peaked too early.
Flip actually cut the joke out!
Dtolnay's reaction (14:40) of (Im paraphrasing) " If MY CHANGE does not work for you, SOMEONE ELSE should make it work" comes off as unfair. He made the change, and now everyone else has to run in circles and react to it.
In addition, the "(as I have done for ... and ... ... which I contribute SIGNIFICANTLY to)" that directly follows, comes off as arrogant and self-congratulatory.
We show who we are not by our actions but by our reactions. IMO
While I agree that this switch to precompiled binaries is a 💩y thing to do, calling it "terroristic" goes a bit too far.
What this whole situation shows beautifully however is a problem that all package-based programming languages have. (Node, PHP, and Go have it too, to a certain degree). It is the fact that you now depend on 3rd parties and these 3rd parties get control over your projects.
In Return, you get to use the results of knowledge you don't have.
All so, that you can keep up with the breakneck speed that everything moves with.
How is it unfair? It's his codebase. What's unfair is every random drama llama coming in and shouting "I demand you make changes to YOUR code because I'm to lazy and/or dumb to do it myself"
The community accuses him of trying to force his agenda on them, yet they don't seem all that bothered that they are doing the exact same thing to him in reverse.
His response is the only correct one. If you don't like it, fork it.
I'm willing to bet that a large majority of the people crying "binary blob bad" downloaded a precompiled OS, precompiled IDE, precompiled rustup, cargo, rustc, etc. Why is that entire supply chain perfectly safe but serde is such a massive security hole?
@@_mr_andersson I appreciate your thoughts. They just confirmed to me, that not learning Rust was the right decision. Just because you don't literally compile everything from a source does not make the criticism less valid.
How is his action unfair? Let's look at an example outside of the coding space.
How would you like it if your gas station sold you the wrong kind of fuel and then told you to "Fix it yourself. It's not their fault. They just changed it to something they like more. If you don't like it, run your own damn Gas Station".
For the people who have trouble running the precompiled versions, this is exactly, what happened. He changed the thing they depend on and now is telling everyone having problems to fix it themselves.
Being the good guy or gal you are, you would probably accept that and just say "Thank you, I'll visit again".
But some people just don't like being forcibly bent over like that, you know.
@@AScribblingTurtle I see you made an attempt at an analogy. Not exactly valid though since no one is paying for serde. Try something like this instead..
AverageRustUser's local gas station has been giving away free fuel for the past few years. Suddenly they change the fuel type and AverageRustUser's car doesn't run anymore. They demand that the station owner switch back to the old fuel. The owner says "No I like this one better, but here are all the materials you need to build and run your own gas station. Completely 100% free."
AverageRustUser replies with "Reeeeeeeee!"
There are some things dtolnay could have done better though. He should have increased the major version, and he should have written a better changelog.
@@_mr_andersson At least we agree on what he could have done better at minimum.
As I said in my post, this problem of people changing the rules on their packages and breaking other projects depending on them is not exclusive to Rust. However other package providers are aware, that they carry a certain responsibility if people start depending on their work.
Changing your "free product", then changing the rules, and then completely ignoring the people who can't use the new changed version, because "Nobody pays you" would be an even bigger dick move. IMO.
Just because you provide something for free does not mean you are not responsible for what you provide.
All the complaints about Rust's compilation time... Why not build the Rust compiler in Rust?
This is an intentionally dumb question btw.
Joke would have landed better if the compiler wasn't actually built in Rust, which it is
@@eleyondfarli but that is the joke?
Due to the rust drama I've decided to learn zig instead.
Learn both. Then when zig is primetime ready you can ditch the clunky one
@@blarghblarghI like that. I shall!!!
@@blarghblarghI'm a dummy in the programming world,so for this i need to make this question.
Why everybody is very exited about zig ?
@@gregandark8571 Because it's very simple and pleasant to use. It's also faster than Rust.
0 days since TheDramaGenerator making clickbait drama.
TheDramaGen
I wonder when Rust community acknowledges that broad standard library is a good thing...
What does this means ???
Didn't Rust having already a standard library,something like on c or c++?
pls explain me,bcz i don't understand the entire situation.
Thanks.
probably when it stops being maintained or developed by anyone handling Rust. When you're directors straight up publicly say shit like "kill all men" "fuck you not sorry" and your organization covers their ass, people tend not to want to trust you with, well, frankly fucking anything.
It's not. Please leave your bloat std lib filled with garbage to yourself.
@@gregandark8571 Similar to C and C++, Rust strives to keep its standard library feature-minimal. This is contrary to most other modern languages starting even with ones like Java, C# and Python, which attempt to ship "batteries included", so there is a standard and accepted way to do most things.
@@gregandark8571 It has a standard library but it covers relatively narrow set of features, for example: you need to install a crate to get regular expressions, serialization etc. Some say it's a good thing, but I disagree. I've been in this industry for 17 years and I don't recall too many cases when I though "damn, I wish I had to search for external dependency" ;)
AGAIN??? What's this time?? too lazy to watch
Video with teej? No thx, I'm good...
The back-and-forth endless circlejerking with those two can get very intense, I don't blame you for skipping. 😹
@@MyAmazingUsername he thinks he is funny. He is not.
@@js-ny2ru I pretty much agree with that. Videos with this pair tends to get really hyper and like giggling schoolgirls.
I find he pretty funny, also his streams are really good, he answers people even in the “dumbest” questions. Give him a try! ;)
Haters gonna hate
I have 2 doughters and both are girls 😁
RUST SUCKS I CAN'T LEARN RUST HELP OH GOD