I recommend creating a random letter generator and deploying it so it inserts random characters on the entire code base... Immediately achieve -100x engineer status ♥
"The interface didn't change." This is only true for the most naïve understanding of what interfaces are. They encompass more than just function signatures. That bit about gnu parallel was dope! Thanks for sharing!
Yeah, with that logic you could replace every function body with an infinite loop. It goes the other direction too though "you can't rely on arbitrary implementation details not to change between patches"
True, but I assume he meant "the (programmical) interface didn't change a single signature/line of code." Dunno the first thing about Python, but I assume it has interfaces..
“I used to say that when being CEO at Apple wasn’t fun anymore, I’d quit. But now I’ve changed my mind - when being CEO isn’t fun anymore, I’ll just fire people until it is fun again.” -- Michael Scott, CEO of Apple, regarding the massive layoffs undertaken to solve the bozo explosion, a few months before he got fired for it
19:55 When I worked at the NAIC we had an application that at one point started failing in production. It would fail 9 times, then succeed, then fail 9 times and then succeed. I traced it back to a dynamic SQL query where the code was interpolating an ID value into the SQL string and we had reached the point where the IDs were large enough to trigger a conversion to scientific notation. Unfortunately, in the conversion to scientific notation, one digit of precision was dropped, so if the last digit was non-zero, the query would fail. If the last digit happened to be zero, it would succeed. Good times...
This is an amazing story. Perfect answer for the interview question about what's the most interesting problem/bug you've faced. Wish i had something similar
Regarding your semver story: defaults ARE part of the interface. If you change defaults, you're changing how someone has to call your function in order to get the same behavior. That's the definition of an interface change.
@@sullivan3503 idk about always. Depends on the default. Rust changing how cargo uses the registries from cloning the whole registry to the sparse protocol as a default isnt really breaking. As all the intended uses of cargo would be fine with the sparse protocol. (They are doing this soon)
I met an engineer like this. They promoted him to principal engineer. He doesn't write code, he just rejects everyone else's code and gets a nice bonus for it. Obfuscating requirements just makes all of that easier.
I worked with a guy like this as well, he always had a problem with everyone else's code and the boss really liked him for that. I thought he was really smart until I dug deep into the code base and had to dig through an ungodly amount of abstractions. It was so insane, whenever you added a new feature he expected you to add 20 layers of abstractions to something simple. It's almost as if he wanted to add extra layers of complexity that didn't need to be there so that A) He could sound smarter and B) he would always have job security if he's the only one that understands this over engineered code.
The lead of my automation team accepts requirements for literally any one who will give them no matter what they are. Not tell any one the requirements were accepted. Not verify if things will be impacted by gathering random requirements from random people with random levels of knowledge. Only sometimes take down meeting notes, and only sometimes accurately. Then be amazed when everyone things the finished product is a worthless pile of crap that almost never runs correctly and that their back log is an endless pile of crap.
That’s what principals and even worse, fellows, do. They just tell you that you are wrong. They shit down all of your ideas. And eventually the company wonders why they haven’t had a new idea in 15 years
You could do this: -have a case-insensitive language - define NULL - define null Sometimes it will get corrected, sometimes it won't Of course they should have different usages but vastly overlapping
The closest thing I've seen to the perfect bug was in a geometric unit test with bad randomisation. It almost never failed when running tests locally, but in the CI pipeline it seemed to almost always fail on master (never on a branch). The fun thing was that rather than generating 10k random polygons, it would generate the same random polygon 10k times 😅 so you had to be really (un)lucky to reproduce it
That test one was soul crushing. I had a coworker who wrote some absolutely horrible test code, so that when I fixed an issue, the entire test file broke and complained. I had to completely rip out the mocking and re-do the file. Great fun!
I used to just CODE THE FIX (ie visibly and indisputably add value), then email the before and after code to whoever owns the broken code and all the obtuse tests, CC my boss and their boss, with "hey, I think there is a defect is your code because A should be B (attached). can you please confirm, update your tests, and commit ASAP as this is a production bug. kthxbai"
had similar experience - fixed a bug in one place, checked the tests for that part - everything fine, created PR - 27 tests in some random parts of the app failing because they were importing some bullshit that was indirectly tied to this bug as working code
I fully agree with your take on threading. I did this a while back on a brute force keyboard layout optimizer. I wrote incredibly optimized code for one thread, but I allowed the isolation of state resources in such a way I can just run it on however many threads I want without an issue. I got insane performance doing that
Dude I once had a job that had me work across a dozen different departments, each with their own rituals, scrum masters, traditions, and software. Every day I had to deal with a hodgepodge of different jira boards and weird other facilitation dependencies that I never care enough to learn the names to. I spent 3/4 of my time simply in pointless meetings, "social times", and trying to update my tasks across everything rather than actually developing and architecting software. Quit after 6 months. Had me wanting to go back to VBA development
As a VBA developer, I feel this in my soul. I will say that not coding in some other language makes hobby projects easier to not hate and it's retroactively made me look at how to use VBA fundamentally differently and now building projects in it is pretty rewarding compared to the crapware macros that dominate big businesses.
The 4:55-5:23 part. Man I have this right now. We got a "manager" who wants to make the most elegant software and behaves like an architect (I have not seen him write a piece of code). He goes around to everyone touting his ideas and every interaction goes like this: "I don't want to tell you how to do this, but I want to understand and ..." then proceeds to criticize you . And when we have a technical discussion he just pulls everyone into technical vortex of ideas that does not go anywhere, it's tiring to have any technical meetings with him. And my biggest issue is he knows a few things well, and in other things like Frontend/Infra (originally backend developer) he just goes full Dunning-Kruger effect. It's really toxic and he pushes us for to reach unnecessary deadlines (like for a meeting that had little to no importance). And we have already rewritten the backend because of him, next it's gonna be the Frontend or the Infra. :D I am sad because my team is great, I am on a greenfield where we experiment, and figure a lot of things as we go and it's really fun it's just we have "Him".
Think outside the box, be creative, come up with fresh new ideas... but also I'm going to micromanage you and question every line of code that you write. And then when you become demotivated because you know that everything you do will be changed or rewritten they become annoyed.
Speaking of the true magic bug, I had that at work. Sometimes builds would fail because they included invalid code, but when running the build again it would work. After weeks of having that bug pop up here and there, I finally found out that the Console Ninja extension was modifying files in the node_modules, and those files got compiled into the dev build as-is, breaking the build. So if you want fun bugs, have a tool that subtly modifies the output binaries, and introduce fragility there. Now your engineers can't treat the compiler as a pure function, and will go crazy.
On the subject of weird c causing random crash,there was the underhanded c contest. The last entry winner used a dirty typedef tp change the type of the pointer of a function: if you compile the unit of code, it takes f32*; if you include the file it takes f16* (or the otherway around ) The compiler doesn't se a problem. The linker doesn't care. And you got a really weird edge case were the program errors out
Take the logarithm of a random number between 0 and 1, and say it is negative. Every number in this interval has a negative log. But sometimes you'll get a number so close to zero, its log will be a huge negative number, causing an overflow.
Side note about the side note about the GNU Parallel, I definitely agree. What you're really thinking of is coroutines and asynchronous programming. Essentially every part of the program only needs to be concerned with itself and its connections. Then a separate "manager" controls everything. You don't have to try to control "this thread needs to be spawned" or "this process needs to do this now" or whatever, everything just works and it's the greatest.
The fear of experiencing that semver story is why we never bump dependencies without testing. Dependencies are pinned to exact versions, periodically updated in a batch, ran through integration tests, and then allowed to mature on test servers till the next release. We don't rush the process unless there's a CVE affecting our codebase.
The original research by Tom Demarco that referred to some engineers being 10x more productive than others had more to do with environment than skill, computer language or years of experience. The book "Peopleware" describes this. And it recommends people having private, quiet offices. The current trend of putting a bunch of people in the same room, and hiring for super engineers, is opposite to what the research was proposing. This common misconception is similar to the one that made governments standardize on the waterfall model, whose original paper mentioned it as how to NOT develop software.
The trend for putting multiple in large open office spaces is maliciously deliberate. It's there so employee's feel the urge to be productive and not be ratted out by other employee's for goofing off/doing nothing.
yeppers. but that is the thing. without developing tools for the CLI, you just don't know these things. even if you knew the linux philosophy, it doesn't really set in until you get parallel's existence...
@@ThePrimeTimeagen Made my day as a long time Unix/Linux admin to see that. I have actually beaten our dev team''s in house code more times than I can count simply using standard awk(mawk is even faster)/grep/parallel routines, specifically on customer data dumps we need to massage. But that's not exactly their fault they were using Java and Python. Hard to complete with 40+ years of optimized code from some of the best computer scientists. A note about Mawk. It might be one of the fastest text processors that ever existed. The developers even found ways to knock cycles of a lot common routines in the code. Its written in C and beat pure C in lines opened and closed speed form the standard routines.
I'm a data scientist. I was working in R on an old project, and learned that you could redefine the addition operator globally for the interpreter's session. Naturally, I redefined it to randomly add one to the result every now and then. You could fix it by restarting the interpreter session, but if you managed to sneak it into a script that got imported somewhere... :)
The first thing you do with the indispensable employee is fire them. Eventually they will quit or get hit by a bus and you can't have that much risk in your company. Any competent person can untangle the work of an idiot.
@@bkucenski Haha good luck with that. That one employee is usually the only guy making this unique product / service technically possible and feasible, and finding another one like that is zero chance.
@@edgarmedrano2562 you found a very strange "authority" to quote. Jordan peterson is a brain-dead quack, and pretty much everything he mumbles is an utter nonsense.
No, it appeared because of people like Fabrice Bellard (author of QEMU, FFmpeg, a PC emulator written in Javascript, TinyGL, QuickJS, the Bellard's formula, and many other things/awards/world records, etc.).
@@brinckau Curious that Fabrice Bellard is mentioned every single time 10x is discussed. Like he's the only example people can think about. A one in a billion genius (deservedly so). Yet, every freaking bodyshop with 20 employees boasts of hiring 10xers. There is only one Fabrice Bellard out there. There are no other people like him. Certainly not thousands, not hundreds, not even dozens.
@@vitalyl1327 Yes, there are very rare. People with incredible abilities are always very rare, whether the field is programming, math, sport, etc. The distribution of bad/average/exceptional people follows a Gaussian curve.
I'm still a pretty young developer, only 2 years out of uni. I've seen some crazy bugs, let me tell you, and you can get away with a lot in a multi-language environment when, say, you're modifying C code and most developers work largely in Java. Variable names containing a double underscore, or that start with an underscore followed by a capital letter, are reserved for the language / toolchain, but the compiler won't stop you. Back during a co-op, I had a crazy time figuring out why a build wouldn't work when updating the version info handshake from C firmware to a Java host when the version string variable in C looked like firmware__v1_2_3. That __v doesn't do what you might think. I'm sure you can find another sequence of characters that'll fail in a context-dependent way. This last week I was struggling with an issue that didn't show up in our simulated environment, only on our real device hardware, and only for some systems. Turns out we had a linker script specifying the beginning of the stack was aligned to 8-bytes, but not the end... and a stack canary that was an int, and hence wanted to be aligned to a 4-byte boundary. An unrelated change fixed another invisible bug that led to the wrong linker script being used, so the stack was now in a different region of memory that it now shared with global variables. So the linker could just slip a 1-byte global variable in that little offset and kill the canary on startup. Before communications are established, even, so the device can't even send an error message. The common theme with these two changes was that they were small parts of mega-merges. That was sadly missing from the overall -10x list -- get your team into a situation where there's a very large merge of death, so that finding bugs can't be done by just looking at small changelists.
The part about nonsensical dependencies hit home pretty hard. Almost all the projects I've inherited has had this problem. I've found lodash in projects where all it was used for was checking a few strings. I've found form validation & component library combos that create a huge amount of extra code overhead to make work, and all it's been used for is validating that text is text and numbers are numbers.
RE: gnu parallel, I *definitely* recommend using a utility on top that will also take a list of colors and process names as input like so: better_parallel \ -c 'red,blue,green' \ -n 'foo,bar,baz' \ do_foo \ something_something_bar \ hault_and_catch_fire_baz output: [_foo_]: hi *[bar]:* starting *[bar]:* stopping *bar* exited with 0 [_foo_]: this [`baz`]: wazzuuup `baz` exited with 0 [_foo_]: is [_foo_]: program [_foo_]: bye _foo_ exited with 0 And also an option to kill all processes when any one of them exits abnormally. Trust me, it'll make you life nicer to be able to tell what's printing what. For bonus points, get all dispatched commands to output to a different terminal window/tab.
I know nothing about gnu parallel, but your description makes me think "Can that possibly be setup to autofeed tmux so everything is it's own detachable session?"
@@shrikedecil I'm just going through man parallel_tutorial, and curiously this seems to be a built-in feature? quote: *A terminal for every job* Using --tmux GNU parallel can start a terminal for every job run: seq 10 20 | parallel --tmux 'echo start {}; sleep {}; echo done {}' This will tell you to run something similar to: tmux -S /tmp/tmsrPrO0 attach Using normal tmux keystrokes (CTRL-b n or CTRL-b p) you can cycle between windows of the running jobs. When a job is finished it will pause for 10 seconds before closing the window.
I will help you with your quest for entering a heisenbug into your repository: - Step 1: Use a language where you can query line numbers - Step 2: Make an evil macro that only triggers when the usage site line number is in some range. Make it randomly inc some random mem location. - Step 3: A thousand commits later, profit. You swell in pride when that perfectly sensible code some poor dev tries to merge into CI constantly fails. You will not be blamed.
Sure, but easily detectible during a code review. It has to be something that doesn't seem out of place. Of course look at your peers and understand their skills. You have a better chance if you can convince them.
@@dealloc You can get some pretty gnarly macros (Boost preprocessor stuff might be a good example, I recall seeing some other sites talking about how to do branching / complex logic in macros). Stuff like using ## to append a conditional to a root, and then wrapping it in another macro to get the preprocessor to evaluate that root0 or root1 as another macro. It looks gnarly when it's implemented correctly, I'm sure you can slip in a conditional of `__LINE__ > 700` somewhere in the macro call chain. As a bonus, this whole thing happens at compile time and is thus un-debuggable.
I got a language where you can query AND change line numbers at runtime: Commodore BASIC. With some POKEs and PEEKs, you could relink the order of the interpreted lines 😈, change them all to be line 0 🤪 and there were probably other creative hacks which I don't have on the top of my head right now.
I've had something similar happen by chance. It depended on some instruction's alignment with the cache... It worked *most* of the time, but some builds failed every time...
Thanks for heads up on parallel, i will add it as a part of my repertoire. For ad hoc file processing i find "fd -x" to be the most convenient. Runs in parallel
15:06 "Migrate your database frequently": That's very cost-effective wasting technique. Plus, you get a bonus combo with the "burn out" section by cycling this evil tactic in order to end up, after many migrations, with the exact same database spec the whole team started the project with. Just to let them realise how pointless their migrating efforts were. Then ask for migration again.
On every other Tuesday we have a day full of meetings. The morning is wasted with sprint planning (3 hours across 5 people), in the afternoon we have the retro which (1.5 hours across 5 people), then we have sprint planning 2 which is another 90 minutes where we assign the tasks... Finally on Mondays before the sprint planning we have the backlog refinement which is 3 hours (4 people) trying to come up with subtasks for things you can't even know yet. It drives me insane sometimes.
Sure, but who in their right mind pays full attention during backlog refinement? Half of SCRUM ceremonies are implied "work on a little shiny problem on the side, while pretending to pay undivided attention"-time. Unless of course you do it in person, in which case I hope you are on some strong antidepressants.
Nonono, You don't aim simply for random. That's not evil enough. You tie that trigger to a hash of the code itself so it is a "consistent" random bug. Reverting code should fix the bug. Nobody should be able to know WHY reverting code fixes the bug, but it always goes away with the revert... until it happens again.
I heard "Use grep with parallel" and immediately looked it up. Huge game changer. Edit: I love that the man page links to a youtube video and a cheat sheet instead of the usual barely explain what the tool doe appraoch of some other manuals (looking at you, dmenu).
My company's software has a lot do with timing, like dealing with when a store opens, closes, special holiday hours, etc. Some of our PHP tests had things like time("-4 hours") which caused the tests to fail when ran between 0:00-4:00 UTC. I discovered these bugs because I was the only one working at those weird hours and was wondering why super old tests were suddenly failing
Once upon a time I was working on this big old PHP codebase that had originally been written by a team completely oblivious to the fact timezones exist. And now it was being deployed to customers in multiple countries. So being young and keen I fixed a couple of egregiously broken timezone bugs. My reward was spending the next 18 MONTHS exclusively on datetime and timezone fixes....
I'd recommend allowing devs copying and pasting randomly found solutions from stack overflow and elsewhere and then asking seniors to CR that. Even better without testing if that even works. Also randomly naming variables and functions work wonders. Like using a for iterators, x for arrays, 'class' for function names, get for setters...
20:18 When python creates a new int from a literal, it creates a new instance of int to hold the information. But since ints are immutable, cpython actually pre-initializes and caches instances of int for numbers from -5 to 100 at startup. Soo there is a global list of ints just sitting there in cpython, and it seems a cpython module could just... I dunno... reorder them. So that anywhere in your entire codebase that you assigned a literal 7 to a variable, it would store 8 instead. This might not be a fair game to play with interpreted languages with extensions. It can reach the same level of debugging difficulty, but it's not as clever of a prank.
Java has both primitive ints, and immutable boxed object Integers. Using the == comparison operator works as you'd expect for primitive ints, but compares the memory address of boxed Integers. Turns out that Java caches at startup all the boxed Integers from -128 to 127, so an Integer == Integer comparison will "work" if you only write tests with numbers in that range. But the moment you try comparing Integers outside that pre-cached range, all bets are off. I've actually seen this bug in production code...
Introduce side effects. This way it makes it easier to affect it from the outside which is harder to detect. Side effects can be implemented in a way that seem to be pure. Especially in languages that has lesser or no type systems.
Also floating point arithmetic can be a great way to obscure what's going on. Because no-one understands floating points unless they've read the specification (and understood every word of it). Another is if you work with date and time, you can just implement something in a way that makes sense based on people's assumptions about how date and time works. That way the code seems to be true on the surface, but for the untrained eye, you can truly cause havoc. Also you can introduce randomness this way.
@@dealloc Both of these are excellent. I'd add in using floats as Booleans and as many implicit conversions between strings and other data types as possible.
One of my favorite ones is adding 40 layers of security theater on anything you do so even basic tasks take at least several steps from an user standpoint. "Go add this info to the wiki". Great, let me disconnect from our AWS VPN and connect to the internal VPN, that'll take some time, then I'll go log in to the wiki. Oup, wait, 2FA has to hit my phone give it a sec. Alright, now let's open the authenticator app... ah this code is about to expire, let me wait a few seconds for the next one... ah shit, the login page expired, let me reload and try again... ok, we're in, what was the internal project name for this thing again? Oh I lost access to this last week? Let me file a ticket to get it back, gonna log in to servicenow, give 2FA a second to hit my phone... repeat ad infinitum
Xerox PARC pioneered the bozo explosion thing. Their rule was you could not hire somebody unless they were as good as you or better. Steve probably picked it up there, along with the mouse and tablet.
I don't think documents are inherently bad when you're trying to communicate something to others, it's just that they're not always well made and reviewed, or they're used too much. I'd much rather read on my own time what is my colleague's plan for something important (and this is a big factor to the success of this; if you start this process for everything it will slow things down severely), leave a few a questions or comments and wait for him to respond on his own time, than try to sit down in a meeting and do it synchronously. The trick with this is you have to start the planning a bit early, but the advantage of being allowed to marinate your thoughts on an issue are very serious.
19:20 - this actually happens in graphics programming. Some devs call it "Uber Shader", a rendering program that is created to handle every possible case, most of the time, by preprocessor macros. When you try to handle multiple platforms, stereo rendering (for VR), few different rendering techniques, you end up with a code that takes 30 minutes to understand one if statement. Also import order matters, one macro can be enabled in one file, and then disabled in the next file. When trying to understand the code, you feel like a detective. Pin board and huge notebook is a must have :')
20:36 @ThePrimeTime Ideas for the most insane bug factories: - Make routine result depend on *available memory address* at runtime. - Make routine result depend on *last measured call frequency* at runtime. - Make routine result depend on *CPU load at* runtime. - Make routines result depend on *network delay to answer* at runtime. - Make routine result depend on *shared heap state at runtime.* - Make all routines shared methods so their result depend on *their current caller.* - Give all routines internal state so their result depend on *their previous calls* (aka make them silently programmable, my favourite). - Make routine *infer* valid results *from errors* returned by sub-routines. - Make routines *infer* valid parameter values *from non-valid parameters.* In general, apply the "never fail to present some plausible result *no matter how* , *at all cost* ", just like the Russian army. And of course, don't forget to create an _undecorate_ decorator, otherwise it's not fun enough.
I had a boss who enforced code reviews for all engineers, except himself. We'd come back from weekends and there'd be a dozen check-ins. Whole systems would be broken.
If you really want a nasty bug, you can write your program in C and write 1 element past the max index of a vector. You can create bugs that disappear while debugging and only happen in production under certain conditions depending on your code and the compiler. BTW, error / crash will be in an unrelated area of your program. (you can also recreate this problem in javascript/typescript, although it is a bit harder to do)
Or even better, hack the allocs so that they always allocate a few bytes less than requested Though i think valgrind can easily detect illegal memory accesses. So i reccomend tricking valgrind into believing that illegal accesses are totally legal
def change_random_var(): for variables in globals(): if random.random() > 0.001: continue globals()[variable] = "surprise!😂" This function should (unless I wrote a bug in the yt comment) change a random global variable to a string 1/1000th of the time. Function names change Dunder variables change Normal global vars change Split this function up between 4 imports
BoZo explosion! Never heard that before. So true, lived through it, 5 years all went to shit. Company valued obedience and never let the capable ones have power/make-choices
The bug triage stuff is wild. Sometimes you find bugs caused by dependencies doing something stupid in some way where you can't even tell where the source of truth is. Recently was debugging an issue where tooltips overflowed the tooltip container on a graph. Graph was zingchart. It was a custom html tooltip. Zingchart generates the background of the tooltip in the SVG of the chart and puts the html tooltip outside the SVG and lines them up. Took way too long to understand what the hell was going on there. And how to fix it. Even worse there was no way to directly inspect the tooltip, since it was be removed if you loved the mouse or clicked.
This list SCREAMS at me. Im in one of these groups, especially the busywork. If I have to sit through one more damn pointless meeting.....hold on...meeting...
we had a test that failed whenever the test was executed at .999 of a second - the setup would create an x second expectation but the actual was then the next second (didnt take long to find tho , cause the moment i spotted asserts for dates i was looking out for it ^^)
Working at a bank right now. The team I'm at right now is honestly pretty good, I'd say technical expertise/general knowledge is 9.9/10 and team cohesion is like a strong 8/10. But watching this video... You know, especially with the tests. There were some -10x engineers in this team previously, seriously. Some parts of the code are an unbelievable debugging nightmare, just insane
Evil C++ bug idea: 1. Make a utility "tool" that takes "references" to objects from a class that inherits from some virtual class, ideally a class written by someone else, they are going to be your fall guy. 2. Make it an overengineered mess to scare people away from attempting to understand anything about it or use whatever 3. obfuscation methods you want. 3. Sometimes, just sometimes, subtly overwrite the reference to the table that holds the references to the final implementations of their virtual functions, AKA the vtable, with a similar table that has a few functions switched around. Every object of a class that inherits from virtual classes has it's own reference to the vtable, so you don't need to do it for all the objects if you are able to take many as an argument. Try to choose the least called function out of all of them so the bug is less common. Congratulations, you made someone else's code fail inexplicable!
At 4:00 who said "that's how SEMVER works" is wrong and doesn't understand it. If the expected result of a certain call to a function changes after the code change, it has broken the interface's expectation, even if the result was an error. SEMVER is about expectations, not interfaces. It surely was a breaking change, as it BROKE the previously known EXPECTATION.
I had a similar experience with another team introducing breaking changes as a “patch” update. It was infuriating and cost the company money. I personally think that teams that work on isolated projects should take ownership of the impact of their work (ie do more than unit test), or at a minimum clearly communicate changes to teams that consume their product.
What changed in that Workiva's dependency was not the interface, but the contract of a function. These two are not always the same. Default param values are a good example.
1:55 This is by far the most common thing I encountered as an off-shore engineer. Every single damn time, that I asked something to my senior, who is working on the other end, he would say: 'you should know this by now'. Then come to my code review with 'you made the wrong assumption'. Like wtf?
My favorite: 1) Write as much useless information as possible to the log-files. Use at least 15 lines or more, to document every event that happens. 2) Write a tiny line to the logs, upon crash, seg-fault, stack-overflow, etc ... Make sure it's well hidden between all other garbage in the logs. Don't use color coding! 3) Use a custom launcher, instead of the systems default one. Don't tell anyone! Make sure nobody notices, the actual program crashes and restarts in background once in a while! Took us 6 months to find the issue!
You could use prime numbers to intro use the unpredictability. Say, break the code when the length of a list equals a prime number over 100. You can hide the checks in a distributed manner where maybe you can argue that you need to factor the number into primes, and then later you check the number of factors that resulted. Then later you can introduce a bug where you check thay the number of factors is >1 rather than =>1. I think people are not really aware of triple digit prime numbers so it would take forever to pick up on the pattern.
On the series of preprocessor macros to introduce a subtle bug, it would be a really good exercise. If you can break and abuse code, you then know how not to break your code.
The point "Incur 400 hours of bug triage" reminded me of the article "Finding a CPU Design Bug in the Xbox 360". It was a bug caused by the speculative execution of the CPU and if you want to debug stuff, speculative is disabled. A real pain to find xD
I think 2 background threads is good. The danger with trying to use tons of threads is if you don’t perfectly await everything, you can fück the whole thing up and when is the cpu usually the bottleneck anyways? God forbid you multithread a ton of disk writing
I got to thank you for turning my attention to NeoVim and then me converting myself to NeoVIM, I went all in full retard mode and got TypeScript Angualr and Java Spring editing and debugging working. I no longer have to use 2 editors like VSCode and Inteiij. I am loving my NVIM experience and have been diving into learning everything I need to know to work more efficently, kepmaps and code navigation. I am hoping I don't run into any hiccups that would vex me down the road with things breaking on updates.
I am currently in a meeting that will last 3 hours and is planned every monday. There are over 70 engineers in the meeting. The meeting itself is about how to do this super specific thing in a tool that one of the dev teams developed and the other teams don't even have access to. And the language used to explain this thing, that none of us understands or will ever have to do, is riddled with domain specific knowledge and acronyms. I'm 100% sure, that most people in this meeting have no idea what it is about. that's easily over 210hours wasted, just this morning. I have 2 to 5 meetings planned every day this week.
Thanks for the insight on parallel chained processing! It seems a good ideia to make parallel streams of simples apps chaining on another! I'll defenitly dive a little on that one =D
One very very simple trick was missing here: If you have, let's say, 2h of total meeting time per day, you don't want to schedule those in one 2hour block in the morning/afternoon. What you want to do instead is spread them out in 4-5 small meetings with 30 minutes break in between because that's not enough time for most developers to get back into "the zone". This way you can easily waste 4-5 hours per day with only 2hours of meetings!
Your most evil bug needs to use floating point rounding error as the source of randomness. It looks innocent and has no external dependencies; the jank is baked into the silicon. Have two subtly different calculation that should obviously reach the same number, but they don't quite because of floating point error and instruction ordering. And the program works correctly except when the error term is in a very narrow band.
Just having meetings is not bad enough: Have the person that disagrees with everything write a meeting summary that is circulated among attendees and incites discussion about what was actually talked about and decided in the meeting. You can at least 2x the impact of meetings this way.
What you said about C pre-processor directives really hit home for me because I literally had a final exam today where one of the questions dealt with duplicating pre-increment operators inside a ternary statement to get an unexpected result. Yay college lol
19:22 This can preferably be executed by redefining NULL to be non-zero for the scope of a project header (between essential low level libraries and top level code) and then redefining it back afterwards, which will cause segfaults when if-statements stop catching "null" pointers. To avoid detection this "null" value should hold the address to itself (in the heap somewhere), then get randomly rewritten by other stuff in the code later, this needs to happen in an existing malloc'ed scratch disc area and be accompanied by 10k+ changed lines of code to minimize the chances of easily catching it.
Another option is to make the linter ignore your actual malicious file and keep a separate faux file just for linter purposes. This allows hiding really awful macros like "#define const" which strips const from variables, making them mutable, which can be abused in all sorts of ways, bonus points implementing this in multiple steps.
Hmmm, the parallels take, reminds me of what windoZe did with STA (single threaded apartment) stuff. Not sure if that was by design or laziness... While have to admit, the stuff i use to do with working on unix machines (haven't for a while), but loved the ability to have most program to pipe in and pipe out, the composition you could do was awesome... As for ineffectiveness... Create you test framework rely on or need to test historical data, but, make it so the QA environment is wiped clean, well data from prod i pushed into QA... Can't help thinking of the HIPPA and other security issues... 🙂 Also back in the daZe of my first jobs... Back when working on PC's one of the devs if he didn't like you would updated you startup batch file to load a TSR that randomly change characters you typed, and he was really mean, it kept track how much you typed, if you were typing a lot it would flip more, or if you hit the same key trying to debug, it would never flip and all random....
21:40 maybe make it depend on the device it’s run/built on. so one engineer runs into it and then asks someone to take a look and it works for them. or code that passes all the tests/runs fine on staging servers but then fails when run on the prod server
![*Next-door 10x Software Engineer* [FULL]](http://i.ytimg.com/vi/kKAue9DiHc0/mqdefault.jpg)
I recommend creating a random letter generator and deploying it so it inserts random characters on the entire code base... Immediately achieve -100x engineer status ♥
oh i love it
That's all under source control. It's even better when you corrupt datasets that are not tracked.
@@thomassynths made my day xD
Good codewars/leetcode question 😜
A lossy commit hook?
"The interface didn't change."
This is only true for the most naïve understanding of what interfaces are. They encompass more than just function signatures.
That bit about gnu parallel was dope! Thanks for sharing!
Yeah, with that logic you could replace every function body with an infinite loop.
It goes the other direction too though "you can't rely on arbitrary implementation details not to change between patches"
Came to say this!
@@gamekiller0123Thank you I laughed so hard at this
True, but I assume he meant "the (programmical) interface didn't change a single signature/line of code."
Dunno the first thing about Python, but I assume it has interfaces..
Semantic Versioning is about the public API not just "interfaces"
“I used to say that when being CEO at Apple wasn’t fun anymore, I’d quit. But now I’ve changed my mind - when being CEO isn’t fun anymore, I’ll just fire people until it is fun again.” -- Michael Scott, CEO of Apple, regarding the massive layoffs undertaken to solve the bozo explosion, a few months before he got fired for it
The final bozo
The bozo explosion lmao
19:55 When I worked at the NAIC we had an application that at one point started failing in production. It would fail 9 times, then succeed, then fail 9 times and then succeed. I traced it back to a dynamic SQL query where the code was interpolating an ID value into the SQL string and we had reached the point where the IDs were large enough to trigger a conversion to scientific notation. Unfortunately, in the conversion to scientific notation, one digit of precision was dropped, so if the last digit was non-zero, the query would fail. If the last digit happened to be zero, it would succeed. Good times...
I'm so sorry for your loss.
9/10 fails seems easy to repro failing 1/10 would of been better at causing chaos.
😂😂😂
Sounds like an exciting and dynamic workplace environment.
This is an amazing story. Perfect answer for the interview question about what's the most interesting problem/bug you've faced. Wish i had something similar
Regarding your semver story: defaults ARE part of the interface. If you change defaults, you're changing how someone has to call your function in order to get the same behavior. That's the definition of an interface change.
Agreed. Which is at least potentially a breaking change.
@@NathanHedglin If the function is public, changing a default is always a breaking change.
@@sullivan3503 idk about always. Depends on the default.
Rust changing how cargo uses the registries from cloning the whole registry to the sparse protocol as a default isnt really breaking. As all the intended uses of cargo would be fine with the sparse protocol.
(They are doing this soon)
develpers are just afraid of major versions, aah scary big number, we'll be on version 13 if we did this for every change! 😂
@@michaeletzkornmeanwhile, Chrome here on version 150 or something
I met an engineer like this. They promoted him to principal engineer. He doesn't write code, he just rejects everyone else's code and gets a nice bonus for it. Obfuscating requirements just makes all of that easier.
I worked with a guy like this as well, he always had a problem with everyone else's code and the boss really liked him for that. I thought he was really smart until I dug deep into the code base and had to dig through an ungodly amount of abstractions. It was so insane, whenever you added a new feature he expected you to add 20 layers of abstractions to something simple. It's almost as if he wanted to add extra layers of complexity that didn't need to be there so that A) He could sound smarter and B) he would always have job security if he's the only one that understands this over engineered code.
@@BillClinton228 its smart. Power perceived is power achieved. It's not what you know it's what you can prove.
The lead of my automation team accepts requirements for literally any one who will give them no matter what they are. Not tell any one the requirements were accepted. Not verify if things will be impacted by gathering random requirements from random people with random levels of knowledge. Only sometimes take down meeting notes, and only sometimes accurately. Then be amazed when everyone things the finished product is a worthless pile of crap that almost never runs correctly and that their back log is an endless pile of crap.
That’s what principals and even worse, fellows, do. They just tell you that you are wrong. They shit down all of your ideas. And eventually the company wonders why they haven’t had a new idea in 15 years
You could do this:
-have a case-insensitive language
- define NULL
- define null
Sometimes it will get corrected, sometimes it won't
Of course they should have different usages but vastly overlapping
Thanks, Satan.
I won't ruin the like count. But Like.
What the hell
wouldn't case insensitivity mean exclusive upcasing or exclusive downcasing?
#define struct union
#define else
If you do this, I will find you.
The closest thing I've seen to the perfect bug was in a geometric unit test with bad randomisation. It almost never failed when running tests locally, but in the CI pipeline it seemed to almost always fail on master (never on a branch). The fun thing was that rather than generating 10k random polygons, it would generate the same random polygon 10k times 😅 so you had to be really (un)lucky to reproduce it
That test one was soul crushing. I had a coworker who wrote some absolutely horrible test code, so that when I fixed an issue, the entire test file broke and complained. I had to completely rip out the mocking and re-do the file. Great fun!
Do the code and then do the test to test the code not the expected behaviour . then you test that nobody can solve bugs with confidence
I used to just CODE THE FIX (ie visibly and indisputably add value), then email the before and after code to whoever owns the broken code and all the obtuse tests, CC my boss and their boss, with "hey, I think there is a defect is your code because A should be B (attached). can you please confirm, update your tests, and commit ASAP as this is a production bug. kthxbai"
had similar experience - fixed a bug in one place, checked the tests for that part - everything fine, created PR - 27 tests in some random parts of the app failing because they were importing some bullshit that was indirectly tied to this bug as working code
I fully agree with your take on threading. I did this a while back on a brute force keyboard layout optimizer. I wrote incredibly optimized code for one thread, but I allowed the isolation of state resources in such a way I can just run it on however many threads I want without an issue. I got insane performance doing that
Dude I once had a job that had me work across a dozen different departments, each with their own rituals, scrum masters, traditions, and software. Every day I had to deal with a hodgepodge of different jira boards and weird other facilitation dependencies that I never care enough to learn the names to. I spent 3/4 of my time simply in pointless meetings, "social times", and trying to update my tasks across everything rather than actually developing and architecting software. Quit after 6 months. Had me wanting to go back to VBA development
As a VBA developer, I feel this in my soul. I will say that not coding in some other language makes hobby projects easier to not hate and it's retroactively made me look at how to use VBA fundamentally differently and now building projects in it is pretty rewarding compared to the crapware macros that dominate big businesses.
The 4:55-5:23 part. Man I have this right now. We got a "manager" who wants to make the most elegant software and behaves like an architect (I have not seen him write a piece of code). He goes around to everyone touting his ideas and every interaction goes like this: "I don't want to tell you how to do this, but I want to understand and ..." then proceeds to criticize you . And when we have a technical discussion he just pulls everyone into technical vortex of ideas that does not go anywhere, it's tiring to have any technical meetings with him. And my biggest issue is he knows a few things well, and in other things like Frontend/Infra (originally backend developer) he just goes full Dunning-Kruger effect. It's really toxic and he pushes us for to reach unnecessary deadlines (like for a meeting that had little to no importance). And we have already rewritten the backend because of him, next it's gonna be the Frontend or the Infra. :D
I am sad because my team is great, I am on a greenfield where we experiment, and figure a lot of things as we go and it's really fun it's just we have "Him".
yeah, that is always tough to have a bad manager. so many things break down due to management
Think outside the box, be creative, come up with fresh new ideas... but also I'm going to micromanage you and question every line of code that you write. And then when you become demotivated because you know that everything you do will be changed or rewritten they become annoyed.
Speaking of the true magic bug, I had that at work. Sometimes builds would fail because they included invalid code, but when running the build again it would work.
After weeks of having that bug pop up here and there, I finally found out that the Console Ninja extension was modifying files in the node_modules, and those files got compiled into the dev build as-is, breaking the build.
So if you want fun bugs, have a tool that subtly modifies the output binaries, and introduce fragility there. Now your engineers can't treat the compiler as a pure function, and will go crazy.
I dont need this level of paranoia
This is terrifying
That is so much more evil that what I said
js ecosystem is genius brainrot
On the subject of weird c causing random crash,there was the underhanded c contest.
The last entry winner used a dirty typedef tp change the type of the pointer of a function: if you compile the unit of code, it takes f32*; if you include the file it takes f16* (or the otherway around )
The compiler doesn't se a problem. The linker doesn't care.
And you got a really weird edge case were the program errors out
Take the logarithm of a random number between 0 and 1, and say it is negative. Every number in this interval has a negative log. But sometimes you'll get a number so close to zero, its log will be a huge negative number, causing an overflow.
Side note about the side note about the GNU Parallel, I definitely agree. What you're really thinking of is coroutines and asynchronous programming. Essentially every part of the program only needs to be concerned with itself and its connections. Then a separate "manager" controls everything. You don't have to try to control "this thread needs to be spawned" or "this process needs to do this now" or whatever, everything just works and it's the greatest.
The fear of experiencing that semver story is why we never bump dependencies without testing. Dependencies are pinned to exact versions, periodically updated in a batch, ran through integration tests, and then allowed to mature on test servers till the next release. We don't rush the process unless there's a CVE affecting our codebase.
yeah, its real tough :)
Get the f*** out of here with your mature, highly deterministic low-risk approach
The original research by Tom Demarco that referred to some engineers being 10x more productive than others had more to do with environment than skill, computer language or years of experience. The book "Peopleware" describes this. And it recommends people having private, quiet offices.
The current trend of putting a bunch of people in the same room, and hiring for super engineers, is opposite to what the research was proposing. This common misconception is similar to the one that made governments standardize on the waterfall model, whose original paper mentioned it as how to NOT develop software.
The trend for putting multiple in large open office spaces is maliciously deliberate. It's there so employee's feel the urge to be productive and not be ratted out by other employee's for goofing off/doing nothing.
prime rediscovering unix philosophy for multithreading
yeppers. but that is the thing. without developing tools for the CLI, you just don't know these things.
even if you knew the linux philosophy, it doesn't really set in until you get parallel's existence...
@@ThePrimeTimeagen Made my day as a long time Unix/Linux admin to see that. I have actually beaten our dev team''s in house code more times than I can count simply using standard awk(mawk is even faster)/grep/parallel routines, specifically on customer data dumps we need to massage. But that's not exactly their fault they were using Java and Python. Hard to complete with 40+ years of optimized code from some of the best computer scientists.
A note about Mawk. It might be one of the fastest text processors that ever existed. The developers even found ways to knock cycles of a lot common routines in the code. Its written in C and beat pure C in lines opened and closed speed form the standard routines.
I'm a data scientist. I was working in R on an old project, and learned that you could redefine the addition operator globally for the interpreter's session. Naturally, I redefined it to randomly add one to the result every now and then. You could fix it by restarting the interpreter session, but if you managed to sneak it into a script that got imported somewhere... :)
Can we talk about how this is literally also a playbook for extremely high job security for chaotic neutral types
The first thing you do with the indispensable employee is fire them. Eventually they will quit or get hit by a bus and you can't have that much risk in your company. Any competent person can untangle the work of an idiot.
@@bkucenski Haha good luck with that. That one employee is usually the only guy making this unique product / service technically possible and feasible, and finding another one like that is zero chance.
@@VincentPride1986 Nope.
@@bkucenski that's fine, only people with actual talent and intelligence understand that
@@VincentPride1986 You're not as clever as you think.
I believe the funny myth of 10x engineers appeared simply because of the existence of the 0.1x engineers (thanks to all those bootcamps and alike).
According to Jordan Peterson there are always 10% people in companies doing most of the work. Some of them just happen to be engineers.
@@edgarmedrano2562 you found a very strange "authority" to quote. Jordan peterson is a brain-dead quack, and pretty much everything he mumbles is an utter nonsense.
No, it appeared because of people like Fabrice Bellard (author of QEMU, FFmpeg, a PC emulator written in Javascript, TinyGL, QuickJS, the Bellard's formula, and many other things/awards/world records, etc.).
@@brinckau Curious that Fabrice Bellard is mentioned every single time 10x is discussed. Like he's the only example people can think about. A one in a billion genius (deservedly so). Yet, every freaking bodyshop with 20 employees boasts of hiring 10xers. There is only one Fabrice Bellard out there. There are no other people like him. Certainly not thousands, not hundreds, not even dozens.
@@vitalyl1327 Yes, there are very rare. People with incredible abilities are always very rare, whether the field is programming, math, sport, etc.
The distribution of bad/average/exceptional people follows a Gaussian curve.
parallel may have just revolutionized my workflow
you are welcome
I'm still a pretty young developer, only 2 years out of uni. I've seen some crazy bugs, let me tell you, and you can get away with a lot in a multi-language environment when, say, you're modifying C code and most developers work largely in Java.
Variable names containing a double underscore, or that start with an underscore followed by a capital letter, are reserved for the language / toolchain, but the compiler won't stop you. Back during a co-op, I had a crazy time figuring out why a build wouldn't work when updating the version info handshake from C firmware to a Java host when the version string variable in C looked like firmware__v1_2_3. That __v doesn't do what you might think. I'm sure you can find another sequence of characters that'll fail in a context-dependent way.
This last week I was struggling with an issue that didn't show up in our simulated environment, only on our real device hardware, and only for some systems. Turns out we had a linker script specifying the beginning of the stack was aligned to 8-bytes, but not the end... and a stack canary that was an int, and hence wanted to be aligned to a 4-byte boundary. An unrelated change fixed another invisible bug that led to the wrong linker script being used, so the stack was now in a different region of memory that it now shared with global variables. So the linker could just slip a 1-byte global variable in that little offset and kill the canary on startup. Before communications are established, even, so the device can't even send an error message.
The common theme with these two changes was that they were small parts of mega-merges. That was sadly missing from the overall -10x list -- get your team into a situation where there's a very large merge of death, so that finding bugs can't be done by just looking at small changelists.
The part about nonsensical dependencies hit home pretty hard.
Almost all the projects I've inherited has had this problem.
I've found lodash in projects where all it was used for was checking a few strings.
I've found form validation & component library combos that create a huge amount of extra code overhead to make work, and all it's been used for is validating that text is text and numbers are numbers.
RE: gnu parallel, I *definitely* recommend using a utility on top that will also take a list of colors and process names as input like so:
better_parallel \
-c 'red,blue,green' \
-n 'foo,bar,baz' \
do_foo \
something_something_bar \
hault_and_catch_fire_baz
output:
[_foo_]: hi
*[bar]:* starting
*[bar]:* stopping
*bar* exited with 0
[_foo_]: this
[`baz`]: wazzuuup
`baz` exited with 0
[_foo_]: is
[_foo_]: program
[_foo_]: bye
_foo_ exited with 0
And also an option to kill all processes when any one of them exits abnormally.
Trust me, it'll make you life nicer to be able to tell what's printing what.
For bonus points, get all dispatched commands to output to a different terminal window/tab.
Also you can
`npx concurrently`
but it's probably a lot slower than gnu parallel, but hey the superapps demo uses it 🤷
@@KnThSelf2ThSelfBTrue of course there’s a npm package for it 😂
I know nothing about gnu parallel, but your description makes me think "Can that possibly be setup to autofeed tmux so everything is it's own detachable session?"
@@shrikedecil I'm just going through man parallel_tutorial, and curiously this seems to be a built-in feature?
quote:
*A terminal for every job*
Using --tmux GNU parallel can start a terminal for every job run:
seq 10 20 | parallel --tmux 'echo start {}; sleep {}; echo done {}'
This will tell you to run something similar to:
tmux -S /tmp/tmsrPrO0 attach
Using normal tmux keystrokes (CTRL-b n or CTRL-b p) you can cycle between windows of the running jobs. When a job is finished it will pause for 10 seconds before closing the window.
I will help you with your quest for entering a heisenbug into your repository:
- Step 1: Use a language where you can query line numbers
- Step 2: Make an evil macro that only triggers when the usage site line number is in some range. Make it randomly inc some random mem location.
- Step 3: A thousand commits later, profit. You swell in pride when that perfectly sensible code some poor dev tries to merge into CI constantly fails. You will not be blamed.
___line___ in GCC for ... reference purposes
Sure, but easily detectible during a code review. It has to be something that doesn't seem out of place. Of course look at your peers and understand their skills. You have a better chance if you can convince them.
@@dealloc You can get some pretty gnarly macros (Boost preprocessor stuff might be a good example, I recall seeing some other sites talking about how to do branching / complex logic in macros). Stuff like using ## to append a conditional to a root, and then wrapping it in another macro to get the preprocessor to evaluate that root0 or root1 as another macro. It looks gnarly when it's implemented correctly, I'm sure you can slip in a conditional of `__LINE__ > 700` somewhere in the macro call chain.
As a bonus, this whole thing happens at compile time and is thus un-debuggable.
I got a language where you can query AND change line numbers at runtime: Commodore BASIC. With some POKEs and PEEKs, you could relink the order of the interpreted lines 😈, change them all to be line 0 🤪 and there were probably other creative hacks which I don't have on the top of my head right now.
I've had something similar happen by chance. It depended on some instruction's alignment with the cache... It worked *most* of the time, but some builds failed every time...
Thanks for heads up on parallel, i will add it as a part of my repertoire. For ad hoc file processing i find "fd -x" to be the most convenient. Runs in parallel
15:06 "Migrate your database frequently": That's very cost-effective wasting technique. Plus, you get a bonus combo with the "burn out" section by cycling this evil tactic in order to end up, after many migrations, with the exact same database spec the whole team started the project with. Just to let them realise how pointless their migrating efforts were. Then ask for migration again.
On every other Tuesday we have a day full of meetings. The morning is wasted with sprint planning (3 hours across 5 people), in the afternoon we have the retro which (1.5 hours across 5 people), then we have sprint planning 2 which is another 90 minutes where we assign the tasks... Finally on Mondays before the sprint planning we have the backlog refinement which is 3 hours (4 people) trying to come up with subtasks for things you can't even know yet. It drives me insane sometimes.
Sure, but who in their right mind pays full attention during backlog refinement? Half of SCRUM ceremonies are implied "work on a little shiny problem on the side, while pretending to pay undivided attention"-time. Unless of course you do it in person, in which case I hope you are on some strong antidepressants.
16 MINUTES AND YOU ALREADY GOT 1K VIEWS?????? Bro you deserve the attention ur content is great
my man! ty ty ty :)
things have been blowing up lately, and you have no idea how much i trust you
I think I need to work on becoming barely incompetent before I can try to truely reach the stars of true intense anti-competency.
To avoid easy reversion, make sure your bug uses wall-time somehow and set it to only start appearing in a few months
Nonono, You don't aim simply for random. That's not evil enough. You tie that trigger to a hash of the code itself so it is a "consistent" random bug. Reverting code should fix the bug. Nobody should be able to know WHY reverting code fixes the bug, but it always goes away with the revert... until it happens again.
Beautiful
Omfg pure satan
I heard "Use grep with parallel" and immediately looked it up. Huge game changer.
Edit: I love that the man page links to a youtube video and a cheat sheet instead of the usual barely explain what the tool doe appraoch of some other manuals (looking at you, dmenu).
My company's software has a lot do with timing, like dealing with when a store opens, closes, special holiday hours, etc. Some of our PHP tests had things like time("-4 hours") which caused the tests to fail when ran between 0:00-4:00 UTC. I discovered these bugs because I was the only one working at those weird hours and was wondering why super old tests were suddenly failing
Once upon a time I was working on this big old PHP codebase that had originally been written by a team completely oblivious to the fact timezones exist. And now it was being deployed to customers in multiple countries. So being young and keen I fixed a couple of egregiously broken timezone bugs. My reward was spending the next 18 MONTHS exclusively on datetime and timezone fixes....
I have such a desperate need for more in-depth spicy takes on agile/scrum/etc from prime
I'd recommend allowing devs copying and pasting randomly found solutions from stack overflow and elsewhere and then asking seniors to CR that. Even better without testing if that even works.
Also randomly naming variables and functions work wonders. Like using a for iterators, x for arrays, 'class' for function names, get for setters...
20:18 When python creates a new int from a literal, it creates a new instance of int to hold the information. But since ints are immutable, cpython actually pre-initializes and caches instances of int for numbers from -5 to 100 at startup.
Soo there is a global list of ints just sitting there in cpython, and it seems a cpython module could just... I dunno... reorder them. So that anywhere in your entire codebase that you assigned a literal 7 to a variable, it would store 8 instead.
This might not be a fair game to play with interpreted languages with extensions. It can reach the same level of debugging difficulty, but it's not as clever of a prank.
Java has both primitive ints, and immutable boxed object Integers. Using the == comparison operator works as you'd expect for primitive ints, but compares the memory address of boxed Integers. Turns out that Java caches at startup all the boxed Integers from -128 to 127, so an Integer == Integer comparison will "work" if you only write tests with numbers in that range. But the moment you try comparing Integers outside that pre-cached range, all bets are off.
I've actually seen this bug in production code...
I made that bug in production, avoiding my automated tests
@@maxwell-lt and that's why you use .equals in the wrapper class
Introduce side effects. This way it makes it easier to affect it from the outside which is harder to detect. Side effects can be implemented in a way that seem to be pure. Especially in languages that has lesser or no type systems.
Also floating point arithmetic can be a great way to obscure what's going on. Because no-one understands floating points unless they've read the specification (and understood every word of it).
Another is if you work with date and time, you can just implement something in a way that makes sense based on people's assumptions about how date and time works. That way the code seems to be true on the surface, but for the untrained eye, you can truly cause havoc. Also you can introduce randomness this way.
@@dealloc Both of these are excellent. I'd add in using floats as Booleans and as many implicit conversions between strings and other data types as possible.
One of my favorite ones is adding 40 layers of security theater on anything you do so even basic tasks take at least several steps from an user standpoint. "Go add this info to the wiki". Great, let me disconnect from our AWS VPN and connect to the internal VPN, that'll take some time, then I'll go log in to the wiki. Oup, wait, 2FA has to hit my phone give it a sec. Alright, now let's open the authenticator app... ah this code is about to expire, let me wait a few seconds for the next one... ah shit, the login page expired, let me reload and try again... ok, we're in, what was the internal project name for this thing again? Oh I lost access to this last week? Let me file a ticket to get it back, gonna log in to servicenow, give 2FA a second to hit my phone... repeat ad infinitum
Xerox PARC pioneered the bozo explosion thing. Their rule was you could not hire somebody unless they were as good as you or better. Steve probably picked it up there, along with the mouse and tablet.
I don't think documents are inherently bad when you're trying to communicate something to others, it's just that they're not always well made and reviewed, or they're used too much. I'd much rather read on my own time what is my colleague's plan for something important (and this is a big factor to the success of this; if you start this process for everything it will slow things down severely), leave a few a questions or comments and wait for him to respond on his own time, than try to sit down in a meeting and do it synchronously.
The trick with this is you have to start the planning a bit early, but the advantage of being allowed to marinate your thoughts on an issue are very serious.
19:20 - this actually happens in graphics programming. Some devs call it "Uber Shader", a rendering program that is created to handle every possible case, most of the time, by preprocessor macros. When you try to handle multiple platforms, stereo rendering (for VR), few different rendering techniques, you end up with a code that takes 30 minutes to understand one if statement.
Also import order matters, one macro can be enabled in one file, and then disabled in the next file. When trying to understand the code, you feel like a detective. Pin board and huge notebook is a must have :')
20:36 @ThePrimeTime
Ideas for the most insane bug factories:
- Make routine result depend on *available memory address* at runtime.
- Make routine result depend on *last measured call frequency* at runtime.
- Make routine result depend on *CPU load at* runtime.
- Make routines result depend on *network delay to answer* at runtime.
- Make routine result depend on *shared heap state at runtime.*
- Make all routines shared methods so their result depend on *their current caller.*
- Give all routines internal state so their result depend on *their previous calls* (aka make them silently programmable, my favourite).
- Make routine *infer* valid results *from errors* returned by sub-routines.
- Make routines *infer* valid parameter values *from non-valid parameters.*
In general, apply the "never fail to present some plausible result *no matter how* , *at all cost* ", just like the Russian army.
And of course, don't forget to create an _undecorate_ decorator, otherwise it's not fun enough.
I had a boss who enforced code reviews for all engineers, except himself. We'd come back from weekends and there'd be a dozen check-ins. Whole systems would be broken.
lol. This is hell
Complete narcissist
Do a math function that at some point will modulo using the current time in ms. You have a 1 in 1000 chance of doing modulo 0 - breaking your code.
I dont thing that they get to claim that this is how semver works. Changing the defaults is a breaking change, at least in this case.
If you really want a nasty bug, you can write your program in C and write 1 element past the max index of a vector. You can create bugs that disappear while debugging and only happen in production under certain conditions depending on your code and the compiler. BTW, error / crash will be in an unrelated area of your program. (you can also recreate this problem in javascript/typescript, although it is a bit harder to do)
Or even better, hack the allocs so that they always allocate a few bytes less than requested
Though i think valgrind can easily detect illegal memory accesses. So i reccomend tricking valgrind into believing that illegal accesses are totally legal
def change_random_var():
for variables in globals():
if random.random() > 0.001:
continue
globals()[variable] = "surprise!😂"
This function should (unless I wrote a bug in the yt comment) change a random global variable to a string 1/1000th of the time.
Function names change
Dunder variables change
Normal global vars change
Split this function up between 4 imports
BoZo explosion! Never heard that before. So true, lived through it, 5 years all went to shit. Company valued obedience and never let the capable ones have power/make-choices
the one about long builds distracting everybody is basically a summary of my workplace
Occasionally, brother sounds like Michael Scott and I loved it ❤
Haha, the part on holding a retro, hit me right in the feels.
The bug triage stuff is wild.
Sometimes you find bugs caused by dependencies doing something stupid in some way where you can't even tell where the source of truth is.
Recently was debugging an issue where tooltips overflowed the tooltip container on a graph. Graph was zingchart.
It was a custom html tooltip.
Zingchart generates the background of the tooltip in the SVG of the chart and puts the html tooltip outside the SVG and lines them up.
Took way too long to understand what the hell was going on there. And how to fix it.
Even worse there was no way to directly inspect the tooltip, since it was be removed if you loved the mouse or clicked.
I really like that talk about gnu parallel. Hope you can teach more about, to give some concrete examples
This list SCREAMS at me. Im in one of these groups, especially the busywork. If I have to sit through one more damn pointless meeting.....hold on...meeting...
we had a test that failed whenever the test was executed at .999 of a second - the setup would create an x second expectation but the actual was then the next second
(didnt take long to find tho , cause the moment i spotted asserts for dates i was looking out for it ^^)
19:06 "Sensitive to initial conditions" in continuous space + feedback loop: the recipe for garanteed chaos !
Working at a bank right now. The team I'm at right now is honestly pretty good, I'd say technical expertise/general knowledge is 9.9/10 and team cohesion is like a strong 8/10. But watching this video... You know, especially with the tests. There were some -10x engineers in this team previously, seriously. Some parts of the code are an unbelievable debugging nightmare, just insane
One of my former co-workers could write the book on this.
8:11 Multi-process. Threads went out the window a long time ago. A thread doesn't have stdio. A process does.
Evil C++ bug idea:
1. Make a utility "tool" that takes "references" to objects from a class that inherits from some virtual class, ideally a class written by someone else, they are going to be your fall guy.
2. Make it an overengineered mess to scare people away from attempting to understand anything about it or use whatever 3. obfuscation methods you want.
3. Sometimes, just sometimes, subtly overwrite the reference to the table that holds the references to the final implementations of their virtual functions, AKA the vtable, with a similar table that has a few functions switched around. Every object of a class that inherits from virtual classes has it's own reference to the vtable, so you don't need to do it for all the objects if you are able to take many as an argument. Try to choose the least called function out of all of them so the bug is less common.
Congratulations, you made someone else's code fail inexplicable!
At 4:00 who said "that's how SEMVER works" is wrong and doesn't understand it. If the expected result of a certain call to a function changes after the code change, it has broken the interface's expectation, even if the result was an error. SEMVER is about expectations, not interfaces. It surely was a breaking change, as it BROKE the previously known EXPECTATION.
With remote teams Retro is a great way to just get your turn to say you piece. I’m all for retros.
I remember getting yelled at for not having made any progress. I replied "because I'm in meetings all day asking about my progress!"
I had a similar experience with another team introducing breaking changes as a “patch” update. It was infuriating and cost the company money. I personally think that teams that work on isolated projects should take ownership of the impact of their work (ie do more than unit test), or at a minimum clearly communicate changes to teams that consume their product.
What changed in that Workiva's dependency was not the interface, but the contract of a function. These two are not always the same. Default param values are a good example.
I like how he glanced straight at the viewers after saying "the most idealistic engineer possible"
1:55 This is by far the most common thing I encountered as an off-shore engineer. Every single damn time, that I asked something to my senior, who is working on the other end, he would say: 'you should know this by now'. Then come to my code review with 'you made the wrong assumption'. Like wtf?
My favorite:
1) Write as much useless information as possible to the log-files.
Use at least 15 lines or more, to document every event that happens.
2) Write a tiny line to the logs, upon crash, seg-fault, stack-overflow, etc ...
Make sure it's well hidden between all other garbage in the logs. Don't use color coding!
3) Use a custom launcher, instead of the systems default one. Don't tell anyone!
Make sure nobody notices, the actual program crashes and restarts in background once in a while!
Took us 6 months to find the issue!
Dude I just found this channel. This content's so null I might be undefined.
You could use prime numbers to intro use the unpredictability. Say, break the code when the length of a list equals a prime number over 100. You can hide the checks in a distributed manner where maybe you can argue that you need to factor the number into primes, and then later you check the number of factors that resulted. Then later you can introduce a bug where you check thay the number of factors is >1 rather than =>1. I think people are not really aware of triple digit prime numbers so it would take forever to pick up on the pattern.
On the series of preprocessor macros to introduce a subtle bug, it would be a really good exercise. If you can break and abuse code, you then know how not to break your code.
The point "Incur 400 hours of bug triage" reminded me of the article "Finding a CPU Design Bug in the Xbox 360". It was a bug caused by the speculative execution of the CPU and if you want to debug stuff, speculative is disabled. A real pain to find xD
someone suggested making var names longer than whats actually checked by the compiler, so the vars have different names but actually are the same var.
what does that work with? basic?
This post is pure verbal sarcasm. And the mustachio totally got it. **sigh**
Appreciate what you do Barry, thanks so much.
I think 2 background threads is good. The danger with trying to use tons of threads is if you don’t perfectly await everything, you can fück the whole thing up and when is the cpu usually the bottleneck anyways? God forbid you multithread a ton of disk writing
I got to thank you for turning my attention to NeoVim and then me converting myself to NeoVIM, I went all in full retard mode and got TypeScript Angualr and Java Spring editing and debugging working. I no longer have to use 2 editors like VSCode and Inteiij.
I am loving my NVIM experience and have been diving into learning everything I need to know to work more efficently, kepmaps and code navigation. I am hoping I don't run into any hiccups that would vex me down the road with things breaking on updates.
I am currently in a meeting that will last 3 hours and is planned every monday. There are over 70 engineers in the meeting. The meeting itself is about how to do this super specific thing in a tool that one of the dev teams developed and the other teams don't even have access to. And the language used to explain this thing, that none of us understands or will ever have to do, is riddled with domain specific knowledge and acronyms. I'm 100% sure, that most people in this meeting have no idea what it is about. that's easily over 210hours wasted, just this morning. I have 2 to 5 meetings planned every day this week.
Thanks for the insight on parallel chained processing!
It seems a good ideia to make parallel streams of simples apps chaining on another!
I'll defenitly dive a little on that one =D
Defaults are part of the function signature, so yeah changing the default values is a change to the interface (contract or no contract)
I definitely came across some -x engineers in mechanical engineering. It's an amazing thing to witness.
Thank you for GNU parallel. What a slick idea
I am not even a dev
I find this funny as hell 🤣🤣🤣
This even better than standups..
The timeline of where Prime works feels like a Mendala effect.
That video ended up so abruptly. I don't even know the name of the guy to thank...
the name is...
One very very simple trick was missing here: If you have, let's say, 2h of total meeting time per day, you don't want to schedule those in one 2hour block in the morning/afternoon. What you want to do instead is spread them out in 4-5 small meetings with 30 minutes break in between because that's not enough time for most developers to get back into "the zone". This way you can easily waste 4-5 hours per day with only 2hours of meetings!
Would love a deeper dive on higher-order components. 😏 Just lots of emotional baggage to unpack there, would be pretty entertaining.
'You just got kicked in the agile right there' I laughed then i cried.
17:30 "The Bozo Explosion" - thank you, you just educated me. I felt that.
Your most evil bug needs to use floating point rounding error as the source of randomness. It looks innocent and has no external dependencies; the jank is baked into the silicon. Have two subtly different calculation that should obviously reach the same number, but they don't quite because of floating point error and instruction ordering. And the program works correctly except when the error term is in a very narrow band.
Just having meetings is not bad enough:
Have the person that disagrees with everything write a meeting summary that is circulated among attendees and incites discussion about what was actually talked about and decided in the meeting. You can at least 2x the impact of meetings this way.
Parallel makes the best one-liners ever.
What you said about C pre-processor directives really hit home for me because I literally had a final exam today where one of the questions dealt with duplicating pre-increment operators inside a ternary statement to get an unexpected result. Yay college lol
Whhhhaaaaatttt
19:22 This can preferably be executed by redefining NULL to be non-zero for the scope of a project header (between essential low level libraries and top level code) and then redefining it back afterwards, which will cause segfaults when if-statements stop catching "null" pointers. To avoid detection this "null" value should hold the address to itself (in the heap somewhere), then get randomly rewritten by other stuff in the code later, this needs to happen in an existing malloc'ed scratch disc area and be accompanied by 10k+ changed lines of code to minimize the chances of easily catching it.
Another option is to make the linter ignore your actual malicious file and keep a separate faux file just for linter purposes.
This allows hiding really awful macros like "#define const" which strips const from variables, making them mutable, which can be abused in all sorts of ways, bonus points implementing this in multiple steps.
Hmmm, the parallels take, reminds me of what windoZe did with STA (single threaded apartment) stuff. Not sure if that was by design or laziness... While have to admit, the stuff i use to do with working on unix machines (haven't for a while), but loved the ability to have most program to pipe in and pipe out, the composition you could do was awesome...
As for ineffectiveness... Create you test framework rely on or need to test historical data, but, make it so the QA environment is wiped clean, well data from prod i pushed into QA... Can't help thinking of the HIPPA and other security issues... 🙂
Also back in the daZe of my first jobs... Back when working on PC's one of the devs if he didn't like you would updated you startup batch file to load a TSR that randomly change characters you typed, and he was really mean, it kept track how much you typed, if you were typing a lot it would flip more, or if you hit the same key trying to debug, it would never flip and all random....
21:40 maybe make it depend on the device it’s run/built on. so one engineer runs into it and then asks someone to take a look and it works for them. or code that passes all the tests/runs fine on staging servers but then fails when run on the prod server
As someone who enjoys his weekly retro... PAIN