I've worked a Game Tester before, and everytime we receive a new build (AKA an in-development patch), we always had to test stuff in that order: - Regression Testing: confirming that bug fix attempts worked or not, and once in a while, seeing if other non-touched bugs were accidentally fixed - Coverage Testing: Making sure that the core stuff of the game is still functionnal. Using Smash as an example, you'd have people assigned on each game mode, each character, each item, etc. And for each of these, there's a checklist of things to check, such as every attack and special mechanic of a character. Usually there's at least one person whose job is to pretty much speedrun through the main story to make sure it's completable. - Destructive Testing: Given a portion of the game, trying to break it as much as possible with all interactions you can think of. IMO, this is where the true worth of a Game Tester starts. Some people just play around normally, but you find more important bugs by having a deep knowledge of the full game and understanding how stuff can work from a programming perspective. For example, Pirahna Plant's Down B makes a HUGE extension of his hurtbox, so there's definitely potential for bugs there, like having command grabs that teleport the grabber, or a well-timed attack causing Pirahna Plant to clip into the stage. And sometimes you need to think outside the box: Maybe Pirahna Plant keeps its large Hurtbox if the Switch enters Sleep Mode in the middle of the attack! - Ad-hoc Testing: Same as above, but now the tester is free to do what he wants. The issue I see is that sometimes, the game is super big and new builds are very frequent to the point that there's not much time for Destructive Testing. If too much focus is placed on Coverage Testing, you get to a point where your game is definitely completable without any issue, but you surely have bugs on the side that allow you to completely break the game. This could be one of the reasons why Sonic Boom ended up with its famous Infinite Jump exploit done by pausing mid-attack.
When I was a Game Tester, we didn't have Destructive Testing as it's own thing - that was more or less part of the ad-hoc testing and was dependent on whether or not the tester wanted to go that deep into testing compared to just playing the game. Personally, I loved really digging into the game and trying to really break stuff. I had a good handful of bugs that made other testers just go "WTF?!" 😆but a lot of these were edge cases where the average player probably wouldn't encounter it. Still the devs appreciated it and that kept me motivated. Also wanna give a shoutout to game testers that squash soooooo many bugs before the release date that the public wouldn't know.
@@splatbot8091 Depends on the company and probably the game as well. The company I worked for accepted anyone with a high school diploma and decent writing skills.
I mean, don't you already have at least 10 years of experience playing video games? :V I would say to look for 3rd party companies. The bigger your game is, the more testers you need, so it's not uncommon for developpers to hire 3rd party companies. And these companies want to recruit as many people as possible to meet the demand. So instead of asking for years of experience, they create a simple test to see if you're capable of finding bugs.
Back in the NES, SNES, early PS days we literally had to live with it or send the game back to the maker to have it replaced with a debugged copy, so being able to fix the bugs with a patch really is a huge change. I honestly have nothing but respect for the people debugging the huge games nowadays.
The games were a lot smaller and less complex back then as well! Also, without a patch to release, they literally needed to make sure that they fixed everything in order to not release a hot trash.
It actually bothers me greatly when people oversimplify how games were "back in the day" and you had games like SoulCalibur III frying memory data over the CotS mode. It's never been perfect and likely never will be.
@@RangoTheMercenary People need to realize that back the only option to get improved game post purchase was to either wait for a replacement/additional disc or nothing. While nowadays it still sucks to get a buggy game upon release, we at least have the option to know quite soon whether a downloadable update will come, and if it does, the loss is lesser.
I remember that the European version of Digimon World 4 for the PS1 had the worst type of bug, a door you needed to enter in order to continue the game didn't work. Yeah, the game was basically impossible to finish, and you had to live with that the rest of your life.
I often say this: Imagine a game have 100 testers working 40 hours a week for 52 weeks, you get 208 000 hours of testing in a year AT MOST across the entire game (vacations, day off, meetings, all would reduce actual testing time). A game releases, day 1 it sells 1 millions copies and 1 million players plays for 1 hour, that's 1 million hours of testing in the first hour of the game already. Testers do their best, but they can't catch everything!
I'm a programmer but mainly for commercial software than actual video game , This video is really an eyes opener to those who have to program video game. Sound like hell when you have to account for multiple character and interaction not just between character but arena and items. In commercial software there were bugs don't get me wrong but mostly we control what user can see and interact with so it mostly confine to certain amount and I already want to pull my hair out when I can't fix some bugs in commercial software so I gotta give credit to those video game devs.
It's also worth noting that automated unit testing is almost not a thing in games (as far as I'm aware). Factorio has unit tests, but that's the only one I know of.
@@duuqndnah it is for the studios that put more money into QA. A couple have been really expanding those teams last ~5 years too. They just can't do everything and in RPG or settings heavy games are pretty useless. It's more about using automation/bots to assist a person so one tester can test something that otherwise takes like 40 people. It also doesn't save money for video games quite the same way because so many variables you do need a team designing the automated tests more rapidly and with more novel solutions.
@@duuqnd The studio behind Talos Principle has a talk on how they do automated testing in their games. But it's not the same as unit testing, more of an integration or approval test.
One thing that differs commercial software bugfixing from videogames is that they required predictability, for the program to work as intended all times; while games need variance and multiple possible interactions, so there is not a single way of playing. Videogames can be made entirely of "buggy" code if you think about it.
A lot of people do need to realize this. I've seen huge games get made fun of if big bugs end up getting found, but they don't realize that the testers most likely never found the bug in the first place, like the duping glitches in TotK or that Rashid infinite tornado bug. The player-base is a much larger sample size for things to be found.
This is also why snall dev teams have Early Access releases and a _LOT_ of fighting games often try to do a limited weekend beta test. The number of testing hours from that totally eclipse the equivalent of the entire development team ONLY doing bug testing full time on that version for a solid year. QA is all about knowing where it's most likely to look and trying to make sure that bugs aren't common, but it's also difficult to know exactly how users break things until you let them. It's always appreciated having a bit more light shed on QA & Bug testing / fixing to the general public, 'cause it's a hard thing understand from the outside.
As a Disney’s Dreamlight Valley player, I’m surprisingly glad to have purchased Early Access of the game with some extra cool items for a few more bucks. I know Disney’s infamous reputation, but I have this feeling that the money I put to play the game will potentially help with some of the annoying glitches (or at least releases of some cool future features) …Though I know a lot of you guys will disagree and say otherwise
I want to shine a light on Ultrakill here, where the developer was not only aware of the bugs in his game but actually accounted for them and sort of made them unofficial gameplay mechanics (think shotgun parrying) or ways to reach secrets (slam storage) because they were actually in line with the feel of the game. Of course some stuff that was just unintended he did patch, but still.
I LOVE when former bugs evolve into legitimate gameplay mechanics and features. Minecraft's Creeper is a famous one. Also, I believe the Spy class in TF2 was based off of a bug. If you can't beat it, embrace it.
A few years ago I found out something interesting about one of my childhood games, Ratchet and Clank 3. The "Warp Pad" gadget always felt strange to me: it lets you place a warp point and then teleport to it later, but it's obtained extremely late and can only be used in fixed locations, so in practice it's a glorified key for two areas and never used again. It turns out they intended for you to be able to play warp points anywhere with this item, but realized this would require testing every possible place you could throw it in every single level, which would be a nightmare. So they has two choices: either cut it entirely, or heavily restrict it.
Reminds me of Castlevania: Aria of Sorrow's ability Black Panther: It's super speed as long as you hold the button. That's it. But it's given so late in the game, the only place you didn't explore is the final area, now unlocked by killing the boss guarding the ability. Watching AoS speedruns and randomizers I can see why: There are so many wall collision bugs you can exploit and more speed makes it very easy.
0:25 “Art is never completed, simply turned in.” As a computer science student, that talk about enbugging felt like such a call-out. I have so much respect for bug testers, as there’s simply never enough time to find every single way that your game can break. It must be exhausting trying to run through the same pieces of content to make certain that it works.
I was a game tester working at Sega of America in San Francisco. I tested Full Auto 2, a PS3 launch game that got pushed to "launch window." I personally logged over 300 bug reports, although to be fair, I was pushing myself to find as many as possible to break the QA record (which I did.)
I think non-game breaking big are fine, they're what make some game memorable. Something like OOT3D where they kept some popular bugs in when making it is really cool
I still remember being surprised when Super Mario 3D All Stars kept some of the glitches that I used to enjoy playing around with as a child. Sometimes it’s those types glitches that make the game more fun!
@@sirbrandalf7813that’s because they used a Japanese updated version of SM64. Which, c’mon, A) if you care that much about BLJ, you likely already have a version of the game where you can do it, and B) this was neat due to the fact that we finally got the Shindou edition overseas.
@@Space_Australian Wrong warp, Navi Dive, Child Link with a master sword, items anywhere, escaping kokiri forest trough Mido, Adult Link in deku tree and probably more, not all glitch have been ported, but it was the devs intention (as said in an iwata ask) to keep as much as possible to keep the spirit of the game alive. This is also why MM3D sucks compared to OOT3D, far too much was patched and changed
Then there are bugs that just kind of seem to happen at random. Remember the Pokemon X and Y Lumiose City save bug? Before it got patched, some people had no issue when they saved there while others basically bricked their games. They tested Lumiose City worked, and they tested saving worked, but that specific combination coupled with a bit of a randomness factor led to a huge issue that rendered some games unplayable. Yet as far as standard testing for bugs goes, it does pass all the initial checks one would make for it. It was just a hyperspecific example with a random factor added on. I do not envy the job of anyone who debugs and I have utmost respect for them.
This is the reason why devs would ask for logs/files or steps users/players did before leading up to the bug. That way, they can emulate it as best they can before they can do any actual fixing. They mostly can't fix what they can't replicate especially for something "seemingly" random. This is true for software dev in general.
@@RaiOkami Lol I can attest to that. Was a game tester, sometimes a random bug occurred that we couldn't repro. Still wrote up a report for the devs only to get reply back saying "please find steps to actually repro" because they also don't know what's happening from their end.
@@LightsJusticeZ Yep, often bugs like that are hard to fix even in normal software, but sometimes the developer has an idea based on the description of what happened, so always good to include them anyway. Having steps to reproduce is always better though.
Similar case in Metroid Other M (which I quite enjoyed from start to finish, Japanese version), where near the end of the game if you progressed to a certain point then backtracked to the previous save point it left a door permanently locked that you would have reached (and found unlocked) and been able to go through to progress the story... you had to restart your whole gameplay if that happened. Ultraspecific trigger that simply didn't occur during testing.
Yeah, it does. Gamers can be notoriously toxic towards devs, largely due to the lack of communication and the vast knowledge required to develop software. I'm glad Sakurai is doing his part in closing that gap.
@@gonderage I totally agree. As a developer myself, I understand that the general consumer feels that they don’t need to understand how hard game development is, and that’s to be expected, but I hope that more of them could at least have more empathy towards the profession.
Im also a dev-I think most of them do, there’s just some particularly noisy ones that don’t. I don’t think the beige between devs and that part of the audience will ever be built-they’ll just keep thinking we’re greedy and lazy and we’ll keep thinking they’re whiny and stupid lol
I wish Sakurai had made an important distinction when he said how much devs want to fix all bugs before releasing the game... maybe he doesn't want to burn any bridges or maybe it's because he primarily worked with Nintendo and doesn't have much personal history with the likes of Ubisoft and Activision, but it's important to note that, he's right when he says most developers are perfectionists at heart and don't want to release broken games to fix later... but many PUBLISHERS like Activision DO want this. Now, it's sad how so many people don't realize or understand that developers and publishers are two very very different things, but many times they also want two different things as well. Nintendo is a very developer-minded company as those in their higher ranks started out as devs or aren't hired unless they share a mindset with devs, but several other major Japanese companies and most western companies DON'T. Those publishers can and will force an unwilling development staff to just cut off finish-by dates early, and release a game that ranges from bug-riddled to completely broken. But he is right that developers usually have more pride in their work to want to do that, but publishers can override what they want at any time and tell them "just do what we said", and they have to. Just saying that this is a very important point worth clarifying.
"Everyone would rather have their game be perfect from the very beginning". THANK YOU! No one likes to deliver shitty products except those who get the money without actually working on the products
I've managed software in the past. I can relate to this video as the software gets more robust there is more room for bugs. We obviously do the best we can testing everything, but new bugs appear from fixing other ones and it's just such a headache.
Aunomua waged a scorched earth campaign against bugs in _Tears of The Kingdom_ for a full year and people still break the game in new ways on a regular basis. No code can anticipate everything every player in the world will try over potentially millions of years of combined playtime, thousonds of thouse being devoted specifically to finding faults whcih can be exploited. This isn't a testament to the fraigility of the code but the insane neutron-star-heavy load it's under. If you put just about anything, under a proverbial hydralic press, it's only a matter of time before it cracks.
While there are plenty of ways to break Tears of the Kingdom, in the vast majority of cases, you have to be actively seeking them out. The number of notable bugs you're likely to discover if you're not doing it on purpose is really quite impressively low in this game. And really, I think that's the most crucial part to have covered. If a bug is virtually impossible to come across without doing it on purpose, it's typically not really a big deal, unless it completely breaks the game somehow in a harmful way or impacts multiplayer balance.
@@laggalot1012 Except for that food effect duping glitch that could literally be achieved with normal gameplay. It's shocking that one passed QA considering how polished the rest of the game is.
What's so impressive is that the item duplication glitch and some quest progression checks are the only true bug I can think of. Pretty much everything else players think of as "bugs" are actually exploits of the game's intended physics.
Yeah. It's why devs were losing their minds over the things the game could do. It's a programming nightmare to have so many individual elements working together in such a seamless way. Despite the bugs people find it's still incredibly polished for the kind of game it is.
I was with 4 other game designers developing a student project, the scope of our 3d puzzle plateformer grew a lot, after that most bugs were happening because saturn and uranus aligned with gandalf who touched too much wood that day. I remember one where if you spin around too much then an issue in the rotation processing happens, triggers a gimball lock and makes the creature you possess walk on its head. It took me weeks to figure out why it was happening.
Love the animation here XD One of the issues I used to come across as a tester is bugs that have seemingly been solved resurfacing because a fix to another bug has caused further problems. So it is quite ridiculous to stay on top of.
My Turn Based Strategy game has a bug where sometimes you move someone and they snap back to the original location, as if they never moved. But it happens so rarely I can't reproduce it. Luckily it doesn't cost your movement so you can just redo the move. Mad respect to people who have to do this professionally.
I honestly feel humbled and have learned some more valuable lessons then I ever will learn from these videos, about bug fixes, how you can't possibly fix all bugs completely or the game will never release, and how VERY MASSIVE AND BIG bug patching can be on developers, and more. We should all give more respect and support to game developers, creators, ESPECALLY the people involved with patching the game and getting rid as many bugs as humanly possible. We salute you all!
I'd say I'm generally somewhat lenient on bugs, but at the same time, hearing this, I also think it's truly admirable when a game does release with minimal intrusive bugs. Take Tears of the Kingdom as a recent example. In like 200 hours, I've barely stumbled upon bugs at all, or at least ones that I could clearly distinguish as such. A bit of jank here and there with things moving in unexpected ways, but that's about it. In a game like that, where you can freely build your own objects and vehicles and have so many interactive elements, I think it's a true marvel that the game is as light on bugs as it is! That said, there are also games where you can barely go a few hours without stumbling upon obvious bugs or where it even just permeates the experience. I think that's the point where criticism is completely fair game. Obviously, we as players can't personally look into the development process to see what went wrong, but often in situations like this, you can at least tell it went wrong *somewhere.*
Oh yeah Tears is a GREAT example of insane levels of testing. Certain intended methods for Shrines like the moving a foldable bridge via wheeled platform across pools of lava would have definitely caused one of the joints to just fly into space in a good few other games
Some of these things I've heard before, but how extensive those combinations get, as well as grabbing and throwing, were things I hadn't yet considered.
Yes and no, it also depends on the way the game is developped, you could group stuff and have the whole group share properties. Like all the trees are technically the same Tree object with just visuals and colliders changed, it means you could test the all the trees as one Tree instead of having to test all the trees individually. To know if Tree as a group can be cut you'd just need to take 1 weapon that is allowed to cut elements from the Tree group and cut 1 tree instead of having to cut all the trees with all the weapons.
@@sebastienpautot Yeah but in that case you're just copy pasting a code for something that the player can't interact with. What if the player CAN interact with the trees and you give each tree type it's possible interactions? Sure, they might share some code, but they can't be too similar anyway, because if they were you are limiting the players freedom by making him interact with trees in the same way, even if the tree is aesthetically different for instance. However, take what I said as an hyperbole. Ofcourse you can give player freedom without making the game full of bugs. But it's just to point out that the more the player is free of doing, the more likely bugs might happen. I mean, videogames work in a PROGRAMMED environment, and it's impossible to program every possible out come that the free will of a human can come up with.
@@The_D_ray you don't need to program every interactions, you "just" have to let it happen and loosely program behaviours, it can be tough but it's possible. In a game I programmed called Posseslime, all the creatures had the same core, and then creatures had modules for movement, abilities and camera configuration. It allowed us to make and debug individual modules instead of having to debug each creature. Meaning we could debug and provide fixes for the jump ability and all creatures using it would benefit from it.
@@The_D_ray Yes but even if there are a lot of outcomes, in special cases such as emergent systems you only define how certain elements react to each other. So long as those cases work as expected, the the cases should subsequently work according to the laws of how the initial cases work.
what about developers that release buggy messes at launch to lauded critical reception and legions of fans that fix their game for free to the point that it's basically their entire business model like Bethesda? I respect developers that allow mods for their games, but Bethesda has been relying on an engine from 2002, with all of the oldness and creakiness it entails, and marketing themselves on their new features that frequently don't work or are misrepresented before launch. And all of this was only a problem when Fallout 76 came out because, as it turns out, making a multiplayer game on a single player focused engine, and having the bare minimum expectation of a multiplayer game that is functional at launch, was, in fact, too much for the big corporate developer to manage. I don't expect bug free, but I expect some effort, and I expect the game to at least be optimized to run on things that aren't new-gen consoles. (looking at you, Jedi Survivor...)
I'd say that depends on the amount of bugs present, while I do think the Team working on Fallout 76 worked as hard as they could, there is no doubt that Bethesda itself was pretty lazy with the game considering there is a 3 Hour video concerning nothing but bugs for that game. And of course it depends on the Studio as well if an Indie studio made Fallout 76 (and it cam out the exact same) for example it would be more forgiving.
@@fearingalma1550No "what abouts" here. Bethesda works extremely hard on their games for many years, and no one there actually says "the fans will fix it." The modders choose to "fix" what they can see, while introducing new issues, all the time. So no, that isn't at all the exception you thought it would be.
@@fearingalma1550 The stuff that modders "fix" are stuff that would be considered as typical flaws in any standard non-Bethesda video game. With non-Bethesda games, you can't "fix" flaws - even though flaws themselves are subjective a lot of the time - but with Bethesda games you can due to the fact that Bethesda is one of the few developers that actually supports their modding community, unlike Rockstar or something. The way you phrase it makes it seem like a Bethesda game has to be absolutely perfect in order for modders to not be able to "fix" anything, but that's impossible because no game can ever be perfect. There is always something one can "fix" or improve with any game ever made.
This sorta reminds me of that one Penny Arcade comic that mocked the reality show where one winner would be able to work as a professional video game tester, and instead of being paid to play video games normally he instead had to test every point of collision in a Barbie horse game.
Its also worth noting that oftentimes, many players LOVE bug. Take the guardian flings in breath of the wild where using bullet time can send them shooting up at mach 8. Thats an example of a bug people dont want fixed because of its history, practicality, and sheer fun levels. Or the - world in smb 1. That bug is straight up iconic.
I do business consultant type development work in my day-job and do hobbyist gamedev and I want to say THANK YOU. This is something I wanted shouted from the rooftops to the general consumer/YT Commentariat. Even games that you'd think are very well put together can be broken by speedrunners exploiting your physics system, a minor bug that's rarely seen or something else completely crazy the developers could not have intended or forseen (Like hacking your game with Koopa shells! There is no way the devs of Super Mario world could have known that! Or heck how would you even test for that!?) I do want to echo Sakurai's caveat @3:34 for all games, this doesn't excuse them of course and we should still do our best to minimize them but there is a level of "buginess" that is acceptable to the general public (and for the record thats basically minor/rare/unnoticed bugs, as in they dont affect gameplay much, almost never happen, and/or exist outside of normal play, I.E Boundary breaks or only happen when you get the game into a state that was never intended) if you can fix em then sure! Go for it! but again its a bit more complicated than that, especially for games as there's an absurd amount of interactions, but ultimately you have to pick your battles. Also guys don't name names in this one (per the caveat on paragraph 2), Mr Sakurai did ask us not to speak ill of any other developers on his channel in his first community post. Believe me I can list off at least five devs off the top of my head on this particular issue but that would distract from the point he's making.
Yeah, unit and integration tests can only take you so far! Especially with something as dynamic as a video game! Puzzle games are usually possible to have automatic tests for, but something like Smash would be nearly impossible.
Tech Lead from the games industry here: unit and integration tests are almost non-existent in the game industry for a very very good reason. The level of interaction that makes or breaks a system or a feature does need human testers. I know this sounds weird since in other kind of software it’s common wisdom, but it’s absolutely true.
Really appreciate this video for illustrating how tough it can be. Two additional notes that I try to remind folks who play games: 1) coders and QA are usually aware of most of the bugs players find, but we don't often get to decide what is a priority to fix, and there's never enough time to fix it all anyway. 2) even knowing the above is true, no amount of rigorous testing will compare against what 10000 players crashing into it on launch day will uncover. We do what we can with the time we get!
I understand the issue here Mr Sakurai. Being a dev myself, I'm fully aware of bug existence and also the issue of debugging it. Another thing worth mentioning is that bugs could be discovered one at a time. When you're about to release a patch, it's possible someone reported another bug which could delay the patch release. So you have to choose to release patch to solve couple bugs soon or major patch to solve many bugs at once later.
As a software developer for a major tech company, I can relate to this so much. Bugs are just as much of an issue in software development as they are in game development.
Another fantastic video from Sakurai! I have tremendous respect for the game developers who have to deal with getting rid of as many bugs as possible before release. Hopefully, this video and Sakurai's information will help dispel the misinformation. As Sakurai confirmed, every game developer wants to release a perfect game. They do not just want to push a game out the window and fix it later with patches, but the large scope of the games makes it impossible to make a game completely bug free. Kudos to the developers spending extra time on the game after it goes gold who continue to test and make content for the game to release with patches and updates. :)
Nowadays a lot of the testing has to be automated for that specific reason: as Sakurai said, it's literally impossible to have persons testing all combinations again and again. Every time there is a change, the cycle repeats. Automated testing alleviates some of that repetition, but it's still complex to imagine all those scenarios and build tests for them. AI is already helping in that area and it should make all this testing way easier and faster in the near future.
Automated testing is great, though an additional challenge in game development is that a lot of the features are experimental in nature with stuff being added, scrapped, or radically changed on a frequent basis, so a lot of automated tests will become outdated almost faster than you can write them. It's typically limited in its ability to find new bugs too, being mostly useful for regression testing. Determining if something new is a bug generally comes down to someone making a judgement call about what the expected behavior should be, which is inherently impossible to automate.
I’ve witnessed a variety of weird glitches, especially in games you’d think have all the glitches pointed out by blogs and media, but it turns out there’s more that’s hidden very well throughout time. My personal experience was when my Go Vacation (Wii) data got corrupted, I’m still not sure what went wrong, but it definitely made me delete my beloved Mii character just to play it safe. And then I’ve seen a streamer accidentally find a glitch that nobody’s ever seen or at least reported before in Super Metroid (SNES). Those are only a few examples of glitches that nobody’s ever reported or analyzed yet, and I have plenty more examples in my experiences of playing or watching a stream.
As someone with over 12 years in QA -- This is an awesome video :) Another stat to consider, for example, combining all the hours tested over the whole life cycle of Smash Bros Ultimate... Well, at the moment, Smash Bros Ultimate sold 31.7M units. If those players play for just 1 hour on release, that playtime already completely trumps anything the QA team has put in! It is simply impossible nowadays to check every little thing players might do when games are as complicated as they are.
If you think old games were free of downright criminal bugs, check out what happened with Cinder in the first Killer Instinct arcades. They had to manually remove and put new arcade machines (or at least chips, I think?) Into every one in the states or something, it was crazy
Excellent insight, as always. People are so quick to blame devs nowadays for bugs. I think out of "class" Sakurai didn't mention how corporate rushing a game is also a big component of them having bugs despite developers wanting more time
Some bugs/glitches really do occur under extremely rare situations. Imagine if, in a Pokemon game, there's some kind of bug that can happen if you (1) return to a specific zone after (2) Flying away from an NPC in that zone on (3) the _exact frame_ that they try to initiate a Trainer battle with you, and (3) battling a very specific Trainer in a different zone instead of them....
(for those of you who don't follow Pokémon, this is a description of a generation 1 glitch that allowed you to catch a pokémon that was otherwise specific to an event)
I truly respect bug catchers in game dev Q&A. What I don't respect is games that cone out TRULY and utterly bugged like Mass Effect Andromeda where even the simplest mistakes were overlooked and released for full priced. Last ten really had some stinkers when it came to releasing games on time with little to no bugs
Testing for bugs is one of the things you have to do when testing your game. But depending on how big the project is, it's easy to get pretty overwhelmed.
If it's nothing game-breaking and easily resolvable, I like messing around with bugs and glitches. In a way, it adds to the charm of a game, as most bugs end up being unique to the game itself due to the circumstances it's found in. Whether you use the glitch to your advantage or not is up to you, but it's like another option you can use to change up your gameplay, even if by a little bit.
There are certain games where bugs are absolutely riddled throughout the game, and when there are too many, it can become a problematic game to play. (Look at Cyberpunk 2077 for example.) But then there are games where it seems pretty much solid with very few bugs at all; but when you do find one, I feel it's pretty cool. Almost like you've found something new the game has to offer. That goes especially for a game you could have played for a long time.
Unmentioned is work distribution - assigning a developer to: Detect a bug, reproduce a bug, investigate cause of a bug, fix a bug, make sure the fix broke nothing else - means that developer is not working on something else. He might also spend hours on a bug and never find cause of even reproduce it. My favourite bug from Factorio was caused by neutrinos from the Sun causing rare change in specific graphic cards memory, making it that some internal value would become something the game couldn't interpret - Try discovering that!! Rare occurrence on rare hardware :D
The best advice I can give to game developers looking to find bugs in their game is “don’t play your game properly.” I was once exhibiting a game of mine on an iPad and a toddler walked up, mashed the screen a bunch, and crashed the game. So in that instance, when I triggered a menu transition, I would patiently wait for it to complete. Don’t assume your players will do the same. Mash buttons, quickly and randomly. Tap screens all over the place. Try to initiate and cancel something multiple times in a row. You’d be surprised how quickly things go wrong unless the programming is really rigorous.
Not only are there gameplay bugs, but there are also bugs that causes crashes, or even damage the console/computer. Those get much higher priority for bug check and will consume as much resources as they need because recalling disks and having that reputation follow you around are both incredibly expensive
One of my fondest memories was discovering the Jigglypuff Final Smash bug/glitch when you use it on Bridge of Eldin which makes it so she stays at her final smash size. It required specific timing, specific stage and specific characters and items. I can't imagine any one dev during the time would of been crazy enough to test for that.
You best video to date Story Time! Abeit pretty short but I happen to find this rare bug in Sonic Mania of all things that if you save replays in all zone with the main trio and going beyond it with mighty & ray you'll see that you'll lose your ability to save and record new records for online can't erase it or erase the whole TT to get it working which leave starting the game from scratch (technically) was hoping for a fix but by the time I've report it, it was already too late 😓sega and whitehead's team shook hands and when on with their day still remainds to this day🙁 So moral of this story: don't ignore the rare bugs because it can be game breaking in something you won't expect seen a few game devs tackle these years after release yet sega won't let them fix just the one so to say that rare bugs shouldn't be ignored would be the understatement of the decade to say the least.
From my personal experience, at least, in a case like Smash Bros. where there's a million different potential conflicts, the only thing to do is to test each move individually and ensure everything works as intended in a vacuum before proceeding - and I'm sure Bandai Namco does this too. It's just that Ultimate's scale is so astronomical that even with incremental testing and slowly working up to bigger and bigger scopes, when you get to some of the higher level scopes you just can't catch everything. But regardless, if you work up in terms of scope when testing, it should help a lot. As a solo developer (with very little playtester access), my "favorite" instance of bug testing is any sort of deterministic replay system. You have to figure out through basically brute force what states you need to save to a file, which ones can (or need to) be inferred via context, or - worst of all - which parts of your system were developed with a non-deterministic approach or other bugs in the first place, and weed those out first.
It's stuff like this why I refrain from blaming individual employees for bugs and questionnable level design. The only thing that can get me to write an email is if a bug renders continuing impossible by any means. *Glares at NIS Classics*
3:30 "explain, don't excuse", are words to live by. It's not about absolving responsibility, just about making sure people don't need to repeat your mistakes Personally, I dont mind bugs in games that much, so long as they're relatively "stable" bugs. That is, bugs that show up under specific circumstances, and dont cause a soft lock or a similar degree of interruption. Obviously it's better to not have bugs, but there's a clear difference in impact between bugs that a player will often need to search for, and bugs that a player will struggle to avoid
I legit didn't realize that that was how bugs worked... when you put it that way, no wonder bug testing seems kinda stressful :'D I agree with most of the comments here tho, as long as the bugs don't break the game or make it legitimately difficult to play, it's fine. Perfection is only a concept, after all.
Some games definitely release buggier than others, though, and you only get one chance to make a first impression. There's been plenty of games that were train wrecks at launch, only to get fixed later, that are still remembered largely for their bugs.
It's not just multiplication; it's exponential growth. Literally (2^n)-1 potential bugs for n elements. And, as indicated with the Stage Builder example, custom content can draw out bugs that wouldn't manifest otherwise. And even if jerk publishers aren't part of the equation, a buggy early release may be necessary in order to make the money that can keep one well and working, and that itself is a gamble for an obvious reason. It's no wonder games rarely have their own engines anymore in lieu of preexisting ones.
I do like this topic on bugs in games cause it tells us on how it can work though also not work, and lets be real without some bugs we wouldn't have great moments like Roach, 21 second Ocarina of time, or others like it, though I will also point out that games shouldn't come out if it looks broken just everywhere like how Cyberpunk 2077 came out which they learned and have fixed it up now over time though another game I will praise is Baldur's Gate 3 which started in early access and asked fans how they could improve it over it's development.
The mention of "enbugging" reminds me of Dead by Daylight's issue with every patch they release. They fix a major bug, then more bugs start showing up.
QA testers would love to enjoy the games they test as much as any customers do, but they need more sanity than anyone to make sure they don't end up feeling numb over it due to their job. It's also a thankless job, so make sure you thank them.
While of course it would be nice if there were just zero bugs, I'll always respect developers who work to fix any bugs players encounter that negatively impact the experience.
![Games During My Childhood [Grab Bag]](http://i.ytimg.com/vi/mDVaOyzO5pY/mqdefault.jpg)
![Games During My Childhood [Grab Bag]](/img/tr.png)
![Development Secrets of the Original Kirby [Grab Bag]](http://i.ytimg.com/vi/ZUY2AtBD6Sk/mqdefault.jpg)
![Odds and Ends of Supervising [Graphics]](http://i.ytimg.com/vi/wKQ2v9kno9I/mqdefault.jpg)
![More Game Essence Means Less Broad Appeal [Game Essence]](/img/n.gif)
Lol the game over screen from Brawl
With the Subspace Emissary Logo!
And the announcer saying “Continue?”
I've worked a Game Tester before, and everytime we receive a new build (AKA an in-development patch), we always had to test stuff in that order:
- Regression Testing: confirming that bug fix attempts worked or not, and once in a while, seeing if other non-touched bugs were accidentally fixed
- Coverage Testing: Making sure that the core stuff of the game is still functionnal. Using Smash as an example, you'd have people assigned on each game mode, each character, each item, etc. And for each of these, there's a checklist of things to check, such as every attack and special mechanic of a character. Usually there's at least one person whose job is to pretty much speedrun through the main story to make sure it's completable.
- Destructive Testing: Given a portion of the game, trying to break it as much as possible with all interactions you can think of. IMO, this is where the true worth of a Game Tester starts. Some people just play around normally, but you find more important bugs by having a deep knowledge of the full game and understanding how stuff can work from a programming perspective. For example, Pirahna Plant's Down B makes a HUGE extension of his hurtbox, so there's definitely potential for bugs there, like having command grabs that teleport the grabber, or a well-timed attack causing Pirahna Plant to clip into the stage. And sometimes you need to think outside the box: Maybe Pirahna Plant keeps its large Hurtbox if the Switch enters Sleep Mode in the middle of the attack!
- Ad-hoc Testing: Same as above, but now the tester is free to do what he wants.
The issue I see is that sometimes, the game is super big and new builds are very frequent to the point that there's not much time for Destructive Testing. If too much focus is placed on Coverage Testing, you get to a point where your game is definitely completable without any issue, but you surely have bugs on the side that allow you to completely break the game. This could be one of the reasons why Sonic Boom ended up with its famous Infinite Jump exploit done by pausing mid-attack.
When I was a Game Tester, we didn't have Destructive Testing as it's own thing - that was more or less part of the ad-hoc testing and was dependent on whether or not the tester wanted to go that deep into testing compared to just playing the game. Personally, I loved really digging into the game and trying to really break stuff. I had a good handful of bugs that made other testers just go "WTF?!" 😆but a lot of these were edge cases where the average player probably wouldn't encounter it. Still the devs appreciated it and that kept me motivated.
Also wanna give a shoutout to game testers that squash soooooo many bugs before the release date that the public wouldn't know.
Thanks for the insight!
Literally how does one become a tester though. All places require YEARS of experience on a supposedly entry level role
@@splatbot8091 Depends on the company and probably the game as well. The company I worked for accepted anyone with a high school diploma and decent writing skills.
I mean, don't you already have at least 10 years of experience playing video games? :V
I would say to look for 3rd party companies. The bigger your game is, the more testers you need, so it's not uncommon for developpers to hire 3rd party companies. And these companies want to recruit as many people as possible to meet the demand. So instead of asking for years of experience, they create a simple test to see if you're capable of finding bugs.
Back in the NES, SNES, early PS days we literally had to live with it or send the game back to the maker to have it replaced with a debugged copy, so being able to fix the bugs with a patch really is a huge change. I honestly have nothing but respect for the people debugging the huge games nowadays.
The games were a lot smaller and less complex back then as well! Also, without a patch to release, they literally needed to make sure that they fixed everything in order to not release a hot trash.
It actually bothers me greatly when people oversimplify how games were "back in the day" and you had games like SoulCalibur III frying memory data over the CotS mode. It's never been perfect and likely never will be.
@@RangoTheMercenary People need to realize that back the only option to get improved game post purchase was to either wait for a replacement/additional disc or nothing. While nowadays it still sucks to get a buggy game upon release, we at least have the option to know quite soon whether a downloadable update will come, and if it does, the loss is lesser.
I remember that the European version of Digimon World 4 for the PS1 had the worst type of bug, a door you needed to enter in order to continue the game didn't work. Yeah, the game was basically impossible to finish, and you had to live with that the rest of your life.
Although on occasion some bugs made the games better lol
As a former in house QA tester for game dev, Sakurai isn’t exaggerating when it comes to dealing with “add-on” bugs
I often say this: Imagine a game have 100 testers working 40 hours a week for 52 weeks, you get 208 000 hours of testing in a year AT MOST across the entire game (vacations, day off, meetings, all would reduce actual testing time). A game releases, day 1 it sells 1 millions copies and 1 million players plays for 1 hour, that's 1 million hours of testing in the first hour of the game already.
Testers do their best, but they can't catch everything!
I'm a programmer but mainly for commercial software than actual video game , This video is really an eyes opener to those who have to program video game.
Sound like hell when you have to account for multiple character and interaction not just between character but arena and items.
In commercial software there were bugs don't get me wrong but mostly we control what user can see and interact with so it mostly confine to certain amount and I already want to pull my hair out when I can't fix some bugs in commercial software so I gotta give credit to those video game devs.
It's also worth noting that automated unit testing is almost not a thing in games (as far as I'm aware). Factorio has unit tests, but that's the only one I know of.
@@duuqndnah it is for the studios that put more money into QA. A couple have been really expanding those teams last ~5 years too. They just can't do everything and in RPG or settings heavy games are pretty useless. It's more about using automation/bots to assist a person so one tester can test something that otherwise takes like 40 people. It also doesn't save money for video games quite the same way because so many variables you do need a team designing the automated tests more rapidly and with more novel solutions.
@@duuqnd The studio behind Talos Principle has a talk on how they do automated testing in their games. But it's not the same as unit testing, more of an integration or approval test.
One thing that differs commercial software bugfixing from videogames is that they required predictability, for the program to work as intended all times; while games need variance and multiple possible interactions, so there is not a single way of playing. Videogames can be made entirely of "buggy" code if you think about it.
It probably helps being restricted to 2D and not having "freedom" as a selling point.
Imagine if every program worked in the vein of psdoom instead.
A lot of people do need to realize this. I've seen huge games get made fun of if big bugs end up getting found, but they don't realize that the testers most likely never found the bug in the first place, like the duping glitches in TotK or that Rashid infinite tornado bug. The player-base is a much larger sample size for things to be found.
This is also why snall dev teams have Early Access releases and a _LOT_ of fighting games often try to do a limited weekend beta test. The number of testing hours from that totally eclipse the equivalent of the entire development team ONLY doing bug testing full time on that version for a solid year.
QA is all about knowing where it's most likely to look and trying to make sure that bugs aren't common, but it's also difficult to know exactly how users break things until you let them.
It's always appreciated having a bit more light shed on QA & Bug testing / fixing to the general public, 'cause it's a hard thing understand from the outside.
As a Disney’s Dreamlight Valley player, I’m surprisingly glad to have purchased Early Access of the game with some extra cool items for a few more bucks. I know Disney’s infamous reputation, but I have this feeling that the money I put to play the game will potentially help with some of the annoying glitches (or at least releases of some cool future features)
…Though I know a lot of you guys will disagree and say otherwise
@@princesspixel3151That's fine, but a big company like Disney could also afford to do a free public test instead of charging for early access.
I want to shine a light on Ultrakill here, where the developer was not only aware of the bugs in his game but actually accounted for them and sort of made them unofficial gameplay mechanics (think shotgun parrying) or ways to reach secrets (slam storage) because they were actually in line with the feel of the game. Of course some stuff that was just unintended he did patch, but still.
SCRONGBONGLED comes to mind here.
I LOVE when former bugs evolve into legitimate gameplay mechanics and features. Minecraft's Creeper is a famous one. Also, I believe the Spy class in TF2 was based off of a bug.
If you can't beat it, embrace it.
Another example of this is skiing from Tribes. It was originally a physics bug, but it was so much fun they made it a core gameplay mechanic.
@@Amins88The double cherry from 3D World, hecc, the very idea of a FIGHTING GAME COMBO technically started out as a glitch.
@@Amins88Sometimes it's just better to embrace the fun bugs!
A few years ago I found out something interesting about one of my childhood games, Ratchet and Clank 3. The "Warp Pad" gadget always felt strange to me: it lets you place a warp point and then teleport to it later, but it's obtained extremely late and can only be used in fixed locations, so in practice it's a glorified key for two areas and never used again. It turns out they intended for you to be able to play warp points anywhere with this item, but realized this would require testing every possible place you could throw it in every single level, which would be a nightmare. So they has two choices: either cut it entirely, or heavily restrict it.
Reminds me of Castlevania: Aria of Sorrow's ability Black Panther: It's super speed as long as you hold the button. That's it. But it's given so late in the game, the only place you didn't explore is the final area, now unlocked by killing the boss guarding the ability. Watching AoS speedruns and randomizers I can see why: There are so many wall collision bugs you can exploit and more speed makes it very easy.
0:25 “Art is never completed, simply turned in.”
As a computer science student, that talk about enbugging felt like such a call-out.
I have so much respect for bug testers, as there’s simply never enough time to find every single way that your game can break. It must be exhausting trying to run through the same pieces of content to make certain that it works.
Never has a thumbnail perfectly captured what it's like to fix bugs during game development
I was a game tester working at Sega of America in San Francisco. I tested Full Auto 2, a PS3 launch game that got pushed to "launch window." I personally logged over 300 bug reports, although to be fair, I was pushing myself to find as many as possible to break the QA record (which I did.)
I think non-game breaking big are fine, they're what make some game memorable. Something like OOT3D where they kept some popular bugs in when making it is really cool
I still remember being surprised when Super Mario 3D All Stars kept some of the glitches that I used to enjoy playing around with as a child. Sometimes it’s those types glitches that make the game more fun!
@princesspixel3151 they took out BLJ though, which is disapointing. Speedruns can be super interesting!
@@sirbrandalf7813that’s because they used a Japanese updated version of SM64. Which, c’mon, A) if you care that much about BLJ, you likely already have a version of the game where you can do it, and B) this was neat due to the fact that we finally got the Shindou edition overseas.
Name one "popular bug" that didn't get removed in OoT3D
@@Space_Australian Wrong warp, Navi Dive, Child Link with a master sword, items anywhere, escaping kokiri forest trough Mido, Adult Link in deku tree and probably more, not all glitch have been ported, but it was the devs intention (as said in an iwata ask) to keep as much as possible to keep the spirit of the game alive. This is also why MM3D sucks compared to OOT3D, far too much was patched and changed
Then there are bugs that just kind of seem to happen at random. Remember the Pokemon X and Y Lumiose City save bug? Before it got patched, some people had no issue when they saved there while others basically bricked their games. They tested Lumiose City worked, and they tested saving worked, but that specific combination coupled with a bit of a randomness factor led to a huge issue that rendered some games unplayable. Yet as far as standard testing for bugs goes, it does pass all the initial checks one would make for it. It was just a hyperspecific example with a random factor added on.
I do not envy the job of anyone who debugs and I have utmost respect for them.
This is the reason why devs would ask for logs/files or steps users/players did before leading up to the bug. That way, they can emulate it as best they can before they can do any actual fixing. They mostly can't fix what they can't replicate especially for something "seemingly" random. This is true for software dev in general.
@@RaiOkami Lol I can attest to that. Was a game tester, sometimes a random bug occurred that we couldn't repro. Still wrote up a report for the devs only to get reply back saying "please find steps to actually repro" because they also don't know what's happening from their end.
@@LightsJusticeZ Yep, often bugs like that are hard to fix even in normal software, but sometimes the developer has an idea based on the description of what happened, so always good to include them anyway. Having steps to reproduce is always better though.
Similar case in Metroid Other M (which I quite enjoyed from start to finish, Japanese version), where near the end of the game if you progressed to a certain point then backtracked to the previous save point it left a door permanently locked that you would have reached (and found unlocked) and been able to go through to progress the story... you had to restart your whole gameplay if that happened.
Ultraspecific trigger that simply didn't occur during testing.
This one felt more like a lesson for the customers than for students! lol
Yeah, it does. Gamers can be notoriously toxic towards devs, largely due to the lack of communication and the vast knowledge required to develop software. I'm glad Sakurai is doing his part in closing that gap.
@@gonderage I totally agree. As a developer myself, I understand that the general consumer feels that they don’t need to understand how hard game development is, and that’s to be expected, but I hope that more of them could at least have more empathy towards the profession.
Im also a dev-I think most of them do, there’s just some particularly noisy ones that don’t. I don’t think the beige between devs and that part of the audience will ever be built-they’ll just keep thinking we’re greedy and lazy and we’ll keep thinking they’re whiny and stupid lol
bridge will ever be built*
@@gonderage Don't forget that we are paying customers. They are paid to do a job and to provide a product.
I greatly appreciate the number of people panicking in this video. Those drawings are so cute and hilarious. XD
Love and respect to all playtesters finding bugs and coders fixing them!
I wish Sakurai had made an important distinction when he said how much devs want to fix all bugs before releasing the game... maybe he doesn't want to burn any bridges or maybe it's because he primarily worked with Nintendo and doesn't have much personal history with the likes of Ubisoft and Activision, but it's important to note that, he's right when he says most developers are perfectionists at heart and don't want to release broken games to fix later... but many PUBLISHERS like Activision DO want this.
Now, it's sad how so many people don't realize or understand that developers and publishers are two very very different things, but many times they also want two different things as well. Nintendo is a very developer-minded company as those in their higher ranks started out as devs or aren't hired unless they share a mindset with devs, but several other major Japanese companies and most western companies DON'T. Those publishers can and will force an unwilling development staff to just cut off finish-by dates early, and release a game that ranges from bug-riddled to completely broken.
But he is right that developers usually have more pride in their work to want to do that, but publishers can override what they want at any time and tell them "just do what we said", and they have to. Just saying that this is a very important point worth clarifying.
Yeah basically. Though why anybody still supports Activision after literal human rights violations is beyond me.
(Also hi been a while)
1:23 The Subspace Emissary reference absolutely killed me. 🤣
"Everyone would rather have their game be perfect from the very beginning". THANK YOU!
No one likes to deliver shitty products except those who get the money without actually working on the products
Define “perfect” for this topic?
@@princesspixel3151you don’t encounter game-breaking bugs every 3 seconds.
@@princesspixel3151 Completely free of any potential bugs.
I've managed software in the past. I can relate to this video as the software gets more robust there is more room for bugs. We obviously do the best we can testing everything, but new bugs appear from fixing other ones and it's just such a headache.
Aunomua waged a scorched earth campaign against bugs in _Tears of The Kingdom_ for a full year and people still break the game in new ways on a regular basis. No code can anticipate everything every player in the world will try over potentially millions of years of combined playtime, thousonds of thouse being devoted specifically to finding faults whcih can be exploited.
This isn't a testament to the fraigility of the code but the insane neutron-star-heavy load it's under. If you put just about anything, under a proverbial hydralic press, it's only a matter of time before it cracks.
If you're not sure how to write and/or pronounce someone's name, please look it up before posting a comment, out of respect for them.
While there are plenty of ways to break Tears of the Kingdom, in the vast majority of cases, you have to be actively seeking them out. The number of notable bugs you're likely to discover if you're not doing it on purpose is really quite impressively low in this game. And really, I think that's the most crucial part to have covered. If a bug is virtually impossible to come across without doing it on purpose, it's typically not really a big deal, unless it completely breaks the game somehow in a harmful way or impacts multiplayer balance.
@@laggalot1012 Except for that food effect duping glitch that could literally be achieved with normal gameplay. It's shocking that one passed QA considering how polished the rest of the game is.
What's so impressive is that the item duplication glitch and some quest progression checks are the only true bug I can think of. Pretty much everything else players think of as "bugs" are actually exploits of the game's intended physics.
Yeah. It's why devs were losing their minds over the things the game could do. It's a programming nightmare to have so many individual elements working together in such a seamless way. Despite the bugs people find it's still incredibly polished for the kind of game it is.
I was with 4 other game designers developing a student project, the scope of our 3d puzzle plateformer grew a lot, after that most bugs were happening because saturn and uranus aligned with gandalf who touched too much wood that day. I remember one where if you spin around too much then an issue in the rotation processing happens, triggers a gimball lock and makes the creature you possess walk on its head. It took me weeks to figure out why it was happening.
Love the animation here XD One of the issues I used to come across as a tester is bugs that have seemingly been solved resurfacing because a fix to another bug has caused further problems. So it is quite ridiculous to stay on top of.
Love you humbleness. Ever since I became a software QA I understood more the complexity of the developers.
This is one of the most important videos you'vr made, yet it'll most likely fall on deaf ears. Still, thank you.
My Turn Based Strategy game has a bug where sometimes you move someone and they snap back to the original location, as if they never moved. But it happens so rarely I can't reproduce it. Luckily it doesn't cost your movement so you can just redo the move. Mad respect to people who have to do this professionally.
I honestly feel humbled and have learned some more valuable lessons then I ever will learn from these videos, about bug fixes, how you can't possibly fix all bugs completely or the game will never release, and how VERY MASSIVE AND BIG bug patching can be on developers, and more.
We should all give more respect and support to game developers, creators, ESPECALLY the people involved with patching the game and getting rid as many bugs as humanly possible.
We salute you all!
I'd say I'm generally somewhat lenient on bugs, but at the same time, hearing this, I also think it's truly admirable when a game does release with minimal intrusive bugs.
Take Tears of the Kingdom as a recent example. In like 200 hours, I've barely stumbled upon bugs at all, or at least ones that I could clearly distinguish as such. A bit of jank here and there with things moving in unexpected ways, but that's about it. In a game like that, where you can freely build your own objects and vehicles and have so many interactive elements, I think it's a true marvel that the game is as light on bugs as it is!
That said, there are also games where you can barely go a few hours without stumbling upon obvious bugs or where it even just permeates the experience. I think that's the point where criticism is completely fair game. Obviously, we as players can't personally look into the development process to see what went wrong, but often in situations like this, you can at least tell it went wrong *somewhere.*
Oh yeah Tears is a GREAT example of insane levels of testing. Certain intended methods for Shrines like the moving a foldable bridge via wheeled platform across pools of lava would have definitely caused one of the joints to just fly into space in a good few other games
Some of these things I've heard before, but how extensive those combinations get, as well as grabbing and throwing, were things I hadn't yet considered.
As a tester, I am thankful for when I am given a list of all the combinations I have to test in a document. It makes it so much easier to track.
The amount of BUGS is DIRECTLY PROPORTIONAL to the amount of FREEDOM the player has.
That's pretty much what I got from experience.
Yes and no, it also depends on the way the game is developped, you could group stuff and have the whole group share properties. Like all the trees are technically the same Tree object with just visuals and colliders changed, it means you could test the all the trees as one Tree instead of having to test all the trees individually. To know if Tree as a group can be cut you'd just need to take 1 weapon that is allowed to cut elements from the Tree group and cut 1 tree instead of having to cut all the trees with all the weapons.
@@sebastienpautot Yeah but in that case you're just copy pasting a code for something that the player can't interact with.
What if the player CAN interact with the trees and you give each tree type it's possible interactions?
Sure, they might share some code, but they can't be too similar anyway, because if they were you are limiting the players freedom by making him interact with trees in the same way, even if the tree is aesthetically different for instance.
However, take what I said as an hyperbole. Ofcourse you can give player freedom without making the game full of bugs. But it's just to point out that the more the player is free of doing, the more likely bugs might happen.
I mean, videogames work in a PROGRAMMED environment, and it's impossible to program every possible out come that the free will of a human can come up with.
@@The_D_ray you don't need to program every interactions, you "just" have to let it happen and loosely program behaviours, it can be tough but it's possible. In a game I programmed called Posseslime, all the creatures had the same core, and then creatures had modules for movement, abilities and camera configuration. It allowed us to make and debug individual modules instead of having to debug each creature. Meaning we could debug and provide fixes for the jump ability and all creatures using it would benefit from it.
@@The_D_ray Yes but even if there are a lot of outcomes, in special cases such as emergent systems you only define how certain elements react to each other. So long as those cases work as expected, the the cases should subsequently work according to the laws of how the initial cases work.
@@keplar7243 Yes.
Great wisdom as always, Mr. Sakurai. It’s great to hear about this topic from your perspective!
If a game ends up buggy at release and someone says that it's because the developers were "lazy", I automatically think less of that person
what about developers that release buggy messes at launch to lauded critical reception and legions of fans that fix their game for free to the point that it's basically their entire business model like Bethesda? I respect developers that allow mods for their games, but Bethesda has been relying on an engine from 2002, with all of the oldness and creakiness it entails, and marketing themselves on their new features that frequently don't work or are misrepresented before launch. And all of this was only a problem when Fallout 76 came out because, as it turns out, making a multiplayer game on a single player focused engine, and having the bare minimum expectation of a multiplayer game that is functional at launch, was, in fact, too much for the big corporate developer to manage.
I don't expect bug free, but I expect some effort, and I expect the game to at least be optimized to run on things that aren't new-gen consoles. (looking at you, Jedi Survivor...)
I'd say that depends on the amount of bugs present, while I do think the Team working on Fallout 76 worked as hard as they could, there is no doubt that Bethesda itself was pretty lazy with the game considering there is a 3 Hour video concerning nothing but bugs for that game.
And of course it depends on the Studio as well if an Indie studio made Fallout 76 (and it cam out the exact same) for example it would be more forgiving.
@@fearingalma1550No "what abouts" here. Bethesda works extremely hard on their games for many years, and no one there actually says "the fans will fix it." The modders choose to "fix" what they can see, while introducing new issues, all the time. So no, that isn't at all the exception you thought it would be.
or the ceos are a$$holes
@@fearingalma1550 The stuff that modders "fix" are stuff that would be considered as typical flaws in any standard non-Bethesda video game. With non-Bethesda games, you can't "fix" flaws - even though flaws themselves are subjective a lot of the time - but with Bethesda games you can due to the fact that Bethesda is one of the few developers that actually supports their modding community, unlike Rockstar or something.
The way you phrase it makes it seem like a Bethesda game has to be absolutely perfect in order for modders to not be able to "fix" anything, but that's impossible because no game can ever be perfect. There is always something one can "fix" or improve with any game ever made.
This sorta reminds me of that one Penny Arcade comic that mocked the reality show where one winner would be able to work as a professional video game tester, and instead of being paid to play video games normally he instead had to test every point of collision in a Barbie horse game.
The King of Game Development: Bug Testing Never Ends
Its also worth noting that oftentimes, many players LOVE bug. Take the guardian flings in breath of the wild where using bullet time can send them shooting up at mach 8. Thats an example of a bug people dont want fixed because of its history, practicality, and sheer fun levels. Or the - world in smb 1. That bug is straight up iconic.
Reminds me of the time Nintendo had to purposely include bugs in the Ocarina of Time 3DS port for the fans.
@@LightsJusticeZ yep. Stuff like the web mechanics being bugged in the original got carried over
I do business consultant type development work in my day-job and do hobbyist gamedev and I want to say THANK YOU. This is something I wanted shouted from the rooftops to the general consumer/YT Commentariat. Even games that you'd think are very well put together can be broken by speedrunners exploiting your physics system, a minor bug that's rarely seen or something else completely crazy the developers could not have intended or forseen (Like hacking your game with Koopa shells! There is no way the devs of Super Mario world could have known that! Or heck how would you even test for that!?)
I do want to echo Sakurai's caveat @3:34 for all games, this doesn't excuse them of course and we should still do our best to minimize them but there is a level of "buginess" that is acceptable to the general public (and for the record thats basically minor/rare/unnoticed bugs, as in they dont affect gameplay much, almost never happen, and/or exist outside of normal play, I.E Boundary breaks or only happen when you get the game into a state that was never intended) if you can fix em then sure! Go for it! but again its a bit more complicated than that, especially for games as there's an absurd amount of interactions, but ultimately you have to pick your battles.
Also guys don't name names in this one (per the caveat on paragraph 2), Mr Sakurai did ask us not to speak ill of any other developers on his channel in his first community post. Believe me I can list off at least five devs off the top of my head on this particular issue but that would distract from the point he's making.
Yeah, unit and integration tests can only take you so far! Especially with something as dynamic as a video game!
Puzzle games are usually possible to have automatic tests for, but something like Smash would be nearly impossible.
Turn Based Strategies can also benefit from Unit Testing. Basically any game where there is a easily defined "start" point that can "input" data.
Tech Lead from the games industry here: unit and integration tests are almost non-existent in the game industry for a very very good reason. The level of interaction that makes or breaks a system or a feature does need human testers. I know this sounds weird since in other kind of software it’s common wisdom, but it’s absolutely true.
Really appreciate this video for illustrating how tough it can be. Two additional notes that I try to remind folks who play games:
1) coders and QA are usually aware of most of the bugs players find, but we don't often get to decide what is a priority to fix, and there's never enough time to fix it all anyway.
2) even knowing the above is true, no amount of rigorous testing will compare against what 10000 players crashing into it on launch day will uncover. We do what we can with the time we get!
I understand the issue here Mr Sakurai. Being a dev myself, I'm fully aware of bug existence and also the issue of debugging it.
Another thing worth mentioning is that bugs could be discovered one at a time. When you're about to release a patch, it's possible someone reported another bug which could delay the patch release. So you have to choose to release patch to solve couple bugs soon or major patch to solve many bugs at once later.
Makes me appreciate the unbelievable level of polish Nintendo games reach, especially considering how mechanically complex they are.
Your option of the bug in Pokemon Scarlet Violet? Buggy or normal?
This video is such a mood it hurts >.<
Sakurai-san is speaking directly to my soul.
Existential crisis testers leading in to "Continue?" got a good chuckle out of me. Good job, video editors.
Seeing a Subspace reference made me really happy
As a software developer for a major tech company, I can relate to this so much. Bugs are just as much of an issue in software development as they are in game development.
Another fantastic video from Sakurai! I have tremendous respect for the game developers who have to deal with getting rid of as many bugs as possible before release. Hopefully, this video and Sakurai's information will help dispel the misinformation. As Sakurai confirmed, every game developer wants to release a perfect game. They do not just want to push a game out the window and fix it later with patches, but the large scope of the games makes it impossible to make a game completely bug free. Kudos to the developers spending extra time on the game after it goes gold who continue to test and make content for the game to release with patches and updates. :)
Nowadays a lot of the testing has to be automated for that specific reason: as Sakurai said, it's literally impossible to have persons testing all combinations again and again. Every time there is a change, the cycle repeats. Automated testing alleviates some of that repetition, but it's still complex to imagine all those scenarios and build tests for them. AI is already helping in that area and it should make all this testing way easier and faster in the near future.
Automated testing is great, though an additional challenge in game development is that a lot of the features are experimental in nature with stuff being added, scrapped, or radically changed on a frequent basis, so a lot of automated tests will become outdated almost faster than you can write them. It's typically limited in its ability to find new bugs too, being mostly useful for regression testing. Determining if something new is a bug generally comes down to someone making a judgement call about what the expected behavior should be, which is inherently impossible to automate.
Im 13 and starting game development on godot and his videos help
Good luck, man!
Woah... a godotian in the wild. good luck :D
Good luck! Remember to playtest early and often.
I’ve witnessed a variety of weird glitches, especially in games you’d think have all the glitches pointed out by blogs and media, but it turns out there’s more that’s hidden very well throughout time.
My personal experience was when my Go Vacation (Wii) data got corrupted, I’m still not sure what went wrong, but it definitely made me delete my beloved Mii character just to play it safe.
And then I’ve seen a streamer accidentally find a glitch that nobody’s ever seen or at least reported before in Super Metroid (SNES).
Those are only a few examples of glitches that nobody’s ever reported or analyzed yet, and I have plenty more examples in my experiences of playing or watching a stream.
As someone with over 12 years in QA -- This is an awesome video :)
Another stat to consider, for example, combining all the hours tested over the whole life cycle of Smash Bros Ultimate... Well, at the moment, Smash Bros Ultimate sold 31.7M units. If those players play for just 1 hour on release, that playtime already completely trumps anything the QA team has put in! It is simply impossible nowadays to check every little thing players might do when games are as complicated as they are.
That reference to the game over screen from Subspace is amazing, it even uses the Subspace Emissary logo!
If you think old games were free of downright criminal bugs, check out what happened with Cinder in the first Killer Instinct arcades. They had to manually remove and put new arcade machines (or at least chips, I think?) Into every one in the states or something, it was crazy
Excellent insight, as always. People are so quick to blame devs nowadays for bugs. I think out of "class" Sakurai didn't mention how corporate rushing a game is also a big component of them having bugs despite developers wanting more time
Love this channel so much!
Some bugs/glitches really do occur under extremely rare situations. Imagine if, in a Pokemon game, there's some kind of bug that can happen if you (1) return to a specific zone after (2) Flying away from an NPC in that zone on (3) the _exact frame_ that they try to initiate a Trainer battle with you, and (3) battling a very specific Trainer in a different zone instead of them....
(for those of you who don't follow Pokémon, this is a description of a generation 1 glitch that allowed you to catch a pokémon that was otherwise specific to an event)
I truly respect bug catchers in game dev Q&A. What I don't respect is games that cone out TRULY and utterly bugged like Mass Effect Andromeda where even the simplest mistakes were overlooked and released for full priced. Last ten really had some stinkers when it came to releasing games on time with little to no bugs
Testing for bugs is one of the things you have to do when testing your game. But depending on how big the project is, it's easy to get pretty overwhelmed.
I LOVE HOW THE BUGTESTERS JUST STRAIGHT UP DIE
it even happens twice in the video so im guessing the animators thought it was funny as well haha
Sometimes a bug is really just a cool feature waiting to be discovered (like wall jumping in the original Super Mario Bros)
1:30 subspace emmesary reference!!
If it's nothing game-breaking and easily resolvable, I like messing around with bugs and glitches. In a way, it adds to the charm of a game, as most bugs end up being unique to the game itself due to the circumstances it's found in. Whether you use the glitch to your advantage or not is up to you, but it's like another option you can use to change up your gameplay, even if by a little bit.
There are certain games where bugs are absolutely riddled throughout the game, and when there are too many, it can become a problematic game to play. (Look at Cyberpunk 2077 for example.)
But then there are games where it seems pretty much solid with very few bugs at all; but when you do find one, I feel it's pretty cool. Almost like you've found something new the game has to offer. That goes especially for a game you could have played for a long time.
Unmentioned is work distribution - assigning a developer to: Detect a bug, reproduce a bug, investigate cause of a bug, fix a bug, make sure the fix broke nothing else - means that developer is not working on something else. He might also spend hours on a bug and never find cause of even reproduce it.
My favourite bug from Factorio was caused by neutrinos from the Sun causing rare change in specific graphic cards memory, making it that some internal value would become something the game couldn't interpret - Try discovering that!! Rare occurrence on rare hardware :D
The best advice I can give to game developers looking to find bugs in their game is “don’t play your game properly.” I was once exhibiting a game of mine on an iPad and a toddler walked up, mashed the screen a bunch, and crashed the game. So in that instance, when I triggered a menu transition, I would patiently wait for it to complete. Don’t assume your players will do the same. Mash buttons, quickly and randomly. Tap screens all over the place. Try to initiate and cancel something multiple times in a row. You’d be surprised how quickly things go wrong unless the programming is really rigorous.
Thank you for this video Sakurai! Trying to make my way into a game testing career 🙏
Not only are there gameplay bugs, but there are also bugs that causes crashes, or even damage the console/computer. Those get much higher priority for bug check and will consume as much resources as they need because recalling disks and having that reputation follow you around are both incredibly expensive
Loved that brawl game over screen
Oh, the Subspace Emissary
One of my fondest memories was discovering the Jigglypuff Final Smash bug/glitch when you use it on Bridge of Eldin which makes it so she stays at her final smash size.
It required specific timing, specific stage and specific characters and items. I can't imagine any one dev during the time would of been crazy enough to test for that.
I really hope that this puts an end to all the people who say "omg lazy devs never fix bugs these days". It's NOT as easy as you think!
This video is so important, not just for game designers, but for PLAYERS to understand the process.
You best video to date Story Time!
Abeit pretty short but I happen to find this rare bug in Sonic Mania of all things that if you save replays in all zone with the main trio and going beyond it with mighty & ray you'll see that you'll lose your ability to save and record new records for online can't erase it or erase the whole TT to get it working which leave starting the game from scratch (technically) was hoping for a fix but by the time I've report it, it was already too late 😓sega and whitehead's team shook hands and when on with their day still remainds to this day🙁
So moral of this story: don't ignore the rare bugs because it can be game breaking in something you won't expect seen a few game devs tackle these years after release yet sega won't let them fix just the one
so to say that rare bugs shouldn't be ignored would be the understatement of the decade to say the least.
From my personal experience, at least, in a case like Smash Bros. where there's a million different potential conflicts, the only thing to do is to test each move individually and ensure everything works as intended in a vacuum before proceeding - and I'm sure Bandai Namco does this too. It's just that Ultimate's scale is so astronomical that even with incremental testing and slowly working up to bigger and bigger scopes, when you get to some of the higher level scopes you just can't catch everything. But regardless, if you work up in terms of scope when testing, it should help a lot.
As a solo developer (with very little playtester access), my "favorite" instance of bug testing is any sort of deterministic replay system. You have to figure out through basically brute force what states you need to save to a file, which ones can (or need to) be inferred via context, or - worst of all - which parts of your system were developed with a non-deterministic approach or other bugs in the first place, and weed those out first.
It's stuff like this why I refrain from blaming individual employees for bugs and questionnable level design. The only thing that can get me to write an email is if a bug renders continuing impossible by any means.
*Glares at NIS Classics*
1:15 Yeesh! That's a lot!
1:20 That poor team.
3:23 There's an actual term for it?!
3:30 "explain, don't excuse", are words to live by. It's not about absolving responsibility, just about making sure people don't need to repeat your mistakes
Personally, I dont mind bugs in games that much, so long as they're relatively "stable" bugs. That is, bugs that show up under specific circumstances, and dont cause a soft lock or a similar degree of interruption. Obviously it's better to not have bugs, but there's a clear difference in impact between bugs that a player will often need to search for, and bugs that a player will struggle to avoid
"it's not a bug, it's a feature" words to live by.
This explains why it takes marginally longer to release each DLC fighter.
Now I dread to ask you for anything.
I thinks good philosophy I learned from an engineer early on was "optimization is a persistent mindset, not something you do at intervals"
I legit didn't realize that that was how bugs worked... when you put it that way, no wonder bug testing seems kinda stressful :'D I agree with most of the comments here tho, as long as the bugs don't break the game or make it legitimately difficult to play, it's fine. Perfection is only a concept, after all.
Watching this just reminds me of the hours I spent debugging...even simple stuff can be a pain in the butt to figure out sometimes.
Some games definitely release buggier than others, though, and you only get one chance to make a first impression. There's been plenty of games that were train wrecks at launch, only to get fixed later, that are still remembered largely for their bugs.
It's not just multiplication; it's exponential growth. Literally (2^n)-1 potential bugs for n elements. And, as indicated with the Stage Builder example, custom content can draw out bugs that wouldn't manifest otherwise. And even if jerk publishers aren't part of the equation, a buggy early release may be necessary in order to make the money that can keep one well and working, and that itself is a gamble for an obvious reason. It's no wonder games rarely have their own engines anymore in lieu of preexisting ones.
The fact you posted this while in fixing a bug in my game is funny
Gotten it yet?
@BJGvideos Yes its fixed a friend helped me its called Penguin Quest
@@walkervinshorts7268 Sweet!
amazing yet again, thanks sakurai!
I do like this topic on bugs in games cause it tells us on how it can work though also not work, and lets be real without some bugs we wouldn't have great moments like Roach, 21 second Ocarina of time, or others like it, though I will also point out that games shouldn't come out if it looks broken just everywhere like how Cyberpunk 2077 came out which they learned and have fixed it up now over time though another game I will praise is Baldur's Gate 3 which started in early access and asked fans how they could improve it over it's development.
Creating videogames is a never-ending nightmare, kids!
Sometimes a missed bug can make or break a game. Super Mario 64's Backward Long Jump made speedruns more fun and interesting to watch.
The mention of "enbugging" reminds me of Dead by Daylight's issue with every patch they release. They fix a major bug, then more bugs start showing up.
I was thinking the same thing.
Like the hydra. You get rid of one bug, and more take its place...
Software development is harder than you can think. The one time bugs are okay Is games 😊
Very informative video!
They truly are some of the best developers in gaming
QA testers would love to enjoy the games they test as much as any customers do,
but they need more sanity than anyone to make sure they don't end up feeling numb over it due to their job.
It's also a thankless job, so make sure you thank them.
I use to do coding class and I agree that bugs are so annoying to deal with.
0:18 True about the developers, but certainly not most of the corporations
I love this channel!
I can think in some recent games that looks like it was launched without debugging
While of course it would be nice if there were just zero bugs, I'll always respect developers who work to fix any bugs players encounter that negatively impact the experience.
0:38 This line right here is pulling a LOT of weight. But we don't say any more.