Automated Testing of Gameplay Features in 'Sea of Thieves'
ฝัง
- เผยแพร่เมื่อ 8 ก.พ. 2025
- In this 2019 GDC talk, programmer Robert Masella explains why automated testing was the right choice for Sea of Thieves, and how the practice could benefit other developers and studios.
Join the GDC mailing list: www.gdconf.com/...
Follow GDC on Twitter: / official_gdc
GDC talks cover a range of developmental topics including game design, programming, audio, visual arts, business management, production, online games, and much more. We post a fresh GDC video every day. Subscribe to the channel to stay on top of regular updates, and check out GDC Vault for thousands of more in-depth talks from our archives.
As a non-game industry programmer, this is a very refreshing presentation to see. I hope automated testing catches on across the board in game development.
@pixelroutine4609 You have no idea what frustrating errors I've had to deal with in my career. The fact of the matter is, automated testing and writing testable code are the easiest ways to make sure that persistent errors get fixed and more importantly *STAY FIXED*.
As an SoT player, bugs in SoT is a huge meme within the community and we had our fair share of running into said bugs once or twice per month. Most of them are harmless and well documented that you know what to do to avoid them.
As someone who worked on game jams and student game projects, it's a complete wild west between:
- Figuring out the bugs (Is it an isolated type or a serious one that travels up the chain of interactions.)
- Documenting the bugs for recreation (so you know what is causing it)
- Fix the bugs in the current build.
- Make sure new changes in the future build won't trigger an old bug/start a new one.
^ Rinse and repeat all of the above steps.
Just remember:
- The scale of bug fixes is relative to the complexity of the game itself.
* (This also applies to staff sizes and new content releases to keep things fresh or people would get bored.)
- No game is without bugs and no bugs are without a complete carpet comb down of checks.
*(You could spend a week combing a new map/level for collision boxes and people *will* always find new ways to clip through something in some weird places.)
*NOTE:*
- No game test/playtest is foolproof. Human interaction will always be required at the end of the day. (This is where player feedback comes in.)
TL;DR - The devs have always been working hard on the fixes but it's not a perfect system.
Join SoT insiders, and help them pinpoint the issues that AIs can't find.
Man...The dunning kruger effect is on full display in the comments here. The game industry has opened up a lot in recent years on the development process. Instead of bridging the gap between developers and gamers in terms of understanding the process it seems to have spawned an army of armchair developers, most of whom I'd wager have never written a single line of code, let alone been anywhere near AAA production. Yet every time they feel the need to share their "expert" opinion on how those "lazy" devs should've done things differently.
@@dexterdextrow7248 Sea of Thieves currently sits at very positive on steam. Evidently for most players the bugs aren't that much of an issue. But sure hardware differs a lot between players. Some have lower tolerance for jank and some just play the game differently. Granted there are many reasons why some encounter issues that others don't. If your game is buggy by all means leave a critical review. Giving that bad review because your game continuously crashed or something is totally legitimate criticism. Saying this method is useless because your game crashed isn't, because that assessment isn't based on any insight on whether this method contributed to making things worse. More likely this method prevented a far bigger mess of bugs and was simply an efficient method to regularly release builds that were as stable as possible with the time and resources they had. Even as a game developer if your not part of that team it's really hard to say why something went wrong or right, because every game project is a different challenge. Commenting with the presumption that you do know is sharing that "expert opinion" which is born simply of ignorance.
Also this video isn't preaching anything to users because it isn't for them. This is a talk on a game developers conference for game developers on the game developers conference youtube channel. It it presented in a way that it assumes a basic knowledge of game development workings. So no it isn't reasonable to expect "legitimate" criticism here from users. If all kinds of people want to watch GDC lectures and learn, great more power to them. If they want to give feedback on a game, or criticize it, sure. Write on the forums, leave a review on on the store page, Engage a dev on twitter, reddit or whatever if they're open to that. There are tons of places where developers are actively looking for feedback, on forums, early access pages, etc. This isn't one of those places. These comments are confusing correlation with causation, they aren't adding anything to the discussion here and are just cluttering up the space for people genuinely interested in learning and discussing the application and effectiveness of such a method.
Great comment, this is something that has bothered me for quite a while. Seems like we are just getting more and more "gamers" that are self-described experts and just spreads a lot of bad misinformation. That kind of attitude just drives devs away from engaging with their community, which I think most players would actually want.
@@dexterdextrow7248 Lets respond to your points step by step:
"Programming and game development in general has become a much more wide spread skill and occupation, of course more people will have a opinion on it"
- It's fine that people have opinions on matter, what's not fine is that when someone describes themselves as an expert when they really have only watched a video on programming or similar. If you don't really know something then say so, it could be as easy starting the sentence with "I'm no expert, but this could be because of ...".
"if developers try to preach for a solution that gives a result the users don't appreciate"
- The problem is that if you don't understand what's being preached, then how would you know this is the reason for any result you speak of? There could be so many reasons for why a game has a lot of bugs, and describing that to this one thing doesn't make sense. The thing here that's being preached is to developers of the same type of games, we are talking about a tool which can make a lot of things much easier, but of course any kind of tool can be misused, that doesn't mean the tool itself is incorrect.
"firstly, that it "isn't for them" is a bit of an elitist view"
- It's not an elitist view to say that you shouldn't spread misinformation about things that you have no actual experience with, I think for most people this is quite obvious, but something about the internet makes everyone speak like they are experts in the field. GDC is targeted at developers, to understand what's being mentioned fully you need to have some underlying experience with this area. If you don't then don't speak as you understand it fully.
"the worst thing for a consumer is if a large amount of developers adopt a practice that leads to bad or unsatisfying results"
- Absolutely, as long as the consumers understand the practice and what result it leads to, which in this case they obviously don't.
"But if they feel this game is unusually buggy compared to other previous titles from rare or in the same genre, that can certainly hint to this being the case."
- Or it can be 100 other reasons, like the example that you brought up yourself. I'm not going to sit here and say why SoT is buggy, because that can probably only be answered correctly by the team themselves.
"Same goes for all design decisions in consumer products, in relation to games we have very predatory and manipulative micro transactions that are in so very popular"
- And that seems like something that would be very healthy to complain about, because the consumer can see the direct correlation between the causing factor and the result. I wouldn't say the same for in-depth development practices.
@LordSnorky: You are speaking like you're on the other side of the industry (dev). Because we (gamers) don't give a shit about your life as a game dev. We just want good games and sea of thieves ins't one according to the critics. Don't worry, gamers have an exellent "understanding" of devs and greedy corporations who employ them. So far, gamers have been coping and understanding devs a bit too much. Now AAA games are just garbage, gambling games and asset flips. It would just be time for the AAA industry to die to better rebirth.
@@BlueBeam10 I'm not going to bother with someone who begins their comment with "'I don't care about the people who facilitate my hobby. I just want my games!''
Thanks for sharing your experience with automated testing. Definitely, a good path to be considered.
I came to know about the bugs with this game and most of them were physics body interactions.
The crunch curve is pretty cool. These tests will help to avoid that.
Very informative and accessible talk, thank you!
This is so amazing. Thank you for sharing your progress in the world of automated game tests
I find the comments here interesting. I play the game a few hours a week, I haven't hit anything too serious. Bugs? Sure. Occasionally I can't jump or interact with an item, and sometimes there's server issues, but it's far and few between and doesn't spoil the fun for me.
i've sunk 300+ hours into SoT and have rarely had bugs that were game breaking at all.
I’ve put 1000+ hours in this game and there are bugs that can and do ruin the fun for me and a lot of people. For example the hit registration is really broken and rocks, wood launch you in the air for no reason, the animations do not look the same for people on different clients, the hitbox for the character is not accurate, the bullets that do not hit often kills the person. The PVE part of the game is really good and really fun but the PVP part of the game needs a lot of work and should really be focused on right now.
@@BajorDavid my issues mostly came with issues that have (hopefully) been fixed now, but the issues were vault keys wouldn’t work because someone had a tall tale open, so you either have to wait for them to be done, or give up, and the glitch where the shipwreck in a merchant lost ship quest would just not be there, and you could only get the manifest, nothing else.
@@thedarkcod4824 Huh, never had the vault thing and it is my only favorite thing to do in SOT. As for the merchant quest I had that happen once but that was about it. Getting stuck on hitboxes can be a problem and direct melee or gun combat in PVP is to be avoided at all times. Ship battles are good though. Apart from that I have not encountered many game breaking bugs to be honest. Sometimes rowboats dont work, especially after Pirates life update.
2019 talk bruv.
BIG TALK ENERGY! Love the tips and insights.
The idea of releasing software without automated testing is often seen as madness outside of games so this is very refreshing. Yes, I know, there are bad apples who have little or no test coverage, but they generally know they _should_ have.
It's a failure of priority, it sucks.
@@CoreUnderscore it so often is outsinde of game developement as well ^^
I feel like there's a mantra about the importance of automated testing, but I've seen too many cases where the automated tests are garbage, barely testing anything, and engineering leadership just doesn't care.
@@dit-zy It often happens for two reasons. The first; because management sees it as time not spent working on features. In reality, automated tests usually pay for themselves down the line when the code base is much bigger and features touch a lot of points in the code. The amount of debugging and QA time saved with tests proving you didn't break anything is orders of magnitude higher than the time spent writing said tests. The second; tests were introduced late in the development process after a lot of "untestable" code was added.
@@soviut303 very much agreed. Another reason (similar to your first) is that management cares, but is bad at investing in their tech (another common problem). So solutions are poorly engineered, but so is automation infrastructure, making it harder to add or run tests. So tests are worse and cover less surface.
skip to 12:10 if you know the differences between unit and integration tests already
Property-Based Testing mentioned, let's go!
The most intensive such automation test suite I've seen is on Factorio.
This is a VERY good idea, something I wish I had done over past 7 years!
It could be cool to have chapter timestamps in talks
18:05 really clever, though seems it would have been worthwhile to add a feature that logs and compares the time till success to prior test results. Would be useful to flag them if there is a significant, whatever you deem that to be, maybe 10% difference, between how long it used to take and how ling it took this time.
That way, if you know you implemented some new animation which explains it, you can quickly tag it as fine, but if you haven't changed anything that you expected to change it, you may be alerted you've got a bug of some sort before it becomes catastrophic.
Don't really get the hate for the game, I'm just here to learn automated testing
I can tell you it doesn't work cuz the game is a buggy mess that's why a lot of people hate the game
One word: Hitreg
The game is great
@@savagefox3530 Ive been playing for just over 100 hours and have only encountered the single minor bug of being flung through the ocean probably 3 times. It is probably the most solid game ive played in a long while. If you want to have a game to complain about bugs, try something made by Gaijin, like War Thunder.
@@t1e6x12 Do...Do you even play the same game I do?
At 21:39 they show their functional test blueprint, but their SyncClientServer and ThenClient nodes don't exist in Unreal. Does anyone have any more information as to how to make those myself? At 53:10 they talk about faking replication.
Wow, It's very interesting subject
I don't think creating a minimal world for the tests is such a weird thing - as he says the editor has multiple worlds already, you have the main one being edited, you get a cloned PIE world when you do play-in-editor, most of the editor tools (phat, static mesh editor, etc) have their own minimal preview worlds that individual assets can be edited in without affecting the main world, and I think if you implemented a feature like CoD killcams (mini replays) you would probably do it with a separate world too.
what tool can test automation for game, plz ?
Chris Wilson, from GGG, should watch this ;/
Can all this be done with Unity as well? I am new and completely clueless so I thought I'd ask..
Hi, i was wondering, maybe somebody can help me.
What is asset audit checks?
Things like file size, naming, number of polygons etc on any of the game assets/content
I like that he says they didn't do test driven framework, why? Because requirements always change. So if you are doing 2 week sprints on a project where customers don't know exactly what they want, don't do test driven framework.
Yep. Sometimes I let test programmers write tests after a function has been established.
--What does he mean with young studio?--
--Rare has been there since N64.--
--Or is there something that Microsoft did like layoff all the old staff?--
EDIT: Nevermind he said "long" I understood "young".
The cost benefit analysis might get thrown off if you're starting from the point of a human manually checking every regression every change.
You kinda need good abstractions to reduce the complexity. Automated tests can be good, but if you've got a highly coupled nightmare of code all they do is tell you how brittle your software is when every change causes 100 test failures (which is sadly usually the point people say thank goodness they had tests).
It's an interesting topic but tests are code and it takes a lot of experience to do them well and to know when you're getting value. I'm sure as they carry forward they will refine their methods.
Sot and other physic based movement in game should have a button, "Hey I'm stuck, reset position".
many games do. However, if you are truly stuck, SoT has a reliable system where it will put you back on your ship.
@@thefonze7 Reliable?
That sounds terrible. Just teleporting out of combat or whenever youre in danger.
@@ubserd8187 yup! Reliable.
Also - the “black out zones” at certain spots, suck as FOTD, are examples of such a mechanic.
The spots that you can get truly stuck are rare, and in order for them to be effective in combat you would have to have them memorized.
The worst one is at Fotd, you can teleport back to ship in a couple secs
all forts have really ez blackcreeen spots despite the fact they you don't even get stuck, also trees exist, if you haven't been stuck in a tree at least 10 times I guess you just aren't a professional stucker like me
The automated tests forgot to test the hit registration.
They actually playtest their game? You wouldn't know :/
The computer play tests for them... "It doesn't crash, release it!"
Because they didn't! A computer did it for them!
A game infamous for its many bugs is going to teach us how to test a game.
Because this is the only large using automated testing so they would be the only game capable of giving a talk.....
A gamer who doesn't understand anything about game development (and definitely didn't understand or probably even watch this lecture) is going to tell us how to develop games.
@@ninjadodovideos how childish are you? I worked 6 years in QA.
@@Tormentadeplomo And yet your comment demonstrates you either didn't watch or didn't understand the lecture.
@@dexterdextrow7248 Not really, but others have already explained this better elsewhere in the thread so I won't bother to repeat them.
Man I just started the video but from the title alone I can finally understand why in Sea of Thieves nothing works as intended and every patch brings incredible and unexpected new bugs.
[edit]
Rofl, I sent a link to this video to the guy I usually play SoT with and he answered laughing and "I guess AIs don't get on board of ships".
[edit2]
At 50 minutes now, and this turned out to be a bit sad. Especially at the Banjo graph, because it seems to me like the guy (or the team even) really believed in this method, while all I can see is a gargantuan waste of time building, fixing and mantaining a whole system supposed to do something in your place while it's evident by the game something's not working. Not only the game is a shitshow of bugs (many of which I've always been wondering how they got through, on which I joked about but is a real issue) but many have been there from the start of my experience, which would be when the PC Game Pass released so not exactly a couple weeks, and never got adressed, but it's still adding new issues every time.
For instance the day the disney patch dropped we had to "restart the crew" 4 or 5 times just to be able to get on board our ship which had a collision bug I so far had only seen on lifeboats. We finally managed to do the first new quest and the very moment we got out of the instanced part back to the regular sea we both clipped through the boat again and watched it sail away laughing. I have a whole lot of shadowplay videos with the funniest random stuff happening, from chicken cages vanishing through the mast to flying boats and after all this time collisions are still shit and I don't think I had any game session where I didn't have issues walking between bridges on my sloop because the lag fucks collision with the stairs up. Dropping a chest on a lifeboat connected to the ship is something I learned to never do for the same reason, sometimes they just clip through and drop into water. I could go on for ages or you could just go to the reddit to see how much weird shit happens in this game.
It feels to me like they spent a lot of time and effort on a system that ultimately does nothing. I personally thought they just rushed patches out without testing. It would be interesting to hear from the same guy in the current year.
Any chance you could post some of those?
@@zappodude7591 I've got the aforementioned chicken and a couple more I shared on whatsapp, I'm putting them together and uploading them. Might warrant a headphone warning tho.
[edit]
Since I'm here I'll also upload the one about the boat clipping.
[edit2]
th-cam.com/video/Djd4Hp32peQ/w-d-xo.html
[edit3]
Here's patch day
th-cam.com/video/9i7L0i9dZ3o/w-d-xo.html
You have no fucking idea what you are talking about. Automated tests are how world class robust software is developed. Yes, games are software.
@@13b78rug5h You seem to lack the proper ability to read. I never assumed anything and never judged the method itself. I did make fun of it but not because "omg it's automated" but because the game is a shitshow in this regard. I'm judging by the results. Considering you have to generalize so much just to insult me gives me the idea that maybe I'm not the one talking out of his ass here.
You're also implying that every automated system is the same, which I don't think I need to state how silly it is.
Maybe next time think a bit more before commenting?
Is this why the game is always buggy as shit and hit reg is atrocious? Cuz nothing is manually tested
Non of this helped them with hit reg
None of you seem to understand that having automated testing to catch certain bugs doesn't mean they don't ALSO do regular playtesting.
Ah so that is why the hit register is so damn broken in SoT
Let me guess: all of this was done without considering ping or latency from other players - locally, on a private server or something.
Yes, because a triple AAA game studio with hundreds of people didn’t know to account for latency
Wow they should hire you as the new president of the company, nobody there thought of that! You're smarter than everyone there!