I'd pay to review a spec sheet and checklist of completed tasks when they implement a new feature to test. Hows the sever performance affect your QA tests? Do they provide an offline server to bypass those bottlenecks? IMHO we backers are the QA without the internal ticketing system.
You know this is a really good idea, I don't know if that's too sensitive according to CIG, but it would give us a lot of insight. I would be interested, but I don't know how many others may be.
@@NoNaNelson69 Wouldn't that be counter-intuitive with the fact that almost all actions are server side requests? (doors, drinks, KIOSKs). How do they test? And why not on a PU cloned server with identical hardware?
@@zeyosin1771 you would be suprises how muhc power the client has in this game - this in not a server sided game - and even if i needs server requests, other games do that too - running server and client on the same machine (e.g. minecraft) it would also be much easier to change server settings
@@NoNaNelson69 Actually its a very network reliant system, thats why nothing works when the server begins to struggle. This is the opposite of client based actions/operations that don't exist on server side or are updated at expected intervals. You should research the old quanta framework to better understand what im trying to explain. Testing without the environment parameters is useless when confirming a successful resolution.
i was in QA in a leading position for 13 years... and QA is not the problem that CIG faces. Their problem is that they are utterly incapable to get anything done, besides timesinks.
I mean you only have to look at SC's track record of pushing out patches early to know you guys have a hard time getting half of the testing done that is needed before management pushes things out, its over your head and personally I never blamed QA
They have to get the content out to make sure it works. Because you're part of the alpha process the people who are gonna be buying this later my still be in dippers and to them when the game comes out they might only just be getting the technology to play it in their country. We don't know how many 3rd worlds gonna become the next England or America
Funny, when I learned to program for commercial stuff, my teacher said. If you test it, please test as if the cat is dancing at the keyboard and see what happens. He was right, customers do things you can't even imaginewhen you build a program and if you talk to them, most of the time they say "well I didn't do anything special" and you know they have to pushed a whole lot of buttons before the problem could appear. 😄
I appreciated the content and perspective. Your sound was good and production values were high with video matching what was said. One critique: Your video is very distorted. Almost like your FOV slider has changed from default. This gives it a very fisheye quality and is nauseating to watch and very unappealing. This was less important for this video which largely was about the audio content presented but still made me want to click off the video multiple times. Keep up the great work!
Can you give some perspective on why SC bug reports so often don’t get addressed? We very often find bugs that seem like they don’t happen to many people. But it genuinely breaks the game for a small group of people. It seems they devs don’t see these as a big enough issue to look into. I see this all the time and it piles up into a buggy mess of a tech demo.
100%, and this will come from a developer perspective vs a QA one. I will start off by saying that from what they have said, the root cause of most of this is from releasing features too early to allow it to be "playable for players". In addition, the developers made poor solutions which were never intended to last this long but are now integral to the game. So it is mismanagement of the development of the game. QA is probably just as mad as we are, but sometimes, when bugs are reported, the issue is far worse than expected. The developer may look at it and see it'll take x amount of hours and involve an overhaul of certain items. In pre-released products, this can be common, but it's now common to have people play or test in pre-release. It then comes to a management choice. Do we fix it now or use "hacks" in the code to make it a little better. Often, the wise choice is to wait until you redo the system where the underlying issue is. Yes, this means it's because one system was written and designed poorly, which often happens when you go fast. If you follow ISC, you can see they have kinda been tackling some of those old bad systems now, which has caused recent releases to take longer, but the overall stability of the game has improved.
Its normal that bugs are prioritised, as there is finite time to do stuff. Normally there is a threshold below which fixes are not implemented. Things that influence this are how likely is the bug to occur, is there a work around, how long has it been broken. If it's perceived it will be hard to reproduce the bug (report is vague, or requires a highly specific setup, including a chain of actions), this will impact priority too. If the root cause of the bug is complex, or is planned to be worked on separately in the near future, this will decrease priority. if it's in a specialised bit of code, with only one or two expert developers, there is reduced capacity to take on work.
Why do you even need an explanation? Isn't it logical that addressing all bugs may not always be possible or smart if the connected systems are getting replaced for example?
@@Shadowshade-Gaming gnerally the major issue is that the dev team is not testing bugs in the conditions in which they exist. They test them on their servers with their optimized system and ping rate. I seriously doubt any of these devs are taking a Walmart gaming laptop to the local mcdonalds and testing the issues in an open live server.
I would be very interested to see what their automated integration tests look like. A system this complex, the cost to fix must be very high by the time it hits evocati testing.
I would revise the employment status of every single person working QA at CIG. I say that as a person who does QA for a company, and we don't accept mistakes. Although we are held up to federal regulations / standards as we are a med company.
Same, I'm a CSV contractor (computer systems validation) in the pharma industry and there's just no excuse. Though I will say this isn't just directed at CIG. I'd direct it at the entire consumer software & games industries. The stuff they get away with boggles my mind. If I have an HMI that has buttons flipping colours at random it can halt the qualification of ANY system it's connected to until it's fixed for gods sake.
Star Citizen is an alpha under development, they are constantly adding new features, gameplay systems, etc. in a game that already combines thousands of different interconnected things. Every single update is extremely likely to bring-in issues with new features as well as old features that worked perfectly fine before. If they took time to do a perfect QA testing phase to eradicate all bugs for each live release, the development wouldn't never be able to move forward. You said it yourself, they don't have enough time. Same goes for what you said about testing with users. Star Citizen actually has a really strong infrastructure for that with the issue council. Internal tests unveil issues but not all of them, that is why every patch then goes into a testing phase with hundreds of Evocati players basically doing the job of QA testers. Then, once major and game breaking bugs are identified and reported, devs do their best to fix it. But even this is not enough, because some issues are sometimes so niche or happening under very specific conditions, even an Evocati testing environement won't unveil them. This is why each build gets into PTU Testing afterward, with thousands of testing players, and then even the LIVE release sometimes unveils issues that where not apparent during PTU testing. Only having internal QA testing for a game like Star Citizen is just impossible, and the current system CIG is using for QA testing is working extremely well to identify bugs and issues and fix the major ones for each patch to be able to provide LIVE release as stable as possibles all while still moving forward with the development of the game.
@dashbolts101s when bugs remain unfixed then they may cause more bugs later. Imagine building home. You start with foundation level. You are about to start next level but you noticed that part of the foundation is missing. Do you fix it now or after you build next levels over it? This hole in foundation may affect rest of structure above - and will be harder to fix later
You know you are very right, and their QA and developers must be so grateful for the reports of users. Most reports are poorly written, so they still take effort to work on, but literally, the bug council and players are the reason Star Citizen is able to succeed in any form at the moment. If this game ever does get released, it'll most likely have the chance to release with fewer bugs than others because of this. But if it means we as players feel the burnout that the devs and QA feel. Time doesn't only burn funding, but it also kills motivation if you don't have wins. I feel that they didn't get their feet under them until 2021, and I hope it's not too late now.
Well OP... It's only been over TEN years!!! What's another 10... Right!? And bugs only get resolved... When someone actually spends the time to resolve them! Otherwise... Bugs on top of bugs will result! And once a hyper complex project hits a certain threshold of unresolved bugs, the only solution left, will be to dump the entire project and start ALL OVER again!
I have a friend that wouldnt give a QA pass for the loading screen on star citizen. QA is nonexistent, they just covering their asses for features they lied about 10 years ago.
These video's you make about QA are pointless when it comes to Star Citizen. I wouldn't say this game was in early access or even a Beta in the sense most people are familiar with these day. I would say it's more like your being allowed to play a game that's in still in early-mid development, expecting them to test and fix things that are constantly being changed is a exercise in futility and a waste of development time. The sooner people realise this the better. Talking about QA closer to it's completion would make alot more sense, which will most likely be many years in the future unfortunately. It might be a hard pill to swallow for most people but that is the reality. Just be glad there's at least one development studio out there trying to make a truly next-gen game and you get to at least have a peek behind the curtain in its early stages.
Forgive me..but “your company”..you can’t be older than 16-18 years old. What kind of company and how many employees does this supposed company of yours employ?
You really think just because you did something for many years you know what you are talking about? The people who sit on the couch all day and then make youtube videos about something they never did or learned anything about, THEY are the true experts!
Typically, if someone does something for many years, they have more experience than the average Joe, so... Yeah, maybe they do know what they are talking about? Your comment really has the same tone as: "You really think just because you might specialize at something you know what you are talking about?"
I feel like there's some sarcasm here and see a joke. I do want to say that just cause you work in a field doesn't make you an expert even if it's for a long time. I played Cello for a number of years and got to play very difficult pieces only because of what my teacher said. Practice does not make perfect, but perfect practice (with intent to improve) makes perfect.
I'd pay to review a spec sheet and checklist of completed tasks when they implement a new feature to test. Hows the sever performance affect your QA tests? Do they provide an offline server to bypass those bottlenecks? IMHO we backers are the QA without the internal ticketing system.
You know this is a really good idea, I don't know if that's too sensitive according to CIG, but it would give us a lot of insight. I would be interested, but I don't know how many others may be.
pretty sure QA that isnt specifically for net-code is tested on local instances on their PC
@@NoNaNelson69 Wouldn't that be counter-intuitive with the fact that almost all actions are server side requests? (doors, drinks, KIOSKs). How do they test? And why not on a PU cloned server with identical hardware?
@@zeyosin1771 you would be suprises how muhc power the client has in this game - this in not a server sided game - and even if i needs server requests, other games do that too - running server and client on the same machine (e.g. minecraft) it would also be much easier to change server settings
@@NoNaNelson69 Actually its a very network reliant system, thats why nothing works when the server begins to struggle. This is the opposite of client based actions/operations that don't exist on server side or are updated at expected intervals. You should research the old quanta framework to better understand what im trying to explain. Testing without the environment parameters is useless when confirming a successful resolution.
i was in QA in a leading position for 13 years... and QA is not the problem that CIG faces. Their problem is that they are utterly incapable to get anything done, besides timesinks.
Amazing. Didn't know we had PES and 1,000 player server meshing tests back in 2014.
I mean you only have to look at SC's track record of pushing out patches early to know you guys have a hard time getting half of the testing done that is needed before management pushes things out, its over your head and personally I never blamed QA
They have to get the content out to make sure it works.
Because you're part of the alpha process the people who are gonna be buying this later my still be in dippers and to them when the game comes out they might only just be getting the technology to play it in their country. We don't know how many 3rd worlds gonna become the next England or America
Funny, when I learned to program for commercial stuff, my teacher said. If you test it, please test as if the cat is dancing at the keyboard and see what happens. He was right, customers do things you can't even imaginewhen you build a program and if you talk to them, most of the time they say "well I didn't do anything special" and you know they have to pushed a whole lot of buttons before the problem could appear. 😄
I appreciated the content and perspective. Your sound was good and production values were high with video matching what was said. One critique: Your video is very distorted. Almost like your FOV slider has changed from default. This gives it a very fisheye quality and is nauseating to watch and very unappealing. This was less important for this video which largely was about the audio content presented but still made me want to click off the video multiple times. Keep up the great work!
Thanks for the feedback!
Can you give some perspective on why SC bug reports so often don’t get addressed? We very often find bugs that seem like they don’t happen to many people. But it genuinely breaks the game for a small group of people. It seems they devs don’t see these as a big enough issue to look into. I see this all the time and it piles up into a buggy mess of a tech demo.
100%, and this will come from a developer perspective vs a QA one. I will start off by saying that from what they have said, the root cause of most of this is from releasing features too early to allow it to be "playable for players". In addition, the developers made poor solutions which were never intended to last this long but are now integral to the game. So it is mismanagement of the development of the game.
QA is probably just as mad as we are, but sometimes, when bugs are reported, the issue is far worse than expected. The developer may look at it and see it'll take x amount of hours and involve an overhaul of certain items. In pre-released products, this can be common, but it's now common to have people play or test in pre-release.
It then comes to a management choice. Do we fix it now or use "hacks" in the code to make it a little better. Often, the wise choice is to wait until you redo the system where the underlying issue is. Yes, this means it's because one system was written and designed poorly, which often happens when you go fast.
If you follow ISC, you can see they have kinda been tackling some of those old bad systems now, which has caused recent releases to take longer, but the overall stability of the game has improved.
Its normal that bugs are prioritised, as there is finite time to do stuff. Normally there is a threshold below which fixes are not implemented.
Things that influence this are how likely is the bug to occur, is there a work around, how long has it been broken.
If it's perceived it will be hard to reproduce the bug (report is vague, or requires a highly specific setup, including a chain of actions), this will impact priority too.
If the root cause of the bug is complex, or is planned to be worked on separately in the near future, this will decrease priority.
if it's in a specialised bit of code, with only one or two expert developers, there is reduced capacity to take on work.
Why do you even need an explanation?
Isn't it logical that addressing all bugs may not always be possible or smart if the connected systems are getting replaced for example?
@@Shadowshade-Gaming gnerally the major issue is that the dev team is not testing bugs in the conditions in which they exist. They test them on their servers with their optimized system and ping rate. I seriously doubt any of these devs are taking a Walmart gaming laptop to the local mcdonalds and testing the issues in an open live server.
I would be very interested to see what their automated integration tests look like. A system this complex, the cost to fix must be very high by the time it hits evocati testing.
I would revise the employment status of every single person working QA at CIG.
I say that as a person who does QA for a company, and we don't accept mistakes.
Although we are held up to federal regulations / standards as we are a med company.
Same, I'm a CSV contractor (computer systems validation) in the pharma industry and there's just no excuse. Though I will say this isn't just directed at CIG. I'd direct it at the entire consumer software & games industries. The stuff they get away with boggles my mind. If I have an HMI that has buttons flipping colours at random it can halt the qualification of ANY system it's connected to until it's fixed for gods sake.
Star Citizen is an alpha under development, they are constantly adding new features, gameplay systems, etc. in a game that already combines thousands of different interconnected things.
Every single update is extremely likely to bring-in issues with new features as well as old features that worked perfectly fine before. If they took time to do a perfect QA testing phase to eradicate all bugs for each live release, the development wouldn't never be able to move forward.
You said it yourself, they don't have enough time.
Same goes for what you said about testing with users.
Star Citizen actually has a really strong infrastructure for that with the issue council. Internal tests unveil issues but not all of them, that is why every patch then goes into a testing phase with hundreds of Evocati players basically doing the job of QA testers. Then, once major and game breaking bugs are identified and reported, devs do their best to fix it. But even this is not enough, because some issues are sometimes so niche or happening under very specific conditions, even an Evocati testing environement won't unveil them. This is why each build gets into PTU Testing afterward, with thousands of testing players, and then even the LIVE release sometimes unveils issues that where not apparent during PTU testing.
Only having internal QA testing for a game like Star Citizen is just impossible, and the current system CIG is using for QA testing is working extremely well to identify bugs and issues and fix the major ones for each patch to be able to provide LIVE release as stable as possibles all while still moving forward with the development of the game.
@dashbolts101s when bugs remain unfixed then they may cause more bugs later. Imagine building home. You start with foundation level. You are about to start next level but you noticed that part of the foundation is missing. Do you fix it now or after you build next levels over it? This hole in foundation may affect rest of structure above - and will be harder to fix later
You know you are very right, and their QA and developers must be so grateful for the reports of users.
Most reports are poorly written, so they still take effort to work on, but literally, the bug council and players are the reason Star Citizen is able to succeed in any form at the moment.
If this game ever does get released, it'll most likely have the chance to release with fewer bugs than others because of this. But if it means we as players feel the burnout that the devs and QA feel. Time doesn't only burn funding, but it also kills motivation if you don't have wins. I feel that they didn't get their feet under them until 2021, and I hope it's not too late now.
Well OP... It's only been over TEN years!!! What's another 10... Right!? And bugs only get resolved... When someone actually spends the time to resolve them! Otherwise... Bugs on top of bugs will result!
And once a hyper complex project hits a certain threshold of unresolved bugs, the only solution left, will be to dump the entire project and start ALL OVER again!
We are QA
I have a friend that wouldnt give a QA pass for the loading screen on star citizen. QA is nonexistent, they just covering their asses for features they lied about 10 years ago.
These video's you make about QA are pointless when it comes to Star Citizen. I wouldn't say this game was in early access or even a Beta in the sense most people are familiar with these day. I would say it's more like your being allowed to play a game that's in still in early-mid development, expecting them to test and fix things that are constantly being changed is a exercise in futility and a waste of development time. The sooner people realise this the better. Talking about QA closer to it's completion would make alot more sense, which will most likely be many years in the future unfortunately. It might be a hard pill to swallow for most people but that is the reality. Just be glad there's at least one development studio out there trying to make a truly next-gen game and you get to at least have a peek behind the curtain in its early stages.
Forgive me..but “your company”..you can’t be older than 16-18 years old. What kind of company and how many employees does this supposed company of yours employ?
I wanna he fidelity qa but i dont rhink rhey care nuch about fidelity ar rhis point
Let's stop the charade, we all know the backers (current players) are the real QA.
You really think just because you did something for many years you know what you are talking about?
The people who sit on the couch all day and then make youtube videos about something they never did or learned anything about, THEY are the true experts!
Typically, if someone does something for many years, they have more experience than the average Joe, so...
Yeah, maybe they do know what they are talking about?
Your comment really has the same tone as:
"You really think just because you might specialize at something you know what you are talking about?"
I feel like there's some sarcasm here and see a joke. I do want to say that just cause you work in a field doesn't make you an expert even if it's for a long time.
I played Cello for a number of years and got to play very difficult pieces only because of what my teacher said. Practice does not make perfect, but perfect practice (with intent to improve) makes perfect.
@@Shadowshade-Gaming I'm just gonna reference Poe's Law and my possible autism lol