A Software Engineer Reacts To Star Citizen Server Meshing & Replication Layer - CitizenCon 2953
ฝัง
- เผยแพร่เมื่อ 28 ก.ย. 2024
- By popular demand (thanks, both of you ;) I share some thoughts from a software engineering perspective about the Replication Layer and Server Meshing section of the StarEngine talk from Star Citizen's CitizenCon 2953 this October 2023.
2023 CitizenCon presentation on replication layer & server meshing:
• CitizenCon 2953: Shapi...
2021 CitizenCon presentation on persistent entity streaming: • CitizenCon 2951: Serve...
Jake Beal paper on "What the Assassin’s Guild Taught Me About Distributed Computing" from MIT: jakebeal.githu...
Overwatch Developer Update | Let's Talk Netcode: • Developer Update | Let...
Zachery Johnson talks about Blizzard's Diablo II: Resurrected database architecture on the Critical Point War Stories podcast:
• HE LOCKED PLAYERS OUT ...
My main channel: @critical-point
My personal site/contact info: free-dissociat...
![[TH] FS vs BME - VCT Ascension Pacific - Lower Bracket Final](http://i.ytimg.com/vi/wnygqsR2Fig/mqdefault.jpg)
Apparently when the guy who made this first showed Chris Roberts, he was told that if we wasn't a coward he would showcase it live. Well done.
Game dev system design is nuts. Low latency multiplayer games where you are trying to sync everything in real time is a hard problem. Thanks for the content!
IT crew use to say: there's no real time, conceptually and technically. So yes, real time is always a problem (and that's why they use the "ticks"). :(
it's impossible. SC will fail no doubts.
The reason MMO games use regional servers is that the speed of light is finite. Light travel from Asia to America takes more than 100ms, which is enough to destroy competitive gameplay. SC choose to ignore physics is beyond insanity.
@@edward3190 misinformation my friend. The goal of sm is not to overcome latency problems in terms of "letting people that are on other sides of the world play together", but rather to just distribute load by splitting simulation of a gameworld into different processes that can run on different machines. Everyone is well aware that latency will always be a problem, that's why it's highly expected (by devs aswell as community) that there will always be regions (NA, EU, ASIA, ...) in SC. Having ONE global shard is an existing idea, but no one guarantees that smth like that will ever exist (perhaps there are options to achieve something with instancing and concepts of "consistency over time" but that's it from my pov - no real time gameplay interactions between two clients that are on other sides of the world)
distance is approx. 10.000 kms at 299 792 km/s... more than 100 ms.... Suuuuure. ROFL.
@@nicodemiusoverschnaps4759 just shut it if you don't understand real life.
light doesn't travel that fast on Earth, and 10000km... wtf?
"It should not be possible" is usually followed by "How did they do that?"
Dude this was exactly the kind of professional opinion I was hoping to get from this video. Great work, great insight. You got another subscriber here!
The distributed lock and all that is cool, but the really big job was that all state had to be externalised from the game code and copied into the replication layer. That is *all state*, because a server has to be able to take over in-flight or take over from a server that crashed at any moment. So essentially, server meshing changes the whole code base, all code in the game servers. I saw another hint in some other presentation about how things work: They said explicitly that the entity graph is distinct from the replication layer. I agree that the entity graph might well be Neo4J, but I think the replication layer is something else, probably in-memory. Also, in last year's Chairman letter, CR mentioned that the replication layer is horizontally scalable and redundant, so the replication layer does need to handle distributed lock and replication in a cluster.
BTW, I have founded dev companies so I can guess why CR cried when it worked: CR sent off this team to build server meshing many years ago knowing full well what the job would require (he is a good dev on his own merits; he wrote most of Wing Commander himself). Starting a project the size of server meshing is a "bet the farm" project. If the dev team had not been able to get server meshing working, CIG would quite likely be over. So CR had to trust that the dev team were able to do it, entrusting them with the future of CIG and let them forge on ahead through all the delays and the failure of iCache. That is a nerve wracking position to be in when you are responsible for 1200 people's livelihood!
could be memgraph
and when people are constantly calling your game a scam
Thanks for your insight and context :)
My man had a Stargate reference in his video. That always deserves a like. o7
And Enders Game with the Ainsble
Originally the ansible was Le Guin, from whom Card borrowed it, but yes I am a big ol' nerd and have been forever ;)
@@free-dissociation Ursula! Didn’t know lol
Well this example is a very small room. The key is that currently, one server has 5 planets and several moons. Imagine if one server only had to account for one planet. The data strain will be greatly reduced. The transfer latency between planets or moons wont have to be instant as shown here, because planets are so far apart.
Yeah this is something that I wish they had made more clear.
The way I’m picturing this is in the game there is no reason to have three different servers handle three different rooms in the same building, this was just a scale model to show the technology.
Ideally, a server would take care of a large area (as you say, could be a whole planet, a station, etc) and then we wouldn’t notice any lag because we’re in quantum when it happens.
That would also mean the example he mentions where you’re shooting something across three servers is infeasible because of scale.
They don’t mention this specifically on the conference though, so it’s kinda up to our own interpretation. They could’ve said “these rooms would represent different planets, or a 1/4 section of a planet” or something to give a sense of scale of how it would actually look in the final game.
Though the end goal is to have the mesh dynamically scale based on the number of players in a region. If a thousand players show up on microtech, multiple servers are going to have to take care of a single planet. And if they all show up at new babbage, there may very well be situations where rooms are divided up into separate servers. Maybe the designers will make it so server boundaries will never have situations where you can shoot through multiple servers when indoors.
Because they demo shooting across server boundaries I kind of infer that they actually intend for that to be a thing in the medium term.
@uttrik this person gets it!
@@uttrik Oddly this is something MMO's have to deal with and are unable to deal with. Case in point just look at Final Fantasy XIV Endwalker's Login issues, 5000 players all trying to log in at the same time the servers were unable to handle it. Now if this teach was in its growth/maturity stage something like having login issues for MMO's or dealing with crowded areas would just be a thing of the past because the system would handle the issues without a problem.
50:45 To answer your question: Yes, they are. But instead of having a 3D mesh to represent the object it's treated more like a physicalized particule (GPU bound). So you won't see a mesh around those. But they are physicalized.
59:19 That sound like something you went through 🤣
I'm a DevOps engineer, specialized in IAC. I felt that moment... 😅
what do the blue and yellow colors mean for your pic?
A sea of sunflowers under a blue sky. But realistically, the Ukrainian flag. @@toecutter9372
@@toecutter9372something good
> That sound like something you went through 🤣
I have stories. So many stories ;)
Don’t have much to say beyond, very interesting commentary, very cool insights. Definitely subbing in case you do more similar content in the future. Or if you get that interview!
I think it is not using a physicalized bullet, but a time constrained spline, the spline would then be subdivided for each server through the replication layer, which keep the full spline.
Ya they said they went back to hit scanning from physicalized bullets.
this is an incredibly intriguing as an engineer and amateur programmer! Also thanks for introducing me to the term "Nerd Sniping". Love Munroe
"This is not something a AAA publisher would have put up with" this is why Ive believed in this project. This why ive been patient. Theyre doing soemthing phenomenal. I want them to succeed so that all the haters of this project and those who call it a scam have an amazing game to play soon(TM).
Yes, the funding model is the real secret sauce of Star Citizen. No financer would approve of waiting this long to start selling the game if it had been done in the traditional way. All that money sitting in CIG would be money not giving of interest, so the expected profit requirement for the game would grow year on year until they become untenable. At that point the game would have been forced to publish to recoup as much of the loss as possible. It is the main reason why Forbes hates Star Citizen; the funding model entirely cuts out the financer.
I think the game already good with all the current features. Only one system but there's so many things to do. There is only two things that bother me with the current state, and that is the bugs and the NPC AI. If they fix those two, I can say the game already complete even with only one system. With this player already able to play the game, while they working on expanding to a new system and new vehicle internally until it's ready, stable and minimal bugs they can update it then continue expanding internally until ready. Just keep doing it like that, it should be good.
@@r2artCH Unfortunately, having each release mostly bug-free will slow down development quite dramatically. Think of it as building a something like a chest of drawers. You build it with some group of drawers to begin with, polish it, make sure all the nails are tightly nailed in, etc, and give it to the customer. Then you want to add new drawers. To do that you have to break apart much of what you already built to fit the new drawers. Then you have to re-polish it, fix it up, etc to make it look nice again. This is the problem with releasing code under development: If you want it to be bug free and polished, you have to redo the bug-fixing and polish every time you release and that slows down development dramatically. So CIG are doing minimal polish and accepting quite a lot of bugs, especially in systems that are planned to be replaced, to avoid getting all resources stuck in bug-fixing and polish.
this video convinced me to install the free 14 day epo install and try this game even with my poopy 4gb 1050 gtx ti gfx card keep up the content i think the 20k views show there is a high demand for this content and the format it was very real i look forward to see whatelse you will cover as far as other games are concerned. well next convention if yougain alot of traction on youtube mabey they will pay all exspenses so you can go taere nextime to citcon right? :D
Very insightful about the handoff lag.
The best way I can think of hiding handoff lag right now would be to make sure server areas are all sized well beyond the practically implications of player input, relative to the size of the game the range of your bullets is nothing.
Forget about three servers in one hallway, the smallest server zone in v1.0 of server meshing will probably be an entire planet/moon, areas separated by millions of Km.
I would expect handoffs to only take place in the travel between planets at which point the likelihood of players engaging in combat on the border between servers would be effectively zero.
As-is we're limited to about 40 people in each instance of the entire solar system, giving one server to each separated POI in the system would allow an absolute maximum of about 1,200 people (there are around 30 individual points of interest in the Stanton system).
Of course the total would still be limited by the single highest traffic point in the game, but I would expect at least an order of magnitude more players active in the system at once, once server meshing is in Star Citizen will finally feel like a real MMO.
Current instance pop is a flex number of 100. They have had 200 player servers.
@@iiKiboshii-c3z Excellent, my only info was from a gameplay video and I assumed he was correct.
If CIG servers are good for 100 players per server then Server Meshing is going to make Star Citizen really pop off.
@@budthecyborg4575 the thing is we know somewhat they they can already as people have stress tested the servers by bringing all 200 people of a server into a single ship. And barring restricting people to walk (so people would not die off running into each other and colliding) the servers were actually running better than normal in this scenario.
It's because right now the cause of server load is due to all the players being spread out everywhere, meaning the server has to load and account for basically every major landing zone and most points of interests at the same time. All those interactions stacking on each other until something bad happens.
Essentially, CIG's streaming tech can't be utilized to its fullest server side as long as servers functionally will have to keep most areas loaded.
Yeah, it's certainly possible to set up server meshing so that it's very rare for players to cross servers and thus hide server leg. Because they demo'd interaction across servers, I infer that they actually intend for that to work and be a big part of how the game works at least in the medium term.
@@free-dissociation My bet is they need localized server meshing to work before we can have a playable Bengal.
Players have speculated for years about a single Idris meshed across multiple servers but that would never actually be necessary just for an Idris, the Bengal on the other hand is likely to rely on this system.
51:20. I think there is no bullets, just raycasting outputs from player entity with saving the angle (vector) when leaving one zone and receiving it by another zone
You also need a timestamp and an owner, bc the player isn't on the second server.
Very unique take on the topic. Good to hear from ya
I have a feeling that you may have misunderstood the job of the replication layer in all of this
the way I myself understand it is that the RL is there to replicate objects for other servers on other zones
so that each server can do the physics calculations for its own zone of objects, and it just displays the results of what the others did
so its "replicates" objects for the player, and thats how you can shoot across boundries.
When he had the graph representation up, I don't remember seeing a bunch entities being spawned off the zone node, that you'd expect to see if they were physicalized entities, like other entities. Wonder how they are handling that?
Other commenters tell me that CIG moved from physicalized bullets back to hit scanning.
@@free-dissociation Ah, makes sense.
@@free-dissociation I don't think they are doing hit-scanning in the immediate-hit-where-you-aim-sense. I think they are doing bullets on the GPU. At the least for ship combat there is a hard requirement to follow where the shots are moving (those beautiful arcs of fire between ships), so for large scale fleet combat they still need to move bullets between zones. However, I have some vague memory that even for ground, they still trace bullets through the air, they just have some GPU trick to draw them very quickly rather than traditional geometry. BTW, this is easily testable with a sniper rifle.
Overwatch is a bad fucking example, because despite the fact Blizzard said the principle of "Favor the shooter" is in place, what they don't tell you is the server tick rate is 60 or 64hz. So if you are running 120 FPS constant and you take a shot, you miss because the server (aka Replication Layer) misses 1/2 of the information the client sees. They recently changed Sombra's translocator ability which is directly effected by server tick rates. You can do a 180 degree turn throw the translocator, and the "Replication Layer" only sees you turning 90 degrees and as a result your translocator hits the wall next to you instead of where you were actually aiming on the client side. This is the same reason people claim to be behind a wall or corner and they die in a different location than what they see on their client.
TL:DR Overwatch needs a 120hz+ server tick rate to make the game playable for most clients considering how extremely low the hardware requirement to play Overwatch actually is. CIG might be able to get away with this issue at 60hz tick rate because the game requires beast hardware to run. And it is not even optimized yet.
The big worry i had was latency. But then I thought if they cluster the various servers running the replication layer and shards (assuming shards would take on the role of each hosting a couple of containers) that moving from one shard to the next shard should be pretty fast and have pretty low latency between each other even if they have high latency between the servers and us. they may have to figure out clusters and probably if enough people showed up in one spot it would leak outside of the clusters to find any available shard to host containers.
I'm curious how well it will deal with a ship fill of loot from bunkers when it transitions between servers/shards
Yeah that's the real pickle. How does one do that with so many children? It seems possible but when the electrons hit the wires... Like FD mentioned. Protocol is everything if it is to be playable. Simple compressible and elegant.
Not really. The entity within entity within entity capability of the graph database means you just move the shi entity and the associated entities within it go along. This can work for the ship entity as easily as a container entity.
Cool video, Thx! CIG is doing big things, but I still have my skepticism that they are as far as they want us to think! Sidenote: dont worry about background noise. Its narely noticeable!
There are a few glitches in the demo, for example the 'box' in the middle left in the green zone (bottom right view) moves at 58:34 (player in the first person view shoots it) but it does not move in the fpv nor the other 2 views.. and then it suddenly re-appears again in the bottom right view at 1:00:31 - hmm?
I see that too and may be a bug. They did just put this all together as the first iteration of server meshing so I'd imagine they need to tweak the code to get it perfectly in sync with all the different layers of the simulation.
I'd imagine you would need to do some complex checks and those are done on the server side as opposed to the client. These checks will likely be involved with latency between servers as well. It could also improve with time as they work out the tech. It's not so much about running into issues with simulation as it is probably more about how you handle those issues since you will have latency, memory pressure, CPU throttle, crashes, etc. that you have to account for in code at replication layer and simulation layer.
Will just have to wait for the next iteration of the tech and see how dialed in it gets. As it stands its very impressive that they have it working to this degree.
ooh hmm. thanks for pointing those out. I'll have to go over this and see if I think it tells me anything more about there implementation.
The prinicples oft this technology could change how we think about multiplayer.
I been a backer for about 4 years and I'm with you buddy i have a potentially unhealthy obsession with this game myself lol..even when i take a break from playing i watch a LOT of sc stuff lol
So interesting you mentioned the interaction at a distance, this was solved by another massive multiplayer game by making ordinanace that could possibly traverse withotu the player like large bombs or whatever, ownership was passed over to the server, and thereforce would just act in CIG case like a server owned object passing through each zone. This also allows objects that impact large numbers of people independant of the player, this is also basicalyl how i assume a player leavin their ship in an are would be handled if player despawned in combat etc, so I think this will be built in by default.
A minecraft mod called "Create" pulled off a similar thing with trains - travelling through unloaded world chunks. Since the rails are curves, it can follow them regardless of the chunk state.
I'm guessing this pre-calculated curve approach would be the type of implementation for this game, since bullet distances are finite.
The fact they are persisting entities across servers, is just a showcase of this...
In reality... A server is going to swathe a HUGE span of the system... IF and the IF is pretty big here... players find themself on the cusp of a server boundry, yeah... Might be difficult...
But as that could be the edge of a bubble that's millions of meter cubed.. Chances are slim...
Getting to the edge of that bubble would mean you dropping out of QT at the right time
bold of you to assume theres not a lot of dead space in between, in spatial terms. One of their main goals with meshing and containers is to not manage space with nothing in it. The other half of that is to abstract out large chunks of the NPC simulation, so you don't need to spend a lot of server resources on them when there are no players to observe them.
Its not a fixed number of bubbles across space, but a bunch of small roaming bubbles that merge and break off as things move around.
@@freelancerthe2561 I don't think there will be... All that dead space will be on its own shard...
@@freelancerthe2561dead space coincidentally is very easy to calculate as there is nothing there
Right now the edges are going to be fixed. Essentially they will delimit areas of big traffic and assign servers to each.
But the goal is dynamic server meshing, meaning server authority changes based on the actual load demand. So that if a big chunk of load moves from one stellar area to another then servers will gradually cluster up in that area to divide the load. Potentially up to the divide happening within visual contact.
Everything else that does not need to be seen is streamed out, and handed to the background simulation, which is an entirely separate thing that feeds to and from the in game simulation.
We are all waiting for this tech to hit the live servers if they can pull this off. I have never been so addicted to a game ever in my life, this game is like every childhood dream I had of being able to do in a game and then some. 😂 This attempt to do this in a game will be historic for many in this industry.
Players, just like servers, have a streaming container around them. It is likely that servers only need to see around them to facilitate simulation and transitions, but players have variable needs. For instance, you'd expect a player flying at 1200m/s to need a much bigger container to stream entities early enough to avoid popping.
So to answer your question, if all 3 servers in this demo were in a straight line (purple, green, red) and bullets were physicalized, you could imagine a player in server purple shooting at a player in server red. The server purple would manage the physicalized bullet for a while, then server green and then red, potentially handling the collision with another player there. While the shooting player needs to see the result of their shot from server purple to red, server purple doesn't care. So they'd have very different streaming containers.
Would you please do a video with Mr Tybio about the 5 toughest problems in computer science CIG is taking on or 1 video for each of the 5 with Mr Tybio?
The room used in this was is just for demonstration. I doubt they will use small areas per server in end use. They are more likely to have a city as one server and maybe the inside of a large ship being one server. So movement between servers wouldn't happen as often as you are assuming.
They have stated that the meshing is intended to be dynamic based on load in the future. You may be correct that the initial implementation might not have this issue, but they have set themselves up to need to tackle this issue in the future. An example they have given in the past is a single ship, like an Idris, having multiple dynamic meshes.
"With Dynamic Server Meshing, it’s possible that large ships such as a Javelin could have their own dedicated server assigned to run the authoritative simulation for that ship and everything on it. However, we’re trying to avoid having inflexible rules about how entities get assigned to processing resources, so that might not always be the case. It comes down to efficiency in terms of both processing speed and server costs. If we had a hard rule that each Javelin and everything in it gets its own server, then it wouldn’t be very cost-efficient when a Javelin only has a handful of players on it. The same rule also wouldn’t be efficient in terms of server processing speed if there were hundreds of players all crowded into the same Javelin, as the rule would prevent us from distributing the processing load across multiple servers."
robertsspaceindustries.com/comm-link/transmission/18397-Server-Meshing-And-Persistent-Streaming-Q-A
Can someone give an idea, at what point the replication layer could struggle updating the entity graph with the appropriate tick rate (lets say 30)? Because for my understanding, if meshing would work as intended, this would become the next bottleneck, or am I wrong here. As I don't have a great understanding of game engines and server tech, I don't really know if like the entity graph update is 100 times less compute intense compared to the engine work (physics etc). Hope this isn't a stupid question...
You're right, this *is* gonna be a limitation, and it's gonna depend a lot on specific implementation details. I unfortunately don't feel like i have a good intuitive sense of where and when it's likely to bottleneck. Updating the entity graph on a transaction-by-transaction basis is definitely less compute intensive than the physics simulation, but because the global entity graph has to be able to receive and process updates from *everybody* currently playing, not just the players in one region of one shard, it's still a hard problem. Really the best practice with optimization stuff like this in computing is just "try it and find out."
@@free-dissociation I think the answer to this is that the entity graph is not the replication layer. The replication layer might be caching things for the entity graph. The replication layer is also redundant and horizontally scalable, and the entity graph is the persistent storage. As long as the replication layer is able to scale horizontally, it can take the load for tens of thousands of simultaneuous players in a single shard.
the replication layer uses a queue to update entity graph I think, per 2021 cit con presentation at least, shoudn't have a fixed (30 ticks per second) update rate that way. Your other question I don't know enough about how this works to answer.
We saw a demonstration that carved up the game space by 3d spacial boundaries because it's easier to visualize in a demo. However, given what you know about the tech, what additional technical challenges do see regarding removal of the spacial boundries for servers? The use case being that multiple servers could then handle a single 3d gamespace, thus potentially increasing the max player count in close proximity.
You're correct about the fact that running all servers and the replication layer on the same machine, it eliminates the latency issue. That said, the latency between game servers and the replication layer, assuming running on the same data center, should be negligible. Assuming the game servers simulations run at approximately 60 ticks per second, that gives around 16ms per tick. With an average network latency of, say, 1ms within the same data center, and standard client-server synchronization techniques applied (client prediction, server reconciliation), it is very possible that no latency would be visible in this very simple demo.
Wellllllll. It doesn't _eliminate_ the latency issues, unfortunately-I wish it did. There's all kinds of nonsense that gets in the way. (Whatever nonsense the scheduler has decided it's getting up to that day is the one that I remember biting me personally.) But I'll certainly take same-machine latency over even same-datacenter latency.
Awesome dose of information! Thanks!
Great video brotha. Dont worry about background noise :) not much you can do abiut a hungry roommate.
It's good to watch this after attending CitCon with my buds that arent in tech. Im neither a game dev or a regular developer, but as a systems admin I have some familiarity with data systems. Your comment about localizing the inter-server communication to something like a kubernetes cluster is spot on, if i had to guess. This falls in line with their initial idea of geographic shards (e.g. a shard for NA, SA, Europe, and Asia, etc), but that's gonna be a huge undertaking if they want the solution in finality to have everyone on a single shard. The laws of physics work against you there, so im very curious to see what directions they take with minimizing the latency on handoffs.
We are already able to play in distant regions if we like (you can select region manually when joining the game) and it works, though admittedly 0.5s ping is a bit "bouncy" 🙂. Nevertheless, unless you are doing combat, it is usually fine. An issue with shards is that they are so isolated from each other. You will never meet people who play on other shards. This is an issue for large global orgs with people wanting to play in their local shard, yet hang out with org members from other parts of the world. One solution could be to take advantage of the Amazon infrastructure and high performance networks between their DCs to set up a "follow the sun"-shard. So, if you log in to the "follow the sun"-shard, you will be in the region that is currently in it's prime time, but as the 24 hours move on, your server gets migrated to new DCs around the globe. There might also be local shards, for those that always want perfect ping, but if you are OK with high ping during off-peak time of day, you could make your home in the "follow the sun"-shard.
you had 4 minutes of talk before the actual content of the video started, i'd recommend you to keep that a little shorter (cuts do help) so people keep watching. or you use timestamps so people can jump directly to the mainpart what means a lot of people will not see what had come before ^^
i don't have time to listen to this right now but chapters or timestamps in the description for the main points you wish to make would help a lot
Thank you and well done
I think you just became a Star citizen youtuber!
Was this live? I'd be interested to watch the show live with you on twitch or youtube or something in the future.
Yes, it was done live at the event.
@@thePrisoner1000 not what he was asking.....
I think it is about 10 hours total. I cant find one thing they talked about that i thought was "bad" or "missleading"
And i am a bit picky.
The presentation was live on Twitch originally then uploaded to YT, this reaction was a recorded video.
Problem 1: I suspect that there's additional communication between each of the adjacent servers that allows them to notify their neighbors of any raycasts that may potentially penetrate an entity not under their authority. The question is what happens when there's latency between each server? What if the bullet has a large enough range to go through, say, 3 different servers? That would make the experience even *worse* than using a single server because you'd be tripling the latency cost. They'd definitely need to avoid placing many of these zones in wide open areas.
Problem 2: If there are players that know the boundaries of each of these servers, what happens if that group all decides in a single moment to move between each one rapidly? Depending on how computationally expensive the "trade off" algorithm is a large enough group of players could potentially cause the replication server to lag or crash outright (or even cause a nasty deadlock depending on how they have things set up).
Problem 3: This is a space simulator that allows you to travel hundreds of times faster than the speed of light. What happens when player 1 leaves server A and transitions to server B, but before the replication server is able to handle that and server B knows that it has player 1, player 1 is now transitioning from server B (at least it's zone), to server C, but it's still under server A's authority?
Does server A still claim to have player 1 and so update the transaction to say server C instead of B? I'm sure there's something that could blow up there in the wrong circumstances, or potentially a teleportation exploit. Maybe I'm mistaken.
As with all things new, something's bound to break! But G** bless them for being the first ones to try!
> The question is what happens when there's latency between each server?
And the first rule of distributed systems is, there's *always* latency between servers.
> If there are players that know the boundaries of each of these servers, what happens if that group all decides in a single moment to move between each one rapidly?
This is a pretty common tactic that EVE Online corps have used to turn battles in their favor, from the launch of the game, so yeah-a big problem, and one CIG will have to address.
> What happens when player 1 leaves server A and transitions to server B, but before the replication server is able to handle that and server B knows that it has player 1, player 1 is now transitioning from server B (at least it's zone), to server C, but it's still under server A's authority?
Probably, the client is not allowed to initiate the transition to server C before the server acknowledges the authority transfer to server B. Hope they tested the heck out of that state machine!
7:16 CIG dont have a habit of avoiding bugs during live demos... where'd you get that? It's on the bingo for one of their live demos to crash. XD It almost always messes up for them, the amount of times its crashed entirely during the demo has been a meme. lol
It would be interesting know when a 30k Happens is the crash a single service on a server or the physical server. That would drastically change the tech. If it's a service or VM that crashes you have the replication layer on the same physical hardware. If the replication layer is on different hardware the transitions will be much more impressive. not to mention whatever tech they have for spinning up new "Servers" as needed.
It's just the game server service process crashing, it's not the underlying hardware. (Think how much that would have cost CIG back when we were getting 30ks every day in game if it were the hardware ;)
@free-dissociation I agree that's how it works currently. But it may not be in full server meshing.
We are star citizen players and we want your opinion on server meshing. 07 see you in verse!
transition is most likely when its almost done- if you look at armistice zones, its basically the same how to server detects once ur in or out
you enter when you are fully in and leave when fully out
They have had bugs on every demo they have done for their live stuff tbh. Their video stuff course not so much. But even the pre-recorded video for Star Engine was buggy which why they been fixing it since :)
SeconfLife has been doing this for a very long time. Not saying that they do it better or that their engine would be up to running such a complex game like Star Citizen, but there is something to be said that they had this distributed model running more than a decade ago. I wonder how the two solutions compare?
They do not.
@@CSKefka They do to.
Yeah, EVE Online as well. I know a tiny bit about EVE's internals and nothing about Second Life's and would love to talk with people who know them better. EVE at least does handoff between servers at the solar system level, and it's pretty clear to me that CIG want and need something much more fine-grained here.
@@free-dissociation Yes, definitely finer grain for CIG. From wikipedia "Each full region (an area of 256×256 meters) in the Second Life "grid" runs on a single dedicated core of a multi-core server." I was originally thinking they were doing a region per server instance, but it looks like it is a region per core. Interesting. So the big explosion of core per cpu, probably really benefitted them. I have a dual cpu EPYC server running in my rack at home that has 48 cores total, that I pieced together off of eBay from Chinese decommissioned hardware for like $1k. It is like 3 generations old EPYCs and used RAM, but it runs like a champ and I guess it could run 48 regions of SecondLife, crazy!. I imagine their server footprint, to support the same number of users, gets smaller every year.
@@sevilnatas And in this day and age individual cores have gotten so performant that it's entirely possible they could run multiple server processes per core. That was one of my surprises from the interview I did with Zach Johnson-he said that Diablo II Resurrected was able to run that way: th-cam.com/video/UupPvHocUQg/w-d-xo.html
How could I get a job with cloud imperium games?
I mean that I don't have that many skills and I am in Jordan so the only place I can can learn from is the internet
Security IT engineer talks about game :D
Please tell me your uncle is Jason Issacs, because that would make perfect sense in this reality. BTW thanks for your expertise and taking the time to unpack all the details that even CIG's show cannot do. +1 sub
I would be curious with how big CIG decides to make these zones. If the red zone is 1 Million meters across for a whole planet, you wouldn't be able to shoot across three zones like you had mentioned. The weapons and scanning don't have those kind of ranges at the moment.
the solar system its self is a giant zone, they can nestle everything .. they just need to link them all to servers now.. fiddly but its more or less done :))
As someone who doesn't know much about programming, what does this mean in regards to server player limits?
Say the server limit is 100 and call this demo a city. Without meshing you could only have 100 people in this city so would only see and interact with 99 people. From what I gather of it, with the city now being 3 zones and a different server having authority over each zone then you could have 90 people in the purple zone and 60 in the green zone so if you were in the purple you would now be able to see and interact with 149 people and not just the 99 you could when it was just 1 server with no meshing.
Is that correct or am I way off?
If that is correct then the servers still have the 100 limit so what happens if there are 100 people in the green zone and you move from the purple zone to the green zone?
You're correct! The goal here is to be able to substantially increase the player count on what we've called "servers" up to this point-what CIG are now calling it a "shard", and it is made up of multiple servers.
Ideally in the case you describe you can see and interact with all 200 people (100 in the green zone and 100 in the purple zone, each on their own server) seamlessly as though they were all on a single server.
@@free-dissociation So what will happen when you move into a zone which is already full? Is there some kind of dynamic system where it will split that full zone into smaller zones (which now have their own server and limit) until there server in control of the area your entering is no longer full?
If something like that is the case I would think there would have to be some kind of noticeable latency with the amount of times they might have to move people from server to server if an area is full and after the split other people have already entered it making it full again and needing additional splitting and so on o.O
in future, since you're recording a youtube video in 4k.. you should either upload in 4k as well so we can read the text in the video easier.. otherwise reduce the source video to 1080p :)
A mounted SHURE SM7B will help with your roommate situation. You will want to set a gate though. You can gate through OBS but having a gate on the audio interface is nicer for monitoring
For those who don't speak Engineer "brave choice" = "bad choice" unless you know something they don't...
I think alot of changes of aproch over the years have come from a shaky architecture.. Originally I think the engine and data sets where being optimised for optane, possibly with SPDK.. Now though I think they have had to migrate to CXL, this will possibly be better latency and scale wise, definitely it will be better for support and implementation going forward though.. 👌
Also.. Shooter calls the shot I don't think will be relevant as the shots are raycast and outcome it self is authoritive and know instantly by both parties.. 🤔 (Will still be a micro second faster for the party taking the shot but not in any relevant way compared to older z axis rounding calculations which does have its inherent issues in group play)
You are a citizen. We are here and held the line!!! o7
Why there will be huge latency if both servers and replication layer using same infrastructure inside same claster?
"Huge" here is relative to the latency of doing this in the memory of a single process. The rough rule of thumb in systems is CPU registers -> CPU cache -> RAM -> Disk -> Network, each step is an order of magnitude slower. Lots of asterisks as tech has evolved but that's the intuition.
I'm not really surprised that it works in a test setup. You can get anything to work in a test setup. What I'm wondering is whether it will keep working when this gets upscaled to thousands of simultaneous players and millions of individual objects being streamed between different clusters and the interactions between all those entities. Time will tell...
player can see it because the replication layer is now showing all clients all the logic ,, but the server its self has streamed out the data.. saving a fuck load of memory
the replication layer
1:01:30 In prior interviews developers have talked about how they have a desire to make something amazing in gaming, from what their choice of words and expressions showed it was their way to 'make a mark on history'. They then go onto talk about how they never would get those opportunities with other companies like AAA game studios because of deadlines and quota's thus the ability to work on a "passion project" while getting paid for it can be a large reward in itself instead of millions of dollars, notoriety can also be very profitable in the end.
I don't mean this harshly by any means but just so you know.
Thank you for the video and your thoughts & opinions, I always like hearing from a third party because you approach it from a different scope which can lend to ideas/thoughts that may have been overlooked.
Also with those views you may as well make more content like this, might even get paid for it after all.
They run on Amazon cloud servers, And the internal high speed network on the datacenter should minimize latency between servers
You could pool objects such as bullets so they are just reusing bullets instead of creating and replicating them across. Just pool and use them and send them back to the pool.
my english not that good
you said youy make perfect the enemy of the good?
if so you just described my life
haha mine too, all too often ;)
"distributed system is the hard part" That's why we have all been waiting for this game for 12 years now. I can remember way back at the beginning when CR was talking about having 2 cap ships fighting and each was a server and shooting from one ship to the other would involved hand offs between 3 different servers at the shot moves. That is why this demo might be the biggest step in the development of this game. This demo showed everyone who understood and has been waiting that it may actually be possible.
You don't worry about latency to much in a datacenter like Amazon between servers today if you pay to want this bandwith. Where I work our servers hand off highly active memory between servers when they switch hosts in the a network easily with no notice of latency really. As long as you keep the server in the same building or nearby buildings because distance with the speed of light that data travels is a thing. The real deal is the speed that the replication layer (very high speed database connection) service can handle all the traffic from all the static servers and in the future when server meshing and it will be object containers like outposts, cities, planets, ships or part of the solarsystem like shown in the demo, where it was one room divided into 3 object containers. But this tech I can't see why can't be scaled up as long as the replication layer can keep up. CIG really are doing what every MMO game company would dream of doing but won't invest the time or money on (we are talking minimum 5 years of R&D on a dedicated team at CIG and external database company). And also the bandwith/speed today in modern and good networks today in the top hosting centers are way faster than a single computers memory speed/bandwith.
> Where I work our servers hand off highly active memory between servers when they switch hosts in the a network easily with no notice of latency really.
Interesting. Is this like a compute host which is shutting down transparently snapshotting a running VM and transferring the snapshot to another compute host where it starts up and seamlessly takes over without user-visible downtime? I know there's a lot of wild stuff that's possible at the hardware level, increasingly supercomputer level stuff, but I haven't gotten a good read how deep in bed CIG are with AWS to pull stuff like this off, or really how much they need to. And IME often the cloud providers _promise_ this but whether it _actually works_ is another question.
And yeah, when I talk with folks at other MMOs companies it sounds like they haven't invested the R&D and are running stuff in paradigms that are 5-10 years behind current, so I'll be curious if CIG are really pushing the envelope like that here.
My thoughts and question:
- bullets: while they are going through the RL so that other servers see it, I doubt they are going to the PES graph db (= no write spam), I assume they ban some entities that have a short ttl and no purpose.
- your comment about latency on real servers is true, however a DGS will usually serve maybe a system, a planet, or a city. Cases where it will be assigned to a space ship will probably be rare, and I doubt they will go "lower" than that, so we will rarely observe any latency from that. If it comes to indeed splitting ships/UGFs into multiple servers, I'm sure there are some solution (your idea with k8s pods on the same machine is a good one tbh)
- question: when sending objects between two adjacent DGS, I understand that both have a copy of the object, and only one has authority. Do you think both DGS run the simulation for that object though, when they are close to the "border" at least, so that handing over the authority is as smooth as possible?
Thank you for your videos!
> - bullets: while they are going through the RL so that other servers see it, I doubt they are going to the PES graph db (= no write spam), I assume they ban some entities that have a short ttl and no purpose.
Yeah agreed that bullets aren't going to go back to the global entity database. Even there though the write spam on the replication layer strikes me as high, and my read is that they *have* to go through the replication layer. It's possible CIG have worked out some hack with raycasting as other commenters here have suggested.
> - question: when sending objects between two adjacent DGS, I understand that both have a copy of the object, and only one has authority. Do you think both DGS run the simulation for that object though, when they are close to the "border" at least, so that handing over the authority is as smooth as possible?
I think naïvely I would try without this, just doing the handoff between physics simulations when the authority gets handed off, to simplify the process. I can imagine there being enough randomness in certain kinds of physics simulations that two servers might simulate the same object very differently, but there I think the better solution is to make the physics simulation behave as close to identically across all servers by e.g. stuffing the necessary random seed into the object data.
@@free-dissociation Oh good point about the randomness of physics sim. I guess we'll never know 100% how things work unless someone from cig tells us, but I would love to have all these details.
It sure is fascinating to watch your channel and see how someone else thinks. I work in IT but nothing to do with gaming.
Subbed because I wanna see you lose your damn mind when it's in-game! 😂 GO, CIG!
Is this technology something that is ground breaking even for worldwide day-to-day life operations like product inventories for big retailers, warehouses, logistics, finance, and definitely other small to massive games that might benefit from server meshing? What CIG have accomplished, would it have a moderately big or massive impact in some industries?
My read is that it's pretty limited to virtual worlds-ops stuff like you're discussing is already pretty well handled by existing database tech, and this doesn't improve on that. But, if this works, then many other virtual worlds at the scale and complexity of Star Citizen may follow.
"U didnt have all the pain, im sorry"
To be honest you should show a bit more of respectoto CR. HE is himself programer and he did himself a large portion of the Zone System which is the fundamental piece of tech in which this is holded on, and the only rock solid thech in the entire SC world..
(and he worked on several other pieces of thech in SC, I just named one because its mentioned a lot in this video)
You should really see the 2015 ish interviews of CR in several youtube channels, I remeber one in gammer nexus and such, and he going REALLY into technical details on how he (and they) did the Zone Sytem. I can tell you that he went way more technical than u did in your videos because he knows what he is talking about.
Lets also not forget that he was programing and doing games even before game engines existed.
This has to be AI + quantum combined. This is beyond human comprehension. Besides that, there is a curvature visible. IMO😵💫
*Ok...this is the Oasis🤔 Right?
I´m really curious on the full implement on HDR, DLSS 2.0 (3.5) Raytracing and Vulcan. Did i remeber correctly that they said q4 2023 for those thingis?
To be honest. I think the way at the Moment ist Just a Server maybe for a Planet and surroundings moons.
So for the Moment you would have 4 Servers vor crusader and 4 for Pyro.
And the Transition ist easy hid in the quantum jumps.
So 100 Players in every Planet System is quit good for now.
And if you need you seperaten a moon for more Players there.
And If a Planet is empty from Players you Stream the hole Thing Out and shut the Server down.
Throu the quantum jumps the streaming ist very good Hidden.
The Same way you could so spacebattles in a bubble.(Server)
So i dont think they will start streaming every room an Split two rooms in two Servers when there are Just 100 people.
Have a good day Guys.
I don't think they have build the test level that way because they can't interract with an object accross a server (from purple to red through green) but because it illustrated better the difference between client streaming and server streaming. Because of proximity and line of sight, if all 3 zones were in a line, none would have been unloaded and they wanted to show in lemons terms what server meshing is doing (and I feel they fail for the most part).
Regarding getting across a server (from purple to red through green again) I feel like it should be quite easy :
Imagine all zones in a line. I'm in purple and the replication layer give to purple all assets in green and red. You can suppose that hit boxes are included (as they demonstrate already) so do a raycast from my gun shooting to see if I hit anything. ho... nothign in green... ok... ho I hit objectRedX in red... my client send to the replication layer : hit objectRedX.
The replication layer send information to served red (the same way it did for server green) and voilà.
and raycast are cheap and efficient so.... why not.
What's your thoughts on that ?
I took that they were deciding which data should be streamed in to which servers based on player presence and physical adjacency rather than line of sight, so if all the zones were in a line as you describe and both players were in the zone at one end, they would actually see the objects in the zone at the far end get unloaded. But I could have misunderstood that.
I understand the same. My comment was in context of that demo where the 3 regions in a line would be always loaded because of proximity. you would "see" objects in the red zone from the purple one. unloading based on adjacency alone would create weird unloading for the player@@free-dissociation
@@daganael There are things called vis areas (from what I recall) , these things especially for static environments are set up in advance, and even on ships. It would make sense to drive things based on that, both on the client and on the server->server
You can see the transition latency at just before 56 minutes in when Ben jumps crossing into the red area. That is going to be a real trick to hash out further complicating the already complicated. Doing that a thousand times per tick in say a fleet battle that moves from one zone to another is going to be monstrously difficult without wasting a lot of resources. A better idea would be to dynamically adapt zones (In a give and take between servers) based on context and call load something like you mentioned on a protocol level. In space, you can see as far as your angular resolution will allow. With fast projectiles you can touch them. In order for large fleet battles to be possible, I can't imagine any way to do it but with intelligent adaptive zoning. Easy peasy lemon squeezy for a single player game. Whole different animal for MMO. Perhaps AI could be of use as a co-processor to the replication layer to adjust zone load dynamically and feed that back to the actual servers and clients (without hallucinating) while doing that in literal milliseconds. Latency budget of any game is as close to zero as possible but if you get to 100ms territory, you've already broken it. It has to feel like at least 30ms latency or lower on client assuming you have a perfect connection.
The idea is super cool and that it is pretty much seamless is not trivial, I imagine, but holy hell what bandwidth requirements that must balloon into when you're trying to simulate this granularly. I feel like doing all of that and keeping it seamless is like borderline insanity pushing the limits of the actual internet's infrastructure especially in terms of latency and stuttering problems like Ben's border crossing. That doesn't even include gameplay "what actually happens in game" problems they have to solve.
Solid take. Kind of confirmed what I already suspected or at least we have the same suspicions, yours obviously more knowledgeable. I love that there's an Assassin's Guild at MIT of LARPing nerds who pull out the stops and make everything feel super immersive.
Makes me think of optical circuits and their potential. In phase 1, out of phase 0 not just one at a time but for each band of wavelength. The transition is the trigger just like a very very tiny interferometer. How to apply those principles to something as stable as silicon. Hmm... Maybe some day we'll have x-ray or higher frequency circuits and look back on the archaic use of the electron's influence to do calculations.
😮 Amazing. This vlog looks like a West Anderson movie feels... Maybe this software engineer should moonlight as a Cinematographer? Seriously.
I do appreciate your insights and agree it is a big deal. No Triple A(All)A(Acumen)A(Absent) would do this.
can you please teach me computer science? :D
A game called Starbase already did server meshing in a way before this. Though, that game is completely dead.
Hi Kevin. Most comments that speak authoratatively on server meshing just don't actually seem to grasp the essential realities regarding just how descreet, elastic, and granular the actual back-end systems are going to have to be -at scale - in SC. A 'server' as is commonly thought of just is no longer possible. SC will have a cloud solution which divides and subdivides so that a given 'base physicality coordination server' might contain the entities that could be a 1,000,000 km cube or a thousand 1000 km cubes of the same volume region, depending on activity and entity load within it. The entities required are streamed into each tasked server as needed. The 'replication layer' is another elastic subset of servers which handle the graph database work. They did not go into that in the presentation as it would have only confused people. Especially as the entity offline storage layer beneath the replication layer which holds the master entity records which are 'checked out' by the replication layer servers, updated and saved every x ticks is yet another distributed server system so that a replication layer server crash only risks losing a fraction of a second of updated entities or at most a second or three in the absolute worst case. These replication layer virtualized servers in the cloud must also be elastic and subdivideable and spawnable so that they stay ahead of latency, just as the 'physicalization' servers must do as well for the same reason. SC servers must work this way because they cannot 'cheat' like most games do. The gameplay operates within as accurate of a simulation as they can achieve; not a game that pretends to be a bit simulationish. There are likely to be other layers of servers connected to the physicalization and replication servers to handle simulation systems to run atmosphere, wind, volumetric clouds, weather, water, and other simulated resource systems, for example. All these are feeding results to the main physicalization 'servers' coordinating with the clients. Plus, it seems likely that those others have their own entity graph databases so that everyone sees the same elephant shaped cloud form and 'feels' the same wind consistently. Now add more layers for the Quanta system for NPCs and their simulated related activity, virtual or physicalized.
Ugh. Everything old is new. The replication server is nothing more than a glorified IRC server with a messaging and hierarchicy and filters. You could also describe it as an event system similar to windows or Javascript events. I don't understand why it's taken them years to re-engineer something that's open source and stable already. Though I will say, this is the most fun part of the entire Star Citizen project. The real work will come from it's application. If it takes 5 years to build a chat server, the stuff attaching to it should take another 50.
You say "right heavy" and I'm not sure if you mean "write heavy", because both of those are valid things to say but have different meanings.
Is that in the captions? It should be "write heavy"-I let TH-cam autogenerate the captions, and they are never great. Thanks for the heads up, I'll go fix it.
Lol
Aren't we all
Lol
This game is just evil for your wallet 😅
Hey, most microwaves will 'Silence' the beeps if you hold down the '2' button for about 5 seconds.
and hold it back down 5 seconds to turn it back on
Give it a shot :)
Thanx!!! that worked for me.... 😁
Stop saying sorry for things in the background. It's hardly noticeable.
CIG isn't forth coming on their failures, and only give us Soon(tm). It's good to have an understanding of what is going on.
All this begs the question: Aren't we all more interested in 'how it works' than the actual game itself? ;)
Both
I have been known to say that I like Star Citizen kind of in inverse proportion to how polished, 'content'-ful, and triple-A it gets. *Way* more than half the fun for me has always been knocking around it trying to figure out how it works this week. ;)
Great commentary but ffs man, 4 minutes of rambling to get to the content...
One thing that bugs me is how overhyped backers get over the slightest progress. When the question was asked ”does it work at scale” zyloh said that it does. Zyloh is not a software developer, but everyone took his word for granted. But if it actually worked at scale (particularly the scale needed for this game), THATS what would have been shown.
This presentation was just oa visual step further from last years citcons powerpoint presentation of how they imagined it would work.
I fucking hope that they pull SM off so we get this badass game. But backers are too fucking naive and give CIG more goodwill than they deserve.
They already tested it in June with Evocati in very limited capacity. That's how they knew it worked.
@@Billy-bc8pk Keywords there being "very" and "limited".
SM is nowhere near proven to work at the needed scale.
We can disregard Zyloh as an authority; that's fine. The point is that they had a plan and they've (mostly) implemented it. We're excited to see whether it fulfills the hopes. That's not hype. It's the risk we knowingly took to advance the state of the art. And it's the risk nobody else had the balls to take. You're welcome to doubt, but all it will get you is a job at EA cranking out the same old shitty games.
@IdleWorker It worked with all the necessary systems of the PU in place, and the results -- from the leaks -- indicated that it was a success, hence why they moved forward with a demo (and optimisations) they worked on over the summer in preparation for CitizenCon.
But we'll get to test it for ourselves in the preview channel next week.
@@Billy-bc8pk Yeah but the problem is that on a smaller or "limited scale" you can get any system to run fine. But there is a reason why we dont have shit like resource management systems in yet even though we got to see a vid on it like 3 years ago.
They havent managed to make it work at scale.
Same with SM.
Im glad we're gonna get to test the systems soon, but dont be surprised if the system crashes and burns and they have to start over (remember icache?).
Again. What we saw at citcon in regards to SM was just a 3D presentation of the same powerpoint presentation we got at last years citcon. So curb your hype. And dont give CIG new money for concept jpegs.
ja so he dont know anything i wasted my time
Trying too hard
Who? This guy or cig?
server meshing is 20 years old tech in EVERY mmo
nothing new
K. Would like to see your 20 year old example of shooting something real time from one server to another with another player in the background hopping around.
Im not a professional programmer but have been to a university for programming and maths and i programmed this type of tool almost 15 years ago. So this was not at all impressive to me. Not this FAR in development. And on the 3d graphics and server development side. From scratch too mind you. If anything this example is rather very elementary. Game development has existed for a long time. There are tens of thousands of games on Steam. You are going to argue that this basic programming hasn't been utilized. Yeah, I doubt that dramatically. It's not a hard problem. It's work but it ain't hard.
if it's so simple, then why has no other MMO done it? or why didn't you sell this supposed tool to them.... The other only other game in existence that i know of that had something similar was a game by microsoft 10-15 years ago for passing off physic objects to multiple servers on the same map, which ended up only being for their demo of the game as the released version of the game removed it.
@@resresres1I did this myself. I looked at my work as elementary and classify their work as supposedly complicated. Clearly that is NOT the case. One person versus a team and game development has been going on for years. I did this based on information I garnered from many many other long developed sources. AND I am not a professional and was a student when I did this. So I don't know what to tell you but this is a bag of goods you and everyone else is buying for nothing. Insane.
@@jeremiahbond2810 let me rephrase what i posted earlier. I simply don't believe you made something similar to what they showed. If you had, then surely one of the hundreds of thousands of programmers after you would have done it as well.
@@jeremiahbond2810 The reason this is so complicated is because of how Star Citizen manages entities and resources. Having two servers talk to each other is basic - but exchanging nodes on a graph database in real time to represent 3D objects streaming in and out on the server side is a whole other deal. Whatever you did almost certainly was not what we see here - there's a lot more going on behind the scenes man, and PES is evidence of this.
@@resresres1 😂 Well good for you. Because I don't care. I wonder why everyone uses a couple game engines. 😂🤣
suggestion, watch the video first without constantly pausing, that would be reacting.. and then do a review of the entire thing with pauses.. because you are not able to properly react with the constant pauses
50:46, the bullets are ray casts which is incredible to me. There was a interview with one of the developers and he said that raycasts are working but projectiles are not working atm
sorry, do you happen to know anything more about this interview? Whose interview or a link?
"lightning bolt, lightning bolt, lightning bolt"
Fucking subscribed.
I do whole-heartedly believe that the dog-leg is there because they wanted to show the stream-out mechanic.
If they let us shoot from one region into a different region across a third region then the two outer regions will probably not stream each other out.
How that works with dynamic server meshing where you can't know what zone has "view" into other zones will have to be decided (I'd bet on distance between the zone edges)
I think the idea is that at scale, one server is Microtech, the other server is Arccorp, and the third server is Crusader. The distances between server zones is so vast, that this won't even matter to begin with and the idea is you'd only ever need to interact with things in the nearest server over anyways, and never need to reach through the next server to reach the third.
I agree the dog legs purpose was to allow them to showcase what happens when game objects go out of view.
One thing I think keeps getting overlooked is that from the engines point of view all movable objects have an origin point in the parents reference view. So when you move an object you only need to update the objects origin point, and not every point that makes up the mesh.
The boundary box for each server mesh then can be a simple coordinate check if that point has crossed the boundary and can be handed off ownership. This prevents a lot of calculations and the issue of portal slices of game objects.
Then the players graphics card can do the heavy lifting of matrix calculations on the objects mesh to camera view.
It is also more than likely the same values used to reference in the replication layer and storage.
You just need to know the object ID, the object origin point in parent coord values, parent object a list of child objects, and list of mod values. I know they said it's being stored in a graph database but given objects can't be inside themselves there is no potential for circular paths and so it's just a tree problem.
this show is only meant to early wh, oh baker reassurence
@@dawnphantom2439 The end goal is much smaller, but in the demo they kept it limited to the 2-3 fields which is just a place holder. To start out it's going to be one system per node, then likely planet, then probably regions of planets, POIs, etc. down and down as the tech evolves and population increases. The end goal is for it to have "as many as needed" to handle the population of an area. Ie if one area is more popular it may have several smaller zones with each master having a further view where as a low pop area might only have one zone covering a larger area because less people go there.
Remember thease zones will be significantly bigger , think station , quarter solar system size to this issue isn’t really a big deal , only pes is the biggest thing
Is managing these transactions in a node DB compared with RDBMS so much harder , or is it that we have 40 years experience and mature methods and patterns associated with managing RDMS. I am so old I remember switching from PICK to Dbase III early ORACLE had all sorts of undocumented features. Thanks for doing this BTW , i much prefer this level of insight with house mates making late night snacks than glossy videos that gloss over the issues 🎉
Speaking from experience of converting an RDMS to GDMS It's a little of both. SQL has the advantage in write speed for realtime transactions especially if you don't have varchars fields and fully rigid rows of reasonable size. A true GDB like Neo4j destroys SQL in search/read speed regardless of database size the moment in SQL you would have to use the Join or Union keyword. The way you model and query the data are very different mindsets between the two when designing them and your apps.
@@eltreum1 So RDMS are faster for ACID transactions, but GDMS are better at reading records and significantly equally fast irrespective of size of the DB.
So will some sort of caching / queuing system be needed. Providing it was a real working demo - the focus will be to make it stable - then make it fast.
@@richardmoore4194 A couple years ago they implemented what they call icache as a caching layer that is the foundation to the replication layer. It allowed servers to crash and us not lose ship and cargo every time if our cargo made it to cache before the server crashed. GDB isn't drastically slower on writes, but you do have to plan your workloads if you want consistent speed per transaction.
EvE online did a similar thing to address the DB load/response time. The entire DB was replicated in a mirrored pair of RAMSAN cache arrays connected to the server cluster via Fiberchannel backplane. The RAMSAN also had read replicas on SAS storage arrays for backups and read queries that did not need real time priority. NVME and new gen RAM storage arrays now eliminate disk seek time mostly if you need it.
SC is a little different and a game that will play itself. There is a universe simulator that is just a machine learning system that calculates the macro economy and state of the universe stats and designated target metrics and employs NPCs to balance things as players affect systems. That is mirrored to/from the replication layer and other backend services that are like APIs in a microservice system with the replication layer as a data broker. The game servers will be assigned authority over parts of the universe and any players or NPC that travel there so the live state is ran with distributive computing.
NPCs the players can't see are running as microservice agents on other server pools streamed off the gameplay servers interacting with the universe system at the speed of a player for travel/work through APIs to interact with the replication layer. When NPCs are in sight of a player they are streamed back onto the gameplay servers so you can see and interact with them. That is part of the plan to get capacity and scale is offload the gameplay server anything a player is not going to see so the universe keeps moving. Crimes and cargo NPCs keep working to modulate the security and economy to a target in response to players messing with things.
Sorry for long post you should watch this vid where their engineer shows the stuff in action. th-cam.com/video/2muGWtX8e7g/w-d-xo.html
What a lot of laymens don't understand is that the entity view is actually just the graph database. It's an advantage of using one, like neo4j, that you can easily present the information. Most of my friends thought it was a custom tool, (which is definitely is modified, but i digress) that they made for debugging because they had never used a graphdb.
Firs time watcher. "Perfect is the enemy of the good." Never heard that before! "..the truth will set you free." Definitely an empowering statement.
You're a very soft-spoken bro. In fact, so much so that I could hardly hear you xD (edit: at least in the beginning)