How Stardock's Elemental: War of Magic Failed | War Stories | Ars Technica
ฝัง
- เผยแพร่เมื่อ 27 ก.พ. 2018
- Brad Wardell, founder of Stardock Corporation, details the difficulties in the development of Elemental: War of Magic, a game that shipped in a largely broken state. A memory error caused the development team to pull features from the game in order to fix the issue, but the concern wasn't resolved by the time the game released. As a result, the original vision for Elemental: War of Magic was never fully realized.
Connect with Ars Technica:
Visit ArsTechnica.com: arstechnica.com
Follow Ars Technica on Facebook: / arstechnica
Follow Ars Technica on Google+: plus.google.com/+ArsTechnica/...
Follow Ars Technica on Twitter: / arstechnica
How Stardock's Elemental: War of Magic Failed | War Stories | Ars Technica - เกม
I remember playing and testing the Alpha and Beta with Elemental. And Brad was responding to everyones responses on the forum and any other online source. And Stardock never wanted to abandon the game. They worked and released major patches and updates to the community for free. It was an impressive sight to see such dedication to the fans and to the final product. Ultimately it would take them several years to figure out the issue. But still i am impressed with Brad and Stardocks dedications.
i really love this series
same, same
Same. As someone who enjoys video games, technology, the idea of game development and programming, its really interesting to see the struggles some of these games had during the early days of pc gaming. I hope this series is successful enough to continue, as I feel like the history of gaming will only become more and more important as gaming becomes more popular.
Ditto and Amen.
I'm really digging the series. It provides insightful behind the scene struggles on how games are made... I must admit sometimes I have no idea what they are talking about... but I sorta get what they mean.
awesome series overall. and looking forward to more.
god damn, this is a horror story for any game developer
Let's just put the "memory fragmentation is not something commonly known" thing aside, the real lesson learned should have been: *Never solve a bug, without understanding the bug*. It is words to live by, and you don't end up butchering a game.
Well said, but hindsight's 20/20... They tested a fix, fix seemed to help, they deployed the fix across the game and it worked but it wasn't until after launch that they realized the extent of the issue. I can see unintentionally falling down that path... You live, you learn.
The only thing that probably would have saved them would have been to just to wait. Blizzard had done this many times. Like Titan and Star Craft Ghost. They may have ended up cancelling but they knew the limits of technology and were realistic about what they can achieve. Most game companies just crank them out bugs and all. I'm looking at you Bethesda and EA...
Easier to say than do, unfortunately. In programming language, there are a million ways to skin a cat. No one way is better than the other, it really just depends on how complex the system is and what you are using it for. Part of understanding a bug is knowing the bugs limitations and constraints. It's a back and forth thing. You won't always be able to identify where a bug is coming from, and secondly it may not even throw you an error. It could simply be the way the game was programmed or the limitations of the OS. They were doing waht development tells you to do like JS vets said.
Yeah well, what if you *think* you understand the bug but you don't and you only realize this later? As someone already said, hindsight is 20/20.
fix a bug... and now you have 99 new problems.
I must admit Ive gone down this route, and even wrecked a game the same way. What a great video, and to be honest I dont think much about memory fragmentation either.
Aw Brad, it's all right. Seeing Elemental fail like that after I had the collector's box in my hand... Well it stung pretty bad at the time. But I'm over it. I still have the box. All is forgiven. :)
The Mad Tinkerer I also still have my collectors edition but I have to say they handled it well and even gave us the next two games for free as an oops I’m sorry we royally screwed up majorly apology
:) Cute comment, made me feel nice inside. I can only hope I see this mentality in my player base when the time comes... Good on ya.
Joshua Durben -Didn't realize it was 2 games... Damn... Good on them.
These days companies do that to devoted fans to get you into a new game on purpose...
Remember ethics? Those were nice.
Man I remember the disappointment when it came out. But legendary heroes (the 3rd in the series) is still probably my favorite fantasy strategy game. (Actually reinstalled it last week for another go)
Maybe if brad had taken a look at manager instead of waiting until the game f**king shipped.
And that is why modern game engines have profilers, it helps zero in on the problem much easier. Even with that said, its heartbreaking to work so long and hard, to give up on so much on your vision and not be able to ship something both you and players are happy with.
Unfortunately in engine tools will not show the fragmentation, as you have to instrument the malloc implementation instead. No matter how many tools you have, there's usually one more issue that hides out of sight and needs one more tool.
Ars - this series is fanatic, please keep it going! The Thief one lacked any technical details which was a bit of a bummer tho, not as thorough.
This is an awesome format
Thanks for this story. I remember playing Elemental at launch and it's nice to finally know what went wrong
So this is why Galactic Civilizations 3 was 64bit only, Brad didn't want a repeat.
Recognizing and owning it and moving on, this series is awesome.
Surprisingly I know about memory fragmentation because my old boss had an issue with his developers causing memory fragmentation and blaming it on the servers being too weak. He had to look at the source code and prove the bad coding practices were causing all the memory to be eaten up.
I hope there is something from Westwood. I love the original Command & Conquer. LONG LIVE THE BROTHERHOOD OF NOD!
Swormi that game was phenomenal.
YES PLEASE! WAR STORIES COMMAND & CONQUER (pre-EA) :D
Good call, would be nice to hear about their trials and tribulations. Its like a mine-field out there for small studios (when they started that is)...
Yes!
EA will not agree to do a video about how they butcher the C&C series.
"If you played for a long enough amount of time, it crashed 100% of the time."
Shipped game.
Well, I guess harsh reviews and fan's rage is better than just tossing 2 years worth of hard work into the trash bin, right?
@@ScienceDiscoverer "a late game is only late until it ships, a bad game is bad forever", Seamus Blackley
Lol yeah, I had to replay that part, thought I was missing something.
@@geraq0 the thing is that was said by a company that pretty much has an infinite budget. sometimes you dont have a choice.
When you run out of development money, you have the choice between shipping and facing the consequences of shipping a bad game, or going out of business permanently.
This is the only one of this series where I haven’t heard of the game. I think I know why.
Only one where I have heard of the game, but I'm into niche stuff.
Stripping systems away from a game that is all about systems hurts like hell. It must've been tough for those guys, but I'm glad they pulled through.
Thanks Ars Technica for this great, great series!
This series is fascinating. Please keep it going!
Awesome series! I'm really happy to hear this broken down, I bought this game while I was in the army, and while I wasn't mad, I was definitely disappointed.
More of these series, please!
Please continue to expand this series. There are tons of people like me that love anything about old games :)
Memory fragmentation was my guess but only because of what was mentioned about the custom memory allocation/deallocation system. Having made a task management implementation designed for an Arduino, where memory fragmentation is a much bigger issue, I can definitely appreciate the issue. I'm wondering if you applied any of the same mitigation techniques? Such as allocating blocks of common sizes that can be more easily replaced than arbitrary-sized allocations? Should be much easier to apply something like that with all of the memory headroom, compared to a microcontroller with only 1KB to work with.
These issues can still occur in 64-bit addressing space with virtual memory since the mapping to a much more limited physical address can suffer from fragmentation issues. I've seen OOM exceptions thrown in 64-bit Linux where allocating a 256MB block failed in spite of having over 4 gigabytes free. The easiest solution IMO is to just avoid using gigantic arrays. An array containing 2^20 elements can be represented by 2^10 small arrays containing 2^10 (1024) elements each. That introduces a layer of indirection along with a bitshift and bitwise-and for random access, but it tends to be a trivial overhead in the context of the logic performed inside game loops (and most of the critical loopy paths in games tend to be sequential in nature).
Man I am so empathetic for that position, how terrible to feel that error dawning on you. thank you for having the mettle to take an interview and turn it into a learning experience for us!
I love this series and I hope you all continue to do more! This kinda' stuff is super cool in its own right, historically relevant to the gaming medium, and something people can potentially learn from all in one. It's neat.
+Ars Technica Can you do a *War* *Stories* episode about Westwood Studios in the early-to-mid 1990's? I'd *LOVE* to watch a video about Command & Conquer, Dune, and or Lands of Lore etc...! I love this series, keep up the excellent work! :)
really hope i get to see more like this, the problems in game development and what is learn from them is not talk about enough
Considering what they were aiming to create it’s a shame things didn’t work out.
I just found this game in a thrift store still wrapped, it had never been opened. I now sort of see why it's like this but I never even heard of this game. It was also only 75cents.
These are great stories, keep em coming!
More of this series please.
This is an awesome series of videos!
Would love to see more of these for sure , great series. I would like to see an extended version on Star Wars Galaxies. Always found that game a good topic for discussion.
Really good series! Please make more!
this is an awesome series so far hope to see a lot more of it
Lovin' these. Keep em coming :)
I really appreciate his honesty in this, not an easy thing to do.
Please do more of this series
Loving this series keep it up!
Great video & great series!
I distinctly remember buying this game along with a new laptop around 2010, installing it and rage uninstalling it the same day because it was basically unplayable with an underwhelming design and buying Civ 5 the next day instead. This is still the most broken game I think have ever had the misfortune of spending money on.
kudos for admitting the mistake and the way you handled it
some studios should take a note ...
tbf it's much easier to admit a mistake 8 years after the fact as opposed to 8 days/weeks
I absolutely love elemental War of magic, I just wish more was done after release..
Love this series. Also love Fallen Enchantress, but found the game so late that I only ever encountered the drama that surrounded Elemental briefly by reading about it, but never interacted with it myself.
The idea of this game sounds absolutely amazing. I'd like to see them remake it with modern technology in mind.
More videos from this series!
Such a fascinating series! Would love to hear from Bethesda or Obsidian on how they have had to solve design issues.
They’ll need to learn how to solve their issues first. Maybe in another several years.
Super interesting, thanks for sharing
Best series on youtube! :)
So why not release a remastered edition on the 64 bit format?
They did.
Fallen Enchantress Legendary Heroes is the name of the game. They had to ditch the Elemental name cause it was an anchor around their neck.
I want to see more of this series.
This series is fantastic
Really fascinating thank you
Great stuff, not all battles are won, and all you can do is learn and improve
This is amazing how interesting this videos are
I guess they didn't come from a console background. Memory fragmentation is what we were fighting from day 1 on ps2 games because you only have 32meg and that very quickly fragments if you do anything dynamic.
Memory fragmentation was even an issue on the Amiga.
The moment he mentioned that they're close to hitting the 2gb border and keep unloading and loading individual assets... I knew exactly where this is going. Oh well.
MY GOODNESS! the idea was really good
I genuinely loved the Elemental, and it didn't even gave me all that much trouble either. And i do hope Enchantress will deliver as well, i've bought it a while ago, but still didn't have the chance to play :D
More valuable lessons can be learnt from failed projects
Can use the Windows 3.1 solution: memory is a handle, not a pointer. When you want to read the contents to "lock" it, read it, and "unlock" when you're done. Behind the scenes another mechanism defragments the memory.
Bethesda games have shocking memory management too but by far the worst program on nearly everyone's computer at this moment is Windows 10 photo preview. This default Jpeg viewer can bring any system to its knees given an a few hours of operation, its impressive work.
Just saw these by accident, thrilling!
Great perspective; reminds me that human beings try to create these amazing projects and sometimes the bugs are just too strong. There are just so many layers, what can you do.
In 2018, Stardock sued the creators of Star Control, in order to obtain the right to use their characters, story lines, and races; after purchasing the rights to the "Star Control" trademark name from Atari, claiming their ownership applies to all supporting assets. The creators assert that Atari only ever owned the name, and the supporting assets were never Atari's to sell. Stardock then sued the creators to keep them from creating their own sequel to Star Control 2 (under a different name). There are various counter-claims, such as Stardock not having the rights to sell Star Control 1 and 2 through Steam. Stardock also claimed the original authors didn't create their games. Litigation is ongoing.
why did this serie stop? i've just discovered it and it's awesome
- ok men I love you! I also participate in this generation of the beginnings in which they started in Brazil in the world of games, very early with IBM PCs. Thank you very much and big hug from the gamers of Brazil! Let's move on!
Wow, learned something here. First time hearing about memory fragmentation.
It's sad too with the second game, they brought in someone who had a masterful mod for Civ IV by the name of Fall from Heaven for the writing, but it still had issues....
Doing a CS degree, this is a great series
The lesson: next time hire a senior engineer. It was 2010, data oriented programming was already established by almost a decade. We were already developing games with custom allocators, cache friendly big unfragmented buffers, data oriented pipelines, reusable components and entities and much more. ECS was already a thing (2000-2008).
Bingo! I am not even a software engineer and even I know the benefits of thread pools and memory pools. If I had to guess, on a modern system using the stack is at least 3x faster than using the heap. If you want dynamic allocation and you want it to run fast use a custom aloocator that puts the data on the stack. Writing a quick partial defragmentator that only runs when there are available cycles sounds like it would be a fun project. I imagine the framework would be a lot of opaque pointers, which is still probably quite a bit faster if they are pointing to a memory pool on the stack than non pointer data to the heap.
I imagine allocating in chuncks that line up with cache line sizes of the cpu are ideal so the cpu only has to do one load for a bunch of data instead of a bunch of tiny loads would massively reduce latency which is everything to a video game, so being strategic of where you put your data matters making it even more of a fun project.
@@Danielle_1234 Well one of the reasons the stack is so fast is because it is restricted to allocating and deallocating memory in a strict push/pop LIFO pattern. That allows allocation and deallocation to be done by simply incrementing and decrementing a stack pointer. The other reason it's fast is because it's generally only used to allocate a small total amount of purely contiguous, thread-local memory. As a result, you get locality of reference (both spatial and temporal) constantly accessing the same contiguous regions of memory over and over with minimal cache misses when accessing memory relative to the stack pointer. What you are describing to automatically defragment memory in the background sounds very similar to what garbage collectors do, although they do so at a heavy cost to performance.
I've been making games since 15 years, now.
Small... games.
Because that's my nightmare : to be caught pants down on a technical bug I couldn't fix, halfway through the project (or worse, in this specific case).
I like to conceive games.
I don't want to take that kind of risk.
So, I stick to small games, even if I could grow and take on bigger projects easily.
But damn, hitting such kind of technical wall... the horror indeed!
Get yourself some allies. Maybe a consultant with a lot of low level experience. Have your feelers out in the industry for people you can ask when stuck. You don't have to fight an impossible bug alone. Mostly, people will be willing to help with insight if you ask nicely, they won't do the actual job for you but they also won't charge you. Having dealt with malloc fragmentation before, I don't know how they haven't recognised the issue just a handful weeks into encountering it. Everyone has blind spots, but it's still on them that they haven't reached out to industry experts.
Certainly a single perspective of the post-mortem; game post-mortems are awesome.
The idea of the game still sounds really cool.
I hope they get the opportunity to re-do something like this at some point.
I really like the idea and style of this game.
Pre-Allocation is the way to fix and im glad they were able to finally implement it in later games.
ye, pretty sure that's what they did with old gameboy and gba games. those games never crash unless you yank the cart out.
More videos please!
The sequel fallen enchantress was a great game. I was so excited about the original too. Oh well. They made good on it.
Comment section is 60% composed by legendary game devs apparently. Anyway, comments cringe aside, this series is awesome, pls give us moarrr
Don't you know, they downloaded Unity or Unreal once and tried to follow a tutorial, they ARE experts...
Sorry Brad, but the 3D argument does not fly: "But strategy games had only recently embraced 3D engines"
... erm...
Emperor: Battle for Dune (2001). Earth 2150 (2000), Warzone 2100 (1999), Warcraft 3 (2002), Age of Mythology (2002) - this list goes on and on.
That said, Stardock has shown a lot of class in handling things after the game was released, especially giving away Fallen Enchantress to everyone who owned Elemental. Hats off for that.
Yeah, their response was a huge part of why I don't think badly of them. They really did try to make up for it. Though it was a bit late, then AoW 3 was revealed and...yeah.
It sounds more like the company was too busy guessing at what the issue was instead of taking the time to debug the issue.
To my best of knowledge this was commonly known. I understand that many large games have their own memory manager for this reason.
That's around the era when awareness started spreading and certainly hasn't reached all the corners. It also depends on the developer background. If they were more PC based but never had to work on a web browser, they wouldn't be aware. If they cut their teeth on Dreamcast or PS2, systems where memory remapping support was disabled, they very much would. Web browser replace the system malloc for their process to manage fragmentation, while solutions in games aware of the issue and trying to limit it tend to be more ad hoc.
I want Elemental II and I want it NOW!!!!
👍👍👍 love War Stories
This game looks like it was just ahead of it's time. They should try it again the concept looks very good.
It's easy, to criticize in hindsight, but this was a classic case of addressing the symptom, not the root cause.
It's clear it had to do with memory, because of the error message, but it also seems clear nobody in the team was a low level programming specialist. It would have been a great idea to swallow the pride, and hire a specialist to help figure this out. Of course, nowadays there's just so many more resources than back then.
Lessons: Recognize when you need to address the root cause. Recognize when you need external help.
Of course, they still shipped and sold it anyway.
@@cipherpunk7409 It was basically fraud.
I would give them a pass. The real crooks are Bethesda, EA, Activision/Blizzard.
Did they actually publish it themselves?
The alternative was putting the whole team out of a job, which is kind of what happened anyway.
Well, it's nice to hear him admit the game bombed. I recall being excited about the game. Upon reading the reviews, I dodged a bullet by not buying it. At the time, Stardock defended it which was not the thing to do. Thankfully, there games since have been anywhere from good to amazing, and I think we can forgive a company for a single bad game.
I played the game and liked it. The only problem was that there was no way I could win. You are plopped down on a map where everyone else is already fully established, and you are just figuring out how steel works. I ended up surviving matches by racing to the terraforming spells, and building an ocean between me and any opposing kingdom. The imbalance between what the player had and what the npcs had was so extreme that I never incountered their memory problem.
Memory pools are pretty standard so gutted for them they didn’t realise until too late.
His undershirt is made of pure white light.
Love it. But really, why did you release it unfinished and buggy?
On 32-Bit Windows you can use PAE to allocate more than 2GiB with trickery. Oh and memory fragmentation sucks bad on 32-Bit and is one of the reasons why you usually use pool and arena allocators. And sorry, memory fragmentation is a known issue in Game Dev, especially on 32-Bit. Now with 64-Bit, it's less of an issue as the memory space (Not physical memory!) is so vast that fragmentation isn't that big of a deal. However, you still need to check on your stuff.
Damn this looks better than 99.99999999% on whats on steam. When can I preorder this?
Newb mistake. Memory fragmentation was one of the first things I learned about dealing with when managing memory on Mac OS 8.0 in the late 90's.
Or any old Mac from the era of their Handle based memory management systems =) Or any new fangled C++ with the move semantic built into some of the data structures... Pool memory allocators, etc... I totally agree with you, there's no excuse to be blindsided by this.
He said you don't find a lot of tools out there for memory fragmentation. I disagree. I mean there are IDE's and tools to find all kinds of memory issues. Yeah, this seems like something that wouldn't require a massive overhaul to fix. I mean you could have patched it right?
well, the question is if those tools were there 10 years ago.
think you're missing something - they did eventually fix the crashing bug, but by then they had stripped out so many features of the game it sucked. Which he said himself. In other words in trying to fix the bug they watered down the game and features so much it was no longer a fun game. Then later on they figured out the memory issue, fixed it but by then the game was dead. Fallen Enchantress was much better
"...he moved on to be lead developer for adobe premiere pro"
It's genuinely hard for me to understand how they could be talented enough to (usually) make great games and yet not realize it was memory fragmentation. As soon as he said they were running out of memory even though they had memory available, I thought, "Oh, it's fragmented." Additionally, he made it seem like fragmentation is an utterly unsolvable problem. There are a number of options for allocating memory to minimize fragmentation, such as first-fit or worst-fit allocation. You can also split your heap up into blocks of fixed sizes, based on your application's allocation tendencies. Furthermore, it's possible to defragment memory that has become too fragmented, though that's more complicated. I've worked in embedded systems for years and I've implemented heap allocation algorithms to do all of those things, so I know it's not only possible, but reasonably easy. If I can do it, it's not that hard.
Console developers coming from PS2 or Dreamcast were as aware as embedded devs and had to design for fragmentation avoidance up front, given the MMU is disabled. But these guys had PC background.
A quick and dirty fix on PC would be just to do what Chrome did at the time, and steal tcmalloc from it. With their 800mb real footprint they'd be just fine.
Huh something i hadn't thought of until just now, IN SPITE of having dealt with this exact problem in the past, is that the GPU driver will create memory map allocations inside your process to communicate with the GPU, corresponding to all the buffers and textures that you want to upload to the GPU and shaderstuffs and more stuff, and they'll just hang around for the potentially useful life of the buffers. Interleaved mmap requests from the malloc implementation and from the GPU driver will necessarily partition and fragment the address space, without occupying any physical memory. While you expect people to be somewhat aware of fragmentation internal to and inherent to malloc, this inter-system layer of fragmentation is probably not something anyone spoke about until application malloc pool sizes grew close to a gig and GPU object pool sizes grew beyond a couple hundred MB. Suddenly, the address space gets really really tight! Worse so if you have mmapped file IO anywhere. And you don't have any leverage over what the GPU driver is doing there, none at all! It's not documented anywhere either, it's just a conclusion you come to painfully and then headdesk. Most of the time, it'll look to you like the crash is coming from the depths of the GPU driver, since you have by then gotten extremely careful with your allocations, or they are mostly caught by an existing pool within the malloc implementation so your direct ones don't cause a crash. The crashes are extremely nondescript, like whoops the GPU driver thread crashed again well out of context of your invocations, there is no stack trace between what the driver is doing and what you're asking it to do.
Sins of a Solar empire was really fun so I'd say stardock made up for it. I like the admission to failure too, better than companies blindly following the product
Publisher not developer.
If your watching this any chance of doing a remake on elemental?