C++ is like chemistry. You spend a respectable amount of time learning the rules and a ridiculous amount of time learning about exceptions and edge cases.
I myself program in C++ and what you did there is just amazing. Honestly you're approaching legend status in the C++ community. Keep up the incredible work!
@@KleptomaniacJames all I know is every time I post c code for feedback or help(not often I do) I am just told to use c++. So there cohesive about that.
Dude, you’re actually a freaking legend. I code in C++ too; around an year ago I’d asked a question about reflection on SO. Turns out it didn’t exist. ‘Oh well’, I thought to myself and gave up on the thing I needed reflection for. But you actually adding reflection TO THE LANGUAGE. LEGEND FR.
Wow. I personally am a JavaScript and C# programmer, so I guess I never really thought about all the work that goes into this stuff. I'm honestly really impressed. Honestly, you're one of the best programmers I've seen in a long time. I think we could all learn a bit on your approach to problems. Keep up the good work!
Both C# and JavaScript are infinitely easier to build this in because C# has a runtime that you can inspect and JavaScript and interpreter. C++ is harder because it’s compiled not to byte code that has a runtime but native machine code. You could compare it like at runtime having C# change it’s runtime or JavaScript change it’s interpreter. Most people who were in need of reflection in C/C++ would actually create their own little run time where a structure would be either read from a header file (like I did once to know where to move within the pointer of the object) or created their own types at runtime. Like JavaScript really - then reflection is super trivial.
Let's see... *Builds an OS from scratch *Has an affinity for old-style cpu's *modifies a popular programming language to suit his needs You're becoming more and more like Terry Davis with every passing day.
@@Perseagatuna He needed a language that could be used without a POSIX runtime. I'm currently building a non-POSIX runtime in C++, but since he also needed a new compiler that runs on his exotic kernel anyways, I guess he decided why not make a new language for it too.
All this to catch an obscure and very rare instance that **MIGHT** happen if the user spills liquid on a specific part of his keyboard, while performing an ancient ritual to summon a spirit that could, potentially, force the RNG gods to grant a special favor when farming a specific item in a completely and unrelated game. The art of code efficiency, sometimes, can take you to a very dark place.
I've been working with C++ since at least 2003 and I can say that this was quite impressive to dive into the intermediate language of the compilers. Some of the best developers and engineers are those who design their own tools! Job well done!
In one of those ironic life moments, i actually just spent a month working on a reflection project in c++... and im genuinely impressed you created an even more convoluted solution. Brilliantly done!
I've actually also written a janky reflection system too, but my method used extreme preprocessor abuse rather than anything remotely smart like templates or your method of automatically fetching data. I had a macro that took in the name of an existing class, and then a varadic collection of other macros. That main macro created another secret class with a static constructor and some metadata variables. The varadic arguments in that macro then had to be yet more preprocessor macros which were primarily used to list off fields that could be found in the actual class, as well as give them metadata. Through abuse of static constructors (Which ran before main() ), and the typeid(xyz).name(), offsetof_s() and sizeof() functions, as well as just horrific abuse of casting, I had a workable reflection system that met the needs of the project, with each macro essentially building up part of an internal type database that will be fully populated with all relevant types once main() itself runs. Since typeid().name() gave me a string name of the field's type, I could then also slot some type converters into a hashmap to handle serializing/deserializing anything that wasn't an int and void* casting it into the class instance at the right spot thanks to my knowledge of the field offsets and sizes. Later on, the field definition macros even let me add extra metadata that let me add networking support by marking replicated fields, from which a subsystem would grab, compare and generate packets automagically! It even sort of worked! Ah, university... Got to love being given a complex 3D project where the only library you're allowed to use is DirectX and yet it needs a load/save system... No idea if my project would have worked on any compiler other than MSVC, but it got me a high grade, that's for sure.
That is the default approach for most serialization libraries. You just put a bunch of macros, and it generates the necessary code to serialize and deserialize it.
I made my own compile-time typeid().name() because I wanted to (it was for listing arguments of commands for a command processor thing I was working on, which was also 90% compile-time abstractions). I had to do some black-magic template stuff with pointer type parameters, static constexpr locals, and the builtin __FUNC__ macro. What a pain. I also added support for including type specifications/qualifiers/modifiers in the resulting string.
What would be useful to have in your serialization is to address an occasion when you have a class, and some time later in the development you change field names or datatypes. After that old save files will be unusable. You can serizalize some version number for each type, say by default each version 1, then for each changed class you increments it version. Then in your code you check if version for a class from a save file matches your source code class version, and if not you could implement some per-class conversion function to keep backward-compatibility.
that's exactly the plan :) the serializer can also serialize a schema this way (names/hashes of field names and class bases in the order in which they were written) to allow for smooth upgrades.
Don't forget that if you incorporate a library that is GPL (even v2) and distribute it (freely or commercially) you need to make available to the end user any change you made to the source code.
Wrong, only the original source would be under GPL. This is why many companies are able to use GPL’d code with their stuff strapped on top.. it doesn’t make the whole file GPL’d. Would have to make the original GPL’d code available upon request though.
I salute this level of insanity! My solution to this was just to store the game state in a single struct and then I can do a single file read or write to save or load. I really wish C++ would add reflection, every now and then I want to do something that turns out to be completely impossible.
This was the magic behind Warcraft's save system. Most games had really long load and save times back then (and today, _especially_ if you account for the difference in drive speeds); Warcraft instead had the game state as a single contiguous state in memory that could easily be read or written in a single block. Unfortunately, for Starcraft, they decided to switch to C++... :D It's especially funny when you consider that many game design patterns today lend themselves pretty handily to this approach - coming back to the good old "keep data separate from logic".
@@LuaanTi It's not like you couldn't do it the C way just because you're using C++... (unless you're relying on a parser, but you guys are talking about the total opposite solution).
@@zvxcvxcz It's a bit of a problem if you're using inheritance (and pointers, which Warcraft avoided). Starcraft had to deal with the same problem and they ended up having to remap all the pointers before each save and after each load - which was of course _very_ slow and error prone. Starcraft was buggy as hell :D Of course, you _can_ use as little C++ as possible. And you can still separate data from code in C++. It's just that C++ was kind of built on the exact opposite mindset.
@@LuaanTi That works as long as you don't try to transfer data between architectures. Granted, nowadays many/most PC (not necessarily Windows, just non-console) games are built for x86, including on Mac, but if that save file was copied to, say, an older Mac system, suddenly all your multibyte values are gibberish.
I've enjoyed C++ with reflection for 15 years or so now -- Qt's "moc" (meta-object compiler) generates reflection data for classes. I used this back in about 2008 to build a tool to transparently connect signals to methods on a remote instance over a network connection. More recently I've also used it to automatically bind C++ classes to Lua, to allow scripts to control C++ software. And in another project, moc-based reflection lets me expose methods as web API endpoints.
This is, by far, the worst, most convoluted, most unmaintainable c++ solution for anything I've ever seen... and it's beautiful. I have no words. They should have sent a poet.
Serialization by reflection is only cool on paper in my experience because it adds a big headache on top of the already existing headache that is migration / data layout changes between versions. thats why i prefer explicit serialization amd for the reflection part i usually just use an enum based type registry.
That’s awesome!! I love when folks are able to go “there’s surely an easier way to do this…” and then roll-out a completely custom system to accomplish what they’re looking for. This “meta” programming is incredibly powerful and I love seeing it when it takes place. Thanks for such a well-documented version! :3 Is this something like Unreal Engine’s reflection system under the hood? Or is this this completely different :o
I love this solution. Re-inveting the wheel is somewhat frowned upon by programmers, but I think it should be the opposite, it's precisely what's necessary to get a deep understanding of things.
It's funny, you're a big inspiration to me in terms of skill but it's also intimidating to see just how far away I am. I still need to learn so much to be able make the software that I dream about making.
Why am I even here? I mentally checked out like three videos ago. I have no idea what's going on. So why am I still so captivated and entertained by this?
I went back and forth on reflection for the past year and a half precisely for serializing components in my tiny game. At first I did it all by hand, but it became too much repeated code (serialize in, serialize out, add to the imgui inspector...) so I said to myself, it can't be that difficult to do it automatically... Well, down the rabbit hole I went. I looked at all the macro approaches first, and left it like that for a while. Then tried the clang library but was too intimidated by it. Now, since I only need reflection for component data, which should be literal types, I use something similar to the PFR library which decomposes them into structured bindings which can be used like a tuple. You can iterate over them while compiling and save/load everything. The only missing part were field names, so I wrote a quick and dirty python script that generates a header that is optionally included where the components are declared and creates an array of names that you can use in the reflection library. It's not perfect, but it's the best I could do and actually I'm really happy with how it works now for my needs, specially compared to other previous solutions. However, I must say, what you did here is black magic. Knowing firsthand the hurdles of implementing reflection, this was amazing to watch. And everything you accomplished is seriously mind blowing (as are all your c++ experiments to be honest). So wow, thank you so much for making this incredible videos!
Anything that has to do with parsing the mountain of syntax that is C++ is an absolute feat of software engineering. You have my respect. 40 years of stacking features on top of features have made it virtually impossilbe for a single person to know the full syntax of the language.
I hear more recent programming languages like Zig let you do this sort of metaprogramming very easily, and at compile time rather than runtime. Even in C/C++, I've seen people successfully solve this exact problem with code generation (writing C programs that _literally_ write out C programs). This avoids having to update serialisation code manually every time e.g. an entity gets an extra field added, but without having to massively bloat your data structures with runtime type information (which is what I understood to be your approach). Either way, very entertaining video as always.
As someone whose had to support runtime reflection in a giant (3M+ lines of code) c++ system with some code predating c++98, and a c++ expert, this is both extremely impressive and genuinely horrific. I really hope we finally can get decent reflection into the core language soon, but I’m growing cynical about the standards committee. It seems to be over full of obscurantist weenies who’re more terrified of breaking ancient junk than delivering well-designed new features or improving the core language. As if the mere possibility that some ancient code on some IBM mainframe- that can’t even run a 21st century OS anyway- not compiling under newer c++ language versions is a valid reason to screw with or forgo useful language changes. It’s akin to having a master body charged with updating the English language that’s a mix of Oxford literature snobs, legalese experts, and elite Shakespearean actors who don’t want their rare mastery of Ye Olde English to lose value. You do have the occasional normal, fluent speaker in there but too often they’re drowned out by the weirdos and “better not take risks” crowd. At this point it may make more sense to define a successor “C*” language like c++ originally was for C. A Rust with better syntax, say.
Object de-serialization from a human-readable format is actually quite an interesting topic, as you basically have to write a complier for a non-turing-complete language -- the savefile.
My kind of person. A few months ago I wanted to make an audio processing program because the current ones werent good enough for me. Long story short, as a direct result, im now trying to make my own little x86-64 desktop operating system from scratch.
I had a similar but less successful story 20 years ago. I was making a game in C++ and wanted to save & load all the different objects. The system I built first was inflexible - any time I added a member variable that needed saving, any old files would no longer work. So I thought I'd try saving everything as the code needed to recreate everything. I tried making a scripting system for this purpose but took ages and got nowhere. Then I found an existing open source C++ scripting system, improved it with some asm to do the function calls I wanted. I used plenty of macros to shorten all the things I was doing in every class. These days I use JS and other modern languages. I'm throwing JSON data around all the time. Luxury!
JSON data is gross, not a luxury. Anyway, in the interim we have gained some obvious robust solutions, e.g. there is good serialization in the Boost libraries now, but 20 years ago... you were just like 2-3 years too early.
Crazy skills needed to pull this off in C++. At university way back in the 90s I did make a C-interpreter in a compiler course using lex and yacc. That was pretty fun as I could modify my code in real time as it was running to experiment myself towards the output (but often messing up the data in the process), C is way simpler though so making reflection support for that would have been way easier for sure.
And here I am, seeing if I can store the locations of the values of compile-time variables as written in machine code, so I can modify them during runtime without recompiling :)
you should take these detours when writing code! if your goal is to learn and become a better dev, you shouldn't be ashamed of them. you learn a lot more stuff and it's *way more* fun to write code and contribute that way, leading to you becoming a better dev. keep it up jdh!
@@bluesillybeard for low-level stuff I think D is a bit of an underrated language. It can be used for low-level stuff or high-level things and it has a lot of features that other languages should have
Before I start waffling: I guess you didn't have to bother handling anything too weird (like SDL objects, or whatever graphics interface you're using) as you only want to save game data, so I'm guessing you can ignore pointers to external library stuff (more weird than STL) ? Or did you have to write edge case stuff for all those too ? Now to the waffle: Genius ! I looked at the LLVM backend thinking it'd be a great tool to write my own language, but then I decided I didn't have the time (the skill or the insanity). C++ is the first language I learnt (I hesitate to say learnt, does anyone know all it's knoocks and crannies) , it has made learning everything else real easy.. Anyway some fantastic craziness there, delivered with perfect understatement.. well done !!
I've not followed it In a while, but c++23 was supposed to get reflection. At least it was one of the more popular proposals when the c++23 spec was first being announced. But this is awesome and truly amazing work, looking forward to what you do next
I locked myself outside the house but the first floor's window was open, so to get in what I did was... build an electro-mechanical elevator powered by geothermal energy of course... what else would you expect?
I've built a C++ parser from first principles before. It was... hard. Very hard. But it parsed its own source code, including the STL includes! It would have eventually become a complete compiler, but the "C++ Grandmaster" class I was taking fell apart because the instructor ghosted.
Ooh! I did this too, but for typescript which was waaay easier. I don't think I would have done this for C++ due to it being too fragile. So I have reflection for serialising data but it's purely runtime (and much easier because I define specifically how to save each type, I just made it so I only have to decorate types to make them save/load properly). I also have a compiler plugin that adds proper mixin support. So I feel like I've done both sides, but in easy mode.
I'm in the process of writing a remote DWARF debug info server/parser. Stack unwinding wasn't too bad even if I couldn't use libunwind's built-in remote unwinding for... reasons. Then came the part of parsing actual debug info so I could see where and how variables are stored and... Your comment about edge cases upon edge cases is soo true for this as well 🤣
Good lord... This is giving my PTSD of when I worked with C++ xD The absurd complexity of everything yet the surprising lack of "common" features and those compile times... The compile times especially hit me hard, after getting used to regular C, where quite often I'd just recompile whole projects while testing, just because it was fast enough to not bother me. Was also WAY less error prone to the kind of wacky compilation anomalies only C/C++ have (as far as I know).
I'm genially impressed by your approach, dedication and knowledge. All that said, I don't think writing compiler extension it the best idea :) I think one of the approaches you could take is just use some interaction with Rust, because it's already has decent serialization without much of boilerplate. Also for my personal needs I implemented somewhat similar to Unreal Header Tool, which just scans file for macros/tags and then do runtime serialization, so you not writing whole C++ parser, but a small part of it. I did a very very basic stuff, so only few types and containers are recognizable, it doesn't work well with templates and custom spacing/aligning etc, but I think you can make it pretty generic too.
Careful: reflection-based serialization is somewhat fragile. Half the work in serialization libraries is making sure you can still load a file from several versions ago, and you still can't really do that automatically. You need to bear in mind backwards compatibility. For example, I add a new field to an existing structure, but then load an old gamesave from before I added that field. Do I have a default value? Does it make sense?
You know, im not very smart, but from what i've learned with coding so far is- "If you can't figure out a way to do it, even a hacky way, you probably shouldn't."
I wonder, could you take a snapshot of the memory as a dump of sorts, and then allocate that same state again? Another idea could be the use some sort of custom base class or template that could record all needed data. Perhaps with a touch of boilerplate code generation. Not saying every language couldn't use proper reflection features, especially cpp. Anyways, very impressed that you got it working, well done!
I have actually looked into this approach for my own project, but decided against it due to platform incompatibility. Since this reflection system is inherently tied to CLang, you cannot use it with, for example, MSVC, which i needed for multi-platform support. I instead made a templatized reflection system that generates type names and some basic information at compile-time from within C++ itself, and then you can explicitly reflect certain additional aspects of your types such as attributes, inheritance, functions, etc. at runtime during plugin initialization
you can actually use this on systems which clang doesn't support - the plugin then will just emit source code which can then be used by the target compiler just as any source would be included in the final build
@@jdh That is true, but that still means that you cant just compile with MSVC from the start. I'm making a game engine so I can't really force the user to run CLang plugins and *then* compile with MSVC :p Also personally, I am just not a fan of external tools for codegen :P, which is why i went with compile-time generation of reflection data. You can actually do a lot of similar reflection stuff at compile-time with templates, like using `__PRETTY_FUNC__` + some lightweight constexpr parsing to generate a demangled string name for any type, generating casts for integral and floating-point types, enum to underlying type casts, type-erased range and tuple-like type binding, type-erased string-like type binding, etc.
If you want to do some weird stuff like that, you might want to consider trying the Scopes programming language. It also has compile time execution. I wonder, if it's possible to create a similar system for scopes without hacking the compiler. Scopes even caches compiled functions into object files.
Ah, I remember reading an article or watching a video recently on how C++ is virtually unparseable. I mean, it of course is, it works etc., but it all has some "hacks" and "limitations". I mean, Templates being turing complete is one issue, but there are of course various magical things in there as well...
My solution to this was to define > operators for every object and struct in my game, similar to how std::cin and std::count does it. From there, my game data was human readable and writing a parser was fairly simple. istream and ostream made serialization easy with ambiguous implementation!
Risks .. compatibility over code changes? - does the seralizer emit to somethign like json or whatever, that uses simple named key/value/recursives, or is it just dumping out binary of the objects in question? which is to say .. do you have the risk that over time, if you release updates to the game, will it invalidate their existing save stuff since it was autogenmerated to match a certain set of code, and now the code coudl be significantly different, but you'd want the saves to be compatible?
I'm also curious about this. If a compiler update can break it, or I can't transfer data between different types of computers this becomes much less useful...
I wish youtube brought to your videos more attention so you could kind od get at least some money back for your time you sept on this. But at least, man, your knowledge, despite coding for 8 months as you said, is quite immpresive.
I personally would have gone for those approaches first: 1/ macros 2/ manual serialize() deserialize() in every object I'm surprised you didn't think of using macros.
it's definitely faster to add syntax coloring to your Vim for a #define named STRUCT_SERIALIZABLE that colorize as a struct. Than to hack the compiler the way you did. In any case kudos for the hard work and ingenuity.
C++ is like chemistry. You spend a respectable amount of time learning the rules and a ridiculous amount of time learning about exceptions and edge cases.
Wow, that’s an amazing metaphor
So its basically the French of programming languages?
@@___aZa___ perhaps it's the English of programming languages
@@Snail5008 Nah, French.
@@Ozzianman Fr*nch
I myself program in C++ and what you did there is just amazing. Honestly you're approaching legend status in the C++ community. Keep up the incredible work!
@ThinkHam in no way practical or reusable but nonetheless badass
I find myself doubting the existence of a cohesive c++ community
@@KleptomaniacJames all I know is every time I post c code for feedback or help(not often I do) I am just told to use c++. So there cohesive about that.
💀
@@nickgennady "Hi, I want to C!"
- But have you tried C++ yet?
Watching this video made my will to learn C++ go from 7 to 0.
That's a bit like watching Rallye and it depleting your will to drive
Dont let it discourage you, what he did was complete overkill and something you should never do anyway.
If anything, this encouraged me to learn more
@@user-ry4ip9ps9x cuz youre gey
That's a good thing. Learn Rust instead, it's a much more sane language.
Dude, you’re actually a freaking legend. I code in C++ too; around an year ago I’d asked a question about reflection on SO. Turns out it didn’t exist. ‘Oh well’, I thought to myself and gave up on the thing I needed reflection for. But you actually adding reflection TO THE LANGUAGE. LEGEND FR.
JDH, you've made an extremely janky, bodged system here that's "temporary."
And I love you for it.
Wow. I personally am a JavaScript and C# programmer, so I guess I never really thought about all the work that goes into this stuff. I'm honestly really impressed. Honestly, you're one of the best programmers I've seen in a long time. I think we could all learn a bit on your approach to problems. Keep up the good work!
well, for JS you have... JS Object Notation :^)
Javascript? You poor soul. I hope God will some day have mercy with you.
@@graealex fr
Both C# and JavaScript are infinitely easier to build this in because C# has a runtime that you can inspect and JavaScript and interpreter. C++ is harder because it’s compiled not to byte code that has a runtime but native machine code.
You could compare it like at runtime having C# change it’s runtime or JavaScript change it’s interpreter.
Most people who were in need of reflection in C/C++ would actually create their own little run time where a structure would be either read from a header file (like I did once to know where to move within the pointer of the object) or created their own types at runtime.
Like JavaScript really - then reflection is super trivial.
@@graealex JS is a great language and I want to use it in the future
Let's see...
*Builds an OS from scratch
*Has an affinity for old-style cpu's
*modifies a popular programming language to suit his needs
You're becoming more and more like Terry Davis with every passing day.
Terry Davis utterly despised C++.
@@Cons-Cat isnt that why he made holyc?
@@Perseagatuna He needed a language that could be used without a POSIX runtime. I'm currently building a non-POSIX runtime in C++, but since he also needed a new compiler that runs on his exotic kernel anyways, I guess he decided why not make a new language for it too.
@DannyLeWasTaken I have a video on my channel named LibCat Lecture which explains the motivating fundamental problems at the start.
All this to catch an obscure and very rare instance that **MIGHT** happen if the user spills liquid on a specific part of his keyboard, while performing an ancient ritual to summon a spirit that could, potentially, force the RNG gods to grant a special favor when farming a specific item in a completely and unrelated game.
The art of code efficiency, sometimes, can take you to a very dark place.
I've been working with C++ since at least 2003 and I can say that this was quite impressive to dive into the intermediate language of the compilers. Some of the best developers and engineers are those who design their own tools! Job well done!
I don’t know what I was expecting you to use to make a save system, but I definitely know it wasn’t this, incredible job!
In one of those ironic life moments, i actually just spent a month working on a reflection project in c++... and im genuinely impressed you created an even more convoluted solution. Brilliantly done!
I've actually also written a janky reflection system too, but my method used extreme preprocessor abuse rather than anything remotely smart like templates or your method of automatically fetching data.
I had a macro that took in the name of an existing class, and then a varadic collection of other macros. That main macro created another secret class with a static constructor and some metadata variables. The varadic arguments in that macro then had to be yet more preprocessor macros which were primarily used to list off fields that could be found in the actual class, as well as give them metadata. Through abuse of static constructors (Which ran before main() ), and the typeid(xyz).name(), offsetof_s() and sizeof() functions, as well as just horrific abuse of casting, I had a workable reflection system that met the needs of the project, with each macro essentially building up part of an internal type database that will be fully populated with all relevant types once main() itself runs.
Since typeid().name() gave me a string name of the field's type, I could then also slot some type converters into a hashmap to handle serializing/deserializing anything that wasn't an int and void* casting it into the class instance at the right spot thanks to my knowledge of the field offsets and sizes. Later on, the field definition macros even let me add extra metadata that let me add networking support by marking replicated fields, from which a subsystem would grab, compare and generate packets automagically! It even sort of worked!
Ah, university... Got to love being given a complex 3D project where the only library you're allowed to use is DirectX and yet it needs a load/save system... No idea if my project would have worked on any compiler other than MSVC, but it got me a high grade, that's for sure.
That is the default approach for most serialization libraries. You just put a bunch of macros, and it generates the necessary code to serialize and deserialize it.
I made my own compile-time typeid().name() because I wanted to (it was for listing arguments of commands for a command processor thing I was working on, which was also 90% compile-time abstractions).
I had to do some black-magic template stuff with pointer type parameters, static constexpr locals, and the builtin __FUNC__ macro. What a pain. I also added support for including type specifications/qualifiers/modifiers in the resulting string.
What would be useful to have in your serialization is to address an occasion when you have a class, and some time later in the development you change field names or datatypes. After that old save files will be unusable. You can serizalize some version number for each type, say by default each version 1, then for each changed class you increments it version. Then in your code you check if version for a class from a save file matches your source code class version, and if not you could implement some per-class conversion function to keep backward-compatibility.
that's exactly the plan :) the serializer can also serialize a schema this way (names/hashes of field names and class bases in the order in which they were written) to allow for smooth upgrades.
Your breaking the laws of physics for a game that might not be released amazing
foreshadowing is a literary device used to hint at future events
doing metaprogramming in c++ with templates involved must be one of the most terrifying jobs in CS history..
Template only libraries are fun stuff :P
"Babe wake up new jdh video just dropped"
It’s like my favourite tv series posting a new episode this game dev vlog
Don't forget that if you incorporate a library that is GPL (even v2) and distribute it (freely or commercially) you need to make available to the end user any change you made to the source code.
Wrong, only the original source would be under GPL. This is why many companies are able to use GPL’d code with their stuff strapped on top.. it doesn’t make the whole file GPL’d. Would have to make the original GPL’d code available upon request though.
I salute this level of insanity! My solution to this was just to store the game state in a single struct and then I can do a single file read or write to save or load. I really wish C++ would add reflection, every now and then I want to do something that turns out to be completely impossible.
This is my exact thought on how to do saves, setup a file to write the save data to on exit by calling a function to write the save information
This was the magic behind Warcraft's save system. Most games had really long load and save times back then (and today, _especially_ if you account for the difference in drive speeds); Warcraft instead had the game state as a single contiguous state in memory that could easily be read or written in a single block. Unfortunately, for Starcraft, they decided to switch to C++... :D
It's especially funny when you consider that many game design patterns today lend themselves pretty handily to this approach - coming back to the good old "keep data separate from logic".
@@LuaanTi It's not like you couldn't do it the C way just because you're using C++... (unless you're relying on a parser, but you guys are talking about the total opposite solution).
@@zvxcvxcz It's a bit of a problem if you're using inheritance (and pointers, which Warcraft avoided). Starcraft had to deal with the same problem and they ended up having to remap all the pointers before each save and after each load - which was of course _very_ slow and error prone. Starcraft was buggy as hell :D
Of course, you _can_ use as little C++ as possible. And you can still separate data from code in C++. It's just that C++ was kind of built on the exact opposite mindset.
@@LuaanTi That works as long as you don't try to transfer data between architectures. Granted, nowadays many/most PC (not necessarily Windows, just non-console) games are built for x86, including on Mac, but if that save file was copied to, say, an older Mac system, suddenly all your multibyte values are gibberish.
I haven't seen anything like that in my life. This is what I call persistence, dedication, and devotion.
I think you're a genius man.
I've enjoyed C++ with reflection for 15 years or so now -- Qt's "moc" (meta-object compiler) generates reflection data for classes. I used this back in about 2008 to build a tool to transparently connect signals to methods on a remote instance over a network connection. More recently I've also used it to automatically bind C++ classes to Lua, to allow scripts to control C++ software. And in another project, moc-based reflection lets me expose methods as web API endpoints.
I love that this guy took a simple idea of saving game state, and went off the deep end with it. Kudos my crazy ass hacker brother!
This is, by far, the worst, most convoluted, most unmaintainable c++ solution for anything I've ever seen... and it's beautiful.
I have no words. They should have sent a poet.
Serialization by reflection is only cool on paper in my experience because it adds a big headache on top of the already existing headache that is migration / data layout changes between versions. thats why i prefer explicit serialization amd for the reflection part i usually just use an enum based type registry.
I love GRPC serialisation. For me there is no other serialisation anymore
I use std::any inputs to a single `serialize()` function and a gigantic switch case. Works pretty well, and is fairly readable, if a bit long.
You did in a few weeks what hasn't been done for years by the C++ standard committee. Very impressive!!
That’s awesome!! I love when folks are able to go “there’s surely an easier way to do this…” and then roll-out a completely custom system to accomplish what they’re looking for. This “meta” programming is incredibly powerful and I love seeing it when it takes place. Thanks for such a well-documented version! :3
Is this something like Unreal Engine’s reflection system under the hood? Or is this this completely different :o
I am beyond excited for this game! I’ve always wanted a game like this!
I'm rewriting all my C++ in Rust.
It's hard at first, joy afterwards.
Serialization in Rust is very easy.
The more I watch this video, the more I understand, and the more I commend you for attempting something like this.
other game devs: nooooo how do i add an object in unity
this guy: reprogramming his compiler to make his game work
I love this solution. Re-inveting the wheel is somewhat frowned upon by programmers, but I think it should be the opposite, it's precisely what's necessary to get a deep understanding of things.
It's funny, you're a big inspiration to me in terms of skill but it's also intimidating to see just how far away I am. I still need to learn so much to be able make the software that I dream about making.
15:30 wow that was a blast from the past
Why am I even here? I mentally checked out like three videos ago. I have no idea what's going on.
So why am I still so captivated and entertained by this?
I went back and forth on reflection for the past year and a half precisely for serializing components in my tiny game. At first I did it all by hand, but it became too much repeated code (serialize in, serialize out, add to the imgui inspector...) so I said to myself, it can't be that difficult to do it automatically... Well, down the rabbit hole I went.
I looked at all the macro approaches first, and left it like that for a while. Then tried the clang library but was too intimidated by it. Now, since I only need reflection for component data, which should be literal types, I use something similar to the PFR library which decomposes them into structured bindings which can be used like a tuple. You can iterate over them while compiling and save/load everything. The only missing part were field names, so I wrote a quick and dirty python script that generates a header that is optionally included where the components are declared and creates an array of names that you can use in the reflection library. It's not perfect, but it's the best I could do and actually I'm really happy with how it works now for my needs, specially compared to other previous solutions.
However, I must say, what you did here is black magic. Knowing firsthand the hurdles of implementing reflection, this was amazing to watch. And everything you accomplished is seriously mind blowing (as are all your c++ experiments to be honest). So wow, thank you so much for making this incredible videos!
Most great coders make things look easy.
JDH makes things look hard.
No hate.
Anything that has to do with parsing the mountain of syntax that is C++ is an absolute feat of software engineering. You have my respect. 40 years of stacking features on top of features have made it virtually impossilbe for a single person to know the full syntax of the language.
I hear more recent programming languages like Zig let you do this sort of metaprogramming very easily, and at compile time rather than runtime.
Even in C/C++, I've seen people successfully solve this exact problem with code generation (writing C programs that _literally_ write out C programs). This avoids having to update serialisation code manually every time e.g. an entity gets an extra field added, but without having to massively bloat your data structures with runtime type information (which is what I understood to be your approach).
Either way, very entertaining video as always.
This is incredible. I don't have anything really productive to add, but I just wanted to say how impressive this is.
Your dedication to doing things yourself instead of settling for what others have already made is truly remarkable! Excellent video!
"Why take 5 minutes to do something when you can spend 6 hours trying to automate it" - Some Guy
As someone whose had to support runtime reflection in a giant (3M+ lines of code) c++ system with some code predating c++98, and a c++ expert, this is both extremely impressive and genuinely horrific.
I really hope we finally can get decent reflection into the core language soon, but I’m growing cynical about the standards committee. It seems to be over full of obscurantist weenies who’re more terrified of breaking ancient junk than delivering well-designed new features or improving the core language. As if the mere possibility that some ancient code on some IBM mainframe- that can’t even run a 21st century OS anyway- not compiling under newer c++ language versions is a valid reason to screw with or forgo useful language changes.
It’s akin to having a master body charged with updating the English language that’s a mix of Oxford literature snobs, legalese experts, and elite Shakespearean actors who don’t want their rare mastery of Ye Olde English to lose value. You do have the occasional normal, fluent speaker in there but too often they’re drowned out by the weirdos and “better not take risks” crowd. At this point it may make more sense to define a successor “C*” language like c++ originally was for C. A Rust with better syntax, say.
me: "oh I also code in C++!"
also me after watching this: "do you, though?"
Object de-serialization from a human-readable format is actually quite an interesting topic, as you basically have to write a complier for a non-turing-complete language -- the savefile.
God, please have mercy for C++ programmers, they need to handle that instead of fread/fwrite with POD C struct :D
My kind of person.
A few months ago I wanted to make an audio processing program because the current ones werent good enough for me. Long story short, as a direct result, im now trying to make my own little x86-64 desktop operating system from scratch.
I had a similar but less successful story 20 years ago. I was making a game in C++ and wanted to save & load all the different objects. The system I built first was inflexible - any time I added a member variable that needed saving, any old files would no longer work. So I thought I'd try saving everything as the code needed to recreate everything. I tried making a scripting system for this purpose but took ages and got nowhere. Then I found an existing open source C++ scripting system, improved it with some asm to do the function calls I wanted. I used plenty of macros to shorten all the things I was doing in every class.
These days I use JS and other modern languages. I'm throwing JSON data around all the time. Luxury!
JSON data is gross, not a luxury. Anyway, in the interim we have gained some obvious robust solutions, e.g. there is good serialization in the Boost libraries now, but 20 years ago... you were just like 2-3 years too early.
Crazy skills needed to pull this off in C++. At university way back in the 90s I did make a C-interpreter in a compiler course using lex and yacc. That was pretty fun as I could modify my code in real time as it was running to experiment myself towards the output (but often messing up the data in the process), C is way simpler though so making reflection support for that would have been way easier for sure.
And here I am, seeing if I can store the locations of the values of compile-time variables as written in machine code, so I can modify them during runtime without recompiling :)
This is exactly why I respect game devs. Appreciate the video being seasoned with memes💯
i wish i could understand anything said in this video, being able to do that myself would probably bring me a lot of joy
you should take these detours when writing code! if your goal is to learn and become a better dev, you shouldn't be ashamed of them.
you learn a lot more stuff and it's *way more* fun to write code and contribute that way, leading to you becoming a better dev.
keep it up jdh!
Reflection: 10,000 LOC, 7 days work
Adding serde methods: 1000 LOC, 2 days work
Chad
Qt is c++, with reflection via its metacompiler. I use it for all my desktop applications where I need performance.
Me, quietly suspecting for a while I’d enjoy game design a lot more than game dev: hmm
You, doing this:
Me: I WANT OFF THIS RIDE RIGHT NOW!
honestly, you usually wouldn't have to deal with things like this in most languages. C++ is just a complete clusterfuck
@@thezipcreator Agreed. Tried to learn C++, but it was too much. I think I'll stick to either C or Rust for any low-level stuff.
@@bluesillybeard for low-level stuff I think D is a bit of an underrated language. It can be used for low-level stuff or high-level things and it has a lot of features that other languages should have
@@thezipcreator I'll make sure to check it out at some point
Before I start waffling:
I guess you didn't have to bother handling anything too weird (like SDL objects, or whatever graphics interface you're using) as you only want to save game data, so I'm guessing you can ignore pointers to external library stuff (more weird than STL) ? Or did you have to write edge case stuff for all those too ?
Now to the waffle:
Genius ! I looked at the LLVM backend thinking it'd be a great tool to write my own language, but then I decided I didn't have the time (the skill or the insanity). C++ is the first language I learnt (I hesitate to say learnt, does anyone know all it's knoocks and crannies) , it has made learning everything else real easy.. Anyway some fantastic craziness there, delivered with perfect understatement.. well done !!
I've not followed it In a while, but c++23 was supposed to get reflection. At least it was one of the more popular proposals when the c++23 spec was first being announced. But this is awesome and truly amazing work, looking forward to what you do next
I locked myself outside the house but the first floor's window was open, so to get in what I did was... build an electro-mechanical elevator powered by geothermal energy of course... what else would you expect?
entertainingly overengineered
_Mumbo Jumbo_ school of redstone :-)
I've built a C++ parser from first principles before. It was... hard. Very hard. But it parsed its own source code, including the STL includes! It would have eventually become a complete compiler, but the "C++ Grandmaster" class I was taking fell apart because the instructor ghosted.
omg this game came so far since i last watched one of your devlogs!!!!!!
waiting for: i built my own custom human to serve me every need
Ooh! I did this too, but for typescript which was waaay easier. I don't think I would have done this for C++ due to it being too fragile.
So I have reflection for serialising data but it's purely runtime (and much easier because I define specifically how to save each type, I just made it so I only have to decorate types to make them save/load properly).
I also have a compiler plugin that adds proper mixin support.
So I feel like I've done both sides, but in easy mode.
I'm in the process of writing a remote DWARF debug info server/parser. Stack unwinding wasn't too bad even if I couldn't use libunwind's built-in remote unwinding for... reasons. Then came the part of parsing actual debug info so I could see where and how variables are stored and... Your comment about edge cases upon edge cases is soo true for this as well 🤣
This game is looking amazing! I'm really looking forward to seeing it
reflection is pog
I program in C++ and find your work incredible.
5:15 "That looks like progress again, but actually isn't".
Mood.
Good lord... This is giving my PTSD of when I worked with C++ xD
The absurd complexity of everything yet the surprising lack of "common" features and those compile times...
The compile times especially hit me hard, after getting used to regular C, where quite often I'd just recompile whole projects while testing, just because it was fast enough to not bother me. Was also WAY less error prone to the kind of wacky compilation anomalies only C/C++ have (as far as I know).
I'm genially impressed by your approach, dedication and knowledge. All that said, I don't think writing compiler extension it the best idea :)
I think one of the approaches you could take is just use some interaction with Rust, because it's already has decent serialization without much of boilerplate.
Also for my personal needs I implemented somewhat similar to Unreal Header Tool, which just scans file for macros/tags and then do runtime serialization, so you not writing whole C++ parser, but a small part of it. I did a very very basic stuff, so only few types and containers are recognizable, it doesn't work well with templates and custom spacing/aligning etc, but I think you can make it pretty generic too.
"The deep abiding goal of every engineer is to take a thing and make it do a thing it wasn't supposed to do."
-Reddit.
Careful: reflection-based serialization is somewhat fragile. Half the work in serialization libraries is making sure you can still load a file from several versions ago, and you still can't really do that automatically. You need to bear in mind backwards compatibility. For example, I add a new field to an existing structure, but then load an old gamesave from before I added that field. Do I have a default value? Does it make sense?
the game looks so cool
You know, im not very smart, but from what i've learned with coding so far is- "If you can't figure out a way to do it, even a hacky way, you probably shouldn't."
Greenspun's Tenth Rule embodied
I wonder, could you take a snapshot of the memory as a dump of sorts, and then allocate that same state again? Another idea could be the use some sort of custom base class or template that could record all needed data. Perhaps with a touch of boilerplate code generation. Not saying every language couldn't use proper reflection features, especially cpp.
Anyways, very impressed that you got it working, well done!
The "replace pointers with IDs" technique, which I already saw in other libraries, caught my attention...
I have actually looked into this approach for my own project, but decided against it due to platform incompatibility. Since this reflection system is inherently tied to CLang, you cannot use it with, for example, MSVC, which i needed for multi-platform support.
I instead made a templatized reflection system that generates type names and some basic information at compile-time from within C++ itself, and then you can explicitly reflect certain additional aspects of your types such as attributes, inheritance, functions, etc. at runtime during plugin initialization
you can actually use this on systems which clang doesn't support - the plugin then will just emit source code which can then be used by the target compiler just as any source would be included in the final build
@@jdh That is true, but that still means that you cant just compile with MSVC from the start. I'm making a game engine so I can't really force the user to run CLang plugins and *then* compile with MSVC :p
Also personally, I am just not a fan of external tools for codegen :P, which is why i went with compile-time generation of reflection data.
You can actually do a lot of similar reflection stuff at compile-time with templates, like using `__PRETTY_FUNC__` + some lightweight constexpr parsing to generate a demangled string name for any type, generating casts for integral and floating-point types, enum to underlying type casts, type-erased range and tuple-like type binding, type-erased string-like type binding, etc.
If you want to do some weird stuff like that, you might want to consider trying the Scopes programming language.
It also has compile time execution. I wonder, if it's possible to create a similar system for scopes without hacking the compiler.
Scopes even caches compiled functions into object files.
It takes loyalty to parse through the inner workings of C++ and still say the makers are smarter than you
So you've single-handedly added a reflection system to C++!? I am at a loss for words here.
I will just have to keep it to myself that I'm enjoying this kind of content.
-so did you watched anything interesting yesterday?
- cat videos
holy memory safety batman
As someone who's compiled language experience is mostly Go, this video makes me want to cry from fear and anxiety, lmao.
Thiis is extreme procrastination in the same sense that climbing a rocky mountain on a dirt bike is extreme sport.
Ok this was just absolutely ridiculous and amazing thank you for blessing us with this
Ah, I remember reading an article or watching a video recently on how C++ is virtually unparseable. I mean, it of course is, it works etc., but it all has some "hacks" and "limitations". I mean, Templates being turing complete is one issue, but there are of course various magical things in there as well...
Small brain: Saving data by serialising arbitrary structs
Big brain: Saving data by 'fwrite'-ing arbitrary structs
My solution to this was to define > operators for every object and struct in my game, similar to how std::cin and std::count does it. From there, my game data was human readable and writing a parser was fairly simple.
istream and ostream made serialization easy with ambiguous implementation!
I like your funny words, magic man
and here i am using c and saving things with a custom file format
That's not quite my first solution but you do you
This is crazy. This man transcends us.
Your work is incredibly impressive
Really impressive, keep up the good work 💪💪
I'd wager the heat death of the universe would win between those two, so good choice here.
the -saddam hussein-- ROBOT hiding place got me
Risks .. compatibility over code changes? - does the seralizer emit to somethign like json or whatever, that uses simple named key/value/recursives, or is it just dumping out binary of the objects in question? which is to say .. do you have the risk that over time, if you release updates to the game, will it invalidate their existing save stuff since it was autogenmerated to match a certain set of code, and now the code coudl be significantly different, but you'd want the saves to be compatible?
I'm also curious about this. If a compiler update can break it, or I can't transfer data between different types of computers this becomes much less useful...
You didnʼt mention where the ships that the historical Archimedes (probably didnʼt) use reflection to burn were… they were in the C.
how did I miss that
I wish youtube brought to your videos more attention so you could kind od get at least some money back for your time you sept on this. But at least, man, your knowledge, despite coding for 8 months as you said, is quite immpresive.
I personally would have gone for those approaches first:
1/ macros
2/ manual serialize() deserialize() in every object
I'm surprised you didn't think of using macros.
it's definitely faster to add syntax coloring to your Vim for a #define named STRUCT_SERIALIZABLE that colorize as a struct.
Than to hack the compiler the way you did.
In any case kudos for the hard work and ingenuity.
He's stated if this wasn't for TH-cam he would have already did that and have been done with it. TH-cam is his excuse for getting freaky with it.
@@mitlanderson I know, I heard him saying that. However in his approach he's listed the steps he took, and the two I've listed are missing.
Basically ”Declare war on god and win, or sacrifice 3 children”