Serialising/Deserialising Structured Memory To Disk, AKA Savin' Stuff
ฝัง
- เผยแพร่เมื่อ 18 พ.ย. 2022
- In this tedious and long video, I talk about an easy to use file format that's great for serilizing and deserializing data, i.e. Saving Things!
Source: Coming Soon!
Patreon: / javidx9
TH-cam: / javidx9
/ javidx9extra
Discord: / discord
Twitter: / javidx9
Twitch: / javidx9
GitHub: www.github.com/onelonecoder
Homepage: www.onelonecoder.com - วิทยาศาสตร์และเทคโนโลยี
This reminds me of a recent video I saw where this guy was trying to create a save system for his game so he went on a journey to implement reflection in C++ by messing with the compiler code itself. This, by comparison, is far more straight forward lol.
Gotta respect jdh's determination to do things his way. It's a story as inspirational as his plant-growing robot protagonist.
I watched the same exact video that you were describing a few weeks back. It was quite impressive when he started to manipulate the compiler itself to read/write the data in the manner he wanted.
Yh ,I remember. That was pure madness... But he did an entire program memory dumper which could store any program at any point like an emulator.
Also it was not ideal for human read.
jdh is cool
@@hide-347 That video was a roller coaster of emotions. Awe, frustration, confusion, acceptance,
Since about 2004 I have been using a similar file format to input configuration parameters. But I hated how I accessed the data from within the application. Back then I was more worried about wasting RAM space. Now you have given me some ideas on how to improve and modernize my code.
thanks *big thumbs up*
David, thanks so much for addressing this topic. Serialization/Deserialization is something I've heard a lot about but isn't something that isn't covered in most textbooks,
Very interesting video. I don't do a lot of programming but always enjoy your content
As always, amazing video. Clear code, great teaching, and having an example we've seen before in another video is so cool!! Great mate
Welcome back to TH-cam 🐱❤
The techniques presented in this video are very powerful indeed! Great video as always.
If I may suggest something; try to add links when you refer to your past videos. For example, you mentioned a video about "a retro menu system" but if I'm interested to watch it then I need to browse through the list of videos on your channel.
Great to know that you are still producing the standalone "educational" videos.
Keep up the good work.
Great content.
Why the quotes?
Wow just as I was done watching all of your videos, you release a new video. Nice timing!
Just the topic I wanted you to cover. thanks so much!!!
About time!. That's really nice, thank you!
As always been your videos full of practical working n full of knowledge and something to learn new concept even seasoned programmers.
Good to see you again!!
thank you very much, I'm waiting for new videos about the finalization of the format
I'm super excited to see your video today! 😎👍
this man and his oneNote can explain everything
I prefer passing the indent level as an argument to the lambda; that saves you having to decrement it. But then I also prefer having a publicly accessible function accepting a file pointer rather than a name, since that simplifies dumping to log files, standard output, etc.
Yes - this code was written for teaching, not for being exactly what you'd use in a production environment.,
Awesome, :) always a pleasure to watch Your videos :)
"Human generated garbage"
-Javidx9
Yes! My weekend is saved :)
Good and interesting video as usual. Just a small comment on the fact that your format can easily be "broken", for example if some strings contain an equal sign, or worse, a double quote sign (or a dot for your property chaining with dots). The case of a string containing a double quote sign needs to escape the double quote sign with an escape character, for example, the backslash sign. The escape character then needs to be itself escaped when written. And when reading, escape characters need to be recognized, removed, and then handle properly the following character.
@Abraham Johnathan you're definitely on the right track; I suspect these routines are safe because there's a minimum of manual pointer manipulation, invalid references construct zeros, etc.
Running this through a fuzzer would probably find crashes instantly but not code execution.
Input parsing is notoriously difficult to get right. (There's entire industries built around this: sqlite3, protobuf, thrift, flex and yacc, etc. Some are horribly unsafe, some are solid.)
@Abraham Johnathan I was not talking about memory corruption or anything like that. What I meant is that with the code as it is now, if a property or property value contains some equal or double quote sign, what you will read back from the file will not be what you initially wrote.
For example a property named a=b with value c and a property named d with a string value e,"f,g",h will be written
a=b = c
d = "e,"f,g",h"
and will be read as property 'a' instead of property 'a=b', and with value 'b = c' and property d with some value (not sure exactly what the value will be, it will contain 'e', but then you get what the reader thinks of as a closing double quote, so it's possible that the reader thinks that 'f,g' or 'f,g,h' is a new property name, would need to double check the code to see what exactly happens).
Another great video.
For my data file utility I used the object dot notation each parameter, and didn't worry about making it easy for humans to edit 😁
Based on advice on the Internet, I found using std::stringstream to be a really easy way to parse / tokenise a string based on a delimiter.
A huge point not addressed is what is the "best" / recommended way to make an object serialisable but minimise the coupling between the objects being serialised and the Class doing the serialisation. Here you've added the serialisation code into the object, i.e. coupling the object to the datafile class; but what about a "shapeserialiser" class that knows what data each shape wants to save and how the datafile will save and restore that data and then create Shapes from the datafile; or possibly a Structure of data names and values that is passed between the Shapes and the datafile; or are there better ways?
Also what I do when loading a data file with an unknown number of objects that can have variable amounts of data, is to create an object add it to a Vector or Map, then get a pointer to the object in the container and then add the data to the object referenced by the pointer.
Thus, when the data ends I don't have to add the last object outside of the main main data reading loop. Good when adding an object requires a number of steps.
ty, with some effort made example work, happy!
Thank you!
fantastic!
That sounds like JSON with extra steps! :D
Good work though, very informative on how to write your own data structure - I would honestly went with JSON as it's standardized and why reinvent the wheel anyway but that's me. Kudos for great thinking!
100% agree but it's a fascinating intellectual exercise. That said, he did point out he's been using this format for 15 years, and while JSON was certainly a thing back then, it wasn't as ubiquitous
@@jgcornell of course! Never said anything is wrong with it!
Hi David, thanks for the video. I have a question that I think is related to this tutorial's content. I want to build a OpenType font making software. I have experience in programming but I have never done a large scale software design and programming before. Would you give some pointers and tips about where to start and how to proceed? Thanks in advace and take care.
Thanks Javid , still working on SIMD for you should have it to you soon. John Ireland 🇮🇪
Excellent.
Did not expect u to serialise this way
I'm not sure if I maybe missed the part where you covered that edge case, but what if there's a quotation mark in the string that I want to save? I think you'd need to add a quote escape sequence.
Super cool stuff with saving data in a line by line fashion so it could read in correct order using a stack. I was wondering while you were writing the save function how are you going to read it ?. OLC rocks !
Man, I've been trying to write a yaml parser (there isn't a good pure-lua one) for a bit and it is making me appreciate JSON more and more.
You want a string? Wrap it into quotes. One kind of quotes.
One kind of mapping, one kind of list.
Python's whitespace may be funky, but zen of python is correct - there should only be *one* way to properly do something.
i like the style of your video.
There's some stuff in here that's not been featured in your previous videos. What's std::function? The inline keyword? Apologies if these are too basic.
I have dealt with this file format quite a lot (Kerbal Space Program save games, where I made an editor) and always wondered if it has a name and some standard implementation along with it. I did not know a name and you didn't mention one either. But I also had some fun implementing my own parser.
Just to annoy you or someone else: How about the quoted strings having quotes within themselves 😁
Oh my, all this child talk and array indexing just reminded me of a project i'm working with where im reading json data and building a class that can grow and build its own tree system where its hooks object on itself or data depending of what its read.
Is there any particular reason why you wouldn't treat simple strings as a list of one string (apart from waisting a few bytes for map->index pair)?
Javidx I wish you were in South Africa man. I want to be your student. I am really obsessed about learning the technology behind CAD/CAM systems and your videos really helps me. Are you ever gonna visit SA?
This reminds me a bit of device-tree sources.
Ah! the lost art of game saving...
Finally!!!
Javid remember the graphics engine series WE NEED A PHYSICS ENGINE SERIES NOW!
heh, nice work using reference_wrapper to maintain the modern C++ idiom. One of the big failings of C++ I feel is the poor implementation of references, i.e., that they can't be reassigned (understandable but occasionally irritating from a programmer perspective) and that you can't readily pass around the reference objects themselves without having to resort to what I consider a kludge, having to use a STL structure to hold them. It's always seemed to me that references and object moves should be handled properly by the language syntax without having to rely on work-arounds in the STL.(you could argue that std::move is just an alias for an && cast and therefore moves are handled properly by the language, but we always use move and the proper way to do that is to not omit the std namespace identifier. It must confuse newcomers no end.)
Great video.
Is there a reasion you write `if (file.is_open()) { ... return true; } return false;`
Instead of writing `if (!file.is_open()) return false; ... return true` to reduce indentation? Or is that just the style you are used to
I'd argue the '!' is more easy to overlook. So it's less readable as a result.
@@ScibbieGames One could also write "not", but in such cases I often tend to just put a space between the ! and its argument to make it more visible: 'if (!file.is_open())' --> 'if (! file.is_open())'
that's unreadable
@@nimi5965 if you can't read an early return and a negated statement then you should probably focus on the fundamentals of programming my guy.
@@anarchoyeasty3908 i know how to read i'm just saying why the unnecessary complication
Can you build a simple compiler? That'd be cool. Its a very confusing subject for me probably do to the multiple ways of implementing it.
could you explain the design decision behind checking your vector for space then resizing by one instead of just using push_back? I imagine it's because you want to be more in control of space utilization at the expense of time efficiency?
Assume the vector is empty, then the first thing the user does (and no reason why they wouldn't) is set an item at location 5. I need the vector to contain at least 6 elements.
BTW, if you make your separator a single char or C-style string then you can construct a string as e.g. *std::string(count, char)* or *std::string(count, c_str)*. That way you don't need to do the loop yourself. Less typing!
Yeah!!!!
1. Indentations can be precomputed/cached and then reused after.
2. Strings should always be in quotes. Comma or space are just two special cases, there are many more.
3. Parsing was preatty weak and not very effecient. You said yourself that whitespace in invisible yet you divide your pair by lines. CR/LF conflicts will arise one day and display it all wrong or corrupr the while file. Also parsing by just finding the "=" symbol is not very reliable, you should condider making a lexer for that purpose and check for tokens, not for symbols. Also you suppose your strings start and end with the quotation matk which makes it impossible to include escape characters like \" which may be required some day.
4. Adding elements to collections seems unhandy, a simple Add(...) will do. No need to go through the whole data structure twice.
5. How is than better than JSON?
Qite an interesting video. However, in a production code I would have preferred boost::serialization
I find JSON serializing to work almost this way, and there are a lot of very fast JSON parsers already. Seems like a case where rolling your own might be a bad idea?
Edit: As others have pointed out, teaching concepts is the idea here, not using existing libraries. Should have thought about it more :)
I've used #include before. The biggest issue is you have to pre-process out the comments and trailing commas. And of course in JSON you can't mix subscript arrays with object notation. I've been tempted to just throw LUA into my project, but giving your program arbitrary code execution just to have a nice config file format seems a bit overkill. :P
There's one very good reason to use a JSON library in a game engine though, and that's glTF files.
Not necessarily especially when it comes to 3D Graphics, Game Engines... Yes it is quite a good amount of work up front however rolling your own can provide several benefits to you. You have full customization of your own tools that are centered around your own user defined objects. You can customize it by adding new features at will helping to make it easier to allow the user(s) of your product the ability to update to a newer version with ease with less chances of it breaking your application. This also has the benefit that you are not tied to some "standard" or external library or API so that in future cases if they update to a newer version of their library or API and you try to incorporate those changes into your own code base, you don't have to search through your entire source code to find every instance you are using it to make sure all of the changes have been made. When you have a Game Engine that was written / designed by you and or your team as opposed to a 3rd party one, there can be 100s of thousands of lines maybe even a million lines of code you'd have to go through. By writing your own, this decouples the need of relying on that dependency. More than just that, you as the developer of that project has a better understanding of the internal workings and it also allows for more fine tuning. The caveat to it is, there's more work involved and a little more code to maintain. In truth it all depends on the project's current requirements and specifications. This also depends on who you are working for. A given company may require you to use a specific library and or format because they have an existing code base where they have invested millions. A different company may refuse the use of outside libraries due to many reasons such as copyright or the fact that they have their own in house tools. So having the ability to roll you own isn't always a bad idea, but having the ability to do so is just another tool added to your tool box. In the long run it better than not knowing how to do it and just relying on existing libraries. There are pros and cons to both using both approaches. However, there is always an added benefit of knowing when and where to use which approach. Now if you are creating a project from a source of a hobby, then either approach you choose is just as valid as the other. There is nothing wrong with using 3rd party libraries, and there is nothing wrong with writing or designing your own. Also by creating your own file format, it will strengthen your ability to implement file parsers either being in binary or text format. They are great exercises! There is also one more added benefit of rolling your own, creating your own file format is that if you don't release it to the public then it makes it harder for others to "hack" your project because they don't know how your storing your objects in memory. If you use something that is well known like json or xml and they can read it, it's easy for them to manipulate the text file and break it. However, in this video he was using a text file parser as opposed to a binary one. And the intent here is to make the text file human readable. So this is something else to consider. It could also serve as a system of security or anti-cheat by having your own format that hasn't been disclosed or released to the public where it's only available to the team who is working on that project.
Even after reading your edit, sure you rushed to conclusions but I still don't understand would it be a bad idea just because you personally find json similar?
I spent 3 hours trying to write my own HTML parser and gave up, mainly due to the fact that a lot of HTML is slightly malformed (e.g they use the opening node like but never close it with ). Would be really cool if you could do a video on that too!
sadly you will need to implement a tree, html works only this way
if the html is malformed a new node is not added, if is ok, just add a new children node
Learn about context-free grammars
html is special
What if a string contains a close curly brace or escaped double quotes, will serialization and deserialization break?
Good one! Then also think about how to distinguish the string "javidx" from javidx (with and without the quotes respectively). How about strings containing whitespace at the start/end or within, or that have one or more embedded " characters ;-)
Hi Javid can you do a video on creating non blocking timers, for triggering events, having events take a certain amount of time etc. I’ve used win32 set timer function but it just runs concurrently and waitable timers block the game loop.
IIRC standard win32 timers are not intended for multimedia purposes where precise timing matters. People use high definition timer functions of WinAPI.
You need to run the timer asynchronously, and then communicate to the other thread when appropriate.
26:24 LooOOOL
24:24 I think there's no necessary another function for indentation, it's enough just one line:
file
what if you want so store the character = " in your system? """ it is not going to work, does it?
Can you make a vídeo explaining those keyframes? C++ have more than 90
Wisdom's sound.
one technique I've seen is have a boolean for whether you're saving or loading... if you're clever, the serialization and deserialization code can be the exact same function, just with the bool flipped... haven't tried this yet but I want to
Well it's not to say that it couldn't be done, however, you want to call the open/close functions as minimal as possible. So to have a function branched base on this flag would be tricky because the iostream variants for reading and writing would have to be defined after this bool variable is known. Have an external standalone read and write function that are independent of each other does mitigate the dependency of having to know that value of boolean flag variable. When you open a file stream for reading versus one for writing they are different stream type objects. And the mechanics of reading info into the stream from disk or memory is different than writing it out to disc, memory, or console....
David just created NBT 2
at 25:50 I had a small issue where the ending curly brackets would line up at the end and not go back. I resolved it by subtracting from the indent amount.
It looked something like this:
some_node:
{
name
{
nickname : "jav"
}
}
after my fix it now looks like this but I think I missed something cuz he didn't do this in the video:
some_node:
{
name
{
nickname : "jav"
}
}
any help would be appreciated.
this video is amazing though, I'm probably just dyslexic or something lol.
11:06 I'm not a c++ programmer, so this code may be fine, but serializing floating point numbers without taking current locale into account has bitten me in the ass so many times I've lost count. Always make sure that your code uses strictly defined locale settings for (de)serialization, otherwise you may be very unpleasantly suprised when your file that works fine on one PC refuses to be read (or worse, silently reads garbage) on another.
I prefer to use XML, since it's used in lots of professional software (like GTA V)
Ah XML: the file format that is simultaneously hard to read as a human and surprisingly hard to parse reliably in code. I for one am glad it's a thing of the past.
@@markmuir7338 Ah XML critics: people that have never really understood the format, but now build tools for other languages that achieve the exact same thing.
@@graealex Yeah, people will criticize XML for being unreadable when it's obvious they've never even tried it. I think it has valid criticisms to be made, but readability isn't one of them.
@@IndellableHatesHandles I think it is actually more readable, because contrary to most formats, the closing tag repeats the name of the opening tag. If you have a complex JSON or YAML, where it is just curly braces or just the indentation level, it gets confusing very fast. Plus I feel like using attributes in a different way than elements also makes it easier to read.
Besides that, I think XML makes all the right compromises, people underestimate the abilities and tools surrounding it, yet they claim to not be able to only use a subset of it... and then they rebuild exactly the same tools for a different language, because it turns out, the people that built it for XML didn't do so out of boredom, but because there is a need for it - for example to have schema, or transformation languages.
Even the problem of verbosity has already been solved, not once, but a few times (EXI, EBML, WBXML, BML and a few others).
How about yaml or JSON?
can you make a video about saving stuff but in binary format?
How about write it into binary? any suggestion?
So this is what Gilfoyle is doing after Silicon Valley
If you wanted to have a simple almoust binary format... use msgpack
reminded me of nginx config format
I ran across an XML parser a while ago that avoids extra string allocations by loading the entire file into contiguous memory and adding null terminators to all the values. The C++ equivalent would be string_view, but I think that requires a standard upgrade to C++20. It seems like this would be a good use of std::variant also, so you could directly assign nodes to ints, doubles, and strings in code. An .append() method would also be useful for automatically managing the indices.
But all in all, this is a pretty neat implementation. I like the use of std::stack instead of recursion. Preserving the order in the file is something I wouldn't have thought about, but in that case you'd want to preserve the comments, too, if you ever had to overwrite something that users might manually edit.
I'm not certain but I think string_view might be accessible with C+17 at a minimum however, the C+17 compiler may not provide all of its functionality or features. It's been a while since I've done any programming and C+17 is the latest compiler I have installed on my pc. I don't have C++20 or newer, and I remember being able to experiment with string_view a little bit. So it may be partially supported and I was using the MSVC compiler as opposed to GCC or Clang.
@@skilz8098 Ah you're right, it is a C++17 thing. I guess I was thinking of ranges. 20 is awesome though, you should try it. CppCon has a lot of good talks going over the new stuff. In the past month it's all about 23 though, so you'll have to scroll back a ways.
Wooooooooooooooooooooooooooooooooooooo
C++ for Life
Looking at the kind of file-reading api's that are coming into play as we speak and very recently, where you separately allocate a chunk of space for the data to be read in and then get the entire buffer filled from the OS - this makes little sense. Parsing data like this for games leads to significantly higher loading times than simply reading the buffer from disk with a memory-prepared format (which then doesn't require any parsing). Why go through and parse every four bytes or sometimes even smaller when you can read tens or hundreds of megabytes without having to load all of that data into a cache, process it only to store it someplace else in memory.
I mean it's great for learning some basics about serialization and good presentation in regards to that. I didn't mean to come off completely as negative as I sounded :)
First time I heard about serializing and deserializing when I read about java Serializable interface..I don't really know what that is..from the description it says convert data into an array of bytes..
I don't really understand that..
Hopefully after watching this video I will understand it better
If you know what JSON (for example) format is. Then Serializing is converting your objects to this format as a "string", Deserializing is the opposite process. As I get. XD
I bet it's harder than this and saving data in various binary formats means it too, but in general, it's something like this. And if it's not, then it's just converting date from your objects, maybe even some other data from variables and this kind of stuff into some ASCII text format.
Java has reflection, so it has no trouble analyzing classes and defining a way to put individual objects and their fields into a binary stream. Besides saving and loading data from disk, this is also useful to pass data over network communication, although for interoperability, many other, often human-readable formats are used. In other applications where bandwidth is at a premium, more compact formats like ASN.1 are used. A lot of stuff in telephony and telematics use that format.
hi david, how old are you?
Long time no seen your video :) Hows your baby, is he or she?
mmhh... maybe I am a minority, but I'd like to ask setup and usage on visual studio: I come from VS Code and a lot of terminal so jumping to a full IDE was a big jump that left a lot of confusion. Seeing that you use them a lot maybe you can show us some tips and tricks.
VS has a lot of options, but most can be ignored. You don't have to use every feature, but it's nice to have them there once you do need them.
Edit: if you have specific questions, reply here and I'll help you (or share contact info, discord, email, etc.)
@@theRPGmaster Can I contact you on discord or reddit?? I wanted to learn more about cmake integration, source control and in general useful feature to OOP..
@@stefanopilosio6838 You can always search for that online. Documentation helps.
Also, you don't need to use Visual Studio if you don't want to. That's up to you. Using a editor with a terminal is a choice that many professionals make.
Comments in config files are indeed useful. It's amazingly annoying that JSON has taken off as a config file format since it doesn't officially allow comments. Of course they say that it's only for data transfer over the network - well then please would people like Douglas Crockford make a public announcement NOT to use JSON as a config file format? But no, instead you get people making JSON interpreters that break the format and allow comments anyway, so that some others will complain about it being an invalid JSON file. sigh.
Looks like someone just invented JSON.
why haven't I seen this yet?
32:38 Yeesss, python shouldn't had ever been a thing imo lol 😈
DirectX 12 D3D tutorial? :P
Conspiracy theory: Height and Name seem to be true, so I assume age is also true, but it's probably hex instead of dec!
Hi. I know it's off topic, but Im currently on a m1pro 16inch 32gb. It's definitely a decent machine. However most of my tasks require Parallels18 Virtualization to run windows exe's, but even then they don't run native inside parallels, because I use the pre release ARM win11.
MacOS a closed system, I find myself not using all my potential nor the systems capabilities and because of this my plan is switching back to Windows.
I'd say im an intermediate lvl C# user(2.year at Uni) and I like Assembly too(unusual I know). I also like the RE topic, which I found inconvenient doing on the ARMvers.Windows.
I never owned a Lenovo or IBM device, however I had a few dell maschines.
From the design perspective, Thinkpads are just way beyond anything a have seen so far including MacBooks. It just feels premium and professional.
The specs I would go for is a 12700h, QHD165 panel RTX a2000. ( I have a rtx 3060ti desktop I can use as an eGPU for heavier tasks in the near future, maybe some gaming although im not fascinated by games, but rather crating things).
Do you recommend the Thinkpad X1extreme or P1 gen5 models over something else like a dell XPS or my MacBook in the first place?
The whole mess of special casing for if the string contains the list separator character is why I always go for simpler binary formats. "Looking pretty" is very, very low on the list of priorities.
How often do you need to read one of your files to find why it's not loading correctly? Or create a test file to feed into your program? You can do this with binary files but then you probably need to write a separate translator into a text format. It's usually easier, if slower and less space-efficient, to just use text instead. For game saves, however, you might want some form of obfuscation anyway, just to discourage someone from going in and changing their number of lives to 999999.
when I wanna get saved I ask JESUS
?
Then you are expecting a life full of read write errors.
@@Yleski probably a joke about jesus quicksaving became he came back 3 days later
JSON External Saving Unbreakable System?
@@TheWeepingCorpse gold lmao
the file format looks like a cross between json and yaml
I absolutely agree, whitespace should be invisible. The designer of Python is daft, and I can't wait for Python2 to die. If only Python3 would too.
DAY 1 of asking for a video about metaballs and isosurfaces in c++.
Javid is 24? Is that hex?
💯
I think I detonated a WWII German neutron bomb.
-W pedantic is upset about the semicolon after the namespace closure. On a positive note, javid invented Json before it was cool.
pls start doing rust videos!
valve uses this way in their source engine for decades to load files like models, textures, and work on UI
Anybody knows good video explaining how to create multi-file cpp projects ?? Thank You in advance. :)
only God saves, people saving is in bits only.
@39:43
Hi ..is line in the description : "Source: Coming Soon!" == true ? click link : keep waiting :) ;
Lol sorry, it's there in the PGE repo /utilities, I need to update the link
hi,
you scare me..
what should we do about it?