Click this link sponsr.is/bootdev_dreamsofcode and use my code DREAMSOFCODE to get 25% off your first payment for boot.dev. That’s 25% off your first month or your first year, depending on the subscription you choose.
I wrote Go professionally for years. I've been writing Rust for work lately and enjoying how easy it is to embed assets into the built binary and wishing Go could do the same and apparently it CAN and has for years haha. Thank you for making this video, I would have had no idea otherwise. I completely missed this one in the release notes.
The embedded files are not loaded into memory. you can embed a 10gb file and try to read it and it's still not loaded into memory. I've tried to see at what point it does, but it seems go is doing some amazing magic to minimize memory usage with embedded fs.
i am not sure if go does anything to optimize, but the linux kernel doesnt load the whole executable on startup. rather, it is in like swap like position - looks like it is in ram but stored in disk
Being relatively new to Go, just a couple of days ago I had to include migrations via my Dockerfile in the exact way that you showed in this video. After doing it, I immediately wonder if there was a better way to achieve the same result and was going to check that on Monday. The perfect timing of you releasing this video today is kind of hilarious for me lol. This video is a godsend. Your Go content has made me appreciate the language so much more and I'm enjoying it a ton. Thank you so much!
@@fikrirahmatnurhidayat4988 For sure! Depends on your use case tho. My PG model is just a couple of tables for Timescale. I'm using Firebase for everything else. So if I ever need to include a new migration, I can just add it to my repo, restart the service and voila. Now if my relational model ever gets more complex, I'd rely on a separate CI job for sure.
go:embed is one of my favorite features of go. It works really well with static compilation since you can create a completely self-contained executable that doesnt need to read any other files at runtime
Loving this feature, mostly using it to embed HTMX and other static web assets into my binary and ship it as a single binary. Great to learn new stuff about Redis and much more through your video, thanks! Also thank you for all that Go content!
8:40 by the way this can be resolved by configuring tree-sitter properly. You can get highlighting in a language that's not the main file's language as long as there is a way for treesitter to identify the chunk. It works fairly well with SQL queries embedded as strings.
At work we have been using embed to hold database fixtures for testing. also quite useful. and saves you from error checking inside of tests. The compiler will do it for you :)
I'll definitely be using this. I had an issue where I wanted to create a demo iso for our device and this meant adding a 30000 line NMEA file for simulation. I didn't want to have to install the file alongside the binary just for this demo package so for the meantime I just inserted it directly in the Go source file. As you can probably imagine this wasn't great when editing the file, especially running any sort of formatting/lsp 😅
Go embed is also really useful in tests using mocked http responses. If you reuse the responses, its very clean to just use a named variable and return the bytes, string or a reader from the embedded data instead of loading it here and there. While this could blow up the binary, it's only added when running go test if it's crammed into a test file. You've just gained the same benefits for testing, which you named for production! Separate files, but nothing goes missing and you will know from the get go
I embedded a file called ".version" and before recompiling it increments the version number automatically using Bash. This also allows me to keep track of how many times I have recompiled during development.
@@dreamsofcode for simple, and i mean really simple projects you could embed classless css too. Greetings from Argentina, i learn a lot with your channel.
Hey, great video! Storing the file inside the binary doesn't fix the security issue, it just makes it harder to exploit If you open the code even in notepad/vim you will see the plain text if the file somewhere inside, and will be able to modify it Same with a checksum
Not a go expert by any mean but had to do a small once desktop app and didn't know how to do it. I settled on embedding a sqlite DB in the go executable and copy it to the install directory if it does not elrqdy exist. This way I had a working DB out of the box. I also embedded htmx et my html template so I could do the interface using web technologies. The application, when started was running a server on localhost and launching a Web view browser. Was kind of hacky pour ok for a personnal project
it is super nice because you can stick your whole website in the exe But what you are actually seeing here is go accepting its destiny as a compiled java replacement and making fat jars as exes
This reminds me a lot of the .jar files that Java produces. They are basically .zip files and the content can be accessed from within the running program.
I mostly use Davinci resolve for any animations I do, but I have an editor now who uses After effects and premiere. Most animations are done by them now and I just do any final edits or videography!
What is the command prompt value (PS1) . I like how it reacts. Adds separation for visual and then cleans it up to continue. or.... was this multimedia animation. [still cool]
This is my zenful zsh configuration! I have a whole video on my second channel about it. If you search for "Dreams of Autonomy zsh" then it should show up!
I've been running migrations at app startup for a while now and haven't really had problems with it but I've seen some people online arguing that it would be better to run migrations in CI/CD so you don't accidentally push application code reliant on migrations that won't run. What're your thoughts on that?
Personally I'm fine with either approach. I think there's some pros and cons for either way. Personally I like migrations at app start up as my automated tests should pick up if there's an issue, and blue green deployments can prevent any bad state!
The one thing I wonder about embed is whteher or not the plain text is stored in the built binary. I would love to embed things like API keys or similar in a CLI tool I built but I'm not sure if it's safe to embed that or not
API keys are never safe to embed in a binary, but to answer your question, yes, it is just copying the bytes without encryption. Assuming your API key is a string of ASCII-compatible characters, it will be quite visible to anything looking at the memory.
I guess the `go install` commands etc. don't provide a clear way on how to package text and config files. Of course, other build tools can do that, but other languages don't have this issue (Java and (I think) Python come to mind).
Templ is yes! Ive been using html/template more because of some quirks about templ that I don't like... That being said, I did just get templ indentation to work correctly and it's much more usable now... Soooo maybe I'll be using it more!
Go's philosophy is pragmatic, not purist. I don't necessarily like the idea of comments that execute either, but if you sort of shift your perspective a little, are they really comments? They start with two slashes, yes, but they are specifically ignored by the godoc tool that generates documentation from comments. The Go ecosystem tools recognize them and add linting (in the case of go:build) or even syntax highlighting. They could have been added officially with some sort of curly brace syntax, but they happen to start with two slashes. They are really only comments in the eyes of the language parser.
This is a really great nod to go and I didn't realize that you could do embed single file to string. That being said. I do think there are reasons not to do it and I'll be devils advocate. 1. I have no opinion other than Redis burned me at my job when they changed license. 2. I'm stealing this one. This is brilliant. 3. I'm not a fan of this because sometimes people need to do rebranding. Though their cache is going to be more evil about this up the pipeline outside of our control. Still even though go is fast at compiling it's nice not having to give users that kind of privileged who do design work. 4. This one swings back to your earlier argument. The moment images and video gets stuffed in the static directory it gets spooky to untangle. Thanks for making this I honestly swore embedded off, but for me #2 is a slam dunk.
@@pearl911 it is restricted. you can't do normal file IO in a const fn. you *can* use one of those macros above and do something with the data in a const fn, though.
Click this link sponsr.is/bootdev_dreamsofcode and use my code DREAMSOFCODE to get 25% off your first payment for boot.dev. That’s 25% off your first month or your first year, depending on the subscription you choose.
I wrote Go professionally for years. I've been writing Rust for work lately and enjoying how easy it is to embed assets into the built binary and wishing Go could do the same and apparently it CAN and has for years haha. Thank you for making this video, I would have had no idea otherwise. I completely missed this one in the release notes.
I'm glad I'm not the only one who overlooked it haha!
The embedded files are not loaded into memory. you can embed a 10gb file and try to read it and it's still not loaded into memory. I've tried to see at what point it does, but it seems go is doing some amazing magic to minimize memory usage with embedded fs.
i am not sure if go does anything to optimize, but the linux kernel doesnt load the whole executable on startup. rather, it is in like swap like position - looks like it is in ram but stored in disk
Oh that's awesome to hear! I must have been reading an old source on it when doing my research.
A binary with size 10gb is not deployable 😅😅
Being relatively new to Go, just a couple of days ago I had to include migrations via my Dockerfile in the exact way that you showed in this video. After doing it, I immediately wonder if there was a better way to achieve the same result and was going to check that on Monday. The perfect timing of you releasing this video today is kind of hilarious for me lol. This video is a godsend. Your Go content has made me appreciate the language so much more and I'm enjoying it a ton.
Thank you so much!
Or just have seperate ci/cd script to migrate, I found it more useful, since I can inspect the migration via CI/CD, and rollback them when needed.
@@fikrirahmatnurhidayat4988 For sure! Depends on your use case tho. My PG model is just a couple of tables for Timescale. I'm using Firebase for everything else. So if I ever need to include a new migration, I can just add it to my repo, restart the service and voila.
Now if my relational model ever gets more complex, I'd rely on a separate CI job for sure.
@@fikrirahmatnurhidayat4988 agreed, doing migrations on app startup has always caused headaches in my experience
I really appreciate the amount of Go content you're putting out!
go:embed is one of my favorite features of go. It works really well with static compilation since you can create a completely self-contained executable that doesnt need to read any other files at runtime
if your not currently doing voice over work then you should... you have that perfect story telling voice.
Thank you!
Loving this feature, mostly using it to embed HTMX and other static web assets into my binary and ship it as a single binary.
Great to learn new stuff about Redis and much more through your video, thanks!
Also thank you for all that Go content!
8:40 by the way this can be resolved by configuring tree-sitter properly.
You can get highlighting in a language that's not the main file's language as long as there is a way for treesitter to identify the chunk.
It works fairly well with SQL queries embedded as strings.
do you have highlights scm for sql inside go?
At work we have been using embed to hold database fixtures for testing. also quite useful. and saves you from error checking inside of tests. The compiler will do it for you :)
this video is really came in its time thank you so much for this information
I always use embed to embed my database migration. It’s so great.
I'll definitely be using this. I had an issue where I wanted to create a demo iso for our device and this meant adding a 30000 line NMEA file for simulation. I didn't want to have to install the file alongside the binary just for this demo package so for the meantime I just inserted it directly in the Go source file. As you can probably imagine this wasn't great when editing the file, especially running any sort of formatting/lsp 😅
I've used embed for a few quick and dirty hacks in the past. It is an absolute life saver when you need it.
Go embed is also really useful in tests using mocked http responses. If you reuse the responses, its very clean to just use a named variable and return the bytes, string or a reader from the embedded data instead of loading it here and there. While this could blow up the binary, it's only added when running go test if it's crammed into a test file. You've just gained the same benefits for testing, which you named for production! Separate files, but nothing goes missing and you will know from the get go
I embedded a file called ".version" and before recompiling it increments the version number automatically using Bash. This also allows me to keep track of how many times I have recompiled during development.
git describe > .git_version works nicely if you have used git tags.
Seems like this would be pretty good for embedding static data or scripts into CLI apps
i embed htmx and js/css files, i learned that way, never look for another solution, it just works.
Ohh embedding htmx is a good idea
@@dreamsofcode for simple, and i mean really simple projects you could embed classless css too. Greetings from Argentina, i learn a lot with your channel.
Thanks for this video. I am going to update my project.
Have you considered making a video about the pre-release security checklist you have for secure go web applications?
This is a great idea! I'll add it to my list
Heyy this is great, i had no idea there was such a package
Your videos are so darn useful, thanks!
Excellent! thank you for sharing.
it’s like rusts ‘include_bytes!’ or ‘include_string!’ macros
Hey, great video!
Storing the file inside the binary doesn't fix the security issue, it just makes it harder to exploit
If you open the code even in notepad/vim you will see the plain text if the file somewhere inside, and will be able to modify it
Same with a checksum
2:35 BRO, HOW DID I NOT KNOW ABOUT THIS I WAS EMBEDING ALOT OF SHIT MANULAY DDAAAAMMNNN!!!!!!!!!!!!!!!!!!! THANKKKKKYOUUUU
I'm building an LSP client that requires spawning a NodeJS process with the LSP server. This solution perfectly fulfills the task.
Dreams, you're still using that banana
Banana cursor supremacy
That shit is bananas
This reminds me I want a banana logo mod for my MacBook.
okay, i like this, you also inspired me to make a module with a similar behaviour for python too
I think the good old GNU linker ld also has this ability with -b binary
I currently use goose for migrations it’s very useful and I find it simpler than migrate
Thanks for this. I have a use case for this following your 4…. I have an index.html and then js and css files.
Not a go expert by any mean but had to do a small once desktop app and didn't know how to do it.
I settled on embedding a sqlite DB in the go executable and copy it to the install directory if it does not elrqdy exist. This way I had a working DB out of the box.
I also embedded htmx et my html template so I could do the interface using web technologies.
The application, when started was running a server on localhost and launching a Web view browser.
Was kind of hacky pour ok for a personnal project
it is super nice because you can stick your whole website in the exe
But what you are actually seeing here is go accepting its destiny as a compiled java replacement and making fat jars as exes
A year or so ago I was using some OpenGL in Go, and found that the embed package was a great solution for the shader files.
Oh this is a great idea!
This reminds me a lot of the .jar files that Java produces. They are basically .zip files and the content can be accessed from within the running program.
wow, that's amazing
time to embed models
now Activision can convert COD from a 250gb game folder to a 250gb game executable
Nice package, apart exact FS mode, something similar is very popular in Windows world (acros many languages), it is called resources.
Where did you get that vim notepad at 7:11 its really cool and I want one
I designed it myself! You can find it on my website dreamsofcode io in the store
It may be useful for creating installers.
I would use this for swagger docs files
that's cool. thanks
Nice Video, quick question ? What du you usually use to animate ?
I mostly use Davinci resolve for any animations I do, but I have an editor now who uses After effects and premiere. Most animations are done by them now and I just do any final edits or videography!
@ Nice, always looks neat! Thanks for the reply
I heard about `embed` from pocketbase repo
What is the command prompt value (PS1) . I like how it reacts. Adds separation for visual and then cleans it up to continue.
or.... was this multimedia animation. [still cool]
This is my zenful zsh configuration! I have a whole video on my second channel about it.
If you search for "Dreams of Autonomy zsh" then it should show up!
It makes deployment just so much more easier... copy over one file and you're done.
I've been running migrations at app startup for a while now and haven't really had problems with it but I've seen some people online arguing that it would be better to run migrations in CI/CD so you don't accidentally push application code reliant on migrations that won't run. What're your thoughts on that?
Personally I'm fine with either approach. I think there's some pros and cons for either way. Personally I like migrations at app start up as my automated tests should pick up if there's an issue, and blue green deployments can prevent any bad state!
The one thing I wonder about embed is whteher or not the plain text is stored in the built binary. I would love to embed things like API keys or similar in a CLI tool I built but I'm not sure if it's safe to embed that or not
API keys are never safe to embed in a binary, but to answer your question, yes, it is just copying the bytes without encryption. Assuming your API key is a string of ASCII-compatible characters, it will be quite visible to anything looking at the memory.
@ForeverZer0 thank you! I assumed this was the case but needed some verification
I guess the `go install` commands etc. don't provide a clear way on how to package text and config files.
Of course, other build tools can do that, but other languages don't have this issue (Java and (I think) Python come to mind).
Aren't Templ's templates included during compilation by the library itself?
Templ is yes! Ive been using html/template more because of some quirks about templ that I don't like...
That being said, I did just get templ indentation to work correctly and it's much more usable now... Soooo maybe I'll be using it more!
What browser does he use?
Feel like I found an Easter egg🌟
Hey Mate, can you make a video related to nixos and highdpi screens ? Have you anything related to this in your configuration.nix ?
I can! I currently use a hidpi screen with hyprland, will be doing some videos on it on my second channel!
@@dreamsofcode thanks mate, appreciate that
So would this be a good package for renaming Gmail attachments, mainly PDFs?
If you need game to teach you code , you cursed for life , and if for kids that's great product
Are comments that execute code a good idea?
it's as bad as preprocessors
@@nathanp3366 yeah I'm not happy about that either, go is a perfectionist's nightmare it seems
Go directives are pretty common in the language. CGo as well is all comment based.
@@dreamsofcode I don't think that is really an answer to the question.
Go's philosophy is pragmatic, not purist. I don't necessarily like the idea of comments that execute either, but if you sort of shift your perspective a little, are they really comments?
They start with two slashes, yes, but they are specifically ignored by the godoc tool that generates documentation from comments.
The Go ecosystem tools recognize them and add linting (in the case of go:build) or even syntax highlighting.
They could have been added officially with some sort of curly brace syntax, but they happen to start with two slashes.
They are really only comments in the eyes of the language parser.
Hey dream ! Van you please reveal what your terminal emulator is ?
Looks like tmux
Tmux as the multiplexer, Alacritty as the emulator
lol, functional comments, looking forward for comment injection into opensource go projects
I love Go
This is a really great nod to go and I didn't realize that you could do embed single file to string. That being said. I do think there are reasons not to do it and I'll be devils advocate.
1. I have no opinion other than Redis burned me at my job when they changed license.
2. I'm stealing this one. This is brilliant.
3. I'm not a fan of this because sometimes people need to do rebranding. Though their cache is going to be more evil about this up the pipeline outside of our control. Still even though go is fast at compiling it's nice not having to give users that kind of privileged who do design work.
4. This one swings back to your earlier argument. The moment images and video gets stuffed in the static directory it gets spooky to untangle.
Thanks for making this I honestly swore embedded off, but for me #2 is a slam dunk.
Great video! But skip the popup bubble sound
Thanks for the feedback! I will reduce it in future
Its amazing, but I think its super weird to have comments that execute.
彼の手書きは悪いです
8:45 skill issue
go-bindata was years before embed and still do the job better than it
I'll check it out!
Rust has that too btw.: `let str: &'static str = include_str!("../html/index.html");`
Although, it just always returns a &'static str
In Rust you can use `const fn` to do something with data in compile time.
Rust has include_str!() for UTF-8 encoded files and include_bytes!() for anything else.
@@adicthreex3530 is that really how that works? i thought that it super restricted what you could do
@@pearl911 it is restricted. you can't do normal file IO in a const fn. you *can* use one of those macros above and do something with the data in a const fn, though.
And Java had that 20+ years ago with fat-JAR.
take a look at goose. alot better the go-migrate
I've heard it's pretty good. I'll add it to my list to check out!