One of the only backup tutorial videos I’ve seen that actually covers restoring as well. One of the biggest short comings of most peoples back up routines is they don’t know how to restore and don’t test the restore pretty much ever Love the videos 👍
Suggestion for anyone watching this video: look up autorestic, it's basically a wrapper for restic that makes it easier to configure and allows to stop containers before making a backup and restart them after. I make a backup everyday at 4am to an nfs share from my truenas which pushes daily to backblaze b2 storage.
Hello Jim, Thank you for providing your docker-compose. Everything has started and performed the snapshot. I have tried restoring the snapshot, and that has also mostly worked. I have a small issue with running this with elevated privileges. I am running docker desktop. When I perform docker exec restic restic restore --target /tmp-for-restore XXXXX it will not restore all the files. Some of my files, for example for Homepage, or Tandoor databases throw errors - apparently permission is denied. And this is true, since to perform a manual backup of these containers, and others - such as NGINX - I need to copy them under root privileges, because the folders are marked as locked. Do you think there is a way to circumvent this? EDIT: The solution was to backup to an external drive, but restore to the main docker directory where the compose resides.
I think Duplicati does all three in one, can be docker'd and is available on win,mac,unix and accessible by browser. I only got into this yesterday, but some other channel just recovered his whole promox docker using duplicati from online mega storage. Because of this and the ease of use, I gave it a try. I was also interested in Restic because of your other backup video. But this seems less intuitive, thank though.
Great video and explanation, I was able to get it running very easily. Is there a way to also get notifications in case it fails to work? Don’t think it was covered yet it’s pretty important for security
Unfortunately there is no native notification system that I am aware of. You could create one with a script though, something I hope to do in the near future.
so helpful! thank you. can this be used to create a restic server of sorts? not sure how to get my other linux devices to backup to this without putting a docker compose on each one...
@@Jims-GarageI actually used your video for a non-proxmox setup. I am so grateful for this. I just know some folks set up a restic server in a docker on the main host and then send backups to it from other devices. I am not sure how to do that, but honestly, this is good enough and is working amazingly well on both my devices.. so I don't think I care lol. Thanks again!
Great, thanks! Where would I get the backup id for restoring when restic container itself is lost? Any chance of restoring backups with a fresh restic instance?
Great demonstration, looking forward to implementing the solution. However, I do have a question about the containers where databases are running like postgres, mariabd, etc. I hear that we should be shutting down the containers before the backups are being taken. Does restic support this? or maybe you have a better way of doing it.
Thanks, I appreciate the feedback. If you're using sqlite you should be fine (as it's a file). However, using mariadb mysql etc, it's best to use the in-built backup method for that solution (typically you can exec into the container and do it - possibly via Cron). Otherwise, stop the container and run restic (again can be done as a Cron).
Is the docker-compose file correct? The restic main job has a repository env_var of "/restic", then later you bind a local folder on the host to the docker container volumes section but in the "check" and "prune" containers there is no binding of the host folder to the container (there is no volumes section at all), and the logs of both services gives me "0 snapshots" found.
@@Jims-Garage is it optional? if I want to check and prune it "must" be done on the same repository of the restic backup, checking a different repository has no sense. am I correct?
@@Jims-Garage I have checked the official repos. Its seem not using local storage as a place to backup. I think we still need to mount the folder to prune and check as well
@@Jims-Garage Thanks for the prompt reply. From some reading I did on Reddit, seems Restic is indeed a better, more efficient and reliable solution, with the only downsides being the lack of a good GUI and occasional spikes in RAM usage. But I guess none of the those apply to server backups.
you sure the docker-compose file is correct? in the other 2 containers, there's a restic repository = /restic but this path is not mapped on an actual volume like in the 1st one, so what they're supposed to check? it seems like the other 2 are using something that in kubernetes world is called "ephemeral storage"... which will be lost on k8s pod restart...
@@Jims-Garage exactly, but it's mounted only to the 1st container in the docker compose... how do the other 2 know about that if not explicitely added as a volume to those 2?
@@squalazzo ok, looks like it has changed in more recent versions. Previously it would default to that location. Check here for the latest version github.com/djmaze/resticker/blob/master/docker-compose.example.yml
Very nice tutorial, was able to set it up, I have my portainer_data in a docker volume which is in a different location so i was able to map both my docker folders and the docker volume separately into the volumes path following your tutorial. I did a test restore and everything is working well. I do have a question, recovery makes sense when original vm is up and running (and I AM making backups of that vm). However, I'm having trouble thinking through how to use restic if I am restoring to a fresh machine without the original docker running. Will I be able to do this directly from command line with my encryption password without the docker layer?
@@Jims-Garage thanks! I'll experiment with this from a separate computer that has that path mapped to do a test restore just so I understand my options a bit better. Cool stuff!
I would recommend using SQL's inbuilt backup tool, and outside of that, use restic when the container is shutdown. Using it whilst in use can result in failed backups or data loss. You could script to shutdown prior to backup.
Those K8s configs I see in homelabs and small company job offerings is like taking that Apache military helicopter to go for an Moskito. Lunacy. I mean the technology is great but...
@@tobi5106 depends. It demonstrates that you can understand complex systems. A bit like a degree, rarely used, but demonstrates being able to operate at a specific level.
Anyone have any idea why my snapshots are 200 bytes for a docker data over 9gb? It keeps happily making backups, but they're clearly not the correct size. It says "processed 208179 files, 9.270 GiB in 0:12" and then makes a snapshot thats tiny
This is great and working. I got all my containers going just as you have outlined above. There is one thing though, I like to get a status emailed to me after a backup regardless of success or failure. I'm looing for a way to do that with this docker implementation. Is there an easier way?
@@Jims-Garage That is exactly what I'm going to do. Looks pretty straight forward to get accomplished. The gem I got from this video though, was restic. So slick and better than just running an rsync. Thanks again!
I'm obviously missing something, but it seems to me that: If the original backup captures ALL the data available to be backed up, and all subsequent backups only capture the incremental changes, and all backups are eventually pruned (when they pass their 'use by date' in other words) eventually you will only have a lot of incremental pieces of data still saved, so, when you want to restore and Restic Restore runs back through all the available backups that exist, it may find less than the total amount of data required, because the original 'Full' backup was Pruned long ago. Where am I going wrong with this logic?🙄
You specify a backup schedule, those are full backups. Between those dates you specify an increment backup schedule. So I do weekly backups, daily incremental backup (aka snapshot).
Hi there, I seem to be getting an error with the backup, below is the error log..... /etc/dockervolumebackup/conf.d was found, using configuration files from this directory. /root/entrypoint.sh: source: line 18: can't open '/etc/dockervolumebackup/conf.d/*': No such file or directory
One of the only backup tutorial videos I’ve seen that actually covers restoring as well. One of the biggest short comings of most peoples back up routines is they don’t know how to restore and don’t test the restore pretty much ever
Love the videos 👍
Thanks, I totally agree. Just as important, if not more so, to test you can restore!
Suggestion for anyone watching this video: look up autorestic, it's basically a wrapper for restic that makes it easier to configure and allows to stop containers before making a backup and restart them after. I make a backup everyday at 4am to an nfs share from my truenas which pushes daily to backblaze b2 storage.
Also, check out backrest, stupid name so google doesn't help much unless you search "backrest git" it's a webui for restic
You sir are a gentleman and a scholar... thank you
Glad to help
thanks for the demo and info, have a great day
You're welcome, Chris.
Great video! So concise and to-the-point
Thanks 👍
Looking forward to this one!
Hope you like it!
Brilliant. Thanks Jim!
Hello Jim,
Thank you for providing your docker-compose. Everything has started and performed the snapshot. I have tried restoring the snapshot, and that has also mostly worked.
I have a small issue with running this with elevated privileges. I am running docker desktop. When I perform docker exec restic restic restore --target /tmp-for-restore XXXXX
it will not restore all the files. Some of my files, for example for Homepage, or Tandoor databases throw errors - apparently permission is denied.
And this is true, since to perform a manual backup of these containers, and others - such as NGINX - I need to copy them under root privileges, because the folders are marked as locked.
Do you think there is a way to circumvent this?
EDIT: The solution was to backup to an external drive, but restore to the main docker directory where the compose resides.
Good job, glad you sorted it.
I think Duplicati does all three in one, can be docker'd and is available on win,mac,unix and accessible by browser.
I only got into this yesterday, but some other channel just recovered his whole promox docker using duplicati from online mega storage. Because of this and the ease of use, I gave it a try.
I was also interested in Restic because of your other backup video. But this seems less intuitive, thank though.
Great video and explanation, I was able to get it running very easily. Is there a way to also get notifications in case it fails to work? Don’t think it was covered yet it’s pretty important for security
Unfortunately there is no native notification system that I am aware of. You could create one with a script though, something I hope to do in the near future.
How can I backup multiple directories with different tag for each one?
so helpful! thank you. can this be used to create a restic server of sorts? not sure how to get my other linux devices to backup to this without putting a docker compose on each one...
This is designed for docker containers. I recommend Proxmox backup server for other Linux VMs (I have a video)
@@Jims-GarageI actually used your video for a non-proxmox setup. I am so grateful for this. I just know some folks set up a restic server in a docker on the main host and then send backups to it from other devices. I am not sure how to do that, but honestly, this is good enough and is working amazingly well on both my devices.. so I don't think I care lol. Thanks again!
Great, thanks!
Where would I get the backup id for restoring when restic container itself is lost? Any chance of restoring backups with a fresh restic instance?
Yes, redeploy and keep your keys the same. Should be fine.
Hey James! Are there any benefits for installing restic using docker as opposed to a regular binary install?
IMO yes, the configuration is saved as code. Every time you deploy it it's the same (and quicker).
@@Jims-Garage Ah I see, Thank you!
Great demonstration, looking forward to implementing the solution. However, I do have a question about the containers where databases are running like postgres, mariabd, etc. I hear that we should be shutting down the containers before the backups are being taken. Does restic support this? or maybe you have a better way of doing it.
Thanks, I appreciate the feedback. If you're using sqlite you should be fine (as it's a file). However, using mariadb mysql etc, it's best to use the in-built backup method for that solution (typically you can exec into the container and do it - possibly via Cron). Otherwise, stop the container and run restic (again can be done as a Cron).
Is the docker-compose file correct?
The restic main job has a repository env_var of "/restic", then later you bind a local folder on the host to the docker container volumes section
but in the "check" and "prune" containers there is no binding of the host folder to the container (there is no volumes section at all), and the logs of both services gives me "0 snapshots" found.
Yes, it's correct. You can validate against the official repository.
@@Jims-Garage is it optional? if I want to check and prune it "must" be done on the same repository of the restic backup, checking a different repository has no sense. am I correct?
@@RamiKattan the prune and check containers are optional. You don't need to use them. They do need to use the same folder though.
@@Jims-Garage I have checked the official repos. Its seem not using local storage as a place to backup. I think we still need to mount the folder to prune and check as well
@@altgr391 i thought the same... how can they work without the volumes mounted?
Great video! Do you have any experience with borg backup and how it compares to Restic?
Sorry, never used it. One to add to the list... I did use Restic for a few years and it was flawless.
@@Jims-Garage Thanks for the prompt reply. From some reading I did on Reddit, seems Restic is indeed a better, more efficient and reliable solution, with the only downsides being the lack of a good GUI and occasional spikes in RAM usage. But I guess none of the those apply to server backups.
@@georgebobolas6363 agreed, set it and forget it. You don't really need a gui.
If you have Mutiple instances of Portainers do you need to run rustic on all or can you just pull from all of them
You need restic per docker instance.
And do you point the back up from to where your /config are? Because it keeps failing to find anything when I do...
@@Geekydrones you need to backup to somewhere that exists
Sorry correct thats my synology nas share, but, where am I pulling from.
you sure the docker-compose file is correct? in the other 2 containers, there's a restic repository = /restic but this path is not mapped on an actual volume like in the 1st one, so what they're supposed to check?
it seems like the other 2 are using something that in kubernetes world is called "ephemeral storage"... which will be lost on k8s pod restart...
Yes, as demonstrated. The /restic is an internal folder within restic container that you map to your backups on your host.
@@Jims-Garage exactly, but it's mounted only to the 1st container in the docker compose... how do the other 2 know about that if not explicitely added as a volume to those 2?
@@squalazzo ok, looks like it has changed in more recent versions. Previously it would default to that location. Check here for the latest version github.com/djmaze/resticker/blob/master/docker-compose.example.yml
How would you backup to sftp? I have tried but can't get it to work
I've not tried but I suspect you mount an sftp to the host and then map it to the container locally.
Very nice tutorial, was able to set it up, I have my portainer_data in a docker volume which is in a different location so i was able to map both my docker folders and the docker volume separately into the volumes path following your tutorial. I did a test restore and everything is working well. I do have a question, recovery makes sense when original vm is up and running (and I AM making backups of that vm). However, I'm having trouble thinking through how to use restic if I am restoring to a fresh machine without the original docker running. Will I be able to do this directly from command line with my encryption password without the docker layer?
Thanks. Yes, restic can be run as a local application. You don't need Docker, but you will need your encryption keys.
@@Jims-Garage thanks! I'll experiment with this from a separate computer that has that path mapped to do a test restore just so I understand my options a bit better. Cool stuff!
@@victor2410 I love restic, awesome program 😎
Nice video. Do you have any experience doing this with sql db's without corruption?
I would recommend using SQL's inbuilt backup tool, and outside of that, use restic when the container is shutdown.
Using it whilst in use can result in failed backups or data loss. You could script to shutdown prior to backup.
@@Jims-Garage jesus- that was quick. Thanks - I'll try this week with your video. I appreciate your help
@@50_Pence no worries, let me know how you get on.
Those K8s configs I see in homelabs and small company job offerings is like taking that Apache military helicopter to go for an Moskito. Lunacy.
I mean the technology is great but...
@@tobi5106 depends. It demonstrates that you can understand complex systems. A bit like a degree, rarely used, but demonstrates being able to operate at a specific level.
Anyone have any idea why my snapshots are 200 bytes for a docker data over 9gb?
It keeps happily making backups, but they're clearly not the correct size.
It says "processed 208179 files, 9.270 GiB in 0:12" and then makes a snapshot thats tiny
Have the files changed much since the original backup? Remember it's differential backups, not full.
@@Jims-Garage that was the first backup
@@damo_c hmm, that doesn't look right, possibly a permissions/access issue. Can you jump onto Discord and paste the logs?
@@Jims-Garage away for the weekend with no access, will do when I get back. Thanks!
This is great and working. I got all my containers going just as you have outlined above.
There is one thing though, I like to get a status emailed to me after a backup regardless of success or failure. I'm looing for a way to do that with this docker implementation. Is there an easier way?
There's no native support but you could integrate it into a Cron job etc. there are some user scripts on the restic forums.
@@Jims-Garage
That is exactly what I'm going to do. Looks pretty straight forward to get accomplished.
The gem I got from this video though, was restic. So slick and better than just running an rsync. Thanks again!
@@rickhernandez2114 you're welcome, it's an awesome tool 🔥
Isn't part of the backup process the backing up of the docker-compose/stack configurations from Portainer also important?
Absolutely, I cover that in the next video using rClone to Google Drive
I'm obviously missing something, but it seems to me that:
If the original backup captures ALL the data available to be backed up,
and all subsequent backups only capture the incremental changes,
and all backups are eventually pruned (when they pass their 'use by date' in other words)
eventually you will only have a lot of incremental pieces of data still saved,
so, when you want to restore and Restic Restore runs back through all the available backups that exist,
it may find less than the total amount of data required,
because the original 'Full' backup was Pruned long ago.
Where am I going wrong with this logic?🙄
You specify a backup schedule, those are full backups. Between those dates you specify an increment backup schedule.
So I do weekly backups, daily incremental backup (aka snapshot).
Thank you for the instant response again. Aha, I misheard that all subsequent backups were incremental - will watch again.
Hi there, I seem to be getting an error with the backup, below is the error log.....
/etc/dockervolumebackup/conf.d was found, using configuration files from this directory.
/root/entrypoint.sh: source: line 18: can't open '/etc/dockervolumebackup/conf.d/*': No such file or directory