ฝัง
- เผยแพร่เมื่อ 30 ต.ค. 2024
- SD cards in Raspberry Pi computers wear out and die after a while because they are not made for computer usage, and many write cycles. In addition, the SD cards can become corrupted if the Pi loses power without a proper shutdown. We have several possibilities to avoid this disaster. Only one is simple and cheap. But, unfortunately, it has its caveats. So let’s get an overview and use a nearly unknown Raspbian feature to help us solve the problem.
Links:
Raspberry Pi 4: amzn.to/3kBF5nv
Barefoot method 1: / make-your-raspberry-pi...
Barefoot Method 2: www.raspberry-...
Adafruit's method: learn.adafruit...
The links above usually are affiliate links that support the channel (no additional cost for you).
Supporting Material and Blog Page: www.sensorsiot.org
Github: www.github.com...
My Patreon Page: / andreasspiess
Discord: / discord
If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission of your purchases to buy new stuff for the channel
My Amazon.com shop: www.amazon.com...
For Banggood bit.ly/2jAQEf4
For AliExpress: bit.ly/2B0yTLL
For Amazon.de: amzn.to/2r0ZCYI
For Amazon UK: amzn.to/2mxBaJf
For ebay.com: ebay.to/2DuYXBp
www.facebook.c...
/ spiessa
www.instructab...
Please do not try to email me. This communication channel is reserved for my primary job
As an Amazon Associate, I earn from qualifying purchases
#no#midroll#ads
there is another option using write-behind caching on EXT4 so it caches all the writes to ram and then using the COMMIT=xx seconds mount option it will write the changes from ram to disk every so often, the best of both worlds
Good point! And it seems to be easy. Similar to log2RAM, I think.
It doesn't ensure that writes will be cached to ram though. Just that it will cache up to "commit" seconds of writes before automatically syncing the filesystem so if your system suddenly crashes, you know that you have lost at worst this amount of writes. Applications can request their writes to be immediately recorded to the disk/SD Card. Applications who use databases (such as SQLite) or unbuffered logs will do that frequently.
@@_gawen i forgot to mention, that its a setting to enable journal_data_writeback using "tune2fs" on ext4 filesystems, as well as "nobarrier". this will run it all to ram first at the cost of data security and reliability, obviously
I have a pi3 running on the same sd card I started with, probably 6 years of running 24-7 with not a single issue. Pi4, has already burned 3 sd cards up in less than 2 years.
Can't for the life of me remember what commands I did to limit the card writes, or whatever I read at the time, but it's worked. Wondering if this is it...
I was able to reduce my sd card consumption drastically by logging to my remote homeserver. (almost) No local logs anymore. There is another advantage in remote logs: you can find suspects, even after the device in question died. Since there is a homeserver up and running 24/7, it was just a matter of some simple configuration.
Remote logging seems to be an interesting alternative (or extension) to this concept. Maybe I have to try it too...
@@mixxx2005 For anyone considering this: make sure to put security in your network config. Unless you want a random device in the wild to give an attacker access to the files on your home servers/computers.
Have any details on how you did this? I'd love to do the same.. have an NFS share or similar on a pre-existing NAS that would support logging from all of my various PIs, which could then run completely read-only.
@@SteveJones172pilot I enabled one of my Linux hosts to accept remote syslog messages, and enabled all clients to log to this remote logging facility. This is quite straight forward. How you have to do i depends on your software, of course.
This came just as I needed it!
Releasing 24 dashboards driven by raspberry pi's next week and was worried about configs being changed and other (python) code config that needs to write to the memory! Much appreciated Andreas!
Indeed this should prevent bad things from happening.
SD cards are not by definition unreliable. They come in different flavors for different use cases. If you pick the right one, an SD card can be more reliable than a big cheap SSD.
The most important criteria are the cell technology used and the Bill of materials.
Cell technology:
* Most consumer cards use TLC or QLC (3D) cell technology. A single cell can typically be rewritten approx 200 to 300 times.
* Some consumer cards advertise with 'long endurance' or 'industrial' terms. Often such cards use MLC cell technology (but this is not guaranteed!). In this case a single cell van typically be rewritten approx 2000 to 3000 times. This type is also often used for EMMC memory and modules.
* For more demanding environments (automotive, medical, extreme conditions,..), often (p)SLC cards are used. For these type of cards, a single cell can be typically rewritten up to 30000 times. That's >100 times more than a typical consumer card!. You will not find these cards in a regular (online) shop. Cards utilizing this technology often have more advanced flash controllers onboard providing additional security and features compared to consumer grade cards (e.g. power failure protection, better wear leveling...).
Bill Of Materials:
* Most consumer cards do not have a fixed Bill Of Materials (BOM). This means the internal components can change without notification and the manufacturer will choose whatever is available and the cheapest that complies more or less with the specifications. So for one batch of cards, the cells can be TLC, and a few months later, QLC becomes cheaper and the manufacturer switches to QLC components, or uses another flash controller... This is also the reason why almost no manufacturer will give the complete specifications of the used components for these cards.
* Most manufacturers also offer fixed BOM cards. These are cards where the used components are fixed and will not change without notice. This guaranties that if you buy the same card next year, it will behave exactly the same as the ones you already have.
An example of a 'cheap' (p)SLC card with fixed BOM is the SDSDQED-016G-XI model by Western Digial/Sandisk.
It is a 16GB SLC card with an extended temperature range (-40+85C) and an endurance of 480TBW. It costs about 30 euro's at Mouser. This is typically cheaper than a usb->sata/M2 adapter + decent SSD.
For comparison: a Kingston A2000 SSD of 500GB has an endurance of 360TBW. And if you only look at smaller very low cost ssd's, then the endurance is often 40 TBW)
So for projects that do not depend on very high speed disk access or require a lot of storage, such SD card is unrivaled. Both for price and reliability.
(The explanation above is a bit simplified... there are other factors playing also a role, but this comment is already long enough for now :-)
Thank you for bringing light into the dark of the SD cards! So far I have only used SD cards from online suppliers. Good to know other variants exist.
Concerning SSDs: Even my home automation server only writes MB and not GB because I store sensor data and not videos. So with my Raspis, I can live with very low TBW values (other than with my TH-cam production machine ;-) ) I also only use the smallest SSDs I can get because the volume is small (
Best comment ever! Thanks.
I see that there is also a SDSDQEC with up to 3200 TBW. Search for "brochure-removable-flash-storage.pdf" for a good summary of current offerings.
I also tend to install ZRAM on every pi I own. It removes the need for swap space on the sdcard and it allows you to fit about 50% more stuff into the physical available RAM.
I never used it because I only have 2GB and bigger Pi4. So memory is no more a big problem for me. Probably good for a Pi zero if it does not consume a lot of CPU.
@@AndreasSpiess I even use it on my rpi 8gb model. Sometimes I notice raspian swapping stuff out even with minimal applications running and plenty of main mem free.
@@misteragony For some reason I think the default "swappiness" value is a bit too high in raspbian (how likely the os is to use swap). I adjusted it to 10 from its default 60 and have only noticed minimal swap usage since then, but it is still there for emergencies when you are running close to the limit (I have also done this on my laptop with similar results).
It is also very useful to mount the filesystems with noatime,nodiratime options in order to avoid updating access time in the file and directory attributes each time the file or directory is accessed, no matter it was just read.
Another viewer also mentioned noatime. I am not aware of this feature. So I have some reading ahead ;-)
@@AndreasSpiess This is how I use these mount options, on each and every extX filesystem which resides on SD, SSD or CF on my machines.
root@raspberrypi:~# cat /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=60650d97-01 /boot vfat defaults 0 2
PARTUUID=60650d97-02 / ext4 noatime,nodiratime 0 1
@@AndreasSpiess I just noticed on the mount(8) man page that nodiatime is implied if noatime option is used. There is always something new to learn. From the mount(8) man page:
noatime
Do not update inode access times on this filesystem (e.g. for
faster access on the news spool to speed up news servers).
This works for all inode types (directories too), so it
implies nodiratime.
With the newer Raspberry Pi 4 with the higher RAM this overlay makes a lot more sense now. I guess you could use read/write Boot drive and periodically copy the overlay to the SD or to an external network drive for data that may need to be kept longer term. Thanks for the great information.
You are right, many combinations are possible if you include network disks in your setup.
YES! Finally I can cut the power to my rasppi whenever I want without the fear of destroying the SD-Card!
True. I like this, too!
I tried it with hdd. I've had to re-set it up twice. 😒
I use a different approach, and I find it much simpler and scalable: I use a NAS, so shared directories to save logs and data. So I get automatically backups (I need to set up only once for the base directory in the NAS), fast disk (but it depends also on network), cheap disks. And I'm able to split or merge services and rpi much easier.
Storing the files on a NAS is for sure a good thing for many applications (if you own one).
When a Raspberry is booted without connection to the NAS (e.g. NAS is down, or no network connection), the Raspberry runs without NAS connection forever! No retries to connect later.
Solution: autofs mounts the NAS directories when needed, and releases the connection after 5 minutes when not in use, so this also costs less memory. Thus a retry is provided after connection-loss.
When speed is not needed, you can make a NAS from a Raspberry and 2 USB disks (even USB-thumbs) as RAID-1 using mdadm.
@@AndreasSpiess It is super easy to make your own NAS, just install Linux on a machine (could be another raspberry pi) and configure a RAID 5 (preferably not on USB, which is too unreliable for it, as it can disconnect randomly from time to time, which breaks the RAID and makes it go into degraded mode) and then share certain folders via NFS, very easy. The advantage is that you don't have a proprietary NAS like Synology that is both inflexible, hard to fix when it breaks and expensive and often uses a non standard proprietarily modified Linux.
I have a NAS as well and I prefer using it as well. I just need to get 1 that's a few teraflops my problem is the $.
@@PvGeens Sounds like I will need to be careful with doing that.
IIRC, OpenWRT uses OverlayFS, but their implementation is pretty static without lots of hackery. Really neat the Pi Foundation was able to simplify it down to a toggle switch! 👍
As a Linux noob I was not aware of it until it was simplified ;-)
Thanks Andreas, I was worrying about my sd card in. my headless Raspberry Pi 4 recoding NOAA weather images 24/7 so I made images of the card if it fails, I haven't had any card fail yet but you saved the day and me worrying, thanks for this tip will implement it : )
As said: A backup is always good! But your application probably is a good use case to reduce the chance for failure.
Add commit=300 as a mount option in /etc/fstab. EXT4 default is to commit all writes every 5 seconds. If you mount with commit=300 it will cache writes in memory then write them to disk once every 5 minutes. You can reduce your SD wear by 60x by adding that line.
Good point. Thank you. I will try it
Thanks Andreas for the tip. What I do is to use a high endurance SD card, specially designed for high number of writing cycles ( they are used in security cameras). Is a very lazy solution though.
I hope this works (because I ordered a few of them after comments of other viewers...)
@@AndreasSpiess I personally have been lucky using Samsung pro endurance in Rpi for a while after testing them 1+ year on dashcam. Only 1 out of 10 have made one file corruption (as far I have noticed). Before that the Sd cards was not saving all the files or saving them empty. Just a note, that bigger sizes have bigger large ratios before failure.
Other option is to add some USB drive, which comes in micro size, so fairly extends out of devices itself
@@AndreasSpiess Nothing is failure-free. I used to work with MDVR (mobile video recorder) and moved from a Sandisk Ultra type SD to a SanDisk high endurance. From a batch of 20 units, we used to have one SD dead every 2 weeks; after the change the first failure happened 6 months after. Using Transcend High Endurance SDs in three Raspi working as LoRa Gateways for one year, no problem so far.
@@jdsan6009 Thanks for the info about Transcend SD card. I will try that one next time. Currently using SanDisk 128GB High Endurance in my 4K dashcam. Working fine so far. My old 64 gig SanDisk SD card worked fine for 2 years before I replaced it. I always get the biggest SD card that the device can support to extend the life out of it.
@Andreas Spiess. You will like to know Tesla uses industrial type SD Cards to save logs. Here is the reference th-cam.com/video/ODdIRr5RzI8/w-d-xo.html
Thank you! Extremely useful, extremely inexpensive, reversable and can be done in seconds. Love it! The overlay wins for me.
Glad it was helpful!
Very useful video, Andreas👍- this setup is indeed ideal for 24/7 applications that do "something" and send the results to a remote server or such. Or another idea: Run Minikube on the Pi4 and push containers to it from remote. That way the Pi would become application-agnostic?
So far I never used Kubernetes. Maybe a project for winter ;-)
Instead of a SSD, you can also use a network filesystem and redirect things there through symbolic links. Handy if you have a NAS on the same network. Beware that SAMBA/CIFS does not work well for that and some services will randomly fail if their data are put on a SAMBA mount. NFS on the other hand works fine with everything.
You can also use the iotop command for instance 'iotop -o -b -a' to log the cumulative writes of the different processes so you can see who is writing and how much.
Thank you for your input. Remote logging indeed would be a possibility. Maybe even with these switches (if you want to keep your logs)
After 2 SD cards busted, I put an SSD for my Home Assistant RPi4 and it is running since then very reliable, hope I don't jinx it. But very good tip to make the SD card last longer.
I also use an SSD on my server. Also for speed...
I use RAMFS and mount it over the top of the log directories. Then, when a program writes to the log files in the log directory they are intercepted by RAMFS instead. When the max size is reached they are discarded. They are never written to the SD card. My SD card is in its 4th year of operation (as a LoRa gateway) with no problems so far.
I never used RAMFS, only log2ram which seems to have a similar strategy. This is another valid way to solve this problem.
I have a bunch of raspi 3's running on the field and they have been working nice for the past 4 years with this tmpfs trick. A couple of sd-cards have became corrupt during this time, but I suspect its due to power outages, since after dd'ing they seem to work ok again.
I would love to see any scripts or configs you have to accomplish this.. Sounds cool.. Also sorry you're "not" mark Knopfler.. it would be cool to be Mark Knopfler! :-)
Can you write the command for setting this up?
I have found this:
sudo nano /etc/fstab
And added
tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0755,size=20m 0 0
Is this correct or are there other directories?
Thx.
Thanks Andreas. I watch your videos and like them every time but this is the first time I comment. I did not know about this feature and it is very useful for me because my children use Pi 400 for school. They love to mess with the OS and I have to reimage their systems often. Well, not any more. Thanks again
Perfect use case, I think...
In addition to these tips I would strongly recommend getting an SD card that's more suitable for this kind of usage. I've personally had good results with the WD purple SD cards, I've got 3 pi's that have been running 24/7 for about 3 years now on an unmodified raspbian installation with full logging enabled (and one of them is constantly writing due to a pihole installation). So far I haven't had any issues. But... I would recommend partitioning only a small part of the SD card (i.e. 25% or so) so the wear leveling algorithms can do their thing and make sure your SD card won't run out of working blocks.
Thank you for sharing your experience. Because other viewers also recommended these WD purple cards a few are in the mail by now. I hope they implemented wear levelling. The descriptions I found were not clear. Just that they monitor "bad cells".
@@AndreasSpiess I've read on reddit that they are supposed to have wear leveling according to WD suport. Anecdotally however, I've had many SD cards fail before I switched. Even high quality ones
I do not believe having a small partition would make any difference to the wear leveling since it uses all the cells to write data regardless what file system you use. The idea is get the biggest card the device can support to extend the life out of it.
@@Darkk6969 You're right, assuming you have TRIM enabled it won't make any difference in that aspect. But if you only partition a small part of the disk you leave the SD card with enough space to keep wear-leveling without any worries and you can't accidently fill up the disk with logs or other files. Given how cheap memory cards are and how much space you actually need, this has been a good tradeoff for me.
Sandisk do a range of rugged IoT targeted SD cards using more robust flash. Makes sense.
I've usually mitigate writes to the sdcard by making sure the noatime and nodiratime option is on the sdcard in the fstab and mounting a small flash drive to the /var directory, and using a larger flashdrive for the home directory (you could put both partitions on one flashdrive). This should eliminate all writes to the sdcard except when installing and updating software. Not on PIs, but on PCs, I frequently use a virtual disk and keep my work on flashdrives, and have scripts with clickable icons to setup, load, and when my session is done, save and destroy the virtual disk.
Thank you for sharing how you do it. Virtual environments are also a good thing if you want to protect the original.
Sometimes, at good points, I hit the save work icon, and it saves only the files that have changed, using rsync, and I continue on.
Great tips, nice video!
To be a bit pedantic, overlayFS itself doesn't mean writes are put in RAM. Rather, overlayFS is one piece of the puzzle - it allows redirecting writes to somewhere else, and when combined with tmpfs (RAM-based filesystem) it achieves the reboot-to-discard-temporary-changes feature.
You are right, of course!
I've been using raspberry pi for quite some time It didn't realize that was there. I really enjoyed this video. I really enjoyed listening to you on the ham radio workbench podcast as well. Thanks
There were a lot of discussions in the past about this problem.
It was also a pleasure to be guest in the workbench podcast.
I saw this option before in dietPI, I think it's a cool feature, glad the ported it on raspbian too
Another viewer asked if it is available on dietPi too. So here we have the answer...
@@AndreasSpiess Not sure if it is exactly the same. I know that in DietPI logs are not saved directly to SD and when this functionality was introduced, it was not yet in Raspbian. It is a really interesting concept to preserve SD life expectancy
Clever ways to save on writes :) Phones uses F2FS file system to save on writes. F2FS supports ZSTD compression. Image with bear minimum of applications would write less compared to stock Rasbian.
There are MLC SD cards that last ~10x at same capacity compared to most micro SD cards in the wild. Jet MLC cards cost ~4x (on SSDs the price difference is more reasonable ~2x). Higher capacity cards will last longer (grows lineary with the size). As for disconnecting the drive unsafely - I had luck with usb sata bridge and microSD readers, they usually go before the storage.
Thank you for the additional info!
Thank you Adreas for the easy to understand ex[;amation of the Overlay File System. It's just what I needed for my weather station which uses an ePaper display controled by a Pi Zero W.
That seems to be a perfect fit!
Is it a perfect fit, though? You mentioned in the video that 2G of RAM is recommended, and the Pi Zero W only has 512M.
I don't know how the Overlay File System decides how much RAM to use: is it dynamically allocating memory based on the amount of data written, possibly running out of memory over time? Unfortunately, Googling doesn't return any tips on reducing its memory usage, and I couldn't find confirmation that anyone is using it with a Pi Zero W successfully.
I guess the only way to be sure is to try it.
Thanks for pointing out the option, though: I didn't know it existed.
This is a discussion about two topics:
1. How flash works.
2. BC/DR.
Lets solve 'two' first. If your data/system is critical make sure you take/keep a remote copy and keep it off site. End of story. 'Local resiliency' doesn't help you if your data center - or in this case raspberry pi - goes up in smoke.
For 'one' let's talk about flash's 'peculiarities.' Flash devices - and by the way both memory cards and SSDs work the same way - have the following characteristics:
1. Fixed block size for writes. Unlike spinning disks flash can has a fixed write size. i.e. If a device writes at 32kb and your write is 4kb. The device writes 4kb of data and 28kb of zeros. If your write is 128kb the same device writes 4 × 32kb writes.
2. SLC, MLC, TLC. Denotes the number of bits in a cell. 1, 2 or 3 respectively. Additionally, it indicates 'burn rate' how quickly cells those cells die. The greater the number of bits in a cell the faster they wear out. Most consumer grade SSDs and flash cards are MLC.
3. Overprovisioning. Every consumer flash device SSDs and flash cards are overprovisioned to take wear and tear into consideration. SSDs have wayyyyyy more than Flash cards as they have to deal with the writes an OS throws at them. So don't buy an SSD buy a bigger memory card. It will last a lot longer, be easier to deal with and work out cheaper.
4. Fade. As magnetic media is hit by 'bit rot' flash gets hit by 'fade.' Over time the bits flip or the cells fail and the files become unreadable. Magnetic interference and heat accelerate this process so keep your pi cool.
4. Device firmware. Not all flash devices are created 'equal' and - no disrespect to the engineers who do this - the firmware on the a given generation of devices often gets written in a hurry. So the lifespans of two different cards with the same size advertised size and identical workloads may differ drastically in terms of lifespan.
5. Reads are free, writes are poison. High levels of random writes are absolute cancer for flash. Everything slows down if your device is bangoed with high write traffic. So if you are logging tons of data? My advice send it to a NAS backed by spinning disk DON'T store it locally on your memory card it will kill your flash device quickly.
So in summary:
1. Keep a back up.
2. Buy a bigger memory card than you think you need.
3. Keep your PI cool.
4. For all random write traffic send it to spinning disk.
5. Check the product reviews for angry consumers complaining about 'failures within weeks of purchase.'
How do I know this? I was employee 186 for the best enterprise flash array vendor on the market. As an aside - but related to this - 'middle out' from the 'Silicon Valley' series is based on a real algorithm that optimised wear and tear on MLC using meta data references to avoid writes if possible.
Thank you for your additional information. Everything is true. Do you have information about the wear-levelling of SD cards? I read a lot that they do not have such an algorithm and manufacturers explicitly forbid to use them in file system scenarios (and refuse warranty as one viewer wrote).
Just a few additional thoughts about the use cases of the video:
I talk about Raspberry Pi computers which are not often used in data centers nor do they have to deal with lots of data, because its performance is weak compared to servers. On the other hand its data often is not very valuable for the long run or does not change a lot. Many of our Pis just do some data transfers (sensors, gateways). or are used to drive displays, etc. and are sometimes operated in harsh situations.
Fortunately, we will hardly hit any of the specified transfer volumes of SSDs. However, we all experienced crashes of SD cards. Sometimes also because the power was switched off during a bad time which corrupted the FS.
great features
Thanks for sharing your experience with all of us 👍😀
Thanks for watching!
Indeed interesting. Guess I should make work of adapting my RPi asap.
If you have one which does not have to keep data it is a good thing.
thats a neat trick, plus is more secure, becasue if you loose your rapsberry pi whilst transporting it, you wont have to worry about still being logged into google etc and saved passwords. just enable it on a new install after you have setup wifi and customized the dektop
Good point!
Great hint, Andreas! Thank you!
You are welcome!
So, why isn't there an option to write the ram files back to sd on close? Thus dramatically reducing the writes
My guess is it’s too hard to guarantee that. If people expect their data to be saved when the pi reboots then they will be angry if 100’s of megs get erased when it looses power or crashes. If the system expects files to be written on shutdown then the file system would be corrupted on ever single unexpected reboot. It’s much safer to just assume all data is temporary in the RAM disk.
@@CodeMonkeX Yes I see if the rewrites were automatic, there could be a problem. My thoughts were for a Manual/programmatic option - a simple 'do you want to save' or cmdline save
I am sure a lot of things can be done. But, as a Linux noob I am happy to have a simple way. All my use cases can be appointed to one or the other scenario: No longterm storage needed --> SD card or longterm storage needed --> SSD
Interesting question. That would be a nice option. I'm all about those SSD drives because of the speed. I only use SD cards for the portable builds like my hotspots.
In some way a similar functionality can be achieved with "apt install log2ram". If I'm not mistaken "log2ram" is installed by default in armbian-os, on raspberry-os we can install it ourselves.
Wow..I didn't know about this feature..?...thanks!
You are welcome!
This was a great video.. I had been wondering why there wasn't an easy way to do this.. Next step would be to map a network drive and use it for logging and other data you dont want to lose, and it would be the best of both worlds! Thanks!
Agreed. Other viewers also suggested that method. So far I never used it.
Great video, as always. Also, a great tip I will try both ways. Overlay and with a small SSD drive.
I use both ways, too. For different use cases.
thanks to the guy with the Swiss accent, another great tip and information
Happy to help!
Thank you for these useful tips. Much easier that I had imagined
Great comments on this video too. :)
I agree, I have a lot of knowledgeable viewers and commenters!
Thank you very much! It was useful and interesting 🙂
Glad to hear that!
Nice Video, learned something new. Now it would be great if specific log operations could be made to go to the SD card and the majority just to ram...
You an du that by using RAMFS or log2RAM to only store logging in RAM. But it needs a little more Linux knowhow. Or you mount a network drive for data as other viewers suggested.
Excellent information, I already put UPS boards on all my Pi’s but this will is a great trick.
UPS is also a good think, But not cheap ;-)
@@AndreasSpiess No they are not, but I wont run a PI without one, I use the PiSuger2 on all of mine, they come with a remote web interface for configuration and status, totally worth it.
Looks like a good solution!
I still prefer to use an ssd. You can buy a 128 GB M.2 disk for 20-30 € and a small enclosure for a similar amount. On the other hand I don’t have Scottish ancestry….
For me it is mainly about the form factor. As you write, the prices are ok these days.
Thanks Andreas,
I believe a short lifecycle is typical for low-cost SD.
I use several SD cards in raspberries for 3 years already, incl my Majordomo home automation (decades of mqtt sensors, etc.) and had issues while using sandisk-like SDs).
After changing to less cheap I've forgotten about it.
Thank you for sharing your experience. It seems you were lucky. At least I found mentioning Sandisk and other branded SD cards in the forums complaining about failures. Maybe the operating systems also got more robust against corruption of SD cards.
Please let us know which are the BAD and the GOOD SD cards you used.
@@paulcohen1555 Sandisk were the worst, now Samsung, no issues yet.
I can't remember particular models, sorry.
@@paulcohen1555 I don't think it would be a healthy data because of the chip shortage. Even good SSD brands like samsung started to use less expensive less durable components. It's safe to assume that same tactics are deployed in the SDcard production.
Day of installation:
# ls -ld /lost+found
drwx------ 2 root root 16384 Nov 29 2017 /lost+found
excellent... i wondered what that setting was for but never looked into it!...thanks Andreas
You are welcome!
Useful AND interesting! I was unaware of the read only setting.
Glad you like it!
@@AndreasSpiess I realized I was unknowingly using the read only feature on my pistar hotspot. 🤣🤣 I thought it was part of the project and didn't realize that was a feature of the OS.
AFAIK Pistar does not use Raspbian and therefore probably implemented a different concept. But it seems to work ;-)
I have the same SD-card running since my Raspi 3B+ was new. 2018? 19? Never got corrupted (headlesss lite raspian doing DNS and Cups). I still did that, thanks.
So you were lucky and got a good SD card, I think.
Very very good video and here I was wondering why my card would die after a few months of usage. Thanks a lot for this video. Now to the terminal window!
I hope this will solve your problem. Other viewers commented also that they were successful using "endurance SD cards"
Write files to USB Flash & update as needed if data logging with minimal data loss. ...There exists a "HAT" that goes under the Pi and is screwed on, supports an M.2 SSD via a small USB 3.0 adapter. The bonus: you get SSD speeds with low power draw.
Do you know if USB sticks use wear-levelling? Otherwise we probably have the same issues.
Good that the "hat" goes below the Pi4. I still see too many "top hats" which create heat issues.
Another astonishingly simple option is to use a bigger SD cad. Let's say that your card is 80% full, then there are 20% storage/repair capacity left. If you chose an SD card with twice as much capacity, you'll get 120% storage/repair capacity left relative to your original one. This leads to a 6 times as much lifetime of the SD card as before.
I agree with Jan. But bigger cards usually also are faster (at least in the past).
@@AndreasSpiess
Wear-Leveling If Not done by Hardware is a Problem a OS should handle... But Looks Like they dont.
Maybe writte a Script to do monthly a copy of the Files that have IO writtes and rename the old File to dead Disk space +timestamp
If i build a System i Always keep more then 50% Disk free
And If the Disk get to 20% free i search for Upgrade Options.
Have a nice Day
And greatz from Germany
opo
Add autofs against an NFS box with your homedir, we did diskless workstations that way for years back in the 90’s.
You are right. Remote file systems can be a good solution instead of an SSD if you need to keep the changing data.
Not Pi related but Linux related. In the early days of SATA SSDs I had one at 16GB from Imation. I had the most 'write intensive' volumes mounted on an LSI RAID volume (e.g. /tmp /var/ and SWAP). It lasted over 6 years in 24/7 use before it was replaced by a new computer. I did pull the drive before sending the computer to freegeek. Those are the 'high write cycle' mounts for a system drive. Data drives are not addressed.
Thank you for sharing your experience. So far I also had no problems with SSDs, also not on the PC. Even if I do some video editing. For the Pi I do not expect any issues because the data volumes are very small (mostly sensor data and logging).
Hey there,
i had the same problem.
I solved it by just using a MicroSD to SD Adapter in a SD to MicroSD. ^^
The Different is that y can switch the writeprotection on at the MicroSD to SD Adapter.
Old SDs had a small switch at one side. The smaler MicroSD did not have one.
Sometimes its simple like that.
Regards Richard
Good idea! However, the OS has to support it...
Great. I just needed this.
Very good!
Very useful and simple, many thanks.
Glad it was helpful!
Thank you this was a simple but effective trick!
You are welcome!
after the loss of 2 SDcards, ive switched to using a USB Stick in my Pi, worked sofar without issues.
Thank you for sharing your experience.
great video. happy christmas
Thank you! Happy Holidays, too.
Great find!
I'm going to have to check to see if it works on my OctoPi server 😃
I never tried so far. But it should not need to keep a lot of data after reboot.
Perfect; I'll set the SD to RO, and write data to a NAS.
That should work and protect your installation
The overlay system looks like a cool feature but I almost never use raspbian. Tbh, I've never had a problem with tweaking fstab but this is handy to know to recommend to others as it's simpler to explain.
If you use different distros I assume you are an experienced Linux user and do not need such a simple solution. For a Linux noob like me it is different ;-)
@@AndreasSpiess not sure I’d call myself experienced but I am comfortable with *nix. Like I said, didn’t know about this feature and I will recommend it to those that don’t really care about what’s under the bonnet. Thanks for sharing.
Useful and interesting as usual.
Glad you think so!
Nice!, this will also improve the performance. I am thinking about to create a minimum raspbian version to improbe the boot speed and memory usage on a Raspberry Pi Zero.This will help me for sure.
And it is needed for the Pi zero. I do not like it because it is very slow. Maybe with your changes it is ok...
I feel you should look at using a network drive so you do not need to use so much RAM. One simple way to make a network drive is to use an old desktop as a Linux box and have it share the hard drive to the raspberry pi.
You are right. Other viewers also mentioned remote logging as an opportunity. I assume it would need at least some Linux knowledge.
Years ago, my daughter had been using my core 2 duo laptop running ubuntu 12LTS booting(with pre-loaded apps) from the dvd drive loading it on ram for several years till the internal cooling fan conked out. You could still save on the hard drive but everything else resets after shutting down.
Others also suggested that the overlay FS was used for USB stick booting which was probably similar to your concept. So nothing new, but still useful and now integrated into Raspbian (good for noobs like me)
Very educational content. Just wanted to point out that I'm personally using High Endurance SD by SandDisk which meant for CCTV application (run 24/7 + consistent read & write application). So far works well (more over 8 month with multiple time of trips by thunder storm - in turn will be a sudden power loss) for my simple LAMP project without overlay file which is running 24/7 when ever possible.
Maybe can give those a shoot for extensive test?
Other viewers also commented about those SD cards with different results. I do not plan such a test because I do no more use SD cards for heavy usage. I changed to SSDs, also because they are faster.
True, I've lost a few SD cards due to unexpected power off
Thank you for sharing your experience!
Another thing to do is to enable periodic fstrim operations to discard deleted blocks. This applies to all flash based storage: SSDs as well as SD cards.
Would this not increase the wear?
@@AndreasSpiess It spreads the wear more evenly as it allows the flash controller to know which blocks are free and can be reused instead of having to reuse (erase, reprogram) the same flash block when it runs out of spare unused blocks. Wear levelling.
Are you sure that this is true for SD cards. I heard that their main problem is that they have no wear levelling implemented.
@@AndreasSpiess It really depends on how the card is made. The better ones do implement the TRIM operation. Alternately have the operating system use a more flash friendly filesystem for logs, eg. f2fs
Brilliant way to make a Live OS
True!
I'm testing the overlay file system on 11 Raspberry Pi, running camera applications.
Users were instructed to just unplug the power without powering off the system, so I could see if this was a viable solution.
So far no issues with sd card corruption.
2 of the Raspberry Pi were running without interruption all day (9-18) for 3 month and they still work perfectly fine.
I don't know if i'm lucky or if it works well, so take it with a grain of salt.
Thank you for sharing your test data. For me it looks like it works as designed!
thanks! this helps me a lot!
You are welcome!
• 2:15 - Another disadvantage of SSDs is that as flash-media, they wear out like a memory-card does, so it's no better.
• 3:19 - The write-protect switch on SD cards is just a piece of plastic, the write-protect function is only in the card-reader/writer, and not all card-readers (*cough*Chinese*cough*) honor the switch, so it doesn't guarantee to protect a card from being written (this is the same as with floppy disks).
• 4:22 - This makes the system even more vulnerable to data-loss than ever. 🤔 You assume reboots are intentional.
1. SSDs use wear leveling which was not the case for SD cards. This is why they were not specified for computer usage.
2. You are right
3. I do not agree because I only use it for relatively stable applications. So reboots for configuration changes are very rare.
It was interesting. I do have an original Pi 1 that has the operating system on an attached powered Hard Drive. Just uses SD card to boot.
I only started with the Pi3.... I tried once to attach a HDD but had to solve the power issue...
@@AndreasSpiess yes, definitely works. Actually, I use nfs mode to develop o/s image updates, then use squashfs to compress it and send it over the network and into ram from a server at boot time. Runs as normal, no sd card required. Needs 2gb ram minimum. I have 5 pi4s on my network doing the home automation (node red and home assistant) - I can't have part of my house dropping out because of a worn out sd card!
Yes, good and useful video.
However, I disagree slightly on the SSD; they do wear out.
Harddisks are less likely to wear out (choose CMR, *not* SMR; choose WD-RED, not WD-BLUE).
But ... There is another option which isn't extremely complicated: Use NFS for storing your log-files. You don't have to set up netboot (which I just attempted yesterday and failed). You can make your fstab auto-mount your NFS shares and then have your 53 Raspberry Pi's (including Raspi Zero) log to one server.
You'll then have the convenience of looking on one computer rather than 53. ;)
-That is if you have your RasPi's connected to your network (they don't have to have internet access, though).
Remote logging is a good alternative if you really are interested in logs. I rarely use them (for finding errors) on such simple installations I use SD cards.
SSDs usually specify a max amount of data stored before they die. The data on my Raspberries are very, very small compared to these numbers because they consist of sensor readings and alike, not videos (different to my TH-cam production machine)
@@AndreasSpiess What I'm concerned about is writing. Writing kills SD-cards, MicroSD-cards, USB-sticks and SSD drives - not the amount of data stored.
But as you're indicating; if writing a lot of data, the card / SSD will die quickly compared to if you write only a little data.
But repeatedly writing small files and deleting them (eg. compiling) is bad for all such devices. My brother's two SSD drives just died recently (mounted in a Mac Pro). Those drives didn't last more than a few years. Even WD-GREEN harddisks last longer (they're no longer available; WD-RED is much better anyway).
NFS can also be good for data-logging and that 'family' (including analyzing) - this also opens up doors for off-loading analyzing to the server instead of the device. :)
Now you mention video; I know you can actually attach cameras to the RasPi, which means you could make smart surveillance via NFS shares too. ;)
thanks for the knowledge
You are welcome!
The overlay technique is actually pretty old. They used it for bootable Linux CD-ROMs back in the day.
Good to know. And now they made it simple to use it.
Disable file date updates. This will beat the heck out of sd cards. It will also speed up access times.move logs to ramdisk and dump and rotate logs to external storage.
Agreed. I assume that the instructions I mentioned do such things.
Thanks for the info
You are welcome!
Nice info! Thanks!
Glad it was helpful!
This should be a feature of laptop and desktop computers as well. All should come with two separate disk drives, one for the OS and application programmes, set to read only, and one for user data, set to read/write. This would solve so many problems.
It would bring meaning back to the phrase, "switch it off and on again" to fix it. It would make installing viruses far more difficult.
The procedure for updating the system components could disengage the user drive and allow the system drive to become read/write. The procedure could be made to automatically generate a system backup, before making any system changes. By isolating the user drive, any trojan horse would not be able to lie in wait on the user disk, for an opportunity to corrupt the system, and once returned to normal operation the read only mode of the system disk would stop any new virus corrupting the backup files. I would like to see the system install feature and its associate disk write locking to be a user controlled hardware switch. A physical interlock not dependent on some software sub programme, which might be hacked.
The current practice of allowing writes to the system disk is effectively producing self modifying code, which is really bad practice if you want a stable system.
Chromebooks do this with the guest account.
@@macemoneta one of the most advertised features of chrome books is the level of virus protection it provides.
nice and useful trick, thanks for the info
Always welcome
The medium article instructions, while time consuming are better when it comes to usability
With overlay fs, you need to reboot every time you need to make any changes and then reboot again to return
Yes, you need to reboot twice for changes.
thank you for the job
You're welcome!
What would be the difference when leaving the boot partition on read/write while having overlay enabled?
I never thought about that :-(
I have few application that move files from 1 folder to another in my Pi. I assume that the overlay file system will revert everything back on restart?
I getting a SSD for my 3B+ soon, which is a better solution for my situation (i think)
1. Yes, it will revert to exactly the initial state
2. An SSD is a good thing. However it will not be very fast on the USB2 of a Pi3B+
You can enable iscsi drive from network for writing.
Unfortunately, I do not understand :-(
@@AndreasSpiess As far as I understand, Raspberry Pi 4B supports iSCSI in the boot-firmware, so it might be able to just boot off of an iSCSI disk.
Alternatively, one could OverlayFS mount an iSCSI volume on top of the SD-card.
(and if you want to make things complex, which is off-topic, then you could include docker and mount network-shares only inside containers).
Are SD cards that fail in this mode of operation covered by the warranty?
In short no. I made a claim for SanDisk memory stick. They asked for the use case. I said it had been used to transfer files and to write a large Linux ISO file. I think they interpreted the ISO file for a complete working OS system as they replied:
"Please note that the pen drive you are using has not been manufactured for use in devices or applications that perform constant read and write operations like car cameras, security cameras, bootable drives etc."
They did replace it as a one-time courtesy. I normally buy SanDisk but I worry about buying a fake. I only buy those in a full retail blister package but you never known what happens to the ones that fail the quality test in the factories.
@@Alan_UK I still believe in SanDisk for quality but as they said the product have limitations (and IMITATIONS).
They probably can check the operation history kept somewhere in the drive.
Anyway, I learned something from your case.
Thank You
You are welcome!
Are any of these options available on the old Raspberry Pi model B boards (not Pi 3B but the original one)? Would it for instance be possible to boot from SD card and then re-mount to an SSD on USB? I'm not a unix expert, so commenters forgive me if my question is very stupid.
The feature presented in this video is a software feature and should be available also for older models. But I do not own such boards anymore. So I do not know if there are restrictions.
@@AndreasSpiess thank you kindly for your reply, Andreas.
Does this only work with Raspberry Pi 4? I don't see the Performance options on my 3B+ which is running the latest OS.
I now tried it with a Pi3 (desktop version) and it was available as with the Pi4.
@@AndreasSpiess It seems to be missing from the lite version. I’ll try again with the full desktop version. That opens up a lot of possibilities for automotive use.
I had to update the firmware and then was able to get the the latest OS installed. I have it working the Pi I used for Fruitbox (jukebox software) and it works great! This should greatly prolong the life of the card. Thank you!
Does this also help in the event of a sudden power loss? I imagine as long as there's no writes occuring at the time of loss it could be okay?
Yes. Because nothing is written directly to the disk (or SD card). So, your Pi always starts with a "fresh" image
If you're going to go to all this effort just build a completely in-ram system with buildroot. A very basic system with python3 etc is going to be about 16-32MB as an initramfs. You can stick that on an SD card, have it load into memory at boot and never do any writes to the SD card.
That is an interesting idea. Would it also run all the examples I mentioned or only Python programs?
@@AndreasSpiess
You can run anything you can on a normal distribution. You just have to select the pieces you need for your build. Packaging your own bits is easy too. IIRC Tesla use buildroot in their cars.
An example: I have a beaglebone black based LoRaWAN gateway in my roof. It's impossible to go an fix that thing without getting out ladders etc. So it has just u-boot in it's eMMC. It downloads a buildroot based image over TFTP on power on and runs from that in memory.
If there is a problem and I need to update the OS I can just build a new image, load it to the tftp server and then power cycle the board via it's PoE port.
Sounds like what we call "OTA" on our ESP32s ;-) I did not know this exists, but it is very handy for situations like yours!
@@donpalmera Is that also know as PXE boot?
What will happen if we set Overlay: Enabled, but Boot Partition: Read-write? Any use case?
Hmm, I might be wrong, but I don't think it'll write to the partition.
You might be able to doubly-mount a partition on the overlay disk if you wish to write to it.
Eg. On your overlayfs...
mkdir /mnt/seethru; mount /dev/sda2 /mnt/seethru
... don't know if it'll work of if it needs some fiddling.
I never tried it (because I did not have a use case).
Disabling the extensive logging and writing to disk is also useful to extend the life of all types of disk drive, especially handy for extending the life of limited-write SSD's that also degrade rapidly like the sd cards. For any High Volume Data logging try using Mechanical HDD's for a much longer service life.
Good to know that the old HDDs still have some advantages...
1. download Raspberry Pi Imager
2. Flash sd as usb boot
3. Flash image to usb
4. Insert sd+usb and boot
5. green screen, unplugged power and sd card
6. Plug the power and go
Unfortunately, I do not understand:-(
How does Overlay File System in Read-Write mode works? As in RO mode but writes on SD Card before shutdown or restart? Or is just a temporary solution when updating and just passes the data through RAM to SD Card? Thank you
As shown in the video all changed data is lost when you reboot the Pi.
Thank you. Also for all the videos on your channel. I learned a lot of useful stuff.
Where is the problem? Is it because of excessive opening and closing of files or short write bursts? I was planning to use SD cards for data logging temperature once a second and leaving the file open all the time. Any help here? :)
It is because the same SD card region is written on and on (probably especially with logs, but I do not know). Maybe your file will extend to new cells on the SD card and will not cause problems if you do not reboot.
WD line Purple of microsd cards for video security has a write cycle of up to 16tbw for a capacity of 32gb.
Thanks for the tip. I ordered a few of them ;-)
I actually use a very small USB stick. it’s easier, yes I loose one of the USB ports, but it’s faster than an SD, very compact (no cable), cheap and everything works as usual
Good point. But so far nobody was able to tell me if USB sticks implemented Wear levelling. Otherwise they will encounter the same issues as SD cards.
You can even get rid of any local storage by booting them over the network and have the rootfs on a nfs share
You are right. I never tried it so far.
I would like this, but is there some way to make certain write process still work, like apt-get upgrade and for my moonraker to keep Klipper and mainsail up to date? Perhaps this is the case, but I am afraid to turn it on and have issues.
As shown in the video, all changes are lost during reboot. But you can switch it it off, do your updates, and switch it on again. I would try it with a copy of your productive SD card.
Does the SSD have the same corruption issue as the SD cards resulting from power outages? I have corrupted Octoprint a few times from someone (me) cutting the power before shutting down. I am definitely going to try the Raspian Function. Thanks!
Corruption often comes from the file system (the computer cannot finish a transaction) and can also happen to the SSD.