Expanding FreeNAS Beyond a Single Chassis - Part II
ฝัง
- เผยแพร่เมื่อ 2 ส.ค. 2024
- This is the second of a 2 part video series covering a project to expand my FreeNAS system beyond a single chassis. In the previous video, we covered my HBAs, reconfiguring PCIe slots, and syncing PSU states between multiple chassis. In this video, we cover independent fan control for the main and secondary chassis along with some new monitoring software I made.
Part I can be found: • Expanding FreeNAS Beyo...
Here is a detailed article covering the system from top to bottom: jro.io/nas/
Here is the github repo with the fan control scripts: github.com/edgarsuit/FreeNAS-...
These are the previous videos I did on my FreeNAS system:
• Making a quiet Supermi...
• Supermicro SC846 Cooli...
Here's the original fan control script I used:
www.ixsystems.com/community/t... - วิทยาศาสตร์และเทคโนโลยี
I have been reading the article on your website. Crazy amount of detail, and love the scientific approach. Please keep up the videos!
Nice video and audio work on this. I hope you'll document more projects about your setup.
i really wish there were more of these videos on youtube. please make more videos about anything and everything in your rack!!! i really like that you go into just enough detail to make things interesting for people such as myself, not too hard to understand for the laymen, and yet not dumbed down as to be repetitive. please please do more videos. i would love to see an overview of your workstation in the rack as well as your hypervisor (i think you said that it was proxmox). I'm planning to buy a server rack for all pc and home theater equipment. it will be expensive but so worth it.
Wonderful videos of your FreeNAS expanding adventure. Really nice you take the time to make them and they are very informative and useful. Have been using your web-site and your videos to build my own FreeNAS solution. Looking forward to see more from you and your development of your storage solution.
Great stuff man! Hope you can find the time to teach us more!
I really enjoyed following the project. The explanations, the camera work, and commentary were top notch and your implementation seems pretty creative. Thanks for sharing!
Awesome! Just came across your videos, first your cooling mod one. I have an old supermicro running xcp-ng with a freenas vm. Im excited to try some of these tips! Keep it up man.
Nice and informative videos, enjoyed Part 1 and 2. Loved the implementation of the rpi and display combo. Nicely done. I wish your external fan mod was actually something you could purchase for front intake server chassis and home use.
Your videos are SPECTACULAR! The details your provide and the clarity with which you relay information is really impressive. Thanks and keep it up.
Impressive, you do all things that i wish i could do my servers. Thanks for sharing it with us.
an amazing project! I can tell you have done a ton of research and realized you need to wear many hats to undertake a project that is so far outside the norm. I have gone down that rabbit hole a few times and started to chuckle as you mentioned the arduinos. I ended up there also, eagery looking at that those cheap little devices can do. In the end, the communication problems over 5V I2C to 3.3V rPi was too much of a pain and I also abandoned the arduinos and went strictly rPi. It was a learning curve for sure but in the end I think we can both agree we are batter techies as a result. I look forward to future videos!
Hi Jason, great video. I actually came up myself with a solution very similar to the one shown in the video, but I hadn't figured out how to control the fans. Thank you very much. Keep up the great work man
Thank you for putting the scripts up online. Normally when I see videos like this the person just says that they edited or used a certain script then doesn't give any more details.
very nice and a lot of information for us watchers :) keep up the good work!
Hy Jason, thank you for this interesting videos, very well made and interesting even for novices! Keep it up!
This is dope, I have no use for any of this but it’s so well done it’s hard not to watch. Sub’d!
Great work! I was expecting you to use an Aquaero instead of the Pi route but seems to have worked out well.
Awesome!
Great video and great golden doggo
The most essential element of any homelab is a good doggo
Very informative, and quite well made :) I'll definitely be coming back for more . I have a DS4243 that I bought from ebay that isn't even unboxed; I have no idea what to do with it :)
great videos! nice see the yt algorithm make nice score for the server and automation lovers.
Great Job
Congratulations on overcoming all the technical problems in amazing build. A very impressive budget too - you have certaintly squeezed every dollar of value here. Like many others.... I'm curious what is it for? Is the system the purpose in itself?
Thanks for sharing m8.
Agreed, you have achieved what other can not even dream of.
I am a bit curious on what kind of data you have on a such large storage ? Just curious !!!!
What could be cost of the entire system ?
I'm curious to know how much current you're pulling with the entire rack setup and how you have the power configured. Trying to rap my head around my own power needs and seeing you have a fully built system that has been working successfully for years I'm hoping I could pick your brain or perhaps inspire a video going over what hurdles you ran into (or perhaps jumped right over!) Thanks for sharing these videos and taking the time to do them so well. I really appreciate it. Take care!
As I watched the video further I just thought how nice it would be if fans (and the movement of air) could be completely silent since running the fans full speed would be sooooo much easier.
Super cool
Great video. Was wondering what desks you had there.
Thank you!
They're the Bekant desks from IKEA
@@JasonRose Thank you for the reply!
I thought my rig was out-of-hand, but you're running almost twice the storage as my setup. Curious as to why you're running an LGA v3 CPU. Wondering what I'm missing using xeon 2630v1 dual CPU setup; very low TDP. I'm using Windows 2012 server and only 32Gb ram so maybe that's it along with the fact I'm only using the setup as a file server and other rigs for hypervisor and Plex server. Very interesting. I'll have to read your webpage to see what adjustments I might need to make. Btw, love the fan shrouds although too pricey for me but maybe in the future; who knows. Great work!
Hey man first off just wanted to say great job! You definitely had fun with this project and I'm not sure of your skills prior to this project but I assume you've learned a ton as well which seems essentially to me the biggest reason you made certain decisions , that and for the fun of tinkering !!
Im not sure your understanding or knowledge of Ceph, but I think you would absolutely love it and just fall down the (admittedly sizable) rabbit hole just learning ceph clusters and all that comes with it !!
Curious if you have ever heard of 45Drives? That's actually where I work and you were dead on and informative with your overview on the LSI cards and SAS. Still to this day I smirk at jbod. Give us tech guys the ability to name things , and that's the type of things we come up with :) but yeah although I think there were quite a few simpler ways to achieve your end goal I can absolutely 100% appreciate the route you took to get there. It included a ton of really fun ingenuity and outside the box thinking !
I'm glad you enjoyed the video! I've most certainly heard of 45Drives and I'd be interested in playing with ceph at some point but just haven't had the hardware to do it yet.
What is your typical ambient temp in your office space? Wow, a lot of work! I have historically just built storage servers, stuck them in a data center and walked away. Maybe physically touch them once a year if needed. If I had to do anything of what you went through in this project, I wouldn't be in IT. Great Video, well done!
Pretty sure the monitoring could have been done with SNMP.. but you're an Engineer. So I am not going to hold that against you. That's a beautiful custom setup.
Well, at least the dog seems to like it.
Cool videos, I'm just curious to know what kind of data your storing that you need a set up like this?
How gorgous that rackcase with UPS's and 24 bay file storage is. I see why you have chose your network switches on the back to mnimize UTP cables length. How deep is that rackcase? I have a 37U and its 800mm deep,but I have IBM Lenovo x3650 m3 server that doesnt fit the rackcase because no one told me it is the depth of front steel beam and rear steel beam that needs to be 700mm -800 in order to rackservers to have enough space and close the rear door. I'm thinking of getting a 42U APC that is 1070mm deep,but I also am considering a 1200mm deep rackcase,what you think?
I plan on buying the SC847 rackcase which has 36 bays, 24bays in front will be for mass storage while the 12 rear will be SSD array for VM's backup,snapshots,network configs on cisco switches and perhaps a L2ARC SSD and SSD 512GB in RAID1e for redundancy and a few 10Gbit SFP+ NIC's and a daughter quad port NIC for LACP. And maybee even get a 900p as the bootdrive.
I'm just getting into looking for supermicro motherboards that supports this kind of storage + ssd's and maybe some NVME drives. I dont really know which motherboards to chose from since its so many and confusing since the SC847 SKU info mentions that certain motherboards are designed for the chassis,but when I click on them I see some are 2011 socket motherboards. I'm kinda want a single socket Supermicro MB and pref a EPYC for all the lanes.
But that is gonna be big project and probably going to end up at 10k dollars ++before I have the parts and such. But I love servers,switches and any rackmounted units. I also plan on getting a rackconsole with a KVM switch.
How big is your UPS's? VA with the amount of powerplugs? Are you doing 1 powerplug from each of the PSUs to each of the UPS's? So its even distributed in case 1 UPS dies?
Do you plan on getting some rackmount servers with WS2019/2016 for windows services?. Please go into detail what kind motherboard and components you have there. Also do you have any network monitoring sw for survering your network?
Awesome stuff. You need a pop filter for your audio, though :-)
Super awesome build! I have the same chassis. I've already swapped my PSUs out for a normal PSU from a desktop and the rear fans with noctuas. However, after adding new drives (for a total of 24) the drives got hot on the lowest speed control on the BIOs and I had to ramp up and it's louder than I'd like. Which cooling modification(s) would you recommend doing for the most impact so I can lower the noise and keep the same airflow (or better airflow)?
Loved the video, but you should really introduce the dog!
I have kicked around SSD development, just curious if you think that there would be any other bottle necks aside from those network related.
You have done well.
I did a video the other week on hardware and bottlenecks.
Now this is the issue which maybe you find later.
The bus for the das is what it is.
We will say 100gb/s.
If you link is 100gb/s
But the board is 80gb/s then you get a issue.
The data , plus process, plus control (memory, net, input , GPU, etc) exceed the system bus and therefore creates a bottle neck.
Also Would you (in hindsight) have been better off making a DAS and a load balancing controller between the shelves?
Could you maybe touch on how you did the PSU setup between chassis again?
Looking to do something similar but im a total nub lol
Great informative video Jason. I know you made this for FreeNas, but do you think something like this work for UnRaid?
Good to hear from you again Wil! I have no doubt that at least the HBA and PSU stuff would work for Unraid (assuming it has the drivers for the HBAs I used, which it should). The fan control stuff might be a bit more tricky; you'll have to translate some of the shell commands run in the main Python script to their Unraid equivalent, but the basic idea should hold up just fine.
@@JasonRose Thank you so much for the response. I really appreciate it.
Have you looked at Grafana for your information display? joined with influxdb and telegraf it would really improve the visual look and allow a history of how the setup is doing.
Only briefly, but I do want to experiment with it a bit more. This whole project took like 6 months and I need some time away from it before I go back in and try to improve things, but it'll happen eventually I'm sure.
Three questions:
1) In your JBOD expansion shelf, you were able to use a single SAS3 cable to the SAS3 backplane. In order to simplify your cable management on your head node, could you have done the same?
2) If you need to expand to more shelves, would you be directly connecting them to the external ports on the LSI 9305-16e or would you be daisy-chaining the shelves to each other?
3) Knowing what you now know in regards to the ever growing storage needs, would you have preferred to gone with something that could have done 30-45 drives (horizontally like you have (e.g. Supermicro SC847) or would you have preferred one of those upto 90 drive bay, top loading chassis instead (either for the head node or for the expansion shelf)?
Thanks.
1) Yes, I could have done that, but it would have been more expensive. As I mentioned in part 1, the SAS3 backplanes are expensive.
2) I'll be directly connecting them to the 16e HBA. Daisy chaining would create its own bottlenecks and I've got plenty of SAS ports.
3) Those top loading shelves are extremely expensive and require really loud fans to cool, so I'll probably always stick with the 24 bay front-loading shelves.
@@JasonRose
Thank you.
But it is possible, so that's good to know.
It appears you are powering the fans directly from the IO pins on the Arduino? I did not realize those could source so much current.
Nah, they're powered from the +12V on the PSU. The PWM out pin on the Ardunio (and on the Pi) only has to do a few mA at +5V.
What is the case that is above the shelf holding your mini tower and below your sc846? The one with three fans in the front and the power button on the top left.
Put up some sound dampening in your office, it'll get that ambient noise way down.
I'm curious what you did to modify the Supermicro MCP-290-00073-0N Cable Management Arm. Any pictures/description of how you made it fit the 846?
Just one thing I noticed, you said I TWO C (i2c). The electrical engineers I know say I-squared-C :)
With that much storage, How much does it cost to have an off site backup?
Arduinos are not ideal for any serious project, for a variety of reasons. You would have a much better time with a Nucleo board from ST. The price is comparable, but the tools are far better and so is the performance. You can also use FreeRTOS which is a multi-tasking OS. Also, whatever chip you choose, you need to make use of the watchdog timer if you want simple, secure recovery in case your code freezes.
How r u handling heat from rack
how did you get the drive temps to the raspberry´s ? i have a similar idea of controlling the fans, but i cant find a direct way to get the drive temps out of freenas.
Python's sockets module, one script running on the FreeNAS and one on the rpi. Check out the code in the github for more info.
What do they do in industry to control temperatures of thousands of drives? Just run all the fans full speed 24/7/365?
Will freenas be able to tell me if hardrive fails?
Just curious, with all this expansion, what do you have in the Supermicro Mini tower??? And how does it tie into the existing setup?
That's a FreeNAS mini system I use for testing nightly builds, new features, etc etc. It's not tied into the main FreeNAS system at all.
@@JasonRose OK... I have a small low powered Xeon system built in the same case that will be a ESXi or Proxmox system soon
What the model of the raspberry pi touch screen?
Do you still recommend 3d hubs? I tried to use the site but it wants my work email. I don't know why it won't let me buy without using one.
Great video but something weird is going on with your audio. Sounds like it keeps flipping between a stereo channel and mono. Or some very strange phasing going on with the mic you were recording with.
So the raspberry pi is controlling the fans in the expansion chassis? Are they also providing the power to the fans?
Yep, exactly. They don't power the fans, just the PWM line; the 12V is pulled from the CPU. RPi can't output very much power.
@@JasonRose Can you do an in-depth video on how you have this wired up? My visualization of this muddy at best and my experience with wiring is spotty.
You have hundreds of TB of storage. It's on FreeNAS, so it's protected via redundancy... but what about backup? How do you handle that?
I use rclone to encrypt everything and back it up to a cloud provider
Your content is in part or above the highest quality videos out there about technology. Trying to reach an audience with such a broad distribution on technical knowledge is really hard but you take the time and effort to reach beginners when is necessary and cut corners when it makes sense. Never going to miss another one of your videos.
Also, why don't you use certbot and get a certificate for your website? jro.io/
Thanks for the feedback, glad you enjoyed the videos! I've been meaning to set up a cert on my site for a while but just haven't gotten around to it. It's all static content, so my motivation has been pretty low, but I'll get it set up in the next month or so probably.
Hi, can you link to your reddit account so we can go through your posts? (in relation to this project) and maybe a follow ;)
Check out NetData. You might find your dashboard heaven.
Very familiar with netdata, it's good stuff. No HDD temp info in there though.
What are the helper arms that carry your power, data & ect called that you use for back of the rack management. My Rack looks like a Ball of wire's and I can not pull server's out with out disconnecting them. Here's a link ibb.co/4Jt8MvZ
They're called cable management arms: store.supermicro.com/2u-3u-4u-cable-management-arm-mcp-290-00073-0n.html
I undertook two dozen's of Freenas Installations. This is surely overwhelming to me. I am amazed with your work. Can a distributed filesystem can be installed over Freenas ????
I have also stumbled upon a project on www.ovios.org/ which is quite similar to Freenas but without any kind of bloat. Request for your review. Only expert like you can do justice with this.
Thanks! And no, FreeNAS is ZFS only.
What is your day job and why do you need so much equipment?
lol "need". It's my hobby, I don't need any of it, but I enjoy playing with it.