Dell PowerEdge R410 build PART 5 | Expanding FreeNAS storage by building Supermicro 836 JBOD / DAS
ฝัง
- เผยแพร่เมื่อ 20 ต.ค. 2024
- This is the 5th video in the Dell PowerEdge R410 build series. Up until this point, we've been getting the hardware setup and putting FreeNAS on a USB/mSATA SSD to turn this R410 into a NAS device. In this video, I'm going to expand the storage of this R410 NAS with an external JBOD / DAS / Disk shelf chassis with 16-bays. I'm using the Supermicro 836 chassis, and converting it to a JBOD. First, I'll show how to build it using "traditional" methods, so you'll understand the fundamental components needed to build your own JBOD. However, I'll rebuild it again using "unconventional" approach that costs much less.
If you're interested in using the Dell R410, be sure to checkout the rest of this video series:
• Dell PowerEdge R410 bu...
If you'd like to support this channel, please consider shopping at my eBay store: ebay.to/2ZKBFDM
Link to Lenovo 03X3834 SAS-2 expander: ebay.to/2OMoLTg
Link to Supermicro CSE-PTJBOD-CB2 controller: ebay.to/48uaUVk
Here are some affiliate eBay search links for some of the products I used in this video:
eBay search for Supermicro CSE-PTJBOD-CB2: ebay.to/38CS3uP
eBay search for Supermicro CSE-PTJBOD-CB3: ebay.to/3lcT41T
eBay search for Supermicro X9SPU-F: ebay.to/3tm96cD
eBay search for Supermicro 836: ebay.to/3rKRxSV
eBay search for Supermicro BPN-SAS2-836EL1: ebay.to/3rLw7VH
eBay search for Noctua NA-RC7 low noise fan adapters: ebay.to/3rNmwOf
eBay search for SFF-8087 to SFF-8088 PCI adapter: ebay.to/30I7mhy
eBay Partner Affiliate disclosure:
The eBay links in this video description are eBay partner affiliate links. By using these links to shop on eBay, you support my channel, at no additional cost to you. Even if you do not buy from the ART OF SERVER eBay store, any purchases you make on eBay via these links, will help support my channel. Please consider using them for your eBay shopping. Thank you for all your support! :-)
SUPER super brilliant. Seriously -- this guy deserves 100k subscribers ...
INCREDIBLY knowledgeable. Absolutely subscribed!
Thanks for the sub! Please share! :-)
He is super knowledgeable. I asked for advice on what to buy on his ebay store and taught me about mistake I made on my sas setup and pointed me to what I needed. Highly recommend his store.
AGREED, he has supported me every step of the way, extremely knowledgeable, and extremely hard working guy.
This video & the links have helped me figure out wtf hardware our old IT manager left us, but never finished setting up. Now I know we have 4xCSE-826 chassis with the CSE-PTJBOD-CB3 boards...Neat. We have 40TB worth of disks on order...hopefully it all works out. Thank you!
Glad this helped! Thanks for watching!
Very interesting that expander is able to function standalone without cpu and ram. Thanks for get videos and explanations on all videos I would like to say most of us probably never delt with enterprise hardware. I've built alot of computers but never touched this type of stuff. What's really intriguing is the amount of bandwidth your capable of getting out of these servers, ram and thread counts. There is some interesting possilities owning atleast one server.
Glad these videos have given you some ideas! Lots of possibilities for sure! Thanks for watching!
Super happy I found this Video.. thx... Built two and running well...
Glad it helped! Thanks for watching!
Beautiful, this certainly helped out and helped me understand what I was unsure of. Thanks for the help over the message earlier. Found my chassis and will grab whatever I need that you have from your store.
Thanks again.
Glad it was helpful!
Would you ever be willing to throw together a series on how to create your own SAN? A NAS is great for file storage, but I've wanted to build a SAN for block-level storage.
I'd like to see the process where you break down the hardware needed, such as what it is and why it's needed, and how everything interacts and communicates with each other to form the SAN.
That's an interesting idea and I might consider it. However, most of my audience I think are better off using a NAS. Home networks are fairly common now, and some even run 10Gb networks. and if you want block storage there's always iSCSI (or AoE?) . I also don't really have the hardware here to demo a SAN.
nonetheless, I appreciate you sharing your thoughts and suggestions. keep'em coming! :-) thank you for watching!
Thanks for this!! I didn't realize till now how easy it was to do the sas expander with the supermicro board. I have now ordered both!
Glad it was helpful!
@@ArtofServer All setup now, but I have a weird issue. The IPMI has the system as off but everything is on in there, I am thinking this might be the fact that I am using a really old PSU but just wanted to check with you.
I'm not sure about the backplane that you have in your 3U box, but the BPN-826-EL1 & EL2 backplanes that I have have fan connectors on them that, a bit after power on, they ramp the fans down, rather than leave them blasting at full speed.
The great part about using something like a R410 as a FreeNAS 'controller' and then have an external JBOD enclosure is recoverability. If the R410 were to suffer a massive failure and you had to get back up, you could find pretty much any machine that would take the HBA(s) and boot media and you'll have your FreeNAS back up in no time. If you have everything inside a Dell or Supermicro (or whatever) box and the motherboard were to fail, you're pretty much stuck until you can find a proper board for the case.
I originally started with a 2u Supermicro box with a dual socket Supermicro board and an external 2u JBOD, but I converted the first enclosure over to a JBOD and started running FreeNAS on a R620. I've since migrated away from the R620 to a FC630, which was the easiest of migrations as all I had to do was shut it down, move my SAS cables and boot drives and turn on the FC630.
Thanks for sharing! That's a good point about the expander backplanes with fan headers.
Thanks. I plan on moving from my turnkey QNAP NASes to supermicro rack chassis system.
I'm a big fan of supermicro! They are great for custom diy builds.
Can you do the same with a 10Gig ethernet card on both with the host running ubuntu and supermicro running freenas then do the same? Having the supermicro's hard drive show up in the host OS as just hard drives.
There's no real comparison. 8x SAS-2 lanes has 48Gbps of theoretical bandwidth while 10Gbe has only 10Gbps theoretical. Your solution could have the Supermicro running a light OS exposing the disks via iSCSI which you can then map on the host but that'll be limited to 10Gbps minus the comparatively high overhead and latency of all the translations
Is there some trick to getting the Supermicro motherboard to stay on when powered without a cpu or memory? I’ve tried both a X9SCL-F and X10SLM+-F and they just continually power off, then back on, then off again and repeat that without ever staying on. Any thoughts or tips?
No, I didn't have to do anything special... that sounds like you might have some other problem going on...
When I use the Sunmicro case 836 with motherboard only setup, I have the impression that the PSU is making far more noise compared to loading the motherboard with CPU (at the start). Once the CPU starts getting hotter, the PSU goes into overdrive.
Do you have any suggestion in keeping the PSU also in low noise? In this setup the PSU is making too much noise. (I tried with 1 PSU / 2 PSU, and I have tried 4 different PWS-2K02P-1R modules to see if some might be more silent)
In your setup do you use the PWS-2K02P-1R or a different PSU
Is it an idea to put in a pws-920p-sq or a PWS-1K28P-SQ ? Do you have an idea which is the more silent of both? Which one would you recommend?
I would use the PWS-920P-SQ. The 1K28P version has a wider connector that is not compatible with all PDBs, but the 920P is compatible with all PDBs of that form factor. If you know your PDB can take the wider connector, then you can consider either, based on your expected power load.
How do you synchronize power supplies such that the JBOD chassis is available on boot?
there are many ways. if the enclosure has remote management, you can send a signal either via IPMI LAN command, or SNMP or whatever protocol might be available. it could also be very crude and you just have a cable from the main server into the enclosure that activates the PSU.
Great series! :D
Thanks! 😄
I have an X10SLL-F in my JBOD case (Antec VSK4000E). IPMI is working with CPU and RAM. I tried to use this to run my fans and power without CPU and RAM but after removing the CPU and RAM, it would just constantly power cycle, IPMI showing on and off cycling without ability to manage remotely. Any thoughts? Also, I just aquired a SM 825-TQ as a chassis upgrade so I wonder if this will allow the motherboard to function properly for this purpose because maybe the motherboard needs certain things plugged into it that the antec didn't have but the 825-TQ will? Thanks
no, not all motherboards will work, and maybe the X10SLL-F is one of those that will not just power on without CPU/RAM. i don't think the 825TQ will help in this case. probably should just use a CSE-PTJBOD board.
@@ArtofServer sorry I didn’t realize at the time that the 825TQ is just the backplane but I also have the super micro chassis with it as well now. So the antec case will be replaced. I wonder if it’s worth leaving the cpu and ram in as I’m not sure about it’s power consumption when being used for this purpose.
For the Lenovo 03X3834 SAS-2 expander, if I am connecting SATA drives, are they at 6Gbps or 3Gbps? In my researching before buying from your store I am finding conflicting data out here. Thank you for a very informative and helpful video/s
if the SATA drive is 6Gbps capable, then the link between that SATA drive and the SAS-2 expander should be 6Gbps, unless there are signal integrity issues that causes it to try a lower link speed.
Nice info, thx!
You bet!
Oh snap ! Just wanna know how to do this... Thank you from Belgium ( IT's me who buy a H710p IT MODE flashed from you ) Still waiting for it looking at the postman everyday 🤣😂
Glad I could help! Thanks for your support! :-)
Art of Server Hello there, just receive my H710 today and allready tested. Working nicely in my second r720xd unraid server allready build my array and now performing a parity check ! Everything Is fine 👌🏻 Thank you so much 👍🏻
@@jj-icejoe6642 Awesome!! :-) Thanks for your support JJ!
Silly question, what do you need to SAS expander for? Can you not directly plug the cables from the backplane onto the internal to external card that you've got? You will need two of those but I suppose the extra one is cheaper than the expander itself?
The HBA card only has 8 SAS lanes, so if I just plugged it directly into the backplane, I would only have enough connections for 8 drives. Using the SAS expander, I can connect to all 16 drives using just the 8 or 4 SAS lanes.
What's the difference between this setup and those ready-made solutions such as DELL PowerVault or HP StorageWorks? What are advantages or the selling point of these commercial products?
The PowerVaults and StorageWorks are not very easy to customize or modify (although, I have a future video coming up on that subject)... and by default, they are pretty loud. Building a custom DAS/JBOD chassis allows you to better control the system, like fan control, expander vs direct-attached, etc.
Of course, if the operating parameters work for you, the commercial solutions are easy and ready to go. Probably the simplest way to get going. A custom build takes time, perhaps experimenting with different parts, etc., and may cost more in the end, considering I'm seeing MD1200 going for very little cost these days.
Very interesting series. I am building my own r410 now because of your channel. Could one use a h200 card to connect to a jbod? Because my machine came with one and since I am using the sas 6ir it is now not being used.
Thanks
Great to hear that this series helped you with your R410! :-)
The H200 is an internal card... so no straight forward way to use that to connect to JBOD.
@@ArtofServer
Thank you for your reply. I know it is an internal card, there is also the h200e with external connectors. I was wondering if there is a difference other than the ports? I know it is physically difficult to route the cable from the internal version to outside the server. But if this were possible could I use the card or are they different inside too? Firmware or hardware?
Thanks
This Lenovo Expander you have. Can you clarify that it will work with Unraid (obviously together with an Unraid compatible HBA), I guess it should? Is it 8x PCIE 3.0? Do you get near 6 Gb/s SATA speed on attached SSDs?
The Lenovo expander is like the IBM 46M0997 I've spoken about in the past. So, watch those videos and they equally apply.
the expander card is *not* a PCIe device... it just uses the slot for power. the data connection is via the HBA you connect to it. as long as your HBA works, the expander works with it.
just watched your series. there is something missing. i see many cards out there that do not have "to controller" labeled on their ports. they are listed as 24 or 32 port cards and yet that is 100% of their ports. so they count the interconnects to the HBA card as "ports" so a 32 port card can run 28 drives with a single connection to the HBA OR 24 drives with 2 connections? you dont show how to hook the cards up, how their daisy chain works, what the bandwith limitations are as you start chaining more and more cards.
SAS expanders are like network switches for SAS protocol. There are no "input" or "output" ports or "to controller" ... that is just suggestive for the particular setup in a particular server. Any port, up to 2, can be used to connect upstream to the SAS controller. The rest can be used to connect to target devices or other SAS expanders. I have an entire playlist on SAS expanders here:
th-cam.com/play/PL28eVGz5vFQ-pn6eFBC6AmfbL3yPcBDV7.html
maybe some of those vids will help you out.
If you want to better understand the bandwidth considerations, perhaps this video will help:
th-cam.com/video/Q4e8kmuGm6o/w-d-xo.html
Sorry if I'm misunderstanding, but are you slowing the fan speed down via IPMI or is there something in line that lowers the PWM voltage?
In this demonstration, I used noctua fan resistors to slow the fan speeds. I mentioned using the BMC/ipmi if you add a low power cpu to enable that functionality.
@@ArtofServer Sorry for the 2-year bump on this. Have a question along this same train of thought.
If you have an X9 F motherboard with CPU and RAM, and set the fan profile (say to "Optimal speed" which defaults to ~30%), and then power off, yank the CPU and RAM, will it still adhere to that profile, or will it just spin up to max speed all the time?
Do you have a link to flashing the latest firmware on the Lenovo sas expander?
Not specific to the Lenovo, but the process is the same as in this video: th-cam.com/video/Lw4TTI_HYqM/w-d-xo.html
@@ArtofServer thank you
Thank you so much man I really needed this kind of video.
Question, how many GB of Ram do we need per TB for FreeNAS? My guess is 1GB per 1TB according to google
Glad you found this helpful!
Scaling ram with storage is not linear. In the end, it depends on your use pattern. I would say minimum of 8gb, but monitor how your ARC behaves and add ram as necessary. Of course, more ram usually means higher performance.
@@ArtofServer Yeah I was thinking about that, thank you!
Hi! Thanks for your thorough videos. Are you planning on covering the firmware update procedure for the 03X3834? Does it even need it? I got my hands on one and am curious. Not sure where to find the firmware for it. If you had a link, I'd appreciate it. Thanks!
So far, of all the units I've received, they already have the latest firmware. I don't know if Lenovo just always sold them that way or I just got lucky. I compared the firmware on the ones I received to the latest download available from Lenovo. Also, I've tested the Lenovo card and did not encounter any issues like I did with the IBM 46M0997, so I'm pretty confident the current firmware is good AFAIK. The IBM one had issues where certain drives wouldn't show up during boot up.
If a firmware update is necessary, the same procedure I showed for the IBM expander should also work.
@@ArtofServer Awesome! Thank you for all your expertise.
So I tried the hack of no CPU and ram and I press power, nothing happens. When I go into ipmi and try and power on, I get please wait for a moment after acpi system status has changed. Help?
Are you using the same exact motherboard or a different one?
@@ArtofServer x9dri-f
The funny thing is I started to put the CPU in while BNC was running the system beeps and some lights came on so I took it off right away and unplugged it put the CPU and it booted I am using a low watt cpu but it would have been nice not to use one at all but the bonuses obviously fan control without putting resistors
Would an HP iLo Motherboard, or Dell IDrac Motherboard be another option to use without the processor or the ram for this remote boot and fan control features?
No idea. Maybe try it and report back!
So my X9DRi-F can run without CPU and ram? I plan to use my 4u as jbod as well
You'd have to try it out
Another alternate power/fan control option is an arduino ethernet with a cuple temperature probes for full range pwm fan control and remote power toggle.. can run on 5vsb psu output.. i intended to make a proper pcb but i only needed 1 so i only have my hacked together in use prototype, a rough schematic, and the code i wrote for simple html browser based power/pwm fan override control.. total cost was about $25 in chinese knockoff components.. you can also add a 4-wire quick connect to the host server power button and led leads to make them both auto power up simultaneously with either button via the arduino.. an easier way if u just want power toggle is get a tiny 5v relay, solder the coil leads to a usb cord and the NO(normally open) and C(common) to Gnd and the small (usually green) power trigger wire on the jbod psu... then it will dumbly follow the power state of whatever u have the usb cord plugged into... if u have questions u know how 2 reach me..
that's awesome. i hope you can share the details of your solution some day!
IPMITOOLS
Hi, do you have "HBA H200e in IT mode" at your store? Thanks
No, I don't. But I have 9200-8e which is same in function.
could you do any type of gpu/cpu passthrough on those array of disks?
На бэкплейне для дисков есть разьемы для подключения кулеров.
Да, эти разъемы для вентиляторов на объединительной плате возможны. Но они только запускают вентиляторы на полной скорости. Нет управления вентилятором. Спасибо за просмотр
Could you not use a raspberry pi to start this up?
Well, what can't you do with a RPi? :-) Sure, there are many ways to run a JBOD chassis... the $20 CPU-less M/B is hard to beat on price.. even vs. RPi.
Wouldn't this prevent your from adding 10g networking?
Yeah, one has to choose what to do with that 1 available PCIe slot...
@@ArtofServer is there a a good platform you would recommend with multiple PCIe slots for this use case? I have an R410 I was going to use but I was planning on using it for iSCSI to a vm host. So I was worried about bandwidth on a gig connection.
Yeah, check out the R510/12-bay or R720XD/12-bay.
I hear ya! give a vendor some business and BOOM!! they get greedy!! That sucks bro!!
Nature of business I suppose... Thanks for watching!
Wish there was an low profile option for an expander card.
I haven't used it before, but there is a low profile option by Intel. Search for RES2SV240.
@@ArtofServer Thanks, i never heard of that one before. Unfortunately it seems unreasonable priced compared to a SAS controller.
@@ArtofServer I knew without even looking at the fact that there were replies that of COURSE you'd not only know of one, but, knew the damned part by HEART. omg, seriously. Amazingly knowledgeable. I cannot even begin to say how grateful I am to have found a channel with this level of clarity and knowledge. Just a WEALTH of info.
Why not just get a SAS card that has internal and external connectors?
That's certainly another idea. The only LSI card I know with internal + external would be the 9207-4i4e. The 9200-8e would provide 2x SFF-8088 ports for more bandwidth or greater expansion capability.
@@ArtofServer I'm guessing he kinda missed the point; your premise is a device which doesn't even require a CPU ... ergo, even IF the cards were the same price, one has a substantially steeper set of requirements to be able to work... when the CPU's role here is little more than a fan controller (or with IPMI, to time the sequence of initialization).
So based on consumer drives typically being much cheaper the 9207-4i4e, 9217-4i4e (which i just bought for my r410) flashed in IT and without the premium of the 9207 but same card both operating at pcie 3.0 . 9212-4i4e, which if you have the non backplane r410 would be a great option. The rest of the higher line up with sas 3 connectivity would be problematic in the 410. But there's definitely some options especially with the 9212 and 17 under 30 bucks.
hi i would like to buy the setup you have in this video.
th-cam.com/video/qccpopxc_Uo/w-d-xo.html
but i am wondering if this would work with windows 10? plug and play? or would i have to go into the hba and set it up? if so in windows it would see 24 drives? want to use it for mining. so this will work for that? will be using exos 16tb hdd's.
Yes, with that setup, the 24x HDDs should just show up in the OS as individual disks.
@@ArtofServer will it be plug and play?
if i buy from you?
ok price for those cards hba + jumper cable 8087 to 8087 and 6x8087to 8087 cables 1m long shipped to thailand?
is there a limit on drive size? like can i load this out with 24 16tb drives?
how the fuck did you get that x9spu for 20$ when they are all 400$... I want one :/
do you know what, if any, functions are lost using a full board with IPMI vs the cb3 board?
one difference is that the CB3 has the IPMI port on the inside, not at the IO space, and the cable/bracket assemble for getting it to the PCI slots is...like 250$ :/
LOL.. there was a time when they were dumping those boards and I got one.
I don't know for sure.. but i think a full IPMI enabled mobo has the same features as the CB3 if you add a low power CPU. I think the CB3 can moderate the fans if you run the backplane I2C to the CB3, whereas the IPMI enabled m/b without CPU runs the fans at full speed and you'd have to manually intervene on the fan speeds.
@@ArtofServer ya, that's about what i figured too. still cheaper than the CB3 board cuz...wow.