45drives Visit: ZFS, Snapshield Ransomware Protection, and Fun Homelab Projects!
ฝัง
- เผยแพร่เมื่อ 7 ก.ย. 2023
- Wendell went all the way to Sydney, Nova Scotia to hang out with 45Drives! Whoa!
**********************************
Check us out online at the following places!
bio.link/level1techs
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Intro and Outro Music By: Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons.org/licenses/b... - วิทยาศาสตร์และเทคโนโลยี
It was so fun meeting you, Tim, (other) Jeff, and all the other folks 45Drives brought to the summit. Can't wait to see where they go next, and I do think they're on to something with the Junior model. Or I just love it's based on a Raspberry Pi 😂
ooo can't wait for your video too
A smaller more affordable 45 Drives chassis would be amazing. that $800 price point sounds very attractive.
oh shit, how many beers did you try?
"..based on a Raspberry Pi.."
The Pi foundation decisions (which made sense at the time) and global circumstances have made every maker I know (including myself) 'meh out' of anything Pi related.
You can get a legit ITX/m-ATX x86-64 NAS board for the same price as a good Pi. And you don't have to include 4 pints of black-market blood or a handjob with the price just to find one available. Who cares if it idles at 8W instead of 3W?
I would be WAY more interested if they got a supplier for one of those low power 8-Ecore embedded CPUs with 8(12?) real PCIe lanes.
These things sip power (5-20W) and have a reasonable enough BOM that they could compete with an 8GB Pi in price.
Hell yeah! Coal burners, unite!!
Great meeting you and everyone else! I had such a great time!
If they can make stuff that is more affordable for the homelab community. I will be jumping on that quickly. I have been wanting to upgrade to this kind of stuff, but has always been out of my price range. Super excited to hear they are looking at us.
This is only for people with money and you sound like a poor person, please get out of here.
idk why but I'm geeking out that you visited my neck of the woods in Nova Scotia! Sydney is a sleepy town 100%.
That little homelab box is so attractive right now. 10ish drives in little more than an SFF box. I want one or two or three...
$800 for the homelab 15 bay case and backplane seems kind of high. Not sure how many they hope to sell.
I had no idea 45Drives was based in NS! Nice to see my home province represented in my favourite hobbyist community. Did you catch the "u" in favourite there? That's how you know I'm Canadian.
Welcome to the province! I feel like a celebrity is visiting Nova Scotia!
we had a public meet and greet on the second day of the event, too bad you weren't able to attend!
have their 15 drive storinator, but got it barebones and fully customized it, fans, power supply, motherboard and all, working great as a case with a drive cage.
Got to talk to 45Drives this week. Mentioned you and they said you were there last week.
I've been to Sydney, NS. Very nice folks there!
this is good shit and is why i subscribe, just when i think i have out grown most of your content you come out with some REAL shit
Would be fun to get one of those NVMe servers (populated) into the hands of Jens Axboe as well.
Cheers to the decade I've been watching Wendell on TH-cam. It's crazy to say that. Thanks for everything
Also shout out to Cincinnati and that glorious chili
I am so excited for the homelab chassis. I been looking around to replace my current norco 4U. Those look great!
The 15 drive home lab machine is super cool, that price is pretty steep tho!
If it includes all server hardware (cpu, ram, mobo, etc.) minus non-boot drives, it's a pretty good price for $800 I think. Plus whatever engineering they put into the drive bay board(s), building a similar server out of consumer parts is about the same price in recent experience (albeit idk what specs they plan to give it)
I might have miss interpreted what was said about bringing your own parts in the few clips I've seen of this product so I'm still not sure what's included.
The company I work for has a ton of their older 60 and 45 bay servers and they're built like absolute tanks. $800 for a bare bones kit with no mobo seems like a lot but I could see it being a reality. Hopefully 45drives makes a video on it and gives us a run down. I'd definitely like to pick one up.
Yeah the home lab will be built with the same process and tools as their enterprise equipment, so the build quality will be pretty much the same.
$800 is only for the case and backplane, according to their site "45homelab".
Containers, isolation, ZFS. Would be cool to see Illumos backing their software stack.
Shocked that the Canadians let you leave.
He had to fight his way out
Would love to run ZFS on my M1 Mac and maybe get back some extra storage space and maybe have a GUI for friends and family who are less technical with the command line but would also want the benefits of ZFS
That junior case is freaking awesome
Great video
All support for competing with Synology for the home lab nas.
I never understood why people like Synology so much, from what I know about it, its so overrated
For a turnkey solution they work remarkably well. They have great support. You just don’t get a lot of hardware for your dollar.
@@keyboard_g that’s what confuses me. I made a much much faster and tricked out NAS with a $160 for ryzen + motherboard and $70 for used 64gb RAM and unraid. So what would be better? Synology or a system like mine ? Like I’m genuinely curious
Synology is for people who don’t want to build and maintain their own system. Although, I will credit them. Mine is years old and apps and the OS still get security patches.
Drilling holes in harddrives? Wait, it's not common practice to take them out to a shooting range for target practice? 😛
45Drives is in Canada so. . . . .pew laws are weird and getting worse there.
But in the freer parts of the USA, a 308 remote hole puncher for HDD disposal is a DOD certified data destruction method! And fun!
@@NPzedI think Airsoft guns are still legal here. For a while, anyway. 🤷🏻♂️
And then apply thermite, just to be sure.
I really like some of those cases. They look very helpful in the future.
Very interesting, i would definitely be interested in a 45Dr barebone, but they also need to support U.2 drive cages maybe E.1 as well. The main disadvantage of any case nowadays is that they increase case size but reduce the number of drivebays and almost none of them have backplanes for normal HD's, U.2/3 and E.1.
Neat! As an ex CNC machinist I'd love to see the inside of the machine shop. But yeahh usually you're not allowed to take videos in them.
If you are interested how some of the work is done at 45 Drives/ProtoCase - check out old LTT video when Linus got his chassis made by them (quite some time ago) for his server rack mounted PC. Video is called "The birth of my custom gaming case - Personal rig update 2015 Part 3"
If custom building from raw metal was really easy everyone would be doing it! But a community effort might help expand current functionality and demand. Sorta like open source but for enclosures. Mountain mods had an idea year's ago but unfortunately does not provide as much customization as building your own arcade cabinet from scratch including the buttons and joys even coin slot placement decals, Unameit.
Awesome to see you enjoyed Cape Breton!.. I'm down in Halifax, so cool to see all this haha. ;)
Love the idea of a storanator jr. would like a itx case that could hold a gaggle of 2.5" drives that could be easily accessible. That said I would prefer the drives not be exposed like that but could be a nifty concept for someone who hot swaps drives a lot I suppose.
I definitely am interested in the homelab project for the future. Nothing against my cases holding my servers now, but I'd love to get to a rack with one of those in the future.
This is only for people with money and you sound like a poor person, please get out of here.
Interesting that Protocase is doing ITAR things
Full custom electronics enclosures with the supply chain mostly or completely within a Five Eyes / NATO country? Of course they are doing ITAR stuff...
Wendell in my home province, weird!
This was an extremely cool video
Didn't see Ricky from Trailer Park Boys on that Nova Scotia trip.
I call bs on that trip/
As someone who has deployed Ceph in production at a time which was to early, I can say I'm excited about Ceph improvements but still a bit burned.
CEPH can be a blessing for HA and a curse if not setup correctly. I've deployed ProxMox V5 with CEPH a few years ago. While it worked well it had performance issues. Never able to solve it. I later rebuilt the ProxMox V7 cluster with regular ZFS. That's been running great for me. I may revisit CEPH on our secondary backup cluster at a later date.
@@Darkk6969 First thing to know: Ceph and similar solutions which replicate in a similar way depends on high performance networking, if you don't have it, it's gonna be terrible.
Super interested in those smaller form factor 8-bay and ssd units!
13:19 as an electrician this gives me anxiety.
I am not seeing water drip mitigation at all...
45Drives Homelab gonna be really really nice I wouldn’t mind paying a bit for something actually good.
it's hilarious how much i'd love a storinator jr chassis to use with one of my RK3588 boards
I wonder if they could do something with that storanator junior and allow access via USB Type-C? Especially if you can then car out that storage on your host machine, Windows, Linux, macOS, in a pinch a mobile device.
21:56 - It’s Avian Geerling!!! 😅😅😅
Zee? No, no, no. It's zed!
Man wish I could visit for a tour.
Would love to see 45Drives/ProtoCase fill the gap left by CaseLabs
what homelabbers are actually looking for: power efficient, low idle power draw(less than 15w), **'quiet**, all flash/nvme 1or2u enclosure and short depth less than 20"
One can get an 8TB PCIe Gen4 NVMe Solidigm D7-5520 for around 610 USD+tax (I even saw the price down to ~510 for a short period). Grabbing three of those drives for a RAIDZ1 flash array resulting in 14TB useable (23TB raw) zpool is definitely within many homelab budgets. The trick is sourcing the right "low power" brain (CPU, MB, etc) and chassis. . .especially if you want enough PCIe lanes for highspeed NICs and maybe a second zpool down the road.
Americans don't pay enough for power and the TH-camrs you see have a lot of physical space
45drives a al barsha south drive tour
a specific use case is mentioned in this video - performant redundant configuration of nextcloud with mention of k8s as a possibility. please excuse my n00bness here, but what would be a reasonable solution for this. As Wendell stated nextcloud wasn't really designed for multisystem scailability, and I was curious what level1 or others in the community have done to make something like this work well... I'm JUUUST starting to learn about scale-out solutions (just recently built my own 5-node proxmox cluster and would love to use it for homeprod and homelab stuff for learning and moving my webapps home for self-hosting)
Cluster of Storinator JR lol yes please
What was that little sata board in 10:55?
I modified a 4U shelve for ATX motherboard setup this year and now almost regret it. (-:
Thank you for wondering autofocus.
Right after talking about the Storinator Jr, at the 16:41 mark in the video, a machine is shown with over 15 internal hard drives in a computer case...what product / machine is this??
That's going to be the homelab product
Speaking of RAID, can you do a VROC update running 4 to 8 NVMe (u.2 or u.3) in a RAID 10 or RAID 5 please??? 🙏
I'm curious what work of theirs is used by cloud computing vendors (like AWS, GCP, DO, etc)
The Junior 🩶
Cool ..tech ..travel channel
what is that microphone brand and model?
I wonder if the HL15 will be in the budget of home users. I have a feeling it won't be given the price of their other servers.
$800 for the case with backplane...
Whoa whoa whoa slow your roll. You guys know about Cincinnati chilli. Say what now?!
why not run deduplication tables in a sql server instead of a giant hash table??
or better yet, a recursive dns tree that can be queried without having to run mass comparisons??
And what is the Cockpit licensing issue on non-45drive hardware?
uploaded 56 seconds ago but comments 6 days old? youtube are you ok?
It was uploaded a week ago for patreons and made visible for the masses 5 minutes ago i recon.
Would like to see some more affordable homelab systems. I get jealous of the bigger creators, lol. Network Chucks drop yesterday was nuts.
I am drooling over thay $800.00 USDish, AV15 derivative for HomeLab users, this is actually what I am looking for RIGHT NOW, lol. Any idea on a release time frame? 15:45
Is there any way to buy just the AV15 chassis???
Soon (tm)
Hello there. The AV15 chassis is currently available, but, since it is spec'ed at the enterprise level still (e.g. sliding rails and PSU), the cost is just under $2100 US. Soon, as Wendell mentioned, we will release a 15 bay chassis spec'ed for a HomeLab, and the chassis and backplanes alone come in at around $800. There will be an option to add a PSU, or get a fully built system without drives too.
Macphee? Mccafee? 🧐
Risc v home lab
Wendell looks like he got hammered
Four flights is a hammering
Jet leg is real. Flying takes a toll on your health
I just put together a 5-node proxmox cluster and each node has a 2tb pcie4 ssd for ceph with ec and I find this configuration rather slow for vm hosting. I was wondering if there were any good techniques to improve performance (current network configuration is each node has a dual 10g card in a lag to a 10g switch, ceph is on it's own vlan), each node has 64gb of ram, i7-8700. I went with ceph for HA and redundancy but so far, i've been pretty disappointed with write performance (which is down to about 300MB/s vs. 3000MB/s for the raw underlying ssd's). what do?
Are you running consumer NVMe? Samsung pros or something similar? If so, there’s your answer. Otherwise feel free to hop on the Ceph subreddit and ask. My name is mitch hall I’m the chief architect at 45Drives… I regularly help out on the Ceph sun free of charge in my spare time.
Also, erasure coding in general is not advised for any latency sensitive workload like virtual machines. It is best served for large sequential streaming reads and write.
@@mitcHELLOworld I went with mid-range consumer stuff (Crucial P3 Plus) because these boards are only pcie3, and I benchmarked them individually before I put the cluster to together and they reach the 3000mb/s throughput I expected and figured it would be fine for this application. I guess I was mistaken?
@@mitcHELLOworld I guess i'll create a new non-ec crush map, get things moved over and re-test, thanks!
@@joshhardin666 unfortunately those p3 drives dont have PLP. You can benchmark them outside of Ceph but they will perform terrble in Ceph. Ceph uses fsync calls on all its writes which completely destroys performance on non-enterprise grade nvmes. To benchmark the true performance of your nvmes for Ceph you can use this fio test : fio --name=/dev/sdX --ioengine=libaio --direct=1 --fsync=1 --readwrite=randwrite --blocksize=4k --runtime=60
Be sure to do this on a blank nvme as if you do it on a disk that’s already an OSD it will corrupt it.
as usual with Wendel - 🤯🤯🤩🤩
yeee 45drives lets make a rack case with space for bunch of drives and sale it for 5000$ .. prove me wrong,,
and heeeeeeeeeeeeeeeeess jeeefffffffff
Very interesting. Although I am getting quite distracted by the camera focusing issues. The close in focus and then sudden blur is very off putting.
Sydney ???
LoL....
Because you need 10 boxes to start a ceph cluster. Thats overkill for homr user
absolutley not good sir! 3 nodes is more than enough to get the full Ceph experience - learning, getting the high availability and the redundancy, and for a homelab if its built right you will be able to get more than enough performance.
For a homelab if you simply want to learn Ceph and get more familiar with it a single node is more than enough, or you can even virtualize 3 nodes on a single node as well for learning.
Hellooo my name Jeff
I... I... Me...Me... Myself... I... Irene... This is where I draw the line. Wendell Officially dropped the ball and was blinded by the lights and no, I can't sleep until I feel my endless dawns non-responsive to my panicking requestes for support getting blinded by the f**ks. in the night.
10k iops on ceph. That is pathetic. We are doing routinely doing 500k iops on our small cluster with 10 nodes (about 1.2PB raw capacity), each with 14x 14TB hdd drives, and 2 NVMe drives under combined with OpenCAS as a fast transient layer (especially for writes and frequent reads), plus metadata and wal on NVMe too separately. We are mostly limited by our 2x10Gb NIC, than by ceph. Ceph is great. I do not agree that Ceph got more streamlined in recent release. There are improvements, some things that required tuning, now work automatically. But overall it is not a big difference. If you are serious about ceph you need to use custom provisioning and config tools, because ceph orch (bundled method), is really limited and inflexible in key areas. Other than that ceph is great. Not perfect, (cephfs data corruptions still can happen for example, and radosgw scales very badly to big S3 buckets as another example), but it is getting better. There some other features (like multitenancy, io priotization, offsite replication) which we do not use, but they are interesting.
We do lot of data ingestion non-stop, lot of archival data, lot of transient data, lots of containers, log files, databases, analytical workloads, etc. And it works. Not HPC setup, but we do care about availability, because big clusters, drives and machines will die. More often than you think.
And we were doing this for like 5 or 6 years on ceph.
Notably, 10K random IOPs on ceph can be quite challenging on a single thread. Obviously, any sufficiently large cluster will be able to soar well past that number by increasing incoming clients requesting threads.
There is, however, a hard limit on cephs possible performance in a given thread. In general, this is a combination of network latency and drive commit time.
Ceph has definitely gotten more streamlined over time, especially with the addition of cephadm and dashboard improvements.
Easy there big fella with the pathetic talk haha. The videos name is just can we achieve 10k iops because someone posted the question is it possible to achieve 10k iops in Ceph - and ourselves and Clyso got involved to answer the question and use it as a chance to inform and educate people.
If you actually watch the video, you’ll see that we were able to actually achieve about 100k read IOPS / 40K write IOPS simultaneously with 4 threads with a 4 node cluster.
We’ve made lots of content on openCAS as well - we’ve actually even made one with an Intel dev on the subject.
There have been some really good changes made from Octopus to reef and performance improvements as well. My colleague and I were at Cephalocon in Amsterdam a few months ago and there are some really amazing things on the horizon - things that will go great with our new NVMe UBM U.3 Stornado ;)
I'm not sure if I agree with you about "More of the thing we should be doing." vis-a-vi putting manufacturing out in the middle of nowhere.
The USUAL reason for doing this, is tax breaks. Which SHOULD. NOT. BE. A. THING.
Neighbors should not be competing with each-other to low-ball getting (needed) tax income.[1]
Putting manufacturing out in the middle of nowhere increases fuel usage for zero benefits to anyone other than share-holders/executives (Who we should NOT be catering to.)
If there are OTHER reasons to move somewhere? Sure!
If there is cheap/clean(ish) light-rail access? Great!
Does the project massively increase residential/light-commercial need and create a Boom Town? Nope! No no no.
[1] Taxes are necessary and serve a purpose.
Roads benefit everyone, even if you don't drive. All your goods sure do. As does your fire/police/telecom/etc.
Schools benefit everyone, even if you have no kids. Educating children is necessary to keep society going.
Privatizing public works is a mistake.
The goal of a public water treatment plant is to treat waste water.
The goal of a private public treatment plant is to make (investors) money.
The goal of a publicly funded space program is...space stuff. For the people.
The goal of a privately funded space program is profit. "The people" no longer get much say. We don't "own" the results. And the main "customers" are gov anyway so you are still paying for it with taxes. Is it cheaper to develop because you can cut corners now? Sure. (Though it should be noted that a lot of the groundwork has already been laid by the public space programs. If they had to develop everything from scratch it might not be.) Is it cheaper in the long run? Not overall, no.
Hey, don't disagree with anything you said. Just wanted you to know that 45Drives and its sister company were founded in Cape Breton by owners from the area, in part because they saw the economic decline as coal and then steel left the area, and they wanted to create well paid manufacturing and tech jobs. That mission is just as true today. We definitely aren't some tax break line on a bigger balance sheet!
@@darryllgillard9237 Oh I get it. There will always be edge cases and extenuating circumstances.
I was mainly commenting on the way that Wendel framed it.
"...they found a way to BRING high-industry here...sleepy town...This is exactly the kind of entrepreneurialism and...high-manufacturing kind of thing that we should see more of in North America." Emphasis-mine.
I strongly disagree with "bringing" industry anywhere.
If it starts there, or if it is (responsibly) replacing a dying local industry? Sure! Go for it!
As long as it isn't being done as a tax write off or something even worse like a state giving exceptions to environmental regulations it if fine. (And yes, there literally are states that will try to bring in business by having more lax enviro regulations...)
P.S. I wish I could actually afford your stuff but I'm on a grad-student budget.
I'm especially excited about the NVMe stuff.
It would be amazing to see an implementation that bifurcated down to 2 lane links. (or even single lanes)
Consumer NVMe drives (and NAND flash as a whole) are getting so cheap that it would be really nice for us HomeLab people to have an easy way to connect up a bunch of drives.
A PCIe 3.0 x2 link is "only" 2GB/s but that is still fantastic for a single drive in a flash-based NAS. Overkill even.
Sure, it can be done with PCIe switching, or with enterprise hardware using retimers, but that inflates budgets astronomically.
I'm imagining a 2U chassis designed like your 4U ones.
24/48/72 M.2 slots mounted vertically in slots like your HDD systems do now.
An 8/16 core EPYC could easily handle that many PCIe lanes and still have some to spare for a 100/200Gbit NIC.
Think about it...
I so dearly wish they'd plunk an Alder Lake N300 into the Storinator Jr. I will be there day one if they do.