This storage cluster is WEIRD! (Mars 400)

แชร์
ฝัง
  • เผยแพร่เมื่อ 23 ธ.ค. 2024

ความคิดเห็น • 374

  • @KristanSmout
    @KristanSmout ปีที่แล้ว +402

    Jeff, ya dummy!

    • @zombie_pigdragon
      @zombie_pigdragon ปีที่แล้ว +69

      Nobody's gonna run one of these things in production!

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +75

      Haha not only topically relevant, but FIRST!

    • @YodaComedy
      @YodaComedy ปีที่แล้ว +3

      40th like

    • @pearcomputers
      @pearcomputers ปีที่แล้ว +1

      :D subbing, twice :)

    • @ubbgn
      @ubbgn ปีที่แล้ว

      @@JeffGeerling paying license fees? lmao

  • @superslammer
    @superslammer ปีที่แล้ว +368

    You lost me at licensing :) If this becomes a brick if you dont pay licnese fees, that becomes a problem. I remember Datto doing that to many customers. Their bare metal restore servers were literally bricks.

    • @jani140
      @jani140 ปีที่แล้ว +16

      Not different from cloud services and saas. I've been told, thats what people want theese days.

    • @markarca6360
      @markarca6360 ปีที่แล้ว

      Also NetApp storage, which requires licensing in order to use their hardware with protocols like NFS, iSCSI, and SMB/CIFS.

    • @DallanLoomis
      @DallanLoomis ปีที่แล้ว +2

      I use 3 datto servers are my proxmox cluster XD
      2 use standard asrock mobos, 1 uses a modified asrock mobo that took an asrock mobo bios chip

    • @PeterRichardsandYoureNot
      @PeterRichardsandYoureNot ปีที่แล้ว +17

      What world are you living in? Anything that is commercial and made for enterprise class is by subscription. It’s just the way they make money and keep the development cycle alive.

    • @korkenz1eher
      @korkenz1eher ปีที่แล้ว +29

      ​@@PeterRichardsandYoureNotthe development of ceph itself isn't funded by that subscription. the hardware itself is a bit meh (hot swapping disks is not possible) and the software itself is just a wrapper around ansible + cephadm.

  • @denzelx
    @denzelx ปีที่แล้ว +17

    I bought your ansible book in 2018 which, in effect, got me to my job where I'm now. There, I'm responsible for a giant ceph cluster with 1250 OSDs . Enjoy your time with ceph, it can be a scary software when something goes sour :D ....and also a big thank you for helping me become a cloud engineer!

    • @KimYoungUn69
      @KimYoungUn69 ปีที่แล้ว +3

      Send money not words

  • @johnlaurencepoole6408
    @johnlaurencepoole6408 ปีที่แล้ว +64

    So any use of this "appliance" requires payment of monies to Mars in terms of license fees, even after an initial subscription? If I am correct, then the forever pay aspect of cloud storage is an integral part of this solution. I think the concept of fully owning your own hardware inherent in a Raspberry Pi cluster must not be forgotten or lost.

  • @sbc_tinkerer
    @sbc_tinkerer ปีที่แล้ว +51

    Jeff, Pi clusters aren’t crazy. The people who build them are. I should know. I have two of them. Keep the posts coming and stay well!

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +13

      Takes one to know one lol

    • @jhvhest
      @jhvhest 10 หลายเดือนก่อน

      Only thing that is crazy are the prices. This hardware Is not in my budget anymore. It only becomes more expensive.

  • @mpuppet1975
    @mpuppet1975 ปีที่แล้ว +96

    The most complex raspberry pi setup doesn't exi.......
    This is above my pay grade.

  • @zippytechnologies
    @zippytechnologies ปีที่แล้ว +3

    Ceph is my fav for homelab - and now I have a small cluster at the office too! Finally some love.

  • @lukasb95
    @lukasb95 ปีที่แล้ว +4

    Wow. I've introduced to the Ambedded in 2018, back then when the Ambedded's founder husband was teaching me Openshift & Ceph. He told that her wife is starting a ARM-based ceph cluster in a box. At that time, i believe it is using Cortex A7 or A53, but with almost the same rack chassis and 8 node config.
    in 2019, I've met her on the Taiwan Computex showcasing the Mars 400

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      This chassis is that! And it's pretty neat! Would love to see them build a new 1U clustered model with more speeed.

  • @robertmoore3058
    @robertmoore3058 ปีที่แล้ว +21

    As a "very large utility company" we had a 24 pi cluster doing predictive math for peak shaving.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      Shame Sony took that away from the Playstation. Oh well their loss.

    • @Roxor128
      @Roxor128 ปีที่แล้ว

      Could the work have been done on a GPU using OpenCL, or was the job something branch-heavy, necessitating the use of the cluster?

  • @ScottPlude
    @ScottPlude ปีที่แล้ว +11

    favorite quote today,"....but my home network is only 10GB..."

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +4

      Hehe... someday I'll upgrade to 40 or 100 Gbps...

    • @TheOfficialOriginalChad
      @TheOfficialOriginalChad 5 หลายเดือนก่อน +1

      To be fair, 10Gb switches can be picked up for under $100 nowadays, NICs for $50. It’s super accessible if you need it to be.

    • @ScottPlude
      @ScottPlude 5 หลายเดือนก่อน +1

      @@TheOfficialOriginalChad My wife (sorry, I spelled BOSS wrong) uses the network for browsing on her phone. She sees no benefit to 10GB.
      I hope I can sell her and get enough to pay for a switch! HAHA

    • @TheOfficialOriginalChad
      @TheOfficialOriginalChad 5 หลายเดือนก่อน

      @@ScottPlude 😂

  • @happilicious
    @happilicious ปีที่แล้ว +28

    Love to see how much you progressed on your Ceph journey. Keep up the good work!

  • @HeieiX
    @HeieiX ปีที่แล้ว +7

    Funny story. A couple years ago I wanted a NAS but I wanted it open source. I was thinking of a TruNAS build until a Jeff on TH-cam mentions “ceph cluster”. The video of your build wasn’t even out yet but I was off to Google to learn about Ceph and before long I decided to build a 3 node Ceph cluster out of small PCs for my home lab. So far so good!

    • @jeremyjedynak
      @jeremyjedynak 8 หลายเดือนก่อน +1

      If you have a 4th node in the cluster, in the event that one node becomes unavailable -- intentional maintenance or unexpected node failure -- you can still operate in a cluster state (three nodes) that can absorb a single node failure.

  • @dfbess
    @dfbess ปีที่แล้ว +113

    I think your videos are very informative.. So I would never call you a dummy..just tease you about hoarding all the Pi's..lol

    • @Roy_1
      @Roy_1 ปีที่แล้ว +4

      How much of the total Rpi in existence does he have?
      What if it's like 10-15%?
      I realize that's probably not true, but it would be interesting to calculate.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +10

      @@Roy_1Heh, judging by the millions that are out there, it's less than .0000001% :)
      I'm not even in the triple-digits! (And still well below 50 total)

    • @Enixious99
      @Enixious99 ปีที่แล้ว +2

      ​​​@@JeffGeerlingFair I think a lot of us honestly think you have like 150 of them which in hindsight seems a bit unrealistic haha

    • @D3M3NT3Dstrang3r
      @D3M3NT3Dstrang3r ปีที่แล้ว +1

      Jeff takes the " You will never know till you try" And I love it. Instead of asking why he asks why not and tries it. That's my type of fun!

    • @D3M3NT3Dstrang3r
      @D3M3NT3Dstrang3r ปีที่แล้ว

      ​@@JeffGeerlingYou know you have hoarded all the cm4s you just wont tell us 😅

  • @KiraSlith
    @KiraSlith ปีที่แล้ว +15

    The Mars 400 really makes for the textbook example of why industrial ARM has taken so long to take off, and why Ampere has done so well in contrast. It has no resale value, the company behind it has no real reputation to speak of and they gave it a heavily used product name, it has no refurb or reapplication opportunity, and it's running on non-distribute closed blobs with minimal oversight. It's an expensive e-waste hulk to dispose of the moment Ambedded decides to drop support in any form or fashion and by extension a ticking time bomb for any business that decides to rely on it for mission-critical tasks, not to mention a security risk. As a CSE, I wouldn't want that thing anywhere near any network, cluster, or farm I'm responsible for.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว +4

      It's running on basically stock Ubuntu + Ceph. You have full access to the innards of their software. It happily runs several other stock Linux deployments if you want to reuse the box.
      Our next closest option at the time was over a quarter of a milion quid and ran proprietary software.

    • @Nephiaust
      @Nephiaust ปีที่แล้ว

      It could be the same for any other company...... Take Cisco and their $%$&%y HyperFlex product line; my last job just installed it ~3 years ago and Cisco announced this year that they are dropping it completely. And that product is so properity its not funny..... to the point where you cant do half the standard VMware management without using their backwards and for the most part broken services. Add to the point that unless you have an active support with them, you dont even get root access to the services on your own network (they recently changed it but *unless* you rebuilt or fresh installed, it wasnt easy to get the password).

  • @xXfzmusicXx
    @xXfzmusicXx ปีที่แล้ว +10

    Firstly I just want to say I love this. It is awesome to see more arm clusters. That being said, I think starting at $3,500 this feels a bit lacking. Especially the hard drive mounting seem super clunky. The other thing that kinda grinds my gears is why documentation is "liscense" based. Where you need to pay for support to get their documentation. Actual support I understand, but the manual and faq etc. seems a bit excessive.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +4

      Can't disagree here. Also note this machine was launched in 2019 I think. It's been a while and their newer boxes are a bit friendlier.

    • @penguinnh
      @penguinnh 10 หลายเดือนก่อน +1

      @@JeffGeerling Speaking of ARMs, has anyone noticed yet that you have three of them?

  • @KG4JYS
    @KG4JYS ปีที่แล้ว +16

    The lack of vibration dampening is a concern in any data center. Vibration can be a major issue when you fill out a rack if you're still using spinning disk media. However, it might not be significant if you only have a single 1u device with 8 drives. When you scale it up, that's when you're going to have problems.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +5

      It's a good reason to only buy Exos or other rated HDDs, they are built for high-vibration environments. And racks can (and should) have a little isolation... but they still have vibration induced by fans, other drives, etc.)

    • @xani666
      @xani666 ปีที่แล้ว +7

      No, that's a very small problem.
      Much, much bigger one is lack of hot-swap. ANYTHING down in that server and you are for server-out operation.
      So the entire benefit of having 1 OSD per server (not having to worry about having multiple OSDs down when server dies), is entirely moot here, as you need to take it out to do even simple disk swap.
      Like, you could risk it doing it when it is on but that needs those rails that allow cables to be still connected when rail is fully extended... but that's PITA to cable and have shit airflow
      Also having option of hot-swapping (even if it was a pair-at-a-time) would also solve vibration as those trays almost always have some elasticity in...

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      @@xani666 Nope, as long as your cables are long enough you can access the cabinet to change hard drives, M.2 drives or even an entire node without a power down.

    • @xani666
      @xani666 ปีที่แล้ว

      @@kevinthorpe8420 we had few servers mounted like that but it's just extra work to install that also blocks airflow, all to make servicing more difficult and time consuming. It's not worth to save few bucks, and looking at price of those you won't even be saving all that much.
      You can get say S5TH | D52T-1ULH for similar money but with drive shelf built into the server and 12 slots instead of 8 in 1U

    • @georgebrandon7696
      @georgebrandon7696 ปีที่แล้ว

      @@kevinthorpe8420 yeah. You can have cables long enough to slide it out. However, as mentioned, it's a PITA to cable manage, and it hinders airflow at the back of the rack. Congrats. You solved the storage access issue with even bigger issues. A solution like this is fine in a 1U chassis. IF you have an identical setup in another rack already to go to switch over when SHTF. Preferably three identical. This way you still have redundancy when one is down. Think Kubernetes. Think ETC clusters. Think anything HA.

  • @Scriven42
    @Scriven42 ปีที่แล้ว +2

    It's nice to know I'm not the only one that _still_ forgets sudo sometimes.
    It's been 30 years, I'll remember eventually! :D

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +2

      I think my command history is like 90% up-arrow, and 9% 'sudo !!'

    • @Scriven42
      @Scriven42 ปีที่แล้ว +1

      @@JeffGeerlinglolol! Command History is such a wonderful gift to the world. I can't remember how many times I just grep the history for that command I did 2 weeks ago!

  • @IndaloMan
    @IndaloMan ปีที่แล้ว +5

    I love watching Jeff videos as it reminds me of the challenges I faced when writing acceptance tests for 5ESS-PRX switches in Saudi Arabia 40 years ago. They ran UNIX RTR and if one of the two main processors failed, theoretically only one call was lost. Never managed to prove that!

  • @tomholroyd7519
    @tomholroyd7519 ปีที่แล้ว +5

    We opened up a vendor's box and there was an RPi 4 compute node in there, doing all the work

  • @ChrisHufnagel_Polymath
    @ChrisHufnagel_Polymath ปีที่แล้ว +7

    I ran Ceph on old Dell gear years ago. Very cool fs. 10gb network is required to get decent IO out of it.

  • @Darkk6969
    @Darkk6969 ปีที่แล้ว +1

    I've ran CEPH on my ProxMox 7 node cluster a few years ago. While it worked well but it had performance issues whenever it's re-balancing. From what I understand that CEPH recently has gotten alot better at that so may give it another try on my backup Proxmox cluster at work. CEPH is great for what it is but you really have to know it's in and outs to get the most out of it. Which is why companies like this offer support contracts.

  • @omegatotal
    @omegatotal ปีที่แล้ว +2

    Thats such a great little 10Gbit switch from Mikrotik!

  • @NicoDsSBCs
    @NicoDsSBCs ปีที่แล้ว +11

    That's sexy gear. I wish more people knew about these ARM solutions. I know people in IT maintaining tons of servers and they have no clue what an ARM SoC is. Many of their tasks could be offloaded by low-end to mid-range ARM devices consuming about 10 x less. But changing somebody their views takes time. This video can help for sure. 👍

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +5

      And this server is actually an older model-the newer ones are much faster, and even more power-efficient.

    • @sazma
      @sazma ปีที่แล้ว +4

      Is it though? No SAS. Very poorly designed drive bays. License(s?). More drive bays than you can use. And 5 rusty spinners are going to give you maybe 800MB/s sequential writes. Doesn't seem very sexy for a base price of $3500 plus licensing.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      @@sazma Why SAS? Each node has internal Flash, an M.2 and possibly an HDD or SSD. The aggregation is at the network layer. It's a textbook Ceph deployment with lots of small servers handling a smallish amount of storage each. The only major issue of this is slow rebalancing times. You can load 8 HDDs on them depending on the role of the specific node. Since this is a single chassis I would guess that the three missing HDDs are the bays for the management nodes. If you have the recommended minimum three chassis then you'll have one HDD missing per chassis.
      I agree it's not full enterprise grade kit. But it fills a niche that a few years ago nobody else supported.

    • @sazma
      @sazma ปีที่แล้ว

      @@kevinthorpe8420 Because it's Enterprise price. Perhaps you're right about 8 hdds, but from what I understood from the video, 3 supervisor/mon/whatever are suggested, so that seems like a possibility rather than a likely scenario. It's a fun tech demo, for sure, but it's definitely WAY overpriced and FAR better solutions can be had for FAR cheaper.

    • @ramdynebix
      @ramdynebix ปีที่แล้ว

      @@sazmaClaiming it can be had cheaper/better somewhere else without proof is just hot air moving.

  • @kevinthorpe8420
    @kevinthorpe8420 ปีที่แล้ว

    We've been running those for a few years. They do what it says on the box. At the time we started this journey there was nothing in the price range to touch them, or space (8 nodes in 1U), or power ~100W per chassis. We've started having issues with hardware failures now they're older and we've overloaded them so we've had to move some of the ceph nodes (index nodes, not storage) off this hardware, but it still works. Our biggest problem by far is that support in a UK time slot is not easy to manage. If they had a support centre in Europe then I think we'd be much happier.

    • @georgebrandon7696
      @georgebrandon7696 ปีที่แล้ว

      "At the time we started this journey there was nothing in the price range to touch them" And that right there is why some people don't understand the price isn't actually expensive. Just the power draw alone pays for itself within a year compared to running multiple Ampere platforms. (Which weren't really available at the time these things were first a thing.)
      I swear those who complain about prices have never actually worked in a medium to large business to know what those budgeting reports look like. I see this same thing with professional camera equipment from hobbyist users. Complaints about how a camera body costs $6000 or a lens costs $10,000.
      A few grand for a piece of equipment is pennies on the dollar when compared to wages at home. Not to mention that money spent makes money. And many (most) companies don't have shoestring budgets. They actually have profit margins worth a darn.

  • @Soraviel
    @Soraviel ปีที่แล้ว +2

    Finally, i was waiting for something like this again

  • @judsonleach5248
    @judsonleach5248 ปีที่แล้ว +2

    How do I get extra arms, Jeff?!? - Reminds me of a "Rick & Morty" episode on "Gazorpazorp"!!!! LOL - Love You, Sir!

  • @TheRolfFR
    @TheRolfFR ปีที่แล้ว +1

    I don't even know what ceph is but this video is fascinating 0oO

  • @jonathanbuzzard1376
    @jonathanbuzzard1376 ปีที่แล้ว +29

    I would imagine the dual 10Gbps ports out the back are so you can connect them up with multi-chassis link aggregation to the upstream switches.

    • @jeremyjedynak
      @jeremyjedynak 8 หลายเดือนก่อน

      Also, network math within the device:
      8x nodes with 2x 2.5GbE ~= 2x 10GbE

  • @RonLaws
    @RonLaws ปีที่แล้ว +1

    Link aggregation is still worth it sometimes! even though the core of your network is only 10Gbe, having the storage linked to the switch at X2 (or higher) to the switch means that a single node on the network can only ever consume 50% (or less) of the total available throughput, it's good at least for multi user access. in your case though maybe not worth it, but just thought i'd mention it!

  • @socketwench
    @socketwench ปีที่แล้ว +5

    I really need to spin up an ARM powered Ceph cluster of my own when I'm finished with the Switchwire...

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +3

      Definitely! It's gotten a lot better-I remember a few years ago it was a struggle to get it running. Now it's almost on par with the x86 experience, and you fight with the normal things you fight with in all clusters, instead of incompatibility issues.

    • @socketwench
      @socketwench ปีที่แล้ว +2

      @@JeffGeerling Hopefully it won't generate enough heat to melt the 3D printed server rack it's in. I really should have used ABS instead of PLA for it...

    • @boneappletee6416
      @boneappletee6416 ปีที่แล้ว +1

      @@socketwench Sounds like it's time to prep another rack... ;)

  • @TheOfficialOriginalChad
    @TheOfficialOriginalChad 5 หลายเดือนก่อน

    7:30 Vibration dampening is NOT for “portability”, it’s to dissipate the forces from spinning disk drives. Multiple drives tend to resonate, and even though these forces are small, they decrease life of drives significantly.
    I’m guessing the dampeners were just missing, because I have never seen a disk caddy without dampening.

  • @WallaceReen
    @WallaceReen ปีที่แล้ว +1

    I maintain our companies ceph storage with >6TB and confused about the "disk per host-ratio" in this setup. There is a sweet spot with disks@hosts and sites. I would be scared to run and maintain this in production.

  • @FSK1138
    @FSK1138 ปีที่แล้ว +1

    this chanel has opened my eyes to low power computing
    i now have about 10 "compute devices" ... nothing abouve 65w
    i am going to pick up 2 pi 5 to replace 4 of those devices
    i hope one day to have everthing running of battery and solar

  • @joshk1058
    @joshk1058 ปีที่แล้ว +1

    you are living a week in the life of some of my testing! fun surprises sometimes.

  • @hcjkruse
    @hcjkruse ปีที่แล้ว +4

    Sillicon Graphics (SGI) Purple 😅. What do you use Ceph for?

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +4

      Kubernetes / distributed storage.
      Right now in my 'production' homelab I'm still running mdadm RAID and Samba. But I'm considering moving my main storage to Ceph at some point. Not sure if 'forever' but it would be nice to run a local bare-metal Ceph cluster in production. Most of my usage has been on managed instances, and they're less fun :) (and more expensive).

  • @domramsey
    @domramsey ปีที่แล้ว +5

    I understood about 4% of that. 4.5% tops. But it's a pretty box.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +4

      Honestly if there's one takeaway, it's that manufacturers should have a go at different color schemes for the front of their boxes :D

    • @domramsey
      @domramsey ปีที่แล้ว

      @@JeffGeerling I have a Cobalt RaQ4 from the 90s that used to host all my websites. It died years ago, but it's such a cool looking blue 1U case with green lighting. I can't throw it away. I should really fill it with Pi's.

  • @jimcabezola3051
    @jimcabezola3051 ปีที่แล้ว

    You make me want to try this for myself! Now THAT'S awesome influencing! Mahalo for you videos!

  • @TheScrider
    @TheScrider ปีที่แล้ว +2

    That is the reason is pretty hard to find one for our projects, he bought it all

  • @aatheus
    @aatheus ปีที่แล้ว

    This is a really cool storage appliance for a homelab! I didn't know that I needed one until you reviewed it
    And thanks for the free eBook!

  • @Allgone-b4k
    @Allgone-b4k ปีที่แล้ว

    Sick video Geff very nice lay out wonderful design

  • @minigpracing3068
    @minigpracing3068 ปีที่แล้ว +1

    Do they ship these with a list of installed MAC addresses so you can set IP reservations before plugging it in?
    How are you liking that CRS309? I have two of these, one for my lab (XCP-NG on used HP servers) and one for work's production XCP-NG (as top of rack on new Supermicro servers).

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      I didn't ask about pre-configuring, but the one they sent me was configured with a couple customizations out of the box, so it's probably an option to at least have them send you the MACs.
      I love the CRS309-I have three of them now, they're quite handy!

  • @moonbastic
    @moonbastic ปีที่แล้ว +1

    Nice video. Though I don't work with Pis, I find the whole cluster concept fascinating.

  • @PeterRichardsandYoureNot
    @PeterRichardsandYoureNot ปีที่แล้ว

    Who needs this? Raises his hand …slowly at first, and then yells “I DO!”

  • @sandmanxo
    @sandmanxo ปีที่แล้ว +1

    While I have no good use for this I can see the value in a pi cluster as a learning tool for those still using on prem hardware. My place of employment went to the cloud a few years ago though so we can pay a lot more for the same server specs as we had in our still running data center. Government work at its finest lol

  • @louwrentius
    @louwrentius ปีที่แล้ว +1

    ☺️😌 he showed my blog diagram about Ceph 👍🌷 I’m so honored

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      Thank you for publishing it!

  • @gokhansarioz7150
    @gokhansarioz7150 ปีที่แล้ว

    Hey man, the reason you are seeing 55k on FIO under Linux terminal is Linux BuffMem cache, i recommend you to remove your test file each time you run the test that way its not able to cache inside of your memory. Awesome video.

  • @OneHappyCrazyPerson
    @OneHappyCrazyPerson ปีที่แล้ว +3

    I think this is the bare minimum for Klipper !

  • @EmmanuelGoldsteinUK
    @EmmanuelGoldsteinUK ปีที่แล้ว

    @5:48 did Jeff suggest that keeping a larger number cool is 'Orwell and good'?

  • @Rick-vm8bl
    @Rick-vm8bl ปีที่แล้ว

    Love the concept, less excited in the licensing aspect of it but appreciate that I as a lowly home labber am not the audience for this product.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      Buy is and don't licence it then. It won't brick. You can still install stock Linux. You just don't get support.

  • @RonJohn63
    @RonJohn63 ปีที่แล้ว

    4:54 Kids these days with their "full-sized" 3.5" drives. Back in my day, we only had 5.25" drives in 3.25" high by *8 INCHES DEEP* bays... and we liked it. 5MB should be more than enough for anyone!!

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      Meh, back In my day a 160MB drive was the size of a washing machine drum. Which was the style at the time. (ICL 2900 series)

  • @itssoaztek4592
    @itssoaztek4592 ปีที่แล้ว

    It's amazing how well done your videos are, I always learn something. And it's just so much fun to watch, not the least because you have a great talent in sharing knowledge.

  • @sysadmin-info
    @sysadmin-info ปีที่แล้ว +1

    I will have to look for the previous video where you presented this six CM 4 compute module mother board. Looks interesting. I would like to learn Ceph just for fun and curiosity. Anyway this 1 U server looks promising, but for the company the version with hot swap drives that you can easily change on the fly is much more convenient. Of course it depends and this is just my private opinion. Thanks for sharing.

  • @TheJimmyCartel
    @TheJimmyCartel ปีที่แล้ว +1

    Hey jeff, i just picked up an poweredge r815 server with 4 opteron 6378 cpus in it totalling 64 cores.
    Any fun projects ideas i can play with?

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      Proxmox would be what I'd throw on there. Figure out how to wrap as many of your homelab services into it (then don't forget to back up everything too!)

    • @TheJimmyCartel
      @TheJimmyCartel 11 หลายเดือนก่อน

      ​​@@JeffGeerling hey coming back weeks later to let you know proxmox has been GREAT!
      trying to run an OS natively and running a task across cores on more then one CPU had some serious latency bottle necks I couldn't believe! Proxmox has been a great learning experience, and is a great use case for a multi socketed system. THANK YOU!
      Gonna make an optiplex cluster next haha
      (PS you should show more people proxmox)

  • @sourcilavise3788
    @sourcilavise3788 ปีที่แล้ว

    Hello Jeff, I'm currently training for the RHCE exam with all the ansible stuff. I was just thinking because you're mentioned in the book wrote by Sander van Vugt for the exam prep. Do you know him or it just happened that he mentioned you as author for ansible roles (not directly but as geerlingguy) ?

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      I've communicated with him before, but I don't personally know him-I think there are probably more than 50% of my Ansible-related contacts who I've never actually met in person :(

  • @GreenCinco12Official
    @GreenCinco12Official ปีที่แล้ว

    Is it the camera or do the m.2 ssd look bend af?
    good to see at 2:50

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      They have a bit of a stance to them. They're fine at that angle, I've seen much worse! Would be nice to have them perfectly flat though.

  • @gurshejsingh2016
    @gurshejsingh2016 ปีที่แล้ว

    Hello jeff, can you explain how to check the computational capacity / calculation per second of a computer / server. Is this good benchmark to select better system for application or web server.

  • @jdl3408
    @jdl3408 ปีที่แล้ว +1

    What was the sequential read and write performance?

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +2

      See later in the video-about 200 MB/sec.

    • @jdl3408
      @jdl3408 ปีที่แล้ว +2

      Ok, I thought that was still random. Would have expected closer to 500MB/s sequential reads out of 5 SATA drives.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      @@jdl3408 - Yeah, in a typical RAID on a single system, you could probably get 250-300 MB/sec with 5 drives, but this is running a kind of 'raid over ethernet', accounting for data integrity and all that... so it has a bit more overhead built in.

  • @olivier2553
    @olivier2553 ปีที่แล้ว

    I like the sound of the fans with a picture of still fans...

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      Those are actually solid 'ducts' on the back of those server fans (they're deep!). The front has the actual spinning fan blades, then at the back I believe the ducts help create a more turbulent flow to make sure the air back there mixes (or it might be something to help with static pressure... or both!). But the best thing is if you touch that your fingers don't get chopped off. Not quite the same on the other side of those fans!

  • @janjansen6263
    @janjansen6263 ปีที่แล้ว

    Does ceph + (offsite backup or tape) count as 3-2-1 backup? There are more than 2 copies on ceph and the offsite backup is a different storage media

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว +1

      Ceph clusters can be multi-location and you can define the crush map to keep copies in both locations. However that can slow things down if the network link is slow. There are also replication strategies you can use for DR backups or archiving.

  • @Felix-ve9hs
    @Felix-ve9hs ปีที่แล้ว +1

    Does this cluster also support IPv6?

    • @jdl3408
      @jdl3408 ปีที่แล้ว +1

      I’d like to know this as well. This should all be IPv6 single stack.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      That's... a good question and something I didn't test.

  • @jacekruzyczka3058
    @jacekruzyczka3058 ปีที่แล้ว

    Nice appliance, Jeff! Do I understand it the right way: Ceph is a storage area networking (SAN) solution implemented completely in software?!

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      Indeed it is. The software on the MARS400 runs basically stock Ubuntu and Ceph. You can build your own cluster if you have a pile of servers kicking around but that's a pile of servers if you need the resilience, then rack space and, power.

  • @AWIRE_onpc
    @AWIRE_onpc ปีที่แล้ว +1

    I would really like to see you run ceph on a 45drives server. It would be cool to see over 2 pb of data storage linked together.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      For now my HL15 and XL60 servers are going to get deployed a bit more traditionally. But follow Network Chuck - he has a set of 45Drives servers he's working on building into a massive Ceph cluster!

  • @StormWarningMom
    @StormWarningMom ปีที่แล้ว

    The Ansible part of the video was your "Squirrel!" moment 😂
    Also, I understood maybe 25% of the whole thing, maybe because I don't know much about networking. Regardless, it was a fascinating video. 👍

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      Haha I know; I was setting the thing up and was (pleasantly) surprised to see ansible output right in the UI they built!

  • @uweburger
    @uweburger 10 หลายเดือนก่อน

    How do you upgrade this to a newer ceph version, what linux is running on these nodes, does it run a vanilla kernel?

  • @terrorpup
    @terrorpup ปีที่แล้ว

    I feel stupid, I thought you said the vpn software was called timegate, I am glad you posted the url here, because I did look for timegate, was like how is a time sheep software vpn software. Thanks Jeff

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      Heh, you could travel through time with timegate!

  • @pcislocked
    @pcislocked ปีที่แล้ว

    7:18 then what's the point of having 8 bays? I feel like I'm missing something...

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว +1

      I'm guessing the three nodes without drives are the management nodes. If you have more chassis then those would be spread over several chassis and you'd use those nodes as OSD nodes with a HDD. You'll be missing three HDD in an entire ceph cluster. It's just that Jeff has the whole cluster on one chassis.

  • @boneappletee6416
    @boneappletee6416 ปีที่แล้ว

    I've unfortunately not yet had the pleasure of running a CEPH cluster... but I'd definitely love to set one up!:D

  • @sidewinder7602
    @sidewinder7602 ปีที่แล้ว

    Twingate doesn't inherently offer enhanced security by default. If you gain console access, such as through SSH for debugging purposes, you can easily move to other services and machines within the same Layer 2 network segments, bypassing TwinGate's access control. In contrast, I prefer Zerotier, which also supports ingress and egress filtering on a per-network basis, though it's not stateful, but it's free and can be self-hosted. NOW ... I'm getting myself a MARS400 for the office - definitely some good stuff! 😉

  • @janmonson
    @janmonson ปีที่แล้ว

    Just running 4 of my Dell R710 servers is killing me on power. My last utility bill was $600. And most of them are usually just at idle. I am afraid to power up my entire 20 server cluster. The monthly cost savings of just a few of these machines would pay for themselves in short order. The only concern would be running my own software and os. I don't want to be locked into a specific ecosystem. I will have to do some research, but it might be worth moving over to these.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      ~100W per cabinet. I think the Mars-200 was 100W and the MArs-400 is 105W.

    • @janmonson
      @janmonson ปีที่แล้ว

      @kevinthorpe8420 my R710s have 850 watt redundant power supplies. Even at idle, they are probably pulling close to 100 watts. And there are 20 of them. Even with just the 4 running and pulling 100 watts plus the 14000 BTU AC unit cooling them, it is kill us on the cost of electricity. Cooler, quieter, and cheaper, always looks better.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      @@janmonson Ours run more like 500W each. So you are clearly wasting a lot of computer doing sweet feck all.

    • @janmonson
      @janmonson ปีที่แล้ว

      @@kevinthorpe8420 Web, email, cloud, and plex. Have an R310 running a private Minecraft Java server as well.

  • @TerminalWorld
    @TerminalWorld ปีที่แล้ว +1

    Over 400 pages?
    Oh my Jeff...

  • @stevekirkham5193
    @stevekirkham5193 ปีที่แล้ว

    Liked 'cause the nice mention of Network Chuck! Already subbed.

  • @Saturn2888
    @Saturn2888 ปีที่แล้ว

    I couldn't switch to a regular PC PSU for the Storinator so far that I've seen. My Storinator is the 60-drive model and it seems to have larger power requirements.
    Also I run all SSDs in one of them and bought 8 hybrid converters to do 128 2.5" drive bays. But logistically, there are many issues.
    1. How to connect all the SAS ports without expanders?
    2. How to fit everything in the case with all those wires?
    3. There are only 6 connectors for drives, but I need 8.
    4. The PSUs only have 50A on +5V (250W) each whereas I need about 15A with some headroom per 16 drives.
    Doing all this, I found some 12V to 5V converters from Corsair to do 20A each (100W). With 6 of these, I'll be in he safe zone.
    Logistically this is a pain to setup, but I'll be doing it this weekend.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +1

      Ooh! If you do get through those challenges please ping me via email... I've been working on getting all the parts together for the same thing. I talked to the 45Drives folks about it a bit a couple months ago too.

  • @tsuzot8192
    @tsuzot8192 ปีที่แล้ว

    Is the "appliance" still usable after your year runs out, and you decide not to renew the license?

  • @autohmae
    @autohmae ปีที่แล้ว

    This is pretty good, the 4GB of memory per node worries me though. If it's just the storage node, maybe it's fine.

  • @annieworroll4373
    @annieworroll4373 ปีที่แล้ว +9

    Hmmm Twingate looks interesting. I've been considering setting up my Mac Mini to do that sort of thing, Twingate sounds like a much simpler solution than fighting with Apple bullshit from 2006, and at least the hardware side for it would be cheaper than buying basically anything x86.
    I'll need to come up with something to do with the ancient Mini though.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +6

      I used my 2011 Mini for a few years as a simple file server. Some of the Intel minis could run Linux and even do okay with some things like Jellyfin or Plex!

    • @annieworroll4373
      @annieworroll4373 ปีที่แล้ว +1

      @@JeffGeerling I should look into more options for it, though mine is a fair bit older than yours.
      I do know I can upgrade the CPU and storage, if only there were quad cores that would run in it, that would probably be better for any likely task I'd put to it since I'm not a vintage Mac software enthusiast where I'd run into lots of stuff that isn't multithreaded enough to benefit from the extra cores.

  • @lilbits3136
    @lilbits3136 ปีที่แล้ว

    Can you use some of these clusters to crack the latest TI calculator signing keys for us? I can't afford to build one, myself.

  • @HaydonRyan
    @HaydonRyan ปีที่แล้ว +1

    Would love to see perf tests on this with sata ssds

  • @AndersHass
    @AndersHass ปีที่แล้ว +1

    Looking forward to someone selling Raspberry pies with dedicated GPUs too lol

  • @xani666
    @xani666 ปีที่แล้ว

    One node per drive seems entirely excessive, especially for spinning rust. Fine for learning/testing, sure, but as you already want at least 3 nodes just the traditional 1U server with 8 OSDs each will be less fuss and probably even cheaper.
    No hotswapping is also concern for anything datacenter. And worst case might be having to turn out other 7 OSDs to replace one drive (if say your setup doesn't have cables long enough to take the server out while running

    • @another3997
      @another3997 ปีที่แล้ว

      Surely, when you start having multiple nodes per drive, you start running in to drive cache limitations, bus contention, latency and therefore an increased potential for data loss? If every drive served two or more nodes, sooner or later something will have to wait to be written or read. Faster performance and ultimates reliability are usually at odds with each other, so you choose the est combination for your needs.

    • @xani666
      @xani666 ปีที่แล้ว

      @@another3997 That is literally the reverse of what I said, I said "one server with 8 OSDs", as in server with 8 drives attached to it.
      One node per one disk is kinda wasteful. You're paying "OS tax" for each OSD, and 8 nodes, even cheap ARM ones, do cost more than one entry level intel box. Limiting amount of disks per server might make sense if you're running into performance issues (say a bunch of fast NVMe OSDs), but otherwise is wasteful.
      Like I said, it's nice if you have test cluster cos you can get a bunch of machines in small amount of rack space, but if you're testing you might want to use VMs anyway, far easier to make changes to that setup.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      Yeah, it would make a little more sense to have 2 bays per node but this was designed to hit a price point so it's all pretty standard. The having 8 nodes in 1U adds more resilience.
      And nearly everything IS hot-swap, even the nodes themselves can be shut down individually and swapped with the chassis still live. In any case the minimum recommended install is three chassis. It's really designed for full racks of these things.

  • @kgottsman
    @kgottsman ปีที่แล้ว +10

    Thanks Jeff. This video is informative and enjoyable. I know Ambedded is partnering for marketing purposes, but you don't make it a sales pitch. Great tech talk and honest reviews/testing. These kind of videos are why I have you in my notifications.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว +8

      Yep! They were kinda surprised I wanted to test their hardware. I mean, I would be too haha :D
      It's not like the common homelab or Pi user would buy enterprise storage appliances! But I thought it was a unique enough machine, and had some interesting parallels to the Pi clusters I build a lot, and they were game!
      They're a neat company, and I know the engineers there do solid work :)

  • @Karthig1987
    @Karthig1987 ปีที่แล้ว

    I don't know much at all about anything and I enjoy your videos a lot. Never touched a raspberry pi even once lol.
    I love purple.

  • @philippesimon2756
    @philippesimon2756 ปีที่แล้ว

    Love your videos Jeff. Lol! Calls himself a dummy while successfully setting up a Pi cluster server. Me: watches videos on YT with a Pi 400. P.S.: I'm getting 1080p playback without a hitch on YT now. Perhaps the latest updates updated the Mali driver?

  • @jnelson4765
    @jnelson4765 ปีที่แล้ว

    Perfect thing to run a homebrew VTL on.

  • @Ollital
    @Ollital ปีที่แล้ว

    "You're crazy"
    "No poor people are crazy, Jack. I'm eccentric."
    (Speed)

  • @herauthon
    @herauthon ปีที่แล้ว

    if RPi evolution is ongoing - what does it mean for a multi-pi server ?

  • @jackbennett2760
    @jackbennett2760 ปีที่แล้ว +1

    hmm buy 3 seems like a cop out for HA advice. If I get 3, why not get 5, don't bother with the dual power supplies & networking and make it cheaper and simpler. Dual top-of-the-rack switches and UPS would be a better use of that, especially if the dual supplies aren't on separate powers trips and RCDs, How HA is HA would be an interesting series cobbling together all the "hobbyist" gear to beat the enterprise. I'm curious how the air gets through that thing when the HDD backplane looks like the full height.

    • @YeOldeTraveller
      @YeOldeTraveller ปีที่แล้ว

      Nice thing about Ceph is that you can define redundancy policy to be node, rack, site, etc. aware. All the redundancy makes the whole environment more resilient. You get improved throughput as a nice side-effect. Also, one starts with what one can, and improves things as you go.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      Space and power. The nearest option we had to three of these at ~300W was 7U and over a kW

  • @goldark3
    @goldark3 ปีที่แล้ว

    put soft foam in the gap to avoid hitting the hard disk.

  • @rfitzgerald2004
    @rfitzgerald2004 ปีที่แล้ว

    Great video Jeff and looks like some interesting hardware. Who do you think would be the target audience for an machine like this? I'm not familiar with Ceph myself so don't know of it's benefits, but looking at your speeds I can't help feeling that a regular server would outperform in transfer speeds and come at a lower cost, although with higher power usage.

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      Generally if you needed some extremely reliable archive storage, you might consider a Mars 400 (or three!). For higher performance (like for active use, or for VMs in a cloud), you would want their faster machines (or other faster hardware, generally), so you could get lower latency and higher throughput.
      But for almost all home users, and for a lot of small business use cases, a typical NAS would be quite adequate.

  • @FlaxTheSeedOne
    @FlaxTheSeedOne 10 หลายเดือนก่อน

    Would love to see the documentation include IPv6, further, its 2024, can we switch to screwless designs for harddrives already?

  • @LockonKubi
    @LockonKubi ปีที่แล้ว +2

    A simple answer as to why do anything is "fun"

  • @alanjrobertson
    @alanjrobertson ปีที่แล้ว

    How does Twingate compare to Tailscale compare to Cloudflare? Maybe one for a comparison video?

  • @TalpaDK
    @TalpaDK ปีที่แล้ว

    Now they only need to become widely available cheap on the second hand market with good support for running Debian on the individual arm nodes

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      They used to run CentOS and now run Ubuntu. You're fine. Debian probably works out of the box. I know Ubuntu does.

  • @elwinroyale
    @elwinroyale ปีที่แล้ว

    So was it now 10 gigabit ethernet ports or 2.5 gigabit?

    • @JeffGeerling
      @JeffGeerling  ปีที่แล้ว

      The rear ports are 10G. The internal connections to all the nodes are 2.5G.

  • @aeleequis
    @aeleequis ปีที่แล้ว +5

    you are truly the Raspberry Pi Lord. Your videos are awesome.
    EDIT: i actually commented before watching the video, even if there are not actual Raspberry Pis, i still think you are awesome

  • @MarcoGPUtuber
    @MarcoGPUtuber ปีที่แล้ว +1

    0:06 well there's your problem, RSJ has taken over your account again!

  • @skylerparker8871
    @skylerparker8871 ปีที่แล้ว

    Twingate is the best!

  • @AB-Prince
    @AB-Prince ปีที่แล้ว

    I think computing companies should try the transputer archetecture again. the idea being that you have a proccessor with it's own onboard ram and switch to allow for any number of processors to work in parallel.

    • @another3997
      @another3997 ปีที่แล้ว

      The transputer was, and is, an interesting concept, however, it didn't succeed in getting a foothold. A modern cluster of networked nodes could in theory make use of that concept, making each node equal and responsible for sorting things out with it's neighbours, but I imagine there's a lot of latency involved in the process. Even if the architecture were scaled down to modern CPU die sizes, would they be any faster than current multicore, multiprocessor designs? Bus contention and latency are big issues in all high performance systems.

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      @@another3997 That's how the computers on the USS Enterprise work. So maybe.Edge computing is a big thing nowadays.

  • @alabamacajun7791
    @alabamacajun7791 ปีที่แล้ว

    I get why YT keeps sending me these videos of advertisements for products I would never buy. This is big money. 💵

  • @kstenders
    @kstenders ปีที่แล้ว

    i dont see the use for a centralized ceph cluster board. the point of ceph is that it is not central. If the power supply fails, or the power distribution board fails, all your ceph nodes are down at the same time..

    • @kevinthorpe8420
      @kevinthorpe8420 ปีที่แล้ว

      These are NOT designed to run solitaire. They are designed as bulk storage to fill whole racks or aisles. The fact that one single chassis can be a cluster is just a lovely by-product.

    • @kstenders
      @kstenders ปีที่แล้ว

      @@kevinthorpe8420 I can easily run 2x 4u jbod Chassis with 44 drives (hot swappable) or even more in storage pod form, with a single 1u storage head and some das raid Controller. You can then put this 4 times into a rack. So depending on your drive capacity you end up with something like 8-16PB of storage. With non hot swappable you can easily go above 20PB. Something like this is what you do if you need storage. Not those weird multiple cluster nodes in one Chassis Setups

  • @jleuthardt
    @jleuthardt ปีที่แล้ว +1

    thanks for causing the pi shortage