"please excuse the mess" _everything looks pristine, dust-free, brightly lit, neatly sorted into labeled boxes, and every cable is managed_ You tell this place is run by Germans, lol
if Hetzner engineer come visit my company data center here, they sure will get heart attack seeing all the dangling wires that we call spaghettis monster here. hah
Hey Roman, really enjoying the content you're posting on the EN channel. This video was super interesting, thanks for taking the time to work with Hetzner and letting us have a look at their operation. So cool!
@@der8auer-en I was wondering why all of your videos were no longer in both German and English, I thought maybe you gave up on making English videos and then I had the thought "what if he made a separate channel?" and I checked your channel info. I usually only browse on TV so I must have missed the message that you had a separate English channel. Was there a video where you announced this? If not , might want to make it more visible. Love your videos!
Hi Roman! I’m from Portugal and I have worked at Hetzner for 4 years in Falkenstein. Thanks for showing up the DC and all the found memories that I have from their and from the Hetzner team.
@D R They are doing hosting. Maybe you met a black sheep of their customers. If you have trouble with traffic from them contact their abuse department. They are very fast and professional and will sort it out. They don't like being abused either.
I've been a happy Hetzner customer for almost 10 years now. Have many many servers with them including some of the models shown in this video. I've had a few hard drives fail (as they always do) and Hetzner replaces them extremely quickly, I'm talking under 30 minutes. Really excellent service and my only desire is for them to open more datacenters in more countries. North America (Canada, USA etc) would be wonderful as-well as Asia. I love their service and just want more regions for redundancy.
I've had them for 7 months and were amazing, their choice of drives used also is good. (I got an 7K400 6TB when I asked, thank god not a seagate BS) Everything just worked, including when we asked for a hard reset THEY even did the reset, and ALSO "replaced server cpu fan as it was faulty", i never even knew it was, after they did it i saw pretty low temperatures too... really good support in general
Its not really good company for any serious bussiness, sometimes they shutdown entire company account and wipe all servers and basically tell you to go f. yourself. I know they do this alot, because they have cheap servers and some shady people are using their service, but still, they shutdown alot of people for no reason really, and with no way to defend or get back their data. Not very good place to run your bussines, also their anti-ddos sucks ass unfortunately. But for testing their servers are alright, but I would never run there anything critical.
@@eth_saver "sometimes they shutdown entire company account and wipe all servers and basically tell you to go f. yourself" you mean summerhosts? basically the hosts which resell servers from companies like Hetzner, mind you Hetzner is been in the business for QUITE some time now, I don't think they will ever do such a thing like one day notice to full company death. The Anti-ddos and stuff is true, their network isn't as good as competitors like OVH. I dunno about servers randomly shutting down with no reason, it HAD to violate TOS and then I can understand if it did, if you are respecting all the rules and not doing illegal BS I dont think it'd just shut down out of the blue I mean, the serious business is also just how it all works, you can just see the price and understand, it's not meant to have the best quality, hetzner is more like "quantity" over "quality" situation. I mean who uses consumer cpus in servers kek. serious business stuff should be placed in AWS or so, that's where they truly exist and are SUPER reliable
@@eth_saver we rent our servers at hetzner for over 3,5 years now. No problems at all. Good support, great hardware, awsome pricing, almost no downtime (i mean like maybe 4s/ 3 months or something like that for maintenance). The part of DDoS protection is true, but you can build decent protection yourself if you have the knowledge for it. I just wish that they would built a datacenter in Canada.
What an awesome facility. I love that they're using their heads for the cooling solutions instead of just spending a ton of money on, and wasting a lot of energy with, air conditioners -- even using the saw-tooth roof that some old metal foundries used to channel hot air up and outside via the angled ceiling to the high windows (I bet those high windows even face north or northeast, to avoid as much direct sunlight/heat as possible).
15:30 Most times the DC from the batteries goes to the UPS. 3 phase UPS uses "double conversion" and what that means is that they take the AC from the mains and then convert it to DC (and that is where the batteries get connected) and then again the signal gets converted from DC to AC. This gives them a big benefit in terms of mains failure - 0 transfer time between mains and battery back-up, and the second benefit is the fact that this technology eliminates every "defect" in the mains - higher voltage, lower voltage, DC offset, swells... The DC bus on the latest gen of UPS devices is around 400-500VDC in order to achieve better efficiency (around 97% efficiency). Some new data centers uses DC power to directly power the servers. This is done in order to gain efficiency by eliminating the DC to AC invertor in the UPS and the AC to DC conversion in the server PSU. APC has some great White Papers on this topic as well as on cooling (as in warm countries it is impossible to use the outside air to cool the servers).
@@1pcfred It depends. A lot of data centers have dual mains and they switch between them using automatic transfer switches that takes a few ms of time to switch (easy covered by the UPS). The diesel generator is a back-up of the back up most of the times. And the requirements for battery back-up is 5 minutes in order to be able to start the diesel generator and let it spin to the required rpm and it also needs to sync some times with the other diesel generators (usually they work in a N+1 redundancy configuration - if your data center needs 1MW, for example, you have 3*500kW diesel generators in order to be able to provide 1MW if one diesel generator fails for some reason). It takes around 1 minute but with diesel generators you never know.... they require a lot of maintenance and you never know what issue you can have with them.
I actually doubt they are using double conversion UPS'es in such a datacenter, because the efficiency is significantly lower than the line interactive/standby type. Normal good healthy SMPS power supplies have a capacitor bank big enough to handle the 5-20ms switching time of a standby UPS. As you are mentioning, many datacenters uses -48v DC systems as it is much much more efficient as you skip the conversion entirely. It would have been really nice if we were able to see how the batteries were wired, to get an idea of they are used. One thing they could be doing, (but i don't believe they are), is running ~230v DC directly to the power supplies, SMPS power supplies doesn't require AC, they run fine on DC, this way they would eliminate conversion AND be able to use standard PSUs. (only issue is that 230v DC is dangerous...)
@@HomelabExtreme another reason for 48V would be it is below the 'safe' limit of 50V (in most countries) so you dont need to treat it as 'live', use an electrician etc.
As satisfied customer of Hetzner it was really nice to see how they datacenters look like. I have been working as engineer/specialist for hosting company myself, so I have quite lot if information how datacenters are runned. I have been customer for Hetzner, for few years already, and I do have auctioned server and several cloud servers. Videos like this make those big hosting companies to feel more aproachable.
We have multiple virtual servers at Nurnberg but from the website Hetzner seemed like a small company, which it still is compared to AWS, but it's much bigger and profesional that I anticipated. I'm really impressed.
With these smaller companies, whats the point in buying their services compared to just running everything off AWS and Azure? My only experience with servers is running a minecraft server off google cloud tho. But im still confused how these smaller companies remain competitive yk?
@@honkhonk8009 Azure is beyond expensive, it can suck you dry out of your blood in no time. Hetzner is great, cloud machines are very powerful and cheap, ideal for small websites. We have dedicated hardware machine with Proxmox and it also works great. You have also unlimited internal network and with hardware machine even external, for the same price.
Very nice, been hosting my servers there for quite a few years now so it was nice to see how their setup looks. This also explains why my nvme drives are hitting 90c on regular basis.
This is the coolest server farm I have ever seen. I especially like the old school tower servers with desktop hardware! This is what I always wished to have what I was a teenager, haha. It's like a home lab on steroids! Love it.
Thanks for the most interesting tour. I am amazed by the size of this operation and the total amount of computing power contained here. I am from the old school of mainframe computers. First started working with computers back in 1965. One computer in an air conditioned room with a raised floor to circulate cold air. One CPU and 24K locations of memory in the typical system I first encountered.
Hetzner is awesome, I got a great deal on a box through their reverse auction. You can't get a box with specs that are even close in the states. Those guys rock! Their dashboard and load process is pretty slick too! These guys know what they're doing.
Well sounds like they're incorporating in the US, or having to charge VAT tax to US citizens now. While this doesn't mean the end of the world, in a price competitive industry, it definitely affects their edge they had.
As a long-time customer of Hetzner myself (and regardless of how old this video is as of writing) this has been a really exciting and interesting video. 👍
Hey der8auer. Just wanted to drop by and thank you for all the great videos, and especially a special one as a fieldtrip to a datacenter. Really cool. Hoping for more like this. Greetings from Sweden.
I'm studying for my CCNA 200-301 and Linux + currently , I have a fascination with servers and hope to be working in a data-center like this some day...thanks for showing this
I know almost nothing about data centers or various components apart from recently upgrading my current laptop to a newer one after 9 years and having to "learn" a bit about various components to understand what sort of various GPU CPU etc are in recent models. This video was really fascinating, well presented and insightful into how a good data center operates and all the various considerations with cooling, ventilation, power use and testing and building are involved. Thank you so much.
Videos and channels like these are doing great job at keeping TH-cam to be useful and educational platform. Hats off to all participants of this video. Great Work! || Tolle Arbeit! Greetings from Slovakia! 🇸🇰 ☮ 🇩🇪
This is great. It's interesting that Hetzner went with Asus to build their boards, I keep thinking maybe other brands aren't so bad but in the end I always stick with them. Though Supermicro is cool if you want to build your own server, I recommend it!
It is interesting to see where my old server was hosted, I never knew it was just a tower. Now I've moved onto their AX51 servers which if they're in that newer part of the building, they look very clean
I rented 400+ cloud VMs in Hetzner for many years and had no issues and I like the price and quality I get for this price. Thank you, Hetzner! Also rented a few Dell servers and SX292 servers, and I had no issues at all within about 8-10 years of rent history. But I heard stories about low server quality in Hetzner, maybe because I never rented low-price servers I didn't notice that. In any case great provider, thanks for this video! It would be very interesting to see more about Hetzner cloud structure, what is based on (hardware and software)...etc
This was quite ineresting. I'm a Sys Admin for ~25 years and there is always something to learn. Amazing that they custom build everything. What I would give to have that ability in my Data Centers.
Perfect video dude. I love these organisation of a data centre. I can remember the days I lived in Germany, how they care about thinks. In Germany, nothing is impossible, they didn't care what other say, they just made it right and precise. I'm missing those time really... Thank you too Hetzner for such of trust you give to this guy.
The main reason you wouldnt run 12V out to your servers is cable losses. You could, though, easily deign your PSUs to take AC or DC (most switchmodes immediately rectify it after all) saving you those inverter stage losses. Thanks heaps for the amazing insight (and to Hetzner too)
Yup, that's why it's usually 48V systems in data centers and then converted into 24V or 12V in racks. However I don't know if it would be 48V directly from batteries.
@@insoYT That makes a hell of a lot of sense! why 48V? the legal safe voltage for exposed installations is 50V at least in the US, you dont have to treat it as 'live'
Depends, if you were to design a system where batteries were separated "per server" in theory you could run everything off 12v but it would be an absolute mess and a lot of unnecessary thick cables.
Very cool to see somewhere like this. I can understand what they mean about trying to tidy things up with the HDD test racks because if you're using that kind of setup a lot (which they clearly are) then balancing hard drives and connecting cables like that is going to be frustrating (I worked on radar kit where we had a similar test setup and it was a real pain). A frame with a backplane would work better, a row of slots you can plug the drives into with LEDs at each HDD position.
Very nice tour, I bet a lot of the viewers have stuff hosted at Hetzner, including myself! I also like how their newest building is constructed using an IFA W50 - also some efficient use of old hardware :)
I would be interested in seeing a report on fans much like Backblaze does with drives. They have to have awesome data on these. Love the fact they use almost exclusively air cooling. So much to like about their approach to this.
With the standing HDD's being "unorganized" It looks nice and straight, but really easy to fall over, shocks you don't want to give HDD's while running.
Thanks for the intro, the compact servers with ATX motherboard that optimized for cooling is cool. And now I know why Hetzner's price is so much cheaper than other providers.
Well, he said they are already using green energy. I wonder if solar just isn't enough, or consistent enough (night time) vs. wind or hydro. It would be interesting to know why they aren't using some solar, though. I'd guess that it's too cloudy on average, that the investment into solar at that location just wouldn't pay off.
@@47CryXMA Maybe too much resources needed for solar panels? I guess that the consistency is also a big factor, the power output is changing with the second and that is bad for the power net in general. If they use thin film panels they could partly fix the consistency. Those panels start generating energy at very low light but the max power is also low.
Maybe they angled the roof away from the sun to keep things cooler? Edit: Actually based on google maps it looks like that's not what they were going for.
Man, building servers there would be my DREAM JOB! Do you know if they are recruiting? Even better would be them expanding into Ireland so I wouldn't have to move!
@@mrmotofy Not likely as hardware keeps evolving ;) First PC I've built I was 11 and trust me when I say that this has never gotten old for myself, quite the opposite, new hardware and/or technologies coming to market keep things interesting! Like when FLASH finally came of age on SSDs instead of USB sticks. There's always something new or improved around the corner and sometimes some deeply disruptive changes like the switch from AT to ATX that resulted in a considerable upgrade cycle although this may not be the best example.
@@wskinnyodden Yea...but building these servers is gonna be very repetitive as they seem to be virtually the same. So some might have a 100Gb Nic instead of 10Gb...big deal yes I remember AT vs ATX, I picked an ATX as it seemed the way it was going.
@@mrmotofy Well, that is just a challenge on how to get as many as possible going at the same time efficiently and correctly. Wouldn't be a first, I still recall the first time I had 10 systems on the work desk at once being installed and set up each for a different customer. (Christmas time for a computer shop back then)
Trabalhei no data center da Embratel nas olimpíadas do RJ irmão, foi meu último job depois que me mudei pra EUA, queria muito voltar pra área também seria um sonho tá num lugar desse de novo
Шикарно. Спасибо за подробное видео. У меня в подвале тоже микро дата-центр - 6 старых системников стоят на полках и работают серверами. Сегодня как раз собираюсь поставить сетевой фильтр с раздельными выключателями для удобного включения\выключения.
You really need to talk about this EN channel more on the main channel, I don't even know until I speculate that you won't upload no English videos and I found this in the channels tab.
I think I was talking about it in the last 3 or 4 english speaking videos and then just started linking below the German video. Hope the word eventually just spreads
Wow. I though datacenters or any kind of enterprises are running only xeons and epycs. I always wondered why intel, amd or nvidia artificially lock up their consumer hardware. I though that no one is willing to run consumer hardware for enterprise use so it seemed stupid to me that they bother to lock up certain enterprise features. This changed my perspective. And AMD or Intel must be quite mad seing 3900x being used instead of epycs which cost twice as much :D same for i7 instead of overpriced xeons. Very very interesting video.
@@PablumMcDump Thanks for the correction. I went to their website a see now that I can rent specific servers with consumer HW, I thought that they are using the ryzens and i9 for cloud. Still though I didn't know this existed.
The racks of your standard consumer grade desktops being utilized as servers just goes to show you don't need to get a Dell PowerEdge or something crazy to start your own server. So when people say they can't afford to buy a server, I'll just send them the timestamp 2:35
as a dutchman i got a question. ever been to the amsterdam data center? supposedly the biggest data (or busiest data center in europe some even say in the world). i have seen videos on it and read alot about it. and its very impressive. most game servers you use in germany are hosted on that data center. Blizzard and steam for example are located in amsterdam. its pretty interesting. anyway nice video very informative.
@@El_Marzocco we just got a new huge google data center in the north of the Netherlands in Groningen. Microsoft just build another one in the netherlands in Middenmeer its in the province of north holland. well they got green light to build it its not their yet but it will come. its in the province of north holland. But the reason why they do this is because we are a tax evasion country one of the biggest in the world. so its cheap for big companies to build their business here. they dont have to pay taxes and what not. something they had to do in any other country in europe. so easy choice to build it here. also we have the best fiberoptic infrastructure in the world. maybe that got something to do with it as well.
Is keeping these old equipments like i7 920 really environment friendly? I don't mean to judge that, just courious. I guess there is a comparation between recycling these low power efficiency device and keeping running on them.
I was surprised by that too. These are OLD 4 core CPU’s with ancient architectures and - relative to today’s standards - terrible IPC. They could probably consolidate an entire aisle of these ‘servers’ into a single rack of modern 64 core Milan servers and just run 4-core VM’s on them if that’s all the customer wants. And even if they want a more consumer oriented platform, modern Threadripper or even Ryzen (to a lesser degree) will provide similar levels of consolidation.
@@benjaminlynch9958 If you're not familiar with the way Hetzner works, you buy the entire machine as a baremetal/dedicated server. When you no longer want it, you can auction it off online via their marketplace. It's sort of in their business to not mess with bought dedicated servers and let them use them until they fail, even auction them off to other hetzner users. I'd imagine the cost of trying to migrate everyone off those old servers (Forcing downtime, trying to migrate disks, and trying to match the same low prices) just isn't worth it. It is kind of nice that you can buy one dedicated server from them, and as long as you keep paying, you can use it virtually forever. Of course, the downside of wasting energy and space.
You think manufacturing hardware has zero environmental impact? Think again. Making electronics is very dirty business. Including slave labor to mine the minerals. People are in bondage so you can have a few more FPS. How dare you!
@@1pcfred Didn't I mention the comparation between recycling and keeping? Old products are less energy efficient, and energy requires labour, too. Why not think carefully before judging someone?
@@1pcfred calm down, what do you get by telling that to the people, do you think i want to know such information when i buy a fucking ssd or a new cpu i need? no one said the world is perfect and the blame can't fall on customers
I would have imagined they just sold old hardware because it's just not power efficient. even if it's fully functional they can do the same thing with 15W that the 920 does with 130W
As long as you have the space and you're not paying for AC then power probably isn't that expensive to make it worth upgrading the whole lot. I'd guess they only retire hardware when it's just too slow to do anything useful.
Right now on their server market place - the oldest CPU is i7-2600 - if you get that server I guess they will let you run it for at least 2 years. So the i7-920 is just running because you do not want to piss of your customer - but the wattage is too high. However likely people using such an old server have that server idling most of the time - so it doesn't matter too much that is has a high TDP. Next likely those 95W 2600 are phased out. Lately the push with free setup on AX servers but not EX is clear as well - they must have figured overall power use is much less compared to same price EX (which likely has lower hardware cost). Not everyone maxes out the CPU of their servers. E.g. I have a AX-41NVME which is idling 99.9% of the time running just two websites. I cannot go virtual/cloud because I need 10TB hdd for downloads on those websites. Cannot find anything cheaper than hetzner. I run a second server to compile stuff, which needs 6TB hdd, and 1TB NVME space - it's only used 1/6 of the time. rest is full idle. Also when not compiling I need the hdd to store lots of data. Again I do not see a cheaper solution. Yes I could use one bigger server instead of two - and run one in virtual machine. But I fear reliabilty so websites needs it's own server. Likely many people are like me - only maxing out some certain aspect of their server - and not having it run 100% for much of the time - hence Hetzner can offer them soo cheap. I would love if Hetzner lowers prices and charges electricty directly to customer - but likely it's too complicated so they just keep it simple and lose money on customers constantly maxing out their server. If you maxed out your 920 you would already have upgraded. And on idle likely the power difference is not much - hence they can afford to keep them running. On the other hand - virtual servers/cloud will always be on very modern/energy efficient hardware.
They *are* still power efficient - given that most are idling or on low load. I think that's a sensible use of resources as they're using very little electricity. Also, junking perfectly good hardware in useful service is certainly unnecessarily wasteful. I applaud their approach with these older midi-towers.
I once did a blockchain project for a german, based on a 64 Core dedicated machine in Hetzner, had to do setup using KVM with booking kvm slot via support, very nice pricing if you are willing to build your own VPC with custom os and shared VMs..
@@oliver9089thats such blatant misinformatio . We in developing countries have started to switch our homes and businesses to solar, because the government is inept to do it themselves. We got our return on investment within three years after installing them.
@@askeladden450Не путайте Африку/Индию и ваши домашние цели, с промышленными комплексами Германии. Если у вас это вполне выгодно, то тут это может вовсе не окупится, говорю как житель России
Ich habe gerade das deutsche Video gesehen und einfach großen Respekt dafür, dass Du das ganze parallel auf Englisch gemacht hast anstatt subtitles oder eine kürzere Version mit voiceover.
"please excuse the mess" _everything looks pristine, dust-free, brightly lit, neatly sorted into labeled boxes, and every cable is managed_
You tell this place is run by Germans, lol
lmao ikr
if Hetzner engineer come visit my company data center here, they sure will get heart attack seeing all the dangling wires that we call spaghettis monster here. hah
Just don't check the basement...
Hetzner is useless company
german quality be like
Hey Roman, really enjoying the content you're posting on the EN channel. This video was super interesting, thanks for taking the time to work with Hetzner and letting us have a look at their operation. So cool!
Thank you!
@@der8auer-en I was wondering why all of your videos were no longer in both German and English, I thought maybe you gave up on making English videos and then I had the thought "what if he made a separate channel?" and I checked your channel info. I usually only browse on TV so I must have missed the message that you had a separate English channel. Was there a video where you announced this? If not , might want to make it more visible. Love your videos!
Hi Roman! I’m from Portugal and I have worked at Hetzner for 4 years in Falkenstein. Thanks for showing up the DC and all the found memories that I have from their and from the Hetzner team.
Ena pá, um tuga que trabalhou na Hetzner? Estamos a ficar muito conhecidos lá fora hehehe. Cumprimentos!
@D R They are doing hosting. Maybe you met a black sheep of their customers. If you have trouble with traffic from them contact their abuse department. They are very fast and professional and will sort it out.
They don't like being abused either.
It's not about you calm down
@@mansart26 nobody talked to you go away from me
I love all of the in house testing! What a cool company.
I've been a happy Hetzner customer for almost 10 years now. Have many many servers with them including some of the models shown in this video. I've had a few hard drives fail (as they always do) and Hetzner replaces them extremely quickly, I'm talking under 30 minutes.
Really excellent service and my only desire is for them to open more datacenters in more countries. North America (Canada, USA etc) would be wonderful as-well as Asia. I love their service and just want more regions for redundancy.
I've had them for 7 months and were amazing, their choice of drives used also is good. (I got an 7K400 6TB when I asked, thank god not a seagate BS)
Everything just worked, including when we asked for a hard reset THEY even did the reset, and ALSO "replaced server cpu fan as it was faulty", i never even knew it was, after they did it i saw pretty low temperatures too... really good support in general
ashburn coming soon broooo cyamonnnn
Its not really good company for any serious bussiness, sometimes they shutdown entire company account and wipe all servers and basically tell you to go f. yourself. I know they do this alot, because they have cheap servers and some shady people are using their service, but still, they shutdown alot of people for no reason really, and with no way to defend or get back their data. Not very good place to run your bussines, also their anti-ddos sucks ass unfortunately. But for testing their servers are alright, but I would never run there anything critical.
@@eth_saver
"sometimes they shutdown entire company account and wipe all servers and basically tell you to go f. yourself"
you mean summerhosts? basically the hosts which resell servers from companies like Hetzner, mind you Hetzner is been in the business for QUITE some time now, I don't think they will ever do such a thing like one day notice to full company death. The Anti-ddos and stuff is true, their network isn't as good as competitors like OVH. I dunno about servers randomly shutting down with no reason, it HAD to violate TOS and then I can understand if it did, if you are respecting all the rules and not doing illegal BS I dont think it'd just shut down out of the blue
I mean, the serious business is also just how it all works, you can just see the price and understand, it's not meant to have the best quality, hetzner is more like "quantity" over "quality" situation. I mean who uses consumer cpus in servers kek. serious business stuff should be placed in AWS or so, that's where they truly exist and are SUPER reliable
@@eth_saver we rent our servers at hetzner for over 3,5 years now. No problems at all. Good support, great hardware, awsome pricing, almost no downtime (i mean like maybe 4s/ 3 months or something like that for maintenance). The part of DDoS protection is true, but you can build decent protection yourself if you have the knowledge for it.
I just wish that they would built a datacenter in Canada.
What an awesome facility. I love that they're using their heads for the cooling solutions instead of just spending a ton of money on, and wasting a lot of energy with, air conditioners -- even using the saw-tooth roof that some old metal foundries used to channel hot air up and outside via the angled ceiling to the high windows (I bet those high windows even face north or northeast, to avoid as much direct sunlight/heat as possible).
Good point, then they could put solar on the south facing roof.
@@ArthurEmbleton солнечные панели здорово нагреваются
That is so cool. Awesome of Hetzner to give you such wide ranging access as well. Thank you for this :)
yeah kinda weird usually its off limits to any kind of filming
Oh shit, one of those might be my server box!
Same!
It's like when you get a child sponsorship and they send you the picture of the child.
Sadly mine are in Hetzner's Finland datacenter. I used to have servers in Falkenstein. So one of those could be my old server.
Oh yeah, I seen it 3rd row 4th down haha
Lol, what a surprise. New "Antics" video coming....?
15:30 Most times the DC from the batteries goes to the UPS. 3 phase UPS uses "double conversion" and what that means is that they take the AC from the mains and then convert it to DC (and that is where the batteries get connected) and then again the signal gets converted from DC to AC. This gives them a big benefit in terms of mains failure - 0 transfer time between mains and battery back-up, and the second benefit is the fact that this technology eliminates every "defect" in the mains - higher voltage, lower voltage, DC offset, swells... The DC bus on the latest gen of UPS devices is around 400-500VDC in order to achieve better efficiency (around 97% efficiency).
Some new data centers uses DC power to directly power the servers. This is done in order to gain efficiency by eliminating the DC to AC invertor in the UPS and the AC to DC conversion in the server PSU.
APC has some great White Papers on this topic as well as on cooling (as in warm countries it is impossible to use the outside air to cool the servers).
The batteries are just there to tide you over until the generators kick on. A few seconds.
Ah, nice! Thank you
@@1pcfred It depends. A lot of data centers have dual mains and they switch between them using automatic transfer switches that takes a few ms of time to switch (easy covered by the UPS). The diesel generator is a back-up of the back up most of the times. And the requirements for battery back-up is 5 minutes in order to be able to start the diesel generator and let it spin to the required rpm and it also needs to sync some times with the other diesel generators (usually they work in a N+1 redundancy configuration - if your data center needs 1MW, for example, you have 3*500kW diesel generators in order to be able to provide 1MW if one diesel generator fails for some reason). It takes around 1 minute but with diesel generators you never know.... they require a lot of maintenance and you never know what issue you can have with them.
I actually doubt they are using double conversion UPS'es in such a datacenter, because the efficiency is significantly lower than the line interactive/standby type.
Normal good healthy SMPS power supplies have a capacitor bank big enough to handle the 5-20ms switching time of a standby UPS.
As you are mentioning, many datacenters uses -48v DC systems as it is much much more efficient as you skip the conversion entirely.
It would have been really nice if we were able to see how the batteries were wired, to get an idea of they are used.
One thing they could be doing, (but i don't believe they are), is running ~230v DC directly to the power supplies, SMPS power supplies doesn't require AC, they run fine on DC, this way they would eliminate conversion AND be able to use standard PSUs.
(only issue is that 230v DC is dangerous...)
@@HomelabExtreme another reason for 48V would be it is below the 'safe' limit of 50V (in most countries) so you dont need to treat it as 'live', use an electrician etc.
As satisfied customer of Hetzner it was really nice to see how they datacenters look like. I have been working as engineer/specialist for hosting company myself, so I have quite lot if information how datacenters are runned. I have been customer for Hetzner, for few years already, and I do have auctioned server and several cloud servers. Videos like this make those big hosting companies to feel more aproachable.
We have multiple virtual servers at Nurnberg but from the website Hetzner seemed like a small company, which it still is compared to AWS, but it's much bigger and profesional that I anticipated.
I'm really impressed.
With these smaller companies, whats the point in buying their services compared to just running everything off AWS and Azure?
My only experience with servers is running a minecraft server off google cloud tho.
But im still confused how these smaller companies remain competitive yk?
@@honkhonk8009 Azure is beyond expensive, it can suck you dry out of your blood in no time.
Hetzner is great, cloud machines are very powerful and cheap, ideal for small websites.
We have dedicated hardware machine with Proxmox and it also works great.
You have also unlimited internal network and with hardware machine even external, for the same price.
@@keenmate9719 AWS has all that and more.
Very nice, been hosting my servers there for quite a few years now so it was nice to see how their setup looks. This also explains why my nvme drives are hitting 90c on regular basis.
This is the coolest server farm I have ever seen. I especially like the old school tower servers with desktop hardware! This is what I always wished to have what I was a teenager, haha. It's like a home lab on steroids! Love it.
Now I know i can sleep well since my servers are all there taken cared of like a baby. So great! Loved it.
Ahh reminds me of the good old days, before everything went corporate.
Makes me like them more seeing the normal and custom gear.
what do you mean "before everything went corporate"?
@@tddt2227 I think he meant before companies started using AWS and companies no longer directly managed the hardware of their servers.
@@Gamesational1: Guessing that's AWS.
Always fascinating to see data centers that work on scale. Great video.
11:51 I deeply relate to the lone memory chip that didn't make it into the cool chip club inside the funnel.
Been a customer of Hetzner for years so really love this!!
A++
Thank you for this and taking the time to produce, VERY VERY interesting and well produced.
Thanks for the most interesting tour. I am amazed by the size of this operation and the total amount of computing power contained here. I am from the old school of mainframe computers. First started working with computers back in 1965. One computer in an air conditioned room with a raised floor to circulate cold air. One CPU and 24K locations of memory in the typical system I first encountered.
Very cool tour. Awesome how they do all that in house testing and 3D printing. And lol at their “not perfect” cable management. Beats mine!
My very first server ever was in the Hetzner datacenter. Nice to see the company is still doing well
Hetzner is awesome, I got a great deal on a box through their reverse auction. You can't get a box with specs that are even close in the states. Those guys rock! Their dashboard and load process is pretty slick too! These guys know what they're doing.
Well sounds like they're incorporating in the US, or having to charge VAT tax to US citizens now. While this doesn't mean the end of the world, in a price competitive industry, it definitely affects their edge they had.
@@ryanpaaz They just have to be careful not to be blown up like the pipeline or robbed with fines like VW when entering that US market...
i love hetzner, always used their services for hosting. nice to see how it works internally, thanks for the video! :)
Building those super slim and barebone server cases in house is genius. Must be saving them a fortune.
As a long-time customer of Hetzner myself (and regardless of how old this video is as of writing) this has been a really exciting and interesting video. 👍
I've been a Hetzner customer since 2014, love the company, prices etc. I currently have an AX-51-NVMe with a 10Gbps connection :)
why not use AWS? what was the determining factor?
Die Mühe die ihr da rein steckt, respekt!
Wenn nicht, denn...
OMG! I still use i7-920 too. It is still ok at gaming and gets the job done.
Hey der8auer. Just wanted to drop by and thank you for all the great videos, and especially a special one as a fieldtrip to a datacenter. Really cool. Hoping for more like this.
Greetings from Sweden.
I'm studying for my CCNA 200-301 and Linux + currently , I have a fascination with servers and hope to be working in a data-center like this some day...thanks for showing this
I know almost nothing about data centers or various components apart from recently upgrading my current laptop to a newer one after 9 years and having to "learn" a bit about various components to understand what sort of various GPU CPU etc are in recent models. This video was really fascinating, well presented and insightful into how a good data center operates and all the various considerations with cooling, ventilation, power use and testing and building are involved. Thank you so much.
Videos and channels like these are doing great job at keeping TH-cam to be useful and educational platform. Hats off to all participants of this video. Great Work! || Tolle Arbeit!
Greetings from Slovakia! 🇸🇰 ☮ 🇩🇪
Fantastic content, love to see servers from other countries and their different mentality!
Not much different, quoting Elon musk, "There is only one rule, Physics".
This is great. It's interesting that Hetzner went with Asus to build their boards, I keep thinking maybe other brands aren't so bad but in the end I always stick with them. Though Supermicro is cool if you want to build your own server, I recommend it!
Been a Hetzner customer for years, this was great, thank you!
Say he is well known for breaking things. Linus laughs at this.
It could be that I accidentially killed my 12900K today by dropping a screw on the mainboard. So much about that
@@der8auer-en seen worse humblebrags in my life
@@der8auer-en lol no day 1 review for you
He is the European version of Linus 🤣
@@der8auer-en That sucks, those are expensive too.
It is interesting to see where my old server was hosted, I never knew it was just a tower. Now I've moved onto their AX51 servers which if they're in that newer part of the building, they look very clean
Hetzner - What an awesome company. I've been a loyal customer for many years. Thanks for bringing this great video to us.
I rented 400+ cloud VMs in Hetzner for many years and had no issues and I like the price and quality I get for this price. Thank you, Hetzner!
Also rented a few Dell servers and SX292 servers, and I had no issues at all within about 8-10 years of rent history. But I heard stories about low server quality in Hetzner, maybe because I never rented low-price servers I didn't notice that. In any case great provider, thanks for this video! It would be very interesting to see more about Hetzner cloud structure, what is based on (hardware and software)...etc
This was quite ineresting. I'm a Sys Admin for ~25 years and there is always something to learn. Amazing that they custom build everything. What I would give to have that ability in my Data Centers.
I've been using hetzner for 3 years, and I love it. keep up Hetzner
Perfect video dude.
I love these organisation of a data centre.
I can remember the days I lived in Germany, how they care about thinks.
In Germany, nothing is impossible, they didn't care what other say, they just made it right and precise.
I'm missing those time really...
Thank you too Hetzner for such of trust you give to this guy.
Super cool to see Hetzner insides... very good provider for years :)
The main reason you wouldnt run 12V out to your servers is cable losses. You could, though, easily deign your PSUs to take AC or DC (most switchmodes immediately rectify it after all) saving you those inverter stage losses.
Thanks heaps for the amazing insight (and to Hetzner too)
Yup, that's why it's usually 48V systems in data centers and then converted into 24V or 12V in racks. However I don't know if it would be 48V directly from batteries.
@@insoYT That makes a hell of a lot of sense! why 48V? the legal safe voltage for exposed installations is 50V at least in the US, you dont have to treat it as 'live'
Your analog phone run on 50 Vdc
@@user-jn9dl9px6r what's that?
Depends, if you were to design a system where batteries were separated "per server" in theory you could run everything off 12v but it would be an absolute mess and a lot of unnecessary thick cables.
Awesome. Thank you for the tour, Roman.
Very interesting! Such a high level of hardware engineering! I would love to work in such a data center!
didnt know you had this English spinoff channel, I use hetzner so appreciated the video, thank you.
That was an awesome tour!!!
Thanks for sharing - this is a refreshing look at a greener server farm.
Very cool to see somewhere like this. I can understand what they mean about trying to tidy things up with the HDD test racks because if you're using that kind of setup a lot (which they clearly are) then balancing hard drives and connecting cables like that is going to be frustrating (I worked on radar kit where we had a similar test setup and it was a real pain). A frame with a backplane would work better, a row of slots you can plug the drives into with LEDs at each HDD position.
Very nice tour, I bet a lot of the viewers have stuff hosted at Hetzner, including myself!
I also like how their newest building is constructed using an IFA W50 - also some efficient use of old hardware :)
awesome! Just wish this was longer and even more detail, 60mins+ of going this stuff through would not get boring :)
Awesome video, really interesting to see how they manage all their servers!
Definitely one of my favorite episodes. This has place has nerd written all over.
I'm using Hetzner servers for 14 years now... nothing but the best!
I would be interested in seeing a report on fans much like Backblaze does with drives. They have to have awesome data on these. Love the fact they use almost exclusively air cooling. So much to like about their approach to this.
This is my new favorite video from Der8auer, thank you for this.
With the standing HDD's being "unorganized"
It looks nice and straight, but really easy to fall over, shocks you don't want to give HDD's while running.
FABULOUS tour, der8auer! You're a genius!!
This is awesome!
I have 4 servers at hetzner :D. Very happy with them for the last 3 years. Had OVH before.
9:18
"I would like to touch things, but I tend to break things, that's why I'm not going to do it"
Immediately takes hold of a cooler
it's pretty hard to break a heatsink eh?
was less worried than touching a Threadripper again
@@mscd9676 I did drop my heatsink and bent the hell out of the fins, so...
Thanks for the intro, the compact servers with ATX motherboard that optimized for cooling is cool. And now I know why Hetzner's price is so much cheaper than other providers.
Contest, derBauer vs. Linus on who can drop and break the most.
So nice to actually see the datacenter, where I have rented a server for so many years :D
Nobody is wiser than engineer. Except German engineer...
Fantastic tour of the data center, thank you.
Would there be potential for solar panels on these roofs just thinking about the angle.
i was just thinking that, why they hadn't covered the roofs in solar panels already!
Well, he said they are already using green energy. I wonder if solar just isn't enough, or consistent enough (night time) vs. wind or hydro.
It would be interesting to know why they aren't using some solar, though. I'd guess that it's too cloudy on average, that the investment into solar at that location just wouldn't pay off.
@@47CryXMA Maybe too much resources needed for solar panels? I guess that the consistency is also a big factor, the power output is changing with the second and that is bad for the power net in general. If they use thin film panels they could partly fix the consistency. Those panels start generating energy at very low light but the max power is also low.
Maybe they angled the roof away from the sun to keep things cooler?
Edit: Actually based on google maps it looks like that's not what they were going for.
I personally think it'd be a waste of money in Germany. It's cold around 3 seasons of the year.
Amazing content! I was very curious about hetzner datacenter!
Man, building servers there would be my DREAM JOB! Do you know if they are recruiting? Even better would be them expanding into Ireland so I wouldn't have to move!
It would get old
@@mrmotofy Not likely as hardware keeps evolving ;) First PC I've built I was 11 and trust me when I say that this has never gotten old for myself, quite the opposite, new hardware and/or technologies coming to market keep things interesting! Like when FLASH finally came of age on SSDs instead of USB sticks.
There's always something new or improved around the corner and sometimes some deeply disruptive changes like the switch from AT to ATX that resulted in a considerable upgrade cycle although this may not be the best example.
@@wskinnyodden Yea...but building these servers is gonna be very repetitive as they seem to be virtually the same. So some might have a 100Gb Nic instead of 10Gb...big deal yes I remember AT vs ATX, I picked an ATX as it seemed the way it was going.
@@mrmotofy Well, that is just a challenge on how to get as many as possible going at the same time efficiently and correctly. Wouldn't be a first, I still recall the first time I had 10 systems on the work desk at once being installed and set up each for a different customer. (Christmas time for a computer shop back then)
Trabalhei no data center da Embratel nas olimpíadas do RJ irmão, foi meu último job depois que me mudei pra EUA, queria muito voltar pra área também seria um sonho tá num lugar desse de novo
Шикарно. Спасибо за подробное видео. У меня в подвале тоже микро дата-центр - 6 старых системников стоят на полках и работают серверами. Сегодня как раз собираюсь поставить сетевой фильтр с раздельными выключателями для удобного включения\выключения.
выгодно ли иметь мини-центр обработки данных?
This is heaven, right?
loved this video man, very well explained and interesting.
You really need to talk about this EN channel more on the main channel, I don't even know until I speculate that you won't upload no English videos and I found this in the channels tab.
I think I was talking about it in the last 3 or 4 english speaking videos and then just started linking below the German video. Hope the word eventually just spreads
Thank you guys I needed this
Wow. I though datacenters or any kind of enterprises are running only xeons and epycs. I always wondered why intel, amd or nvidia artificially lock up their consumer hardware. I though that no one is willing to run consumer hardware for enterprise use so it seemed stupid to me that they bother to lock up certain enterprise features. This changed my perspective. And AMD or Intel must be quite mad seing 3900x being used instead of epycs which cost twice as much :D same for i7 instead of overpriced xeons. Very very interesting video.
Hetzner isn't "enterprise grade", it's inexpensive hardware at an inexpensive monthly charge that doesn't make you sign a long-term contract.
@@PablumMcDump Thanks for the correction. I went to their website a see now that I can rent specific servers with consumer HW, I thought that they are using the ryzens and i9 for cloud. Still though I didn't know this existed.
This was a really interesting video. Thanks for making it!
What is the model name of those blue IP KVM devices that are daisy chained on the back of the midi towers?
What an awesome setup. The layout of some of these Data Centers always amazes me with how uniform everything is.
GO GERMANY!🤟
In Seinäjoki Finland we have datacenter wich heats city with district heat coming from servers
The racks of your standard consumer grade desktops being utilized as servers just goes to show you don't need to get a Dell PowerEdge or something crazy to start your own server. So when people say they can't afford to buy a server, I'll just send them the timestamp 2:35
Id love to work here
Appreciate your tour. I am going to build my own server with mini tower PC😂
as a dutchman i got a question. ever been to the amsterdam data center? supposedly the biggest data (or busiest data center in europe some even say in the world). i have seen videos on it and read alot about it. and its very impressive. most game servers you use in germany are hosted on that data center. Blizzard and steam for example are located in amsterdam. its pretty interesting. anyway nice video very informative.
@@El_Marzocco we just got a new huge google data center in the north of the Netherlands in Groningen. Microsoft just build another one in the netherlands in Middenmeer its in the province of north holland. well they got green light to build it its not their yet but it will come. its in the province of north holland. But the reason why they do this is because we are a tax evasion country one of the biggest in the world. so its cheap for big companies to build their business here. they dont have to pay taxes and what not. something they had to do in any other country in europe. so easy choice to build it here. also we have the best fiberoptic infrastructure in the world. maybe that got something to do with it as well.
Very nice video ! Thank you for this, I really liked the lead acid battery room small tour :)
Is keeping these old equipments like i7 920 really environment friendly? I don't mean to judge that, just courious. I guess there is a comparation between recycling these low power efficiency device and keeping running on them.
I was surprised by that too. These are OLD 4 core CPU’s with ancient architectures and - relative to today’s standards - terrible IPC. They could probably consolidate an entire aisle of these ‘servers’ into a single rack of modern 64 core Milan servers and just run 4-core VM’s on them if that’s all the customer wants. And even if they want a more consumer oriented platform, modern Threadripper or even Ryzen (to a lesser degree) will provide similar levels of consolidation.
@@benjaminlynch9958 If you're not familiar with the way Hetzner works, you buy the entire machine as a baremetal/dedicated server. When you no longer want it, you can auction it off online via their marketplace. It's sort of in their business to not mess with bought dedicated servers and let them use them until they fail, even auction them off to other hetzner users. I'd imagine the cost of trying to migrate everyone off those old servers (Forcing downtime, trying to migrate disks, and trying to match the same low prices) just isn't worth it.
It is kind of nice that you can buy one dedicated server from them, and as long as you keep paying, you can use it virtually forever. Of course, the downside of wasting energy and space.
You think manufacturing hardware has zero environmental impact? Think again. Making electronics is very dirty business. Including slave labor to mine the minerals. People are in bondage so you can have a few more FPS. How dare you!
@@1pcfred Didn't I mention the comparation between recycling and keeping? Old products are less energy efficient, and energy requires labour, too. Why not think carefully before judging someone?
@@1pcfred calm down, what do you get by telling that to the people, do you think i want to know such information when i buy a fucking ssd or a new cpu i need?
no one said the world is perfect and the blame can't fall on customers
Excellent hoster, never had a problem, support is always awesome and the adminpanel is great :-)
I would have imagined they just sold old hardware because it's just not power efficient. even if it's fully functional they can do the same thing with 15W that the 920 does with 130W
As long as you have the space and you're not paying for AC then power probably isn't that expensive to make it worth upgrading the whole lot. I'd guess they only retire hardware when it's just too slow to do anything useful.
@@Gractus only because they have relatively few left. If they had a building full i7-920 machines the power difference would cost a ton
Right now on their server market place - the oldest CPU is i7-2600 - if you get that server I guess they will let you run it for at least 2 years. So the i7-920 is just running because you do not want to piss of your customer - but the wattage is too high. However likely people using such an old server have that server idling most of the time - so it doesn't matter too much that is has a high TDP. Next likely those 95W 2600 are phased out. Lately the push with free setup on AX servers but not EX is clear as well - they must have figured overall power use is much less compared to same price EX (which likely has lower hardware cost). Not everyone maxes out the CPU of their servers. E.g. I have a AX-41NVME which is idling 99.9% of the time running just two websites. I cannot go virtual/cloud because I need 10TB hdd for downloads on those websites. Cannot find anything cheaper than hetzner. I run a second server to compile stuff, which needs 6TB hdd, and 1TB NVME space - it's only used 1/6 of the time. rest is full idle. Also when not compiling I need the hdd to store lots of data. Again I do not see a cheaper solution. Yes I could use one bigger server instead of two - and run one in virtual machine. But I fear reliabilty so websites needs it's own server. Likely many people are like me - only maxing out some certain aspect of their server - and not having it run 100% for much of the time - hence Hetzner can offer them soo cheap. I would love if Hetzner lowers prices and charges electricty directly to customer - but likely it's too complicated so they just keep it simple and lose money on customers constantly maxing out their server. If you maxed out your 920 you would already have upgraded. And on idle likely the power difference is not much - hence they can afford to keep them running. On the other hand - virtual servers/cloud will always be on very modern/energy efficient hardware.
They *are* still power efficient - given that most are idling or on low load. I think that's a sensible use of resources as they're using very little electricity.
Also, junking perfectly good hardware in useful service is certainly unnecessarily wasteful. I applaud their approach with these older midi-towers.
I once did a blockchain project for a german, based on a 64 Core dedicated machine in Hetzner, had to do setup using KVM with booking kvm slot via support, very nice pricing if you are willing to build your own VPC with custom os and shared VMs..
Why are those southern facing roof surfaces not covered in Solar Panels?!
Because solar panels don't generate enough electricity to offset the cost of purchasing/installing/maintaining solar systems.
But it would reduce the load on the Grid and therefore the environmental effects even by a small amount @@oliver9089
@@oliver9089Well that’s a complete lie.
@@oliver9089thats such blatant misinformatio . We in developing countries have started to switch our homes and businesses to solar, because the government is inept to do it themselves. We got our return on investment within three years after installing them.
@@askeladden450Не путайте Африку/Индию и ваши домашние цели, с промышленными комплексами Германии. Если у вас это вполне выгодно, то тут это может вовсе не окупится, говорю как житель России
Amazing! Thanks for the tour 👍
It's called climate cycles not change. Nothing has changed in millions of years and there is nothing we can do about it.
Awesome video -- Amazing place! Would love to see more "tour" videos of different types of tech facilities in the future!
This brilliant. More places in the states should use this model.
I run several webservers and application servers on virtual machines in that data centre, it's cool to see inside. Thanks!
Ich habe gerade das deutsche Video gesehen und einfach großen Respekt dafür, dass Du das ganze parallel auf Englisch gemacht hast anstatt subtitles oder eine kürzere Version mit voiceover.
nice walkthrough Roman!
i dont know why but the shot at 13:33 reminds me of that one scene from harry potter where they try to find that prophecy globe :D
Best servers. We are using them over 5 years. Never had problem with Hetzner.