Cheap home lab build: virtualization server, GNS3 for CCNA-CCNP rack, and ZFS
ฝัง
- เผยแพร่เมื่อ 6 ต.ค. 2024
- Here I build a virtualization server which can be rack mounted inside an IKEA Lack table converted to host a Cisco CCNA-CCNP home lab (with GNS3 for Cisco router emulation) and virtualization lab for studying. This video is a shorter, updated and compressed version of a video which I uploaded half a year ago about building a cheap virtualization host for home use, i.e. not involving expensive industry grade server components. The VT-x (AMD-v) virtualization extension and VT-d (IOMMU) feature for direct PCI-e device direct bypass/passthrough to VMs are discussed. I also focus on features which you will miss if you do not invest into a server-grade motherboard. In particular, SR-IOv for single-root I/O virtualization is only supported on newer server-grade motherboards, which are still very expensive. The HP NC364T quad gigabit network cards are a good choice to start your labs. However, these cards do not support VMD-q, which is nowadays becoming more frequently used. The cards work fine for GNS3 as breakout cards to real Cisco 3750 switches. Other quirk of these quad cards is, that they have a PCI-e switcher chip on the board, so you need to bypass both NICs from an island, and you need to turn on "unsafe-bypass" support in your hypervisor. For storage, I had very good success with ZoL (ZFS on Linux), where mechanical disks are used in RAIDZ, and an SSD partitioned in two. The first SSD partition is used for ZIL (ZFS intention log) and the larger partition for L2ARC (Level 2 Adjustable Replacement Cache) as a read cache, which really helps when you have little RAM in your server (mine so far only has 16GB of RAM). Thus, ZoL and ZFS turned out to be so good, that I would no longer spend money for a RAID controller card. Furthermore, the motherboard I use has plenty of native SATA ports, so I was able to save money not having to buy an HBA (Host Bus Adapter for ZFS). Running ZFS without ECC RAM for home lab is totally fine, since the data has no real value. The hands on experience with a home lab is PRICELESS!!!
I have successfully tested this rig working with Linux KVM (Kernel based Virtual Machine) hypervisor and also with ESXi 5.5. With ESXi 5.5 the only issue I have bumped into was, that the driver for the Realtek onboard NIC had to be baked into the installation ISO.
COMPONENTS I HAVE USED:
==========================
CPU: AMD FX8350 (has AMD-v, IOMMU and ECC support, and 8 corelets)
CPU cooler: Alpenföhn Brocken with 12cm fan (very quiet, fits into the small case)
MoBo: Gigabyte 990FXA-UD5 Rev. 3 (IOMMU support and AMD-v work fine)
MoBo also has lots of PCI-e lanes routed to the PCI-e slots, so VM passthrough is working
MoBo does not support SR-IOv, so you cannot share a PCI-e card among multiple VMs
RAM: 2* 8GB of Crucial Ballistix Sport DDR3 (no ECC, should extend it to 32GB)
Video: old 32-bit PCI VGA (Gigabyte MoBo does not boot without video!!!)
Network: 3* Quad Gigabit HP NC364T + onboard Realtek GbE LAN
Storage: 32GB USB stick for hypervisor + 4 mechanical disks in RAIDZ-1
Storage read/write cache: 128GB SSD partitioned for ZIL and for L2ARC
Case: cheap ATX chassis, bought used, converted to be mountable in an IKEA "Lack" table rack
PSU: Corsair VS550
IPMI substitute: native serial console
InfiniBand: still hunting for cheap Mellanox full-height QDR cards with VPI and cables
Hi thanks for this vid, i am at the starting point of walking this career path and your videos sure help a lot,. I will use your vids as my guide.
Hi Marvin. Thanks for the feedback.
To be honest with you, if I would be now building a lab, based on the stuff I learned from the build you see in this video, I must admit that I would rather build something based on a server board. Namely, due to my job I am interested in SR-IOv and other related technologies, and these cheap boards do not support it. If you can afford it, go for an Intel Xeon used motherboard, especially if you can place the server somewhere, where the noise does not bother you.
The comment I deleted was about desktops and bare metal hypervisors.
Well, the main advantage of Linux KVM is exactly that you can actually use the underlying Linux system as a desktop, while the VMs are running on the same CPU, using the kernels built-in functions to provide them the virtualized environment.
thx for the video. I was waiting for the update.
I am now working on a video where I show the KVM installation (based on Debian Linux) on top of this machine. This weekend or next week this video should be also up.
Tried the new VMware ESXi 6 as well, but they have dropped most of the hardware support and officially now ESXi works only on the newest fanciest and super expensive hardware.
Good video!
Thanks for the feedback!
As you see, I am trying to do it cheapskate way, and spend as little money on the machine as possible.
Good vedio
I hesitant
for choosing one of these workstation (Lenovo S30 & HpZ420 and HPZ600) to make a virtualization - cisco switches- Routers)
Thanks for advance
I never had a Lenovo, so I cannot give you feedback on that. However, I had one of these HP Z600. If you are using the right Xeon processors and the latest BIOS, it will even support VT-d. However, for me the machine was way too loud. Especially that small fan cooling the chipset was screaming at a high pitch. If noise is not an issue for you, then it might be a workable option.
Good video.
Thanks for the positive feedback.
For sure you could have done a better woodworking than an IKEA table ;)
If you find issues with the build, or things which I explained obviously wrong, just let me know so I can improve it in the future.
Subscribed,. I like the content and love your accent, what's your main language ? :-)
Thanks for subscribing and commenting.
My native language is a secret ;) I speak 4 languages (not all 4 on a native level, only fluent), so I was influenced by many accents.
Would it be possible / would it make sense to have a system like this for your desktop system alongside the vms or is it really necessary to have two separate machines for those? That would save some space and definitely some money.
Really like your videos btw, please do keep up the great work!
First of all, thanks for the feedback!
My original idea was EXACTLY like what you suggested, and with Linux KVM it could have totally worked. However, the issue is, that the switches are really loud, and this is why I have built this second machine. This way the server is in one room with the switches in a rack, and in the other room I have a quiet desktop which is almost noiseless, so that I can concentrate on the topics I am learning. Also, this way I have easier physical access to the cabling.
I will soon upload a video where I explain this, and why I put the second machine together and moved the server hardware into a small rack-mountable chassis.
Thank you for your quick reply!
Can you tell me if there are noticeable differences between ESXi and KVM in terms of performance? I think I would prefer a single machine setup for now to avoid additional costs.
I am wondering if the desktop os couldn't be just run as a vm on ESXI once having set up everything, that would be really nice (Only for a learning environment, of course).
Oh and yes, noise is a real issue, I totally agree. I'm still fighting the noise level of my pc :) (it's an FX 8370E build btw)
KVM is a free open source alternative, wheres ESXi is a closed source proprietary solution.
This means, that they work completely differently, in fact, KVM differs even from Xen hypervisor, which is also based on Linux.
Thus, comparing KVM directly to ESXi is a bit of apples to oranges. Still, KVM is surprisingly fast in my view, and I cannot really tell the difference between the responsiveness between the VMs. However, I must state that I am using my lab purely for learning and never for production, and for studying the performance is not much relevant.
Other question is what you would like to study. Namely, when it comes to high availability, ESXi is still the best option out there, so many companies use it instead of Xen or KVM.
Running the OS for your primary desktop inside ESXi is a bit tricky, since you will get no direct video output from your desktop OS, except if you use VT-d, or in your case IOMMU, and you passthrough a video card to your desktop VM. For comparison, VNC or other solutions (except Nvidia Grid) are just too sluggish, I have tested that already.
In stark contrast to ESXi and Xen, Linux KVM is a Type-1 hypervisor based purely on Linux kernel. It is almost like a hosted Type-2 hypervisor, but still with full Type-1 bare-metal features. Thus, you can totally use your underlying virtualization server to run a full blown Linux with graphical desktop on it, and also use the remaining resources to run VMs, including passthrough of PCI-e devices. In my view this is why KVM is a really good option exactly for what you would like to achieve. I will make videos on the software setup as well, so you can check how easy (sometimes complex) it is to set it up.
Why do you put so many network cards in the computer rather than just running a vlan capable switch?
The number of cards is not even enough for the type of scenarios I would like to study on this system.
Namely, when GNS3 is running without a "brakeout" switch, every router interface will use an RJ45 port. Also, every VM generating traffic and sending data through the GNS3 network will also have an interface bound to a real network port, so that I can interface the lab with real Cisco switches.
Even when one is doing simple Virtualization labs, it is really tough to separate iSCSI traffic from management and VM-network traffic. The cards I use are old and slow (only 1Gbps), so I also need LACP bonding.
What would you say is a resonable priced mobo with SR-IOv support these days?? It is for my personal home lab. I plan on using Proxmox? I would like to use both Containers n VMs which is my reason for considering Proxmox.
I would prefer a system which is fairly easy to use and is fairly secure and has good community based support. Any suggestions or recommendations?
Whenever you choose the software what you will use for your lab, *you should pick the software with which you can get a job.* Proxmox is relatively rarely used, so if you are not at least 80% sure that you can get a job with Proxmox, rather look into VMware or KVM. On KVM you can also run both containers and VMs, no problem. In addition, KVM does not have a special VM (Dom0 on Proxmox is this special machine) which is used for management. As for the MoBo, I would be hunting for a more modern dual socket Intel Xeon board if I would be you, preferably from Supermicro. If noise is not an issue for you, a Dell or an HP server in 2U chassis is also a good and inexpensive choice (only about 400-500Euros). If I would have a basement or a large apartment, that would be the way I would be building my system. For standard ATX cases I could not find so far any MoBo with SR-IOv support.
thank you for the nice explination. may you please compare between virtualization server and qotom small computers? I am looking for 4 to 6 virtual servers and I am not sure if it will be cheaper to build virtualization platform or owning several qotom computers. I would appreciate your feed back.
I am not familiar with the "qotom" computers to what you are referring. Can you put in a link for a more accurate description of what these machines are?
thank you for answering back. qotom is tiny computer with j1900 processor and 4 Intel Gigabit LAN ports, up to 8GB RAM GB sata port WIFI, for about $150 to $270 depending on configuration. it can be found on amazon shop called qotom and their web address is qotom.net . Some guy bloged about runing lxc vps on j1900 cpu. I need small lab plus some lightwhight vps for home office file server ,pbx and vpn server. power consumption is some thing to consider. I would appreciate if i can talk to you if you can help out with my choices. thank you again
Samy, look up MAXTA.
For CCNP & CCIE Lab what kind of hardware need to buy in 2020.
Cisco have *severely* changed the certifications now. I would need to go through the list of things they require to put together a lab.
@@DonkeyLearningIT hope you will recomend the hardware and the software so.
what abt intel alternative instead of amd.
I am not familiar with the newer Abit boards. The old Abit boards until the capacitor fiasco were alright, and made for extreme overclocking.
However, for virtualization you need special features from the BIOS, which might not be available from Abit.
I have purchased the AMD board, since the equivalent of the VT-d, VT-x, multiple PCI-e lanes and other virtualization features are available only on Intel Xeon CPUs and server grade MoBos.
but normally everybody says do vmware and virtualization in intel for study purpose. i agreed AMD equivalent is there and and it will be sheaper too.
AHHH, just deleted someones comment BY ACCIDENT, by hitting Remove instead of Reply. SORRY man!!!
There is no way to put it back, another YT fail. Just hit the wrong button, and it is all gone, without any warning...
Shit happens :)
The comment you deleted still appears in my notifications, so I just copied it:
He is using a bare metal hypervisor. There is no usable desktop interface
with ESXi and such hypervisors. You can still create a workstation with
Windows 10 and purchase VMware Workstation to create the networks
virtually and run nested ESXi. However, his goal of creating a LAB is to
simulate as close as possible a corporate or business environment to
learn ESXi. I would go one further and add a SAN IP network and create a
NAS for hosting the VMs over NFS or iSCSI. This will help you learn how
the product might be used in the real world.
Many thanks! YT should totally fix this issue, this is a total fail. You get no warning whatsoever, the comment is just gone... And the two buttons are right below each other, so on mobile you have to be super careful where to tap.
+Henry Eng
ZoL (ZFS on Linux) will be used running on the main KVM kernel to provide NAS and SAN capabilities to underlying VMs. This is why ZoL is so awesome, because it can be used both for NAS via NFS and one can also create block devices on it, and export those as iSCSI targets for the VMs, and your VMs can boot from it. In addition, it has SSD read and write caching via ZIL and L2ARC.
Magyar vagy?
???
En/De please.
Where are you from?
I am just a Donkey beamed down from the future through the Pleiades star cluster by the ruler of the planet Omicron Persei 8 to overtake the Internet of the earthlings ;)
In our cosmopolitan wold countries/borders in many sense no longer matter.
I move around very often, so I speak 4 languages (only 2 fluent, the 2 others just intermediate level).
Why not just buying a cheap server like the this dell R710? www.ebay.co.uk/itm/Dell-PowerEdge-R710-3-5-Intel-XEON-5500-5600-Serverschmiede-Server-Konfigurator-/322028560042?_trksid=p2141725.m3641.l6368
There are several reasons, where the main ones are:
1.) such machines generate LOTS of noise, which is simply unbearable in an apartment. Plus my neighbors would call the police after a couple of hours. I would call them too in a reverse situation...
2.) such machines do not support VT-d, or the BIOS has severe limitations when it comes to VT-d
3.) the machine you linked does *NOT* contain CPUs, RAM, or any other cards. It only has the PSU and the MoBo in. When I add the price of the components what I need+shipping, I can build a lot better machine for myself.
4.) the I/O ports are very limited. I would simply not able to built in as many I/O cards, especially when it comes to high throughput cards like InfiniBand
Donkey Learning IT the machine I linked certainly has all you can ask, there's a configurator in the description. where you can add all the CPUs, add-on cards, memory, hdds, etc. As for the noise, it is quiet if you manage to keep low temperatures(for example air conditioning in the room) but if not they are extremely loud, yes. It sure supports virtualization and it's an extremely popular server for home virtualization laboratories from the research I did and several people use a pair of them as virtualization nodes for vsphere and azure cloud. The eBay article linked even has vsphere 6 installation in the configurator.
Here in German speaking countries there is simply no air conditioning since the "renewable energy" costs are ridiculous (0.28Euros/KWh, above 200KWh per month price jumps up to 0.35). Even a small apartment costs a lot for rent, above 30% of my salary, and electricity cost would kill you. This is what *socialism* is all about, darnit! You have no idea how poor we have became within the EU due to overtaxation and austerity.
If I would be able to afford the luxury of having my own house and basement (it will never happen), for sure I would have an industrial server, and would not care about noise either. In my previous apartment I had an HP server before, and it was way-way too loud. Previous place had three rooms, but I moved to an other city where prices are even higher, and cannot afford more than two rooms. This is why I had to sell the HP server and build my own. Eight 6cm fans spinning at 3000RPM in a small room is no joke, not even talking about the four Cisco switches in addition.
Indeed I can understand that, it's a shame though because if not for the noise that seems a very decent server and affordable too.
Also to vouch. For people with constraints, the R610 is a much quieter option. The Xeon L,E series have VT-d but is based on the CPU. I have a friend that runs it under his bed and the cooling in that is fine.The CPUs can be run with the L5630's which are lower power. The RAM usually is quite cheap with RDIMMs which can have higher density ($30 for 2x8GB KIT u to 12-18 slots), and the L5630's were $20 for a pair. The R610 supports 2 PCI-E and the R710 has 4 PCI-E.