วีดีโอ

Over where?

@@publicsectordirect982 Ukraine

​@publicsectordirect982 my guess China......

@@publicsectordirect982 Ukraine

Hi there. Thanks so much for the comment! Yeah this is definitely a good point. I agree that more VRAM is typically better even if you actually have lower throughput, better to be able to do the task even if its slow than not because you dont have enough VRAM for your usecase.

Hi there. So I have a video actually related to this. th-cam.com/video/RUW3Ay5rsCY/w-d-xo.html Check this out. In the video, I should how to adjust the fans manually. You could put yours manually on the lowest setting and you may be able to turn them off completely. However, please be carefully when doing this. It is never good to run your GPUs or the rest of you system too hot for too long. Another alternative might be to get some snail blower fans and install them that make less noise then lower the naive fans to the lowest setting. Something like the following may work for you. www.ebay.com/itm/186459922797?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=2ucjrgqosnm&sssrc=2047675&ssuid=xv3fo9_stiq&widget_ver=artemis&media=COPY

Hi there. Thank you so much for the encouraging feedback. Really appreciate it!

Hi there! Of course. So glad that you enjoyed the content!

Hi there. So glad this is what you needed! Thanks for the comment!

Hi there. Thanks so much for the kind words and the comment! It really depends on the use case, but overall for most people I would say that the P40 is actually the better choice. The only GPUs I currently have that support NVlink are my RTX 3090s. Yes, I have them connected via NVLink.

Hi there! Thanks so much for the comment. So, my take here is the NVIDIA GPUs are going to be much easier to work with at this stage. I have head from some of my viewers that AMD GPUs can and do work. It is just a lot more of a pain to work around bugs and the learning curve is steeper. NVIDIA is more or less plug and play when it comes to AI/ML/DL, but that is also why you pay a premium. I guess what I would say is if you want the easier route or don't have time to do much trouble shooting NVIDIA might be a better way to go. However, from the sounds of it you are more partial to AMD GPUs and have many other workloads besides just AI. In your case, it may be a better idea to go with AMD GPUs because you will get better price for performance for all of your other workloads then deal with the pain of setting up you AMD GPUs for your specific AI use case. The 7900XT is definitely a powerful card and can handle AI tasks, though you might need to use specific software or frameworks that support AMD GPUs, like ROCm. Creating a VTuber model sounds like an interesting project! I would recommend maybe starting with programs like DeepFaceLab for deepfake-style video or some stable diffusion flavors to generate images as a starting point. For generation, tools like Blender for 3D modeling could be helpful and for real-time animation you might could use VMagicMirror or VSeeFace which can utilize your GPU to bring your VTuber model to life. Hope this helps!

Hi there! Thanks so much for the question. So for all of those things on a low budget. I would probably recommend the p100. You could also go with the p40 for more VRAM. The p100 will handle quantization more efficiently and have high significantly higher throughput which will be important when working with LLMs. The p40 has higher VRAM to start with so you can load larger models, but the fp16 performance is really bad so its through put will be a lot worse (theoretically). As budget options though, I think these could still put some of the smaller open source LLMs within reach for you to start experimenting with. Hope this helps!

Hi there. Thanks so much for the comment! This is great information. Thanks so much for sharing. I did know that floating point ops were different fundamentally than other operations though I was not 100% sure why. This makes a lot of sense! Also, if you know the correct formula for calculating FLOPs in general please let me know.

Hi there. Thanks so much for your comment. The process is normally as easy as: 1) Creating a bootable drive with whatever OS you want to install 2) Plug in the usb 3) Power on the machine and boot into BIOS 4) Adjust the bios settings so that the usb device is the first boot choice (although many times you dont even need to do this) 5) Let the machine boot normally, and you should be prompted to install the OS. 6) Install the OS 7) Restart machine and boot into BIOS 8) Adjust bios setting so that the new drive where the OS is installed is the first boot option 9) Let the machine boot normally, and you should have a working fresh install of whatever OS you choose.

Let me know if this does not work for you. I can try to help you trouble shoot from there.

@@TheDataDaddi I was able to solve the issue but the things is that I originally tried to plug in a 2.5 ssd drive to the back of the server and that didn't work in terms of booting, I was able to install the os on it but when restarting the bios said no bootable drive was found. my solution was to use a usb to Sata adapter and plug that in the internal usb port and that worked and one could then boot, but the drives are detected in ubuntu and can save files to them so it's weird

Hi there. Thanks so much for the question! You should be able to connect if they are the traditional PCIe 8 Pin (6+2) GPU connectors. The 4028GR-TRT has 8 12 EPS power cables that support most Tesla GPUs naturally. From what I can tell, the M10 uses the more traditional 8 pin connector. You could buy an adapter from 12 V EPS to PCIe 8 Pin (6+2) power. I think the following should work for you. a.co/d/hMUEDvs Hope this help you! Please let me know how it goes.

@@TheDataDaddi thanks for your prompt reply. I am wonder if instead of 6+2 pin, I can use 4+4 pin since I have it at home.

@@wasifmasood969 Unfortunately, I do not think the 4+4 pin will work in your case. You can certainly try, but it will likely not fit.

Hi there. Thanks so much for you question! Yeah so this is a really good question. It really depends on the size of model you are trying to run. For example, to host Llama2 70B for FP16 you need approximately 140GB of VRAM. However, you could run quantized versions with much less. Or you could always work with the smaller model sizes. In terms of GPUs, I would recommend GPUs that have at least 24GB VRAM. I have been looking at this a lot for my next build, and I think I actually like the RTX titan best. The RTX 3090 would also be a good choice its FP16 performance just isn't as good. I think the P40/P100 are also great GPUs for the price, but for LLMs specifically they may not be the greatest options because the p100 has only 16 GB of VRAM and the p40 has very poor FP 16 performance. Another off the wall option is to look at the V100 SMX2 32GB. Since these are are SMX2, they are cheaper, but there are a lot fewer servers that they will fit in. The only one I know of off the top of my head is the Dell C4140/C4130. From my research, they the SMX2 GPUs are also fairly tricky to install. Anyway, these are the routes I would go to make a rig to host these models locally. I will eventually build a cluster to host and train these models locally so stay tuned for videos to come on that subject if you are interested

Hi there. Thanks so much for the question! I think it depends a lot on your use case, but I would say if you plan on using it primarily for AI workloads the 3090s would be a better choice because of the higher VRAM and ability to support NVLINK. However, if you want to focus more on gaming and some AI work loads I would choose the 4060s and put more money towards other components like a better CPU. You could also just go with a single RTX 4090. This would give you great performance for AI workloads and gaming with budget enough for other high quality components.

Hi there. Thanks so much for the comment! I am not sure I understand the question. If you could give me, a bit more context that would be great!

This will be coming very soon! Stay tuned.

Hi there. Thanks so much for you comment. So glad that the content was useful for you!

Hi there. Thanks so much for your comment! I would agree that for anything below fp32 operations these GPUs would be quite slow. However, the GPU is less than $200 dollars for 24GB of VRAM. So, if you are wanting to experiment with larger models cheaply, I think these GPUs still have good value.

Hi there! I think this would be a good cheaper way to start experimenting! Do be aware through that training or fine tuning most of the largeer open source LLMs will be out of reach even with a setup of this magnitude. However, you could likely host some of the smaller one or quantized versions locally. Hope this helps. Cheers!

Hi there. Thanks so much for the comment! Really appreciate your positive feedback!

Hi there! So I have actually found a better cabling strategy. The updated cabling should be included in the video description. Please have a look there if you are interested in going that route. Cheers!

Hi there. I am so glad you enjoyed the content! Really really appreciate the donation. Really helps the channel!

Hi there. Thanks so much for the comment! I am glad to here you can confirm! Especially as hardware prices keep increasing I think these are actually becoming even more relevant for those who are budget conscious. Funny you mention this. I am actually working right now on a benchmarking suite to enable reliable comparison between GPUs for different models. There is not a reliable open source benchmarking solution for GPUs so I am trying to create one (or make steps toward it at least). As soon as I get something decent, I will make a video series on it and start using it to benchmark GPUs in a real way with respect to individual models.

Hi there. Thank you so much for the comment! So, I forget what length I set my rack at, but the Supermicro SuperServer 4028GR-TRT is 29" long so I would set it about 3" to 5" longer than this to comfortably fit the server. To my knowledge, servers are pretty much all the same width. One thing to keep in mind though is not all servers are the same length so if you every buy others in the future they may be longer so it is a good idea to set you rack a few inches bit deeper than the longest server you plan on housing.

Hi there! Thanks for the questions. Couple of suggestions/comments here. 1) Pretty much all consumer grade motherboards and CPUs that i know of besides the threadripper series CPUs and compatible motherboards will not be able to support both GPUs at the full x16 lane bandwidth (this route is incredibly expensive). Not having the full x16 lanes for both GPUs might be okay with you. I just wanted to make you aware. With some CPUs you can get have configurations like x16x8 or x8x8. 2) For the CPU, it also depends one whether you want to go the DDR4 or DDR5 route with respect to RAM. I would personally recommend DDR4. It is cheaper, you have more options, and is supposedly more stable at this stage (do some research of your own here as things change very fast these days). I think both AMD and Intel CPUs are fine. I would recommend at least 12 cores, but the exact CPU really depends on you budget. Some cost effective suggestions might be: AMD Ryzen 9 5950X Intel Core i9-12900K

@@TheDataDaddi I also struggle finding the motherboard as you mentioned. I found a solution that using the X11DPI-N mainboard and I have to use intel xeon gold 6138 in this way which only support pcie 3.0. Thank you for your response and I really appreciate it.

@@jack-nguyen Yeah I did as well when I was researching for this video. Eventually, I just switched over to using servers because this normally isn't an issue with server mobos and CPUs. I like the idea though! That should CPU and mobo should work great for you. What case/chasis are you planning on using? Of course man! I am always happy to help.

Hi there. Thanks so much for the comment! I have also been interested in non NVIDIA solutions. The ARC GPUs have certain interested me. However, I would caution you. If you leave the NVIDIA ecosystem, it is like going into the wild west so just make sure you are prepared. Here is a Reddit thread that might shed some light. www.reddit.com/r/MachineLearning/comments/z8k1lb/does_anyone_uses_intel_arc_a770_gpu_for_machine/ If you do decide to go the ARC route, please let me know how it goes for you. I would be super curious to better understand where those GPUs are in terms of AI/ML/DL applications.

Hi there. Thanks so much for the question! The reasons I prefer servers over OC builds: 1) Price for compute is almost always better (main reason) 2) Ability to support more cores and higher CPU RAM 3) Remote management tools like IPMI or IDRAC 4) Generally more stable and built to run forever with out being turned off 5) Built in redundant power supplies and failovers That said. A custom build is always going to be more flexible and likely give you the ability to have the latest and greatest. Overall though, I like refurbed servers because I find they provide the price to performance.

Hi there. Thanks so much for your questions! 1) For the NVME issue you will likely need to go into the bios and enable pcie bifurcation x4x4 or x4x4x4x4 depending on which pcie slot you have it in. If I remember correctly you will need to boot into bios. Then go to chipset configuration>north bridge and here you can change the pcie slot configurations. If you google super micro 4028GR-TRT block diagram this should should you which pcie slots are which. This this and let me know if it works for you. 2) The native power cable in the server is a male 12V EPS connector. This is designed to be used with Telsa series GPUs. However, you can by an adapter that will convert into the 8 pin connector that the RTX series GPUs expect. I believe the following should work for you: a.co/d/fQ9ncia Please let me know if you have any other questions!

@@TheDataDaddi Thank you for the quick response.Really appreciate your help and support. I did change the BIOS setting, Under North bridge 4x4 and 4x4x4x4 but still not able to see the NVME SSD in the boot option. Can you please help to resolve the issue? I have ordered the cable from Amazon, hopefully that will fix the 3080 GPU issue :)

@@SanjaySharma-ov1kf Sure! Also make sure that under PCIE/PCI/PnP section the PCIE slot is enabled as UEFI. Another issue might be that you have not have set the bifurcation for the correct PCIE slot. Try it for other possible candidates if you have not already. Even with the block diagrams it can be a bit confusing to figure out which PCIE slot is which. Unfortunately, it could also be bad hardware. I order 1 4 slot PCIE to NVME adapter, and it did not work originally due to a hardware issue.

@@TheDataDaddi Thanks for the hehlp. I will try another options as well for bifurcation. Under PCIE/PCI/PNP I do not have option UEFI but EFI. The PCIE slot is working as I did install Ubuntu on it by booting from Ventoy USB but the NVME drive is not visible while booting.

@@SanjaySharma-ov1kf Sorry I meant EFI. How are you connecting the NVME drive to the PCIE slot?

Hi there. Thanks for the question! So in theory there is not hard and fast requirement for a CPU RAM based on a particular GPU. That said you need enough RAM to hold whatever it is you are trying to load on your GPUs. For example, if you have 16GB of CPU RAM and 32GB of GPU VRAM. If you try to load a 24GB model onto the GPUs you will likely get errors, because even though you have enough GPU VRAM to hold the model you can't fit it into CPU RAM first in order to load it. I am not sure if this is your problem? If not please let me know, and I can dig deeper to try to help you figure out where the error may be coming from.

Hi there. Thanks for the question! I am using Ubuntu 22.04 for all of my servers at the current moment. Might switch over to NixOS soon though.

Hi there. Thanks so much for the comment and so glad you are enjoying the content! Take what I say with a grain of salt because I have not actually tried this myself, but for a 40 billion parameter model at half precision it would roughly require about 80GB to load the associated parameters then there will be some extra memory overhead for other process and space to load data to run the actual inference. So, to be safe lets approximate 100GB ish of VRAM will be needed. With 4 P40 GPUs (4 x 24 = 96GB) you would in theory be able to run inference locally. Caveat here is you would need to use model sharding to split the overall model across all of the GPUs. This should also be theoretically possible. One other thing to note is that the half precision of the P40s is very low. This will likely mean that you will have much longer inference times. Some other options, you could consider would be going with 4 3090s instead or going with 6 (or more) P100s. This should give you the VRAM you need for your use case while also providing better performance. It seems like the 6 p100s would be the best balance of cost and performance. You could also consider using a smaller model and running it at the full precision to better take advantage of the p40s design. So, it seems that for your use case you will need a minimum of 4 GPUs so you will need a different server than the one discussed in this video that can hold at least 4 2 slot GPUs. If you would like some recommendations, here please let me know.

Hi there. Thanks so much for reaching out. You are should be able to download a local copy and change it in anyway that you see fit. I acknowledge that the spreadsheet needs to be updated. I am working right now on a website actually that lists GPU specs and then keeps track of historical price trends. So all of the information in the spread sheet and more should be available updated on a daily basis soon. Unfortunately, I am not comfortable giving you direct editable access to the original version in the Google drive. I apologize. It is nothing personal. I just don't know you well enough.

Hi there. Thanks so much for the comment. So glad you enjoyed the video!

Hi there. Thanks so much for the question. Currently, for my main body of research around browser security I mainly work in computer vision (ConvNets) and also with some of the newer multimodal models like CLIP and FLAVA. In a separate project, I am working with Graph Neural Nets and Graph Databases to ingest and analyze the entire BTC ledger. As of yet, I have not played around much with any of the open source LLMs on this machine because frankly I just have not had the time. It is high on my list of things to do though. So, unfortunately, I cannot tell you exactly what works and what doesn't concretely because I do not have real data to support it. What I can tell you though is that even with 8 24GB GPUs will only get you to 192 GB VRAM. Llama3 70B will need about 140GB for half precision (back of the envelope here) for the model parameters. So pre training or fine tuning are likely out of reach. Honestly, it seems like if you really wanted to pre train or fine tune any of the really large LLMs you would need a distributed cluster. I would love to do this one day, but that will require a lot more time and funds. lol. That said, I do believe that hosting the Llama3 70B locally for inference with model sharding (splitting parts of the model across different GPUs) should be possible with the setup in the video (depending on what GPUs you choose to add of course). Especially if you are okay with some level of quantization, I think it should be highly manageable on this rig. I will try my best to do some experimentation here very soon and make a video on the results. You are not the only who is very interested in this use case. Yes. Lol. This server is loud indeed. It is probably the loudest one I have ever worked with. The noise is not unbearable (for me at least anyway) unless the machine is booting or under heavy load (running 4+ GPUs at max capacity at once). I would definitely not suggest putting it anywhere you will be working all the time as it could get annoying especially if you are constantly running experiments. What I normally do is if I need to work next to the server for extended periods of time, I just wear headphones (mine are over the ear and noise canceling, but any should probably help). This might help you if you don't have any other options, but to work in the garage with the servers frequently.

@@TheDataDaddi thank you so much for responding, great stuff, great channel.

Hi there! So glad you enjoyed the video! I think both are good options, but I think it depends on your use case and the size of VRAM that you need. The VRAM size of the 4060 ti will be enough for most ML/DL outside of some diffusion related and LLM stuff. If you are wanted to go for larger models then the larger VRAM of the 3090 might be worth the extra money. Personally, I love the 3090s, and I normally always recommend getting the most VRAM you can afford. However, if you are not planning on using the extra VRAM then the 4060 ti will also be a great option. Hope this helps!

Hi there. So glad to hear you are enjoying the content! This is the rack I purchased (link below). I have 3 2U servers, the 4U 4028GR-TRT, a 1U switch, and a 1U PSU. It has worked really and been very solid. However, it is a bit expensive. You could probably find something cheaper possibly on EBAY or Facebook market place. www.ebay.com/itm/134077093756

@@TheDataDaddi thanks, can you pls recheck the link you shared?

@@wasifmasood969 Opps sorry. Wrong link. Here is the correct one. a.co/d/18HuxQG

Hi there. Thanks for the question! Yes, the same cabling should work for the K80s as well.

Hi there. Thanks for the comment! I actually just had an in-depth conversation with one of my viewer recently who when the AMD route. He basically said there was a steep learning curve and still a good number of bugs. That said from what I understand AMD GPUs are becoming more viable alternatives for particular AI/ML/DL applications. At this stage though, I still think its too early for most people. I still believe NVIDIA GPUs are the way to go for now.