@@AZisk Go for the Executive tier at $130. If you don't earn enough from the 2% cash back to make up for the extra cost over the $65 Gold tier, Costco will refund the difference at the end of the year. Also factor in the extra year of warranty Costco gives...
Thank you Alex, I was thinking of doing this exact project! One note: by using a hub you are creating a star topology with a 40Gbps bottleneck shared between all machines. If you used a partially meshed ring topology, you could connect each mini to 3 other minis with a connectivity set of 1:{5,2,3} 2:{1,3,4} 3:{2,4,1} 4:{3,5,2} 5:{4,1,2}. I'd be interesting in seeing if this improved performance. Another potential advantage of the mini cluster vs a single M4 Max is that all M4-series chips have the same 16 ANE cores; you might be able to run distributed inference on the neural engine to benefit from that scaling.
thanks a lot! definitely something to consider. however, the 40Gbps is per thunderbolt controller, and I thought that each port had a separate controller - they do not. Very good observation!!
@@AZisk Ahh I had been wondering why it wasn't just daisy chained, bc that was one of the things Thunderbolt was sold on originally. Would only use 1 port each, but yeah if they. would all share the same bandwidth, that might not work out like we wanted.
The EXO cluster can only expand available memory, not improve inference performance, because each token needs to be processed sequentially on every Mac mini. For a single request, parallel processing is not possible. For example, if you have one Mac mini with 64GB of RAM, its processing performance would be better than two Mac minis with 32GB each.
@@razor-b2d unrealistic? Some companies are already doing it, hence why some people had issue with the power button being switched, as some companies have server racks with the Mac minis.
Not a bad idea. I think Mac OS Server failed because of timing. Apple was not as popular with the Dev community as today. The Mac Mini is probably the most popular device for DevOps. Even AWS bought 1000s and rack them. (Yep you can run Mac OS in AWS). It should not be too complicated for them to use the MacPro and make it rackable. 1U unit, shallow depth, with necessary ports to create clusters. I think Alex is onto something. Clusters make sense, they provide redundancy. I hope Apple is watching and paying attention. Alex what would be interesting is the application side. What could these rackable Mac Mini be used for apart from running LLM test? The footprint and low power consumption is great. Need to find the market for it. Any ideas?
For the bottom mac being hotter, try giving it space under it like you have the others above, might be heat soaking because it cant dissipate the heat like the others can
Exolabs actually processes the parts of the model segmented onto each Mac sequentially, not in parallel which means it's slower than it would be running it on one machine with a lot of ram due to connection delay. However, if Exolabs supported Mixture of Experts models and allowed the experts to be split between devices, that would give insane performance when using all experts compared to doing it on one device because all the experts could be run in parallel.
@blisphul8084 Yes, but even if you only have 8 computers, you can still use the model with two experts per computer (if you have enough ram). You could also just elect to use fewer experts, and the result wouldn't be that much lower quality. You could also have all the experts loaded onto separate computers and still only actively use a couple of experts per token if you want to use less electricity. To clarify, "using fewer experts" only refers to how many experts are activated at a time for each individual token, MoE models still generally use all the experts across multiple tokens, so you need them all loaded somewhere.
@@blisphul8084 Not as much, MoE are already fast because they are sparse models and do not activate all their weights in every run, so they run pretty fast in a single computer already. The problem is fitting them into memory because all the weights have to be on memory (you do not know in advance which will be used). For example a 8x7b mixtral has the speed of 14b model but needs as much RAM as a 56b model. Maybe you could bring it down to 7b speed if all this worked like that, but it would be a 2x increase in speed not 8x. But such a set up would allow for running bigger models that would not fit into RAM normally.
To add onto this: Evaluating the performance of distributed systems is hard. Pipeline Parallel inference: This involves processing one layer on one device, and the next layer on the next, and so on. It’s probably the most intuitive form of parallelism, and means that your system’s RAM is essentially the sum of all constituent components. Note, that your total token/s will be equal to either the slowest machine, or the network bandwidth (whichever comes first). Asynchronous (Batched) Parallel Inference (Exo goes here): This is like PPI, but you are able to execute asynchronously, which lets you add together the bandwidth of your systems, potentially using batches, which lets you increase your token/s by essentially adding together your device’s compute, but it does incur a latency penalty, and each additional device makes the latency worse. Pretty good if you have a huge batch of prompts to go through or something. Lower bandwidth between devices doesn’t decrease token/s but does worsen latency, to my knowledge. Tensor Parallel: Matrix multiplies aren’t interdependent. What this means is that if I calculate the product of the first, third, fifth, and seventh rows on one device, I can calculate the second, fourth, sixth, and eighth on another device, and then just synchronize the product at the end. There’s a bit more to it (you can optimize communications by not synchronizing and actually just sending the products directly to the next layer), but that’s the basics. This requires less bandwidth than pipeline parallelism for LLMs at least (once the weights are loaded, at least), and does let you add together the bandwidth, capacity, and compute of the constituent devices, but it does have two issues. 1) You need a power of 2 number of devices for most tensor parallel implementations, and you also run into issues with certain calculations, like soft max, which require synchronization, which limits how wide a tensor parallel setup can go. It does improve latency, and improves throughput, though, so it’s pretty based. If you have two GPUs, you could run a model literally just twice as fast as not doing tensor parallel, and the same goes for CPUs. I think beyond 4 devices you have to think pretty carefully about how fast the interconnect is, though. Linear Transformers scale way better, so hypothetically if you had like, 64 CPUs for some reason you could probably get pretty close to linear performance improvements for each CPU in the network. Asymmetrical tensor parallel: I’ve never seen this, but in theory it’s possible. You don’t necessarily have to split the tensors evenly between devices. If you have a slower device (ie: CPU) you should still be able to offload, say, 10-20% of the model to that device, while doing the rest on GPU, which doesn’t sound great, but keep in mind it gives you a 10-20% speed up compared to just GPU, and it also reduces the VRAM required by that GPU. This means that you could probably run a 28GB model on a 24GB GPU, for instance, and run it a little bit faster than if you had the same GPU with 28GB of VRAM. Neat. Hybrid: If you have a tensor parallel setup configured as one device and then a second step in the pipeline going to another device, this would be a hybrid setup. To my knowledge no frameworks support this out of the box and would require some careful setup that would probably be very custom to your specific topology. You would expect the latency characteristics to be roughly equal to the ratio of tensor parallel and pipeline parallel components in the network. I guess it would make the most sense if you’re trying to pair several small devices (ie: 4 Raspberry Pi’s) with a more powerful device like a GPU. I’m guessing there’s probably other types of parallelism out there, and there’s also probably some learned parallelism types that might substitute elements of the communication with learned neural networks, or such, but to my knowledge those are the main types for Transformers. CNNs are where parallelism gets **really** fun, though.
A Mac Mini cluster with MLX and ExoLabs makes sense if you like extending context for the models. If you just need the model for a "Hello" or "Write a story" query, a single machine would be sufficient for the task.
I think the argument is that mini cluster with MLX and ExoLabs would not increase inference speed because inference are still done sequentially? But its definitely make running super big models at least possible with the scalability of memory resources
Correct me if I am wrong. According to your video: 1.If the LLM model can fit in just one Mac mini pro, we will get maximum tokens per second. Adding more Mac minis actually decreases TPS, rather than increasing it. 2.If the LLM model is too big for a single Mac mini, the only way to run the model is by using an exo cluster. However, in this case, the TPS will be very low. So I am wondering wouldn't it be better to buy a MacBook Pro M4 Max with 128GB RAM instead of 5 Mac minis? It might be cheaper, and the performance would be much better than an exo cluster with Mac minis (maybe 2 or 3 times on TPS?) no offense, just want to understand
Should totally be possible, the config might require some clever hackishness though, but then you might even get link aggregation to work and include WiFI and BT just for the halibut.
CUDA clusters have 'unified' memory between GPUs thanks BlueField DPUs, so one GPU could be connected to 1TB of memory easily. It's important to have all GPUs working ON THE SAME MEMORY during training. Not all models are compatible with clusters & separate memory spaces.
This kind of investigation is great and truly valuable. It’s a significant contribution to the community. This could become a rabbit hole once you start running tests. Thanks for sharing!
Can you run more tests and show a chart with the results per $ spent? We are trying to figure out if it makes more sense to buy 4 base models in a cluster vs spending the same amount on a maxed out m4 pro or max.
If you’d like a quick overview that will be roughly correct but without a nice looking chart: Apple products are pretty well balanced between Compute, and Memory Bandwidth for the purposes of LLMs. In other words, you can basically look at the memory bandwidth per second, divide that by the cost, and that gets you your price to performance. You can test it, but it will match up quite closely with tokens per second. If your goal is to get the most tokens per second, that’s fine, but there’s other considerations. If you need to run larger models and don’t really care as much about the speed (ie: You need to ask an oracle a really important question every now and then), your best price to performance will be consumer PCs with maxed out RAM configs, running on CPUs. If you need to run at the most tokens per second, then usually GPUs or specialized accelerators (Tenstorrent) will give you orders of magnitude better performance. I will note that a hybrid approach is possible where the CPU runs the oracle that does the overall planning and analysis while the smaller fast models essentially take the “monkeys and Shakespeare” approach and just generate quickly until they get it. This works surprisingly well, don’t knock it until you’ve tried it. The final case is where you need reasonably fast tokens per second in the largest possible model. This is where Apple’s Silicon makes sense, but do keep in mind it won’t be cheap no matter how you slice it. The exact best deal will depend heavily on how much RAM you need for your model specifically, and there’s multiple points of overlap. Again, your best shot is to take the memory capacity you need for your model, and to measure that against every possible config that can run it, and then divide that against the memory bandwidth to cost ratio of each valid configuration. Rather awkwardly, you’ll run into a slightly different answer for every model size.
I’ve always imagined trying LLM clustering using a Mac, and you’ve turned that imagination into reality-I’m so thrilled! Additionally, if there are any Thunderbolt network issues, it would be great to create a video showcasing how to use a 10G network instead. This video is incredibly valuable and practical. Thank you so much for your hard work in making it. I truly appreciate it! ❤❤❤
i think this will be slower, lets say machine A wants to communicate with machine E then the information has to go through B, C and D. Correct me if I'm wrong
The performance problem of the cluster is related to the network bandwidth. Even PCIe cannot effectively handle the speed of data exchange, which is why NVLink was introduced.
The main issue I have right now is that most libraries, papers, or any production models, like SAM, generative 3d geometry, etc, they are all built for CUDA, at work we try to explore every model, but when we try to use mac the answer most of the time is to "fall back to CPU" which makes it unusable and slow, LLM stuff is like hello world at this point lol, really cool to see your experiments :)
Awesome bit of bench work Alex. You always amaze me how you make things can be hard to unpack and make them wholly engaging and great opportunities for learning. Keep. It. Up.
It's interesting that Exo stunts performance over MLX so much but is more convenient to use. If you were going to get a mini to supplement your 128GB M4 Max MacBook, would you just go for the base model mini at 24GB? At $800 its 60% cheaper than the 4090.
Question: 🙋🏻♂️is it possible to “daisy chain” the thunderbolt? I remember seeing a video of servers having their 10gbps Ethernet in daisy chain configuration to faster their replications and backups
Hey man I have a question, I want to make my own personal AI assistant on my own pc but I honestly don’t have much of an idea of how to go about this, so I wanted to see if you would be willing to help me at least get a basic LLM started and possibly teach me how to starting training it myself and then I can take it from there. Or if anyone here in the comments has experience with machine learning and has some time to help me out and is willing to take on this project with me I’d really appreciate it!
I am a PhD student conducting research in computer vision. I would like to ask about the performance of using a Mac Mini cluster for computer vision tasks such as diffusion, object detection, and image segmentation. The video mainly demonstrates inference. If it were used for training, how would its performance compare to NVIDIA GPUs? Additionally, I noticed that MLX also has code applications for computer vision.
Thank you for sharing the cluster idea Alex. I was excited just watching you try the different configurations! Definitely interested in any further networking experiments. Thanks again!
Brilliant Alex!!! Thank you for doing this! To be expected that the thunderbolt hub slows things down.... too much contention. I think we will have to wait for M4 ULTRA with 128/192Gb. Maybe worthwhile to run 2x M4 Ultra using EXO.
The Jumbo packets are more about using less bandwidth for protocol overhead. The TCP/IP header (20 - 40 Bytes), then the payload. So it's saving about 84 bytes if you are filling every jumbo packet. Great for sending a lot of data quickly. At 40Gb/s and assuming we get all of the pipe, you are saving like ~40MB/s in TCP packet overhead. Nothing to sneeze at.
@@Makronauta The MTU is the max transmission size. If the frame isn't full, it doesn't actually transmit a bunch of white space nulls. It only sends what it needs to. Having a constant MTU size is more about not requiring the edge router / switch from having to convert from 9000 MTU size, and splitting them up into 1500 MTU size packets costing CPU time. If you send a 9000 sized MTU packet and it's less than a 1500 sized packet in the payload, it will not need to spit the packet.
I think another part of the reason the lowest unit is the hottest is that it is closest to the table. It has the least airflow underneath and I also suspect the metal chassis is absorbing more heat here.
Just curious. But wouldn't a raspberry pi cluster or PC Mini cluster (running linux) be cheaper? What is the advantage of a mac cluster? I understand wanting to develop on Macs, but why a cluster?
Why did you just overlook MLX by itself? (09:40) I would have greatly appreciated more testing + data, because at 280 tokens, it was smoking all the other setups; combined or not. To say, that might conclude that the hardware might not be the issue, just your testing procedure / exo. Thx anyhow.
Fun fact, WiFi ethernet packets are up to 64kB, by default, with my 2020 wifi 6 router. When streaming videos with DLNA protocol it uses maximum length WiFi ethernet packets.
wouldn't it be possible to connect the 1st to the 2nd, then the 2nd to the 3rd, the 3rd to the 4th and finally the 4th to the 5th instead of using a hub?
I run Qwen 2.5 Coder 32B on my macbook pro without a cluster and its extremely fast and doesnt seem to be an issue at all. I just use run it with MLX 4bit. On a m3 Max 48gb, 16-Core CPU, 40-Core GPU.
Very cool Alex! It would be very interesting to see the same topology with a Mac Studio or Pro, M2 Ultra be the hub of the cluster, with it's 800 GB/S bandwidth and eight Thunderbolt 4 ports. Great content!
Do you think to make,in the future, a cluster of M4 for rendering? If Blender support it, to understand the distance in Gpu performance? Or even some problematic high video decoding as happen with the Canon Cinema Raw?
Very interesting video and set up. Maybe the bottom Mac runs hotter because if doesn't have the same vertical space to push out air as the suspended macs above have.
Try setting up the cluster in a hierarchical scheme, I've had experience with distributed computing and while this adds CPU overhead, model workloads are primarily GPU, so it doesn't affect running the models. One master node, 3 Sub-Master Nodes, followed by upto 3 worker nodes each per sub-master node. I want to see what kind of workloads it can accomplish
A few years ago I myself personally believed in this idea, I dreamed of the concept of a mini PC with very powerful APUs that (ideally) will be in the form of cells that can be connected to each other and like honeycomb and thus automatically create clusters and having collected a whole cabinet of such you can organize a home computing lab. I think the M4 is the first step towards such a future because of its balance of performance and very small size. Thank you for the video (I got a question, if you compare this computing to the analog in the form of GPU, how much would it be in price?)
You will need to quantise it, it needs more than 64gb ram. Then at q8 quantisation it needs around 32gb of ram, so you do not need 64gb ram in the first place and with 48 you should be good. I bet it is still gonna be slow, unless you quantise it further, in which case 64gb is even less needed. But in general even if it was 60gb and it fit there, it would not work well. M4 pro does not have the TFLOPS nor memory bandwidth to really support running such large models on a machine. It should be simple math if anybody took the time to do them. Wait for a mac studio max, which is prob gonna be slightly more expensive. Or buy a bunch of base macminis once one has figured out how to solve the network congestion issues, 3-4 m4s should end up being faster than 1 m4 pro (in the video 2 m4s were around the performance of one m4 pro, which makes sense in terms of TFLOPS and memory bandwidth).
Just to make sure I get it. All of this is good if what we are looking for is to have our own LLM running in our own devices, right? I mean, renting GPU cloud at AWS or at any other service provider is usually cheaper and the power you can reach is almost limitless at a fraction of the cost at least in the short run. Right?
M4 mac mini Daisy chaining /clustering -->> Does it improve any extreme heavy video editing (especially the ones a single standalone unit M4 mac mini can not handle)!!!
Not sure if this is possible with ThunderBolt bridge setup, but can you create a mess network, where each of the four Mac Minis are connected to each other. Ideally, all three TB ports on the back could be ganged together as one interface with a single IP address, so you can still use the same setup with with each Mac having a single IP address in the cluster. Hoping that this helps improve direct node to node communication.
Hot air go on top,my guess is that the bottom machine is the hottest more for an inefficiency over the distributions of the load from the software plus certain tasks require to be running on single core. So probably an M4 put on the bottom to replace the M4 Pro position would result in the hottest too,maybe just a little less than the M4 Pro.
The good thing though about exo is that you can plug in just whatever of the new ecosystem. So if you have the m4 128 that needs more you can provide that through some n-tuple of mac minis. Or you have multiple workstations for an office and your need for intensive local inference is rare. Or in future when we all unnecessarily buy M5 devices 😢
off topic but do you recommend the m4 12-core pro MacBook Pro instead of the 14-core pro? better battery life? can do video editing, music production and coding? keep for 5 years at least, what are your thoughts? I will get the 48gb ram and 2tb ssd
I have a Data Modeling and Internet Programming course this coming spring semester. I think this would be a great project for the students. Once a cluster is setup, can one machine be setup to be a web server so a private network of users can ask questions. Would love to train a cluster model for my CS department.
Why didn’t you use those usb-c ports in front to link, their speed is still 10GB/sec, unless I missed something? Dose thunderbolts hubs have all thunderbolt ports to make a chain without breaking speed of data transmission?
Instead of using a hub, isn‘t it possible to setup a daisy-chained thunderbolt bridge? What would be the network contention with that setup if possible at all?
Wouldn't the use case be to fit very large models into maxed out RAM Apple silicone? As in, a quantized version of Llama 405b running on 4 connected machines with 128GB of RAM each? ChatGPT tells me that an 8-bit version of 405b needs 486 GB of VRAM to run. Given that the A100s top out at 80GB of RAM, you'd need, what, 6 or 7 of those to run the same quantized model locally? Am I anywhere near a reasonable idea or am I misunderstanding something?
The M4 Mac Mini is discounted NOW: ⚡ amzn.to/3CJnjds
Costco has them for $500
@@vinfasi don’t have a Costco membership. How much is that?
@@AZisk $65 in the US. BTW, Costco has the M4/24/512 model for $889.
@@AZisk Go for the Executive tier at $130. If you don't earn enough from the 2% cash back to make up for the extra cost over the $65 Gold tier, Costco will refund the difference at the end of the year.
Also factor in the extra year of warranty Costco gives...
Not true, backordered is way different than discontinued. They’re just backordered now due to popularity.
Thank you Alex, I was thinking of doing this exact project! One note: by using a hub you are creating a star topology with a 40Gbps bottleneck shared between all machines. If you used a partially meshed ring topology, you could connect each mini to 3 other minis with a connectivity set of 1:{5,2,3} 2:{1,3,4} 3:{2,4,1} 4:{3,5,2} 5:{4,1,2}. I'd be interesting in seeing if this improved performance. Another potential advantage of the mini cluster vs a single M4 Max is that all M4-series chips have the same 16 ANE cores; you might be able to run distributed inference on the neural engine to benefit from that scaling.
thanks a lot! definitely something to consider. however, the 40Gbps is per thunderbolt controller, and I thought that each port had a separate controller - they do not. Very good observation!!
@@AZisk Ahh I had been wondering why it wasn't just daisy chained, bc that was one of the things Thunderbolt was sold on originally. Would only use 1 port each, but yeah if they. would all share the same bandwidth, that might not work out like we wanted.
I'd be super interested in seeing this
Hehehehe this brought me back to the old times where I come from... old 1 mbit network setups... damn at that time it was amazing...
The EXO cluster can only expand available memory, not improve inference performance, because each token needs to be processed sequentially on every Mac mini. For a single request, parallel processing is not possible. For example, if you have one Mac mini with 64GB of RAM, its processing performance would be better than two Mac minis with 32GB each.
MacOS servers should be a thing again with M model chips honestly
for sure
Unrealistic
just give me a good linux integration
@@razor-b2d unrealistic? Some companies are already doing it, hence why some people had issue with the power button being switched, as some companies have server racks with the Mac minis.
Not a bad idea. I think Mac OS Server failed because of timing. Apple was not as popular with the Dev community as today. The Mac Mini is probably the most popular device for DevOps. Even AWS bought 1000s and rack them. (Yep you can run Mac OS in AWS).
It should not be too complicated for them to use the MacPro and make it rackable. 1U unit, shallow depth, with necessary ports to create clusters.
I think Alex is onto something. Clusters make sense, they provide redundancy.
I hope Apple is watching and paying attention. Alex what would be interesting is the application side.
What could these rackable Mac Mini be used for apart from running LLM test?
The footprint and low power consumption is great. Need to find the market for it. Any ideas?
For the bottom mac being hotter, try giving it space under it like you have the others above, might be heat soaking because it cant dissipate the heat like the others can
According to the Apple video, the air flows in and out of the bottom, so this makes sense.
Or just lay the rack on its side?
@@rp_npc This is the way. Also what MacStadium does as well.
2000$ in computers, 500$ in Thunderbolt cables.
I like the narrative style without getting too deep into the technical weeds and letting the screen do that talking.
Exolabs actually processes the parts of the model segmented onto each Mac sequentially, not in parallel which means it's slower than it would be running it on one machine with a lot of ram due to connection delay. However, if Exolabs supported Mixture of Experts models and allowed the experts to be split between devices, that would give insane performance when using all experts compared to doing it on one device because all the experts could be run in parallel.
So essentially, if you have a 16 expert model, you can get speed ups until you hit 16 computers?
@blisphul8084 Yes, but even if you only have 8 computers, you can still use the model with two experts per computer (if you have enough ram). You could also just elect to use fewer experts, and the result wouldn't be that much lower quality. You could also have all the experts loaded onto separate computers and still only actively use a couple of experts per token if you want to use less electricity. To clarify, "using fewer experts" only refers to how many experts are activated at a time for each individual token, MoE models still generally use all the experts across multiple tokens, so you need them all loaded somewhere.
Except for training, aren’t most tasks sequence dependent?
@@blisphul8084 Not as much, MoE are already fast because they are sparse models and do not activate all their weights in every run, so they run pretty fast in a single computer already. The problem is fitting them into memory because all the weights have to be on memory (you do not know in advance which will be used). For example a 8x7b mixtral has the speed of 14b model but needs as much RAM as a 56b model. Maybe you could bring it down to 7b speed if all this worked like that, but it would be a 2x increase in speed not 8x. But such a set up would allow for running bigger models that would not fit into RAM normally.
To add onto this: Evaluating the performance of distributed systems is hard.
Pipeline Parallel inference: This involves processing one layer on one device, and the next layer on the next, and so on. It’s probably the most intuitive form of parallelism, and means that your system’s RAM is essentially the sum of all constituent components. Note, that your total token/s will be equal to either the slowest machine, or the network bandwidth (whichever comes first).
Asynchronous (Batched) Parallel Inference (Exo goes here): This is like PPI, but you are able to execute asynchronously, which lets you add together the bandwidth of your systems, potentially using batches, which lets you increase your token/s by essentially adding together your device’s compute, but it does incur a latency penalty, and each additional device makes the latency worse. Pretty good if you have a huge batch of prompts to go through or something. Lower bandwidth between devices doesn’t decrease token/s but does worsen latency, to my knowledge.
Tensor Parallel: Matrix multiplies aren’t interdependent. What this means is that if I calculate the product of the first, third, fifth, and seventh rows on one device, I can calculate the second, fourth, sixth, and eighth on another device, and then just synchronize the product at the end. There’s a bit more to it (you can optimize communications by not synchronizing and actually just sending the products directly to the next layer), but that’s the basics. This requires less bandwidth than pipeline parallelism for LLMs at least (once the weights are loaded, at least), and does let you add together the bandwidth, capacity, and compute of the constituent devices, but it does have two issues. 1) You need a power of 2 number of devices for most tensor parallel implementations, and you also run into issues with certain calculations, like soft max, which require synchronization, which limits how wide a tensor parallel setup can go. It does improve latency, and improves throughput, though, so it’s pretty based. If you have two GPUs, you could run a model literally just twice as fast as not doing tensor parallel, and the same goes for CPUs. I think beyond 4 devices you have to think pretty carefully about how fast the interconnect is, though. Linear Transformers scale way better, so hypothetically if you had like, 64 CPUs for some reason you could probably get pretty close to linear performance improvements for each CPU in the network.
Asymmetrical tensor parallel: I’ve never seen this, but in theory it’s possible. You don’t necessarily have to split the tensors evenly between devices. If you have a slower device (ie: CPU) you should still be able to offload, say, 10-20% of the model to that device, while doing the rest on GPU, which doesn’t sound great, but keep in mind it gives you a 10-20% speed up compared to just GPU, and it also reduces the VRAM required by that GPU. This means that you could probably run a 28GB model on a 24GB GPU, for instance, and run it a little bit faster than if you had the same GPU with 28GB of VRAM. Neat.
Hybrid: If you have a tensor parallel setup configured as one device and then a second step in the pipeline going to another device, this would be a hybrid setup. To my knowledge no frameworks support this out of the box and would require some careful setup that would probably be very custom to your specific topology. You would expect the latency characteristics to be roughly equal to the ratio of tensor parallel and pipeline parallel components in the network. I guess it would make the most sense if you’re trying to pair several small devices (ie: 4 Raspberry Pi’s) with a more powerful device like a GPU.
I’m guessing there’s probably other types of parallelism out there, and there’s also probably some learned parallelism types that might substitute elements of the communication with learned neural networks, or such, but to my knowledge those are the main types for Transformers.
CNNs are where parallelism gets **really** fun, though.
Thanks
hope you enjoyed. thanks so much
Thanks! Very interested in how this all plays out.
Glad you enjoyed it! and thanks a lot!
A Mac Mini cluster with MLX and ExoLabs makes sense if you like extending context for the models.
If you just need the model for a "Hello" or "Write a story" query, a single machine would be sufficient for the task.
@@MrBlogbar but…. Speeeeeeeeeeeeeed
I think the argument is that mini cluster with MLX and ExoLabs would not increase inference speed because inference are still done sequentially? But its definitely make running super big models at least possible with the scalability of memory resources
Can’t wait for the MServe(Xserve)
Correct me if I am wrong. According to your video:
1.If the LLM model can fit in just one Mac mini pro, we will get maximum tokens per second. Adding more Mac minis actually decreases TPS, rather than increasing it.
2.If the LLM model is too big for a single Mac mini, the only way to run the model is by using an exo cluster. However, in this case, the TPS will be very low.
So I am wondering wouldn't it be better to buy a MacBook Pro M4 Max with 128GB RAM instead of 5 Mac minis? It might be cheaper, and the performance would be much better than an exo cluster with Mac minis (maybe 2 or 3 times on TPS?)
no offense, just want to understand
Isn't that what he says at the end of the video?
Its exactly what he says at the end of the video.
For batch size 1 (= one request at a time), yes. However, adding more Mac minis will improve TPS if you are dealing with higher batch size.
@@philippe.139 I see, parallel requests will be processed simultaneously, thanks
@philippe.139 a MacBook has better performance than a desktop MacPro? Or is it about performance per dollar?
Can you daisy chain the thunderbolt connections? This would eliminate the hub.
I thought that way to love to know the answer
Should totally be possible, the config might require some clever hackishness though, but then you might even get link aggregation to work and include WiFI and BT just for the halibut.
@@mpsii surprised this wasn’t his solution. There’s precedent for this.
@@abb0tt good to know, I assumed you could daisy chain via Thunderbolt but did not know for sure
@@JeremyAndersonBoise google "High-speed 10Gbps full-mesh network"
CUDA clusters have 'unified' memory between GPUs thanks BlueField DPUs, so one GPU could be connected to 1TB of memory easily.
It's important to have all GPUs working ON THE SAME MEMORY during training.
Not all models are compatible with clusters & separate memory spaces.
Yeah pretty sure multiple NVidia cards connected with nvlink will be a lot better then this solution
Obviously but not all of us are millionaires
@ it doesn’t change the fact that most downloaded model can’t work in clusters
@@FuZZbaLLbee Of course, and as he said, an Nvidia H100 would be better, but this was a bang for the money kind of test.
@@BiP00 I was thinking multiple second hand 3090s connected via NVlink
This kind of investigation is great and truly valuable. It’s a significant contribution to the community. This could become a rabbit hole once you start running tests. Thanks for sharing!
Finally we get to see the results! You've hinted and showed the racked minis in many prior videos, I was going mad!
You could really see the joy on you face in this video. Like a kid in a candy store. :)
it was a fun one for sure
@@AZisk this video is a porn for geek!
Thank you for exploring this so expertly.
Can you run more tests and show a chart with the results per $ spent? We are trying to figure out if it makes more sense to buy 4 base models in a cluster vs spending the same amount on a maxed out m4 pro or max.
If you’d like a quick overview that will be roughly correct but without a nice looking chart:
Apple products are pretty well balanced between Compute, and Memory Bandwidth for the purposes of LLMs. In other words, you can basically look at the memory bandwidth per second, divide that by the cost, and that gets you your price to performance. You can test it, but it will match up quite closely with tokens per second.
If your goal is to get the most tokens per second, that’s fine, but there’s other considerations.
If you need to run larger models and don’t really care as much about the speed (ie: You need to ask an oracle a really important question every now and then), your best price to performance will be consumer PCs with maxed out RAM configs, running on CPUs.
If you need to run at the most tokens per second, then usually GPUs or specialized accelerators (Tenstorrent) will give you orders of magnitude better performance. I will note that a hybrid approach is possible where the CPU runs the oracle that does the overall planning and analysis while the smaller fast models essentially take the “monkeys and Shakespeare” approach and just generate quickly until they get it. This works surprisingly well, don’t knock it until you’ve tried it.
The final case is where you need reasonably fast tokens per second in the largest possible model. This is where Apple’s Silicon makes sense, but do keep in mind it won’t be cheap no matter how you slice it. The exact best deal will depend heavily on how much RAM you need for your model specifically, and there’s multiple points of overlap. Again, your best shot is to take the memory capacity you need for your model, and to measure that against every possible config that can run it, and then divide that against the memory bandwidth to cost ratio of each valid configuration.
Rather awkwardly, you’ll run into a slightly different answer for every model size.
I’ve always imagined trying LLM clustering using a Mac, and you’ve turned that imagination into reality-I’m so thrilled! Additionally, if there are any Thunderbolt network issues, it would be great to create a video showcasing how to use a 10G network instead. This video is incredibly valuable and practical. Thank you so much for your hard work in making it. I truly appreciate it! ❤❤❤
Booyy ooo boyyy ooo boyyyyy!! This is one heck of a video about M4 mac mini! Such a creative video man! Loved it!
thx!
I loved it too!
If it's Thunderbolt Bridge, can you try daisy-chaining Thunderbolt connections to get rid of the bridge?
@@Alexthesurfer this 👍🏻
i think this will be slower, lets say machine A wants to communicate with machine E then the information has to go through B, C and D. Correct me if I'm wrong
@@zeeshanrabbani8125 you're not wrong
But star connect is possible with 4 macs. I think he mentioned that and I think that would get rid of the bridge.
I was thinking of the sane thing
The performance problem of the cluster is related to the network bandwidth. Even PCIe cannot effectively handle the speed of data exchange, which is why NVLink was introduced.
Great demonstration of what stack of Mac Mini's is capable of.
The main issue I have right now is that most libraries, papers, or any production models, like SAM, generative 3d geometry, etc, they are all built for CUDA, at work we try to explore every model, but when we try to use mac the answer most of the time is to "fall back to CPU" which makes it unusable and slow, LLM stuff is like hello world at this point lol, really cool to see your experiments :)
Lm studio on M1-M4 should work great
Awesome bit of bench work Alex. You always amaze me how you make things can be hard to unpack and make them wholly engaging and great opportunities for learning. Keep. It. Up.
It's interesting that Exo stunts performance over MLX so much but is more convenient to use. If you were going to get a mini to supplement your 128GB M4 Max MacBook, would you just go for the base model mini at 24GB? At $800 its 60% cheaper than the 4090.
Question: 🙋🏻♂️is it possible to “daisy chain” the thunderbolt? I remember seeing a video of servers having their 10gbps Ethernet in daisy chain configuration to faster their replications and backups
I just bought mine today! I connect it to my 100 inch Samsung smart TV mounted to my bedroom wall! It's awesome! I love it so much! 🥰
I was waiting for this video for so long, thanks Alex. ❤
me tooo
Me 3 😂
I commented this on an earlier video, but please compare two 32GB minis running on exo vs. one 64gb mini. Because both setups cost the same amount.
The bottom computer in the rack has less ventilation. Increase the space and see if you get cooler results.
Font rendering on Mac OS is so smooth!!!
Hey man I have a question, I want to make my own personal AI assistant on my own pc but I honestly don’t have much of an idea of how to go about this, so I wanted to see if you would be willing to help me at least get a basic LLM started and possibly teach me how to starting training it myself and then I can take it from there. Or if anyone here in the comments has experience with machine learning and has some time to help me out and is willing to take on this project with me I’d really appreciate it!
I am a PhD student conducting research in computer vision. I would like to ask about the performance of using a Mac Mini cluster for computer vision tasks such as diffusion, object detection, and image segmentation. The video mainly demonstrates inference. If it were used for training, how would its performance compare to NVIDIA GPUs? Additionally, I noticed that MLX also has code applications for computer vision.
@10:30 I wonder if you flip the macs horizontally, may it better?
Thank you for sharing the cluster idea Alex. I was excited just watching you try the different configurations! Definitely interested in any further networking experiments. Thanks again!
Not all the windows opened on the screen. Great 👌 video sir Alex
Brilliant Alex!!! Thank you for doing this! To be expected that the thunderbolt hub slows things down.... too much contention. I think we will have to wait for M4 ULTRA with 128/192Gb. Maybe worthwhile to run 2x M4 Ultra using EXO.
M4 Ultra will have 256GB. There's also supposed to be a chip coming that'll supersede the M4 Ultra with 512GB of memory.
its not particularly about parallel processing but vector operations
11:11 just put it above the others and you'll get the answer
Best video in regards of M chips and machine learning/LLM. Keep it up. I love those kind of videos!!!
The Jumbo packets are more about using less bandwidth for protocol overhead. The TCP/IP header (20 - 40 Bytes), then the payload. So it's saving about 84 bytes if you are filling every jumbo packet. Great for sending a lot of data quickly. At 40Gb/s and assuming we get all of the pipe, you are saving like ~40MB/s in TCP packet overhead. Nothing to sneeze at.
If the packets are large enough, otherwise it's just a lot of empty packets to wait for
@@Makronauta The MTU is the max transmission size. If the frame isn't full, it doesn't actually transmit a bunch of white space nulls. It only sends what it needs to. Having a constant MTU size is more about not requiring the edge router / switch from having to convert from 9000 MTU size, and splitting them up into 1500 MTU size packets costing CPU time. If you send a 9000 sized MTU packet and it's less than a 1500 sized packet in the payload, it will not need to spit the packet.
@@Dygear Thanks for the explanation!
What about doing a cluster of two m4 base model Mac mini's as a home computer?
What is a good name for Mac mini cluster supercomputer?
Orchard?
the vents on the bottom one is almost toughing the tables. The other ones have more space to blow hot air out.
I think another part of the reason the lowest unit is the hottest is that it is closest to the table. It has the least airflow underneath and I also suspect the metal chassis is absorbing more heat here.
Just curious. But wouldn't a raspberry pi cluster or PC Mini cluster (running linux) be cheaper? What is the advantage of a mac cluster? I understand wanting to develop on Macs, but why a cluster?
Is it possible to connect in Ring topology using Thunderbolt ?
Why did you just overlook MLX by itself? (09:40) I would have greatly appreciated more testing + data, because at 280 tokens, it was smoking all the other setups; combined or not.
To say, that might conclude that the hardware might not be the issue, just your testing procedure / exo.
Thx anyhow.
the best review ever ! Thanks Alex.
I was waiting for this one for weeks!!
Took me long enough, right?
I’m just happy it’s here 🎉
Fun fact, WiFi ethernet packets are up to 64kB, by default, with my 2020 wifi 6 router. When streaming videos with DLNA protocol it uses maximum length WiFi ethernet packets.
wouldn't it be possible to connect the 1st to the 2nd, then the 2nd to the 3rd, the 3rd to the 4th and finally the 4th to the 5th instead of using a hub?
I run Qwen 2.5 Coder 32B on my macbook pro without a cluster and its extremely fast and doesnt seem to be an issue at all. I just use run it with MLX 4bit. On a m3 Max 48gb, 16-Core CPU, 40-Core GPU.
But if it works on the neural engine it should be much more powerful??
Very cool Alex! It would be very interesting to see the same topology with a Mac Studio or Pro, M2 Ultra be the hub of the cluster, with it's 800 GB/S bandwidth and eight Thunderbolt 4 ports. Great content!
M4pro has a Thunderbolt 5
Do you think to make,in the future, a cluster of M4 for rendering? If Blender support it, to understand the distance in Gpu performance? Or even some problematic high video decoding as happen with the Canon Cinema Raw?
omg, that power draw is actually amazing!
It is possible to make mini cluster to run Adobe Lightroom to run/export faster ?
It would be interesting to see how it would work if each machine had internal 10Gb ethernet, connected through a 10Gb switch.
You could try putting the cluster on it's side and/or place a fan on the back..
Very interesting video and set up. Maybe the bottom Mac runs hotter because if doesn't have the same vertical space to push out air as the suspended macs above have.
This video, the explanation and the tests were so well done. First time on your channel, now I have subscribed. Thank you.
Very interested to see results of support for jumbo packets!
Try setting up the cluster in a hierarchical scheme, I've had experience with distributed computing and while this adds CPU overhead, model workloads are primarily GPU, so it doesn't affect running the models. One master node, 3 Sub-Master Nodes, followed by upto 3 worker nodes each per sub-master node. I want to see what kind of workloads it can accomplish
A few years ago I myself personally believed in this idea, I dreamed of the concept of a mini PC with very powerful APUs that (ideally) will be in the form of cells that can be connected to each other and like honeycomb and thus automatically create clusters and having collected a whole cabinet of such you can organize a home computing lab. I think the M4 is the first step towards such a future because of its balance of performance and very small size.
Thank you for the video
(I got a question, if you compare this computing to the analog in the form of GPU, how much would it be in price?)
install the macs upside down to help thermals?
Impressive - wasn’t sure how performance would scale when clustered. Continue to be impressed with Apple Silicon.
What’s the performance of the mini pro 64GB on qwen 2.5 32b ? That’s the combo I planned on buying. Thanks !
You will need to quantise it, it needs more than 64gb ram. Then at q8 quantisation it needs around 32gb of ram, so you do not need 64gb ram in the first place and with 48 you should be good. I bet it is still gonna be slow, unless you quantise it further, in which case 64gb is even less needed. But in general even if it was 60gb and it fit there, it would not work well. M4 pro does not have the TFLOPS nor memory bandwidth to really support running such large models on a machine. It should be simple math if anybody took the time to do them. Wait for a mac studio max, which is prob gonna be slightly more expensive. Or buy a bunch of base macminis once one has figured out how to solve the network congestion issues, 3-4 m4s should end up being faster than 1 m4 pro (in the video 2 m4s were around the performance of one m4 pro, which makes sense in terms of TFLOPS and memory bandwidth).
Why didn’t you try to run a 400b model?
The grand finale ?
Just to make sure I get it.
All of this is good if what we are looking for is to have our own LLM running in our own devices, right?
I mean, renting GPU cloud at AWS or at any other service provider is usually cheaper and the power you can reach is almost limitless at a fraction of the cost at least in the short run. Right?
M4 mac mini Daisy chaining /clustering -->> Does it improve any extreme heavy video editing (especially the ones a single standalone unit M4 mac mini can not handle)!!!
Not sure if this is possible with ThunderBolt bridge setup, but can you create a mess network, where each of the four Mac Minis are connected to each other. Ideally, all three TB ports on the back could be ganged together as one interface with a single IP address, so you can still use the same setup with with each Mac having a single IP address in the cluster. Hoping that this helps improve direct node to node communication.
The size and power consumption for that compute is amazing.
correct me if i am wrong but having a setting with MLX was 3x faster than Exo ?
Hot air go on top,my guess is that the bottom machine is the hottest more for an inefficiency over the distributions of the load from the software plus certain tasks require to be running on single core.
So probably an M4 put on the bottom to replace the M4 Pro position would result in the hottest too,maybe just a little less than the M4 Pro.
Is the rack 3d printed? Can you share the file if possible?
The good thing though about exo is that you can plug in just whatever of the new ecosystem. So if you have the m4 128 that needs more you can provide that through some n-tuple of mac minis. Or you have multiple workstations for an office and your need for intensive local inference is rare. Or in future when we all unnecessarily buy M5 devices 😢
This configuration for a LLM model will be suitable doing the merge of the machines using Kubernetes?
Please make the same comparison, but between 1 MacBook M4 Pro and a cluster of 2-3 M4 Mac Minis for rendering videos
off topic but do you recommend the m4 12-core pro MacBook Pro instead of the 14-core pro? better battery life? can do video editing, music production and coding? keep for 5 years at least, what are your thoughts? I will get the 48gb ram and 2tb ssd
Would it be possible to daisy chain Macs via thunderbolt? Wouldn’t this network topology allow more elements in the network and at greater speed?
Can you use two thunderbolt ports ?? For 80gbps ??
I have a Data Modeling and Internet Programming course this coming spring semester. I think this would be a great project for the students. Once a cluster is setup, can one machine be setup to be a web server so a private network of users can ask questions. Would love to train a cluster model for my CS department.
Hey , can i use the Mac Mini M4 cluster, for rednering 3d? or is it better only for llm?
Hi Alex!
I want to know the information of the case(for stacking) of mac mini.
Would it have worked better with the 10G ethernet models and a 10G switch?
Can you send the link for the tinychat project?
Great content! Any chance you could create a detailed video or article on setting up a Mac Mini cluster for those of us who are network-challenged?
Hey Alex great video… Do you have a link to the rack that holds the New Mac Minis.
why don't you try a serial setup between the macminis, instead of installing a central machine via bridge?
Why didn’t you use those usb-c ports in front to link, their speed is still 10GB/sec, unless I missed something? Dose thunderbolts hubs have all thunderbolt ports to make a chain without breaking speed of data transmission?
would be interesting to get a comparison with the new Nvidia Jetson considering they consume only 25w
Are these benchmarks using M4 ANE?
Instead of using a hub, isn‘t it possible to setup a daisy-chained thunderbolt bridge? What would be the network contention with that setup if possible at all?
Wouldn't the use case be to fit very large models into maxed out RAM Apple silicone? As in, a quantized version of Llama 405b running on 4 connected machines with 128GB of RAM each? ChatGPT tells me that an 8-bit version of 405b needs 486 GB of VRAM to run. Given that the A100s top out at 80GB of RAM, you'd need, what, 6 or 7 of those to run the same quantized model locally? Am I anywhere near a reasonable idea or am I misunderstanding something?
Whats to stop the 4090 from using the 1tb of ddr5?
Alex you are doing a great job. The best Mac-related channel for devs.
Do you have an airconditioner?
I do believe the heat from top devices and it being the hub is causing it to have a high temperature
Why would someone want to run an LLM at home? What can you do with it?