This is the kind of video we want to see not fucking imbecilistic cinebench runs!! By the way Cinebench is the least fair benchmark for apple silicon!! And yet most people since they are ignorant believe cinebench is the gold standard. M3max is faster than a OC 14700k and rivals a 14900k, beating everything in more memory bound algorithms(for example prime numbers or water wave simulation)
You didn't pin them to the same random seed, so they're generating different text, so it's hard to compare the elapsed time because they're not generating the same number of tokens.
The M1 & the M3 are producing ***almost*** identical text. While from a benchmarking perspective I guess consistency would be ideal, surely the nature of ML is that a process should do what it needs to do to get a workable result; you shouldn't make machines follow the exact same path if one can jump high fences but another can burrow under them. What's important is how fast they complete the task asked of them, not whether they did so in an identical manner.
I was hoping someone would post this kind of comparison. It seems the unified memory is a huge advantage for running larger llms. One thing I did not understand at the final 70b test, what was the amount the memory used by the m3, could you have gotten away with 64gb only instead of 128gb? Thank you for the effort you put in creating and sharing this test. Subscribed!
@@Anderson-dy8mlI doubt because Falcon 180B is quoted as requiring 400gb of VRAM to infer on, which means you'll need at least 5 A100's before it gets off the ground in its original
@@Joe_Brig I’m thinking an M2 Mac Studio with 64gb would be better suited for this , on long term, heard other ppl commenting about the fans getting louder than normal on M3 MacbookPro s when running inference.
I’ve been searching EXACTLY for this! Thank you. Subscribed and looking forward to those next videos on stable diffusion and your grandma’s clone (if I understood well). Thanks bro!
If AI consumes 35GB of RAM with 70 billion parameters, you are left with 29GB of available RAM, and in my opinion, models will become more performant and specialized, similar to Mistral AI models with 7B models that are very performant relative to their size. So, 128GB is a security if you can afford it, but 64GB is sufficient
Nvidia cards are so expensive now. The are blinded by the demand and feel they can get away with anything. I wish they stay in their lalaland for longer while competition develops and pulls the rug from under their feet. The reason Nvidia can get away with this is because most AI/ML software is optimized to run on Nvidia. AMD/Intel/Apple/Qualcomm or a consortium should should build a killer GPU / machine that can slay Nvidia. They need to work with one or two AI projects - Say Stable Diffusion and Llama - optimize these two to run crazy fast on their hardware. Some ridiculous out-performance over Nvidia is needed to shake things up. Only then will AI/ML libraries and ecosystem will consider optimizing for non-Nvidia hardware.
Great idea, but I hoped this would be a better-executed test, with actual benchmark numbers. And that you show which model you used (which quantization). E.g. llama.cpp prints out the token/s numbers at the end, and you can switch between CPU and GPU with parameters on start. With llama-2 single-user inference's response-token generation, the performance largely depends on memory bandwidth, and not very much on GPU-speed. The 4090 has 2.5x the memory-bandwidht of the M3 Max chip (even though its compute is much faster), and 5x vs. the M1/M2 Pro (the M3 Pro is slower in this due to lower bandwidth than an M1/M2). This largely should determine their performance results. I totally agree, that Apple silicon is great a) because of comparatively cheap large GPU-available memory and b) because of approx. 1/10th the power-draw (heat/noise).
@@Teluric2 „nobody“ is wrong, quite a few people care about power-consumption during inferece. AI inference is moving towards the edge. If you want raw power for training, etc., most use a cloud datacenter for AI.
@@Teluric2This is a laptop and everybody should care about the power consumption. This machine is not even meant to compete against a machine with 4090 but apparently there are tasks where it exceeds.
Great comparison! The parameter always gets bigger. Based on your result, I'm thinking the upcoming 5090 won't be my next purchase. More likely M4 with 128GB or 256 RAM will be my next stop.
Youre comparing an apple 128Gb RAM with more than enough space to run 70b with a PC w/ 32Gb ram setup that’s far less than the recommended spec for running 70b which is around 40Gb RAM minimum. Compare the M3 to a PC with 128Gb in Ram. This is comparison is silly. The 128GB PC will be both cheaper to build and youd probably still be able to get a 4090 with the cash left over.
great test. thank you. for all of those working with LLM models and are considering M3's this is what we're looking for. People should understand these are q4 gguf models but still it's a very relevant test and it's good to see the Mac's unified memory working. I'd love to see how would be to run an interface like streamlit along with RAG on instructor XL and ollama or textgen and see if the Max can handle all of them together.
It was interesting to see that the Mac struggled with using that much memory via the GPU, you could easily see that there is a bottle neck that slowed down the system big time.
Would love to see someone test and benchmark MPS (Metal Performance Shaders) for pytorch for training Ml/AI models with a focus on transformers and LLMs. Some of us do more than just inference and the availability of GPUs for doing ML/AI development is just insanely bad.
Potential problems with this comparison: 1. Have you used GGUF format for all models on 4090 instead of GPTQ for smaller models? 2. Have you used AutoGPTQ instead of ExLamav2 (much faster inference) model loader for 4090? 3. Have you used part (up to 51 layers) of GGUF 70B model for 4090 acceleration? Also comparison would be much cleaner if you add tokens/second for all your runs. Other than that, the comparison is pretty useful.
I think ollama defaults to GGUF for everything. Not sure how well it deals with models that don't fit on the GPU, but it still uses some video ram. I've only played with changing the number of layers given to the GPU using text-generation-webui.
A big difference that would make a next great video: Prompt Processing Time. I found using 4000 character strings as prompts was much much slower on the M1 Max vs 4090 for example.
I doubt most people here are looking for a comparison based on fairness. We are looking to see what is currently available in the market, likely for use in our home labs.
I read that the larger SSDs were faster than the smaller ones, peaking with the 4TB chips. If one regularly hits deep into swap space, I wonder how much it helps sustain performance.
Great content quality: like and subscribe from me :) For me the big question after watching all this is: is 70b worth it? I know that comparing essays could be boring, but... are they any better? Or generally does 70b model give more relative results? Asking, because for the current moment seems like gpt3.5 performs better then gpt4: listen to instructions more carefully, less hallucinate, etc.
This guy: “So you can see the 4090 finishes a little faster than the M3 Max. Just a few seconds. Not a big deal.” Everyone else: “4090 >SLAMS< M3 Max!!!”
Omg, finally a channel using these machines for proper accessible AI instead of TH-cam content creation, keep up the great job and a video how to get models up and running would be awesome 👏
Apple is crazy 🤣. This laptop running 70 billion LLM parameters with ease. Imagine buying two of these laptops and doing a cluster... So the M3 is more cost effective in training LLMs than to a used A600s?
I was about to buy a 128g or 256h ram pc with rtx 4090 but it seems it is now the right choice after watching this video for running 80b locally. What would you say?
By the way GPTQ is faster than GGUF. But GPTQ obviously only runs on GPU, so no good if you want CPU and system RAM offloading. If I want speed I use GPTQ if I want large models I use GGUF. GGUF in this test makes sense since your using CPU's. But you do raise an interesting point the Max seems to have comparative inference speed as the 4090 even if the performance is a little restricted using GGUF. Maybe the 4090 is about 30-40% faster but only if the model fits.
if you are interested in inference, why are you avoiding quantization and just use the blazing fast Nvidia hardware (rather than buying 4 times as many gpu, buy one and quantize the model)? gpu to gpu, the m3 max 40 core is about the same as a 4060ti, if I understand correctly, and quantization is mostly only bad for training. edit: I think I see the answer, for the 70b model, even quantized to 3 bits it is still just over 26GB
ollama defaults to q4_0 quant. that said i would have liked to see the limit for the m3 max. It should be able to handle full precision 70:b, but i wonder if it'll generate fast enough to be usable.
Well, who's not impressed with the M3 MAX 128GB ? But Apple's price at $7,000 is unrealistic and as an added bonus the OS runs slightly more apps than Linux, yikes ! Waiting a few seconds more in getting the result will save anyone thousands in purchase costs. A very interesting channel.
damn i didn’t know openai, microsoft used macbooks m3 for their ai services…. ah no they use nvidia datacenter cards witch take the best performance per watt
As an owner of an M3 Ultra with all the features maxed out, I believe this comparison is extremely biased. The price of a single machine like mine equals a machine with 4 x RTX 4090s, so the comparison should be based on price, limiting ourselves to power consumption is undoubtedly interesting if we need to integrate our solution into a system with energy limits, but in that case, we are not comparing performance by loading a model that will require a lot of power, therefore, energy.The only and ultimate reason that seriously justifies buying the M3 is to be able to easily carry a high-performance model for demonstrations at clients' but really not for daily use, it makes no sense, the smaller models have almost the same performance so the size of the memory does not have immense interest.
This is the best real world test on youtube, i want to buy the 128gb ram m3 max 14 inch with 2 tb ssd only it is expensive €4779,- here in europe. is this laptop good for the next 10 years of programming, video editing, rendering,
@@technopremium91 Yeah the quality looks stellar, I also use my iPhone to record, but it's only since the last update of Mac OS that they force you to use that center stage
The big question is, do you notice any difference in the quality of response on the 70b parameters, or on MiXtral vs 7b mistral? I am wondering if it's worth upgrading my 16gb m1 Mac.
What's your suggestion about SSD volume when choosing 128G M3 Max? It is said different SSD has different transfer speed, which would influence the performance. Thanks~
nice trick with the terminal but why not to run the models with --verbose flag to get statistics to compare? all your compare charts are not accurate since finish time is not the same text generated on them.
Any ideas about which model I'll be able to comfortably run on MBP M3 Max 30 cores with 36Gb memory? I'm trying to figure out which version of M3 to buy. 40GPU-cores and 128Gb is 400mb/s as far as I see, and 30GPU cores with 36Gb is 300mb/s, so i'm concerned a bit if I really should spend extra money for 40GPU cores or there won't be a big difference.
I know others already said this a few times, but I just wanted to parrot. This actual usage comparison for real world workloads in large memory environments is truly what many people look for instead of all of the bull$hit repeated, bare-minimum usage, scenarios. A hearty thank you for showing true limits between a 4090 and a large memory M3 max. Cheers!
Thank you for the videos, very well done and informative. Have you tried doing LLM fine tuning with the M3 Max 128GB unified memory? I have seen people running the LLM models but using NVIDIA cards for fine tuning them. I'd be very interested to learn your thoughts about it
Thank you for this very interesting video. I’m just looking this kind of benchmark because I’m interested in buying an Apple Mac Book Pro M3 Max 128 Gb for LLM models. Great video!
Would you be able to test the Capybara-Tess-Yi 200K context across these devices? I was really impressed with the M3 Max's performance with the 70b model here, and I think it'll also really steal the show for making use of the whole 200k context window of the 34b model I mentioned. That 128gb of RAM is gonna be needed to make use of the whole window size with actually usable speeds
Idk what TH-cam is doing for your video bro but this is the EXACT video I was looking for in search but it didn’t come up. Didn’t appear till much later on my for you page. Glad I found it though, you did an excellent job
Very useful, thanks! I wish we had more benchmarking videos for AI models of various sizes, so that people can set their expectations of what they can get at different budgets.
Your test results concerning timting do not really matter, because you would have to count the tokens. Obviously they created each a different text, so a different amount of tokens.
Wow, it's possible to run 70b (even though quantized) that fast on a CPU?! I thought 70b was out of question because when I tried 33b on CPU it was painfully slow and there's no way a 4090 will load 70b. I was considering dual 4090 or A6000 with the latter only having half the computing power of a 4090. Obviously these prices for these GPUs are insane and the the power requirements for the 4090 are also insane. But if I could get away with a high memory CPU system I'd be totally willing to do this instead! How many tokens per second will was it actually per second with that Mac?
@@Pyriold Well, I've read up on this topic now a little. Seems the test is not really accurate. According to some posts I read, this more or less CPU based configuration will take minutes to before even starting to respond when the context has already been populated to say 50% or like a few thousands tokens. Also benchmarks of CPU based setups where like 10-15x slower than any GPU based setup provided there was enough total VRAM to load the model (like in a RTX 4090 + RTX 3090 dual GPU setup). All numbers I found so far were around 1-3 tokens/s which also matches my experiences so far for CPU based setups. Given how powerful some 7b models recently became though, I more and more doubt that's worth it to invest a lot of money to run 70b models. Especially as these AXXXX GPUs are total rip-off no matter how you put this compared to the already very expensive high end gaming GPUs. Though of course it's probably a different story if you are building an actual data center.
@@mariantocana8472 Thank you. I'm pretty sure he only runs Llama 2. There was no Llama 3 when this video came out. I'm curious if Llama 3 70B runs similarly.
Yes, there’s how TH-camrs should present the 128gb model, not with video editing or benchmarks
I agree.
Yes. And not with gaming as well
What was your setup on the mac? Did you also try gpt4all too?
@@m.goedeker7381 receiving my MacBook Pro 16’ M3 max 128gb tmr. My usage is a bit different: SDXL (a lot of machine learning though)
This is the kind of video we want to see not fucking imbecilistic cinebench runs!! By the way Cinebench is the least fair benchmark for apple silicon!!
And yet most people since they are ignorant believe cinebench is the gold standard.
M3max is faster than a OC 14700k and rivals a 14900k, beating everything in more memory bound algorithms(for example prime numbers or water wave simulation)
Sold on the M3 Max. That 70B test. Damn.
@test12382but then it’s a desktop setup, M3 max is in Laptop which is much more impressive
@test12382 nothing can compete m2 ultra, we can't comapre desktop vs laptop. but for a laptop m3 max is really impressive.
you could look into tesla p40 servers 2 p40s should be able to run llama 3 70b at reasonable quantization
You didn't pin them to the same random seed, so they're generating different text, so it's hard to compare the elapsed time because they're not generating the same number of tokens.
The M1 & the M3 are producing ***almost*** identical text. While from a benchmarking perspective I guess consistency would be ideal, surely the nature of ML is that a process should do what it needs to do to get a workable result; you shouldn't make machines follow the exact same path if one can jump high fences but another can burrow under them. What's important is how fast they complete the task asked of them, not whether they did so in an identical manner.
I was hoping someone would post this kind of comparison. It seems the unified memory is a huge advantage for running larger llms. One thing I did not understand at the final 70b test, what was the amount the memory used by the m3, could you have gotten away with 64gb only instead of 128gb? Thank you for the effort you put in creating and sharing this test. Subscribed!
The 70b use around 35gb of memory, so 64gb will be more than enough at least for llama2 70b
I just ran the llama2:70b on a m3 64gb 14" and it works well. However, the fans did ramp up. No fans on smaller models like the 34b code lama though.
@@rubencabrera8519 do you thingk 64gb is enough to run Falcon 180B model?
@@Anderson-dy8mlI doubt because Falcon 180B is quoted as requiring 400gb of VRAM to infer on, which means you'll need at least 5 A100's before it gets off the ground in its original
@@Joe_Brig I’m thinking an M2 Mac Studio with 64gb would be better suited for this , on long term, heard other ppl commenting about the fans getting louder than normal on M3 MacbookPro s when running inference.
I’ve been searching EXACTLY for this! Thank you. Subscribed and looking forward to those next videos on stable diffusion and your grandma’s clone (if I understood well). Thanks bro!
same
Would be good to see fine tuning (Autotrain / PEFT) limits on Llama2 models for M3 MAX 128GB
Would love to see this too!
What do you think 64Gb will be enough or still the better 128?
If AI consumes 35GB of RAM with 70 billion parameters, you are left with 29GB of available RAM, and in my opinion, models will become more performant and specialized, similar to Mistral AI models with 7B models that are very performant relative to their size. So, 128GB is a security if you can afford it, but 64GB is sufficient
128gb or regrets
Nvidia cards are so expensive now. The are blinded by the demand and feel they can get away with anything. I wish they stay in their lalaland for longer while competition develops and pulls the rug from under their feet. The reason Nvidia can get away with this is because most AI/ML software is optimized to run on Nvidia. AMD/Intel/Apple/Qualcomm or a consortium should should build a killer GPU / machine that can slay Nvidia. They need to work with one or two AI projects - Say Stable Diffusion and Llama - optimize these two to run crazy fast on their hardware. Some ridiculous out-performance over Nvidia is needed to shake things up. Only then will AI/ML libraries and ecosystem will consider optimizing for non-Nvidia hardware.
Great idea, but I hoped this would be a better-executed test, with actual benchmark numbers. And that you show which model you used (which quantization). E.g. llama.cpp prints out the token/s numbers at the end, and you can switch between CPU and GPU with parameters on start.
With llama-2 single-user inference's response-token generation, the performance largely depends on memory bandwidth, and not very much on GPU-speed. The 4090 has 2.5x the memory-bandwidht of the M3 Max chip (even though its compute is much faster), and 5x vs. the M1/M2 Pro (the M3 Pro is slower in this due to lower bandwidth than an M1/M2). This largely should determine their performance results.
I totally agree, that Apple silicon is great a) because of comparatively cheap large GPU-available memory and b) because of approx. 1/10th the power-draw (heat/noise).
+1 for this comment. I noticed that the results are not really the same.
Nobody prefers power over performance . Users who need power dont care about power bill.
@@Teluric2 „nobody“ is wrong, quite a few people care about power-consumption during inferece. AI inference is moving towards the edge. If you want raw power for training, etc., most use a cloud datacenter for AI.
@@aziz9488Meanwhile living in his S-hole in the middle east or North Africa 🤣🤣🤣
@@Teluric2This is a laptop and everybody should care about the power consumption. This machine is not even meant to compete against a machine with 4090 but apparently there are tasks where it exceeds.
Thank you for this comparison! I've been searching for this for a long time and couldn't find it anywhere concrete.
Glad I could help!
Great comparison! The parameter always gets bigger. Based on your result, I'm thinking the upcoming 5090 won't be my next purchase. More likely M4 with 128GB or 256 RAM will be my next stop.
hope the M3 Ultra make a huge shift in performance.
I have this same CPU/RAM combination. I've been able to run up to the 120B Goliath models, q4 quantization. Very fast inference.
Could you share the link of the Model Version you use?
Can you repeat this video for llama 3? That will be awesome!
Could you test the M3 max with the OpenChat 3.5 model and the Falcon 180B model?
Amazing video & comparison, thank you! Subscribed! :)
I will try!
@@technopremium91 thank you! :)
@@technopremium91 awesome. subbed!
Outstanding video - thank you for taking the time to do this. It’s exactly the comparison I was looking for.
Hello, would the Macbook Pro 64GB RAM one be able run the 70b model, or do I absolutely need 128GB RAM?
Youre comparing an apple 128Gb RAM with more than enough space to run 70b with a PC w/ 32Gb ram setup that’s far less than the recommended spec for running 70b which is around 40Gb RAM minimum. Compare the M3 to a PC with 128Gb in Ram. This is comparison is silly. The 128GB PC will be both cheaper to build and youd probably still be able to get a 4090 with the cash left over.
If you can’t fit the model in the VRAM then you’re no longer comparing the Max to a 4090, you’re comparing it to the inadequate CPU setup you have
Not Apple fan here, but M3 Max performance with 70b was impressive, indeed.
great test. thank you. for all of those working with LLM models and are considering M3's this is what we're looking for. People should understand these are q4 gguf models but still it's a very relevant test and it's good to see the Mac's unified memory working. I'd love to see how would be to run an interface like streamlit along with RAG on instructor XL and ollama or textgen and see if the Max can handle all of them together.
It was interesting to see that the Mac struggled with using that much memory via the GPU, you could easily see that there is a bottle neck that slowed down the system big time.
Would love to see someone test and benchmark MPS (Metal Performance Shaders) for pytorch for training Ml/AI models with a focus on transformers and LLMs. Some of us do more than just inference and the availability of GPUs for doing ML/AI development is just insanely bad.
Your 5950x should have 128GB RAM instead of 32GB. Just not much RAM left after loading the 70B model.
Potential problems with this comparison:
1. Have you used GGUF format for all models on 4090 instead of GPTQ for smaller models?
2. Have you used AutoGPTQ instead of ExLamav2 (much faster inference) model loader for 4090?
3. Have you used part (up to 51 layers) of GGUF 70B model for 4090 acceleration?
Also comparison would be much cleaner if you add tokens/second for all your runs.
Other than that, the comparison is pretty useful.
I think ollama defaults to GGUF for everything. Not sure how well it deals with models that don't fit on the GPU, but it still uses some video ram. I've only played with changing the number of layers given to the GPU using text-generation-webui.
The point is comparing Apples to Apples. It wouldn’t be a useful comparison if the 4090 is using a different setup.
@@PythonPlusPlus apples to apples are same used models. optimizing setup and partial gpu offloading is a common sense.
A big difference that would make a next great video: Prompt Processing Time. I found using 4000 character strings as prompts was much much slower on the M1 Max vs 4090 for example.
Comparing a 4090 to an M1 Max is kind of unfair, and it depends a lot on the RAM as you can see in the video.
I doubt most people here are looking for a comparison based on fairness. We are looking to see what is currently available in the market, likely for use in our home labs.
Thanks for that! Could you compare the speeds for stable diffusion? There’s nothing out there comparing SD performances of M3 Max and 4090.
Thats actually my next video, working on it now. I will be uploading tomorrow.
@@technopremium91 heres a like and a sub, thanks to you minority out there who actually tests what matters and not video editing
@@technopremium91 Oh that’s really great. Thanks a lot for your reply and your time and effort!
Hey man! I was here for self gratification for having bought an M3 max but your grandma experiment got me subscribed. Can't wait!
Thanks for the sub, thats a project that took me quite some time, but i am ready to share, working on the video and I will upload early next week.
update 7 months later?
Yeah, I have an M2 Ultra (only 64GB RAM) and it performs similarly. Shared memory FTW
I read that the larger SSDs were faster than the smaller ones, peaking with the 4TB chips. If one regularly hits deep into swap space, I wonder how much it helps sustain performance.
I think in this case the amount of RAM is the real bottleneck, could you repeat the same test but with at least 64GB RAM on the RTX4090?
M3 max with 128GB will cost more than 5000 USD.
For that kind of money, a Windows workstation will run Llama2 7b/13b/70b *MUCH* *FASTER* .
😅no it’s not. Only Nvidia gpu runs ai, but a gpu with 128gb rams is costing ur entire house baby.
Great content quality: like and subscribe from me :)
For me the big question after watching all this is: is 70b worth it? I know that comparing essays could be boring, but... are they any better? Or generally does 70b model give more relative results? Asking, because for the current moment seems like gpt3.5 performs better then gpt4: listen to instructions more carefully, less hallucinate, etc.
This guy: “So you can see the 4090 finishes a little faster than the M3 Max. Just a few seconds. Not a big deal.”
Everyone else: “4090 >SLAMS< M3 Max!!!”
Omg, finally a channel using these machines for proper accessible AI instead of TH-cam content creation, keep up the great job and a video how to get models up and running would be awesome 👏
💯I could not agree more.
Apple is crazy 🤣. This laptop running 70 billion LLM parameters with ease. Imagine buying two of these laptops and doing a cluster... So the M3 is more cost effective in training LLMs than to a used A600s?
use quicktime recording and it will use 90% less resources
You just helped me so much. Thank you.
I was about to buy a 128g or 256h ram pc with rtx 4090 but it seems it is now the right choice after watching this video for running 80b locally. What would you say?
Damn while watching your video my computer start lagging😂😂
By the way GPTQ is faster than GGUF. But GPTQ obviously only runs on GPU, so no good if you want CPU and system RAM offloading. If I want speed I use GPTQ if I want large models I use GGUF. GGUF in this test makes sense since your using CPU's. But you do raise an interesting point the Max seems to have comparative inference speed as the 4090 even if the performance is a little restricted using GGUF. Maybe the 4090 is about 30-40% faster but only if the model fits.
if you are interested in inference, why are you avoiding quantization and just use the blazing fast Nvidia hardware (rather than buying 4 times as many gpu, buy one and quantize the model)? gpu to gpu, the m3 max 40 core is about the same as a 4060ti, if I understand correctly, and quantization is mostly only bad for training.
edit: I think I see the answer, for the 70b model, even quantized to 3 bits it is still just over 26GB
ollama defaults to q4_0 quant. that said i would have liked to see the limit for the m3 max. It should be able to handle full precision 70:b, but i wonder if it'll generate fast enough to be usable.
Tests on Shitbuntu with fat generic binary code with less memory, than srapple, working on khAMD "brilliant" CPU... very good test...
Well, who's not impressed with the M3 MAX 128GB ? But Apple's price at $7,000 is unrealistic and as an added bonus the OS runs slightly more apps than Linux, yikes ! Waiting a few seconds more in getting the result will save anyone thousands in purchase costs. A very interesting channel.
Please show us UNQUANTIZED memory usage for the 70B Llama on the m3 MAX. thank you very much!
damn i didn’t know openai, microsoft used macbooks m3 for their ai services…. ah no they use nvidia datacenter cards witch take the best performance per watt
M3 max outperformed 450w 4090 10 times even though have lost in other tests. Good job done there)
can you run 70b on m3 max 64 gb ram?
32gb of ram are too small which is not even enough to run windows 10/11, my PC has 192gb ddr5 + 4090 to run larger model as well.
As an owner of an M3 Ultra with all the features maxed out, I believe this comparison is extremely biased. The price of a single machine like mine equals a machine with 4 x RTX 4090s, so the comparison should be based on price, limiting ourselves to power consumption is undoubtedly interesting if we need to integrate our solution into a system with energy limits, but in that case, we are not comparing performance by loading a model that will require a lot of power, therefore, energy.The only and ultimate reason that seriously justifies buying the M3 is to be able to easily carry a high-performance model for demonstrations at clients' but really not for daily use, it makes no sense, the smaller models have almost the same performance so the size of the memory does not have immense interest.
It is impossible to find that mac now i hope next year M4 pro model provides me an alternative.
so the bottleneck on pc with rtx4090 is the RAM, you could have 64gb of RAM on the custom pc no?
This is the best real world test on youtube, i want to buy the 128gb ram m3 max 14 inch with 2 tb ssd only it is expensive €4779,- here in europe. is this laptop good for the next 10 years of programming, video editing, rendering,
So annoying that the iPhone continuity camera only work with center stage when recording on the main sensor..
It was annoying for me too, but that was the best way i have for recording because of the quality.
@@technopremium91 Yeah the quality looks stellar, I also use my iPhone to record, but it's only since the last update of Mac OS that they force you to use that center stage
The big question is, do you notice any difference in the quality of response on the 70b parameters, or on MiXtral vs 7b mistral? I am wondering if it's worth upgrading my 16gb m1 Mac.
Cant wait for the m4 max macbook pro! Or even the m4 ultra/extreme mac pro!
Lies, 70B ru s just fine on the 4090 using LM Studio. OLLama is unnecessary and inferior.
so basically if I buy an m3 max I get somethig better tthan an rtx4090?? :)
Why not using Mac Studio with M2 Ultra and 192 gig of ram to run 140b
What's your suggestion about SSD volume when choosing 128G M3 Max? It is said different SSD has different transfer speed, which would influence the performance. Thanks~
nice trick with the terminal but why not to run the models with --verbose flag to get statistics to compare? all your compare charts are not accurate since finish time is not the same text generated on them.
Any ideas about which model I'll be able to comfortably run on MBP M3 Max 30 cores with 36Gb memory?
I'm trying to figure out which version of M3 to buy.
40GPU-cores and 128Gb is 400mb/s as far as I see, and 30GPU cores with 36Gb is 300mb/s, so i'm concerned a bit if I really should spend extra money for 40GPU cores or there won't be a big difference.
can you make a few example of interesting ML projects that would require to use mre than 64 GB or RAM?
I know others already said this a few times, but I just wanted to parrot. This actual usage comparison for real world workloads in large memory environments is truly what many people look for instead of all of the bull$hit repeated, bare-minimum usage, scenarios. A hearty thank you for showing true limits between a 4090 and a large memory M3 max. Cheers!
I tested with my M1 Max 64gb and it works quite well.
Thank you for the videos, very well done and informative. Have you tried doing LLM fine tuning with the M3 Max 128GB unified memory? I have seen people running the LLM models but using NVIDIA cards for fine tuning them. I'd be very interested to learn your thoughts about it
Thank you for this very interesting video. I’m just looking this kind of benchmark because I’m interested in buying an Apple Mac Book Pro M3 Max 128 Gb for LLM models. Great video!
What about two 3090s? With a second-hand price of about $700, could be an interesting option to run the larger models.
What if you bump up the ram to 128gb on the PC
Would it be better to set the AI model to the same seed at a temperature of 0 on each computer, so they get the same output?
I don't quite understand why on mac it's running on CPU instead of GPU?
Its gpu, cpu wouldnt be as fast
"Next video we are gonna be cloning my grandma"
🗿
Vraiment merci beaucoup, ce comparatif m'aide pour mon choix le
Would you be able to test the Capybara-Tess-Yi 200K context across these devices? I was really impressed with the M3 Max's performance with the 70b model here, and I think it'll also really steal the show for making use of the whole 200k context window of the 34b model I mentioned. That 128gb of RAM is gonna be needed to make use of the whole window size with actually usable speeds
what model did you use? Can you share the Link?
i wanna. see 3090 in sli versus m3 max
you get new subcriber,
thank you
The M3 max did beast out on that 70B model.
Thx man, i searched for this benchmark comparison so long. Greetz!
Well done! Straight to the point.
only if the M3 max are not so expensive :(
Paired the 4090 laptop with a crap CPU. Thats too bad
Shut up and take my money, Apple! 😁😁
Idk what TH-cam is doing for your video bro but this is the EXACT video I was looking for in search but it didn’t come up. Didn’t appear till much later on my for you page. Glad I found it though, you did an excellent job
so everyone can run, just change te velocity?
you just sold me on getting the m3 max pro for my project. thank you!
It would be interesting if you could limit the power envelope of the discreet GPU to match the apple soc
This is exactly what I was looking for.
Great video very informative.
Subscribed
I got the 70b model running 22t/s with 2 3090s with the Aphrodite engine, can you do a test with the m3 max?
Try the 400b model as well
What parameters would you use with an M3 Max with 38gb ram?
I was running it with ollama which use the gpu by default.
Very useful, thanks! I wish we had more benchmarking videos for AI models of various sizes, so that people can set their expectations of what they can get at different budgets.
Please do the same test on Llama3 70b
Your test results concerning timting do not really matter, because you would have to count the tokens. Obviously they created each a different text, so a different amount of tokens.
Could you enable rebar expandable memory on the rtx 4090 to be able to use system memory curtly by the GPU? And rerun the tests?
He has 32gb ram only.
Cant wait for m3 ultra even tho i wont buy it.
thank you for sharing the result
nice video
Wow, it's possible to run 70b (even though quantized) that fast on a CPU?! I thought 70b was out of question because when I tried 33b on CPU it was painfully slow and there's no way a 4090 will load 70b. I was considering dual 4090 or A6000 with the latter only having half the computing power of a 4090. Obviously these prices for these GPUs are insane and the the power requirements for the 4090 are also insane. But if I could get away with a high memory CPU system I'd be totally willing to do this instead! How many tokens per second will was it actually per second with that Mac?
The M3Max is not only a CPU, it has a lot of GPU cores and dedicated neural hardware as well.
@@Pyriold Well, I've read up on this topic now a little. Seems the test is not really accurate. According to some posts I read, this more or less CPU based configuration will take minutes to before even starting to respond when the context has already been populated to say 50% or like a few thousands tokens. Also benchmarks of CPU based setups where like 10-15x slower than any GPU based setup provided there was enough total VRAM to load the model (like in a RTX 4090 + RTX 3090 dual GPU setup). All numbers I found so far were around 1-3 tokens/s which also matches my experiences so far for CPU based setups. Given how powerful some 7b models recently became though, I more and more doubt that's worth it to invest a lot of money to run 70b models. Especially as these AXXXX GPUs are total rip-off no matter how you put this compared to the already very expensive high end gaming GPUs. Though of course it's probably a different story if you are building an actual data center.
Get simply 2 3090 - faster than single 4090, cheaper, 48GB memory
How much wattage uses?
Nice channel, pretty professional, got subscribed
Thanks for the sub!
How is the 128GB M3 Max on Llama 3 70B 4bit? I have the 48GB M3 Max and it is very slow, probably one token every 10 seconds.
In this video when author runs "ollama run llama3:70b" that means he runs by default Q4_0 quant model.
@@mariantocana8472 Thank you. I'm pretty sure he only runs Llama 2. There was no Llama 3 when this video came out. I'm curious if Llama 3 70B runs similarly.
finally a yt that actually does real testing