LocalAI LLM Testing: Part 2 Network Distributed Inference Llama 3.1 405B Q2 in the Lab!

[@tbranch227]

Congrats! You unlocked a masssive achievement running this on your own hardware!!! All hail the AI and the kilowatts we feed them

[@RoboTFAI]

Skynet is possible in your basement 🦾

[@marekkroplewski6760]

Dad! Where did my gaming rig go!!! Now listen up there Junior, this is for science. Just don't tell your Mum. And you can have the car keys for Saturday.

[@RoboTFAI]

Better than stealing their GPU's out of their rigs right? 😂

[@animationgaming8539]

I liked every comment on this video!

[@RoboTFAI]

Thanks!

[@ckckck12]

What I see is 1/3 working time caused by using a model built on close to 6 times the data points. That's nice! And... Enabled by distributed mode.
Am I correct? Is there a way that the quant factor affects the computation of those general factors (time/tokens vs size) that would make this more 1:1?
Already it's nice you can get the big models in.

[@twinnie38]

Really impressive, congrats ! Do you know the impact of a limited PCIe bus (1x , 4x GEN3) for those GPU cards ?

[@nickmajkic1436]

Would you be able to make a tutorial on getting lovalAI working in kubernetes?

[@RoboTFAI]

Sure, I think that's overdue at this point!

[@_zproxy]

wild. does this work for llava images too?

[@unsaturated8482]

Damn

[@mckirkus]

What's the network bandwidth? I wonder what could be done if you connected to a bunch of buddies with gigabit symmetrical fiber connections.

[@RoboTFAI]

As much as you can pump for distributing the model - during inference it's really only about 10-20 MB/s per node

[@bechti44]

around 4 Times faster than cpu only... But around 100x more expensive...

[@RoboTFAI]

It's just money and power! like always.... 😁

[@andriidrihulias6197]

First

[@RoboTFAI]

Congrats!

[@Anurag_Tulasi]

It would be more intelligible if your results mention (Higher is better or Lower is better) beside the chart headings.

[@RoboTFAI]

Thanks for the feedback!