How to Use Open Source LLMs in AutoGen Powered by vLLM

In this video, I would like to talk about creating agents in AutoGen with Open Source LLMs.
USEFUL LINKS:
Colab notebook for AutoGen w/ GPT-4 - colab.research.google.com/dri...
Colab notebook for AutoGen w/ Phi-2 - colab.research.google.com/dri...
Tutorial on Medium: levelup.gitconnected.com/addi...
AutoGen Docs: microsoft.github.io/autogen
vLLM Docs: docs.vllm.ai/en/latest/models...
MY CONNECT:
Buy me a coffee - ko-fi.com/yeyuh
Business Inquiries - [e-mail]
X: x.com/Yeyu2HUANG
Discord - / discord
Email Subscription - yeyu.substack.com/
Exclusive service - ko-fi.com/yeyuh/tiers

[@JDWilsonJr]

This is spot on. Thank you for making the video and explaining so well.

[@yeyulab]

Thanks

[@Beenee_AI]

Great ! You are really good at what you do!

[@yeyulab]

Thank you!

[@jaoltr]

Terrific video! Thank you for sharing your knowledge.

[@yeyulab]

Glad it was helpful!

[@joeclacher445]

Unreal video! Looking forward to testing various models instead of GPT!

[@yeyulab]

Thanks, would like to see the performances as well.

[@truliapro7112]

How to use autogen with aws bedrock models ?

[@KodandocomFaria]

Is it possible to use powerinfer instead of vllm? If possible which one would be faster ? Perhaps a good video to make by comparing those two inference tools

[@yeyulab]

Looks Powerinfer is a pretty new inference tool. It cannot be supported in Autogen directly right now but If you can run Uvicorn to serve its inference, there maybe a chance. Thanks for the recommendation.

[@Nick_With_A_Stick]

I don’t believe you have it in your youtube tag’s but you should fill our your youtube tags with things like “ vLLM tutorial” as k looked for one and came up very very short, and this would’ve been much mote useful. Thanks for the vid!

[@yeyulab]

Good suggestion, thanks!

[@shubhamgarg5007]

Hey, thanks for the in depth explanation. While its great that we can use Autogen along with open source models using vLLM, is there any chance we could use Gemini API along with autogen?

[@yeyulab]

There is an on-going branch of AutoGen working on Gemini integration. Soon you can use it I think. github.com/microsoft/autogen/tree/gemini

[@shubhamgarg5007]

@@yeyulab Yeah, I checked it but it has no commits since the last 2 weeks and I doubt its one of their top priorities as of right now. I couldn' find any online resources to use Gemini's free api with autogen either.

[@yeyulab]

Free Gemini API is really useful I agree. Let me check with their team.

[@current.undone]

thanks for sharing. Can vLLM be installed on Mac? Please help if it can as Max Studio has all the musles needed to do the heavylifting 🙂

[@yeyulab]

vLLM does not support MAC backend at the moment and I guess the reason is that they want to maximize the throughput of generation by V100/H100 GPUs.