Microsoft Phi-2 2.7B LLM RAG Medical Chatbot llamaindex Colab Demo 2.7B better than 7B ,13B LLMs

[@sujithanagasuri6151]

sir can you make a video with some other dataset?

[@malleswararaomaguluri6344]

hi sir, can you make a video on offline mode in our windows laptop with our own data (due to data privacy)?

[@RitheshSreenivasan]

just create a python environment on your windows laptop and download the model and run locally. nothing special needed to be done

[@malleswararaomaguluri6344]

@@RitheshSreenivasan sir, I am facing issue how to enable gpu while running llm model. And library conflicts while using accelerate and bitsandbytes. I tried the same in Google colab it is working fine. But in my system I am facing the above issues. Please suggest me how to handle this

[@RitheshSreenivasan]

Check your CUDA installation. Create a clean python virtual environment and install the libraries

[@pavankumarpenjandra]

Thank you very much

[@Storytelling-by-ash]

really awesome video, thank you very much! I feel we are getting empowered everyday more and more and dont need to depend on OpenAI or other cloude service.

[@camilovega1343]

Hi, great video!
But i cant import HuggingFaceLLM.
It appears and error in this part:
from llama_index.prompts.prompts import SimpleInputPrompt
system_prompt = "You are a Q&A assistant. Your goal is to answer questions as accurately as possible based on the instructions and context provided."
# This will wrap the default prompts that are internal to llama-index
query_wrapper_prompt = SimpleInputPrompt("{query_str}")
llm = HuggingFaceLLM(
context_window=4096,
max_new_tokens=256,
generate_kwargs={"temperature": 0.0, "do_sample": False},
system_prompt=system_prompt,
query_wrapper_prompt=query_wrapper_prompt,
tokenizer_name="microsoft/phi-2",
model_name="microsoft/phi-2",
device_map="cuda",
model_kwargs={"torch_dtype": torch.bfloat16}

[@RitheshSreenivasan]

Checkout llamaindex library installation. May be a previous version would work