LangChain in Production - Microservice Architecture (incl. FastAPI and Docker)
ฝัง
- เผยแพร่เมื่อ 21 ก.ค. 2024
- Ever thought about build a real world application with LangChain, with multiple microservices, a real Frontend Framework and RestAPIs to handle conversations? Then this video is for you!
Code: github.com/Coding-Crashkurse/...
0:00 Microservice Architecture Chatbot
2:45 Quick tipps for LangChain Ecosystem in Production
3:45 Code Walkthrough
10:37 Checkout the App!
![[TH] VCT Pacific Stage 2 - Playoffs Grand Final // GEN vs DRX](http://i.ytimg.com/vi/Su6gA_UPs9s/mqdefault.jpg)
i always see these comments under like every tutorial video even the ones that give no really useful info, so any praise under any educational video kinda devalued to me which i suppose might be the case for the creator themselves too.
ive never written such comment partly for that reason and MAN i just want you to know that this video is the most appreciated of the ones about langchain to me, i literally cant begin to describe how useful for me it is, been searching for it for a really long time. you deserve the happiest life a human can imagine and i just hope some day i'll run into you to buy you a beer or two. thank you.
Thank you so much for that comment. Reall makes m day to Read something like this :)
You did a containerized, modular deployment of a LangChain implementation. Very nice! Man here made me smash that 'like'. He made me.
This is very cool! Would be keep to see more of this!
This is absolute gold! You are helping me make my dream project right now, thanks so much!
Thank you for that nice comment :)
excellent ! production readiness score 8/10
Subscribed you channel, waiting for more full project based tutorial on AI, Langchain, Python, Data science. Keep up the good work ❤👍❤
You are the best my man! Such a great video
Nice work!!! there are a lot of notebook but no so much information for real world. its really valuable!!!
Thank you. Will soon release a video on deployment on azure
Great job!! Thank you very much!
Great Work! Finally a Real World App on youtube
thank you very much :-)
This makes sense for very large projects, a monolith for most use cases is 100 percent fine. Microservices is for an app that is used constantly by many users and the app makes a lot of money. Also why would a restaurant pay monthly for so many containers? If it is a chain it is fine, for small business this is very impractical. Following best practices for the sake of it when it does not provide a benefit, and from the business perspective does not make sense is not a good idea.
BTW the design is good, and the tutorial was presented very well, just don't tell people to use microservices for a restaurant app.
Yes, this Implementation is for a Restaurant chain of course. Thats why i made that Statement about hybrid search.
Well done!
Amazing 👍👍👍
Excellent work, it's great to see another approach more oriented to a real application, I wonder if you could make a video of the deployment in web services like AWS and GCP, there are many tools and I don't know which one to choose in these cases, it would help a lot to see how you execute it. Greetings from Colombia.
I will release a project on azure Sohn:)
did u release it already?@@codingcrashcourses8533
@@codingcrashcourses8533Did you also create a video about this? :)
@@joepsterk5176 i did a full udemy course and released a soll video
super helpfull
what is I want to use a model like Llama2, how can I change OpenAI API for Llama2?
how does your pgvector and redis scale? you can't have multiple docker containers of these services.
a scalable cloud service for redis and pgvector might be more desirable which can handle huge load
you can at least for redis. Redis has got Redis Sentinel which is a cluster solution which master and slave agents. For PGVector I am not sure how much load it can handle, but PostGres is a well established DB which normally can handle a lot of traffic with a single instance.
Have you worked with LangChain's agents at all in production? I've been trying to spin up a ReAct agent with access to tools for querying a Postgres DB and Pinecone vector store, but the implementation has been sort of lackluster. I'm using their ConversationalBuffer memory populated from a DynamoDB message history, but the agent really struggles with follow-up questions and using context from the conversation to form queries.
I'm wondering if its better to just unroll everything down to a simpler LLM chain and forgo the Memory class like you've done here.
Follow up questions are worth a video by themself. Got no solution I am totally happy with. In my company we tried going back in the history n-1 and adding that again, since similarity search won´t give you back data.
In general I am not a big fan of agents - they just are unreliable. I would probably use OpenAI function calling for querying a SQL Database. My Agents were too dumb to write good SQL. But try it yourself, I am also just a random dude trying out stuff
Thanks a lot! Quick question - where did you install pydantic, langchain, and the other libraries that are not in the Dockerfile?
I installed it directory with pip install inside Docker. Most dependencies have subdependencies which get installed on the fly then
there are multiple dockers, which docker do you install this package? I run the python file and get an error about write into the database@@codingcrashcourses8533
Is there any risk of PII leakage in this implementation if I am using other hugging face llm instead of OpenAI ?
Outstanding work! This is one of the best tutorial I've found on how to build real-world LLM apps.I think it's really cool how you use Redis for preserving conversation history. However, I'm wondering if there might be issues exceeding the OpenAI token limit? Do you have any ideas to overcome this issue? Keep up the good work!
VERY good point. You could do that by cutting off the conversation if it grows longer than 10 interactions (or any other useful number). Would be worth looking at how ChatGPT handles that
Thanks for your response! I completely agree, keeping the recent X number of interactions might be sufficient for smaller projects. However, for more complex, production-ready apps, I believe using something like ConversationSummaryBufferMemory from Langchain might be a more robust solution. It would be really fascinating to see how this functionality could be incorporated into ConversationService. Looking forward to future updates with this in mind.
Once again, excellent work with this super-detailed tutorial. If you were to create a similar system for several restaurants, what would the architecture be like? Or would you simply create an instance for each restaurant?
Thank you. I would probably use a single vectorstore and save the Restaurant name or id as Metadata. And use that Info to Filter before the vectorsearch. Each Website would then send that Info in a cookie or the request header
@@codingcrashcourses8533 Thanks for the quick reply, excellent suggestion. Do you have any means of communication where we can exchange ideas other than youtube?
@@Almopt currently i dont have any other channels :)
nice video. I can echo the sentiment of others. Most videos miss out on important things or basics for a non experienced beginner. Do you think you could add a video in actual production (live production). Second thing just a feedback , the data flow and handshake of services if it is shown in numbers and different colors ( the flow basically ) would be great.
I have a full udemy course ok that :)
@@codingcrashcourses8533 i have to look for that. Have you also experimented with claude (from anthropic)
Amazing! For some reason though, Im having trouble getting this to work with the LangChain(python) Functions Agent. Does it need to be done differently? By chance do you have any examples?
What did you Change? If you replace the llmchain with your agent it should work as well
@@codingcrashcourses8533 hmmm then i think my front end setup is now broken. Does the above work for Streaming? My frontend is SvelteKit which is where my error might be..
Good video.
I'm already going down this path.
Except I'm forced to use local model.
rip
My team is recommending that we deploy our embeddings along with our code inside the azure Function App. I understand using a vectoreDB would be a more better approach. Can you tell me the limitations of keeping the embeddings inside functionsapp instead of a separate vectors store. So that I can make them understand?
How do you retrieve the embeddings and related documents without a vectorstore? You can do it on your own, but why?
How can you make the ai call external functions and get input for those functions from the user. Such as the as askes for an order id.
I would try to archieve that with prompting and the usage of openai function calling. What might you want to know? Delivery status of a package?
@@codingcrashcourses8533 Yes, that would be the perfect example. I actually run an e-commerce store, and I am looking to implement a chatbot to assist users with their requests more effectively. Regular customers often inquire about the estimated delivery time for their orders. In order to assist them effectively, our chatbot will require their order ID to call internal functions to get the delivery time.
Any comments on why you need to separate the conversation ID management into a separate service from the service making the call to open AI?
Good question! Well it depends what you want to do. In the company I work for we store the conversations from multiple channels, Audio and text. So we use this architecture to store all conversations in a single place
Thank you@@codingcrashcourses8533
Is there any particular reason why you use pgvector over FAISS? I saw benchmarks where people proved that FAISS is much faster
Yeah, but does it really matter if you app in 1ms faster or slower? I prefer to be able to use other tables for different stuff, use SQLAlchemy to build an API Layer around my Database. FAISS for me is some kind of big blackbox
Nice tutorials.may i know the VS code theme used.
Material theme
I get an error message "exec /wait-for-postgres.sh: no such file or directory" when I build using docker-compose up. Do you have any idea how I can fix this?
I guess that you use a Windows Format for your file. Change it to unix. You can do that in most editors or with dos2unix (cmd). Just make sure it is formatted as LF, not CLRF
@@codingcrashcourses8533 Thanks. It resolved the issue.
How do I replace old embeddings stored in pgvector to new ones. I have new embedding stored in pgvector, but the chatbot still queries from the old embeddings..
Can I substitute Redis for Chroma or faiss?
No, these are vector databases, which would substitute pgvector. For storing conversations you could use any other document db like MongoDB, but redis is great since its so fast and easy to use.
@@codingcrashcourses8533 Ah okay thanks!
Great video, but it woud be nice if you replace openai to an open as source llm.
Just deploy the Model and change the class you use. Thats what langchain is for
Hey, is this you? Algovibes?
Haha I though the same thing
No that is just that typical german accent :p
@@codingcrashcourses8533 oh damn! you really had me fooled. You guys should collaborate on something as a publicity stunt.
@@jayglookr haha i never heard of him before to be honest. But i know the german accent when i hear it ;)
@@codingcrashcourses8533 Well, it's not exactly 'that german accent', not exclusively. It was also the top quality, very succinct, informative content. Your channels are easily mistakable in that regard as well. Truly great.