Thanks for making this video. Really well explained. Do you have a solution for open-source LLMs that do support functions? Your approach above didn't activate the functions. The older approach that you highlight does work, but is very unstable and usually breaks. I'm trying to find a way of using your approach.
I would like to use this OpenAI functions type agent however I also want to decide how my agent should behave with system prompts. How would be that possible any idea?
As much as I like LangChain they update it and deprecate so much stuff that lots of resources including there own documentation and examples don't work. Great Video though Sam!
What would be great is to hear how to get around having too many tools (IE, a backend with a lot of different endpoints and parameters, etc.) Won't you run out of tokens for that? I wonder how to handle that. The other thing I haven't really seen anyone cover yet (in the 24 hours since it was announced, lol) is a direct comparison with the OpenAPI agents, which do a lot of similar things.
One thing you can do is dynamically change the functions with different calls. They don't have to be in any history etc. I think we will see some hacks of how to optimize for the number of tokens etc.
the agent using OpenAIFunctions actually keeps re-entering it's tools even when I can see it has deduced the final answer. Is there any way it will produce some stopping command for itself?
What if I want to ask for some other information, which is not related to function. For example, if I ask "Hi, How are you. " It should not call the function. How can I do that?
what if I want to make a tool that will make sure to gather all the required information from the user in a conversational way before executing? For example: Main Tool: schedule a call Information required: user name, phone number and email id. first user input: I want to schedule a call. Then invoking this tool but first asking the user all the required information. How can we achieve this? and how can we integrate this workflow with langgraph?
interesting question. Maybe but for most cases I don't think you will need it. There are some exceptions. The paper JSONFormer shows they could improve this but that would require a change to their inference pipeline not just a new model etc.
Thanks Sam, I liked the part for manual invoking, nice explained. I have one issue that I am struggling to solve for a multi-input tool simlar to get_price_change_percent. I am catching the params in the _run method (lets say ticker and days( but I cannot retrive the action input, i.e. the text inside the Tool (for instance Show me the change of Apple stock in the last 90 days). If I am using the query:str as a param in the run it catches the filtered query without the ticker and days...
its a while since I made this vid and answering without reviewing the code. If manually invoking then you should be able to just parse it all out as JSON. Another thing I have done for some things using this is to fire another function that and have that breakdown anything I needed etc. Let me know how you get on.
Thanks for the presentation! I am wondering how can we add history for the conversation. So, having the first prompt with a tool and getting an answer i want to keep asking chatgpt questions related to its answer
Thanks for the great explanation, but I have one doubt - in the agent part wherein you specified your functions as tools, why are we still using OPENAI_FUNCTIONS agent type, why can't it be simple ZERO_SHOT_REACT_DESCRIPTION agent type? I have been trying to understand actual usage of openAI function agent types but can't find any concrete reason as to when to choose openAI function agent over others.
I notice you don't use llamaindex for the indexing step. Do you think llamaindex would provide better results given the attention on the various indexing methods? Thank you.
I am have been working on a set of Llama Index videos and just keep getting distracted by fun releases like this. I do use and like llama index for work projects, so I will get around to making some videos for it soon.
Thank you for your video. You mention that you don't know how it impacts the tokens used but doesn't the OAI API response explicitly report the tokens used? It would be awesome if you could give a rough comparison of the LangChain tokens with tools vs new functions.
Thanks for sharing Sam! Definitely exciting to have an agent that is capable of utilizing many tools. However one constraint could be due to the context of LLMs. Suppose we have 1000 tools for an agent to apply, but it's impossible to feed all of them into one LLM. Any thoughts on how to sovle this problem?
You can make multiple api calls first filtering the tools. For this you can use embedding version of the tools, then the second call to really pick the tool from a filtered smaller pool. Thats what came to my mind but I am curious to know about more for the exact question.
Given how generative model works, I wonder how they were able to incorporate such function feature as input into the inference. It doesn't seem like it is just part of the prompt, given the precise format.
They have just fine-tuned the model with a new token which is the role of 'function' etc. They actually probably have other tokens in there that they aren't telling us about as well.
lol I was just playing with this today. I agree it is interesting, just want to think how to do where it is not just like the other videos out there etc.
You probably don't want to be passing in JSON data directly to this. write a function that searches the JSON and just use that with the OpenAI functions
Could you please tell how to use output of one tool as input for other tool. Like trivial usecase : three tools = [list_directory, get_file_content, explain_file_content] User question : I want you to explain a code in my directory. Ai : Which file ? (here it must use list directory tool ) User : selects a file Ai : Here is the explanation ( it must use get_file_content > explain_file_content ) Could you please help me figure this one out? Much appreciated thank you . Great video content :D
you could write this a single tool with a function and do it that way. the way you wrote the conversation you could just make it 2 tools one that takes a file url as input after the user selects the file. Just make sure to use memory to all the model to see the full part of this conversation
@@samwitteveenai i wrote that as an example but i am interested to know how to keep conversation interactive between user and agent, while it is running tools (where it has three tools outputs of one to be used as input of other tool based on conversation :) )
@@akhilkanduri6202 the main thing is to use a conversational memory to pass relevant info from the conversation responses to the tool. One trick though is to make custom tools that do multiple things etc
@@samwitteveenai thanks for your response ;) and no, i feel it makes it so much more complicated to implement stuff that it wasn't explicitly meant to do. it has some great ideas like the summarized memory (forgot what it's called) but i prefer creating my own routine so i understand what's happening and i'm quicker to react to developments.
@@MarkWernsdorfer I can totally understand where you are coming from. lol there have been a number things recently where I have thought this would be easier to roll my own (and I have for some synthetic data creation projects). I think there are pluses and minuses to all frameworks and I do think the new OAI Functions make some of the LangChain features not as needed.
@@samwitteveenai I really appreciate the engagement! Maybe I can be a little more constructive: the new openai functions can be exploited in ways that are not possible in Langchain. For example making gpt output structured (JSON) data (without even implementing the function defined for the API) that would've been difficult to achieve otherwise.
Yeah I have played a bit with the JSON output it really does allow to get the same at outparsers/guardrails with less tokens. It would be nice if they just end up implementing Jsonformer github.com/1rgs/jsonformer
![F.HERO Ft. JSPKK x ลำไย ไหทองคำ x M-PEE - ไม่สนิทบิดหมด (Thai Riders Anthem) [Official MV]](http://i.ytimg.com/vi/kPJZ0UihBfk/mqdefault.jpg)
I wish I could shake your hand man, thank you.
2 videos in 1 day? It's madness!
Thanks for this. 👍
lots of similar videos in youtube, this channel explains it best and succinctly 👍
Thanks for the kind words. Much appreciated.
Your channel is an absolute gold mine
Amazing content as always, thank you for sharing your knowledge. I would love to see you cover gpt-engineer and go deeper into its functionality
I agree I would love to see that video
He already went pretty deep compared to other tutors on TH-cam
Excellent tutorial Sam, Thank you so much sir
Amazing Mr. Witteveen................... and thanks for also posting the codelab........ example............ :) bye
can you use this in combination with a QA chain agent as of today, with a retriever/docsearch with embeddings and index?
Thanks for making this video. Really well explained. Do you have a solution for open-source LLMs that do support functions? Your approach above didn't activate the functions. The older approach that you highlight does work, but is very unstable and usually breaks. I'm trying to find a way of using your approach.
you generally want to fine tune for this. There are some models like NexusRaven that you can try as well. I am planning a number of vids about this.
@@samwitteveenai Thanks Sam - I'll take a look at it
Thanks for your video. I am wondering how to stream the final answer of OpenAI Function Agent?
The requires using a streaming call if you want it token by token etc.
Yeah their function calling is basically just how jsonformer and such had already worked before, just implemented directly into openai api, cool
I would like to use this OpenAI functions type agent however I also want to decide how my agent should behave with system prompts. How would be that possible any idea?
Thank you sir, very well explained!
As much as I like LangChain they update it and deprecate so much stuff that lots of resources including there own documentation and examples don't work. Great Video though Sam!
Thank you for the additional video.
Do you know any optimized ways to use multiple templates with an agent and multiple tools specific to each template?
What would be great is to hear how to get around having too many tools (IE, a backend with a lot of different endpoints and parameters, etc.) Won't you run out of tokens for that? I wonder how to handle that. The other thing I haven't really seen anyone cover yet (in the 24 hours since it was announced, lol) is a direct comparison with the OpenAPI agents, which do a lot of similar things.
Maybe there's a way to build a custom agent that utilizes tool retrieval and function calling?
One thing you can do is dynamically change the functions with different calls. They don't have to be in any history etc. I think we will see some hacks of how to optimize for the number of tokens etc.
the agent using OpenAIFunctions actually keeps re-entering it's tools even when I can see it has deduced the final answer. Is there any way it will produce some stopping command for itself?
What if I want to ask for some other information, which is not related to function. For example, if I ask "Hi, How are you. " It should not call the function. How can I do that?
what if I want to make a tool that will make sure to gather all the required information from the user in a conversational way before executing?
For example:
Main Tool: schedule a call
Information required: user name, phone number and email id.
first user input: I want to schedule a call.
Then invoking this tool but first asking the user all the required information.
How can we achieve this? and how can we integrate this workflow with langgraph?
Awesome, thanks for the video! is there any way to combine ReACT and function to get more predictable results in desired format?
interesting question. Maybe but for most cases I don't think you will need it. There are some exceptions. The paper JSONFormer shows they could improve this but that would require a change to their inference pipeline not just a new model etc.
Thanks Sam, I liked the part for manual invoking, nice explained. I have one issue that I am struggling to solve for a multi-input tool simlar to get_price_change_percent. I am catching the params in the _run method (lets say ticker and days( but I cannot retrive the action input, i.e. the text inside the Tool (for instance Show me the change of Apple stock in the last 90 days). If I am using the query:str as a param in the run it catches the filtered query without the ticker and days...
its a while since I made this vid and answering without reviewing the code. If manually invoking then you should be able to just parse it all out as JSON. Another thing I have done for some things using this is to fire another function that and have that breakdown anything I needed etc. Let me know how you get on.
Thanks for the presentation! I am wondering how can we add history for the conversation. So, having the first prompt with a tool and getting an answer i want to keep asking chatgpt questions related to its answer
Thanks for the great explanation, but I have one doubt - in the agent part wherein you specified your functions as tools, why are we still using OPENAI_FUNCTIONS agent type, why can't it be simple ZERO_SHOT_REACT_DESCRIPTION agent type?
I have been trying to understand actual usage of openAI function agent types but can't find any concrete reason as to when to choose openAI function agent over others.
if we use serpapi we can get it form google but why do we use those custom fuctions ?
I notice you don't use llamaindex for the indexing step. Do you think llamaindex would provide better results given the attention on the various indexing methods? Thank you.
I am have been working on a set of Llama Index videos and just keep getting distracted by fun releases like this. I do use and like llama index for work projects, so I will get around to making some videos for it soon.
Thank you for your video. You mention that you don't know how it impacts the tokens used but doesn't the OAI API response explicitly report the tokens used? It would be awesome if you could give a rough comparison of the LangChain tokens with tools vs new functions.
Yeah was thinking about doing something like after I recorded this.
Thanks for sharing Sam! Definitely exciting to have an agent that is capable of utilizing many tools. However one constraint could be due to the context of LLMs. Suppose we have 1000 tools for an agent to apply, but it's impossible to feed all of them into one LLM. Any thoughts on how to sovle this problem?
You can make multiple api calls first filtering the tools. For this you can use embedding version of the tools, then the second call to really pick the tool from a filtered smaller pool. Thats what came to my mind but I am curious to know about more for the exact question.
Great videos!
Given how generative model works, I wonder how they were able to incorporate such function feature as input into the inference. It doesn't seem like it is just part of the prompt, given the precise format.
They have just fine-tuned the model with a new token which is the role of 'function' etc. They actually probably have other tokens in there that they aren't telling us about as well.
how to pass system prompt in agent
I cant import MoveFileTool andformat_tool_to_openai_function . I made pip install --upgrade langchain but still cant import
Check the version. I think it should be 200 from memory
Gpt-engineer is a interesting project you should look into
lol I was just playing with this today. I agree it is interesting, just want to think how to do where it is not just like the other videos out there etc.
sir,i have large json file that containing the finance data when i creating the json agent iam getting the token error,please reply,thank you
You probably don't want to be passing in JSON data directly to this. write a function that searches the JSON and just use that with the OpenAI functions
Brilliant❤
what if i want to respond to live chat helping a client, do I need everytime to send back the whole help docs so that I can get a reply?
no just send the chunks from a vectorstore etc. Checkout the videos using Chroma and PDFs to do this
So are you suggesting passing a tool that gets data from a vectorstore?
Very informative
How might we add memory to these agents?
The should work ok with the normal Conversation Memory etc.
You quick. Didn't functions just announce yesterday?
Could you please tell how to use output of one tool as input for other tool.
Like trivial usecase : three tools = [list_directory, get_file_content, explain_file_content]
User question : I want you to explain a code in my directory.
Ai : Which file ? (here it must use list directory tool )
User : selects a file
Ai : Here is the explanation ( it must use get_file_content > explain_file_content )
Could you please help me figure this one out? Much appreciated thank you .
Great video content :D
you could write this a single tool with a function and do it that way. the way you wrote the conversation you could just make it 2 tools one that takes a file url as input after the user selects the file. Just make sure to use memory to all the model to see the full part of this conversation
@@samwitteveenai i wrote that as an example but i am interested to know how to keep conversation interactive between user and agent, while it is running tools (where it has three tools outputs of one to be used as input of other tool based on conversation :) )
@@akhilkanduri6202 the main thing is to use a conversational memory to pass relevant info from the conversation responses to the tool. One trick though is to make custom tools that do multiple things etc
@@samwitteveenai thank you, Ive been going through docu but couldnt quite figure it out yet. I try again and wait for other videos of yours :D thx
this seems super convoluted. i'd prefer an introduction to openai functions without langchain in between.
Just curious you don't use LangChain at all?
@@samwitteveenai thanks for your response ;) and no, i feel it makes it so much more complicated to implement stuff that it wasn't explicitly meant to do. it has some great ideas like the summarized memory (forgot what it's called) but i prefer creating my own routine so i understand what's happening and i'm quicker to react to developments.
@@MarkWernsdorfer I can totally understand where you are coming from. lol there have been a number things recently where I have thought this would be easier to roll my own (and I have for some synthetic data creation projects). I think there are pluses and minuses to all frameworks and I do think the new OAI Functions make some of the LangChain features not as needed.
@@samwitteveenai I really appreciate the engagement! Maybe I can be a little more constructive: the new openai functions can be exploited in ways that are not possible in Langchain. For example making gpt output structured (JSON) data (without even implementing the function defined for the API) that would've been difficult to achieve otherwise.
Yeah I have played a bit with the JSON output it really does allow to get the same at outparsers/guardrails with less tokens. It would be nice if they just end up implementing Jsonformer github.com/1rgs/jsonformer
Or copy out the function-calling code from langchain and use it in your custom library. You'll thank me later.
Good work! :)
Thanks!
Your voice is AI generated right? Because i hear some artifacts while after while
no it is just I have been using a noise reduction on some of them as the recordings were full of noise.
I am going have to ask. Why isn't chatGPT writing this code for you? 😂
How can we add memory to our custom agent? And how do i make it work if my prompt require more than one agent?
Thanks!