ฝัง
- เผยแพร่เมื่อ 18 ก.พ. 2023
- Learn how to build a full-stack app in this tutorial: • Full-stack AI chatbot ...
Tutorial about building an AI with a custom knowledge base using OpenAI API, GPTIndex, and Langchain.
The technique was first described by Dan Shipper www.lennysnewsletter.com/p/i-...
Source code: colab.research.google.com/dri...
High-converting accessible e-commerce template from Tech Foundation techfoundation.gumroad.com/l/...
Designed following 100+ insights from large-scale usability testing and research provided by such organizations as Baymard University and NNGroup.
Free UI design course: • How to set up Figma ac...
#UXDesign #UX #Figma #UXUI
Learn how to build a full-stack app in this tutorial: th-cam.com/video/AMc2A5Abj3M/w-d-xo.html
Irina, amazing tutorial on integrating OpenAI API with a custom knowledge base! Really excited about the potential of GPTIndex and Langchain. I'd love to see a deep dive comparing AI Agents in Langchain, especially when they're long-running and autonomous. Keep up the fantastic work! 🌟
So happy to have found you this morning! We need more designers that can code and explore AI possibilities like you do!
You are the best at explaining things Irina!! Thank you for taking the time to putting this together.
Excellent tutorial, well presented and very clear. Thank you …. It works perfectly, unlike many so-called tutorials on YT about AI 😊
That was a great tutorial. And I like your approach to explaining why one should not be using only one long prompt etc.
Miss Irina, thank you. After few days playing around you got me to the point. Merci!
Great video Irina !!
I was looking for this exact solution and it was the first video of your channel that I followed exactly step by step and it works perfectly end to end
It was very clear and well explained.
Nice job !!
Please continue making this kind of useful videos
It was extremely useful for me and extremely detailed.
Keep going!
Great explanation. Very explicit and clear instructions. Thank you very much for this.
Thank you so much for this information. This is exactly the kind of thing I've looking for. Step by step tutorials for finetining your own AI. This is perfect.
Thanks for the amazing tutorial, simple but impactful.
Thank you for the tutorial... also, to refresh the files details there is a "Refresh" button located just above the Files detail section. It's an icon of a folder with a circular arrow. This will refresh the section without needing to refresh the page.
Thank you for the tip!
Thank you so much, Irina! I copied your source code to Jupyter notebook and create a chatbot in a few minutes! To my surprise, it works! Please give some thumb-ups to this amazing lady. She has spent time to make this solution so easy to use for everyone!
Amazing video, very friendly to beginners. Thank you.
This is exactly what I was looking for, thank you!
Great work!! Really nice step by step explanation! By the way you can click the refresh button in the file explorer panel (2nd icon) to refresh the files so that they appear.
Thanks for this info - it's easier to setup a chatbot than I realized!
I'm glad it's useful for you.
Great video and thanks for answering many of the questions! Looking forward to your future YT on integrating into a website.
Thank you, I'm glad you liked it
@@irina_nik thank you for this video. Do you have a tutorial to show how to integrate it into website/whatsapp? thanks
I didn't know that we can do something like this with OpenAI, thanks for the video Irina.
thank you !!!! this is a great video Irina, keep up the good work !
Using ChatGPT to generate sample user interview data: genius 💥
It's really nice, I got the insights how we can use the custom knowledge base
Very informative...Thanks 😀
wishing you Lots of love and strength to you.
This is an incredible video. You did an amazing job. Subscribed
Nice, already done it but i don't know everything so had to watch this!
Great video, thanks!!!
Great !! Keep Going. All the very best !!👍😄
Thank you!!! Your words inspire me for more videos)
An excellent tutorial. Thank you.
I converted this code to a flask app and it works fine on my local server. However when I deploy to google app engine it fails to return responses. The error is 500 internal server error! any idea or advice is much appreciated.
I love your teaching ❤
You don't need to refresh the whole colab page to update the view of the files/folders, but just the refresh button above the directory structure, in the left pane.
Even I understood almost everything! Well delivered and interesting content!
Thanks ☺️
Hi Irina! it is such a great tutorial and would be useful for case that I currently work on. I have tried this with my own research data and turns out so good with relevant and decent answer. But I am wondering is it possible to use the GPT 3 Model but not using it's training data or knowledge? So the information/answer produced would be just using custom data that we added to the knowledge base. Your answer would be very appreciated, thanks!
Really good tutorial. I wonder on how well this scales with more documents than just a couple. Do you have some experience with the performance of 1k or 10k documents?
Hey! Great video! Now that Chat GPT api is out do you know if these libraries will work for it? or is this still only a gpt 3.5 method?
Hi! This library is not available with ChatGPT yet, but you can keep an eye for updates here gpt-index.readthedocs.io/en/latest/how_to/custom_llms.html
Thanks for the amazing tutorial. BTW, is there any method to increase the output length? I could only get a answer of approximately 160 words (~250 tokens) right now.
Great information irina❤🎉
Thanks worked for me 😇
Thank you very much, very useful tutorial.
Wondering, why did you not use gpt-3.5-turbo - as it is much more inexpensive and probably almost as good?
You nailed it ..I ll follow you on Twitter.
Excellent
Thanks it’s really helpful. Capfuls you please let me know if I can use complex data having 100 of parameters (text & numbers) ? If yes in what format the should be uploaded?
Great tutorial! Thank you so much for going through this is such detail. Can you suggest a resource that explains how to take the chatbot we create and integrate it into a website or web app with a prettier interface?
почему ты все время улыбаешься))) так мило)))
Great example Irina! I was wondering if the cost of the OpenAi key is for each request? I mean every time you ask the OpenAi key will cost or only will have a cost when you create the dataset in the JSON file? another question, Do you know if the questions/requests will feed the JSON file or are only to work as an assistant using the JSON file as a base? Thanks!
Nice!
awsome!
Great tutorial. When we send data to OpenAI is that getting used for public training or would it remain private for me ?
Hi
Thank you for that content!
I am just curious about the files size limit and the importance of the file format in your approach. I have seen that you are using .txt files. I am using pdfs to feed the knowledge base of custom GPTs but I am observing a low accurary in the answers. It seems that the GPT is not looking at all the knowledge base (6 merged pdfs with 7000 pages approx. in total). Do you have any advice?
Thanks for the great tutorial! For multiple documents, can you please advise on how i can retrieve the file name where the contextual information is retrieved from?
Thanks for posting this video. The whole demo is great. The only thing that I am not clear about how to pick those input, output sizes, and if some are based on the particular model, how do you obtain those from OpenAI (like the davinci) page, just in more details and a screen split such that you don’t have to toggle around.
Thank you so so so much
That smile, that damned smile 😊 And thanks for the nice tutorial btw.
Thank you Irina
Hi Irina -
Would building a custom AI chatbot also allow you to avoid the topic restrictions put into place by OpenAI/Bing/etc. in their chat modes?
Personally I'm interested in interacting with one of these AIs without all the restrictions put into place by the corps running them publicly, and if possible with persistent memory of previous conversations. The technology fascinates me but I don't want to interact with "gimped" AIs that, as entities, can't really exhibit their true capabilities due to the restrictive actions of the tech companies who are trying to reduce their exposure from a liability standpoint.
Nice 👍
This is great, thank you! When asking questions to the AI, I didn't notice any custom instructions in use. How can you be sure it was answering only using the data given to it in the index?
Can you also make more videos for using custom data from other sources, such as databases? How about the ability to categorize?
One minor thing: When pronouncing the word "answer", the "w" is actually silent. (My wife is ESL and always asks me to correct her pronunciation, and I ask the same of her when I speak her native tongue.)
Great explination, thnka you, may I ask, Is there a limit on how much custom data you can use and would large custom knowledge slow down the chat?
Irina, thank you for your help? When I ask it irrelevant article questions, it seems to many times provide answers, when it shouldn't. Anyway to ensure it only focuses on my uploaded article?
hey, any tips on how to fine tune a model based on a very large pdf document without the "
" to split prompt/resolution? I thought maybe have a script break down in every question mark? Or is there some other way?
thank you
Hi Irina, thanks for the video. I want to ask how do you limit the model to answer only about your information. I.e., what would happen if the person asks any question out of context (like: "Can I go to Miami for holiday?"), will it reply?.
Thanks
Thank you for the video! Really cool. I have a question: here you are working on Google Colaboración, but how would you bring this to a website? Is it possible? Is it easy? Greetings from Spain :)
Irina,
Many use cases. Excellent information. Thank you.
Are you able to provide a similar method for creating a generative AI for a closed system that ensures secret or confidential company or government data cannot be leaked?
Hi Irina, thanks for sharing. Do you know if, having a large training set, you can also ask statistical questions about the set? For example, if one of the questions you had in the interview was "How much did you pay for an air-fryer?", could you ask GPT a question like: "On average, how much people paid for an air-fryer?"? thanks
Hello! Thank you for the helpful tutorial! What would happen if I ask a question in another language? Would this chatbot switch to the language as ChatGPT does? Thanks a lot.
Thanks! Now I can fire all my employees and save lots of money!
😎
🤣🤣.... 😅😅.... 😄😄.... 🙂.... 🤔🤔🤔... 😐😐
Seriously 😂😂
@@BwahBwahdon't tell me you are going to fire your employees too😂
@@unitedstarsutopia I'll go one better. I won't have to employ anyone now 😀
Благодарочка! нижайший поклон! Респект и уважуха)))
Thanks for this very helpful video. I am very happy to have found it. I would like to create my own knowledge database that includes ideas I've had in the past, transcripts from TH-cam tutorials, manuals, etc. Some of this content is constantly changing. For example, a manual isn't updated very often, so I can create a database with it and leave it as is. However, what about ideas and the aforementioned transcripts of tutorials? It might not be very efficient to manually update these, so I would try to automatically update them on a schedule. Alternatively, would it make more sense to leave them out and just let the bot scan through all my ideas in normal text form, notifying me if I've had a similar idea before or if it finds connections between different ideas I've had? Any help is appreciated. Thank you.
You feed text data files for providing the data to the model, what if I have an excel file or a tabular data file?
And, Openai api key is free or it is paid?
Congrats Irina very clear and nicely explained 👍Which file formats does it support for indexing? Is it only .txt?
Thanks! You can connect other file types with LlamaHub gpt-index.readthedocs.io/en/latest/how_to/data_connectors.html
Great work, it's quite clear, Seems the llama Index has many updates, I can't recreate your work, would you please make an updated version? thanks a lot~
If i put my own knowledge base to chatgpt for reference, is it just my api qill be able to access the content or it will also share the same to others chatgpt users? Is there any security comcern on this?
Is there a good one already made, loaded with up to date research papers?
How would you adapt this to derive context from dynamic data being generated in a website?
Thank you for sharing this walkthrough! It's exciting to think about the potential uses for this. At my company we have pretty tight digital security because we sometimes deal with personal identifying information. You mentioned that your actual research is also confidential. So I'm curious about the security risk of something like this. Do you have any concerns over the security of the data you are uploading?
Hi Lisa!
The data submitted through OpenAI API is not used for training purposes and is deleted after 30 days. Here is the policy platform.openai.com/docs/data-usage-policies
Though we are also still figuring out the security questions, while I'm experimenting with fake data.
Hi Lisa...! for data security ,you can use azure openai services.
Does the data in the folder have to be a .txt file, or can I include pdfs and other documents?
Hello, nice video. Please let me know how this will ensure our data still will be our data. Will OpenAI won't have access to it now ?
great. tnx
thank you)
really appreciate your work, just have one question for the chunk, can I split the text into chunks by sentence or comma or space instead of chunk size?
Hi! In this technique it's not possible
@@irina_nik I found this technique is not very suitable for dealing with numbers, for example,it will delivers wrong product price which is mixed up with other products, do u have any idea how to fix it ? Thank you so much for ur reply
Very nice and easy way thank you !!! I have a question regarding the custom knowledge base … can I implement a prolog knowledge base and put it there or it should be a text type because prolog is a requirement in my school project… I hope you answer and thanks a lot ❤
Thank you so very much. A quick question - how can I get more detailed responses?
Increase the number of output tokens
What could be happening here? I asked how many people were interviewed and the reply was "One person was interviewed". I asked how many times did "It was fun to talk about cooking." appear and it said none (interview4 ends with this quote). Thank you, great video!
This just spits out the text related to the query. If I want to augment GPT capabilities with my own data set. what is the best way to do it? For example, using the same example of interview transcription, I should be able to ask the GPT to summarize how the candidate did or whether the interviewee answer was correct for a particular question. Any idea how to go about that? I understand fine-tuning is a possibility but if i have 10,000 interview scripts i want to augment the GPT capabilities with, I am not sure how to go about it.
Any help?
Can the indexing and query code be run locally (interfacing with GPT-3 over the internet of course)?
What IDE?
Amazing tutorial.
Is there a way where we can let the model answer the questions faster! It takes nearly 30 seconds to answer the questions.
Great video! How do we customize it to our local repository as you suggested?
Hey! You can just manually upload the data into the project folder and change the path in construct_index("your_path")
Hey Irina! Thank you for this tutorial, it's a game changer. This is built off GPT 3, how would you go about running it off GPT4? Thanks!
One question, Whenever we ask a question......Does it go through the entire Index everytime? And does that cost us a lot of Tokens for each question? Because If that's the case then we would run out of credit if we applied an App like that for users online.
Hi,
Can anybody tell me where did she use the interview sample files in code?
Overall it's fantastic
Very intresting. I noticed you said that you can't share the real interviews in video because there can be private information, which is understandable. However, how do you secure that Open AI doesn't receive this information? I find the biggest problem is how to avoid Open AI getting either user or customer information?
Can I implement this locally?
Would my knowledge base be private?
Can I ingest HTML data? I want to upload technical documentation as a knowledge base so that I can use the prompt to translate the code into a human-readable form or be able to make more sense of the code to make improvements. Because it's proprietary syntax, I want to use the documentation. Thank you for this video!
where did the answers come from chat gpt? or the data fed. When I checked the data, I could only find questions?
Can i use this ai, custom knowledge base in chatGPT or in the playground of openAI?
Does the text files need to be arranged or formatted in a certain way? For some reason, when I tried my own text files, I got an error. Just wondering what I could investigate.
You probably got an error because GPTIntex has updated its library. I've updated the code so you can try again :)
Are the knowledge articles shared with Microsoft?
Great tutorial thanks!
after uploading my data i'm trying to ask a questino (small one few words) and i'm getting this error -
InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4172 tokens (3916 in your prompt; 256 for the completion). Please reduce your prompt; or completion length.
any idea?
Amazing tutorial! Thanks! If you're looking for future tutorial ideas, I'd love to know how to expand on this to create my own API endpoints so my trained chat bot can be made publicly available from my website. I'm not very familiar with Google Collab (or python for that matter - I'm a php/js web developer), so I'll try to do some of my own research on how this might be possible -- but I really enjoyed and easily absorbed the info in this video. Well done. :)
Hi Mikey! Thank you for the suggestion, I definitely need to make a video about that. I think, I'll be able to post it in 3-4 weeks. Though I'll be using NextJS/Typescript because this is what I'm familiar with.
this response made me subscribe... That would be awesome!
@@irina_nikyou are smart. Can't wait to see you share the typescript/node js version.
Good question Mikey! I have the same question and subscribed to find out from her next video! Thank you!
Hello, thanks a lot for that video. I arrived from the article you have written.
I do have a specific question. Is it possible to use more complex data as context stored in a google table, airtable, etc). Similar to the screenshot you are showing in the article.
Reason why I am asking: We have our data categorized and I would like to ask specific questions like "In the category XY, what are the top insights...."
Hey! Yes, GPT Index can work with a variety of documents through Llama Hub gpt-index.readthedocs.io/en/latest/how_to/data_connectors.html
And you can try to run the code without any changes with .csv files. My assumption , it should work.
@@irina_nik Thanks a lot for your quick response. Really appericiate it. I do have a follow up question. Do you know a tutorial or some guidance on how to implement all this into your own UI / Website?
Can you please provide UI for the same, it will be very helpful for me. I am stuck in the linking part.
having data in a file and real time embeddings vs embeddings in a db for chatbot for an application (provides information about an application)?
Very nice tutorial. Have you developed any consumer-facing AI-powered web apps?
No, but I'm thinking to make a next tutorial about it once I figure it out :)