Very good discussions ! I have been following LangChain for some time now and is very much aligned with the promising vision for the future you portray ! You were discussing about how a data/BI analyst approaches a query. My two cents on this is their approach is primary domain driven and not entirely technical meta data driven. Therefore, to improve the accuracy and relevance of LLM generated SQL queries, you may consider training models using domain-specific datasets.
I would agree, it's so much domain driven that it takes people years to understand the unique company acronyms and paradigms. I am sometimes jeleaus of people who run SELECT * type of queries. In my case it's a few nested subqueries if you're lucky, and a beast procedure if you're not. Maybe we should keep the expectations in check as the field evolves and grows. The idea of Software 3.0 (1.0 + 2.0 = 3.0) was really cool in the video. Also very having a test suite is a must I would say to be able to run many experiments quickly, even if it's not perfect. Awesome discussion, learned a lot!
One issue I constantly run into is, the model understands the question and generates the correct query, however, gives an incorrect number as the answer as opposed to manually running the query...
maybe if you could include an additional LLM that is tuned for code or more specifically SQL code that is separate from the LLM that knows your data. The additional LLM trained for coding could be used to generate your sql queries?
so far, gpt-4 is still the best LLM for coding as well and especially for SQL queries. One reason could be that, so many people have used chatgpt for writing sql queries.
I am not sure how you would finetune an LLM on your data schema!!! You would be needing QnA pairs for finetuning. Therefore, what makes sense to me is that you curate a dataset in which the Q will be a requirement (e.g, find me the top 5 sold products and order them by their prices) and it's Answer will be an SQL query. You will be needing to create at least a 1000 of examples like these and will also have to make sure that the examples are diverse enuf to include all your tables/columns etc.
Very good discussions ! I have been following LangChain for some time now and is very much aligned with the promising vision for the future you portray ! You were discussing about how a data/BI analyst approaches a query. My two cents on this is their approach is primary domain driven and not entirely technical meta data driven. Therefore, to improve the accuracy and relevance of LLM generated SQL queries, you may consider training models using domain-specific datasets.
I would agree, it's so much domain driven that it takes people years to understand the unique company acronyms and paradigms. I am sometimes jeleaus of people who run SELECT * type of queries. In my case it's a few nested subqueries if you're lucky, and a beast procedure if you're not. Maybe we should keep the expectations in check as the field evolves and grows. The idea of Software 3.0 (1.0 + 2.0 = 3.0) was really cool in the video. Also very having a test suite is a must I would say to be able to run many experiments quickly, even if it's not perfect. Awesome discussion, learned a lot!
One issue I constantly run into is, the model understands the question and generates the correct query, however, gives an incorrect number as the answer as opposed to manually running the query...
maybe if you could include an additional LLM that is tuned for code or more specifically SQL code that is separate from the LLM that knows your data. The additional LLM trained for coding could be used to generate your sql queries?
so far, gpt-4 is still the best LLM for coding as well and especially for SQL queries. One reason could be that, so many people have used chatgpt for writing sql queries.
how to mix both SQL and RAG to answer user question using LangChain?
When should we expect langChain for noSQL?
couldn't you fine tune your LLM with your data model / schema so that the schema does not need to be part of the prompt?
I am not sure how you would finetune an LLM on your data schema!!! You would be needing QnA pairs for finetuning. Therefore, what makes sense to me is that you curate a dataset in which the Q will be a requirement (e.g, find me the top 5 sold products and order them by their prices) and it's Answer will be an SQL query. You will be needing to create at least a 1000 of examples like these and will also have to make sure that the examples are diverse enuf to include all your tables/columns etc.
or maybe, I was just thinking, we could use gpt-4 to help us create 1000s of Text-to-SQL (QnA) pairs for finetuning.