Just posted this above so reposting here This one certainly is a lot easier to use. Both TextGrad and DSPy really focus on treating it as if it were a deep learning network etc. For me, this one is probably better for people who don't have that kind of experience. I've been meaning to make an in-depth video about DSPy for a while, and just never gotten around to it, I do like it as a project.
I can see how something like this would be useful in a RAG pipeline where as new documents are added a LLM instance could create a base dataset representing the content of the document (or the entire knowledge base) and then use that with PW to create extensive prompts that are subsequently used to evaluate chunk/retrieval performance.
Cool! I've just tried the example problems from the Colab notebook using my Llama3.2 3b with a simple prompt "Please solve the following problem:" All the results were correct, with nice concise reasoning steps. To justify the amount and cost of optimization I would like to see some counter examples showing what value this approach actually adds over the baseline.
I'm thinking of sticking Lite LLM support in there, and making it so that you could try it on any model, including local models, et cetera. That would make it much more effective cost-wise to be able to try things out
This tool looks like it could be extremely helpful for my needs. I have to extract documentation from a couple of source code files, organize it all into sections, and output a YAML representation. It’s hard to know where to begin to get an LLM accomplish this task. It needs to be decomposed into stages, and the LLM needs to have a peek at the documentation that it’s generating so it can improve phrasing, do language translation, etc., and within a small context window so the LLM doesn’t choke. I already have a parser written the old fashioned way that can extract individual items from the sources into a generic JSON or YAML format, so I just need that final step of leveraging the LLM. I can write a TOOL to grab any requested part of the JSON and test it within an environment like Open WebUI, but my brain is fuzzy on how to make a reliable system that can be reused. Any tool that helps refine the prompts, whether for the whole process or to build agents, is most welcome! I’m sure this is a common enough problem that there are already some services popping up. But this seems like a good learning opportunity.
This one certainly is a lot easier to use. Both TextGrad and DSPy really focus on treating it as if it were a deep learning network etc. For me, this one is probably better for people who don't have that kind of experience. I've been meaning to make an in-depth video about DSPy for a while, and just never gotten around to it, I do like it as a project.
Enterprise and research don't seem to care about token counts, especially not if they're investing in RL datasets or will be saving tokens in deployment. 🔥💰 I clenched when Sam said he ran it for 20 minutes lol
Just checked, it wasn't that bad it was $13.70. I feel this is pretty reasonable if it is an important prompt etc. The dataset was not small as well. I was thinking of showing this with DeepSeekV3 or Gemini if there is interest which could make it much cheaper etc.
Thank you for the insightful video. Could you elaborate on whether token usage is a concern for users when employing this framework? Also, would you say it’s only effective for specific tasks or use cases, rather than being broadly applicable? How feasible would it be for someone to develop a similar prompt optimization tool independently?
The cost wasn't super high it was $13.70 . You can certainly use it for your own tasks, it would work better for things where there is a clear correct answer. you could develop something like this yourself but this is totally open so you can leverage of this
Just making a simple tool to optimize your prompt to a lvl 3 or lvl 4 prompt will go and long way. Once you start to use lvl 3 and lvl 4 prompts you will really see what the llm is capable of.
The problem is that this method does not work if you do not know the answer to the problem, or the answer might not be precise like this math. None of the real-worl apps I've seen can use this.
Yes, this is a really good point - it works best for things where there is a clear and correct answer. Using it for things like creative writing etc is probably not going to be as good. Ahem That said, I still think it can be used in certain kinds of real-world apps.
Love your videos man, this space is so noisy but your uploads are pure quality.
Thanks thats really appreciated
Id love a comparison between dspy and this! We should start evaluating frameworks
Just posted this above so reposting here
This one certainly is a lot easier to use. Both TextGrad and DSPy really focus on treating it as if it were a deep learning network etc. For me, this one is probably better for people who don't have that kind of experience. I've been meaning to make an in-depth video about DSPy for a while, and just never gotten around to it, I do like it as a project.
I totally agree. It would be good to start doing some kind of evaluation by comparing all of these kinds of things to each other.
I’ve been thinking about this concept for a while; very glad there’s a concept out in the wild
I can see how something like this would be useful in a RAG pipeline where as new documents are added a LLM instance could create a base dataset representing the content of the document (or the entire knowledge base) and then use that with PW to create extensive prompts that are subsequently used to evaluate chunk/retrieval performance.
Cool! I've just tried the example problems from the Colab notebook using my Llama3.2 3b with a simple prompt "Please solve the following problem:" All the results were correct, with nice concise reasoning steps. To justify the amount and cost of optimization I would like to see some counter examples showing what value this approach actually adds over the baseline.
I'm thinking of sticking Lite LLM support in there, and making it so that you could try it on any model, including local models, et cetera. That would make it much more effective cost-wise to be able to try things out
This tool looks like it could be extremely helpful for my needs. I have to extract documentation from a couple of source code files, organize it all into sections, and output a YAML representation. It’s hard to know where to begin to get an LLM accomplish this task. It needs to be decomposed into stages, and the LLM needs to have a peek at the documentation that it’s generating so it can improve phrasing, do language translation, etc., and within a small context window so the LLM doesn’t choke. I already have a parser written the old fashioned way that can extract individual items from the sources into a generic JSON or YAML format, so I just need that final step of leveraging the LLM. I can write a TOOL to grab any requested part of the JSON and test it within an environment like Open WebUI, but my brain is fuzzy on how to make a reliable system that can be reused. Any tool that helps refine the prompts, whether for the whole process or to build agents, is most welcome! I’m sure this is a common enough problem that there are already some services popping up. But this seems like a good learning opportunity.
Compare promptwizard to textgrad and dspy
This one certainly is a lot easier to use. Both TextGrad and DSPy really focus on treating it as if it were a deep learning network etc. For me, this one is probably better for people who don't have that kind of experience. I've been meaning to make an in-depth video about DSPy for a while, and just never gotten around to it, I do like it as a project.
Just curious - do you have any particular use cases for DSPy or TextGrad?
How much did this cost in the end? Seems like this could use a lot of token?
a fortune
Enterprise and research don't seem to care about token counts, especially not if they're investing in RL datasets or will be saving tokens in deployment. 🔥💰 I clenched when Sam said he ran it for 20 minutes lol
Just checked, it wasn't that bad it was $13.70. I feel this is pretty reasonable if it is an important prompt etc. The dataset was not small as well. I was thinking of showing this with DeepSeekV3 or Gemini if there is interest which could make it much cheaper etc.
@ that’s very reasonable, thanks for the info.
Thank you for the insightful video. Could you elaborate on whether token usage is a concern for users when employing this framework? Also, would you say it’s only effective for specific tasks or use cases, rather than being broadly applicable? How feasible would it be for someone to develop a similar prompt optimization tool independently?
The cost wasn't super high it was $13.70 . You can certainly use it for your own tasks, it would work better for things where there is a clear correct answer. you could develop something like this yourself but this is totally open so you can leverage of this
Just making a simple tool to optimize your prompt to a lvl 3 or lvl 4 prompt will go and long way. Once you start to use lvl 3 and lvl 4 prompts you will really see what the llm is capable of.
Great job!✌ Colab link not working
I just checked it it should be ok. drp.li/58ni6
most of the time model under 8B quite confused using too long prompt am i the only one got that problem?
The iterative optimization part is essentially a genetics algorithm.
Yes, in that way it's quite similar to prompt breeder from DeepMind
thanks - very usuful :)
seems like DSPy
indeed it is like DSPy but has some of the cool ideas from DeepMind's prompt breeder and is easier to use I would say
The problem is that this method does not work if you do not know the answer to the problem, or the answer might not be precise like this math. None of the real-worl apps I've seen can use this.
Yes, this is a really good point - it works best for things where there is a clear and correct answer. Using it for things like creative writing etc is probably not going to be as good. Ahem That said, I still think it can be used in certain kinds of real-world apps.