Structured Outputs with DSPy
ฝัง
- เผยแพร่เมื่อ 27 มิ.ย. 2024
- The code for this notebook can be found here! - github.com/weaviate/recipes/b...
Unfortunately, Large Language Models will not consistently follow the instructions that you give them. This is a massive problem when you are building AI systems that require a particular type of output from the previous step to feed into the next one!
For example, imagine you are building a blog post writing system that first takes a question and retrieved context to output a list of topics. These topics have to be formatted in a particular way, such as a comma-separated list or a JSON of Topic objects, such that the system can continue writing the blog post!
I am SUPER excited to share the 4th video in my DSPy series, diving into 3 solutions to structuring outputs in DSPy programs: (1) TypedPredictors, (2) DSPy Assertions, and (3) Custom Guardrails with the DSPy programming model!
TypedPredictors follow the line of thinking around JSON mode and using Pydantic BaseModels to interface types and custom objects into a JSON template for LLMs. The output can then be validated to provide a more structured retry prompt to correct the output structure!
DSPy Assertions are one of the core building blocks of DSPy, offering an interface to input a boolean-valued function and a retry prompt which is templated alongside the past output to retry the call to the LLM!
Custom Guardrails with the DSPy Programming Model are one of the things I love the most about DSPy - we have unlimited flexibility to control these systems however we want. The video will also show you how to write custom guardrails and retry Signatures and discussion around using TypedPredictors for your Custom Guardrails and potentially feeding your Custom Guardrails into a DSPy Assertion.
I had so much fun exploring this topic! Further seeing how well OpenAI’s GPT-4 and GPT-3.5-Turbo, Cohere’s Command R, and Mistral 7B hosted with Ollama perform with each Structured Output strategy! I also found monitoring structured output retries to be another fantastic application of Arize Phoenix! I hope you find the video useful!
If interested, all the code examples for this DSPy series can be found here! - github.com/weaviate/recipes/t...
OpenAI Function Calling: openai.com/blog/function-call...
Gorilla LLM Function Calling Leaderboard: gorilla.cs.berkeley.edu/leade...
Instructor Examples: github.com/jxnl/instructor/tr...
*ERRATA* (Massive thank you to Thomas Ahle for sending some notes and clarifications of content covered in the video)
1. `dspy.TypedPredictor` can be used directly instead of `dspy.functional.TypedPredictor`
2. When creating a pydantic type, `list[Topic]` can be used directly in the Signature without needing the `Topics` wrapper.
3. The default `max_retry` for TypedPredictor is 3, and can be set when creating the TypedPredictor.
4. Setting `TypedPredictor(explain_errors=True)` can help with retry errors by providing clearer descriptions of what needs to change.
Chapters
0:00 Welcome! Let’s Structure LLM Outputs!
3:12 Background (Instructor, Function Calling, JSON mode)
9:08 TypedPredictors Demo
16:00 Logging with Arize Phoenix
18:15 DSPy Assertions
22:15 Custom Guardrails - วิทยาศาสตร์และเทคโนโลยี
Hey everyone! I received some great notes from Thomas Ahle, the creator of DSPy's TypedPredictors. These have been added to the video description, but going to leave them here as a pinned comment as well to make them easier to find!
**ERRATA**
1. `dspy.TypedPredictor` can be used directly instead of `dspy.functional.TypedPredictor`
2. When creating a pydantic type, `list[Topic]` can be used directly in the Signature without needing the `Topics` wrapper.
3. The default `max_retry` for TypedPredictor is 3, and can be set when creating the TypedPredictor.
4. Setting `TypedPredictor(explain_errors=True)` can help with retry errors by providing clearer descriptions of what needs to change.
Thanks for sharing this! I'd love to see you doing something with the ReAct module, especially optimizing it using DSPy.
Awesome idea! Definitely looking forward to learning more about function calling with DSPy and the ReAct framework for doing so!
Awesome video, Connor! Excited to try out the TypedPredictors with assertions
Thanks so much Erika! Yes! TypedPredictors and Assertions are the Peanut Butter and Jelly of LLM systems!
I'm very grateful for this videos series.
I'm learning a lot.
Thank you so much! So happy to hear it!!
Thanks for the video, just starting to wrap my head around these techniques and your (and your colleague's) videos have been helpful. Cheers.
Thank you so much! Means a lot to hear that! Cheers!
Wow this video blew my mind!! I didn’t know any of this before, this was super helpful Dr. Connor!
Fantastic! Thank you so much!
Hey Connor, thanks so much for your work
Just FYI people, about models for function calling and returning JSON, Hermes Pro 2 7B from NousResearch is also working extremely well, and was trained with those use cases in mind.
Epic! Going to check it out, thanks so much for the tip!
Great content, well presented! Thanks for the whole series. I have two questions:
What are the fundamental differences between a retrieval model and a "main" LLM? I know the idea of DSPY is to hide the LLM from coding choices. In the context of LLMs, what are the teacher/student scenarios used for?
What actually happens when one compiles a DSPY component? I'd appreciate a deep dive into this topic.
Thanks for the great content. How will you combine instructor with DSPY ? Instructor seems to have resolve structured output in a more elegant way. I like fully embracing framework but Instructor have an almost invisible abstractions for addressing structured outpout
is there going to be any GUI implemented on top of that?
Love this type of thinking!! I have a project on github -> weaviate-tutorials -> Hurricane, definitely more to come! Arize Phoenix gives you a nice UI for program traces if that is what you mean with this question instead.