Evaluating LLM-based Applications // Josh Tobin // LLMs in Prod Conference Part 2

This portion is sponsored by Gantry.
Website: gantry.io/
A simple, powerful SDK for model instrumentation
Gantry's SDK gives you easy access to all of your production data and metrics, just by adding a few lines of code.
//Abstract
Evaluating LLM-based applications can feel like more of an art than a science. In this workshop, we'll give a hands-on introduction to evaluating language models. You'll come away with knowledge and tools you can use to evaluate your own applications, and answers to questions like:
Where do I get evaluation data from, anyway?
Is it possible to evaluate generative models in an automated way? What metrics can I use?
What's the role of human evaluation?
//Bio
Josh Tobin is the founder and CEO of Gantry. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel.