Yann Dubois: Scalable Evaluation of Large Language Models

[@dheerajagrawal9107]

Here are a few key takeaways from Yan's talk on evaluating large language models:
1. Evaluation is important for identifying model improvements, selecting the best model for a use case, and determining if a model is ready for production.
2. Desired properties of evaluation include scalability, relevance, discriminative power, interpretability, reproducibility, and robustness to gaming.
3. Evaluating LLMs is challenging due to the diverse set of possible tasks and open-ended nature of responses.
4. Common evaluation approaches include:
- Converting open-ended tasks to closed-ended (e.g. multiple choice)
- Reference-based heuristics comparing model outputs to human references
- Human evaluation
- LLM-based evaluation using another model as judge
5. LLM-based evaluation (e.g. Alpaca Eval) can be scalable and correlate well with human judgments, but requires careful design to avoid biases.
6. Challenges include consistency across implementations, contamination of test data in training sets, quick saturation of benchmarks, and incentives to keep using outdated metrics.
7. Future directions may involve more human-in-the-loop approaches (like rubric-based evaluation) and using LLMs to generate more targeted evaluation instructions.
8. Overall, evaluation of LLMs remains an active area of research with many open challenges to address.

[@403saket]

Thank you prof!