‹ All episodes

Deep Papers

Breaking Down EvalGen: Who Validates the Validators?

May 13, 2024 Arize AI
Breaking Down EvalGen: Who Validates the Validators?
Deep Papers
More Info
Deep Papers
Breaking Down EvalGen: Who Validates the Validators?
May 13, 2024
Arize AI
Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators often inherit the problems of the LLMs they evaluate, requiring further human validation.

This week’s paper explores EvalGen, a mixed-initative approach to aligning LLM-generated evaluation functions with human preferences. EvalGen assists users in developing both criteria acceptable LLM outputs and developing functions to check these standards, ensuring evaluations reflect the users’ own grading standards.

Read it on the blog: https://arize.com/blog/breaking-down-evalgen-who-validates-the-validators/

To learn more about ML observability, join the Arize AI Slack community or get the latest on our LinkedIn and Twitter.

Apple Podcasts Spotify Overcast Castro Castbox Goodpods +
Show Notes
Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators often inherit the problems of the LLMs they evaluate, requiring further human validation.

This week’s paper explores EvalGen, a mixed-initative approach to aligning LLM-generated evaluation functions with human preferences. EvalGen assists users in developing both criteria acceptable LLM outputs and developing functions to check these standards, ensuring evaluations reflect the users’ own grading standards.

Read it on the blog: https://arize.com/blog/breaking-down-evalgen-who-validates-the-validators/

Paper: https://arxiv.org/abs/2404.12272

To learn more about ML observability, join the Arize AI Slack community or get the latest on our LinkedIn and Twitter.