About this session
Shipping a model-backed feature without an evaluation suite is shipping blind. Marcus shows how Halcyon builds eval sets from real traffic, gates deploys on regression scores and keeps human review affordable. You will leave with a template for your first 50-case eval and a way to grow it.
What you'll take away
- How to turn production logs into a labelled evaluation set
- Where LLM-as-judge works, and where it quietly lies to you
- How to wire evals into CI without a ten-minute wait