ChatGPT Work의 로그인 연속성·Opik의 LLM 평가 도구·개발자 자동 리팩터링 소식

Greg Brockman

@gdb

ChatGPT Work for using websites which require login:

11 LLM evaluation methods AI engineers must know: (bookmark this) Two eval metrics can rank the same two models in opposite orders, and neither one is wrong. A model that paraphrases the reference can score near zero on BLEU and near the top on BERTScore for the exact same output. Neither metric is wrong because one is measuring wording and the other is capturing meaning. This is why LLM evaluation is fragmented into several methods, depicted in the visual below and grouped by what each one assumes: > Reference-based (ground truth exists): - BLEU - ROUGE - BERTScore > Judge-based (no ground truth): - G-Eval - LLM-as-Judge - LLM juries > Human and deterministic: - Human eval - DAG > Built for agents: - Trajectory accuracy - Multi-turn eval > Run as a gate: - Safety eval To use them in practice, most of these metrics are already implemented in Opik, which is open source (20k+ stars) and runs them over traced production data. You can start using them in a few lines of code. GitHub repo: https:// github.com/comet-ml/opik (don’t forget to star it ) That said, metrics only point at the failing case. The rest of the work is still done manually, like inspecting the trace to see where the span went wrong, editing a prompt or a tool description, re-running, and checking that the fix did not break anything else. My co-founder wrote a walkthrough (with code) that automates this loop using Opik. It explains the full lifecycle where a failing trace gets diagnosed, the fix runs against the exact input that failed, and that input stays in the eval set as a regression case so it does not recur. Read it below.

Peter Steinberger

@steipete

Hit a new record with our autoreview skill. 66 rounds on a gnarly refactor.

This skill is the biggest contributor to my lack of slop, and also the need for the many accounts I have

DogeDesigner

@cb_doge

Elon Musk on early days of SpaceX: "When I started SpaceX, one of my friends got a compilation of rocket failures & made me watch the whole thing. I knew the probability of SpaceX failing was high"