Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Anthropic always publishes 3p benchmarks every time they announce a new model


No, stop right there. Anything published by Anthropic implicitly is not third party. For it to be third party, the third party has to be the one publishing it.


When you're announcing a new model, typically, nobody else has benchmarked it yet, because it hasn't been released yet. You can still run 3p benchmarks on it and publish those results. If other parties later run the same benchmarks independently, and find major discrepancies, that would be a scandal.


And even if they didn't, they have a track record. Even if we did have benchmarks in this case I would still wait until people got there hands on it and formed a more holistic opinion.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: