David Gerard@awful.systemsM to TechTakes@awful.systemsEnglish · 14 hours agoOpenAI o3 beats FrontierMath — because OpenAI funded the test and had access to the questionspivot-to-ai.comexternal-linkmessage-square11fedilinkarrow-up158arrow-down10
arrow-up158arrow-down1external-linkOpenAI o3 beats FrontierMath — because OpenAI funded the test and had access to the questionspivot-to-ai.comDavid Gerard@awful.systemsM to TechTakes@awful.systemsEnglish · 14 hours agomessage-square11fedilink
minus-squareNazlinkfedilinkEnglisharrow-up1·13 hours agoYou were right. I make open models from scratch and I’ve tested the corporate benchmark banks and some of the results they’ve gotten are extremely sus. After the Volkswagen Dieselgate Scandal, I’ve taken metrics not reported by independent auditors with a chonking peanut scooper of salt. For the curious: My best results so far have been like 74% on HumanEval with a 405B, zero-shot induction.
You were right.
I make open models from scratch and I’ve tested the corporate benchmark banks and some of the results they’ve gotten are extremely sus.
After the Volkswagen Dieselgate Scandal, I’ve taken metrics not reported by independent auditors with a chonking peanut scooper of salt.
For the curious:
My best results so far have been like 74% on HumanEval with a 405B, zero-shot induction.