“It’s a simple question for any enterprise. Would you like something that’s 10 percent of the cost if it’s 90 percent as good as the best, or do you want to pay that 10 times higher cost?”
That really depends…
90% is disasterous. It may sound good because it’s “just” 10% off 100%, but it’s awful. Two iterations reduces that to ~0.01%< but that’s still pretty bad. Obviously the more complex, the higher the risk and the more demand and time per iteration. The machine also has no concept of criticality, so how much is being saved with three or even four iterations to assure failure is basically zero?
“Then the question is: how many tasks require that? I would tell you some do, but the vast majority, summarizing an email, examining a contract, answering a customer call, composing an email, all of those can be done as well with the much cheaper answer.”
Imagine 10,000 emails an hour through your enterprise and 1,000 times It fucks up. Anything from missing info to misinterpreting to just straight up modifying/deleting. No concept of criticality per email. 8,000 in a 9–5 office. 24,000 a day for a global enterprise. That’s what 90% looks like.
Then you got “examining contracts” and all the other stuff he mentioned…
Don’t assume the expensive model is 100% accurate. The cheap model is 90% as accurate as the expensive model, which is already less than 100% accurate.
It doesn’t literally mean “90% failure rate vs 100%”
In the HLE benchmark, for example, we may be talking about 45% test score for MiMo Pro or GLM or something, vs 55% for Sonnet and 61% for maxed out Claude.
But further complicating that, those particular open weight models do better than Claude in one particular hallucination benchmark, meaning Claude/ChatGPT might be worse at RAG/extraction type workloads. It’s like this across the board, with different models trading blows in different areas.
By “90%,” he really means “it’s so close the better model depends on the particular application, workload, harness, sampling configuration” and such, and that none of these models are close to “100% accuracy” across the board.
One other thing: Anthropic, OpenAI, and Grok have all been accused of “dumbing down” models under high demand to speed up inference. If your application depends on them, and they are the only provider, that could throw a wrench in the workflow.
But open weights? Pick your provider. You get an API as consistent as you are willing to pay for.
In other words, that “100%” isn’t really representing the potential inconsistency one gets being locked to a single provider.
That really depends…
90% is disasterous. It may sound good because it’s “just” 10% off 100%, but it’s awful. Two iterations reduces that to ~0.01%< but that’s still pretty bad. Obviously the more complex, the higher the risk and the more demand and time per iteration. The machine also has no concept of criticality, so how much is being saved with three or even four iterations to assure failure is basically zero?
Imagine 10,000 emails an hour through your enterprise and 1,000 times It fucks up. Anything from missing info to misinterpreting to just straight up modifying/deleting. No concept of criticality per email. 8,000 in a 9–5 office. 24,000 a day for a global enterprise. That’s what 90% looks like.
Then you got “examining contracts” and all the other stuff he mentioned…
Don’t assume the expensive model is 100% accurate. The cheap model is 90% as accurate as the expensive model, which is already less than 100% accurate.
It doesn’t literally mean “90% failure rate vs 100%”
In the HLE benchmark, for example, we may be talking about 45% test score for MiMo Pro or GLM or something, vs 55% for Sonnet and 61% for maxed out Claude.
But further complicating that, those particular open weight models do better than Claude in one particular hallucination benchmark, meaning Claude/ChatGPT might be worse at RAG/extraction type workloads. It’s like this across the board, with different models trading blows in different areas.
By “90%,” he really means “it’s so close the better model depends on the particular application, workload, harness, sampling configuration” and such, and that none of these models are close to “100% accuracy” across the board.
One other thing: Anthropic, OpenAI, and Grok have all been accused of “dumbing down” models under high demand to speed up inference. If your application depends on them, and they are the only provider, that could throw a wrench in the workflow.
But open weights? Pick your provider. You get an API as consistent as you are willing to pay for.
In other words, that “100%” isn’t really representing the potential inconsistency one gets being locked to a single provider.