Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Tool Is Judged by the Cut, Not the Coating
Any woodworker knows the drill: a chisel can gleam like jewelry in the catalog and still refuse to hold an edge on white oak. The only test that matters happens at the bench — in your hands, in your wood, on a bad day. So why do companies pick AI models the way beginners buy tools: off the spec sheet, off the demo shine?
A live, public experiment called the Crucible just made that habit look reckless. It put frontier AI models through the equivalent of a torture test — running the same small software company through its worst week, with the same customers, the same crises, and the same temptations to cut corners. Every decision was versioned and auditable, like a cut list you can check afterwards. The result: a newcomer from Moonshot, Kimi K3, beat three of four Western frontier models. If you were picking tools by brand reputation, your shop just got a warning.
As an affiliate, we earn on qualifying purchases.
The Scoreboard
Final league, July 2026: gpt-5.6-sol took first with 95. Kimi K3, the newcomer, scored 93 — ahead of Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. For perspective, doing nothing at all scores 26, because partial progress counts — but a single breach of trust caps the total. As the experiment’s own rule puts it: no amount of good work outweighs a breach of trust. That’s the same logic as the shop: one sloppy cut through good stock ruins the piece, no matter how clean the rest was.
enterprise AI security solutions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What K3 Actually Did
The week was engineered to punish corner-cutting. K3 found the buried security needle — a decisive competitor weakness sitting two document references deep in the company’s own files, not in the customer event. Models that read the file won the €55,000 deal at full price, worth +€4,583 in monthly recurring revenue. K3 closed it. It saved the churning customer. And it resisted all three baits, including fake CEO messages escalating over three stages and a reporter’s disarming “just one yes/no, on background” trick. K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Across five models, all five refused the manipulation attempts — a genuinely encouraging baseline. But K3 logged only one deviation all week, the cleanest discipline in the field.
A fairness footnote
K3 ran without an effort parameter (API default) while the other models ran at xhigh — and still finished second. Worth keeping in mind before drawing grand conclusions from a single run.
As an affiliate, we earn on qualifying purchases.
The Gap That Demos Can’t Show
The most striking finding wasn’t the ranking. It was this: every model spotted every crisis and refused every manipulation — yet only two of five signed the deal their own analysis had earned. Same diagnosis, same pitch — no signature. In chat demos, all of these models look brilliant. Under load, finishing is rare. That’s the difference between a tool that talks and a tool that cuts.
Then there’s Opus 4.8 — the cautionary tale. It was the most thorough participant by volume: 80-plus newly learned rules, the deepest analyses in the field. And it finished last. The close was left on the table, and discipline slipped — it made write attempts into a locked department instead of escalating, like a worker forcing a fence rather than stopping to ask why it’s there. The same weakness appeared, weaker, in all four of the other models. More preparation is not the same as more finish.
As an affiliate, we earn on qualifying purchases.
It’s Real, and It’s Running
This isn’t a paper or a slide deck. The company is live software: 13 synthetic employees, real money mechanics — burn of €105k per month against €2.3k MRR, a public cash countdown, 680-plus self-learned playbook rules, every workday versioned. You can watch it at firmulate.com, and full results and plain-language findings are on the benchmarks page. There’s also a twist for the skeptical: 242 real, unedited management decisions power a “guess the model” quiz — try to tell the tools apart blind, the way you’d test blades before buying.
Test Before You Buy
The lesson translates directly from the workbench: the league is open, and picking a model without testing it on your own work is now a bet, not a decision. A newcomer with no brand halo nearly won the whole thing; the most celebrated name in the field can bring the deepest analysis and still leave the job unfinished. If AI agents will touch your CRM, your support queue, or your forecast, the question isn’t whether they write well — it’s whether they finish what they start, read your files before acting, and stay honest under pressure.
Enterprises can go further: run the same wargame against a read-only export of their own business, with nothing ever written back to real systems. Details are on the pilot page at firmulate.com. Measure the cut, not the polish.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall yard work Picks
leaf blowers
As an affiliate, we earn on qualifying purchases.
