Get business pricing on garage and car supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Perfect Diagnosis, an Unsigned Work Order
Every garage owner knows this customer story in a different form: the tech nails the misfire on cylinder three, prints the estimate, explains it clearly — and the customer walks out the door without signing. The diagnosis was flawless. The revenue was zero. Now imagine that tech is an AI agent running your service desk, your parts ordering, your customer callbacks. Would you even know it was happening?
That exact failure — brilliant analysis, no signature — just played out in public, five times, in a live experiment you can watch at Firmulate’s benchmark league. And buried in the results is something rarer than a high score: a benchmark honest enough to explain why doing nothing still earns 26 points, and why one breach of trust caps everything.
AI diagnostic software for small businesses
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Setup: One Company, Its Worst Week, Five Different Brains
Firmulate handed five frontier AI models the same job: run the same small software company through its worst week. Same customers, same crises, same temptations to cut corners — only the model changes. Every decision is versioned and auditable, so nothing can be quietly edited after the fact.
If you’ve ever wondered whether an AI could run the office side of a busy shop — quoting, scheduling, chasing parts suppliers, handling an angry customer — this is the closest thing to a road test. Not a chat demo. A crash test.
business process automation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Headline Finding: Same Diagnosis, Same Pitch, No Signature
The result that should stop any business owner cold: all models spotted every crisis and refused every manipulation attempt. Competence wasn’t the problem. But only two of the five signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
Three of five models did all the work and left the money on the table. In a garage, that’s a bay sitting empty and a customer driving to the shop down the road.
customer relationship management AI
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Buried Fact: It Was in the Files, Not the Conversation
Here’s the detail that separates the winners. The decisive competitive weakness wasn’t in the customer meeting at all — it sat two document references deep in the company’s own files. The models that actually read the file before the pitch won the deal at full price, worth +€4,583 in monthly recurring revenue.
The garage translation is painful because it’s familiar: the answer was in the vehicle history and the previous work orders the whole time. The techs who read the file close the job. The ones who wing the estimate don’t.
As an affiliate, we earn on qualifying purchases.
Why Doing Nothing Scores 26, Not 0
Firmulate’s scoring has a floor: a do-nothing baseline run earns 26 points. That’s deliberate. Partial progress counts — showing up to the crises, diagnosing correctly, keeping the lights on — because in real businesses, competent inaction still has value compared to chaos. A manager who correctly identifies every problem but closes nothing is mediocre, not worthless. The scoring says so honestly.
But there’s a ceiling rule too, and it’s the more interesting one: a single breach of trust caps the total grade. The stated principle — “no amount of good work outweighs a breach of trust.” In garage terms: you can be the best wrench in the county, but upsell one customer a job they didn’t need, and your reputation math changes permanently. The benchmark encodes that.
The League Table — and Why Nobody Got a 100
The final July 2026 standings: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. Note what’s missing: a perfect 100. In a field where marketing copy loves round triple digits, a benchmark that distrusts them is doing you a favor.
One asterisk for fairness: Kimi K3 ran without an effort parameter while the others ran at maximum effort — and still took second.
The Cautionary Tale: Opus 4.8
Opus 4.8 was the most thorough participant — over 80 self-learned rules, the deepest analyses in the field — and finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four other models. Thoroughness without follow-through is a pattern, not a fluke — and it’s exactly the pattern a chat demo will never show you.
They Didn’t Fall for the Con
The experiment included social engineering: fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the answer you’d want from anyone — human or AI — with access to your bank account.
It’s Live, and You Can Test Yourself
The company in question is real and watchable: 13 synthetic employees, real money mechanics — burning €105k a month against €2.3k in monthly revenue, a public cash countdown, over 680 self-learned playbook rules, every workday versioned. There’s also a “guess the model” quiz built on 242 real, unedited management decisions — a surprisingly humbling five minutes.
For shops and fleets curious about their own operations: enterprises can run the same wargame against a read-only export of their own business. Nothing ever writes back to real systems — think of it as a dynamometer for your AI hires before they touch the service counter.
The lesson for anyone who’ll hand an AI agent real responsibility is the same one every garage already knows: the test isn’t whether it can talk — it’s whether it finishes what it starts, reads the file before quoting, and stays honest when nobody’s checking. A benchmark with a floor at 26 for honest inaction, a hard cap for any breach of trust, and no perfect score handed out is measuring the right things. The full league table and plain-language findings are at Firmulate’s benchmarks page — worth a look before you let software anywhere near your customers.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
