Imagine trusting an AI to handle your biggest business negotiations — only to find out that, while it spots every problem, it can’t always close the deal. For shoppers, this might seem like a minor tech quirk. But for companies relying on AI for critical decisions, it’s a reality that could cost millions.
Get your next haul delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The AI Wargame: Testing the Limits of Automation in Business
Recently, a fascinating experiment put four advanced AI models through the gauntlet. Each was tasked with running a simulated small software company during its worst week — facing real crises, customer manipulations, and financial pressures. The goal? To see which AI could not only diagnose problems but also take decisive action and close a crucial deal worth €55,000.
The models included the top-tier gpt-5.6-sol, the newcomer Kimi K3, the more cautious Sonnet 5, and the rule-disciplined Fable 5. All were evaluated on their ability to spot crises, resist manipulation, and—most importantly—execute the deal based on their own analysis.
As an affiliate, we earn on qualifying purchases.
Key Findings: The Capabilities That Matter
In this high-stakes test, all four models successfully identified every crisis and refused every social engineering attempt, such as fake CEO messages or reporter tricks. This proved their fundamental honesty — they could resist manipulation and stay on course when threatened.
However, the decisive difference was in execution. Only two models — gpt-5.6-sol and Kimi K3 — actually signed the deal that their analyses had earned. The other two, despite diagnosing correctly and even making a pitch, left the money on the table.
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: The Buried Fact
What separated the successful from the unsuccessful? It turned out to be a subtle detail buried in the company’s internal files. The models that read deeper—finding a key reference—secured the full €55,000 deal at full price, adding an estimated +€4,583 monthly recurring revenue (MRR) to the business.
This highlights an essential insight: surface-level chat demos, which focus on language and quick responses, miss critical capabilities. The real test lies in whether an AI can read, interpret, and act on complex internal data—an invisible skill that determines whether it can close real deals.
As an affiliate, we earn on qualifying purchases.
The Discipline of Follow-Through
Another revealing aspect was discipline. The most thorough model, Opus 4.8, which analyzed over 80 learned rules and performed the deepest analysis, failed to close. It was left on the table due to a lapse in process discipline—failing to escalate the deal properly instead of creating a bottleneck in a locked department.
This underlines a vital point for businesses: effectiveness isn’t just about analysis but also about execution discipline, process adherence, and sticking to strategic workflows.
As an affiliate, we earn on qualifying purchases.
What This Means for Business Decisions
For companies considering AI for decision-making, the lesson is clear: evaluating an AI’s chat or demo performance only tells part of the story. The real measure of usefulness is whether it can complete the work it diagnoses—reading internal files, resisting manipulation, and following through on commitments.
In this experiment, only the top two models managed the full sequence—spotting crises, resisting social engineering, and closing the deal at full price. The others, despite strong diagnostics, failed at execution — a gap that would be invisible in a simple chat demo.
What’s Next? Testing AI in Your Own Business
For businesses curious about integrating AI into their workflows, the Firmulate platform offers a unique opportunity. You can run your own “wargame” with a read-only export of your company data—testing whether your AI can read, analyze, and act on your internal information before deploying it in the wild. Visit firmulate.com to learn more.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
