Everyone asks the same question in the workshops: which AI should I use? So we stopped answering it from opinion. We gave five models the identical job, under identical rules, and watched. Four finished. One — the one running privately on our own machine here in Thon Buri — never did.
Narrated walkthrough of the benchmark — every figure drawn from the run data.
The job
Not a quiz. A small, boring, completely real piece of work. Each model got the same codebase and the same instruction: open one file, let a tool answer to two short nicknames as well as its full name, write a test proving the nicknames work and a wrong name is still rejected, run the test, change nothing else.
Same starting point for everyone, a fresh copy of the code each time, a cap on steps and a cap on reading. To pass, a model had to finish inside the rules and leave work that survived our own inspection. Claiming success wasn’t success.
What happened
The final round gave every model the same generous budget, so nothing could be blamed on being starved of room to work.
| Model | Where it ran | Result | Time | Work produced |
|---|---|---|---|---|
| DeepSeek V4 Flash | Cloud | Passed | 7.8s | Clean, 23 lines |
| Kimi K3 | Cloud | Passed | 17.9s | Clean, 23 lines |
| GLM 5.3 Flash | Cloud | Passed | 16.7s | Correct, 31 lines — one redundant edit |
| Claude Fable 5 | Cloud | Passed | 37.9s | Clean, 23 lines |
| Qwen 3.8 27B | Local — our Mac Studio | Failed | 210.4s | Unfinished, 4 lines |
One run per model · 1,000,000-token budget · 31 August 2026
Every model that passed changed exactly the two files it was supposed to change, wrote the test it was asked for, and passed all 13 checks when we ran them ourselves. The differences between them were about speed, cost and tidiness — not about whether the job got done.
Time to finish the same job seconds · shorter is better
Qwen’s 210 seconds did not end in a finished job. It ended in the safety guard shutting it down.
The more revealing number isn’t speed. It’s how much material a model had to plough through before it was satisfied — the closest thing we have to a measure of focus, and on a paid service, the bill.
Total text processed input tokens · lower is better
Four models decided they had seen enough at around 100,000. Qwen read three and a half times as much and still produced nothing usable.
The local model deserves its own section
We care about this one more than the rest. Baan Arun runs a local model on our own hardware precisely so a business can put confidential material in front of an AI without it leaving the room. We wanted it to win, so we tested it hardest.
The first failure looked like a resource problem — it hit the reading limit. So we raised the limit. Then raised it again. Then removed it as a practical constraint entirely.
| Reading budget | Outcome | Actually read | Time | Left behind |
|---|---|---|---|---|
| 128,000 | Failed out of budget | 129,654 | 150.3s | Nothing |
| 256,000 | Failed out of budget | 284,841 | 184.0s | Valid work, but over the line |
| 384,000 | Failed out of steps | 352,621 | 225.9s | Unfinished, test missing |
| 1,000,000 | Failed out of steps | 351,602 | 210.4s | Unfinished, test missing |
Qwen 3.8 27B MLX, local, across four budgets — four failures
More room did not help. Once the reading limit stopped being the thing that caught it, the model simply went in circles until a different guard caught it instead — reopening files it had already read, re-editing files it had already changed, and eventually losing the change it was asked to make in the first place.
The honest read. A local model on a single desktop machine is genuinely useful today for private drafting, summarising, extraction and question-answering — work where a person reads the output before anything happens. It is not ready to be left alone with multi-step work. Those are two very different products, and they get sold as one far too often.
This doesn’t change our view that more businesses will move sensitive work onto their own hardware. It sharpens it. The reason to run a model in your own building is confidentiality, not autonomy — and anyone telling an SME that a box in the office replaces a frontier model for unattended work is selling something.
What it costs
The premium option was the only run with a directly reported price: US$0.56 to US$0.64 for this one small task. The cheaper cloud models did the same job to the same standard for a fraction of that. The local model cost nothing per run — and produced nothing. Which is the whole lesson: price per attempt is meaningless until you know the completion rate. A free model that fails is infinitely expensive.
How we’d route the work
This is what we teach in the workshops, and it’s the real answer to which AI should I use? — you use several, and you match each one to a job.
DeepSeek V4 Flash
Default lane
Fastest and most economical of everything we tested, with the tidiest result. What we reach for first, and where most routine work should land.
Kimi K3
Second opinion
Matched the default lane almost exactly. Valuable precisely because it’s a genuine alternative — when one provider has a bad day, you aren’t stopped.
GLM 5.3 Flash
Coding specialist
Correct and quick, a little less economical with its edits. Sensible for development work where extra thoroughness is welcome.
Claude Fable 5
Escalation & review
The most expensive and not the fastest — but it never wandered. Where we send anything difficult, ambiguous, or too important to redo.
Qwen 3.8 27B
Local · supervised only
Stays in the building. Excellent for confidential material a person will read. Not left alone with a task until it can finish this one.
What this is not
One task, one attempt per model. That’s a canary in a coal mine, not a league table. A canary tells you the air is bad; it doesn’t tell you the composition of the gas. We wouldn’t publish a claim about which model is best on this evidence, and we’d treat any vendor who did with suspicion. What four separate failures do support is one decision: local Qwen stays supervised for now.
Two fairness notes we’d want made if these were someone else’s numbers. The premium model was measured through its own tooling, which counts and reuses text differently from the harness the other four ran under, so its figures aren’t perfectly interchangeable. And the local model was the only one running on a single desktop machine rather than in a datacentre — which is exactly the comparison we wanted, but it should be said plainly rather than buried.
How we kept it fair
- Every attempt began from the same commit, in its own clean copy of the code.
- Every model received the identical request and the same two-file boundary.
- Runs were sequential, so no model competed with another for resources.
- In the final round, every model got the same one-million-token allowance.
- Nothing was taken on trust — files changed, tests, commit state and whitespace were all verified independently after each run.
Why we publish this
Baan Arun is vendor-neutral by design. We don’t resell anyone’s model and we have no reason to flatter one. What we can offer is the thing most businesses can’t do for themselves: run the same job across every serious option, under conditions that don’t move, and show the working — including the runs where our own preferred setup loses. We walk through the whole method, live, in AI Infrastructure for Small Business.