The model is one, the harness is five

I kept a running argument going for most of September about which model was strongest. Someone would post a result showing model A handled a tricky problem, someone else would counter with model B doing the same, and the thread would end in disagreement because neither result reproduced on the other person's machine. Two people comparing things they could not see, in conditions they had not controlled, and calling the difference model quality.

The argument ended with a sentence from a colleague who was tired of it: the model is one, the harness is five. Not literally. The point was that model choice accounts for a small share of what you observe, and everything around the model accounts for the rest. Once I accepted that, the disagreements got small enough to test, and testing ended them.

Why model rankings mislead

A benchmark measures a model inside someone else's harness. Their router, their prompts, their tool permissions, their context handling, their container. When you bring that model into your setup, you bring none of those. You bring your own router, your own prompts, your own everything. The number does not transfer, and the failure is not that the benchmark lied. It is that the benchmark measured a different system than the one you run.

I watched this play out with a model everyone agreed was strong. In one teammate's setup it was excellent. In another's it was mediocre at the same task. Same model, same version, same day. The variable was everything else. The one with the better result had spent more time on the harness, and less on the model.

The useful version of model comparison holds the harness constant and changes one thing. That is a smaller experiment and it produces a real answer. Ranking models on feel across different setups produces an argument you cannot win, because the two of you are not measuring the same thing.

The signals that actually meant something

Once I stopped ranking and started measuring, a few concrete behaviors emerged that were worth knowing. They are narrow, and that is the point. A model being good or bad overall is not actionable. A model being good at one specific job is.

One model turned out to be excellent at finding bugs and weak at writing prose. It produced text that read like it came from a completely different model, flat and generic, but it could hunt down defects in a codebase with a precision the writing model did not have. I stopped giving it writing tasks and started giving it bug-hunting tasks, and its value went up immediately.

Another model was cheap and accurate on bounded tasks and dangerous on open ones. Given a clear job it finished cleanly. Given latitude it re-read context, looped, and burned its entire quota window on a task that should have taken twenty minutes. Cheap per token, expensive per task.

A third model, mid-tier and unremarkable in benchmarks, was more complete at medium effort than at high. At high effort it drifted into overthinking and produced a less usable answer. I had been leaving it on high out of habit and paying more for worse output. Effort is a per-role setting I test now, not a global dial I assume.

None of these findings came from a leaderboard. All of them came from running the model on my work, in my harness, and writing down what happened. And I want to state their level plainly: these are patterns I noticed across a few weeks of my own tasks, judged by me, on setups I did not hold constant between observations. They are signals worth testing, not findings. I have not run the clean version yet, same task, same harness, one variable changed, results recorded, and until I do, the sentence at the top of this post is a principle for how to argue, not a measurement of anything.

The quirks that taught me the most

The oddest observation of the period came from a teammate who noticed a model reasoning in Chinese before answering a Vietnamese question. His suspicion, stated as a suspicion and never proven, was that the model had been trained on data distilled from an open-weight model with Chinese origins, and that the reasoning trace leaked the ancestry. Nobody resolved whether this was true. What got resolved was how I treat it: as a hypothesis about training data, not a proven fact, and not something that changed how I used the model day to day.

The more useful lesson was about how easily one stray observation becomes a shared belief. A single line about a model behaving strangely, passed through a group of forty people, becomes consensus within an hour, and consensus within a group becomes a decision about which tools to buy. I started labeling observations by their evidence level, the way you would in any other engineering context. A guess is not a finding, and the difference matters when the finding is going to change your budget.

The same discipline applied to a long argument about two model families. One side had a demonstration where a model failed a logic problem. The other side pointed out that a single problem, however clever, does not evaluate a model's reasoning quality, and that results differed across versions of the same model on the same problem. Neither side had a standard to measure against, so neither side could win. The argument ran for days and produced the sentence that became my rule. The model is one. The standard is what I lack.

Where agents actually fall short

Separating model quality from harness quality also clarified where the real limits are, and the limits turned out to be about tasks rather than models.

Agents write books that read like a product catalog. Given a broad topic, they produce a wide, shallow survey with a chapter on everything and depth on nothing, which is the statistically safe output for any topic at all. Narrow the topic and the output improves immediately. The fix is scoping, not a better model.

Agents generate code that needs reading differently than human code. A teammate described reading about ten percent of it, focusing on the patterns that tend to be wrong, the queries that look too clever, the logic that got complicated, and ignoring the rest. Someone else framed it as a habit rather than a filter: keep asking questions of the code instead of reading it top to bottom. A codebase you did not write and cannot fully hold still has to be trusted somewhere, and the skill is knowing where to check.

Agents produce interfaces with a specific failure mode around mobile. A landing page that looked fine on my laptop was unusable on a phone, and nobody had checked. The design feedback I got included a concrete framework for the fix, a storytelling structure that puts the user's situation in the headline instead of the product's features, a short guided path instead of a wall of options, and a call to action before the testimonials rather than after. That framework did not come from a model. It came from a person who had built and shipped things, and the agents executed it once it was stated clearly.

What I changed

Three practices came out of this period and stuck.

When comparing models, change one variable and hold the rest. Same task, same harness, same day. Record the result. A comparison you cannot reproduce is an opinion wearing a number.

Write down findings with their evidence level. A single anecdote is not a benchmark. A pattern across three tasks is a signal. A reproducible result on a fixed harness is a finding. Most model arguments are two people sharing anecdotes and calling them findings.

Scope work for the agent's actual strength, not the model's reputation. If a model is good at finding bugs, give it bugs. If it is good at bounded tasks, bound the tasks. If higher effort makes it worse, lower the effort. The model's overall standing tells you almost nothing about how to use it well.

The ranking charts are not useless. They calibrate where to start. What they cannot tell me is how a model will behave inside my setup on my work, because that is a property of the setup and the work as much as the model. The score that matters is the one I take myself, on the tasks I actually run, with the harness I actually built.