On model x harness interactions
Today, a blog post purporting to show that models prefer their own harnesses hit the front page of hacker news. It was based on 27,280 agent runs. It reminded me that I had done a similar analysis out of curiosity, based on 926,588 agent runs!
(How do I have so many agent runs? Because there is a ton of benchmark trajectory data available on Hugging Face. Funny enough, this includes the data from the blog post that triggered me writing this.)
About the data
- 75 benchmark strata (counting source-specific versions and subsets separately)
- 72 different model identities
- 54 different harness configurations
Do models prefer their own harness
The obvious effect we’d expect would be models preferring their own harness. To investigate this, I looked at benchmark runs when a model used its own harness against ones that it used a different harness, then charted the pass rate:

As we can see, most models do not perform differently in their own harness. The only outliers here that are statistically significant are highlighted in the above chart and included below:

Is there a strong model x harness effect?
To investigate this, I built a model to predict benchmark performance with the following coefficients:
- Intercept
- Harness
- Model
- Benchmark
- Task (within benchmark)
- Study
- Model x benchmark
- Benchmark x harness
- Model x harness
If there is a strong effect for model x harness, that is independent of the rest of the effects, we would see a strong signal in the model x harness effect. In fact, we do, but the effect is smaller than the benchmark x harness interaction and the model x benchmark interaction. Here is a chart:
