# Published comparison examples

These are six selected run records from the authors of [The Complexity Trap](https://arxiv.org/abs/2508.21433), not TurnScope performance measurements. The paper was submitted on 2025-08-29 and revised on 2025-10-27. The examples compare the published Qwen3-Coder 480B SWE-agent raw-history and observation-masking runs on three SWE-bench Verified tasks.

Source: [author CSV at pinned commit](https://github.com/JetBrains-Research/the-complexity-trap/blob/bf15b5fb7d279679035a007ac9a81084d6b9a89a/auxiliary-data/experiment_instance_costs_sweagent.csv). The immutable raw download is `https://raw.githubusercontent.com/JetBrains-Research/the-complexity-trap/bf15b5fb7d279679035a007ac9a81084d6b9a89a/auxiliary-data/experiment_instance_costs_sweagent.csv`. Commit: `bf15b5fb7d279679035a007ac9a81084d6b9a89a` (2025-11-18). The source contains 7,587 records. SHA-256 of the complete downloaded CSV: `47b550fd7a4dfafb59a269e73f50ade96a9abd9484e32b17498581b4251eff48`.

The CSV is distributed in the authors' MIT-licensed repository. [Pinned license](https://github.com/JetBrains-Research/the-complexity-trap/blob/bf15b5fb7d279679035a007ac9a81084d6b9a89a/LICENSE), SHA-256 `7610ed3916f6674e34b78417894abd57ff538b3cfdda3085e3643d82acbaf31f`. The separate Hugging Face trajectory archive has its own license; these examples derive from the GitHub CSV.

## Mapping and limits

- `instance_id` becomes `task_id`; `turn_count` is retained as the run-level turn count; `outcome` 1/0 becomes passed/failed.
- `cost` is the authors' modeled total dollar amount as published. Equal lower/upper values mean an exact copy of that published scalar, not an exact invoice or an uncertainty-free scientific measurement. These selected rows have zero summary cost. No tool fees or unreported billing details are added.
- `trial_id`, `evaluator_id`, `scope_id`, and `assumptions_id` are explicit TurnScope adapter labels identifying the selected published runs and shared evaluation/accounting context. They were not literal columns in the source and do not constitute independent verification.
- Rounded `mean_prompt_tokens`, `mean_completion_tokens`, and `mean_reasoning_tokens` are retained only in the source rows below. They are not converted into invented exact token totals or per-turn traces.
- These cases demonstrate comparison decisions on supplied historical evidence. They do not establish causality, statistical significance, or that any new retention policy preserves quality. The examples were deliberately selected to show three different decisions, not to estimate benchmark-wide effects.

## Files and expected decisions

| File | Task | Expected verdict | SHA-256 |
| --- | --- | --- | --- |
| [comparison-quality-regression.json](comparison-quality-regression.json) | `django__django-11138` | `regression` | `ff070ebeafc2c4e40230eb13c9b7bc1c62a2eee9841c9a7f98966ba1e6012327` |
| [comparison-within-budget.json](comparison-within-budget.json) | `astropy__astropy-7336` | `within_budget` | `86305168bbb999aaae4862a14ea60a97f71cb58370b081f6f4a56c44a53339d6` |
| [comparison-cost-regression.json](comparison-cost-regression.json) | `astropy__astropy-7166` | `regression` | `8e4a5dd9d3caa4a0e0e70d3f580a4d2181152451904bec3d06e06d61d3c101e4` |

The quality-regression example drops cost from 1.6722 to 0.6003 while changing passed to failed. The within-budget example retains a pass at 0.2193 versus 0.1969. The cost-regression example retains a pass but rises from 0.2180 to 0.7235. All use a zero-percent allowed cost regression.

## Original selected source values

The following CSV preserves every source column for the selected records, ordered baseline then candidate for each example. Only row order and line endings were normalized. SHA-256 of this UTF-8 CSV block including its final LF: `89e9e184b0af24a58dffc5035199d84e0c7138e99117197e36504c06d4201190`.

```csv
experiment,strategy,instance_id,cost,summary_cost,has_summary,summary_count,turn_count,mean_reasoning_tokens,mean_completion_tokens,mean_prompt_tokens,outcome
local-Qwen3-Coder_480B_A35B_Instruct_FP8-baseline_raw.0-verified-500,Raw Agent,django__django-11138,1.6722,0.0,False,0,67,0,127,35303,1
local-Qwen3-Coder_480B_A35B_Instruct_FP8-baseline_N_1_M_10.0-verified-500,Observation Masking,django__django-11138,0.6003,0.0,False,0,63,0,132,11951,0
local-Qwen3-Coder_480B_A35B_Instruct_FP8-baseline_raw.0-verified-500,Raw Agent,astropy__astropy-7336,0.2193,0.0,False,0,33,0,76,14335,1
local-Qwen3-Coder_480B_A35B_Instruct_FP8-baseline_N_1_M_10.0-verified-500,Observation Masking,astropy__astropy-7336,0.1969,0.0,False,0,22,0,97,10260,1
local-Qwen3-Coder_480B_A35B_Instruct_FP8-baseline_raw.0-verified-500,Raw Agent,astropy__astropy-7166,0.218,0.0,False,0,36,0,116,12797,1
local-Qwen3-Coder_480B_A35B_Instruct_FP8-baseline_N_1_M_10.0-verified-500,Observation Masking,astropy__astropy-7166,0.7235,0.0,False,0,72,0,168,12661,1
```

## Source license notice

```text
MIT License

Copyright (c) 2024 John Yang, Carlos E. Jimenez, Alexander Wettig, Shunyu Yao, Karthik Narasimhan, Ofir Press

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```