# Research behind TurnScope 0.3

Reviewed 2026-11-09. These sources motivate experiments and measurement choices. Their reported results are not claims about TurnScope's own performance. This service has not run a new agent-quality benchmark or established that a particular context change preserves task success.

## Papers

1. **Complexity Trap, 2025.** [Primary paper, v3](https://arxiv.org/html/2508.21433v3), [author code and data](https://github.com/JetBrains-Research/the-complexity-trap). The SWE-agent study compares raw history, observation masking, and summaries across five model configurations. Effects vary by model: the thinking Gemini configuration lost solve rate with both reduction methods. This motivates paired task outcomes and a recent-observation retention experiment, not automatic deletion. The public examples use actual author-reported costs and outcomes; they do not reconstruct missing per-turn telemetry.

2. **AgentDiet, 2025/2026.** [Primary paper, v2](https://arxiv.org/html/2509.23586v2). The tested system delays LLM-based trajectory reduction and includes reflection overhead in its cost model. Results concern the evaluated agents, models and software-engineering benchmarks. This motivates keeping optimizer calls, output costs and recovery steps inside the comparison's accounting scope. A hash-based analyzer cannot identify semantic irrelevance just because content recurs.

3. **Hidden Costs of Forgetting, 2026 preprint.** [Primary paper, v1](https://arxiv.org/html/2610.35496v1). The authors test trajectory deletion across two software-engineering agents. Re-fetching and extra turns can offset reductions, and their flat versus caching-aware price assumptions reverse apparent benefits. Their cache model assumes a 1:0.1:5 input/cache/output price ratio and available matching prefixes. This supports cache-aware intervals and freshness checks; it does not establish that all summarizers or provider caches behave that way.

4. **AWO / meta-tools, 2026.** [Primary paper, v2](https://arxiv.org/html/2601.22037v2). Repeated action chains can motivate composite tools, but the studied method uses domain and action semantics to preserve effects. Sparse tool names or repeated paths do not establish independence, safe batching, or equivalent behavior. TurnScope therefore exports reviewable experiment packets with preconditions rather than executable transformations.

5. **AI Agents That Matter, 2024.** [Primary paper](https://arxiv.org/html/2407.01502v1). Joint evaluation of task success and cost reveals weaknesses in accuracy-only agent comparisons. This motivates matching tasks and trials, retaining failures and unmatched records, and making evaluator identity explicit. A single passing paired run is an observation, not statistical proof of equivalent performance.

## Existing tools and telemetry

- **OpenTelemetry GenAI:** [current inference specification](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md), [attributes](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/registry/attributes/gen-ai.md), [OTLP JSON](https://opentelemetry.io/docs/specs/otlp/#json-protobuf-encoding). Input/output counts include their detailed subsets. Missing cache detail is not necessarily zero. Nested client/server inference spans and multiple services must not be summed blindly. These conventions are evolving; this adapter documents and tests its exact accepted fields.
- **OpenInference / Phoenix:** [semantic conventions](https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md), [client export reference](https://arize-phoenix.readthedocs.io/projects/client/api/spans.html). LLM spans expose prompt/completion counts and cache details. Provider-native message text does not establish exact block tokens or complete selection evidence. The import consequently retains usage, leaves block inventories empty, and keeps tool selection unknown.
- **Collector file exporter:** [official implementation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/fileexporter). Uncompressed JSON exports provide an existing JSONL ingestion path. TurnScope's local client processes these files without adding a live collector or storing the original payload on the service.
- **Langfuse:** [usage accounting](https://langfuse.com/docs/observability/features/token-and-cost-tracking). Current usage detail buckets are mutually exclusive, unlike inclusive OTel totals. Directly treating its `input` bucket as total input would undercount. Native Langfuse export support is deferred until its distinct accounting and optional export fields have their own fixtures.
- **Inspect AI:** [usage reference](https://inspect.aisi.org.uk/reference/inspect_ai.model.html#modelusage), [changelog](https://inspect.aisi.org.uk/CHANGELOG), [events](https://inspect.aisi.org.uk/reference/inspect_ai.event.html#modelevent). SDK version 0.3.184 changed input accounting; cache reads/writes must be considered under the correct version. Local response-cache hits differ from provider prompt-cache hits. Native import is deferred rather than guessing from a file-format version alone.
- **Promptfoo:** [standalone assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/). Existing deterministic evaluation harnesses can supply task outcomes without asking TurnScope to judge its own suggestions. The comparator records the caller's evaluator ID and outcome; it does not execute the evaluator or independently verify its claim.

## What this release proves

Deterministic tests can establish accounting, import fidelity, pairing, conservative interval decisions, privacy boundaries and bounded resource use. Published examples can demonstrate that the comparator catches a cheaper failed task and distinguishes a successful cost improvement from a successful cost regression. New evidence is still required before concluding that a proposed change improves a user's agent. Record the same tasks, evaluator, scope, prices and trial identifiers; include all relevant calls and report missing runs.

The original v1 cost model remains a user-rate input/cache-read/output estimate. It does not price cache-write premiums, tools or optimizer services automatically. Imported write counts remain in local import metadata and imported traces have no assumed prices. Use an explicitly attributed total cost measurement for comparisons when those charges matter.

## What 0.3 adds after the second review

The additional paper/tool review and independent Grok critique identified three gaps in the original workflow: successful rows alone cannot establish the intended trial denominator; aggregate usage does not capture cache-write and helper-model costs; and flat turn lists cannot establish ordering, ownership or concurrency.

Version 0.3 addresses those gaps with caller-declared expected-trial manifests, separate execution status and evaluator outcome, model-specific rate catalogs and explicit cache-write classes, and event metadata with ownership, dependencies, timestamps and ordered fingerprints. The local recorder produces metadata; it does not infer missing usage or attest to a caller's completeness declaration. The server remains an analysis service, not a context-rewriting proxy.

These changes improve what can be inspected. They do not establish measured savings or benchmark quality. Incomplete costs remain unknown, failed work remains in observed totals, and the full study verdict takes precedence over a completed-only subset. See the accounting and study guides for exact semantics. The original v1 three-rate trace estimate retains its narrower scope for compatibility.