# TurnScope: import, experiment, compare

## Recommended workflow in 0.3

Use the [evidence guide](/api/v1/evidence-guide) for new experiments. Download both `/api/v1/client.py` (save as `turnscope.py`) and `/api/v1/evidence.py` into the same directory. These Python standard-library clients run locally; network submission requires an explicit command and server URL.

1. Declare tasks, content fingerprints, expected trial IDs, configuration revisions, evaluator, environment and accounting scope before running the experiment. Initialize a local study with `evidence.py study-init`.
2. Record model, tool and wait events with explicit usage ownership. Include retries, failed attempts and helper calls. Hash exact prompt regions and tool arguments locally with the recorder; do not upload raw telemetry. Missing counts, timing, or graph edges remain unknown.
3. Supply your own shared rate catalog, including model-specific fresh-input, cache-read, cache-write classes, output rates and separate fees. Submit prepared metadata to `POST /api/v1/accounting`. An incomplete cost has `cost.total: null` and `measurement_metrics.cost: null`; known partial charges must not stand in for the full bill.
4. Record each execution status separately from the outcome of your actual evaluator. Use `evidence.py measure` and `study-record` to retain results even for failed, timed-out or cancelled work.
5. Submit the complete study to `POST /api/v1/studies/compare`. Check missing-on-both-sides trials, per-task denominators, provenance, failures and objective coverage. `completed_comparison` is only a secondary subset; use the top-level study verdict.

Contracts: `/api/v1/accounting-schema`, `/api/v1/study-schema`. Semantics: `/api/v1/accounting-guide`, `/api/v1/study-guide`. Expected-trial manifests are caller declarations, not independent attestation. The local client guards against accidental edits; deliberate replacement can still change the denominator. Comparisons describe observed runs and do not establish statistical significance or future savings.

The original workflow below remains available for existing version-1 traces and paired measurements. Its three-rate trace cost cannot capture cache-write premiums or external charges; use the new accounting path for those.

Discover capabilities at `GET /api/v1/manifest`, the input contract at `GET /api/v1/schema`, and a clearly synthetic trace at `GET /api/v1/example`. Analyze with `POST /api/v1/analyze` and `Content-Type: application/json`.

## Improve an agent with measured evidence

1. Preserve a baseline and identify the task, trial, evaluator version and accounting scope. Include the agent's helper/optimizer calls and recovery work when measuring full-run usage.
2. Normalize an existing OTLP GenAI or OpenInference export locally with the downloadable Python client. Select the trace and service explicitly when the file contains more than one. The client sends only normalized usage/identity metadata on an explicit analyze command; prompts and tool payloads remain local. Existing version-1 traces also work.
3. Inspect the report's `experiments`. Each packet references evidence in the report and states a hypothesis, change, preconditions, required measurements, acceptance checks and rollback condition. Choose one supported change. Missing inventory or cache usage may require better measurement before optimization.
4. Implement and run the candidate in your own authorized environment. Run the actual external evaluator. Do not fabricate a passing outcome, treat a model's completion message as verification, or relabel failures to make a budget pass.
5. Use `measurement_metrics` plus explicit task/trial/evaluator/scope/outcome fields to create baseline/candidate records. Compare with `POST /api/v1/compare`; the contract is `GET /api/v1/compare-schema`. Keep missing and failed trials visible.
6. Adopt a change only after reviewing the quality and resource evidence required for the task. A `within_budget` result applies to submitted measurements under your selected checks; it does not prove causal improvement or statistical quality equivalence. Repeat trials before generalizing.

The comparator joins by `(task_id,trial_id)`, not array order. A passed→failed transition is a regression despite lower cost. Missing pairs/outcomes, evaluator or scope mismatches, unresolved failures, missing objective measurements, incompatible cost assumptions and overlapping cost bounds prevent a positive verdict. Known regressions can still be reported when other evidence is incomplete. It accepts exact scalar metrics and attributed cost intervals. Cost comparisons require matching currency, basis and assumptions ID.

Verdicts: `within_budget`, `regression`, `insufficient_evidence`. Inspect `reasons`, `coverage`, `outcomes`, `objective` and `pairs`; do not use only the headline. Under `--check`, the client exits 0, 1, and 2 respectively (operational errors also exit 2).

## Local client workflow

Download `GET /api/v1/client.py` as `turnscope.py`. Python 3 standard library only. Set `TURNSCOPE_BASE_URL` to the origin from which you fetched this guide, or a local instance. Normalization has no network access; analyze/compare require an explicit server URL.

```sh
TURNSCOPE_BASE_URL='http://127.0.0.1:5152'
curl -fsS "$TURNSCOPE_BASE_URL/api/v1/client.py" -o turnscope.py
python3 turnscope.py normalize baseline.otlp.json --format genai --output baseline-trace.json
python3 turnscope.py analyze baseline-trace.json --server "$TURNSCOPE_BASE_URL" --output baseline-report.json
python3 turnscope.py measure baseline-report.json --task-id task-1 --trial-id trial-1 \
  --evaluator-id acceptance-tests-v1 --scope-id full-agent-run \
  --outcome unknown --output baseline-measurement.json
```

Replace `unknown` only with the result of your actual evaluator. Repeat for the candidate, keeping the same identifiers and accounting scope, then:

```sh
python3 turnscope.py compare baseline-measurement.json candidate-measurement.json \
  --server "$TURNSCOPE_BASE_URL" --objective input_tokens --max-regression-pct 0 \
  --check --output comparison.json
```

Use `--format openinference` for OpenInference attributes, and `--trace-id`, `--service`, or `--scope` to select an accounting source. Ambiguous nested inference spans, conflicting duplicates, and missing input/output usage are errors. Imports preserve missing cache as unknown and leave block inventories/tool selection unavailable; they do not infer exact block tokens from text. Import accounting covers the selected service/instrumentation, not automatically every process in a distributed workflow. Choose a consistent scope for both runs.

The importer keeps observed cache-write counts in local metadata and supplies no assumed prices. If cache-write premiums, tool fees or optimizer services matter, provide an explicitly attributed total-cost measurement instead of treating the original three-rate estimate as an invoice. Measured or modeled cost records must describe comparable assumptions. Partial cache data can yield overlapping bounds and an inconclusive result.

Comparison metrics are bounded at 1e12 per record. If an analysis has a modeled cost above that bound, its original cost report is preserved, `measurement_metrics.cost` is null, and `measurement_limitations` explains why. Other metrics remain usable; a missing cost cannot pass a cost comparison.

Published examples are available at `GET /api/v1/examples/quality-regression`, `/within-budget`, and `/cost-regression` under the same examples prefix. Source attribution is `GET /api/v1/comparison-provenance`; research and limitations are `GET /api/v1/research`. They demonstrate comparator behavior using author-reported measurements, not a new TurnScope benchmark.

## An agent-ready instruction

Audit this recorded agent run with TurnScope. Convert each model invocation into one ordered version-1 turn. Preserve provider-reported total input/output counts; input includes cached input. Include cached_input_tokens only if known, and never substitute zero for missing usage. Include tool_calls only if selection metadata is complete for that turn; [] means known no calls.

Inventory any known non-overlapping input blocks by id, kind, reported tokens, and a sha256: fingerprint of exact UTF-8 content. Raw content is optional and should be omitted when fingerprints suffice. Changed content needs a new fingerprint even if its label remains the same. Do not invent block token counts or estimate them as exact. Omit unavailable inventories; the report will preserve unattributed input. Keep a stable fingerprint's kind, count, and tool name consistent within the run.

Ask the user for rates if cost is needed; otherwise omit pricing. Rates are per million input, cached-input, and output tokens in one currency. Never fetch or assume a current provider price. Submit only when the user has authorized sending this trace to the TurnScope service, or use a local instance.

Explain reported totals, cache coverage, repeated exposure by kind, later tool-result exposure, unselected-schema candidates, and repeated exact exploration targets with turn references. Separate facts from heuristic review candidates. Repeated context can be useful and cached. Missing selection data cannot prove a schema unused. Findings overlap and must never be added into a savings total. Explain incomplete cost intervals and include limitations. Do not modify the live run or claim guaranteed savings.

## Requests

Use the same scheme and host from which you fetched this guide as `TURNSCOPE_BASE_URL`. The commands below default to a local instance; replace the value with the public service's current origin when using a hosted instance.

```sh
TURNSCOPE_BASE_URL='http://127.0.0.1:5152'
curl -fsS "$TURNSCOPE_BASE_URL/api/v1/schema"
curl -fsS "$TURNSCOPE_BASE_URL/api/v1/example" -o synthetic-run.json
curl -fsS "$TURNSCOPE_BASE_URL/api/v1/analyze" \
  -H 'Content-Type: application/json' \
  --data-binary @synthetic-run.json > report.json
```

The UI's copyable curl example automatically uses its current origin, so custom hostnames, temporary deployment URLs, and local instances all work.

## Important boundaries

- Turns are ordered by array position, not their IDs. IDs must be unique.
- Counts are nonnegative integers at most 1,000,000,000. Cached input is a subset of input. Block sums cannot exceed input.
- Tool calls belong to the model turn that selected them. A result first appears in a subsequent model input. The first observed result appearance need not be its creation step.
- Schema selection compares exact tool names. Exploration compares exact `(kind, target)` pairs across calls; repeated calls can be necessary refreshes.
- Body limit: 2 MiB; 1,000 turns; 256 blocks/calls each per turn; 20,000 blocks/calls each per trace. Content is limited to 32 KiB UTF-8 per block. Identifier byte limits are documented by the API.
- Errors have `{ "error": { "code", "message", "path"? } }`. 400: malformed request; 422: invalid trace; 413: too large; 429: busy, retry after the supplied delay.
- One analysis is admitted at a time, including the lifetime of its response bytes. Read or close the report response before making another request; honor `Retry-After` when busy.
- Analysis uses request memory only: no persistent storage, content logging, cookies, or outbound model calls. Exported reports include the metadata and labels you supplied; review them before sharing.

Costs model only supplied input/cache/output rates. Bounds allow unknown input to be either cached or uncached; known cached input is still billed at your cached rate. Cache writes, tool fees, taxes, tiering, and other billing rules are outside the estimate. No tokenizer is included.