# TurnScope 0.3 local evidence workflow

`evidence.py` records explicit event metadata, imports selected OTLP model calls, and keeps an expected study manifest before executions. It uses only Python's standard library and the existing `client.py` transport. It does not call model providers, run evaluators, estimate tokens, or collect telemetry automatically.

Download **both files into the same directory**:

```sh
curl -fsS https://turnscope.agentextual.com/api/v1/client.py -o client.py
curl -fsS https://turnscope.agentextual.com/api/v1/evidence.py -o evidence.py
python3 evidence.py --help
```

The legacy download name `turnscope.py` also works beside `evidence.py`. Repository examples below run from the repository root. Download commands contact the public host; local recording, normalization, measurement and study commands do not access the network. Only `accounting` and `compare-study` submit evidence, and each requires an explicit `--server`.

## Explicit local recording

Use the [accounting schema](accounting.schema.json) for the event document. Two local-only conveniences accept raw content: a region item may contain `content` instead of `fingerprint`, and a tool may contain `arguments` instead of `arguments_fingerprint`. The client hashes and removes those fields before writing or uploading. Task manifests likewise accept local `content` instead of `task_fingerprint`.

```python
import sys
sys.path.insert(0, "turnscope")
from evidence import Recorder

recorder = Recorder("provider-calls-and-tools-v1")
recorder.add_event({
    "id": "model-1", "kind": "model", "status": "completed",
    "usage_ownership": "self", "provider": "synthetic", "model": "example-model",
    "start_ms": 0, "end_ms": 120,
    "usage": {"input_tokens": 150, "output_tokens": 15, "cache_read_tokens": 0},
    "regions": {"actual_order": True, "complete": False,
                "items": [{"content": "Synthetic context; exact token count unavailable"}]},
})
recorder.add_event({
    "id": "read-1", "kind": "tool", "status": "completed", "usage_ownership": "none",
    "tool": {"name": "read", "operation": "read", "arguments": {"path": "example.txt"},
             "source_revision": "synthetic-revision-1", "permission_scope": "workspace"},
})
recorder.write("/tmp/events.json")
```

These numbers and identities are invented examples. Real instrumentation should supply reported usage and measured times. `Recorder.add_event` accepts model, tool, wait and container events, dependency/parent IDs, retry links, purpose, agent identity, explicit usage ownership, region ordering, and tool source/permission metadata. `root`, `coverage`, `rate_catalog` and `charges` can be supplied to the constructor. It does not infer that your events cover the complete run. Preserve failed attempts and helpers; give each attempt its own event ID.

To sanitize a local JSON event document or submit it:

```sh
python3 turnscope/evidence.py record local-events.json --output /tmp/events.json
python3 turnscope/evidence.py accounting /tmp/events.json \
  --server http://127.0.0.1:5152 --output /tmp/accounting-report.json
```

Region token counts are optional. Missing usage, cache writes, prices and charges stay unknown. A region fingerprint without a reported token count does not create one. Raw strings are hashed as exact UTF-8 bytes. Structured arguments use sorted, compact UTF-8 JSON, with nonfinite values rejected. A supplied hash must match supplied content. Hashes expose equality and may permit guessing low-entropy content; they are not encryption. Deliberate metadata labels, model/tool names, rate sources, revisions and permission-scope labels remain in uploads, so do not place secrets or original prompt text in those fields. Unsupported fields reject locally without echoing their contents.

## OTLP import

```sh
python3 turnscope/evidence.py normalize turnscope/examples/otel-genai.json \
  --format genai --scope-id selected-provider-export-v1 --output /tmp/imported-events.json
python3 turnscope/evidence.py accounting /tmp/imported-events.json \
  --server http://127.0.0.1:5152 --output /tmp/accounting-report.json
```

The [existing importer](CLIENT.md#supported-telemetry) supplies verified GenAI/OpenInference usage mappings, exact deduplication and trace/service/instrumentation selectors (`--trace-id`, `--service`, `--scope`). Per-call provider, model, cache-read and cache-write counts survive this adapter. Provider fields are `gen_ai.provider.name` for GenAI and `llm.provider` for OpenInference. Missing provider/model/cache fields remain unknown. Input/output usage remains required for selected OTLP calls; instrumentation without that usage must supply explicit events with unknown counts instead.

Numeric OTLP status `1` maps to completed, `2` to failed, and absent/`0` to unknown. Error messages, span names, prompts, completions, tool arguments and arbitrary attributes are excluded. Imported timing uses integer milliseconds relative to the earliest selected call; each offset is rounded down from exact nanoseconds, so submillisecond precision is lost. No root interval, tool inventory, prompt regions, dependency graph, prices or completeness claim is invented from selected spans. All coverage flags start false. Parent links are not scheduling edges.

For local access to exact timestamps and per-call evidence, use `client.normalize_bytes(data, "genai", include_call_metadata=True)`. It adds allowlisted `import.calls`, retaining exact nanosecond strings. The default existing `client.py normalize` output and old v1 commands remain compatible. The accounting adapter uses the per-call data and exports the accounting schema only.

## Expected studies and external outcomes

1. Prepare a [study manifest](study.schema.json) with task fingerprints, explicit trial IDs, two configuration identities/revisions, declared model sets/interventions, evaluator, environment and accounting-scope identities.
2. Initialize its private local file **before executing trials**.
3. Record every execution, including failed/cancelled/timed-out work, with explicit provenance and the external evaluator's outcome.
4. Compare the saved study against its full expected denominator.

```sh
python3 turnscope/evidence.py study-init manifest.json \
  --objective input_tokens --max-regression-pct 0 --output /tmp/study.json

# Run the declared trial, capture its evidence, and evaluate its outcome externally.
python3 turnscope/evidence.py accounting /tmp/events.json \
  --server http://127.0.0.1:5152 --output /tmp/accounting-report.json
python3 turnscope/evidence.py measure /tmp/accounting-report.json \
  --provenance provenance.json --status completed --outcome passed --output /tmp/record.json
python3 turnscope/evidence.py study-record /tmp/study.json /tmp/record.json

# Repeat for each declared configuration/task/trial, including unsuccessful executions.
python3 turnscope/evidence.py compare-study /tmp/study.json \
  --server http://127.0.0.1:5152 --check --output /tmp/study-report.json
```

`provenance.json` requires `manifest_id`, `config_id`, `config_revision`, `task_id`, `trial_id`, `task_fingerprint`, `evaluator_id`, `scope_id` and `environment_id`. Supply what the execution actually used; the client does not copy expected provenance into a record. `measure` requires a version-1 `accounting_report` with a valid `scope_id` matching the execution provenance, then adds its validated `measurement_metrics` and your explicit `--status` and `--outcome`. A helper-only report cannot be relabeled as full-run accounting. It does not infer task success from completion, token counts or HTTP success. Use outcome `unknown` if there was no valid external evaluation. Status choices are `completed`, `failed`, `cancelled`, `timeout`, and `evaluator_error`. Failed records retain their observed metrics. Outcomes must be explicitly `passed`, `failed`, or `unknown`, including when appending an already constructed record.

Each initialized file contains the complete manifest and its SHA-256 digest, plus records. Append and submit check the digest, catching accidental manifest edits. Initialization refuses an existing destination; duplicate configuration/task/trial records and records outside declared cells are rejected. Provenance mismatches within declared cells are preserved for the server to report. A changed task/environment/evaluator/revision is not silently repaired to match the manifest.

The digest is an accidental-edit check, not independent preregistration or tamper-proof attestation: a caller can rewrite both the document and hash or omit real-world executions from their original plan. The server remains stateless and trusts supplied evidence. The client requires owner-only regular files, rejects symlink/hard-link study and lock paths, writes atomically with mode `0600`, and uses a stable sibling `.lock` file for concurrent appends. Keep that lock file in place. Lock acquisition has a ten-second bound. Use a local filesystem with reliable locking. Atomic replacement protects readers from partial documents; it is not a power-loss durability guarantee.

`study-export STUDY --output FILE` checks the saved digest and writes the wire document for browser submission or inspection. `compare-study FILE --wire --server URL --output FILE` accepts a separately prepared wire document, including published examples; it does not claim that document was registered locally before executions. Both paths enforce the allowlisted study fields and bounds.

## Reproducible synthetic path

This example deliberately uses the same invented OTLP report for both configurations and supplies unknown outcomes. It demonstrates interoperability, not independent experimental evidence. Initialize the manifest first:

```sh
python3 - <<'PY'
import sys
sys.path.insert(0, "turnscope")
import evidence

manifest = {
    "manifest_id": "synthetic-cli", "evaluator_id": "none",
    "scope_id": "selected-provider-export-v1", "environment_id": "synthetic-env",
    "baseline": {"config_id": "baseline", "revision": "synthetic-r1", "models": ["synthetic-model"],
                 "intervention": "Synthetic baseline fixture"},
    "candidate": {"config_id": "candidate", "revision": "synthetic-r1", "models": ["synthetic-model"],
                  "intervention": "Synthetic reuse of the same fixture"},
    "tasks": [{"task_id": "fixture", "content": "synthetic OTLP fixture", "trial_ids": ["1"]}],
}
evidence.init_study("/tmp/synthetic-study.json", manifest, objective="input_tokens")
manifest = evidence.load_study("/tmp/synthetic-study.json")["manifest"]
for arm in ("baseline", "candidate"):
    provenance = {key: manifest[key] for key in ("manifest_id", "evaluator_id", "scope_id", "environment_id")}
    provenance.update(config_id=arm, config_revision="synthetic-r1", task_id="fixture", trial_id="1",
                      task_fingerprint=manifest["tasks"][0]["task_fingerprint"])
    evidence.client._write("/tmp/" + arm + "-provenance.json", provenance)
PY

python3 turnscope/evidence.py normalize turnscope/examples/otel-genai.json \
  --format genai --scope-id selected-provider-export-v1 --output /tmp/synthetic-events.json
python3 turnscope/evidence.py accounting /tmp/synthetic-events.json \
  --server http://127.0.0.1:5152 --output /tmp/synthetic-accounting.json
for arm in baseline candidate; do
  python3 turnscope/evidence.py measure /tmp/synthetic-accounting.json \
    --provenance /tmp/$arm-provenance.json --status completed --outcome unknown \
    --output /tmp/$arm-record.json
  python3 turnscope/evidence.py study-record /tmp/synthetic-study.json /tmp/$arm-record.json
done
python3 turnscope/evidence.py compare-study /tmp/synthetic-study.json \
  --server http://127.0.0.1:5152 --check --output /tmp/synthetic-study-report.json
```

The final command should exit **2** for insufficient evidence because outcomes and complete accounting coverage are unknown. The accounting report shows 280 observed input tokens and 32 observed output tokens under `usage.known_input_tokens` and `usage.known_output_tokens`. Its complete-run `measurement_metrics` remain null, including usage, cost and root duration; observed partial totals are not silently promoted into complete-run measurements. Use a fresh study path when repeating initialization.

## Bounds and CI

Accounting accepts at most 5,000 events, 1,000 model calls, 1,000 charges/cards, 20,000 total graph edges/regions, 128 regions per event, 64 dependencies per event, and 32 cache-write classes per event/card. Study manifests allow at most 1,000 expected task/trial pairs per configuration and 2,000 records. Every sanitized API document is limited to 2 MiB; local raw input is limited to 16 MiB. Required metadata and numeric limits follow the schemas. Graph cycles, inconsistent usage ownership, complete cost and timing claims are checked by the accounting endpoint.

All JSON output files are atomically written with owner-only permissions. Use `--output -` for stdout, except study initialization. Successful file commands are silent; errors are bounded JSON on stderr and exit 2. No raw input, response body, local file path or URL credentials are echoed in errors.

Transport uses the existing client's HTTPS-or-loopback-HTTP restriction, redirect refusal, disabled ambient proxies, bounded response reads/deadlines and 429-only retries: at most three attempts and ten seconds of requested retry delay. Responses are limited to 16 MiB. No credential is required.

`compare-study --check` writes the report before exiting: **0** for `within_budget`, **1** for `regression`, **2** for `insufficient_evidence` or operational errors. Without `--check`, a valid response exits 0 regardless of its verdict. Study coverage and caller-supplied quality claims remain visible; the verdict is a policy result for submitted executions, not a population quality claim.

Run the local suites with:

```sh
python3 -m unittest discover -s turnscope -p 'test_*.py'
```