# Track B Algonauts Benchmark Protocol

This document defines the reviewer-facing protocol for the Kahlus/NeuroTwin Track B Algonauts 2025 benchmark. The goal is not to claim state of the art by default. The goal is to make the Pair-Operator/NFC architecture falsifiable against strong leakage-safe baselines before spending a full six-A100 run.

## Dataset And Claim Scope

Primary dataset: Algonauts 2025 / CNeuroMod movie stimulus to fMRI.

Required data contract:

- Four subjects: `sub-01`, `sub-02`, `sub-03`, `sub-05`.
- fMRI modality only for Track B first pass.
- Each prepared response array is shaped `[time,1000]` over Schaefer 1000 parcels.
- Stimulus features are aligned one row per fMRI TR, with `stimulus_embedding.shape[0] == signal.shape[0]`.
- Raw or precomputed feature artifacts have a local source path and SHA256 hash.
- `stimulus_to_fmri_response` must be present as a real prepared task, not skipped.

Claim eligibility:

- Real stimulus claims require `require_real_stimulus=true`.
- Feature sources must be real precomputed stimulus artifacts. Hash-only transcript features, synthetic embeddings, placeholders, and self-attested embeddings are plumbing-only.
- The prepared-task evidence gate must verify that `stimulus_feature_hash` matches the referenced source artifact.
- No raw neural data, movie files, checkpoints, credentials, or downloaded datasets are committed to git.

## Splits

All splits are at whole movie, episode, or run level. Random TR/window splits are prohibited.

The local `official` adapter policy is official-compatible:

- If official/test/OOD-labeled response partitions are present, they are used.
- If withheld official responses are unavailable locally, the adapter uses a whole-run local-dev partition:
  - Friends seasons 1-5: train.
  - Friends season 6: validation.
  - Movie10, explicit test, or OOD-labeled runs: test.

The split manifest must preserve the exact record IDs, source hashes, split assignments, and stimulus IDs. Any boundary-buffer or HRF-delay exclusions used by a later feature pipeline must be recorded in the feature/preprocessing manifests.

## Debug Gate

The one-GPU debug gate must pass before any six-GPU sweep.

Required checks:

- Algonauts raw/prepared data are present under an approved cluster root.
- The four expected subjects are present.
- fMRI arrays are finite and 1000-parcel compatible.
- Stimulus features are nonzero, finite, source-path-backed, and SHA256 verified.
- `event_manifest.json`, `split_manifest.json`, `data_manifest.json`, `feature_manifest.json`, `stimulus_manifest.json`, `leakage_report.json`, and `eval_audit.json` exist.
- `eval_audit.json.passed == true`.
- The prepared baseline suite emits finite metrics for `stimulus_to_fmri_response`.
- `paper_mode_gate.json` passes for seeds `0,1,2` on the debug baseline run.
- A short Kahlus debug model run exits cleanly and writes `summary.json`, `metrics.csv`, and final evidence artifacts.

Failure rule: if any required debug check fails, stop. Do not launch the six-GPU sweep.

## Baselines

Minimum local baselines on the exact same prepared tensors:

- `linear_ridge`: primary ridge anchor.
- `autoregressive_ridge`: history-aware ridge anchor when the task supports it.
- `persistence`, `train_mean`, and `random_permutation`: null/floor baselines.
- `neurotwin` or current direct NeuroTwin model.
- `pair_operator_no_pair_state`: direct ablation without pair state.
- `pair_operator_full`: full Pair-Operator/NFC candidate.

External systems such as TRIBE/VIBE/BrainVista are contextual references unless their exact code and feature contract are run on the same split. BrainVista-style local code is a labeled reimplementation/approximation, not exact BrainVista.

## Six-GPU Ablation Sweep

Run three seed waves: `0`, `1`, `2`.

Each wave uses six A100s, one arm per GPU:

- `ridge_anchor`
- `current_neurotwin`
- `pair_operator_no_pair_state`
- `pair_operator_low_rank_pair_state`
- `pair_operator_pair_state_uncertainty`
- `pair_operator_full`

All arms must share:

- identical split manifest hash;
- identical feature manifest hash;
- identical preprocessing policy;
- train-only scaling/PCA/normalization;
- identical metric and statistical-test code.

Changing feature extractors, split files, normalization scope, or held-out records between arms invalidates the ablation.

## Metrics

Primary metric for Algonauts Track B:

- Mean Pearson `r` for `stimulus_to_fmri_response`.

Also report:

- MSE, MAE, R2, Spearman;
- per-seed metrics;
- per-subject metrics when available;
- per-parcel/per-network summaries when available;
- ridge gap: `pair_operator_full - ridge_anchor`;
- architecture deltas:
  - `pair_operator_full - pair_operator_no_pair_state`;
  - `pair_operator_full - current_neurotwin`;
- uncertainty-error correlation for uncertainty arms.

Do not substitute forecasting or pattern-correlation metrics for the Algonauts stimulus-to-fMRI score. Those can be secondary dynamics diagnostics only.

## Strict Pass Gate

The long six-GPU Pair-Operator run is allowed only if the sweep passes all strict checks:

- `pair_operator_full` beats `pair_operator_no_pair_state` by at least `0.01` mean Pearson.
- `pair_operator_full` beats `current_neurotwin` by at least `0.01` mean Pearson.
- Positive direction holds in at least `2/3` seeds.
- The full model is not badly below ridge: `full >= ridge_anchor - 0.01`, or it beats ridge.
- Uncertainty-error correlation is positive and finite.
- No required task is quarantined.
- Evidence/model-card artifacts are clean.

If per-subject artifacts exist, the positive direction should hold in at least `3/4` subjects. If those artifacts do not exist, the reviewer gate should be considered incomplete and the paper should not make a subject-consistency claim.

## Long Run

Only after the strict gate passes:

- model: `pair_operator_full`;
- GPUs: exactly six idle A100s;
- initial steps: `50000`;
- config: `configs/train/algonauts_pair_operator_full.yaml`;
- run root: `/raid/scratch/$USER/kahlus-algonauts-trackb-v1/long` or the stage root selected in the launch script.

Codex/agents must detach after safe launch. Safe launch means:

- tmux or scheduler job is alive;
- correct GPU count and IDs are visible in the container/job;
- persistent root, prepared root, and config paths are visible;
- config materializes with absolute manifest paths;
- audit/preflight passes;
- the training command starts;
- first metrics/log row appears when applicable;
- no immediate traceback, NaN quarantine, DDP mismatch, NCCL timeout, OOM, or `ChildFailedError`.

After safe launch, create a thread heartbeat near the estimated finish time. Do not keep an agent attached while training.

## Artifact Checklist

Every claim-eligible stage must preserve:

- `data_manifest.json`;
- `split_manifest.json`;
- `event_manifest.json`;
- `feature_manifest.json`;
- `stimulus_manifest.json`;
- `leakage_report.json`;
- `eval_audit.json`;
- `baseline_ranking.csv` or `prepared_baseline_suite.json`;
- `paper_mode_gate.json` when paper mode is run;
- `strict_gate.json` for sweep decisions;
- `summary.json`, `metrics.json`, `metrics.csv`, and `metrics*.jsonl` for training runs;
- `diagnostic_report.md`, `RUN_REPORT.md`, `EEG_MODEL_CARD.md` or successor model card when available;
- container image, git commit, config YAML, and command logs.

Evidence zips must exclude raw data, checkpoints, credentials, `.env*`, tokens, SSH keys, nested archives, and tarballs.

## Interpretation Rules

- If the full model beats internal ablations but loses to ridge, the result supports internal architecture progress only. It does not support a better encoding-model claim.
- If ridge beats all Kahlus variants by a large margin, stop and write a negative/diagnostic result.
- If the debug gate fails feature verification, the run is not paper evidence.
- If the long run launches without the strict gate, it is an engineering run, not a reviewer-safe scientific run.