A100 Runbook

A100 is the canonical cluster target for NeuroTwin v1. H100 configs remain compatible high-memory variants, but public docs and acceptance gates should point here first.

Prepare data before training. Cluster jobs must read local prepared manifests and must not download MOABB, OpenNeuro, or other public data during training. Set NEUROTWIN_DATA to a persistent shared filesystem location; prepared benchmark artifacts belong under $NEUROTWIN_DATA/prepared/, not node-local /tmp.

For the NFC pivot, run the local synthetic field-compiler suite before any A100 debug job. Pair-Operator is an ablation/baseline, not the main architecture.

For the first Chapman run, prefer the guarded one-command path:

bash scripts/run_full.sh /path/to/shared/persistent/neurotwin

That launcher prepares MOABB, verifies window_count=18144, materializes absolute manifest paths under outputs/configs/, dry-runs, and submits exactly one A100 smoke job. scripts/cluster/chapman_a100_first_run.sh is a compatibility wrapper around the same path.

Fast Iteration Lane

Use one A100 for short validation runs before spending a multi-day allocation:

PYTHONPATH=src python3 -m unittest
git diff --check
PYTHONPATH=src python3 -m neurotwin.cli eval --suite nfc_synthetic --out-dir outputs/nfc_synthetic --train-steps 1 --seed 0
bash scripts/run_smoke.sh outputs/smoke-head
bash scripts/run_full.sh /path/to/shared/persistent/neurotwin

Then run the 3-seed MOABB paper-mode gate on prepared manifests:

export NEUROTWIN_DATA=/path/to/shared/persistent/neurotwin
export PREPARED_DIR="$NEUROTWIN_DATA/prepared/moabb_benchmark"
export EVAL_DIR="$NEUROTWIN_DATA/eval/moabb_3seed_head"
python3 -m neurotwin.cli eval \
  --suite neural_translation_v1 \
  --paper-mode \
  --seeds 0 1 2 \
  --event-manifest "$PREPARED_DIR/event_manifest.json" \
  --split-manifest "$PREPARED_DIR/split_manifest.json" \
  --window-length 128 \
  --stride 128 \
  --train-steps 3 \
  --out-dir "$EVAL_DIR"

MOABB EEG is expected to skip tribe_style; this gate validates leakage audits, baseline reporting, seed aggregation, and the paper artifact contract. It does not set scientific_claim_allowed=true; that remains an explicit run-summary decision.

Heavy 7-GPU Lane

Start one 7x A100 80GB run only after local tests, the 1-GPU smoke, and the 3-seed MOABB gate pass for the exact committed artifact. Krish’s partner cluster may expose 7x A100 80GB GPUs, but this handoff uses exactly 7 and leaves the unused. Request 12:00:00 wall time for the deep lane and use configs/train/moabb_a100.yaml with steps: 50000 (50,000 configured steps). Short diagnostic runs ending in a few hours are normal for smoke/synthetic/debug configs and are not the deep 12-hour lane. For the heavy lane, the guarded Docker and Slurm helpers consume existing Phase 1 paper-mode artifacts from A100_PAPER_MODE_EVAL_DIR when paper_mode_gate.json passed. If Phase 1 artifacts are missing, they write a paper_mode_artifacts_unavailable marker and do not silently run paper-mode inside the seven-GPU allocation. Only set A100_RUN_PAPER_MODE_IN_FULL=1 to run the 3-seed paper-mode gate inside the full allocation. After training both lanes call python -m neurotwin.cli run finalize, which copies the small paper-mode artifacts into the run directory, writes the run report, runs leakage-demo and identity-probe diagnostics, finalizes evidence_gate.json, and generates EEG_MODEL_CARD.md.

export NEUROTWIN_DATA=/path/to/shared/persistent/neurotwin
export RUN_ROOT="$NEUROTWIN_DATA/runs"
export A100_CONFIG_TEMPLATE=configs/train/moabb_a100.yaml
export A100_RUN_ID=moabb_a100
PYTHONPATH=src python3 -m neurotwin.cli cluster materialize-config \
  --template "$A100_CONFIG_TEMPLATE" \
  --prepared-root "$NEUROTWIN_DATA/prepared/moabb_benchmark" \
  --out outputs/configs/moabb_a100.materialized.yaml
RUN_ROOT="$RUN_ROOT" \
sbatch --ntasks-per-node=7 --gres=gpu:a100:7 --time=12:00:00 \
  scripts/slurm/train_a100.sh outputs/configs/moabb_a100.materialized.yaml

Use short 1-GPU jobs for debugging. Do not retry failed multi-GPU runs blindly; inspect logs, metrics, checkpoints, and manifests first.

Local readiness checks:

PYTHONPATH=src python3 -m neurotwin.cli doctor
PYTHONPATH=src python3 -m neurotwin.cli train --dry-run --config configs/train/neurotwin_v1_a100.yaml
PYTHONPATH=src python3 -m neurotwin.cli estimate --config configs/train/neurotwin_v1_a100.yaml
bash -n scripts/slurm/*.sh

Prepare the first real-data benchmark outside the repo:

export NEUROTWIN_DATA=/path/to/persistent/neurotwin
scripts/prepare_moabb_benchmark.sh

Do not submit A100 jobs unless the benchmark preparation prints eval_audit_passed=True, window_count > 0, and nonzero train/val/test entries in window_counts_by_split. For BNCI2014_001, the locked MOABB benchmark defaults to window_length=128 and stride=128; larger windows can produce zero runnable windows.

Submit training:

export RUN_ROOT=/path/to/shared/persistent/neurotwin/runs
PYTHONPATH=src python3 -m neurotwin.cli cluster preflight \
  --config outputs/configs/moabb_a100.materialized.yaml \
  --run-root "$RUN_ROOT" \
  --require-cuda \
  --require-prepared-windows \
  --expect-window-count 18144 \
  --expect-split-windows train:12096,val:2016,test:4032
sbatch scripts/slurm/train_a100.sh outputs/configs/moabb_a100.materialized.yaml
sbatch scripts/slurm/eval_a100.sh "$RUN_ROOT/<run_id>"
sbatch scripts/slurm/sweep_a100.sh outputs/configs/moabb_a100_seed*.yaml

Every real run should write config, split manifest, metrics, best checkpoint, environment, git commit, split hash, report tables, figure specs, LEAKAGE_AUDIT.json, CLAIM_GATE.json, baseline rankings, seed aggregates, leakage-demo output, identity-probe output, and EEG_MODEL_CARD.md. A passed paper-mode gate means the artifact contract is satisfied; scientific/model claim allowance is controlled by summary.json and stays false unless that summary explicitly allows it. The allowed evidence statement is leakage-proof evaluation and infrastructure validation, not model superiority.