H100 Runbook

H100 is a compatible high-memory variant. Use docs/A100_RUNBOOK.md as the canonical v1 cluster path unless a specific H100 queue is available. Prepare data before training. H100 jobs must not require internet access. Prepared artifacts should live under persistent $NEUROTWIN_DATA/prepared/; do not rely on node-local /tmp for cluster launches.

Local dry run:

PYTHONPATH=src python3 -m neurotwin.cli doctor
PYTHONPATH=src python3 -m neurotwin.cli data prepare --dataset synthetic --split subject --out-dir /tmp/neurotwin_prepared
PYTHONPATH=src python3 -m neurotwin.cli data smoke --dataset moabb --split subject --out-dir /tmp/neurotwin_moabb_smoke
PYTHONPATH=src python3 -m neurotwin.cli eval audit --suite neural_translation_v1 \
  --event-manifest /tmp/neurotwin_prepared/event_manifest.json \
  --split-manifest /tmp/neurotwin_prepared/split_manifest.json
PYTHONPATH=src python3 -m neurotwin.cli eval --suite neural_translation_v1 \
  --event-manifest /tmp/neurotwin_prepared/event_manifest.json \
  --split-manifest /tmp/neurotwin_prepared/split_manifest.json \
  --train-steps 1
PYTHONPATH=src python3 -m neurotwin.cli train --config configs/train/prepared_synthetic_debug.yaml --run-root /tmp/neurotwin_runs
PYTHONPATH=src python3 -m neurotwin.cli train --config configs/train/prepared_synthetic_debug.yaml \
  --run-root /tmp/neurotwin_runs \
  --resume /tmp/neurotwin_runs/prepared_synthetic_debug/checkpoint.pt
PYTHONPATH=src python3 -m neurotwin.cli train --dry-run --config configs/train/neurotwin_v1_h100.yaml
PYTHONPATH=src torchrun --standalone --nproc_per_node=2 \
  -m neurotwin.cli train --config configs/train/prepared_synthetic_debug.yaml --run-root /tmp/neurotwin_ddp_runs

Locked MOABB smoke, when optional MOABB/MNE dependencies and data access are available:

scripts/prepare_moabb_smoke.sh /tmp/neurotwin_moabb_smoke
PYTHONPATH=src python3 -m neurotwin.cli train --config configs/train/moabb_smoke_locked.yaml --run-root /tmp/neurotwin_moabb_runs
PYTHONPATH=src python3 -m neurotwin.cli report --run-dir /tmp/neurotwin_moabb_runs/moabb_smoke_locked

Before any H100/A100 launch, run the locked MOABB benchmark preparation and confirm eval_audit_passed=True, window_count > 0, and nonzero train/val/test window_counts_by_split. The MOABB benchmark and MOABB cluster configs use window_length=128 and stride=128 for BNCI2014_001 compatibility.

SLURM:

sbatch scripts/slurm/train_h100.sh configs/train/neurotwin_v1_h100.yaml
sbatch scripts/slurm/eval_h100.sh neural_translation_v1 runs/<run_id>
sbatch scripts/slurm/sweep_h100.sh

Every real run should write a config copy, split manifest, metrics, checkpoints, environment info, git commit, and result summary. Prepared training also writes metrics.csv, objective-level metrics.jsonl, checkpoint.pt, checkpoint_best.pt, paper-ready tables/*.csv, figures/*.json through nt report --run-dir, honors gradient_accumulation_steps, and enables bf16 autocast when precision: bf16 is configured. When launched through torchrun, prepared training initializes torch.distributed, wraps the model in DDP, writes rank-specific JSONL metrics, and leaves shared checkpoints/reports to rank zero.

For real cluster runs, edit configs/train/neurotwin_v1_h100.yaml so data.event_manifest and data.split_manifest point to prepared local files. Training jobs must not download MOABB/OpenNeuro/other data.