RepShiftBench
A benchmark comprising 1,218 encoder-dataset tasks across text, image, and audio to evaluate few-shot classification generalization across heterogeneous representation spaces.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info repshiftbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| PE-core-L | 94.8 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| SigLIP-2 so400m/384 | 94.7 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| DINOv3 ViT-L/16 | 92.3 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| Embedding Gemma 300M | 84.3 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| stella-1.5B-v5 | 84.0 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| Whisper Large | 64.8 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| HuBERT-large | 57.7 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
| AST audioset-base | 56.5 | repshiftbench | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
RepShiftBench consists of 1,218 tasks formed by pairing 59 datasets with 62 pretrained encoders. It evaluates generalization to unseen datasets, encoders, joint combinations, and modalities.
Scoring compares in-context learners against per-episode Logistic Regression. The benchmark reveals that standard in-context methods underperform simple baselines, highlighting the challenge of reusable prediction across varying representation geometries.