FASHION-RQBENCH
FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval
A scalable fashion retrieval benchmark with 5,380 queries and 44,446 products, evaluating ranking, relevance, sibling discrimination, and paraphrase robustness using hierarchical semantic structures.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info fashion-rqbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
9 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3-Embedding-4B | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| E5-Mistral-7B-Instruct | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Qwen3-Embedding-8B | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| SimCSE-RoBERTa-large | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Contriever-MSMARCO | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| SPLADE++ | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| ColBERTv2 | 0.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Sentence-T5-large | 0.8 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| RoBERTa-large | 0.7 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
FASHION-RQBENCH comprises 5,380 attribute-enriched queries and 44,446 products derived from catalog structure via RQ-VAE. It supports four tasks: ranking, hierarchical relevance classification, sibling discrimination, and paraphrase robustness.
Scoring uses weighted query-relevance and per-attribute metrics. Experiments show dense encoders outperform BM25 in ranking (nDCG@10 0.87 vs 0.77) but struggle with fine-grained attributes like material and style.