Arabic Cultural and Sociolinguistic Knowledge Benchmark
Harbor: pendingThe cost of human expert evaluation is a principal bottleneck to deploying language models in specialized, high-stakes domains.
- Samples
- N/A
- Released
- 2026
- Scores
- 0
Cross-Care
Cross-Care evaluates biases in language models, highlighting significant discrepancies between model logits, pretraining data and real-world disease prevalence, emphasizing the need for integrating real-world data to ensure fairness.
How to run
ltb · cross-care
$ npm i -g lettokenburn-cli$ ltb bench info cross-care
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
0 sourced entriesNo sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Large language models (LLMs) are increasingly essential in processing natural languages, yet their application is frequently compromised by biases and inaccuracies originating in their training data. In this study, we introduce \textbf{Cross-Care}, the first benchmark framework dedicated to assessing biases and real world knowledge in LLMs, specifically focusing on the representation of disease prevalence across diverse demographic groups. We systematically evaluate how demographic biases embedded in pre-training corpora like $ThePile$ influence the outputs of LLMs. We expose and quantify discrepancies by juxtaposing these biases against actual disease prevalences in various U.S. demographic groups. Our results highlight substantial misalignment between LLM representation of disease prevalence and real disease prevalence rates across demographic subgroups, indicating a pronounced risk of bias propagation and a lack of real-world grounding for medical applications of LLMs. Furthermore, we observe that various alignment methods minimally resolve inconsistencies in the models' representation of disease prevalence across different languages. For further exploration and analysis, we make all data and a data visualization tool available at: \url{www.crosscare.net}.