LetTokenBurn
Sign in
Auto71%Harbor: pendingAgentic / Tool UseMultimodal

Kolab

Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

A dataset of 115+ hours of screen recordings from 50 professionals collaborating with AI agents on long-horizon tasks, measuring anticipation of human actions and collaboration quality judgments.

PaperHomepage

How to run

ltb · kolab
$ npm i -g lettokenburn-cli
$ ltb bench info kolab

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

6 sourced entries
HarnessSourceConfidenceLink
GPT-6 Astra +0.7N/AIndependentmediumOct 15, 2026arxiv.org
Claude Opus 5.50.7N/AIndependentmediumOct 15, 2026arxiv.org
Gemini 3.1 Pro0.6N/AIndependentmediumOct 15, 2026arxiv.org
Gemini 3.8 Flash0.6N/AIndependentmediumOct 15, 2026arxiv.org
Qwen3-VL-30B-A3B0.5N/AIndependentmediumOct 15, 2026arxiv.org
Qwen3-VL-32B0.3N/AIndependentmediumOct 15, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Kolab comprises 115+ hours of screen recordings from 50 professionals across 40 occupations, capturing 508 steps involving 90+ software applications. It includes annotations of human-agent interactions, adoption behaviors, and expert ratings of collaboration satisfaction and output quality.

The benchmark evaluates models on forecasting how professionals use agent contributions (adoption/modification/rejection) and predicting collaboration satisfaction scores. Frontier models struggle with these tasks, showing low F1 scores and weak correlations with human ratings, indicating unsaturated performance.