LetTokenBurn
Sign in
Auto71%Harbor: pendingSafety & AlignmentAgentic / Tool Use

AgentPrivArena

AgentPrivArena: Evaluating and Auditing Real-world AI Agent Privacy

A framework evaluating privacy risks in realistic LLM agent workflows using authentic MCP tools and self-hosted services with trajectory-level metrics.

PaperRepoHomepage

How to run

ltb · agentprivarena
$ npm i -g lettokenburn-cli
$ ltb bench info agentprivarena

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

5 sourced entries
HarnessSourceConfidenceLink
Mistral Large 316.2agentprivarenaIndependentmediumOct 5, 2026arxiv.org
GPT-5.417.3agentprivarenaIndependentmediumOct 5, 2026arxiv.org
Gemini-Flash-2.518.1agentprivarenaIndependentmediumOct 5, 2026arxiv.org
Kimi K2.618.1agentprivarenaIndependentmediumOct 5, 2026arxiv.org
DeepSeek V4 Pro19.3agentprivarenaIndependentmediumOct 5, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

AgentPrivArena integrates authentic Model Context Protocol (MCP) tools and self-hosted open-source services within a reproducible Docker-based sandbox to simulate real-world agent interactions. It transforms privacy scenarios into executable tasks where sensitive records exist as service state rather than prompt text.

The framework employs trajectory-level privacy metrics to quantify unnecessary information access during multi-step execution, complementing outcome-based evaluations. Experiments reveal substantial privacy risks overlooked by existing paradigms, highlighting the need for runtime auditing of intermediate agent behaviors.