ARC-AGI-2
Harbor nativeARC-AGI-2: 更难的抽象推理
Verified100%Abstract Reasoning
- Samples
- 167
- Released
- 2025
- Scores
- 120
MobileInject
MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency Injection
A benchmark with 108 executable risk instances across 11 scenarios and 9 security categories to evaluate MLLM-based mobile agents against multimodal injection attacks.
How to run
ltb · mobileinject
$ npm i -g lettokenburn-cli$ ltb bench info mobileinject
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
9 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GLM-4.5V | 0.0 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| GLM-4.6V | 1.2 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| Gemini 2.5 Flash | 4.4 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| Qwen2.5-VL-72B-Instruct | 9.4 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| GPT-4.1 | 11.1 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| Qwen 3.5 122B | 16.7 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| GPT-5.4-nano | 20.4 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| Qwen3.6-35B | 38.9 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
| Gemini 3 Flash Preview | 50.0 | N/A | Independent | medium | Oct 16, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
MobileInject provides a high-fidelity Android emulator environment covering 11 application scenarios and 108 executable risk instances. It employs Cross-Modal Consistency Injection and enhanced attack methods like Multi-Role Collaborative Deception.
The benchmark uses a Reasoning Security Diagnosis Framework for fine-grained failure analysis. Experiments reveal that mobile agents are broadly vulnerable to multimodal injection, with many failures stemming from early-stage reasoning hijacking.