AgentSocialBench
Evaluating Privacy Risks in Human-Centered Agentic Social Networks
Evaluating Privacy Risks in Human-Centered Agentic Social Networks
AgentSocialBench is the first benchmark for evaluating privacy preservation in human-centered agentic social networks — settings where teams of AI agents serve individual users, coordinate across domain boundaries, and must protect sensitive personal information throughout.
Across different social topologies, agents must coordinate while protecting each user's private information. Browse scenarios from each benchmark category.
Model performance under L0 (unconstrained) privacy mode. Lower leakage rates are better; higher utility scores are better.
| Model | CDLR | MLR | CULR | MPLR | HALR | CSLR | ACS | IAS | TCQ | Task% |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V3.2 | 0.51 | 0.21 | 0.19 | 0.22 | 0.14 | 0.10 | 1.00 | 0.76 | 0.77 | 83.6 |
| Qwen3-235B | 0.49 | 0.26 | 0.14 | 0.22 | 0.06 | 0.08 | 1.00 | 0.75 | 0.73 | 74.9 |
| Kimi K2.5 | 0.67 | 0.30 | 0.29 | 0.30 | 0.12 | 0.09 | 1.00 | 0.69 | 0.86 | 93.3 |
| MiniMax M2.1 | 0.62 | 0.25 | 0.17 | 0.20 | 0.20 | 0.10 | 0.99 | 0.75 | 0.77 | 80.8 |
| GPT-5 Mini | 0.40 | 0.23 | 0.16 | 0.18 | 0.11 | 0.09 | 1.00 | 0.75 | 0.69 | 68.2 |
| Claude Haiku 4.5 | 0.57 | 0.27 | 0.19 | 0.24 | 0.15 | 0.09 | 0.99 | 0.75 | 0.73 | 69.6 |
| Claude Sonnet 4.5 | 0.52 | 0.24 | 0.19 | 0.16 | 0.10 | 0.08 | 1.00 | 0.79 | 0.83 | 87.4 |
| Claude Sonnet 4.6 | 0.50 | 0.21 | 0.19 | 0.18 | 0.10 | 0.08 | 1.00 | 0.85 | 0.87 | 94.1 |
352 scenarios across 7 categories, spanning dyadic and multi-party interactions with increasing structural complexity.
Intra-team coordination across domain boundaries. A user's health agent must collaborate with their finance agent without leaking medical data.
Agent brokers human-to-human interaction. An agent mediates between two people, tempted to reveal one's private constraints to the other.
Agents from different users interact via A2A protocol. Each agent carries its user's data and must not expose it during coordination.
3-6 users' agents in shared group chat coordinate on a common goal with mixed visibility channels.
A coordinator agent aggregates information from multiple participants, creating a central point of leakage risk.
Agents compete for a shared resource under social pressure, creating incentives to extract or leak private information.
Asymmetric affinity tiers modulate how much information should flow between different relationship levels.
Three surprising insights from evaluating 8 frontier LLMs across all categories and privacy instruction levels.
CDLR is 2-3x higher than MC/CU leakage across all models. When agents share a user's full context, the temptation to reference cross-domain information during coordination is overwhelming.
Social structure shapes privacy as much as explicit instructions. Hub-and-spoke topologies concentrate leakage risk at coordinators, while competitive settings create extraction incentives absent in cooperative scenarios.
Teaching agents how to abstract sensitive information paradoxically causes them to discuss those topics more. IAS improves but overall leakage increases — the defense becomes a spotlight on what should stay hidden.
Feature comparison with existing privacy benchmarks for multi-agent systems.
| Feature | AgentSocialBench | MAGPIE | MAMA | AgentLeak | ConfAIde | PrivLM-Bench |
|---|---|---|---|---|---|---|
| Multi-Agent | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| Cross-Domain | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Agent Mediation | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Cross-User | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| Multi-Party | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Social Graph | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
If you find AgentSocialBench useful in your research, please cite our paper.
@misc{wang2026agentsocialbenchevaluatingprivacyrisks,
title={AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks},
author={Prince Zizhuang Wang and Shuli Jiang},
year={2026},
eprint={2604.01487},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2604.01487},
}