From Simulation to Decision Support: A Benchmark and Decision Assistant Framework for Multi-Agent Social System
SocialSim-Bench 研究策略互动及基于模拟的决策支持。我参与谈判场景与 benchmark 任务设计,将私有目标和经济约束转化为评价 agent 行为的任务情境。
暨南大学经济与社会研究院 · 经济学硕士在读
导师: 薄诗雨
我关注 AI 如何支持可信的研究,以及科学知识如何被生产、传播和认可。我的工作连接社会科学中的测量问题、科学学,以及支持知识工作的智能体系统。
我对测量的兴趣,始于用语言模型把非结构化文本转化为研究变量。这些实践让我进一步追问:当模型给出一个分数或标签时,我们凭什么把它解释为想要测量的东西,又凭什么在研究中使用它?
我们利用学生信件研究 AI 人格测量,并结合多类人类评分者和后续结果进行验证。自评本身也不完美,因此,与单一评分者一致只是证据的一部分。我设计了 AI 评分管线,并参与检验这些指标测量了什么、提供了哪些额外信息。
We assess the quality of personality measures generated by a large language model from student-written letters. Using data from a pen-pal program in rural China, we compare the resulting AI ratings with student, teacher, and guardian reports for the same children. The AI ratings show high internal consistency and evidence of construct validity. They are modestly related with but distinct from human reports and predict later academic performance and psychological health. Shapley-Owen decompositions show that the AI reports retain an explanatory contribution to both test scores and mental health alongside student, teacher, and guardian reports, offering a low-cost useful personality measurement source in addition to conventional subjective questionnaire reports.
此前,我与 薄诗雨 合作,从采购文本构造医院采购行为指标;也与 王子健 、 邬介然 、 Cynthia Jing Wu 合作,构造企业货币政策关注度与融资约束的文本指标。
我希望将这一测量视角延伸到 LLM 评估,包括对 agent 行为与研究成果的判断。
我关心什么因素影响科学进展,以及一项成果的贡献与它获得的关注之间是什么关系。政治冲击、合作网络和新技术,为研究这些问题提供了不同的切入点。
我将 arXiv 论文与文献计量数据连接,研究地缘政治紧张如何影响科学产出和知识传播。初步结果显示,引用下降并未伴随发表量或所测文本新颖性的相应下降,提示我们需要区分知识的生产与后续传播。
接下来,我希望研究 AI 如何改变研究过程与科学产出,以及网络关系如何影响有价值成果的认可。这些问题也需要回到测量:我们怎样衡量科学贡献与影响?
在实证研究之外,我也建设文献工具、探索研究智能体。这些工作让我从实际使用中思考:知识怎样被获取、保留和使用,AI 提供的帮助又应该怎样评价。
我围绕实际阅读流程独立建设 Frowang:先通过结构化摘要筛选论文,再查看原文证据,并让助手参与精读。Paper API、CLI 和可安装的 skills 将本地 agent 接入同一文献库,用于检索、整理和跨论文工作。
453 名注册用户 · 近 100 名月活用户 · 截至 2026 年 10 月 24 日
我参与 Agent Memory 系统开发,并研究用于文献发现与综合的 Agent Swarm。Swarm 工作关注:当文献规模超出单个 agent 的上下文时,怎样通过并行检索与阅读梳理研究方向的发展,同时兼顾覆盖与相关性。
SocialSim-Bench 研究策略互动及基于模拟的决策支持。我参与谈判场景与 benchmark 任务设计,将私有目标和经济约束转化为评价 agent 行为的任务情境。
Abstract · 英文原文
Most prior LLM-based social-simulation studies ask whether agents appear realistic, socially intelligent, or behaviorally consistent. However, can LLM agents genuinely reason about evolving social dynamics and develop adaptive strategies, rather than merely role-play? Moreover, social simulations are often intended to provide guidance for real-world decision-making. We study the next step: how simulated possibilities can inform disciplined decision support. We present SocialSim-Bench, a benchmark and decision-assistant report-generation framework for multi-agent decision support, comprising two scenarios: social-media interaction and long-term negotiation. Each scenario combines contextual backgrounds, role-specific profiles, private goals, and action constraints. We evaluate agents using rule-based metrics and an LLM-as-a-judge system, while also analyzing the strategies they employ, finding that even the excellent LLMs only perform simple strategies without applying complex methods. We further investigate multiple approaches for transforming simulation results into decision-assistant reports, including direct LLM generation, harness-based generation with agents such as Codex, and our proposed recursive synthesis approach, which transforms simulation traces into decision-assistant reports by summarizing individual trajectories, aggregating recurring conditions and plausible outcomes, and synthesizing actionable recommendations. We evaluate report quality across multiple dimensions. Our results show that simulation-grounded reporting improves decision-support quality in both social-media and negotiation settings compared with no simulation.