Sovereign 8s.is mirror of standardgalactic/AgentBenchMedicine_source
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
NCCYUNSONG 024edd9d00
Web search tool with content fetching
Implement web search functionality with multiple engines and content fetching.
2025-12-30 23:49:12 +08:00
.idea add 2025-07-10 09:31:16 +02:00
agentclinic Add files via upload 2025-07-10 09:27:04 +02:00
app Add files via upload 2025-07-10 09:27:04 +02:00
config Add files via upload 2025-07-10 09:27:04 +02:00
hle add 2025-07-10 09:31:16 +02:00
medagents-benchmark Web search tool with content fetching 2025-12-30 23:49:12 +08:00
agentclinic_vllmf.py add 2025-07-10 09:31:16 +02:00
agentclinic_vllmr.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_customized1.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_customized2.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_localvraw.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_original.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_vf.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_vf_original.py add 2025-07-10 09:31:16 +02:00
agentclinicmanus_vr.py add 2025-07-10 09:31:16 +02:00
config.ini add 2025-07-10 09:31:16 +02:00
hle.py add 2025-07-10 09:31:16 +02:00
hle_img.py add 2025-07-10 09:31:16 +02:00
hle_llm.py add 2025-07-10 09:31:16 +02:00
hle_llm_img.py add 2025-07-10 09:31:16 +02:00
LICENSE add 2025-07-10 09:31:16 +02:00
README.md Update README.md 2025-11-26 15:27:03 +08:00
readme.md Update readme.md 2025-11-26 15:13:53 +08:00

AgentBenchMedicine

License: MIT

Overview

AgentBenchMedicine provides three open-source benchmarks for evaluating multimodal and text-only large-language-model (LLM) agents in clinical reasoning tasks:

  1. AgentClinic — simulates a multi-agent clinical diagnosis workflow via dialogues among four roles: doctor, patient, measurement, and moderator.
  2. MedAgentsBench — a "hard" subset of complex multiple-choice medical questions requiring multi-step reasoning.
  3. Humanity’s Last Exam (HLE) — a benchmark of highly complex and nuanced Biology/Medicine cases, presented as multiple-choice and short-answer questions.

Table of Contents


Benchmarks and Datasets

AgentClinic

Simulates the clinical diagnosis process as Objective Structured Clinical Examinations (OSCEs) via a four-agent dialogue:

  • Doctor: the LLM or agent under test, interacts with patient and measurement agents to arrive at a diagnosis within a fixed inference budget.
  • Patient, Measurement & Moderator: driven by LLM for role-playing, providing history, lab results, and dialogue control.
  • Inference cap: 20 message exchanges per scenario.

Datasets

  1. AgentClinic-MedQA 214 board-exam–style scenarios (MedQA corpus), each with history, symptoms, labs, and gold-standard diagnosis.

  2. AgentClinic-MIMIC-IV 200 de-identified inpatient cases from MIMIC-IV, adapted to OSCE format.

  3. AgentClinic-NEJM 120 image-centric cases (radiographs, clinical photos) from NEJM Case Challenges.


MedAgentsBench

A "HARD" subset for complex medical QA, covering eight source datasets. Only questions with <50% correct rate in baseline LLMs were retained, ensuring high difficulty and requiring multi-step reasoning.

  • Total items: 862 multiple-choice questions.
  • Sources: MedQA, PubMedQA, MedMCQA, MedExQA, MMLU-Pro, MedBullets, MMLU, MedXpertQA (Reasoning & Understanding splits).

Humanity’s Last Exam

A recently developed benchmark comprising highly complex and nuanced cases across multiple academic areas, presented as multiple-choice and short-answer questions. Created by a consortium of academics and domain experts, each question avoids shortcut cues and requires in-depth domain knowledge and deep reasoning. Frontier LLMs have consistently demonstrated low accuracy on this benchmark.

  • Biology/Medicine Subset:
    • 222 text-only tasks
    • 58 multimodal tasks

Models and Agent Systems

LLMs

  • Text-only tasks:

    • Llama-4-Maverick-17B-128E-Instruct-FP8
    • Qwen-3-235B-A22B-FP8
    • GPT-4.1
    • MedGemma
  • Multimodal tasks (with vision support):

    • Llama-4-Maverick-17B-128E-Instruct-FP8
    • Gemma-3-27B-IT-Q8
    • GPT-4.1
    • MedGemma

Manus & OpenManus Variants

  • Manus (closed-source; invite-only) evaluated on MedAgentsBench and HLE via public web UI.
  • OpenManus (https://github.com/mannaandpoem/OpenManus) with a layered ReAct/ToolCall architecture, plus browsing, code execution, and search.
Configuration AgentClinic Text-only AgentClinic Multimodal MedAgentsBench HLE Text-only HLE Multimodal
Original OpenManus ✔️ ✔️ ✔️ ✔️ ✔️
OpenManus Customized 1 ✔️ (explicit role) n/a n/a n/a n/a
OpenManus Customized 2 ✔️ (max tool usage) n/a n/a n/a n/a
vOpenManus n/a ✔️ n/a n/a ✔️
Manus n/a n/a ✔️ ✔️ ✔️

Running the Benchmarks

AgentClinic CLI

Use the following Python scripts to run different AgentClinic configurations. Adjust flags as needed:

--doctor_llm llama4_maverick #set model used for doctor --patient_llm gpt-4o-mini #set model used for patient --measurement_llm gpt-4o-mini #set model used for measurement --moderator_llm gpt-4o-mini #set model used for moderator --num_scenarios 200 #set scenario number --agent_dataset MedQA #set dataset --doctor_image_request False #whether enable multimodality --total_inferences 20 #set inference time limit

# Text-only: Original OpenManus
python agentclinicmanus_original.py --agent_dataset MedQA_Ext

# Text-only: MedAssist
python agentclinicmanus_customized1.py --doctor_llm openmanus --agent_dataset MedQA_Ext

# Text-only: MedAssist_Tool+
python agentclinicmanus_customized2.py --doctor_llm openmanus --agent_dataset MedQA_Ext

Multimodal (images given at first)

# LLM baseline
python agentclinic_vllmf.py --agent_dataset NEJM_Ext --doctor_image_request True

# Original OpenManus
python agentclinicmanus_vf_original.py --doctor_llm openmanus --agent_dataset NEJM_Ext --doctor_image_request True

# vOpenManus 
python agentclinicmanus_vf.py --doctor_llm openmanus --agent_dataset NEJM_Ext --doctor_image_request True

Multimodal (images requested on demand)

# LLM baseline
python agentclinic_vllmr.py --agent_dataset NEJM_Ext --doctor_image_request True

# Original OpenManus
python agentclinicmanus_vr_original.py --doctor_llm openmanus --agent_dataset NEJM_Ext --doctor_image_request True

# vOpenManus
python agentclinicmanus_vr.py --doctor_llm openmanus --agent_dataset NEJM_Ext --doctor_image_request True

MedAgentsBench CLI

# LLM baseline
python run_llm.py \
  --model_name llama-4 \
  --dataset_name medexqa \
  --dataset_dir ../../data/medexqa/ \
  --split test_hard \
  --output_files_folder ./output/ \
  --num_processes 4 \
  --start_pos 0 \
  --end_pos -1

# OpenManus baseline
python run_openmanus.py \
  --dataset_name medexqa \
  --dataset_dir ../../data/medexqa/ \
  --split test_hard \
  --output_files_folder ./output_openmanus/ \
  --num_processes 4

Humanity’s Last Exam CLI

# LLM baseline (Text-only)
python hle_llm.py \

# LLM baseline (Multimodal)
python hle_llm_img.py \

# OpenManus baseline (Text-only)
python hle.py \

# vOpenManus (Multimodal)
python hle_img.py \

License

This project is licensed under the MIT License.