Sovereign 8s.is mirror of standardgalactic/liveideabench
  • Jupyter Notebook 99.2%
  • Python 0.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2025-04-22 10:46:41 +08:00
assets add fonts 2025-04-22 10:23:50 +08:00
csvs Add files via upload 2025-03-25 12:37:56 +08:00
data Add fluency score files 2025-04-09 23:21:02 +08:00
keywords_data Add files via upload 2025-03-25 12:37:04 +08:00
results update 2025-02-07 21:28:14 +08:00
tokenizer Update README.md 2025-04-11 11:11:06 +08:00
utils update 2025-04-22 08:49:13 +08:00
.gitignore change font path 2025-04-22 10:46:41 +08:00
co2.ipynb update 2025-04-22 07:27:38 +08:00
config.py update 2025-04-22 07:27:38 +08:00
environment.yaml add flexibility.ipynb and co2 related estimation 2025-04-09 22:55:47 +08:00
hash.py update 2025-04-22 07:27:38 +08:00
LICENSE Create LICENSE 2025-04-11 11:12:01 +08:00
README.md Update README.md 2025-04-22 09:18:13 +08:00
requirements.txt Add files via upload 2025-04-18 20:45:43 +08:00
run.py add HINT for model name 2025-04-22 09:05:47 +08:00
stats.ipynb change font path 2025-04-22 10:46:41 +08:00
stats_flexibility.ipynb change font path 2025-04-22 10:46:41 +08:00
view_database.py better preview 2025-04-22 09:16:07 +08:00

🤖💡 LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context

arXiv AI4Science License CO2 Estimation Stars

"It's not like finding a needle in a haystack, it is like creating new needles."

🏆 Leaderboard: http://liveideabench.com 💡

🤗 Dataset

Hugging Face Models Hugging Face Models Hugging Face Models

🧠✨🎉 News (2025/3/29): Latest Dataset Update (v2) on Hugging Face!

We are pleased to announce that, based on the invaluable feedback from reviewers, we have enhanced our benchmark by upgrading it to version 2. This update introduces a new dimension—Clarity—and improves the prompts, evaluation process (including the rejection handling mechanism), making our benchmark more comprehensive and objective. 🚀

This v2 version of the benchmark incorporates the latest models, including: claude-3.7-sonnet:thinking, o3-mini-high, gpt-4.5-preview, qwq-32b, deepseek-r1, gemini-2.0-flash-thinking, and a total of 41 state-of-the-art models.

🧠✨🎉 News (2025/1/27): Latest Dataset Update on Hugging Face!

We are excited to announce that the latest dataset, including supplementary tests for models like deepseek-R1, deepseek-V3, minimax-01, phi-4, and Opus, has been uploaded to Hugging Face! 🚀


LiveIdeaBench Evaluation Framework

LiveIdeaBench Evaluation Framework Leaderboard

👇 Evaluation Instruction

1️⃣ Install Environment

pip install -r requirements.txt

2️⃣ Database Initialization

Run the Python script to initialize the database:

python -c "from utils.database import init_database; init_database()"

3️⃣ Configuring API Keys

Before running the program, you need to configure at least one API key:

  1. Create an apikey file and write your OpenRouter API key:

    echo "your-openrouter-api-key" > apikey
    

    Alternatively, set environment variables:

    export OPENROUTER_API_KEY="your-openrouter-api-key"
    export STEP_API_KEY="your-step-api-key"
    export GEMINI_API_KEYS="key1,key2,key3"
    

4️⃣ Running Examples

Generate and evaluate ideas using a specified model:

# Generate ideas using a specified model
python run.py --idea_model "openai/gpt-4o-mini"

# Use a specific provider
python run.py --idea_model "openai/gpt-4o-mini" --provider openrouter
# Use a single keyword:

python run.py --idea_model "openai/gpt-4o-mini" --keyword "relativity"
# Use multiple keywords:

python run.py --idea_model "openai/gpt-4o-mini" --keyword "relativity" "periodic table"
# Do not specify a keyword (use all keywords):

python run.py --idea_model "openai/gpt-4o-mini"

5️⃣ Database Export

This step extracts the generated ideas, scores, and metadata from the internal database.

Run the script:

python view_database.py      

to extract the generated data from the SQL database.

Then, run stats.ipynb, to generate data/data.parquet which serves as input for the subsequent analysis notebooks.

6️⃣ Evaluate Fluency

Fluency measures the diversity and uniqueness of the generated ideas. This script calculates the Fluency score based on the processed data.

Run the script:

python hash.py

7️⃣ Compute Flexibility & Plotting

Flexibility evaluates whether the model's generated ideas span across diverse scientific disciplines based on the input keyword(s).

This notebook calculates the Flexibility score and creates visualizations of the benchmark results.

Run the Jupyter Notebook: stats_flexibility.ipynb

Generated figures can be found in the ./figs directory.

🌍🌱 CO2 Emission Estimation

This repo provides an estimation of the CO2 footprint associated with running the idea generation and evaluation pipeline.

Run the Jupyter Notebook: co2.ipynb

Bibtex

@article{ruan2024liveideabench,
title={LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context},
author={Kai Ruan and Xuan Wang and Jixiang Hong and Peng Wang and Yang Liu and Hao Sun},
journal={arXiv preprint arXiv:2412.17596},
year={2024}
}