- Python 98.5%
- Shell 1.5%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| RealChart2Code_eval | ||
| .gitignore | ||
| hf_dataset_README.md | ||
| README.md | ||
| README_CN.md | ||
| to_excel_by_sub_score.py | ||
RealChart2Code Benchmark
A comprehensive benchmark for evaluating LLM capabilities in generating Python visualization code from real-world chart images. The benchmark contains 1,016 chart tasks sourced from Kaggle datasets, covering 3 task types across 7 chart categories and 3 difficulty levels.
Overview
RealChart2Code evaluates how well large language models can:
- Task 1 - Replication: Generate visualization code from a chart image alone (using synthetic data)
- Task 2 - Reproduction: Generate visualization code from a chart image plus the original data files
- Task 3 - Refinement: Improve flawed visualization code through multi-turn correction with natural language instructions
Benchmark Statistics
| Item | Count |
|---|---|
| Total chart samples | 5,529 |
| Task 1 & 2 tasks | 1,016 |
| Task 3 tasks | 864 |
| Chart categories | 7 (Change, Comparison, Composition, Distribution, Groups, Relationship, Spatial) |
| Difficulty levels | 3 (easy, middle, hard) |
| Visualization libraries | matplotlib, seaborn, plotly, bokeh, altair |
Directory Structure
RealChart2Code/
├── README.md # English README
├── README_CN.md # Chinese README
├── to_excel_by_sub_score.py # Per-sub-metric score aggregator
├── RealChart2Code_eval/
│ ├── evaluate_task1.py # Task 1: Replication evaluation
│ ├── evaluate_task2.py # Task 2: Reproduction evaluation
│ ├── evaluate_task3.py # Task 3: Refinement evaluation
│ ├── run_task1.sh # Task 1 launch script
│ ├── run_task2.sh # Task 2 launch script
│ ├── run_task3.sh # Task 3 launch script
│ ├── get_results.py # Aggregate results across models
│ ├── requirements.txt # Python dependencies
│ ├── prompt_task1/ # Prompt templates for Task 1
│ ├── prompt_task2/ # Prompt templates for Task 2
│ ├── prompt_task3/ # Prompt templates for Task 3
│ ├── data/
│ │ ├── data_task1_task2.json # Task definitions for Task 1 & 2
│ │ ├── data_task3.json # Task definitions for Task 3
│ │ └── selected_chart2code_benchmark_data/ # Benchmark dataset (from HuggingFace)
│ ├── results_task1/ # Pre-computed Task 1 results (from HuggingFace)
│ ├── results_task2/ # Pre-computed Task 2 results (from HuggingFace)
│ └── results_task3/ # Pre-computed Task 3 results (from HuggingFace)
Installation
1. Clone Repository
git clone https://github.com/Speakn0w/RealChart2Code.git
cd RealChart2Code
2. Python Environment
Requires Python 3.10+. We recommend creating a virtual environment:
python3 -m venv venv
source venv/bin/activate
3. Install Dependencies
cd RealChart2Code_eval
pip install -r requirements.txt
Key dependencies:
fluxllm(>=0.2.0) - Batch LLM request client with retry and rate limitingmatplotlib,seaborn,plotly,bokeh,altair- Visualization librariespandas,numpy,Pillow- Data and image processing
4. Download Benchmark Data from HuggingFace
The benchmark data and pre-computed results are hosted on HuggingFace due to their large size (~9.8GB compressed):
Dataset: https://huggingface.co/datasets/zjj1233/RealChart2Code
Option A: Using huggingface-cli (Recommended)
pip install huggingface_hub
# Download all data to RealChart2Code_eval/
huggingface-cli download zjj1233/RealChart2Code --repo-type dataset --local-dir hf_data
# Decompress benchmark data
cat hf_data/benchmark_data.tar.gz.* | tar -xzf - -C data/
# Decompress pre-computed evaluation results
tar -xzf hf_data/results_task1.tar.gz
tar -xzf hf_data/results_task2.tar.gz
tar -xzf hf_data/results_task3.tar.gz
# Clean up (optional)
rm -rf hf_data
Option B: Manual Download
- Visit https://huggingface.co/datasets/zjj1233/RealChart2Code
- Download all
benchmark_data.tar.gz.*files andresults_task*.tar.gzfiles - Place them in
RealChart2Code_eval/and run the decompression commands above
After decompression, you should have:
data/selected_chart2code_benchmark_data/- 5,529 chart sample directories (~12GB)results_task1/,results_task2/,results_task3/- Pre-computed evaluation results (~241MB)
5. API Configuration
The benchmark uses an OpenAI-compatible API endpoint. You need:
- API Base URL: Your API endpoint (e.g.,
https://api.openai.com/v1) - API Key: Your API key
- Generation Model: The model to generate visualization code (must support vision/image input)
- Evaluation Model: The model to judge generated charts (must support vision/image input)
Quick Start
Run Evaluation
Edit the shell script variables and run:
cd RealChart2Code_eval
# Edit run_task1.sh to configure:
# GENERATION_MODEL="your-generation-model-name"
# EVALUATION_MODEL="your-evaluation-model-name"
# API_BASE_URL="https://your-api-endpoint/v1"
# API_KEY="your-api-key"
bash run_task1.sh # Task 1: Replication
bash run_task2.sh # Task 2: Reproduction
bash run_task3.sh # Task 3: Refinement
Run with Custom Parameters
python evaluate_task1.py \
--mode both \
--generation_model_name "gpt-4o" \
--evaluation_model_name "gpt-4o" \
--generation_base_url "https://api.openai.com/v1" \
--evaluation_base_url "https://api.openai.com/v1" \
--generation_api_key "YOUR_API_KEY" \
--evaluation_api_key "YOUR_API_KEY" \
--data_dir "data/data_task1_task2.json" \
--results_dir "results_task1" \
--generation_prompt_path "prompt_task1/benchmark_generate_prompt.txt" \
--evaluation_prompt_path "prompt_task1/eval.txt" \
--max_data_rows 5 \
--timeout 120 \
--max_retries 20 \
--max_qps 15
Key Arguments
| Argument | Description | Default |
|---|---|---|
--mode |
generate, evaluate, or both |
both |
--generation_model_name |
Model for code generation | gpt-4o |
--evaluation_model_name |
Model for evaluation (judge) | gpt-4o |
--data_dir |
Path to task data JSON | data |
--results_dir |
Output directory for results | results |
--max_data_rows |
Max data rows included in prompt | 10 |
--timeout |
Code execution timeout (seconds) | 120 |
--max_retries |
API retry limit | 5 |
--max_qps |
Max queries per second | 20 |
--diagnose |
Enable verbose diagnostics | false |
Task Types
Task 1 - Replication (Image-Only to Code)
- Input: Chart image + category label
- Goal: Generate Python code that replicates the chart using synthetic data
- Evaluation: Visual structure alignment + execution quality
- Script:
evaluate_task1.py
Task 2 - Reproduction (Image + Data to Code)
- Input: Chart image + category label + actual data files
- Goal: Generate Python code using the provided data files
- Evaluation: Visual structure alignment + execution quality + data alignment
- Script:
evaluate_task2.py
Task 3 - Refinement (Multi-Turn Correction)
- Input: Target chart image + flawed code + improvement instructions
- Goal: Fix/improve the code based on natural language correction instructions
- Evaluation: Visual structure alignment + execution quality
- Script:
evaluate_task3.py
Evaluation Metrics
Visual Structure Alignment (8 metrics, each scored 0/1/2)
- Chart Type Consistency - Correct chart type
- Spatial Layout Consistency - Layout and subplot arrangement
- Text Element Consistency - Titles, labels, legends
- Axis Configuration Consistency - Axis scales, ranges, ticks
- Color Scheme Consistency - Color palette matching
- Style and Format Consistency - Grid, borders, themes
- Data Pattern Consistency - Data trends and patterns
- Component Completeness - All visual elements present
Execution Quality (3 metrics, each scored 0/1/2)
- Visual Clarity - Clean, readable output
- Compositional Balance - Proper spacing and proportions
- Typographic Quality - Text readability, no overlaps
Data Alignment (Task 2 only, scored 0 or 2)
- Verifies correct data usage from provided files
Maximum Scores: Task 1/3 = 22 points, Task 2 = 24 points
Aggregating Results
# Aggregate statistics across all models
python get_results.py
# Per-sub-metric summary for a specific model
python ../to_excel_by_sub_score.py <results_directory> -p "score_*.json"
Pre-computed Results
Pre-computed evaluation results (downloaded from HuggingFace) cover multiple models:
- results_task1/: 16 models evaluated
- results_task2/: 13 models evaluated
- results_task3/: 14 models evaluated
Each model directory contains statistics.json with aggregated scores and individual score_*.json files for each task.
Troubleshooting
- 429 Rate Limit Errors: Increase
--max_retriesor decrease--max_qps - Code Execution Failures: Ensure all visualization library versions match (matplotlib, seaborn, plotly, bokeh, altair)
- Font Warnings: Install Arial font or ignore
findfontwarnings (cosmetic only) - API Compatibility: The evaluation requires an OpenAI-compatible chat API format with
messagesandmax_tokensfields
License
This benchmark uses publicly available datasets from Kaggle. Individual dataset licenses are recorded in each sample's complete_metadata.json.