- Jupyter Notebook 99.3%
- Python 0.7%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| data collection | ||
| results | ||
| .gitignore | ||
| README.md | ||
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
This repository contains the data collection scripts and analysis code for the paper:
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
Buxin Su, Natalie Collina, Garrett Wen, Didong Li, Kyunghyun Cho, Jianqing Fan, Bingxin Zhao, Weijie Su
Overview
Can researchers predict the future impact of their own work? We conducted a large-scale experiment at ICML 2023 where 1,342 researchers ranked their 2,592 submissions by expected long-term scientific contribution. We then tracked citation outcomes over 16 months (July 2023 -- November 2024).
Key findings:
- Papers ranked highest by their authors received twice as many citations as their lowest-ranked counterparts
- Self-rankings were especially effective at identifying highly cited papers (150+ citations)
- Author self-rankings outperformed both pre- and post-rebuttal peer review scores in predicting citation impact
- Results held after controlling for confounding factors such as preprint timing and self-citations
Repository Structure
├── data collection/ # Data collection and cleaning scripts (Google Colab)
│ ├── data_clean_collection_google_citation.py
│ ├── data_clean_collection_time_citation.py
│ ├── data_clean_collection_self_cite.py
│ └── data_clean_collection_publication_date.py
├── results/ # Analysis and visualization notebooks
│ ├── Figures_1_&_3_&_Tables_1_&_2_rank_vs_citation.ipynb
│ ├── Figure_2_rank_vs_timed_citation.ipynb
│ ├── Figures_4_&_S_3_rank_vs_citation_conditional_scores.ipynb
│ ├── Figure_S_2_top_k_rank_vs_citation.ipynb
│ └── Figure_S_4_rank_vs_publication_time.ipynb
└── README.md
Data Collection
All data collection scripts are designed to run on Google Colab with Google Drive integration. They collect citation and metadata for ICML 2023 submissions from external APIs.
Scripts
| Script | Description | API Source |
|---|---|---|
data_clean_collection_google_citation.py |
Fetches cumulative citation counts | Google Scholar (via SerpAPI) |
data_clean_collection_time_citation.py |
Collects time-series citation counts in 2-month bins | Semantic Scholar API |
data_clean_collection_self_cite.py |
Collects time-series citations excluding self-citations | Semantic Scholar API |
data_clean_collection_publication_date.py |
Retrieves paper publication (preprint posting) dates | Semantic Scholar API |
Data Pipeline
- Input:
citation_with_rank_cleaned.csv-- contains submission IDs, titles, author information, and self-rankings - Paper matching: Titles are matched to API results using Levenshtein distance (threshold ≤ 20 edits) with up to 5 retry attempts for unmatched papers
- Post-processing: Deduplication, filtering out unmatched entries, and retaining only authors with multiple submissions
- Output: Cleaned CSV files in
Final_Data/
API Keys
- SerpAPI: Required for Google Scholar queries. Set via
SERPAPI_KEYenvironment variable or replace the placeholder in the script. - Semantic Scholar: Used without authentication by default (rate-limited). An optional API key can be configured for higher throughput.
Analysis & Figures
The Jupyter notebooks in results/ reproduce the figures and tables in the paper:
| Notebook | Paper Output |
|---|---|
Figures_1_&_3_&_Tables_1_&_2_rank_vs_citation.ipynb |
Figures 1, 3 and Tables 1, 2 -- Main analysis of self-rankings vs. citation counts (paired t-tests, stratified by acceptance decision) |
Figure_2_rank_vs_timed_citation.ipynb |
Figure 2 -- Time-series citation accumulation across 2-month windows |
Figures_4_&_S_3_rank_vs_citation_conditional_scores.ipynb |
Figures 4, S3 -- Rankings vs. citations conditional on reviewer scores |
Figure_S_2_top_k_rank_vs_citation.ipynb |
Figure S2 -- Top-K analysis of highly cited papers |
Figure_S_4_rank_vs_publication_time.ipynb |
Figure S4 -- Relationship between rankings and preprint posting dates |
Requirements
- Python 3.7+
- pandas, numpy, scipy, scikit-learn, matplotlib, seaborn
- python-Levenshtein
- google-search-results (SerpAPI client)
- requests
Citation
If you find this work useful, please cite:
@article{su2025fantastic,
title={How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review},
author={Su, Buxin and Collina, Natalie and Wen, Garrett and Li, Didong and Cho, Kyunghyun and Fan, Jianqing and Zhao, Bingxin and Su, Weijie},
journal={arXiv preprint arXiv:2510.02143},
year={2025}
}
License
This project is for academic research purposes. Please refer to the paper for terms of data usage.