Sovereign 8s.is mirror of standardgalactic/ICML_2023_Citation
  • Jupyter Notebook 99.3%
  • Python 0.7%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-03-25 01:39:33 -04:00
data collection Initial commit: data collection scripts, analysis notebooks, and README 2026-03-25 01:39:33 -04:00
results Initial commit: data collection scripts, analysis notebooks, and README 2026-03-25 01:39:33 -04:00
.gitignore Initial commit: data collection scripts, analysis notebooks, and README 2026-03-25 01:39:33 -04:00
README.md Initial commit: data collection scripts, analysis notebooks, and README 2026-03-25 01:39:33 -04:00

How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review

This repository contains the data collection scripts and analysis code for the paper:

How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review

Buxin Su, Natalie Collina, Garrett Wen, Didong Li, Kyunghyun Cho, Jianqing Fan, Bingxin Zhao, Weijie Su

[arXiv]

Overview

Can researchers predict the future impact of their own work? We conducted a large-scale experiment at ICML 2023 where 1,342 researchers ranked their 2,592 submissions by expected long-term scientific contribution. We then tracked citation outcomes over 16 months (July 2023 -- November 2024).

Key findings:

  • Papers ranked highest by their authors received twice as many citations as their lowest-ranked counterparts
  • Self-rankings were especially effective at identifying highly cited papers (150+ citations)
  • Author self-rankings outperformed both pre- and post-rebuttal peer review scores in predicting citation impact
  • Results held after controlling for confounding factors such as preprint timing and self-citations

Repository Structure

├── data collection/                # Data collection and cleaning scripts (Google Colab)
│   ├── data_clean_collection_google_citation.py
│   ├── data_clean_collection_time_citation.py
│   ├── data_clean_collection_self_cite.py
│   └── data_clean_collection_publication_date.py
├── results/                        # Analysis and visualization notebooks
│   ├── Figures_1_&_3_&_Tables_1_&_2_rank_vs_citation.ipynb
│   ├── Figure_2_rank_vs_timed_citation.ipynb
│   ├── Figures_4_&_S_3_rank_vs_citation_conditional_scores.ipynb
│   ├── Figure_S_2_top_k_rank_vs_citation.ipynb
│   └── Figure_S_4_rank_vs_publication_time.ipynb
└── README.md

Data Collection

All data collection scripts are designed to run on Google Colab with Google Drive integration. They collect citation and metadata for ICML 2023 submissions from external APIs.

Scripts

Script Description API Source
data_clean_collection_google_citation.py Fetches cumulative citation counts Google Scholar (via SerpAPI)
data_clean_collection_time_citation.py Collects time-series citation counts in 2-month bins Semantic Scholar API
data_clean_collection_self_cite.py Collects time-series citations excluding self-citations Semantic Scholar API
data_clean_collection_publication_date.py Retrieves paper publication (preprint posting) dates Semantic Scholar API

Data Pipeline

  1. Input: citation_with_rank_cleaned.csv -- contains submission IDs, titles, author information, and self-rankings
  2. Paper matching: Titles are matched to API results using Levenshtein distance (threshold ≤ 20 edits) with up to 5 retry attempts for unmatched papers
  3. Post-processing: Deduplication, filtering out unmatched entries, and retaining only authors with multiple submissions
  4. Output: Cleaned CSV files in Final_Data/

API Keys

  • SerpAPI: Required for Google Scholar queries. Set via SERPAPI_KEY environment variable or replace the placeholder in the script.
  • Semantic Scholar: Used without authentication by default (rate-limited). An optional API key can be configured for higher throughput.

Analysis & Figures

The Jupyter notebooks in results/ reproduce the figures and tables in the paper:

Notebook Paper Output
Figures_1_&_3_&_Tables_1_&_2_rank_vs_citation.ipynb Figures 1, 3 and Tables 1, 2 -- Main analysis of self-rankings vs. citation counts (paired t-tests, stratified by acceptance decision)
Figure_2_rank_vs_timed_citation.ipynb Figure 2 -- Time-series citation accumulation across 2-month windows
Figures_4_&_S_3_rank_vs_citation_conditional_scores.ipynb Figures 4, S3 -- Rankings vs. citations conditional on reviewer scores
Figure_S_2_top_k_rank_vs_citation.ipynb Figure S2 -- Top-K analysis of highly cited papers
Figure_S_4_rank_vs_publication_time.ipynb Figure S4 -- Relationship between rankings and preprint posting dates

Requirements

  • Python 3.7+
  • pandas, numpy, scipy, scikit-learn, matplotlib, seaborn
  • python-Levenshtein
  • google-search-results (SerpAPI client)
  • requests

Citation

If you find this work useful, please cite:

@article{su2025fantastic,
  title={How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review},
  author={Su, Buxin and Collina, Natalie and Wen, Garrett and Li, Didong and Cho, Kyunghyun and Fan, Jianqing and Zhao, Bingxin and Su, Weijie},
  journal={arXiv preprint arXiv:2510.02143},
  year={2025}
}

License

This project is for academic research purposes. Please refer to the paper for terms of data usage.