- Jupyter Notebook 90.8%
- Python 9.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
Some checks failed
Run tests / test (push) Has been cancelled
updated figures according to journal guidance |
||
| .github/workflows | ||
| data | ||
| figures | ||
| notebooks | ||
| src | ||
| tests | ||
| .env.example | ||
| .gitignore | ||
| CITATION.cff | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| environment.yaml | ||
| LICENSE | ||
| README.md | ||
TRIPOD-Code
This repository contains the code used in Code Sharing In Prediction Model Research: A Scoping Review.
Table of Contents
Introduction
In this work, we conducted a large scale analysis of research articles and their associated code repositories to quantify current code-sharing practices. This repository contains the code used to obtain our results.
To characterize articles and repositories, we used OpenAI's structured output generation tool. If you are not familiar with it, you can find an example of how it works in this notebook, where we apply it on this repository.
Repository Structure
TRIPOD-Code/
├── data/ # Data files at each pipeline stage
│ ├── 00_references/ # Raw PubMed citations for TRIPOD statements
│ ├── 01_full_text/ # Articles with full text from PMC
│ ├── 02_paper_assessment/ # Article assessment annotations & predictions
│ └── 03_repo_assessment/ # Repository assessment annotations & predictions
│
├── notebooks/ # Main analysis pipeline (run in order)
│ ├── 00_merge_TRIPOD_references.ipynb
│ ├── 01_gather_full_text.ipynb
│ ├── 02a-c_*.ipynb # Article assessment (annotations, eval, predictions)
│ ├── 03a-d_*.ipynb # Repository assessment (annotations, eval, download, predictions)
│ ├── 04_final_analysis.ipynb # Generate figures and statistics
│ └── example.ipynb # Tutorial on structured output generation
│
├── src/ # Source code modules
│ ├── llm_utils/ # LLM interaction utilities
│ │ ├── llm_wrapper.py # Wrapper for OpenAI structured output API
│ │ ├── structs.py # Pydantic schemas (PaperAssessment, RepoAssessment)
│ │ ├── paper_assessment_prompt.py # Prompt for article assessment
│ │ └── repo_assessment_prompt.py # Prompt for repository assessment
│ │
│ └── repo_utils/ # Repository cloning & extraction
│ ├── main_extractor.py # Main entry point (clone_and_extract_tree)
│ ├── repo_providers.py # Platform-specific cloners (GitHub, Zenodo, OSF, etc.)
│ └── tokenizer.py # Text extraction and tokenization
│
├── tests/ # Unit tests
│ ├── test_llm_utils.py # Tests for LLM utilities (mocked API calls)
│ └── test_repo_utils.py # Tests for repository utilities
│
├── figures/ # Generated figures from analysis
├── environment.yaml # Conda environment specification
├── CONTRIBUTING.md # Contribution guidelines
└── CODE_OF_CONDUCT.md # Community code of conduct
Key components:
notebooks/: Sequential pipeline notebooks (00 → 04) that reproduce our analysissrc/llm_utils/: Handles structured output generation with OpenAI's APIsrc/repo_utils/: Clones repositories from GitHub, Zenodo, OSF, Figshare, and DOI linksdata/: Intermediate and final datasets at each pipeline stage
Setup
To use our repository, you first need to follow these steps:
-
Clone Repository and access it
git clone https://github.com/thomas-sounack/TRIPOD-Code.git cd TRIPOD-Code -
Install Conda (if not already installed)
If you don't have Conda installed, you can download it from Miniconda or Anaconda. -
Create the Conda Environment and activate it
Use the providedenvironment.yamlfile to create the environment. Run the following command:conda env create -f environment.yaml conda activate TRIPOD-CodeThis command takes about one minute to run.
-
Add your OpenAI API key
Our datasets at every step of the process are made available. However, if you would like to reproduce our results, or want to run new data through OpenAI's model, you will need to add an API key. For this, rename the file.env.exampleto.env, and paste your API key afterOPENAI_API_KEY=.Note: you can visit OpenAI's website for more information on how to obtain an API key, and pricing. It cost about $150 to run this pipeline end-to-end when this study was realized.
-
Hardware requirements The LLMs used in this repository are being queried through an API. As such, no GPUs are needed to run this code. However, the Gather full text step can be computationally heavy if you decide to re-build the pmid - full text index, which requires a lot of RAM. This project was ran on a computer with 48GB of RAM.
How to use this repository
To reproduce our results, you can run the following notebooks:
-
- Inputs: This notebook uses the data present in
data/00_references. The csv files present in theTRIPOD/andTRIPOD_AIfolders were downloaded from the Pubmed website, each corresponding to one entry of the TRIPOD statements. - Purpose: The notebook aggregates all of the records citing one of the TRIPOD statements into a single file, and removes duplicate entries
- Outputs: The notebook outputs a single file:
data/00_references/merged_references.parquet.br.
- Inputs: This notebook uses the data present in
-
- Inputs: The file generated in the last step:
data/00_references/merged_references.parquet.br. - Purpose: The notebook gathers the full text of the records using the PMC Open Access API. It removes records that are not indexed in the PMC Open Access API.
- Outputs: The notebook outputs a single file:
data/01_full_text/references_with_full_text.parquet.br.
- Inputs: The file generated in the last step:
-
Article Assessment:
-
2a) Create paper assesstment annotations:
- Inputs: The file generated in the last step
data/01_full_text/references_with_full_text.parquet.brand the file containing the annotations of 500 articles:data/02_paper_assessment/raw_annotations_paper_assessment.csv. - Purpose: This notebook adds the full text information to the annotated records, and renames columns with a
gt_suffix to facilitate model evaluation. - Outputs: The notebook outputs a single file:
data/02_paper_assessment/gt_paper_assessment.csv.
- Inputs: The file generated in the last step
-
2b) Evaluate paper assessment prompt:
- Inputs: The file generated in the last step:
data/02_paper_assessment/gt_paper_assessment.csv, as well as the model prompt and structured output schema defined insrc/llm_utils/paper_assessment_prompt.pyandsrc/llm_utils/structs.py. - Purpose: This notebook evaluates the LLM's article assessment using the 500 annotated articles.
- Outputs: The model's outputs (saved in
data/02_paper_assessment/evaluation_paper_assessment.csv) and performance metrics.
- Inputs: The file generated in the last step:
-
2c) Generate paper assessment predictions:
- Inputs: The file containing the full text articles generated in step 1:
data/01_full_text/references_with_full_text.parquet.br, as well as the model prompt and structured output schema defined insrc/llm_utils/paper_assessment_prompt.pyandsrc/llm_utils/structs.py. - Purpose: This notebook runs the LLM's article assessment on the entire dataset.
- Outputs: The model's outputs (saved in
data/02_paper_assessment/paper_assessment_pred.parquet.br).
- Inputs: The file containing the full text articles generated in step 1:
-
-
Repository Assessment:
-
3a) Create repository assessment annotations:
- Inputs: The file containing the annotated repositories (
data/03_repo_assessment/raw_annotations_repo_assessment.csv). - Purpose: This notebook downloads the repositories using the repository utility (
src/repo_utils/), and adds the repository contents to the annotated data, renaming columns with agt_suffix to facilitate model evaluation. - Outputs: The notebook outputs a single file
data/03_repo_assessment/gt_repo_assessment.csv.
- Inputs: The file containing the annotated repositories (
-
3b) Evaluate repository assessment prompt:
- Inputs: The file generated in the last step:
data/03_repo_assessment/gt_repo_assessment.csv, as well as the model prompt and structured output schema defined insrc/llm_utils/repo_assessment_prompt.pyandsrc/llm_utils/structs.py. - Purpose: This notebook evaluates the LLM's repository assessment using the annotated repositories.
- Outputs: The model's outputs (saved in
data/03_repo_assessment/evaluation_repo_assessment.csv) and performance metrics.
- Inputs: The file generated in the last step:
-
- Inputs: The file containing the extracted repository links generated in step 2c (
data/02_paper_assessment/paper_assessment_pred.parquet.br). - Purpose: This notebook downloads the repositories for the entire dataset using the repository utility (
src/repo_utils/). - Outputs: A file containing all the rows of the dataset for which a repository link was found in step 2c, along with the repository contents:
data/03_repo_assessment/references_with_repo.parquet.br.
- Inputs: The file containing the extracted repository links generated in step 2c (
-
3d) Generate repository assessment predictions:
- Inputs: The file generated in the last step:
data/03_repo_assessment/references_with_repo.parquet.br. - Purpose: This notebook runs the LLM's repository assessment on all repositories that were downloaded.
- Outputs: The model's outputs (saved in
data/03_repo_assessment/references_repo_assessment_pred.parquet.br).
- Inputs: The file generated in the last step:
-
-
- Inputs: The LLM article assessment (
data/02_paper_assessment/paper_assessment_pred.parquet.br) and repository assessment (data/03_repo_assessment/references_repo_assessment_pred.parquet.br). - Purpose: Generates figures and analyses using the model outputs.
- Outputs: Article and repository statistics, and figures saved in
data/figures.
- Inputs: The LLM article assessment (
Running this process end-to-end takes about six hours using the existing PMID - full text dictionary. Re-creating the dictionary adds about seven hours to the runtime (on a Macbook M4 Pro).
Reference
If you use this repository, please cite our paper:
#TODO: insert citation
If you would like to cite our repository as well, you can do so using this citation:
@software{Sounack_TRIPOD-Code_2026,
author = {Sounack, Thomas and Giancotti, Raffaele and Barreñada, Lasai and Pollard, Tom},
doi = {10.5281/zenodo.18897514},
month = mar,
title = {{TRIPOD-Code}},
url = {https://github.com/thomas-sounack/TRIPOD-Code},
version = {1.0.0},
year = {2026}
}