Sovereign 8s.is mirror of standardgalactic/TRIPOD-Code
  • Jupyter Notebook 90.8%
  • Python 9.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Thomas Sounack ae31383455
Some checks failed
Run tests / test (push) Has been cancelled
Merge pull request #18 from thomas-sounack/rework_figures
updated figures according to journal guidance
2026-08-25 16:29:11 -04:00
.github/workflows updated workflow to install conda env 2026-03-14 12:21:54 -04:00
data added explanations full text mapping 2026-03-16 11:59:26 -04:00
figures updated figures according to journal guidance 2026-08-25 16:24:10 -04:00
notebooks updated figures according to journal guidance 2026-08-25 16:24:10 -04:00
src added sphinx docs to repo utils 2026-03-16 10:19:08 -04:00
tests added tests 2026-03-14 12:10:32 -04:00
.env.example Tidy .env file content. 2026-03-09 14:42:28 -04:00
.gitignore ignore checkpoints 2026-01-02 23:31:37 +01:00
CITATION.cff added citation file 2026-03-16 13:16:09 -04:00
CODE_OF_CONDUCT.md added code of conduct 2026-03-16 10:10:56 -04:00
CONTRIBUTING.md added contributing file 2026-03-16 10:14:32 -04:00
environment.yaml fixed pip install typo 2026-03-14 12:24:50 -04:00
LICENSE Initial commit 2025-08-04 09:41:18 -04:00
README.md fixed typo 2026-06-12 16:40:57 -07:00

DOI

TRIPOD-Code

This repository contains the code used in Code Sharing In Prediction Model Research: A Scoping Review.

Table of Contents

  1. Introduction
  2. Repository Structure
  3. Setup
  4. How to use this repository
  5. Reference

Introduction

In this work, we conducted a large scale analysis of research articles and their associated code repositories to quantify current code-sharing practices. This repository contains the code used to obtain our results.

To characterize articles and repositories, we used OpenAI's structured output generation tool. If you are not familiar with it, you can find an example of how it works in this notebook, where we apply it on this repository.

Repository Structure

TRIPOD-Code/
├── data/                          # Data files at each pipeline stage
│   ├── 00_references/             # Raw PubMed citations for TRIPOD statements
│   ├── 01_full_text/              # Articles with full text from PMC
│   ├── 02_paper_assessment/       # Article assessment annotations & predictions
│   └── 03_repo_assessment/       # Repository assessment annotations & predictions
│
├── notebooks/                     # Main analysis pipeline (run in order)
│   ├── 00_merge_TRIPOD_references.ipynb
│   ├── 01_gather_full_text.ipynb
│   ├── 02a-c_*.ipynb              # Article assessment (annotations, eval, predictions)
│   ├── 03a-d_*.ipynb              # Repository assessment (annotations, eval, download, predictions)
│   ├── 04_final_analysis.ipynb    # Generate figures and statistics
│   └── example.ipynb              # Tutorial on structured output generation
│
├── src/                           # Source code modules
│   ├── llm_utils/                 # LLM interaction utilities
│   │   ├── llm_wrapper.py         # Wrapper for OpenAI structured output API
│   │   ├── structs.py             # Pydantic schemas (PaperAssessment, RepoAssessment)
│   │   ├── paper_assessment_prompt.py   # Prompt for article assessment
│   │   └── repo_assessment_prompt.py    # Prompt for repository assessment
│   │
│   └── repo_utils/                # Repository cloning & extraction
│       ├── main_extractor.py      # Main entry point (clone_and_extract_tree)
│       ├── repo_providers.py      # Platform-specific cloners (GitHub, Zenodo, OSF, etc.)
│       └── tokenizer.py           # Text extraction and tokenization
│
├── tests/                         # Unit tests
│   ├── test_llm_utils.py          # Tests for LLM utilities (mocked API calls)
│   └── test_repo_utils.py         # Tests for repository utilities
│
├── figures/                       # Generated figures from analysis
├── environment.yaml               # Conda environment specification
├── CONTRIBUTING.md                # Contribution guidelines
└── CODE_OF_CONDUCT.md             # Community code of conduct

Key components:

  • notebooks/: Sequential pipeline notebooks (00 → 04) that reproduce our analysis
  • src/llm_utils/: Handles structured output generation with OpenAI's API
  • src/repo_utils/: Clones repositories from GitHub, Zenodo, OSF, Figshare, and DOI links
  • data/: Intermediate and final datasets at each pipeline stage

Setup

To use our repository, you first need to follow these steps:

  1. Clone Repository and access it

    git clone https://github.com/thomas-sounack/TRIPOD-Code.git
    cd TRIPOD-Code
    
  2. Install Conda (if not already installed)
    If you don't have Conda installed, you can download it from Miniconda or Anaconda.

  3. Create the Conda Environment and activate it
    Use the provided environment.yaml file to create the environment. Run the following command:

    conda env create -f environment.yaml
    conda activate TRIPOD-Code
    

    This command takes about one minute to run.

  4. Add your OpenAI API key
    Our datasets at every step of the process are made available. However, if you would like to reproduce our results, or want to run new data through OpenAI's model, you will need to add an API key. For this, rename the file .env.example to .env, and paste your API key after OPENAI_API_KEY=.

    Note: you can visit OpenAI's website for more information on how to obtain an API key, and pricing. It cost about $150 to run this pipeline end-to-end when this study was realized.

  5. Hardware requirements The LLMs used in this repository are being queried through an API. As such, no GPUs are needed to run this code. However, the Gather full text step can be computationally heavy if you decide to re-build the pmid - full text index, which requires a lot of RAM. This project was ran on a computer with 48GB of RAM.

How to use this repository

To reproduce our results, you can run the following notebooks:

  1. Merge TRIPOD references:

    • Inputs: This notebook uses the data present in data/00_references. The csv files present in the TRIPOD/ and TRIPOD_AI folders were downloaded from the Pubmed website, each corresponding to one entry of the TRIPOD statements.
    • Purpose: The notebook aggregates all of the records citing one of the TRIPOD statements into a single file, and removes duplicate entries
    • Outputs: The notebook outputs a single file: data/00_references/merged_references.parquet.br.
  2. Gather full text:

    • Inputs: The file generated in the last step: data/00_references/merged_references.parquet.br.
    • Purpose: The notebook gathers the full text of the records using the PMC Open Access API. It removes records that are not indexed in the PMC Open Access API.
    • Outputs: The notebook outputs a single file: data/01_full_text/references_with_full_text.parquet.br.
  3. Article Assessment:

    • 2a) Create paper assesstment annotations:

      • Inputs: The file generated in the last step data/01_full_text/references_with_full_text.parquet.br and the file containing the annotations of 500 articles: data/02_paper_assessment/raw_annotations_paper_assessment.csv.
      • Purpose: This notebook adds the full text information to the annotated records, and renames columns with a gt_ suffix to facilitate model evaluation.
      • Outputs: The notebook outputs a single file: data/02_paper_assessment/gt_paper_assessment.csv.
    • 2b) Evaluate paper assessment prompt:

      • Inputs: The file generated in the last step: data/02_paper_assessment/gt_paper_assessment.csv, as well as the model prompt and structured output schema defined in src/llm_utils/paper_assessment_prompt.py and src/llm_utils/structs.py.
      • Purpose: This notebook evaluates the LLM's article assessment using the 500 annotated articles.
      • Outputs: The model's outputs (saved in data/02_paper_assessment/evaluation_paper_assessment.csv) and performance metrics.
    • 2c) Generate paper assessment predictions:

      • Inputs: The file containing the full text articles generated in step 1: data/01_full_text/references_with_full_text.parquet.br, as well as the model prompt and structured output schema defined in src/llm_utils/paper_assessment_prompt.py and src/llm_utils/structs.py.
      • Purpose: This notebook runs the LLM's article assessment on the entire dataset.
      • Outputs: The model's outputs (saved in data/02_paper_assessment/paper_assessment_pred.parquet.br).
  4. Repository Assessment:

    • 3a) Create repository assessment annotations:

      • Inputs: The file containing the annotated repositories (data/03_repo_assessment/raw_annotations_repo_assessment.csv).
      • Purpose: This notebook downloads the repositories using the repository utility (src/repo_utils/), and adds the repository contents to the annotated data, renaming columns with a gt_ suffix to facilitate model evaluation.
      • Outputs: The notebook outputs a single file data/03_repo_assessment/gt_repo_assessment.csv.
    • 3b) Evaluate repository assessment prompt:

      • Inputs: The file generated in the last step: data/03_repo_assessment/gt_repo_assessment.csv, as well as the model prompt and structured output schema defined in src/llm_utils/repo_assessment_prompt.py and src/llm_utils/structs.py.
      • Purpose: This notebook evaluates the LLM's repository assessment using the annotated repositories.
      • Outputs: The model's outputs (saved in data/03_repo_assessment/evaluation_repo_assessment.csv) and performance metrics.
    • 3c) Download repositories:

      • Inputs: The file containing the extracted repository links generated in step 2c (data/02_paper_assessment/paper_assessment_pred.parquet.br).
      • Purpose: This notebook downloads the repositories for the entire dataset using the repository utility (src/repo_utils/).
      • Outputs: A file containing all the rows of the dataset for which a repository link was found in step 2c, along with the repository contents: data/03_repo_assessment/references_with_repo.parquet.br.
    • 3d) Generate repository assessment predictions:

      • Inputs: The file generated in the last step: data/03_repo_assessment/references_with_repo.parquet.br.
      • Purpose: This notebook runs the LLM's repository assessment on all repositories that were downloaded.
      • Outputs: The model's outputs (saved in data/03_repo_assessment/references_repo_assessment_pred.parquet.br).
  5. Final analysis:

    • Inputs: The LLM article assessment (data/02_paper_assessment/paper_assessment_pred.parquet.br) and repository assessment (data/03_repo_assessment/references_repo_assessment_pred.parquet.br).
    • Purpose: Generates figures and analyses using the model outputs.
    • Outputs: Article and repository statistics, and figures saved in data/figures.

Running this process end-to-end takes about six hours using the existing PMID - full text dictionary. Re-creating the dictionary adds about seven hours to the runtime (on a Macbook M4 Pro).

Reference

If you use this repository, please cite our paper:

#TODO: insert citation

If you would like to cite our repository as well, you can do so using this citation:

@software{Sounack_TRIPOD-Code_2026,
    author = {Sounack, Thomas and Giancotti, Raffaele and Barreñada, Lasai and Pollard, Tom},
    doi = {10.5281/zenodo.18897514},
    month = mar,
    title = {{TRIPOD-Code}},
    url = {https://github.com/thomas-sounack/TRIPOD-Code},
    version = {1.0.0},
    year = {2026}
}