Sovereign 8s.is mirror of standardgalactic/Structured-Missingness
  • Python 57.6%
  • Shell 28.9%
  • R 13.5%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-07-31 00:31:02 +02:00
evaluation evaluation files renamed to reflect the current figure structure 2026-07-31 00:31:02 +02:00
cov_matrix.csv [cov_matrix, means] Update template data 2025-04-22 08:52:14 +02:00
LICENSE Initial commit 2025-04-16 17:59:15 +02:00
LICENSE-CC0 [LICENSES] Adjustment of licenses 2025-09-15 10:39:50 +02:00
LICENSES.md [LICENSES] Adjustment of licenses 2025-09-15 10:39:50 +02:00
means.csv [cov_matrix, means] Update template data 2025-04-22 08:52:14 +02:00
README.md [LICENSES] Adjustment of licenses 2025-09-15 10:39:50 +02:00
requirements.txt requirements added 2026-07-27 00:25:18 +02:00
simulate.sh [simulate] main bash function debugged and updated 2025-04-22 08:57:50 +02:00
simulate_00_generate_dataset.py [simulate 00 generate] debugged - functioning 2025-04-22 08:55:42 +02:00
simulate_01_normalise.py [normalise] script to standardise generated data for autoencoders 2025-04-22 08:56:06 +02:00
simulate_02_AutoComplete.sh [AutoComplete] imputation script 2025-04-22 08:56:33 +02:00
simulate_02_Average.py [Average] imputation script 2025-04-22 08:56:43 +02:00
simulate_02_ExtraTrees.py [Extra Trees] Imputation script 2025-04-22 08:56:57 +02:00
simulate_02_MICE.R [MICE] imputation script 2025-04-22 08:57:10 +02:00
simulate_02_MICE_context.R [MICE hieraarchical] imputation script 2025-04-22 08:57:21 +02:00
simulate_03_eval.py evaluation of all methods script 2025-04-22 08:57:34 +02:00
utils_concat_csv.py helper functions 2025-04-22 08:58:01 +02:00

The Art of Not Knowing: Accommodating Structured Missingness in Biomedical Research

This repository contains scripts to a full pipeline for generating, normalizing, imputing, and evaluating simulated datasets with structured missingness. It supports multiple imputation methods including AutoComplete, Extra Trees (ET), and MICE (standard and context-based). The pipeline is modular and designed to run locally or on an HPC cluster using SLURM.


📦 Environment Requirements

This pipeline assumes the following environments and tools are available:

Conda Environments:

  • ML: Python-based environment containing:
    • Python ≥ 3.8
    • numpy, pandas, scikit-learn
    • pyarrow (for MICE output)
  • R_env: R-based environment with:
    • R ≥ 4.0
    • mice, arrow, and relevant tidyverse packages

Other Dependencies:

  • AutoComplete package has to be downloaded and set-up
  • SLURM job scheduler (for cluster execution)
  • Scripts referenced in code_dir:
    • simulate_00_generate_dataset.py
    • simulate_01_normalise.py
    • simulate_02_AutoComplete.sh
    • simulate_02_ExtraTrees.py
    • simulate_02_MICE.R
    • simulate_02_MICE_context.R
    • simulate_03_eval.py
    • utils_concat_csv.py

🧠 Functional Overview

1. Data Simulation

  • Function: simulate_data
  • Description: Generates synthetic datasets using various parameters (samples, overlap, noise, missingness type).
  • Script: simulate_00_generate_dataset.py

2. Data Normalization

  • Function: normalize_data
  • Description: Normalizes simulated datasets for consistent preprocessing.
  • Script: simulate_01_normalise.py

3. AutoComplete Imputation

  • Function: Autocomplete_run
  • Description: Applies the AutoComplete model to impute missing data.
  • Script: simulate_02_AutoComplete.sh

4. AutoComplete Evaluation

  • Function: Autocomplete_evaluate
  • Description: Evaluates AutoComplete imputations against ground truth.
  • Script: simulate_03_eval.py

5. Extra Trees Imputation

  • Function: ET_run
  • Description: Uses ExtraTreesRegressor to impute missing data. Can run locally or via SLURM.
  • Script: simulate_02_ExtraTrees.py

6. Extra Trees Evaluation

  • Function: ET_evaluate
  • Description: Evaluates imputed data from Extra Trees.
  • Script: simulate_03_eval.py

7. MICE Standard Imputation

  • Function: MICE_standard_run
  • Description: Applies standard MICE (Multiple Imputation by Chained Equations).
  • Script: simulate_02_MICE.R

8. MICE Contextual Imputation

  • Function: MICE_context_run
  • Description: Applies a context-aware variant of MICE.
  • Script: simulate_02_MICE_context.R

9. MICE Evaluation

  • Function: MICE_evaluate
  • Description: Evaluates MICE imputations.
  • Script: simulate_03_eval.py

🛠️ Usage

./simulate.sh run_simulations /where/to/save/resluts
./simulate.sh normalize_data /where/to/save/resluts
./simulate.sh Autocomplete_run /where/to/save/resluts 0 1 /path/to/AutoComplete
./simulate.sh Autocomplete_evaluate /where/to/save/resluts 0 1 
./simulate.sh ET_run /where/to/save/resluts 0 1 
./simulate.sh ET_evaluate /where/to/save/resluts 0 1 
./simulate.sh MICE_standard_run /where/to/save/resluts 0 1 /path/to/MICE_libs
./simulate.sh MICE_context_run /where/to/save/resluts 0 1 /path/to/MICE_libs
./simulate.sh MICE_evaluate /where/to/save/resluts 0 1

Replace 0 (run locally) with 1 to submit jobs to SLURM where supported.
Ensure you're running the scripts from the repository folder.


📁 File Naming and Structure

Generated files follow a naming convention like:

samples1000_overlap0.5_noise0.1_noise-blocks.csv
samples1000_overlap0.5_noise0.1_noise-blocks_norm.csv
samples1000_overlap0.5_noise0.1_noise-blocks_blocks.csv
samples1000_overlap0.5_noise0.1_noise-blocks_complete.csv

Output folders:

  • normalized/
  • AC_results/
  • ET_results/
  • MICE_standard_results/
  • MICE_context_results/

License

This project uses dual licensing:

  • Code: GPL-3.0 (see LICENSE)
  • Data files (cov_matrix.csv, means.csv): CC0-1.0 (see LICENSE-CC0)

For complete licensing information, see LICENSES.md.