- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| configs | ||
| data_loaders | ||
| data_prepare | ||
| Feature_extractor | ||
| Networks | ||
| Training Scripts | ||
| utils | ||
| README.md | ||
Prior knowledge-guided multimodal deep learning system for biomarker exploration and prognosis prediction of urothelial carcinoma
© This code is made available for non-commercial academic purposes. For guiding thought only.
Overview
Urothelial carcinoma (UC), encompassing lower and upper tract variants, remains a prevalent and lethal malignancy within the urinary tract. Precise UC survival risk stratification is critical for accurate personalized therapy. Here we present an interactive, interpretable and prior knowledge-guided deep learning system for biomarker exploration and prognosis prediction.
Directory Structure
- Training Scripts: Training Scripts for CTContextNet, MacroContextNet, Interactive SwinUNETR and IM-NCTNet.
- Data_process: Data preprocessing file.
- Feature_extractor: radiographic, macroscopic,textual, and microscopic feature extraction.
Pre-requisites and Environment
Our Environment
- Linux (Tested on Ubuntu 24.04)
- NVIDIA GPU (Tested on Nvidia GeForce RTX A6000)
- Python (3.12.6), PyTorch (version 2.0.0), Lifelines (version 0.27.8), NumPy (version 1.24.1),MONAI (version 1.3), Pandas (version 2.1.2), Albumentations (version 1.3.1), OpenCV (version 4.8.1), Pillow (version 9.3.0), OpenSlide (version 1.1.2), Captum (version 0.6.0), SciPy (version 1.11.3), Seaborn (version 0.13.0), Matplotlib (version 3.8.1), torch_geometric (version 2.4.0), torch-scatter (version 2.1.2), torch-sparse (version 0.6.18).
Environment Configuration
-
Create a virtual environment and install PyTorch. In the 3rd step, please select the correct Pytorch version that matches your CUDA version from https://pytorch.org/get-started/previous-versions/.
$ conda create -n env python=3.12.6 $ conda activate env $ pip install torchNote:
pip installcommand is required for Pytorch installation. -
To try out the Python code and set up environment, please activate the
envenvironment first:$ conda activate env -
For ease of use, you can just set up the environment and run the following:
$ pip install -r requirements.txt
1. CTContextNet (Rad Module)
- Model's Intended Use: To extract CT features from 3D cropped CT scans for multimodal prognosis prediction.
- Inputs:
- Pre-processed and 3D-cropped CT images.
- Format: NIFTI (
.nii.gz).
- Outputs:
- A feature vector representing the radio signature (
.ptfile).
- A feature vector representing the radio signature (
- Potential Failure Modes:
- Out-of-Domain Data: Performance may degrade on CT scans from different scanners or with different acquisition protocols.
- Compute Footprint & Latency:
- Training: Trained on a single NVIDIA GeForce RTX A6000 (48GB) GPU.
- Inference: Inference for a single case is efficient and completes within seconds on a comparable GPU.
2. MacroContextNet (Macroscopic Pathology Module)
- Model's Intended Use: To analyze tissue probability heatmaps of Whole Slide Images (WSIs) and extract macroscopic-level pathological features.
- Inputs:
- tissue probability heatmaps generated by
UCSparseNetand global knowledge probability heatmap fromCONCHvision-language model. - Format: NumPy array (
.npy).
- tissue probability heatmaps generated by
- Outputs:
- A feature vector representing macroscopic pathological information (
.ptfile).
- A feature vector representing macroscopic pathological information (
- Potential Failure Modes:
- Missing Tissue: The model may fail to extract meaningful features if critical tissue types (e.g., tumor, stroma) are absent in the WSI.
- Compute Footprint & Latency:
- Training: Trained on a single NVIDIA GeForce RTX A6000 (48GB) GPU.
- Inference: Inference time for a single case is typically within few seconds.
3. Interactive SwinUNETR (Interactive Segmentation Model)
- Model's Intended Use: To perform precise, interactive segmentation of specific regions of interest (e.g., tumor areas) in WSIs.
- Inputs:
- Nephrographic phase CT images.
- Format: NIFTI (
.nii.gz).
- Outputs:
- A binary segmentation mask for the corresponding CT images.
- Potential Failure Modes:
- Ambiguous Boundaries: Segmentation accuracy may decrease where the boundary between the tumor and surrounding tissue is indistinct.
- Unrepresentative Clicks: If user-provided points do not represent the typical appearance of the target region, segmentation results may be poor.
- Compute Footprint & Latency:
- Training: Trained on a single NVIDIA GeForce RTX A6000 (48GB) GPU.
- Inference/Interaction: Provides real-time feedback, with segmentation latency for a single patch being low enough for an interactive workflow (few seconds).
4. IM-NCTNet (Multimodal Fusion Model)
- Model's Intended Use: To integrate features from multiple modalities (radio, pathology, text) and predict patient prognosis (e.g., survival risk).
- Inputs:
- Pre-extracted feature vectors from the upstream models:
- Radio features (
.pt) - Macroscopic features (
.pt) - Microscopic features (
.pt) - Textual features (
.pt)
- Radio features (
- Pre-extracted feature vectors from the upstream models:
- Outputs:
- A risk score for patient prognosis.
- Potential Failure Modes:
- Feature Quality: The model's performance is influenced by the quality of the input features. Inaccuracies from upstream modules can affect the final prediction.
- Compute Footprint & Latency:
- Training: Trained on a single NVIDIA GeForce RTX A6000 (48GB) GPU.
- Inference: Inference requires few seconds as it operates on pre-computed feature vectors.
Data Format
WSIs and clinical information of patients are used in this project. Raw WSIs are stored as .svs, .mrxs or .tiff files. Clinical information are stored as .csv files. CT images are stored in NIFIT format (nii.gz)
Data Preparation
Generate Cropped CT image file
CT images are cropped in 3D to create the initial input files for the CTContextNet model.
$ cd ./Data_process
$ python CT_process.py
Generate local and global pathological knowledge-guided patch representation
- Create original macroscopic tissue probability heatmaps for MacroContextNet training. WSIs are first processed by UCSparseNet network to get local probability heatmaps, Global knowledge-guided patch representation: create global probability heatmaps.
$ cd ./Data_prepare
$ python UCSparseNet_inference.py
$ python inference_global_probability.py
-
To cut the empty area of combined tissue probability heatmaps and get square input for MacroContextNet training
$ cd ./Data_process $ python cut_heatmap.py
Feature_extractor
-
Subsequently, we generated macroscopic feature, textual feature, radiographic feature, and microscopic features for IM-NCTNet training, respectively.
$ cd ./Feature_extractor $ python micro_feature.py #get Uni microscopic feature $ python macro_feature.py #get macroscopic feature $ python radio_feature.py #get radiographic feature $ python text_feature.py #get textual feature
Training Scripts
In Training Scripts, the train_radio_cash.py script is used to train the radiographic module.
$ cd ./Training Scripts
$ python train_radio_cash.py # CTContextNet training scripts
In Training Scripts, the train_macro_cash.py script is used to train the macroscopic module.
$ cd ./Training Scripts
$ python train_macro_cash.py # MacroContextNet training scripts
In Training Scripts, the train_IM_NCTNet.py script is used to train the IM-NCTNet multimodal model.
$ cd ./Training Scripts
$ python train_IM_NCTNet.py # IM_NCTNet training scripts
In Training Scripts, the train_seg.py script is used to train the interactive Swin-UNETR model;the geo_refine_process.py script is used to refine the segmentation results
$ cd ./Training Scripts
$ python train_seg.py -c configs/config_rnet.json
Data Distribution
DATA_ROOT/
└──DATASET/
├── clinical_information + + +
├── train.csv +
├── valid.csv +
└── ... +
├── WSI_data +
├── train +
├── slide_1.svs +
├── slide_2.svs Source WSI file
└── ... +
├──valid +
├── slide_1.svs +
├── slide_2.svs +
└── ... +
└── ... + + +
├── macro_file + + +
├── train +
├── slide_1.npy +
├── slide_2.npy +
└── ... +
├── valid +
├── slide_1.npy +
├── slide_2.npy +
└── ... +
└── ... +
└── CT_file + + +
├── train +
├── slide_1.nii.gz +
├── slide_2.nii.gz +
└── ... +
├── valid +
├── slide_1.nii.gz +
├── slide_2.nii.gz +
└── ... +
└── feature_file +
├── Micro + + +
├── slide_1.pt +
├── slide_2.pt +
└── ... +
├── macro + + +
├── slide_1.pt +
├── slide_2.pt +
└── ... +
├── radio + + +
├── slide_1.pt +
├── slide_2.pt +
└── ... +
├── text + + +
├── slide_1.pt +
├── slide_2.pt +
└── ... +
DATA_ROOT is the base directory of all datasets (e.g. the directory to your SSD or HDD). DATASET is the name of the folder containing data specific to one experiment.
Acknowledgements
- Prognosis training and test code base structure was inspired by [PathFinder] and[MCAT] .The pre-training weights have not yet been released, but reasonable requests can be made to the corresponding author to obtain them.