- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| 1.CLAM_patch_subtyping | ||
| 2.WSI_subtyping | ||
| 3.CLAM_patch_prognosis | ||
| 4.WSI_prognosis | ||
| 5.CRC100k_patch_multi_class | ||
| 6.WSI_multi_label_prediction | ||
| 7.Patch_multiplex_protein_quantification | ||
| 8.Patch_spatial_gene_prediction | ||
| 9.WSI_bulk_gene_prediction | ||
| .gitignore | ||
| LICENSE | ||
| README.md | ||
Meta-Encoder: A Unified Aggregation Framework for Multiple Pathological Foundation Models in Cancer Detection
Patch-level pathological foundation models
-
CHIEF is pretrained on 60,530 WSIs spanning 19 anatomical sites from 14 cohorts[1].
-
GigaPath is pretrained on 171,189 WSIs across 31 major tissue types from 28 cancer centers[2].
-
UNI is pretrained on 100,426 WSIs covering 20 major tissue types from two hospitals and the Genotype-Tissue Expression (GTEx) consortium[3].
WSI-level pathological foundation models
-
TITAN is a multimodal foundation model trained on 335,645 WSIs using visual self-supervised learning and vision-language alignment with pathology reports, along with 423,122 synthetic captions generated by a multimodal AI copilot[4].
-
PRISM is pretrained on 587,000 WSIs with 195,000 clinical text reports[5].
I. CLAM-based patch-level features fusion for tumor subtyping
In ./1.CLAM_patch_subtyping directory
Data structure
1.CLAM_patch_subtyping
└───CLAM_feature
│ └───BRCA
│ │ FEATURES_CHIEF_gigapath_UNI
│ │ FEATURES_DIRECTORY_CHIEF
│ │ FEATURES_DIRECTORY_gigapath
│ │ FEATURES_DIRECTORY_UNI
│
│ └───NSCLC
│ │ FEATURES_CHIEF_gigapath_UNI
│ │ FEATURES_DIRECTORY_CHIEF
│ │ FEATURES_DIRECTORY_gigapath
│ │ FEATURES_DIRECTORY_UNI
│
│ └───subtype_list
│ │ tcga_brca
│ │p25
│ │p100
│ │ tcga_lung
│ │p25
│ │p100
│ │ tcga_brca_subset.csv
│ │ tcga_lung_subset.csv
│
└───single_FM
│
└───concat_FM
│
└───self_attention_FM
For tumor subtyping prediction on WSIs from the TCGA dataset, we utilized three patch-level foundation models to extract patch features, followed by downstream fine-tuning using the CLAM framework[6].
The WSI features of TCGA-BRCA and NSCLC extracted by these three foundation models are stored in folders FEATURES_DIRECTORY_CHIEF, FEATURES_DIRECTORY_gigapath, and FEATURES_DIRECTORY_UNI under directories ./CLAM_feature/BRCA and ./CLAM_feature/NSCLC, respectively. We run the following script to concatenate the features from these three models and save them in the FEATURES_CHIEF_gigapath_UNI folder.
python BRCA_feature_integration.py
python NSCLC_feature_integration.py
For downstream training with individual foundation model features, we input the extracted CHIEF, GigaPath, and UNI feature vectors into the CLAM framework. we randomly split the data 10 times, with 10% as the validation set, 10% as the test set, and the remaining 80% as the training set.
Single foundation model
In ./1.CLAM_patch_subtyping/single_FM directory
- CHIEF
- TCGA-BRCA (100% data)
python main_BRCA_subtype.py --drop_out --early_stopping \
--k 10 --k_start 0 --k_end 10 \
--label_frac 1 --exp_code BRCA_CHIEF_train_100 \
--split_dir "../CLAM_feature/subtype_list/tcga_brca/p100" \
--weighted_sample --bag_loss ce \
--inst_loss svm --task task_2_tumor_subtyping --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../CLAM_feature/BRCA/FEATURES_DIRECTORY_CHIEF" \
--seed 123 --lr 2e-4 \
--max_epochs 20 \
--patience 10 \
--encoding_size 768 \
--csv_path "../CLAM_feature/subtype_list/tcga_brca_subset.csv"
python eval_BRCA_subtype.py --drop_out \
--k 10 \
--save_exp_code BRCA_CHIEF_eval_100 --models_exp_code BRCA_CHIEF_train_100_s123 \
--splits_dir "../CLAM_feature/subtype_list/tcga_brca/p100" \
--task task_2_tumor_subtyping --model_type clam_sb \
--data_root_dir "../CLAM_feature/BRCA/FEATURES_DIRECTORY_CHIEF" \
--encoding_size 768 \
--csv_path "../CLAM_feature/subtype_list/tcga_brca_subset.csv"
- TCGA-NSCLC (100% data)
python main_NSCLC_subtype.py --drop_out --early_stopping \
--k 10 --k_start 0 --k_end 10 \
--label_frac 1 --exp_code NSCLC_CHIEF_train_100 \
--split_dir "../CLAM_feature/subtype_list/tcga_lung/p100" \
--weighted_sample --bag_loss ce \
--inst_loss svm --task task_2_tumor_subtyping --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../CLAM_feature/NSCLC/FEATURES_DIRECTORY_CHIEF" \
--seed 123 --lr 2e-4 \
--max_epochs 20 \
--patience 10 \
--encoding_size 768 \
--csv_path "../CLAM_feature/subtype_list/tcga_lung_subset.csv"
python eval_NSCLC_subtype.py --drop_out \
--k 10 \
--save_exp_code NSCLC_CHIEF_eval_100 --models_exp_code NSCLC_CHIEF_train_100_s123 \
--splits_dir "../CLAM_feature/subtype_list/tcga_lung/p100" \
--task task_2_tumor_subtyping --model_type clam_sb \
--data_root_dir "../CLAM_feature/NSCLC/FEATURES_DIRECTORY_CHIEF" \
--encoding_size 768 \
--csv_path "../CLAM_feature/subtype_list/tcga_lung_subset.csv"
To simulate a more data-limited scenario, the dataset was downsampled to 25%, followed by an 8:1:1 split. In this scenario, simply adjust the --splits_dir parameter in the above script from p100 to p25.
For GigaPath and UNI, we only need to adjust the --data_root_dir in the above script to point to the respective folders containing the features extracted by each model.
Concatenation
In ./1.CLAM_patch_subtyping/concat_FM directory
- TCGA-BRCA (100% data)
python main_multi_BRCA_subtype.py --drop_out --early_stopping \
--k 10 --k_start 0 --k_end 10 \
--label_frac 1 --exp_code multi_BRCA_CGU_100 \
--split_dir "../CLAM_feature/subtype_list/tcga_brca/p100" \
--weighted_sample --bag_loss ce \
--inst_loss svm --task task_2_tumor_subtyping --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../CLAM_feature/BRCA/FEATURES_CHIEF_gigapath_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 20 \
--patience 10 \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_brca_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
python eval_multi_BRCA_subtype.py --drop_out \
--k 10 \
--save_exp_code multi_BRCA_CGU_100_eval --models_exp_code multi_BRCA_CGU_100_s123 \
--splits_dir "../CLAM_feature/subtype_list/tcga_brca/p100" \
--task task_2_tumor_subtyping --model_type clam_sb \
--data_root_dir "../CLAM_feature/BRCA/FEATURES_CHIEF_gigapath_UNI" \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_brca_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
- TCGA-NSCLC (100% data)
python main_multi_NSCLC_subtype.py --drop_out --early_stopping \
--k 10 --k_start 0 --k_end 10 \
--label_frac 1 --exp_code multi_NSCLC_CGU_100 \
--split_dir "../CLAM_feature/subtype_list/tcga_lung/p100" \
--weighted_sample --bag_loss ce \
--inst_loss svm --task task_2_tumor_subtyping --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../CLAM_feature/NSCLC/FEATURES_CHIEF_gigapath_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 20 \
--patience 10 \
--encoding_size 3328 \
--csv_path "..CLAM_feature/subtype_list/tcga_lung_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
python eval_multi_NSCLC_subtype.py --drop_out \
--k 10 \
--save_exp_code multi_NSCLC_CGU_100_eval --models_exp_code multi_NSCLC_CGU_100_s123 \
--splits_dir "../CLAM_feature/subtype_list/tcga_lung/p100" \
--task task_2_tumor_subtyping --model_type clam_sb \
--data_root_dir "../CLAM_feature/NSCLC/FEATURES_CHIEF_gigapath_UNI" \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_lung_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
Self-attention
In ./1.CLAM_patch_subtyping/self_attention_FM directory
- TCGA-BRCA (100% data)
python main_ca_BRCA_subtype.py --drop_out --early_stopping \
--k 10 --k_start 0 --k_end 10 \
--label_frac 1 --exp_code ca_BRCA_CGU_100 \
--split_dir "../CLAM_feature/subtype_list/tcga_brca/p100" \
--weighted_sample --bag_loss ce \
--inst_loss svm --task task_2_tumor_subtyping --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../CLAM_feature/BRCA/FEATURES_CHIEF_gigapath_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 20 \
--patience 10 \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_brca_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
python eval_ca_BRCA_subtype.py --drop_out \
--k 10 \
--save_exp_code ca_BRCA_CGU_100_eval --models_exp_code ca_BRCA_CGU_100_s123 \
--splits_dir "../CLAM_feature/subtype_list/tcga_brca/p100" \
--task task_2_tumor_subtyping --model_type clam_sb \
--data_root_dir "../CLAM_feature/BRCA/FEATURES_CHIEF_gigapath_UNI" \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_brca_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
- TCGA-NSCLC (100% data)
python main_ca_NSCLC_subtype.py --drop_out --early_stopping \
--k 10 --k_start 0 --k_end 10 \
--label_frac 1 --exp_code ca_NSCLC_CGU_100 \
--split_dir "../CLAM_feature/subtype_list/tcga_lung/p100" \
--weighted_sample --bag_loss ce \
--inst_loss svm --task task_2_tumor_subtyping --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../CLAM_feature/NSCLC/FEATURES_CHIEF_gigapath_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 20 \
--patience 10 \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_lung_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
python eval_ca_NSCLC_subtype.py --drop_out \
--k 10 \
--save_exp_code ca_NSCLC_CGU_100_eval --models_exp_code ca_NSCLC_CGU_100_s123 \
--splits_dir "../CLAM_feature/subtype_list/tcga_lung/p25" \
--task task_2_tumor_subtyping --model_type clam_sb \
--data_root_dir "../CLAM_feature/NSCLC/FEATURES_CHIEF_gigapath_UNI" \
--encoding_size 3328 \
--csv_path "../CLAM_feature/subtype_list/tcga_lung_subset.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
II. WSI-level features fusion for tumor subtyping
In ./2.WSI_subtyping directory
Data structure
2.WSI_subtyping
└───embedding_features
│ └───BRCA
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
│ └───NSCLC
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
└───subtyping_label
│ ...
For foundation models capable of processing entire WSIs, we directly obtain WSI-level feature representations for downstream fine-tuning. Model performance is evaluated via five-fold cross-validation, repeated 10 times with different random splits. The dataset splitting results are saved in ./2.WSI_subtyping/subtyping_label directory.
WSI features are extracted using TITAN and PRISM, yielding feature dimensions of 768 and 1280. The extracted features for TCGA-BRCA and TCGA-NSCLC are stored in the FEATURES_DIRECTORY_PRISM_WSI_features and FEATURES_DIRECTORY_TITAN_WSI_features folders under ./2.WSI_subtyping/embedding_features/BRCA and ./2.WSI_subtyping/embedding_features/NSCLC, respectively. Each WSI's features are saved as a separate .pt file.
Single foundation model
- TITAN
- TCGA-BRCA
python TCGA_subtyping_single_main.py \
--feature_type 'TITAN' \
--cancer_type 'BRCA' \
--batch_size 128 \
--epochs_num 100 \
--n_cycles 1 \
--lr 5e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'v1' \
--alpha_values 0.8
- TCGA-NSCLC
python TCGA_subtyping_single_main.py \
--feature_type 'TITAN' \
--cancer_type 'NSCLC' \
--batch_size 128 \
--epochs_num 100 \
--n_cycles 1 \
--lr 5e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'v1' \
--alpha_values 0.5
For PRISM model, we only need to adjust the --feature_type to 'PRISM', and all other parameters are kept the same.
Concatenation
- TCGA-BRCA
python TCGA_subtyping_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'BRCA' \
--batch_size 128 \
--epochs_num 100 \
--n_cycles 1 \
--lr 5e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'v1' \
--alpha_values 0.8
- TCGA-NSCLC
python TCGA_subtyping_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'NSCLC' \
--batch_size 128 \
--epochs_num 100 \
--n_cycles 1 \
--lr 5e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'v1' \
--alpha_values 0.5
Self-attention
- TCGA-BRCA
python TCGA_subtyping_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'BRCA' \
--batch_size 128 \
--epochs_num 100 \
--n_cycles 1 \
--lr 5e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'v1' \
--alpha_values 0.8 \
--embed_dim 1024 \
--num_heads 8
- TCGA-NSCLC
python TCGA_subtyping_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'NSCLC' \
--batch_size 128 \
--epochs_num 100 \
--n_cycles 1 \
--lr 5e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'v1' \
--alpha_values 0.5 \
--embed_dim 1024 \
--num_heads 8
III. CLAM-based patch-level features fusion for cancer prognosis
In ./3.CLAM_patch_prognosis directory
Data structure
3.CLAM_patch_prognosis
└───splits_csv_file
│
└───splits_json_file
│
└───surv_concat
│
└───surv_self_attention
│
└───surv_single
│ ...
For cancer prognosis prediction on WSIs from the TCGA dataset, we utilized three patch-level foundation models to extract patch features, followed by downstream fine-tuning using the CLAM framework, which was specifically adapted for prognosis prediction tasks.
The survival data from the TCGA BRCA cohort was evaluated using a three-fold cross-validation scheme. For each split, the model was trained on two folds and tested on the held-out fold. Stratification was performed to ensure consistent censoring rates across all folds, preserving the distribution of survival data. To evaluate model robustness, this entire three-fold cross-validation procedure was repeated across ten different random data partitions.
python surv_data_CV_splits.csv
Single foundation model
In ./3.CLAM_patch_prognosis/surv_single directory
- CHIEF
python main_BRCA_surv.py --drop_out \
--exp_code BRCA_CHIEF_train \
--split_dir "../splits_csv_file" \
--bag_loss ce \
--inst_loss svm --task task_3_tumor_survival --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../../1.CLAM_patch_subtyping/CLAM_feature/BRCA/FEATURES_DIRECTORY_CHIEF" \
--seed 123 --lr 2e-4 \
--max_epochs 5 \
--encoding_size 768 \
--csv_path "../BRCA_surv_label.csv"
- GigaPath
python main_BRCA_surv.py --drop_out \
--exp_code BRCA_gigapath_train \
--split_dir "../splits_csv_file" \
--bag_loss ce \
--inst_loss svm --task task_3_tumor_survival --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../../1.CLAM_patch_subtyping/CLAM_feature/BRCA/FEATURES_DIRECTORY_gigapath" \
--seed 123 --lr 2e-4 \
--max_epochs 5 \
--encoding_size 1536 \
--csv_path "../BRCA_surv_label.csv" \
- UNI
python main_BRCA_surv.py --drop_out \
--exp_code BRCA_UNI_train \
--split_dir "../splits_csv_file" \
--bag_loss ce \
--inst_loss svm --task task_3_tumor_survival --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../../1.CLAM_patch_subtyping/CLAM_feature/BRCA/FEATURES_DIRECTORY_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 5 \
--encoding_size 1024 \
--csv_path "../BRCA_surv_label.csv"
Concatenation
In ./3.CLAM_patch_prognosis/surv_concat directory
python main_BRCA_concat_surv.py --drop_out --early_stopping \
--exp_code BRCA_concat_train \
--split_dir "../splits_csv_file" \
--bag_loss ce \
--inst_loss svm --task task_3_tumor_survival --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "../../1.CLAM_patch_subtyping/CLAM_feature/BRCA/FEATURES_CHIEF_gigapath_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 5 \
--encoding_size 3328 \
--csv_path "../BRCA_surv_label.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
Self-attention
In ./3.CLAM_patch_prognosis/surv_self_attention directory
python main_BRCA_self_atten_surv.py --drop_out --early_stopping \
--exp_code BRCA_self_atten_train_v1 \
--split_dir "/data/home/scv7400/run/FM_collaboration/4.1.CLAM_surv/surv_data/splits_file" \
--bag_loss ce \
--inst_loss svm --task task_3_tumor_survival --model_type clam_sb \
--log_data --subtyping \
--data_root_dir "/data/home/scv7400/run/FM_collaboration/CLAM_feature/BRCA/FEATURES_CHIEF_gigapath_UNI" \
--seed 123 --lr 2e-4 \
--max_epochs 5 \
--encoding_size 3328 \
--csv_path "/data/home/scv7400/run/FM_collaboration/4.1.CLAM_surv/surv_data/BRCA_surv_label.csv" \
--features_list "FEATURES_DIRECTORY_CHIEF" "FEATURES_DIRECTORY_gigapath" "FEATURES_DIRECTORY_UNI"
IV. WSI-level features fusion for cancer prognosis
In ./4.WSI_prognosis directory
Single foundation model
- TITAN
python TCGA_surv_MLP_main_single.py \
--feature_type 'TITAN' \
--latent_dim 128 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'TITAN_MLP_surv'
- PRISM
python TCGA_surv_MLP_main_single.py \
--feature_type 'PRISM' \
--latent_dim 128 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'PRISM_MLP_surv' \
Concatenation
python TCGA_surv_MLP_main_concat.py \
--feature_list 'TITAN' 'PRISM' \
--latent_dim 128 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'concat_MLP_surv'
Self-attention
python TCGA_surv_MLP_main_self_attention.py \
--feature_list 'TITAN' 'PRISM' \
--latent_dim 128 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-5 \
--output_root './results' \
--version 'self_MLP_surv' \
--embed_dim 512 \
--num_heads 4
V. Multi-class classification for patches in the HE-CRC-100K dataset
In ./5.CRC100k_patch_multi_class directory
5.CRC100k_patch_multi_class
└───NCT-CRC-HE-100k
│ └───CHIEF
│ └───GigaPath
│ └───UNI
│ └───split_index
│ ...
The HE-CRC-100K dataset contains patches from 9 distinct labels. The dataset has a predefined train/test split, with the corresponding indices and encoded labels stored in ./5.CRC100k_patch_multi_class/NCT-CRC-HE-100k/split_index. Patch-level features are extracted using three patch-level foundation models, and these features are stored in the CHIEF, GigaPath, and UNI folders under ./5.CRC100k_patch_multi_class/NCT-CRC-HE-100k.
Single foundation model
To ensure robustness, each experiment is repeated ten times, and the mean and standard deviation of performance metrics are reported.
- CHIEF
python CRC100k_linear_probe_main_single.py \
--feature_type 'CHIEF' \
--rep_num 10 \
--embed_dim 768 \
--batch_size 512 \
--train_iters 10000 \
--n_cycles 5 \
--lr 1e-4 \
--min_lr 1e-6 \
--eval_interval 500 \
--output_root './results/CHIEF_rep10'
- GigaPath
python CRC100k_linear_probe_main_single.py \
--feature_type 'GigaPath' \
--rep_num 10 \
--embed_dim 1536 \
--batch_size 512 \
--train_iters 10000 \
--n_cycles 5 \
--lr 1e-4 \
--min_lr 1e-6 \
--eval_interval 500 \
--output_root './results/GigaPath_rep10'
- UNI
python CRC100k_linear_probe_main_single.py \
--feature_type 'UNI' \
--rep_num 10 \
--embed_dim 1024 \
--batch_size 512 \
--train_iters 10000 \
--n_cycles 5 \
--lr 1e-4 \
--min_lr 1e-6 \
--eval_interval 500 \
--output_root './results/UNI_rep10'
Concatenation
python CRC100k_linear_probe_main_concat.py \
--features_list 'CHIEF' 'GigaPath' 'UNI'\
--rep_num 10 \
--embed_dim 3328 \
--batch_size 512 \
--train_iters 10000 \
--n_cycles 5 \
--lr 1e-4 \
--min_lr 1e-6 \
--eval_interval 500 \
--output_root './results/concat_rep10'
Self-attention
python CRC100k_linear_probe_main_cross_attention.py \
--features_list 'CHIEF' 'GigaPath' 'UNI'\
--rep_num 10 \
--embed_dim 1024 \
--num_heads 8 \
--encoding_size 3328 \
--batch_size 512 \
--train_iters 10000 \
--n_cycles 5 \
--lr 1e-4 \
--min_lr 1e-6 \
--eval_interval 500 \
--output_root './results/self_attention_rep10'
VI. Multi-Label prediction of tumor biomarkers using WSI-Level representations
In ./6.WSI_multi_label_prediction directory
Data structure
6.WSI_multi_label_prediction
└───biomarker_info
│ └───tcga_brca_mutation_info.csv
│ └───CRC_MSI_mutation_info.csv
│
└───embedding_features
│ └───BRCA
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
│ └───CRC
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
└───splits_five_fold
│ ...
All data were partitioned using five-fold cross-validation, repeated ten times to generate ten distinct data splits. The dataset division results are stored in Folder ./6.WSI_multi_label_prediction/splits_five_fold. Biomarker-related information is saved in Folder ./6.WSI_multi_label_prediction/biomarker_info.
The extracted features for TCGA-BRCA and TCGA-CRC are stored in the FEATURES_DIRECTORY_PRISM_WSI_features and FEATURES_DIRECTORY_TITAN_WSI_features folders under ./6.WSI_multi_label_prediction/embedding_features/BRCA and ./6.WSI_multi_label_prediction/embedding_features/CRC, respectively. Each WSI's features are saved as a separate .pt file.
Single foundation model
- TITAN
- TCGA-BRCA
python TCGA_mutation_single_main.py \
--feature_type 'TITAN' \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 100 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'TITAN_focal' \
--predict_genes 'TP53' 'PIK3CA' 'CDH1' \
--alpha_values 0.6 0.6 0.9
- TCGA-CRC
python TCGA_mutation_single_main.py \
--feature_type 'TITAN' \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 100 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'TITAN_focal' \
--predict_genes 'RAS' 'BRAF' 'MSI'\
--alpha_values 0.5 0.8 0.8
For PRISM model, we only need to adjust the --feature_type to 'PRISM' and the --version to corresponding name, and all other parameters are kept the same.
Concatenation
- TCGA-BRCA
python TCGA_mutation_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 100 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'concat_focal' \
--predict_genes 'TP53' 'PIK3CA' 'CDH1' \
--alpha_values 0.6 0.6 0.9
- TCGA-CRC
python TCGA_mutation_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 100 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'concat_focal' \
--predict_genes 'RAS' 'BRAF' 'MSI'\
--alpha_values 0.5 0.8 0.8
Self-attention
- TCGA-BRCA
python TCGA_mutation_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 100 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'CA_focal' \
--predict_genes 'TP53' 'PIK3CA' 'CDH1' \
--alpha_values 0.6 0.6 0.9 \
--embed_dim 1024 \
--num_heads 8
- TCGA-CRC
python TCGA_mutation_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 100 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'self_attention_focal' \
--predict_genes 'RAS' 'BRAF' 'MSI'\
--alpha_values 0.5 0.8 0.8 \
--embed_dim 1024 \
--num_heads 8
VII. Multiplex protein quantification based on patch-level histological features
Data structure
7.Patch_multiplex_protein_quantification
└───Orion-CRC
│
└───preprocessing_data
│
└───preprocessed_data
│ └───h5_files
│ └───HE_features
│
└───count_prediction_single
│
└───count_prediction_concat
│
└───count_prediction_self_attention
│
└───count_prediction_cross_attention
│
└───count_prediction_contrastive_loss
│ ...
Data download and preprocessing
In ./7.Patch_multiplex_protein_quantification/Orion-CRC directory
We further quantify multiplex proteins from histology images using the ORION-CRC dataset[7], which includes the HE-stained histological images and mIHC of the same colorectal cancer tissue slide. The objective is to predict the cell counts for 15 specific protein markers within each HE-stained image patch.
First, the preprocessed ORION-CRC dataset from a previous study[8] is downloaded. The file ORION_dataset_20x_he_norm.zipis saved in the Orion-CRC directory and subsequently extracted. Additionally, the following metadata files are obtained: slide_dataframe.csv, train_dataframe.csv, val_dataframe.csv, and test_dataframe.csv.
Within the ./7.Patch_multiplex_protein_quantification/preprocessing_data, these CSV files are merged into a single consolidated file named ORION_HE_dataframe.csv. The resulting dataset comprises 37 whole-slide images (WSIs) obtained from different colorectal cancer patients.
python csv_files_merge.py
Extracted features by CHIEF, GigaPath and UNI models are saved in the directory **./7.Patch_multiplex_protein_quantification/preprocessed_data/HE_features**:
python get_Orion_feature_CHIEF.py
python get_Orion_feature_UNI_GigaPath.py --model 'GigaPath'
python get_Orion_feature_UNI_GigaPath.py --model 'UNI'
We selected 8 WSIs to form the test set, with the remaining 29 WSIs allocated to the training set. The corresponding feature files were subsequently converted into HDF5 format for efficient storage and access, and stored in the directory ./7.Patch_multiplex_protein_quantification/preprocessed_data/h5_files.
python construct_HDF5_file.py
Single foundation model
In ./7.Patch_multiplex_protein_quantification/count_prediction_single directory
- CHIEF
python marker_poisson_count_std_main.py \
--model_name 'CHIEF' \
--encoder_name 'CHIEF' \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--epochs 100 \
--patience 10 \
--batch_size 1024
- GigaPath
python marker_poisson_count_std_main.py \
--model_name 'GigaPath' \
--encoder_name 'GigaPath' \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--epochs 100 \
--patience 10 \
--batch_size 1024
- UNI
python marker_poisson_count_std_main.py \
--model_name 'UNI' \
--encoder_name 'UNI' \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--epochs 100 \
--patience 10 \
--batch_size 1024
Concatenation
In ./7.Patch_multiplex_protein_quantification/count_prediction_concat directory
python marker_poisson_count_std_concat_main.py \
--model_name 'Concat' \
--encoder_list 'CHIEF' 'GigaPath' 'UNI' \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--epochs 100 \
--patience 10 \
--batch_size 1024
Self-attention
In ./7.Patch_multiplex_protein_quantification/count_prediction_self_attention directory
python marker_poisson_count_std_self_main.py \
--model_name 'Self_attention' \
--encoder_list 'CHIEF' 'GigaPath' 'UNI' \
--embed_dim 1024 \
--num_heads 8 \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--epochs 100 \
--patience 10 \
--batch_size 1024
Cross-attention
In ./7.Patch_multiplex_protein_quantification/count_prediction_cross_attention directory
python marker_poisson_count_std_cross_attention_main.py \
--model_name 'Cross_attention' \
--encoder_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--num_heads 8 \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--epochs 100 \
--patience 10 \
--batch_size 1024
Contrastive-loss
In ./7.Patch_multiplex_protein_quantification/count_prediction_contrastive_loss directory
python marker_poisson_count_std_CL_main.py \
--model_name 'CL' \
--encoder_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--learning_rate 2e-4 \
--weight_decay 1e-4 \
--contrastive_weight 10 \
--epochs 100 \
--patience 10 \
--batch_size 512
VIII. High-dimensional spatial gene expression prediction using patch-level features
Data structure
8.Patch_spatial_gene_prediction
└───HEST-benchmark
│ └───embedding_features
│ │ merged
│ │ CHIEF
│ │ GigaPath
│ │ UNI
│
│ └──hest-bench
│ ...
└───CRC-inhouse
│ └───embedding_features
│ │ merged
│ │ CHIEF
│ │ GigaPath
│ │ UNI
│
│ └───dataset
│ │ HEST_format
│ │ Visium_format
│ ...
└───Her2ST
│ └───embedding_features
│ │ merged
│ │ CHIEF
│ │ GigaPath
│ │ UNI
│
│ └───dataset
│ │ HEST_format
│ │ legacy_ST_format
│ ...
HEST-benchmark
In ./8.Patch_spatial_gene_prediction/HEST-benchmark directory
The HEST-benchmark defines nine tasks using data from eight human cancer types across nine organs, including eight primary and one metastatic dataset[9]. The downloaded dataset is saved in folder ./8.Patch_spatial_gene_prediction/HEST-benchmark/hest-bench. Three patch-level foundation models were used to extract features from the patches in HEST-benchmark. The extracted features are stored in folders CHIEF, GigaPath, and UNI under 8.Patch_spatial_gene_prediction/HEST-benchmark/embedding_features. Each folder contains 9 subfolders named after the respective tasks, including CCRCC, COAD, HCC, IDC, LUNG, LYMPH_IDC, PAAD, PRAD, READ, and SKCM. For each task, the information and extracted features of every ST sample are saved as individual .h5 files.
We run the following script to merge the features from three models for each sample into a single file, which is stored in 8.Patch_spatial_gene_prediction/HEST-benchmark/embedding_features/merged.
python benchmark_features_merge.py
Single foundation model
- CHIEF
python HEST_single_concat_main.py \
--feature_type 'CHIEF' \
--batch_size 512 \
--epochs_num 50 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/CHIEF_v1'
For GigaPath and UNI model, we only need to adjust the --feature_type to 'GigaPath' & 'UNI' and the --output_root to corresponding folder name, and all other parameters are kept the same.
Concatenation
python HEST_single_concat_main.py \
--feature_type 'Concat' \
--batch_size 512 \
--epochs_num 50 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/Concat_v1'
Self-attention
python HEST_self_attention_main.py \
--feature_num 3328 \
--embed_dim 1024 \
--num_heads 8 \
--batch_size 512 \
--epochs_num 50 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/self_attention_v1'
Cross-attention
python HEST_cross_attention_main.py \
--features_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--num_heads 8 \
--batch_size 512 \
--epochs_num 50 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/cross_attention_v1'
Contrastive-loss
python HEST_contrastive_loss_main.py \
--features_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--batch_size 256 \
--epochs_num 25 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--contrastive_weight 0.1 \
--output_root './results/contrastive_loss_v1'
CRC-inhouse
In ./8.Patch_spatial_gene_prediction/CRC-inhouse directory First, convert the 10x Visium ST data into HEST format, then we extract features of the spot-corresponding patch regions using different patch-level foundation models.
The Visium-format CRC-inhouse samples are stored in ./8.Patch_spatial_gene_prediction/CRC-inhouse/dataset/Visium_format, with each sample as an individual folder containing filtered_feature_bc_matrix.h5, a spatial folder, and the high-resolution histology image (HE_image/histology.tiff). Converted HEST-fortmat CRC-inhouse samples are stored in ./8.Patch_spatial_gene_prediction/CRC-inhouse/dataset/HEST_format.
python CRC_HEST_setup.py
Extracted features by CHIEF, GigaPath and UNI models are saved in ./8.Patch_spatial_gene_prediction/CRC-inhouse/embedding_features:
python get_ST_feature_CHIEF.py \
--model 'CHIEF' \
--bench_data_root './dataset/HEST_format' \
--save_h5_root './embedding_features'
python get_ST_feature_UNI_GigaPath.py \
--model 'GigaPath' \
--bench_data_root './dataset/HEST_format' \
--save_h5_root './embedding_features'
python get_ST_feature_UNI_GigaPath.py \
--model 'UNI' \
--bench_data_root './dataset/HEST_format' \
--save_h5_root './embedding_features'
We run the following script to merge the features from three models for each sample into a single file, which is stored in 8.Patch_spatial_gene_prediction/CRC-inhouse/embedding_features/merged.
python CRC_features_merge.py
Single foundation model
- CHIEF
python CRC_single_concat_main.py \
--feature_type 'CHIEF' \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/CHIEF_v1'
For GigaPath and UNI model, we only need to adjust the --feature_type to 'GigaPath' & 'UNI' and the --output_root to corresponding folder name, and all other parameters are kept the same.
Concatenation
python CRC_single_concat_main.py \
--feature_type 'Concat' \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/Concat_v1'
Self-attention
python CRC_self_attention_main.py \
--feature_num 3328 \
--embed_dim 1024 \
--num_heads 8 \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/self_attention_v1'
Cross-attention
python CRC_cross_attention_main.py \
--features_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--num_heads 8 \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/cross_attention_v1'
Contrastive-loss
python CRC_contrastive_loss_main.py \
--features_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--batch_size 256 \
--epochs_num 10 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--contrastive_weight 0.1 \
--output_root './results/contrastive_loss_v1'
Her2ST
In ./8.Patch_spatial_gene_prediction/Her2ST directory The legacy ST-format Her2ST samples are stored in ./8.Patch_spatial_gene_prediction/Her2ST/dataset/legacy_ST_format. Converted HEST-fortmat Her2ST samples are stored in ./8.Patch_spatial_gene_prediction/Her2ST/dataset/HEST_format.
python Her2ST_HEST_setup.py
Extracted features by CHIEF, GigaPath and UNI models are saved in ./8.Patch_spatial_gene_prediction/Her2ST/embedding_features:
python get_breast_ST_feature_CHIEF.py \
--model 'CHIEF' \
--bench_data_root './dataset/HEST_format' \
--save_h5_root './embedding_features'
python get_breast_ST_feature_UNI_GigaPath.py \
--model 'GigaPath' \
--bench_data_root './dataset/HEST_format' \
--save_h5_root './embedding_features'
python get_breast_ST_feature_UNI_GigaPath.py \
--model 'UNI' \
--bench_data_root './dataset/HEST_format' \
--save_h5_root './embedding_features'
We run the following script to merge the features from three models for each sample into a single file, which is stored in 8.Patch_spatial_gene_prediction/Her2ST/embedding_features/merged.
python Her2ST_features_merge.py
Single foundation model
- CHIEF
python breast_single_concat_main.py \
--feature_type 'CHIEF' \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/CHIEF_v1'
For GigaPath and UNI model, we only need to adjust the --feature_type to 'GigaPath' & 'UNI' and the --output_root to corresponding folder name, and all other parameters are kept the same.
Concatenation
python breast_single_concat_main.py \
--feature_type 'Concat' \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/Concat_v1'
Self-attention
python breast_self_attention_main.py \
--feature_num 3328 \
--embed_dim 1024 \
--num_heads 8 \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/self_attention_v1'
Cross-attention
python breast_cross_attention_main.py \
--features_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--num_heads 8 \
--batch_size 512 \
--epochs_num 20 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results/cross_attention_v1'
Contrastive-loss
python breast_contrastive_loss_main.py \
--features_list 'GigaPath' 'UNI' 'CHIEF' \
--embed_dim 1024 \
--batch_size 256 \
--epochs_num 10 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--contrastive_weight 0.5 \
--output_root './results/contrastive_loss_v1'
IX. High-dimensional bulk gene expression prediction based on WSI-level features
In ./9.WSI_bulk_gene_prediction directory
Data structure
9.WSI_bulk_gene_prediction
└───embedding_features
│ └───BRCA
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
│ └───NSCLC
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
│ └───CRC
│ │ FEATURES_DIRECTORY_PRISM_WSI_features
│ │ FEATURES_DIRECTORY_TITAN_WSI_features
│
└───splits_five_fold
│
└───target_genes_expression
│ ...
All data were partitioned using five-fold cross-validation, repeated ten times to generate ten distinct data splits. The dataset division results are stored in Folder ./9.WSI_bulk_gene_prediction/splits_five_fold. Bulk mRNA expression information is saved in Folder ./9.WSI_bulk_gene_prediction/target_genes_information.
The extracted features for TCGA-BRCA, TCGA-NSCLC, and TCGA-CRC are stored in the FEATURES_DIRECTORY_PRISM_WSI_features and FEATURES_DIRECTORY_TITAN_WSI_features folders under ./2.WSI_subtyping/embedding_features/BRCA, ./2.WSI_subtyping/embedding_features/NSCLC, and ./2.WSI_subtyping/embedding_features/CRC, respectively. Each WSI's features are saved as a separate .pt file.
Single foundation model
- TITAN
- TCGA-BRCA
python TCGA_mRNA_single_main.py \
--feature_type 'TITAN' \
--latent_dim 512 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'single_TITAN' \
--coef_weight 50
- TCGA-NSCLC
python TCGA_mRNA_single_main.py \
--feature_type 'TITAN' \
--latent_dim 512 \
--cancer_type 'NSCLC' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'single_TITAN' \
--coef_weight 50
- TCGA-CRC
python TCGA_mRNA_single_main.py \
--feature_type 'TITAN' \
--latent_dim 512 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 800 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'single_TITAN' \
--coef_weight 50
For PRISM model, we only need to adjust the --feature_type to 'PRISM' and the --version to corresponding name, and all other parameters are kept the same.
Concatenation
- TCGA-BRCA
python TCGA_mRNA_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--latent_dim 512 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'concat' \
--coef_weight 50
- TCGA-NSCLC
python TCGA_mRNA_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--latent_dim 512 \
--cancer_type 'NSCLC' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'concat' \
--coef_weight 50
- TCGA-CRC
python TCGA_mRNA_concat_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--latent_dim 512 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 800 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'concat' \
--coef_weight 50
Self-attention
- TCGA-BRCA
python TCGA_mRNA_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--latent_dim 512 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'self_attention' \
--coef_weight 50 \
--embed_dim 1024 \
--num_heads 8
- TCGA-NSCLC
python TCGA_mRNA_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--latent_dim 512 \
--cancer_type 'NSCLC' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'self_attention' \
--coef_weight 50 \
--embed_dim 1024 \
--num_heads 8
- TCGA-CRC
python TCGA_mRNA_self_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_num 2048 \
--latent_dim 512 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 800 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'self_attention' \
--coef_weight 50 \
--embed_dim 1024 \
--num_heads 8
Cross-attention
- TCGA-BRCA
python TCGA_mRNA_cross_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_size_list 768 1280 \
--latent_dim 512 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'cross_attention' \
--coef_weight 50 \
--embed_dim 1024 \
--num_heads 8
- TCGA-NSCLC
python TCGA_mRNA_cross_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_size_list 768 1280 \
--latent_dim 512 \
--cancer_type 'NSCLC' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'cross_attention' \
--coef_weight 50 \
--embed_dim 1024 \
--num_heads 8
- TCGA-CRC
python TCGA_mRNA_cross_attention_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_size_list 768 1280 \
--latent_dim 512 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 800 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'cross_attention' \
--coef_weight 50 \
--embed_dim 1024 \
--num_heads 8
Contrastive-loss
- TCGA-BRCA
python TCGA_mRNA_contrastive_loss_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_size_list 768 1280 \
--latent_dim 512 \
--cancer_type 'BRCA' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'contrastive_loss' \
--coef_weight 50 \
--common_dim 512 \
--contrastive_weight 0.01
- TCGA-NSCLC
python TCGA_mRNA_contrastive_loss_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_size_list 768 1280 \
--latent_dim 512 \
--cancer_type 'NSCLC' \
--batch_size 256 \
--epochs_num 300 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'contrastive_loss' \
--coef_weight 50 \
--common_dim 512 \
--contrastive_weight 0.01
- TCGA-CRC
python TCGA_mRNA_contrastive_loss_main.py \
--feature_types 'TITAN' 'PRISM' \
--feature_size_list 768 1280 \
--latent_dim 512 \
--cancer_type 'CRC' \
--batch_size 256 \
--epochs_num 800 \
--n_cycles 1 \
--lr 1e-4 \
--min_lr 1e-6 \
--output_root './results' \
--version 'contrastive_loss' \
--coef_weight 50 \
--common_dim 1024 \
--contrastive_weight 0.1
Reference
[1] Wang, X., et al., A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 2024. 634(8035): p. 970-978.
[2] Xu, H., et al., A whole-slide foundation model for digital pathology from real-world data. Nature, 2024. 630(8015): p. 181-188.
[3] Chen, R.J., et al., Towards a general-purpose foundation model for computational pathology. Nature Medicine, 2024. 30(3): p. 850-862.
[4] Ding, T., et al., A multimodal whole-slide foundation model for pathology. Nature Medicine, 2025: p. 1-13.
[5] Shaikovski, G., et al., Prism: A multi-modal generative foundation model for slide-level histopathology. arXiv preprint arXiv:2405.10254, 2024.
[6] Lu, M.Y., et al., Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 2021. 5(6): p. 555-570.
[7] Lin, J.-R., et al., High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers. Nature cancer, 2023. 4(7): p. 1036-1052.
[8] Balezo, Guillaume, et al., MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models. arXiv preprint arXiv:2505.10294, 2025.
[9] Jaume, G., et al., Hest-1k: A dataset for spatial transcriptomics and histology image analysis. Advances in Neural Information Processing Systems, 2024. 37: p. 53798-53833.