Sovereign 8s.is mirror of standardgalactic/fluoresfm
  • IDL 43.7%
  • Python 36.8%
  • C++ 10.3%
  • HTML 2.7%
  • Cuda 2.5%
  • Other 3.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Qiqi Lu 772ac54b20 update
save p value source data
2026-01-29 12:18:08 +08:00
.vscode update files 2025-08-02 17:39:27 +08:00
__pycache__ update 2025-12-31 22:31:00 +08:00
datasets update 2025-05-28 18:01:30 +08:00
markdown/figures update files 2025-08-02 12:03:00 +08:00
methods update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
models update analysis code 2025-12-22 17:33:17 +08:00
others add code for classification 2025-12-17 00:58:47 +08:00
packages update files 2025-08-02 17:39:27 +08:00
utils update 2025-12-31 22:31:00 +08:00
.gitignore add code for classification 2025-12-17 00:58:47 +08:00
0_0_data_cleaning.py update files 2025-08-02 12:03:00 +08:00
0_1_count_patches.py update files 2025-08-02 12:03:00 +08:00
0_2_dispaly_patch_distribution.py update files 2025-08-02 12:03:00 +08:00
0_3_check_file_existence.py update files 2025-08-02 12:03:00 +08:00
0_4_display_patches.py update files 2025-08-02 12:03:00 +08:00
0_5_display_images_test.py update files 2025-08-02 12:03:00 +08:00
0_6_test_ultils_function.py update files 2025-08-02 12:03:00 +08:00
0_7_test_model.py UPDATE 2025-11-27 17:14:38 +08:00
1_0_text_generation.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
1_1_download_embedder.py update files 2025-08-02 12:03:00 +08:00
1_2_text_embedding.py update analysis code 2025-12-22 17:33:17 +08:00
1_3_structural_promot.py update 2025-12-31 22:31:00 +08:00
1_4_prompt_tokenize.py update 2025-11-22 15:56:06 +08:00
2_0_train_it2i.py update 2025-11-29 11:36:09 +08:00
2_1_train_i2i.py update files 2025-08-02 12:03:00 +08:00
3_0_test_it2i.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
3_1_test_i2i.py update analysis code 2025-12-22 17:33:17 +08:00
3_2_test_i2i_finetune.py update files 2025-08-02 12:03:00 +08:00
4_0_result_evaluate.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
4_1_result_evaluate_finetune.py update files 2025-08-02 12:03:00 +08:00
4_2_collect_results_all_sample.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
4_3_analysis_results_each_dataset.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
4_4_analysis_results_each_dataset_resolution.py add code for resolution analysis 2025-12-04 21:35:33 +08:00
5_0_display_metrics_each_dataset.py update files 2025-08-02 12:03:00 +08:00
5_1_display_each_sample.py update files 2025-08-02 12:03:00 +08:00
5_2_display_radar.py update 2025-12-31 22:31:00 +08:00
5_2_display_resolution_scatter.py update 2025-12-31 22:31:00 +08:00
5_3_display_voilin.py update 2026-01-29 12:18:08 +08:00
5_4_1_display_image_live.py update 2025-12-31 22:31:00 +08:00
5_4_display_image.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
5_5_display_multiple sample.py update 2025-08-22 17:33:00 +08:00
6_0_segment.py update analysis code 2025-12-22 17:33:17 +08:00
6_1_evaluate_seg.py update files 2025-08-02 12:03:00 +08:00
6_2_collect_seg_metrics.py add code for resolution analysis 2025-12-04 21:35:33 +08:00
6_2_display_mask_metrics.py add code for resolution analysis 2025-12-04 21:35:33 +08:00
6_3_display_mask_each.py update files 2025-08-02 12:03:00 +08:00
6_4_display_mask.py update files 2025-08-02 12:03:00 +08:00
7_0_display_finetune_metrics.py update 2026-01-29 12:18:08 +08:00
7_1_display_finetune_training_curve.py update files 2025-08-02 12:03:00 +08:00
7_2_display_finetune_num_sample.py update files 2025-08-02 12:03:00 +08:00
7_3_display_finetune_3d.py update fine-tune code 2025-12-06 15:18:06 +08:00
7_4_display_finetune_proj.py update fine-tune code 2025-12-06 15:18:06 +08:00
7_5_display_finetune_iso_wo_gt.py update fine-tune code 2025-12-06 15:18:06 +08:00
7_6_display_finetune_srx.py update analysis code 2025-12-22 17:33:17 +08:00
8_0_collect_datasets_info.py add code for classification 2025-12-17 00:58:47 +08:00
9_0_background_subtraction.py update files 2025-08-02 12:03:00 +08:00
9_1_image_denoising.py update files 2025-08-02 12:03:00 +08:00
11_0_classification_dataset_construct.py add code for classification 2025-12-17 00:58:47 +08:00
11_1_classification_manual.py add code for classification 2025-12-17 00:58:47 +08:00
11_2_classification_dataset_embedding.py add code for classification 2025-12-17 00:58:47 +08:00
11_3_classification_predict.py add code for classification 2025-12-17 00:58:47 +08:00
11_4_display_classification_data.py update 2026-01-29 12:18:08 +08:00
11_5_display_classification_results.py add code for classification 2025-12-17 00:58:47 +08:00
11_6_classcification_function.py update 2025-12-31 22:31:00 +08:00
12_0_cost_eva_generate_test_image.py update analysis code 2025-12-22 17:33:17 +08:00
13_0_analysis_ccp_single_sample.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
13_1_analysis_ccp_dataset.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
13_2_analysis_ER_node_single_sample.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
13_3_analysis_ER_node_dataset.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
13_3_analysis_mt_dataset.py update 2026-01-29 12:18:08 +08:00
13_3_analysis_mt_dataset_single_sample.py update 2026-01-29 12:18:08 +08:00
13_4_analysis_text_confusion_matrix.py update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
__init__.py first submit 2025-02-27 15:50:08 +08:00
bkg_subtraction.png update 2025-06-23 16:40:11 +08:00
constants.py add code for classification 2025-12-17 00:58:47 +08:00
dataset_analysis.py update files 2025-08-02 12:03:00 +08:00
dataset_test-v2.xlsx update code for morphology -aware measurements 2025-12-29 00:42:17 +08:00
dataset_train_transformer-v2.xlsx add code for classification 2025-12-17 00:58:47 +08:00
finetune_checkpoints.py update files 2025-08-02 12:03:00 +08:00
finetune_evaluation_methods.py update files 2025-08-02 12:03:00 +08:00
LICENSE Initial commit 2025-02-27 15:45:40 +08:00
README.md Update README.md 2026-01-03 01:40:44 +08:00
run.sh update files 2025-08-02 12:03:00 +08:00
run2.sh update files 2025-08-02 12:03:00 +08:00
~$dataset_test-v2.xlsx update 2025-10-14 15:02:03 +08:00

Foundation model for fluorescence microscopic image restoration (FluoResFM)

FluoResFM is a foundation model for fluorescence microscopic image restoration, which can complete three tasks, including denoising, deconvolution, and super-resolution, in a unified model. It is an all-in-one model that was trained on diverse fluorescence microscopic images in a supervised manner. The training data covering more than 20 types of biological structures, such as CCP, MT, ER, F-actin, lysosome, and so on. It can be finetuned on new datasets to improve the performance. The model can serve as a preprocessing tool for cellular and intracellular image analysis, such as cell and organelle segmentation.

napari plugin

We have developed a user-frendly napari plugin for FluoResFM (napari-fluoresfm), which can accomplish the image patching, text embedding, model training, model fine-tuning, and model prediction in a interactive interface.

This repository contains the code for data processing, model training and finetune, model evaluation, and results analysis.

DEPENDENCIES

The study was developed and tested on WSL2 with Ubuntu 24.04

  • Distributor ID: Ubuntu
  • Description: Ubuntu 24.04.1 LTS
  • Release: 24.04
  • Codename: noble

We recommend using the Linux distribution to run the code. Because the pytroch and triton packages are more stable on Linux, and less GPU memory and faster training speed on Linux compared with Windows.

The codes were only tested on python 3.11.9. Besides, the following packages are required (mainly):

  • torch : should be installed based on your CUDA version and system. \Rightarrow pytorch
  • torchvision
  • tensorboard
  • triton-windows: if you are using Windows, you need to install the triton package manually. \Rightarrow trinton-windows
  • numpy
  • pandas
  • scipy
  • matplotlib
  • seaborn
  • scikit-image
  • tqdm
  • pydicom
  • cellpose : Cellpoase-SAM was used for cell/nuclei segmentation. \Rightarrow cellpose
  • nellie : Nellie was used for intracellular organelle segmentation. \Rightarrow nellie
  • flash-attn
  • huggingface-hub : huggingface was used for download BiomedCLIP model.
  • transformers
  • open_clip_torch
  • openpyxl
  • tifffile

The text encoder from BiomedCLIP was employed as the text embedding model, the version used in this sduy can be download using the code in download_embedder.py.

MODEL ARCHITECTURE

FluoResFM is based on a text-conditioned U-Net architecture, with a low-quality raw image and its correspongding text prior information (including task type, imaging object, and imaging condition) as inputs, a restored high-quality image as output.

unet Detailed architecture of FluoResFM. FluoResFM takes the low-quality image and its corresponding textual prior information as inputs and outputs the restored high-quality image. It employs a text-conditioned U-Net as backbone, which consists of an encoder, a decoder, and skip connections. The encoder begins with a convolution layer, and the decoder ends with a convolution layer to generate the restored images. Each scale of the encoder and decoder contains a residual block and a text-image fusion block. The text prompt is projected into a text embedding using a text embedder and then injected into the cross-attention layer within the text-image fusion block. Only blocks labeled with “fire” markers are fine-finetuned during the fine-tuning stage.

A small example data and pre-trained model can be download from

  1. Baidu Yun: https://pan.baidu.com/s/16gSqctnA0EqE8zHGTL4yFg?pwd=ypcm.
  2. Google Drive: https://drive.google.com/drive/folders/1WoTo2at_4-9HGW_b7Ia1o_axYQqHqzJA?usp=drive_link

DATA COLLECTION

All the data used for training and testing are publicly accessible. The original datasets can be download from the following links:

Internal datasets for training and evaluation

Dataset Task Image Object Microscopy Link
BioSR DN, DCV, SR CCP WF, SIM Link
ER WF, SIM Link
F-actin WF, SIM Link
MT WF, SIM Link
BioSR+ DN CCP WF Link
ER WF Link
F-actin WF Link
MT WF Link
myosin-IIA WF Link
CARE DN nuclei (planaria) confocal Link
nuclei (Tribolium) multi-photon Link
SR histone (Drosophila) light-sheet Link
nuclei (retina) confocal Link
nuclear-envelope confocal Link
nuclei/membrane two-photon Link
DCV granule WF Link
tubulin WF Link
tubulin confocal Link
DeepBacs DN nucleoid confocal Link
MreB confocal Link
DCV, SR membrane (E. coli) WF, SIM Link
membrane (S. aureus) WF, SIM Link
W2S DN, DCV, SR Mito WF, SIM Link
lysosome WF, SIM Link
F-actin WF, SIM Link
SR-CACO-2 DN, SR histone confocal Link
survivin confocal Link
tubulin confocal Link
FMD DN nuclei (BPAE) confocal Link
F-actin (BPAE) confocal Link
Mito (BPAE) confocal Link
NCC (zebrafish) confocal Link
chromosome (mice) confocal Link
chromosome (mice) two-photon Link
nuclei (BPAE) two-photon Link
F-actin (BPAE) two-photon Link
Mito (BPAE) two-photon Link
nuclei (BPAE) WF Link
F-actin (BPAE) WF Link
Mito (BPAE) WF Link
VMSIM5 DCV, SR Mito WF, SIM Link
RCAN-in DCV, SR MT confocal, STED Link
NPC confocal, STED Link
DN golgi SIM Link

External datasets for evaluation

Dataset Task Image Object Microscopy Link
SIM DCV F-actin WF, SIM Link
MT WF, SIM Link
VMSIM3 DCV, SR Mito WF, SIM Link
VMSIM488 DCV microsphere WF, SIM Link
VMSIM568 DCV microsphere WF, SIM Link
VMSIM647 DCV microsphere WF, SIM Link
RCAN-ex DN F-actin iSIM Link
ER iSIM Link
lysosome iSIM Link
Mito iSIM Link
MT iSIM Link
BioTISR DN, DCV, SR CCP WF, SIM Link
F-actin WF, SIM Link
lysosome WF, SIM Link
MT WF, SIM Link
Mito WF, SIM Link

Datasets used for segmentation evaluation

Dataset Task Image Object Microscopy Link
CARE DN nuclei (planaria) confocal Link
nuclei (Tribolium) multi-photon Link
SR nuclei/membrane two-photon Link
DeepBacs DN nucleoid confocal Link
DCV membrane (E. coli) WF, SIM Link
membrane (S. aureus) WF, SIM Link
SR-CACO-2 DN, SR histone confocal Link
FMD DN nuclei (BPAE) confocal Link
F-actin (BPAE) confocal Link
Mito (BPAE) confocal Link
BioTISR SR CCP WF, SIM Link
F-actin WF, SIM Link
lysosome WF, SIM Link
HL60 DN, DCV nuclei confocal Link
Scaffold-A549 DN nuclei confocal Link
GranuSEG DN nuclei confocal Link
ColonTissue DCV nuclei confocal Link

Datasets used for other tasks

Datasets sued for 3D image restoration, surface projection from 3D volumes, isotropic reconstruction, and super-resolution with 3x, 4x, and 8x up-sampling factors.

Dataset Task Imaging object Microscopy Link
RCAN-3D 3D DCV MT confocal, STED link
CARE-proj Proj membrane confocal link
CARE-iso ISO histone light-sheet link
BioSR-3x SR (×3) F-actin WF, SIM link
TA-GAN SR (×4) synaptic protein confocal, STED link
DL-SMLM SR (×8) MT WF, STORM link

The processed datasets are available upon request from the corresponding author and can be provided via physical transfer if necessary.

DATA PROCESSING

DATA PREPARATION

images and patches

The raw and GT images are first patched into patches with a shape of [num_channel, patch_size, patch_size] and saved into different folders.

  • data_cleaning.py : exclude the patches with only background noise.
  • count_patches.py : count the number of patches in each dataset.
  • display_patch_dsitribution.py: display the distribution of patches in each task and structure.
  • check_file_existence.py: check if the patches exist in the dataset.
  • display_patches.py: display the patches in the dataset to check visually.
  • display_images_test.py: display the images in the internal and external test datasets.
  • test_utils_function.py: test the utils functions used for data preprocessing and data fold and unfold.
  • test_model.py: check the usability of the model and print the information of model.

text

Before training, our code first uses the information in the information xlsx file to generate the text and then embed the text using a text embedder.

  • text_generation.py : generate the text for each datasets using the information in xlsx file.
  • download_embedder.py : download the text embedder from huggingface.
  • text_embedding.py : embed the text using the text embedder.

data information

  • excel_datasets.py : curate the information of each dataset (e.g. dataset name, task, structure, number of samples, etc.).

MODEL TRAINING AND FINETUNE

  • dataset_analysis.py : consists of the dataset name dictionaries for internal and external datasets, finetune dataset, datasets shown in the radar plot, and daatsets for evaluating sementation performance.
  • train_it2i.py : train or finetune the FluoResFM model on the internal/externla datasets (i refers to image,t refers to text).
  • train_i2i.py : train or finetune the model without inputing the text.
  • test_it2i.py : test the FluoResFM model on internal/external datasets.
  • test_i2i.py : test the model without inputing the text.
  • test_it2i_finetune.py : test the finetuned model on sepcific dataset. Need specify the model name and checkpoint name in the finetune_checkpoints.py file.

MODEL EVALUATION

  • result_evaluate.py : evalute the results of each model on each dataset (calculate the metrics of each sample).
  • result_evaluate_finetune.py : evalute the results of each model on each dataset in the finetune stage (calculate the metrics). Need specify the dataset name and model name in the finetune_evaluation_methods.py file.
  • collect_results_all_samples.py : collect the metric value of all samples into a excel file. Each task has a single excel file.
  • analysis_results_each_dataset.py : analysis the results of each model on each dataset (calculate the mean and std of each metric)

RESULTS ANALYSIS

image restoration

  • display_metrics_each_dataset.py : display the metrics of each dataset using box plot.
  • display_each_sample.py : display the restored image (and its error) of each sample in the internal and external datasets.
  • display_images.py : display the images restored by different methods.
  • display_radar.py : display the radar plot of internal and external datasets.
  • display_voilin.py : display the voilin plot of internal and external datasets across different tasks.
  • display_image.py : display the images restored by different methods in the internal and external datasets.

image segmentation

  • segment.py : segment the images using the existed model. Cellpos-SAM to segment the cells and nuclei, and Nellie to segment the organelle.
  • evaluate_seg.py : evaluate the segmentation results using the metrics.
  • collect_seg_metrics.py : collect the segmentation metrics of each dataset. (calculate the mean and std of each metric)
  • display_mask_metrics.py : display the segmentation metrics of each dataset using scatter plot with error bars.
  • display_mask_each.py : display the segmentation results of each sample in the internal and external datasets.
  • display_mask.py : display segmentation results of selected samples in the internal and external datasets.

finetune

Only the first convolutional layer and the last convolutional layer of the backbone are fine-tuned during the fine-tuning stage.

  • display_finetune_metrics.py : display the metrics of the finetuned models. Mutlple datasets, three metrics.
  • display_finetune_training_curve.py : display the training curve of the finetuned models. Single dataset. To present the overfitting phenomenon.
  • display_finetune_num_sample.py : display the effect of numbder of samples used for finetuning.
  • dispaly_finetune_3d.py : display the results in 3D image restoration task.
  • display_finetune_proj.py : display the results in projection image restoration task.
  • display_finetune_iso_wo_gt.py : display the results in isotropic image restoration task without ground truth.
  • display_finetune_srx.py : display the results in image restoration task with various scale factors.

image classfication

The file start with classfication_ is used to implement the image retrieval-based structure type prediction directly from input images.

  • classification_dataset_construct.py : construct the dataset for image retrieval-based structure type prediction, including searching database and test images.
  • classification_manual.py : manually anotate the structure type of each image in the test dataset.
  • classification_dataset_embedding.py : embed the images using the text embedder.
  • classification_predct.py : predict the structure type of each image in the test dataset.

morphology-aware measurements

  • analysis_ccp_single_sample(dataset).py : calculate the CCP diameter in a single sample or a single dataset.
  • analysis_ER_node_single_sample(dataset).py : calculate the ER node number/degree in a single sample or a single dataset.
  • analysis_mt_single_sample(dataset).py : calculate the filament length in MT network in a single sample or a single dataset.
  • analysis_text_confusion_matrix.py : confusion matrix analysis of structure type prompt impact.

Other functions

  • substract_background.py : substract the background of the images using rolling-ball algorithm.
  • image_denoising.py : denoise image using conventional denoising algorithms, such as non-local means.
  • collect_datasets_info.py : collect the information of each dataset to generate Supplmentary Data 1 in our manuscript.

Running time

The processing time for a 256x256 image is about 0.30 seconds, 512x512 image is about 2.73 seconds, and 1024x1024 image is about 7.63 seconds on a Nvidia RTX 4090D GPU.

Warnings and Recommondations ⚠️

  1. When the target metadata is unknow, only using the task, structure, input metadata, and the target microscope type information, FluoResFM can still obtain an acceptable restoration. In such cases, we recommend selecting an existing target microscope type in the training data.
  2. To minimize restoration error as much as possible, especially on data that differ substantially from the training distribution, we recommend fine-tuning the model with one or a few paired samples from the target domain.
  3. Incorrectly specifying the structure type may result in distortion of detailed structures in the restored image, which can lead to inaccurate morphology-aware measurements. To help users recognize the correct structure type, we have implemented an image retrieval-based classifier that can automatically identify the structure type of the input image. The most probable structure type provided by the classifier can serve as a useful reference for users.

LICENSE

This project is licensed under the MIT License - see the LICENSE file for details.