- Jupyter Notebook 72.5%
- Python 27.5%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| geneformer_peft | ||
| img | ||
| pipeline_scBERT | ||
| reproduction | ||
| scfoundation | ||
| scgpt | ||
| tutorial_peft | ||
| LICENSE | ||
| README.md | ||
| requirements.yaml | ||
scPEFT: Harnessing the Power of Single-Cell Large Language Models with Parameter-Efficient Fine-Tuning
This is the official repository for Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT. To reproduce the results from the paper, please visit scPEFT_reproduction.
❗ 9/25/2025 UPDATE We have released the reproduction for benchmarking the linear model on perturbation prediction. Please see here for more details.
🔥UPDATE: We have released several new reproductions for the revised manuscript. Please see scPEFT_reproduction for more details.
Overview
we propose scPEFT, a framework that integrates Parameter-Efficient Fine-Tuning (PEFT) techniques into scLLMs to calibrate them for specialized use cases. Unlike traditional finetuning, which modifies the entire model, scPEFT employs low-dimensional, learnable, and pluggable adapters to customize scLLMs in a separate, reduced-dimensional subspace. The critical role of these proxy adapters is to estimate a ‘model delta’ (standing for changes in some model parameters) for context alignment under the guidance of task-specific objective functions and limited custom data. During the adaptation process, the original scLLM parameters are frozen to preserve pre-learned biological knowledge, while only the smaller adapter parameters are updated. This design reduces the complexity of domain adaptation, enabling higher performance with fewer resources than traditional finetuning strategies of scLLMs in out-of-context scenarios.
Installation
scPEFT works with Python >= 3.7.13. scPEFT is available on PyPI. To install scPEFT, run the following command:
pip install scpeft
For developing, run the following command:
git clone https://github.com/coffee19850519/scPEFT
cd scPEFT
Note: scPEFT is currently built on top of scGPT, scBERT, scFoundation and Geneformer. Please follow their installation instructions to ensure all necessary versioned dependencies are installed. We provide a requirements. ymal file for the environment in which scPEFT was developed.
Get Started
-
Download the backbone model, e.g., scGPT model checkpoint and place it at e.g.,
work_dir/scPEFT/save. We recommend using the whole-human model for most applications by default, which pretrained on 33 million normal human cells. -
The tutorials of scPEFT for downstream tasks in tutorial_peft. Here are the links to the downstream tasks and tutorials mentioned in our article
| Backbone | Downstream task | Link |
|---|---|---|
| scGPT | cell type identification | Tutorial_Identification.ipynb |
| scGPT | batch correction | Tutorial_BatchCorrection.ipynb |
| scGPT | perturbation | Tutorial_Perturbation.ipynb |
| scGPT | cell population discovery | Tutorial_CellPopulationDiscovery.ipynb |
| scGPT | marker gene detection | Tutorial_MarkerGeneDetection.ipynb |
| scFoundation | cell type identification | Tutorial_identification_scFoundation.ipynb |
| scFoundation | perturbation | Tutorial_Perturbation_scFoundation.ipynb |
| Geneformer | cell type identification | Tutorial_identification_Geneformer.ipynb |
| scBERT | cell type identification | Tutorial_identification_scBERT.ipynb |
To-do-list
- Publish to pypi
- Adapting scPEFT for native-attention
- Release scripts for more downstream tasks
- Release scripts for more scLLM backbones
- Adapting scPEFT for flash-attention
- Only retain PEFT-related parameters when saving peft-model weights.
Contributing
We greatly welcome contributions to scPEFT. Please submit a pull request if you have any ideas or bug fixes. We also welcome any issues you encounter while using scPEFT.
Built With
We sincerely thank the authors of following open-source projects:
Citing scPEFT
@article {He2025.04.21.649754,
author = {He, Fei and Fei, Ruixin and Krull, Jordan E. and Zhang, Xinyu and Gao, Mingyue and Su, Li and Chen, Yibo and Yu, Yang and Li, Jinpu and Jin, Baichuan and Chang, Yuzhou and Ma, Anjun and Ma, Qin and Xu, Dong},
title = {Harnessing the Power of Single-Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT},
year = {2025},
doi = {10.1101/2025.04.21.649754},
publisher = {Cold Spring Harbor Laboratory},
URL = {https://www.biorxiv.org/content/early/2025/04/23/2025.04.21.649754},
journal = {bioRxiv}
}
