scPEFT: Harnessing the Power of Single-Cell Large Language Models with Parameter-Efficient Fine-Tuning
This is the official repository for Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT. To reproduce the results from the paper, please visit scPEFT_reproduction.
📣 12/31/2025 UPDATE scPEFT is published in Nature Machine Intelligence
❗ 9/25/2025 UPDATE We have released the reproduction for benchmarking the linear model on perturbation prediction. Please see here for more details.
we propose scPEFT, a framework that integrates Parameter-Efficient Fine-Tuning (PEFT) techniques into scLLMs to calibrate them for specialized use cases. Unlike traditional finetuning, which modifies the entire model, scPEFT employs low-dimensional, learnable, and pluggable adapters to customize scLLMs in a separate, reduced-dimensional subspace. The critical role of these proxy adapters is to estimate a ‘model delta’ (standing for changes in some model parameters) for context alignment under the guidance of task-specific objective functions and limited custom data. During the adaptation process, the original scLLM parameters are frozen to preserve pre-learned biological knowledge, while only the smaller adapter parameters are updated. This design reduces the complexity of domain adaptation, enabling higher performance with fewer resources than traditional finetuning strategies of scLLMs in out-of-context scenarios.
scPEFT works with Python >= 3.7.13. scPEFT is available on PyPI. To install scPEFT, run the following command:
pip install scpeftFor developing, run the following command:
git clone https://github.com/coffee19850519/scPEFT
cd scPEFT
Note: scPEFT is currently built on top of scGPT, scBERT, scFoundation and Geneformer. Please follow their installation instructions to ensure all necessary versioned dependencies are installed. We provide a requirements. ymal file for the environment in which scPEFT was developed.
-
Download the backbone model, e.g., scGPT model checkpoint and place it at e.g.,
work_dir/scPEFT/save. We recommend using the whole-human model for most applications by default, which pretrained on 33 million normal human cells. -
The tutorials of scPEFT for downstream tasks in tutorial_peft. Here are the links to the downstream tasks and tutorials mentioned in our article
| Backbone | Downstream task | Link |
|---|---|---|
| scGPT | cell type identification | Tutorial_Identification.ipynb |
| scGPT | batch correction | Tutorial_BatchCorrection.ipynb |
| scGPT | perturbation | Tutorial_Perturbation.ipynb |
| scGPT | cell population discovery | Tutorial_CellPopulationDiscovery.ipynb |
| scGPT | marker gene detection | Tutorial_MarkerGeneDetection.ipynb |
| scFoundation | cell type identification | Tutorial_identification_scFoundation.ipynb |
| scFoundation | perturbation | Tutorial_Perturbation_scFoundation.ipynb |
| Geneformer | cell type identification | Tutorial_identification_Geneformer.ipynb |
| scBERT | cell type identification | Tutorial_identification_scBERT.ipynb |
- Publish to pypi
- Adapting scPEFT for native-attention
- Release scripts for more downstream tasks
- Release scripts for more scLLM backbones
- Adapting scPEFT for flash-attention
- Only retain PEFT-related parameters when saving peft-model weights.
We greatly welcome contributions to scPEFT. Please submit a pull request if you have any ideas or bug fixes. We also welcome any issues you encounter while using scPEFT.
We sincerely thank the authors of following open-source projects:
@article {
author = {Fei He, Ruixin Fei, Jordan E. Krull, Yang Yu, Xinyu Zhang, Xianyu Wang, Hao Cheng, Mingyue Gao, Li Su, Yibo Chen, Jinpu Li, Baichuan Jin, Yuzhou Chang, Anjun Ma, Qin Ma & Dong Xu},
title = {Harnessing the Power of Single-Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT},
year = {2025},
doi = {https://doi.org/10.1038/s42256-025-01170-z},
publisher = {Springer Nature},
URL = {https://www.nature.com/articles/s42256-025-01170-z},
journal = {Nature Machine Intelligence}
}