MoE-Pipe

MoE-Pipe: A Pipelined MoE Model Loading Framework for Reducing the Cold Start Delays in Serverless Inference
Accepted at CLOSER 2026

MoE-Pipe is a framework designed to significantly reduce the cold-start latency of large-scale Mixture-of-Experts (MoE) models in serverless environments.

By leveraging the inherent sparse activation property of MoE models, MoE-Pipe introduces a multi-level pipelining methodology to overlap the loading of expert weights with ongoing model computation. Instead of a sequential load-then-compute process, our approach orchestrates CPU-GPU, inter-GPU, and intra-GPU overlapping, successfully hiding approximately 65% of the loading time and achieving up to 1.85× improvement in end-to-end latency.

Key Features

• CPU Pipelining: Overlaps disk-to-host memory loading with host-to-device (H2D) transfers.
• Inter-GPU Pipelining: Preemptively schedules H2D transfers across multiple GPUs to hide handoff latency.
• Intra-GPU Pipelining: Decouples computing and loading jobs into asynchronous CUDA streams, managed by an event-driven Expert Manager.

---

📥 Model Weight Preparation

We use Mixtral-8x7B-Instruct-v0.1 for our evaluation. Please download the weights using the Hugging Face CLI before running the experiments:

huggingface-cli download mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --local-dir ./weights/Mixtral-Instruct-8x7B-v0.1 \
  --local-dir-use-symlinks False

---

🛠️ Environment Setup

We evaluate MoE-Pipe on two distinct hardware environments. Please choose the setup that matches your hardware configuration.

1. Volta Node (e.g., 4× NVIDIA V100 16GB)

Due to GPU memory constraints on older architectures, this environment utilizes 4-bit NF4 double quantization with bfloat16 computation.

• NVIDIA Driver: 550.54.15
• CUDA: 12.4
• Nsight Compute: 2025.2.1

Setup via Conda:

conda env create --name moellm -f volta_environment.yml
conda activate moellm

2. RTX Node (e.g., 4× NVIDIA RTX 5090 32GB)

A higher-performance node with sufficient VRAM. Experiments on this node run without quantization.

• NVIDIA Driver: 580.65.06
• CUDA: 13.0
• Nsight Compute: 2025.2.1

Setup via Conda (using provided yml):

conda env create --name moellm_rtx -f rtx_environment.yml
conda activate moellm_rtx

Alternative: Setup from scratch (Click to expand)

conda create --name moellm_rtx python=3.10
conda activate moellm_rtx
conda install nvidia/label/cuda-12.9.1::cuda-toolkit
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu129
pip install transformers==4.51.3 bitsandbytes==0.46.1 accelerate==1.8.1 nvtx
conda install -c nvidia cuda-toolkit

# Reactivate
conda deactivate
conda activate moellm_rtx

---

🚀 How to Run

Volta Node (Quantized)

To run the evaluation with 4-bit quantization on the Volta environment:

conda activate moellm
python main.py

RTX Node (Non-Quantized)

To run the evaluation without quantization (nq_* scripts are intended for GPUs with sufficient VRAM, like the RTX 5090):

conda activate moellm_rtx
python nq_main.py

---

🐳 Docker Deployment

To ensure precise and reproducible measurements, MoE-Pipe can be executed using a lightweight, container-based emulation of a serverless environment.

Example: Running on the Volta Node

# Make sure you are in the root directory of the repository
sudo docker build -t serverless-moe-app -f docker/volta/Dockerfile .

sudo docker run --gpus all --name moe-test --ipc=host \
  -v $(pwd)/weights/Mixtral-Instruct-8x7B-v0.1:/app/weights/Mixtral-Instruct-8x7B-v0.1:ro \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  serverless-moe-app

---

📖 Citation

If you find this code or our paper useful in your research, please cite our work:

@inproceedings{wu2026moepipe,
  title={MoE-Pipe: A Pipelined MoE Model Loading Framework for Reducing the Cold Start Delays in Serverless Inference},
  author={Wu, Zhan-Wei and Tsai, Chih-Tai and Lin, Sao-Hsuan and Ke, Yi-Syuan and Chou, Jerry},
  booktitle={Proceedings of the International Conference on Cloud Computing and Services Science (CLOSER)},
  year={2026}
}