[Bug] full finetune wan-2.1-1.3 model failed

[liuuzexiang]

Describe the bug

Hi, I used the latest code to finetune wan-1.3b model, but the validation video quality is worse.

validation_step_1410_inference_steps_50_video_0.mp4

Reproduction

bash examples/training/finetune/wan_t2v_1.3B/crush_smol/finetune_t2v.sh

Environment

Name: torch
Version: 2.10.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page:
Author:
Author-email: PyTorch Team packages@pytorch.org
License: BSD-3-Clause
Location: /root/miniconda3/lib/python3.12/site-packages
Requires: cuda-bindings, filelock, fsspec, jinja2, networkx, nvidia-cublas-cu12, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-cufft-cu12, nvidia-cufile-cu12, nvidia-curand-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, nvidia-cusparselt-cu12, nvidia-nccl-cu12, nvidia-nvjitlink-cu12, nvidia-nvshmem-cu12, nvidia-nvtx-cu12, setuptools, sympy, triton, typing-extensions
Required-by: accelerate, diffsynth, DistVAE, fastvideo, fastvideo-kernel, flash-attn, peft, test-tube, timm, torchaudio, torchdata, torchvision, xfuser, yunchang

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

Name: fastvideo-kernel
Version: 0.2.5
Summary: Unified CUDA kernels for FastVideo
Home-page:

[jzhang38]

Could it be because you train for so long? I can get OK quality at step 100