Training loss does not converge

[xiaohutuxian99]

Hello! I sincerely appreciate your open - source efforts.

I'm attempting to reproduce the Dasheng training, but I've run into an issue. The reconstruction loss doesn't seem to decline as the number of epochs increases. I've used the default configuration in the current code and experimented with a smaller dataset. I'm wondering if you've faced this problem before?

I'm also curious about the appropriate threshold for the reconstruction loss. At what value can we consider the training to be successfully completed?

I've tried training with the 1.2B model, and I'm uncertain whether the loss reduction is within the normal range.

Moreover, when I used the 86M model, it appears that the training didn't converge.

Could you offer some suggestions on how to address these issues?

86M model training：

1.2B model training：

[RicherMans]

Hey there!
Thanks for the efforts to using the code.
It seems that the 1.2B model training is "normal" ( somewhere 0.3-ish is good), but the 86M is not.

I'd say there are usually two common "fixes" for the problem:
If you have a smaller dataset overall, you might want to adjust the --warmup_epoch parameter and the --optimizer_args '{"lr":0.0001}' parameter.
In your case i'd set the --warmup_epochs to a smaller value, and decrease the learning rate to say 1e-5.

In our experiments we also encountered these loss problems, which we usually fix with the above mentioned method.
Let me know if it helps.

Hope that helps!
Kind regards,
Heinrich

[xiaohutuxian99]

Thank you very much for your prompt reply!

I have modified the configuration according to your suggestion and am experimenting.

[RicherMans]

Hey @xiaohutuxian99 ,
how did it work out?

[xiaohutuxian99]

Thank you for your invaluable guidance !
Using the AudioSet dataset and adjusting the learning rate schedule as recommended, I observed a significant improvement in training stability.

[xiaohutuxian99]

Additionally, the pre-trained Dasheng model only includes Encoder weights. Could you consider open-sourcing the Decoder as well?

This is my first attempt to train an Audio Encoder, and I'm particularly curious about reconstructed Mel spectrogram from Decoder during MAE training. I tried training a Decoder using the pre-trained Encoder, but observed unsatisfactory quality in the reconstructed Mel spectrograms. Could you share examples of the trained model's Decoder outputs at mask_ratio=0.75/0?

Intermediate results of training, approximately 10 epochs of outputs:
mask_ratio=0

mask_ratio=0.75

[RicherMans]

Hey,
thanks for pointing out the details about your training configuration.

So in regards to the decoder: The reconstruction is always about on the same level as you have shown. The decoder is actually not a good decoder if you want to use the model for generation tasks.
A colleague of mine did already perform these experiments to see if the decoder can actually reconstruct the input, but the results were quite average at best.
A good decoder would be to simply attach a hifigan vocoder ontop of the emebddings, that would give you some reasonable results.
We also conducted experiments plainly using the proposed dasheng-base embeddings and run denoising on embedding level with a vocoder to generate waveforms and results were great.

So in short: I currently don't plan to release the decoder because I don't think its even useful at all.

[xiaohutuxian99]

Thank you very much for your reply！

I'm currently carefully studying the Dasheng model. In fact, I hope to determine whether my training process is correct through the pre-trained Decoder. In my experiments, training stability remains an issue, but with your suggestions, it's gradually improving. Thank you again for your open-source contributions.