mixer-tts-pytorch

[Samples]

This repo contains an implementation of the Mixer-TTS model (https://arxiv.org/abs/2110.03584), a compact non-autoregressive text-to-speech model that predicts mel spectrograms from text or phoneme sequences.

Pre-trained weights are available for the LJ Speech dataset. The channel dimensions of the convolutions inside the model were chosen as 384, 128, and 80, resulting in models with 20.6M, 3.17M, and 1.74M parameters.

The pre-trained models take IPA symbols as input. Please refer to here to install phonemizer and the espeak-ng backend.

A simple patch-based discriminator was used in training to reduce oversmoothing in generated mel spectrograms.

Audio samples are available here.

Pre-trained Models

All (3) checkpoint files can be downloaded by running: python download_files.py

Dataset	dim	params	name	link
LJSpeech	80	1.74M	mixer_lj_80	link
LJSpeech	128	3.17M	mixer_lj_128	link
LJSpeech	384	20.6M	mixer_lj_384	link

Training

Single-speaker LJSpeech-style training:

python train.py --config configs/ljspeech-384.yaml

Multi-speaker training:

python train_ms.py --config configs/libritts-128.yaml

train_ms.py is the multi-speaker variant. It conditions the patch discriminator on speaker embedding vectors and includes configs/libritts-128.yaml as an example for ten speakers from LibriTTS-R.

Typical preprocessing steps are:

python phonemize_lj.py
python extract_pitch.py

Adjust the paths in the scripts/configs for your local dataset layout before running them.

Notes

The pre-trained models output the 80-channel mel spectrogram version first proposed by the HiFi-GAN vocoder.

More information about the discriminator's effect in training can be found in the manuscript Arabic TTS with FastPitch: Reproducible Baselines, Adversarial Training, and Oversmoothing Analysis (arXiv | ResearchGate).

References

The model was taken out of NVIDIA's NeMo framework in order to make it easier to modify and have fewer dependencies. An energy embedding and optional speaker and emotion embeddings have been added.

Mixer-TTS in NeMo

Mixer-TTS Paper:

@article{Tatanov2021MixerTTSNF,
  title={Mixer-TTS: Non-Autoregressive, Fast and Compact Text-to-Speech Model Conditioned on Language Model Embeddings},
  author={Oktai Tatanov and Stanislav Beliaev and Boris Ginsburg},
  journal={ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2021},
  pages={7482-7486},
}