CIF-PyTorch

A PyTorch Implementation of the Continuous Integrate-and-Fire (CIF) Module for End-to-End ASR

🚀 Attention! Please refer to CIF-HieraDist for our latest and complete implementation of the CIF-based speech recognition model!

This repository provides a PyTorch implementation of the Continuous Integrate-and-Fire (CIF) module for end-to-end (E2E) automatic speech recognition (ASR), originally proposed in CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition.

📧 If you have any questions, feel free to contact me at hanminglun1996@foxmail.com.

---

Table of Contents

• 1. A Feasible Configuration for the CIF Module
• 2. Tips
• 3. Other CIF Research Work and Resources
  • a. Papers
  • b. Repositories

---

1. A Feasible Configuration for the CIF Module

encoder_embed_dim: 256          # should be the innermost dimension of inputs
produce_weight_type: "conv"
cif_threshold: 0.99
conv_cif_layer_num: 1
conv_cif_width: 3               # or 5
conv_cif_output_channels_num: 256
conv_cif_dropout: 0.1
dense_cif_units_num: 256
apply_scaling: True
apply_tail_handling: True
tail_handling_firing_threshold: 0.5
add_cif_ctxt_layers: False

---

2. Tips

1. Down-sampling. For speech recognition, we usually down-sample the input frame sequence to 1/8 of its length on the encoder side to ensure efficient training of the CIF module. For other tasks, keep the length difference between the CIF input and output within a reasonable range.
2. Length alignment. During training, when the scaled sum of the weights differs from the length of the reference transcription, truncate the reference and the model output to the same length.
3. Avoiding gradient explosion. The scaling strategy during training may cause gradient explosion, since computing the normalization scalar involves a division. Add a small value (1e-8) to the denominator to avoid this problem.

---

3. Other CIF Research Work and Resources

a. Papers

• LLM + CIF

  • X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  • Wav2Prompt: End-to-End Speech Prompt Generation and Tuning for LLM in Zero- and Few-shot Learning

• ASR

  • CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition ·
  • A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognition
  • Knowledge Transfer from Pre-trained Language Models to CIF-based Speech Recognizers via Hierarchical Distillation
  • CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition
  • CIF-RNNT: Streaming ASR via Acoustic Word Embeddings with Continuous Integrate-and-Fire and RNN-Transducers
  • CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training
  • A CIF-Based Speech Segmentation Method for Streaming E2E ASR
  • Improving CTC-Based Speech Recognition via Knowledge Transferring from Pre-Trained Language Models
  • An Efficient Text Augmentation Approach for Contextualized Mandarin Speech Recognition

• ASR Context Biasing

  • CIF-based Collaborative Decoding for End-to-End Contextual Speech Recognition
  • Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection
  • SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability
  • XCB: An Effective Contextual Biasing Approach to Bias Cross-lingual Phrases in Speech Recognition

• Low-resource Speech Recognition

  • Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition

• Non-Autoregressive ASR

  • Boundary and Context Aware Training for CIF-based Non-Autoregressive End-to-End ASR
  • A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation
  • Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
  • Paraformer-v2: An Improved Non-autoregressive Transformer for Noise-robust Speech Recognition
  • E-Paraformer: A Faster and Better Parallel Transformer for Non-autoregressive End-to-End Mandarin Speech Recognition

• Non-Autoregressive Lip Reading

  • Non-Autoregressive Lipreading Model with Integrate-and-Fire

• Speech Translation

  • UniST: Unified End-to-end Model for Streaming and Non-streaming Speech Translation
  • Exploring Continuous Integrate-and-Fire for Efficient and Adaptive Simultaneous Speech Translation
  • Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy

• Spiking Neural Networks

  • Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognition

• Multimodal ASR

  • VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition
  • X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  • CM-CIF: Cross-Modal for Unaligned Modality Fusion with Continuous Integrate-and-Fire

• Keyword Spotting

  • Leveraging Synthetic Speech for CIF-Based Customized Keyword Spotting

b. Repositories

• CIF-PyTorch — A PyTorch implementation of a standalone CIF module.
• torch_cif — A faster PyTorch implementation of CIF.
• CIF-ColDec — CIF-based Contextualization and Collaborative Decoding (ColDec).
• w2v-cif-bert — CIF as a bridge connecting pre-trained acoustic models and pre-trained language models.
• CIF-HieraDist — The official implementation of Hierarchical Knowledge Distillation (HieraDist) for CIF-based models.

---

⭐ If you find this repository helpful, please consider giving it a star.