[Not for merge] Contrastive Language-Speech Pre-training (CLSP) Recipe by yfyeung · Pull Request #2053 · k2-fsa/icefall

yfyeung · 2025-12-30T16:32:08Z

Contrastive Language-Speech Pre-training
Multi-Positive Contrastive Language-Speech Pre-training
Speech–Text Retrieval
Zero-shot Paralinguistic Classification
Speech Style Similarity Scoring
Release the checkpoints: https://huggingface.co/yfyeung/CLSP
Release the dataset: https://huggingface.co/datasets/yfyeung/FCaps
Release the paper: https://arxiv.org/pdf/2601.03065

update soft links

update update update update name update fix update fix multinode multigpu update try to support evaluate update update update update update update update update update fix update update fix fix update update update update update fix fix fix fix fix update update remove barrier to avoid stuck

update update

update update update update update update update update fix update update update update

update update update update update

update update

update Update generate_paraspeechcaps_manifests.py

add attach_caption update update update update

fix update add filter long captions update update fix multipositive clap loss update

update

Expand NO_PATTERN regex to include more terms Optimize long captions selection in finetune_stage2.py update

Redo typo in validation set name update update update update update fix update update update update update update update update update update fix fix fix update fix update update update update

update update update update update

update

support zero-shot classification update update fix

coderabbitai · 2026-01-20T13:44:59Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

🔍 Trigger a full review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

severusyang and others added 22 commits December 3, 2025 12:45

init

7ac6fdd

update soft links

fix utils.py

3f6d6f7

update update

update

e89f669

support strict fixed batch size

2d3d7c8

update update update update update update update update fix update update update update

unwrap model for rank0-only evaluation to avoid NCCL hang

e64bb85

use while-loop for strict batch sync, remove sampler state_dict

5a86b74

use emilia webdataset and split out test_fixedbucketbatchsizeconstraint

1d999df

update update update update update

add paraspeechcaps manifests and support evaluate retrieval

3815692

update update

use short caption from paraspeechcaps

edb01f3

update Update generate_paraspeechcaps_manifests.py

add normalization for paraspeechcaps

a168cfb

add attach_caption

cdd2643

add attach_caption update update update update

support finetune stage 1 and stage 2

0c3cec8

add iemocap manifests

00bcc51

support multi-positive labels

29cf090

fix update add filter long captions update update fix multipositive clap loss update

support export retrieval details

7b2f70e

update

fix

60da35a

Expand NO_PATTERN regex to include more terms Optimize long captions selection in finetune_stage2.py update

add finetune stage3

3c578e5

Redo typo in validation set name update update update update update fix update update update update update update update update update update fix fix fix update fix update update update update

support zero-shot classification

7d76033

update update update update update

update finetune stage 3

c3218d3

update

add ravdess manifest

6ce8fa0

add cremad manifest

098216f

k2-fsa deleted a comment from coderabbitai bot Dec 31, 2025

yfyeung changed the title ~~S3CLAP: Advancing Fine-Grained Speech Tasks Evaluation with Contrastive Language-Speech Pre-training~~ S3CLAP: Advancing Speech Tasks Evaluation with Fine-Grained Contrastive Language-Speech Pre-training Jan 2, 2026

yfyeung changed the title ~~S3CLAP: Advancing Speech Tasks Evaluation with Fine-Grained Contrastive Language-Speech Pre-training~~ [Not for merge] S3CLAP: Advancing Speech Tasks Evaluation with Fine-Grained Contrastive Language-Speech Pre-training Jan 2, 2026

yfyeung changed the title ~~[Not for merge] S3CLAP: Advancing Speech Tasks Evaluation with Fine-Grained Contrastive Language-Speech Pre-training~~ [Not for merge] Speech-CLAP recipe Jan 5, 2026

yfyeung changed the title ~~[Not for merge] Speech-CLAP recipe~~ [Not for merge] Contrastive Language-Speech Pre-training (CLSP) Recipe Jan 5, 2026

yfyeung added 3 commits January 20, 2026 21:44

support zero-shot classification

e935b8a

support zero-shot classification update update fix

add glap evaluation recipe

740b7db

add laion_clap evaluation recipe

8d4b9a3

add paraclap evaluation recipe

a7163fd

yfyeung force-pushed the clsp branch from d19feaa to a7163fd Compare January 20, 2026 13:44

add clsp recipe

ba0b41c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Not for merge] Contrastive Language-Speech Pre-training (CLSP) Recipe#2053

[Not for merge] Contrastive Language-Speech Pre-training (CLSP) Recipe#2053
yfyeung wants to merge 27 commits intok2-fsa:masterfrom
yfyeung:clsp

yfyeung commented Dec 30, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Jan 20, 2026 •

edited

Loading

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments

Conversation

yfyeung commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments

yfyeung commented Dec 30, 2025 •

edited

Loading

coderabbitai bot commented Jan 20, 2026 •

edited

Loading