Pytorch implementation of SoundCTM
  • Python 99.8%
  • Shell 0.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-15 18:40:36 -05:00
audioldm_eval Add files via upload 2024-06-05 11:40:02 -04:00
ckpt Add files via upload 2024-06-05 11:30:30 -04:00
clap Add files via upload 2024-06-05 11:30:30 -04:00
configs Add files via upload 2024-06-05 11:30:30 -04:00
ctm docs: restore the contact number (host IP stays purged) 2026-10-29 06:53:55 +09:00
data Add files via upload 2024-06-05 11:30:30 -04:00
src/hear21passt Add files via upload 2024-06-05 11:31:27 -04:00
tango_edm Add files via upload 2024-06-05 11:36:55 -04:00
tests fix: correct BFP-64 compression claim to 1.98x (i16) and pin it; drop PII 2026-10-27 16:34:34 +09:00
tools Add files via upload 2024-06-05 11:31:27 -04:00
ctm_inference.py Update ctm_inference.py 2024-08-29 10:47:03 -04:00
ctm_inference.sh Update ctm_inference.sh 2024-08-29 10:47:33 -04:00
ctm_train.py Add files via upload 2024-06-05 11:30:30 -04:00
ctm_train.sh Add files via upload 2024-06-05 11:30:30 -04:00
docker2singularity.sh Add files via upload 2024-06-05 11:30:30 -04:00
dockerfile Add files via upload 2024-06-05 11:30:30 -04:00
LICENSE Add files via upload 2024-06-05 19:14:32 -04:00
numerical_evaluation.py Add files via upload 2024-06-05 11:30:30 -04:00
numerical_evaluation.sh Add files via upload 2024-06-05 11:30:30 -04:00
python_accelerate.sh Add files via upload 2024-06-05 11:30:30 -04:00
README.md Update README.md 2025-03-30 21:40:12 -04:00
requirements.txt fix(deps): reconcile training dependency compatibility 2026-11-16 00:38:04 +01:00
teacher_eval.py Add files via upload 2024-06-05 11:30:30 -04:00
teacher_eval.sh Add files via upload 2024-06-05 11:30:30 -04:00

[ICLR'25] SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation

This repository is the official implementation of "SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound Generation"

Contact:

Info

  • [2025/03/30] SoundCTM-DiT is uploaded on SoundCTM-DiT (ICLR'25).
  • [2024/12/04] We're plainig to open-source codebase/checkpoints of DiT backbone with full-band text-to-sound setting and downstream tasks, as well.
  • [2024/02/10] Our paper, updated version openreview from previous version, is accepted at ICLR'25!!

Checkpoints (Current checkpoint is based on previous version.)

For inference, both AudioLDM-s-full (for VAE's decoder+Vocoder) and SoundCTM checkpoints will be used.

Prerequisites

Install docker to your own server and build docker container:

docker build -t soundctm .

Then run scripts in the container.

Training

Please see ctm_train.sh and ctm_train.py and modify folder path dependeing on your environment.

Then run bash ctm_train.sh

Inference

Please see ctm_inference.sh and ctm_inference.py and modify folder path dependeing on your environment.

Then run bash ctm_inference.sh

Numerical evaluation

Please see numerical_evaluation.sh and numerical_evaluation.py and modify folder path dependeing on your environment.

Then run bash numerical_evaluation.sh

Dataset

Follow the instructions given in the AudioCaps repository for downloading the data. Data locations are needed to be spesificied in ctm_train.sh. You can also see some examples at data/train.csv.

WandB for logging

The training code also requires a Weights & Biases account to log the training outputs and demos. Create an account and log in with:

$ wandb login

Or you can also pass an API key as an environment variable WANDB_API_KEY. (You can obtain the API key from https://wandb.ai/authorize after logging in to your account.)

$ WANDB_API_KEY="12345x6789y..."

Citation

@inproceedings{
  saito2025soundctm,
  title={Sound{CTM}: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation},
  author={Koichi Saito and Dongjun Kim and Takashi Shibuya and Chieh-Hsin Lai and Zhi Zhong and Yuhta Takida and Yuki Mitsufuji},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=KrK6zXbjfO}
}

Reference

Part of the code is borrowed from the following repos. We would like to thank the authors of these repos for their contribution.

https://github.com/sony/ctm

https://github.com/declare-lab/tango

https://github.com/haoheliu/AudioLDM

https://github.com/haoheliu/audioldm_eval