Upstream pull mirror of https://github.com/8b-is/MuSTream.git
  • Python 81%
  • JavaScript 17.7%
  • CSS 1%
  • Rust 0.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
8bit-Wraith 5c28f9f90d
Merge PR #12: give streaming memory claims context
Documentation-only correction checked against the three extraction helpers and callers; five executable Python ASTs unchanged. No configured checks or reviews required; no protections bypassed.
2026-10-28 03:57:09 +09:00
docs Initial commit 2026-10-19 11:42:08 +09:00
mustream docs: give streaming memory claims some context 2026-10-28 03:56:16 +09:00
streaming-demo fix: keep rejected decoder settings out of treble 2026-10-27 20:55:10 +09:00
tests fix: keep NaNs out of the export report card 2026-10-28 02:58:33 +09:00
.gitignore Initial commit 2026-10-19 11:42:08 +09:00
LICENSE Add code and model licenses 2026-10-20 15:53:25 +09:00
LICENSE-MODEL Add code and model licenses 2026-10-20 15:53:25 +09:00
pyproject.toml Add code and model licenses 2026-10-20 15:53:25 +09:00
README.md Add code and model licenses 2026-10-20 15:53:25 +09:00
uv.lock Initial commit 2026-10-19 11:42:08 +09:00

MuSTream: A Streaming Music Representation for Online Music Understanding

TL;DR MuSTream is a streaming music representation for online music understanding. A two-stage distillation (capacity, then causality) turns the non-causal MERT into causal streaming models of 26M, 95M and 330M parameters, which process audio as a stream and reach near-offline performance on chord recognition, beat and downbeat tracking and key detection, in real time on a CPU.

MuSTream overview

Highlights

  • Real-time on a CPU. One prediction every 0.2 s, with a block-wise KV cache instead of recomputing past context. On an Apple M1 CPU, MuSTream-26M runs at a real-time factor of 0.09 with 0.22 s latency, and MuSTream-330M at 0.35 with 0.29 s.
  • In-browser real-time interface. The demo page streams chord, beat and key predictions while a song plays, with ONNX Runtime Web on your own device: nothing to install, and your audio is never uploaded.
  • Open-source checkpoints. Six streaming backbones (26M / 95M / 330M, each with p = 0 and p = 15) and their chord, beat and key probes, in PyTorch and ONNX. The released backbones are distilled on the public MTG-Jamendo dataset.
  • Drop-in streaming features. extract_features.py turns any audio file into 75 fps MuSTream features through one streaming session, as in real-time use.
  • Reproducible. Scripts for both distillation stages on MTG-Jamendo (splits included), probe training and evaluation in offline, chunked, sliding-window and streaming modes, and ONNX export.

Quick Start

Real-time interface for music understanding

Open https://yamaha-research-mustream.static.hf.space, choose a task (chord, beat, key or all), upload a song and press Start streaming, or press Try a sample. It runs MuSTream-26M by default; the models run in your browser and your audio is never uploaded. For larger models and running the demo locally, see streaming-demo/README.md.

Streaming representation extraction

Set up the environment with uv and download the released checkpoints into mustream/ckpt/:

git clone https://github.com/yamaha-research/MuSTream.git && cd MuSTream
uv sync && source .venv/bin/activate
hf download yamaha-research/MuSTream --local-dir mustream/ckpt

Extract streaming features from your own audio with a MuSTream backbone (--model 26M, 95M or 330M; --variant r0 for p = 0 or r15 for p = 15):

python mustream/extract_features.py song.wav --model 95M --variant r15
# -> features/song.npz: `features` [L, T, D] at 75 fps (L = 13, D = 768 for 95M)

From Python:

import sys; sys.path.insert(0, "mustream")
from extract_features import load_backbone, load_audio, extract

model = load_backbone("mustream/ckpt/pytorch/backbone/streaming_95M_r15.ckpt", device="cuda")  # or "cpu"
features = extract(model, load_audio("song.wav"))     # torch.Tensor [L, T, D], 75 fps

For chord, beat and key predictions, training the backbones and probes, and evaluation, see mustream/README.md.

More experiment results

  • docs/understanding_performance.md: the paper's results come from backbones distilled on internal data. The released checkpoints are distilled on the public MTG-Jamendo dataset; this page compares the two sets on all three tasks.
  • docs/key_detection.md: full key-detection results, including how fast and how stably the running key estimate converges.

Repository

Path Contents
mustream/ The MuSTream code: backbones (modeling/), the two distillation stages (train_mert_size_distill.py, train_mert_causal_distill.py), MTG-Jamendo preparation (data_processing/), and feature extraction (extract_features.py).
mustream/chord_recognition/, beat_tracking/, key_detection/ The three understanding tasks: LSTM probes, data loaders, training, offline / chunked / sliding / streaming inference and evaluation.
mustream/onnx_export/ Exports a streaming backbone and its probes to ONNX and runs them without PyTorch.
streaming-demo/ The browser demo: real-time chord, beat and key recognition with ONNX Runtime Web.
docs/ Additional results and the overview figure.

License

Copyright 2026 Yamaha Corporation and the MuSTream authors.

For commercial use of the code under other terms, or of the model weights, please contact us.

Acknowledgement

MuSTream builds on the following open-source code: