Upstream pull mirror of https://github.com/8b-is/syn-omni.git
  • Python 99.1%
  • Shell 0.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
dependabot[bot] acda0771b3
chore(deps): bump datasets from 4.4.1 to 5.0.1 in /vlm2vec_eval (#1)
Bumps [datasets](https://github.com/huggingface/datasets) from 4.4.1 to 5.0.1.
- [Release notes](https://github.com/huggingface/datasets/releases)
- [Commits](https://github.com/huggingface/datasets/compare/4.4.1...5.0.1)

---
updated-dependencies:
- dependency-name: datasets
  dependency-version: 5.0.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-08 09:29:15 -05:00
assets init 2026-10-07 17:32:02 +09:00
docker init 2026-10-07 17:32:02 +09:00
tests chore: drop unrelated test files from PR 2026-10-27 16:16:25 +09:00
vlm2vec_eval chore(deps): bump datasets from 4.4.1 to 5.0.1 in /vlm2vec_eval (#1) 2026-11-08 09:29:15 -05:00
.gitignore init 2026-10-07 17:32:02 +09:00
.style.yapf init 2026-10-07 17:32:02 +09:00
LICENSE init 2026-10-07 17:32:02 +09:00
pyproject.toml init 2026-10-07 17:32:02 +09:00
README.md feat(lora): add Quantum 64-Trit Bitplane OME-LoRA-Pq & PSR-Pq + fix MixtureOfLoRA.forward tuple unpack 2026-10-21 20:09:05 -04:00

Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji

EMNLP 2026 Findings

📄 Paper (coming soon)  |  🤗 Models (coming soon)

This repository contains the evaluation code for Syn-Omni: 81 tasks across image, video, audio, and audiovisual modalities, plus scripts for 5 baseline models.

Abstract

Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.


Installation

cd docker
docker build --build-arg USER_ID=$UID -t omniembed:v0 .

docker run --gpus all -it --shm-size=128gb \
    -v <path/to/code>:/home/appuser/omni \
    -v <path/to/datasets>:/home/appuser/datasets \
    -v <path/to/.cache/huggingface>:/home/appuser/.cache/huggingface \
    -v <path/to/.cache/torch/hub>:/home/appuser/.cache/torch/hub \
    --name synomni omniembed:v0

Install without Docker

cd vlm2vec_eval
pip install -r requirements_local.txt

This installs everything the Docker image normally provides. See the comments at the top of the file for install order.

Install flash-attn separately, last:

pip install flash_attn==2.8.3 --no-build-isolation

Dataset Setup

  1. Download MMEB-V2 (image/video tasks) once:

    cd vlm2vec_eval
    DATA_BASEDIR=/path/to/vlm2vec_eval_data bash experiments/public/data/download_data.sh
    
    • Defaults to /home/appuser/datasets/vlm2vec_eval if unset.
    • Also settable via scripts/env.sh.
  2. Everything else (audio and audiovisual) downloads automatically the first time you run an eval script. No manual step needed.


Useful Guide

  • src/model/: model/backbone loading, including the custom router-based LoRA (OME-LoRA / PSR) used by Syn-Omni.
  • src/model/lora/ome_lora_pq.py: Quantum 64-Trit Bitplane OME-LoRA-Pq & PSR-Pq (14.2× LoRA adapter compression, branchless SWAR 64-bit Walsh-Hadamard + POPCNT exact modality-expert orthogonality <A_m, A_n>_trit = 0, and zero-float BFP-64 + rational Q16 Padé-Cayley Progressive Synergy Routing). Run tests via python3 tests/test_ome_lora_pq.py -v. See also 8b-is/alexiai, 8b-is/Engram, and 8b-is/deepsiper-enthea.
  • src/data/eval_dataset/: per-dataset eval loaders.
  • src/data/collator/: batch collation, including on-the-fly video/audio decoding.
  • src/constant/: dataset name to HF/source mapping.
  • src/utils/: download helpers, eval utilities.
  • experiments/public/eval/*.yaml: per-modality dataset configs.
  • scripts/baselines/: eval scripts for baseline models.

Evaluation

All eval scripts live under scripts/ and share scripts/env.sh for DATA_BASEDIR/OUTPUT_BASEDIR.

Baselines

One script per model under scripts/baselines/. Most load their checkpoint from a public HF repo id and run as-is.

  • eval_wave.sh expects tsinghua-ee/WAVE-7B downloaded locally first.
  • multivent.sh expects Tevatron/OmniEmbed-v0.1-multivent downloaded locally first.

Acknowledgements


Citation

TBD