Sovereign 8s.is mirror of standardgalactic/savanna
  • Python 54.5%
  • Jupyter Notebook 37.4%
  • Shell 7.3%
  • C++ 0.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
gabo515 7b819b8609
feat: add LoRA (Low-Rank Adaptation) support for parameter-efficient finetuning (#13)
Add LoRA adapters for Savanna's parallel linear layers, enabling
parameter-efficient finetuning while keeping the base model frozen.

Supported layer types:
- ColumnParallelLinear (Megatron-style tensor parallel)
- RowParallelLinear (Megatron-style tensor parallel)
- TELinear (TransformerEngine FP8, column & row parallel)

Key capabilities:
- YAML-based config (rank, alpha, dropout, target modules, separate LR)
- Freeze base model, train only LoRA params (+ optional modules_to_save)
- Save/load LoRA weights separately per MP rank, or merged into base
- Checkpoint key remapping for loading base checkpoints into LoRA models
- Compatible with activation checkpointing via _ensure_requires_grad fix
- Config validation (r, alpha, dropout, target_modules type checks)

New files:
- savanna/lora.py: LoRA wrapper classes, apply/merge/unmerge/state_dict
- savanna/arguments/lora_config.py: GlobalConfigLoRA dataclass
- tests/unit/test_lora.py: 64 unit tests

Modified files:
- savanna/arguments/arguments.py: register LoRA config in BASE_CLASSES
- savanna/checkpointing.py: LoRA save/load, key remapping, async race fix
- savanna/model/utils.py: LoRA param groups, _ensure_requires_grad
- savanna/training.py: inject LoRA in get_model, freeze base, finetune path

Tested: single-GPU LoRA finetuning of Evo2-7B with FP8/TELinear (10 iters,
loss decreasing, 30% MFU).
2026-02-19 18:52:45 -08:00
configs evo2 1b config 2025-05-07 09:57:33 -07:00
launcher feat: setup commit 2025-02-15 16:55:24 -08:00
notebooks feat: setup commit 2025-02-15 16:55:24 -08:00
requirements fix: pin flash-attn to a version compatible with other packages to avoid warnings 2025-07-16 11:58:31 -06:00
resources feat: setup commit 2025-02-15 16:55:24 -08:00
savanna feat: add LoRA (Low-Rank Adaptation) support for parameter-efficient finetuning (#13) 2026-02-19 18:52:45 -08:00
scripts feat: setup commit 2025-02-15 16:55:24 -08:00
tests feat: add LoRA (Low-Rank Adaptation) support for parameter-efficient finetuning (#13) 2026-02-19 18:52:45 -08:00
tools vortex_conversion 2025-03-10 15:24:08 -07:00
tuner feat: setup commit 2025-02-15 16:55:24 -08:00
.gitignore feat: setup commit 2025-02-15 16:55:24 -08:00
CITE.md chore: add CITE 2025-02-19 00:48:41 -08:00
launch.py feat: setup commit 2025-02-15 16:55:24 -08:00
launch_multinode_slurm.sh feat: setup commit 2025-02-15 16:55:24 -08:00
LICENSE Update LICENSE 2025-02-17 10:17:49 -08:00
Makefile feat: setup commit 2025-02-15 16:55:24 -08:00
MANIFEST.in feat: setup commit 2025-02-15 16:55:24 -08:00
NOTICE Update NOTICE 2025-02-17 10:19:47 -08:00
paper.pdf chore: add paper 2025-02-19 04:37:24 -08:00
pyproject.toml feat: setup commit 2025-02-15 16:55:24 -08:00
README.md fix: wrong link to evo2 repo in README.md (#4) 2025-04-20 23:19:49 -07:00
run_regression_test.sh feat: setup commit 2025-02-15 16:55:24 -08:00
setup.py feat: setup commit 2025-02-15 16:55:24 -08:00
slurm.regression.sh feat: setup commit 2025-02-15 16:55:24 -08:00
train.py feat: setup commit 2025-02-15 16:55:24 -08:00

Savanna

Pretraining infrastructure for research and application of convolutional multi-hybrid models (StripedHyena 2).

What is Savanna?

Savanna is an open-source pretraining framework for alternative architectures. It is built on top of components from other open-source projects, including:

It extends these projects with additional features, including:

  • Multi-hybrid model support (StripedHyena, StripedHyena 2)
  • byte-level tokenization and training
  • tuner for distributed hyperparameters
  • a2a and p2p context parallelism
  • various optimization for graceful scaling on thousand-GPU clusters

A variety of models have been trained and finetuned on Savanna, including:

Savanna is a framework developed and maintained by a small team (< 5 people), tailored to the specific needs of the projects above. With careful tuning of the distributed training strategy and architecture, it achieves high MFU on H100s at the thousand-GPU scale. It is not a production-ready framework, expect rough edges if you don't know what you're doing. It can serve as a good starting point for research on large-scale training of multi-hybrids. We recommend digging around the training configs for the models above: all the details are there.

For inference using multi-hybrids, check out the vortex project.

StripedHyena 2

Install

make setup-env will install the necessary dependencies and detect if you have an active (non-base) python environment.

If TransformerEngine wheel installation fails on your cluster, check the official docs for more details (tip: when using conda, you can get most missing headers easily with conda install -c conda-forge cudatoolkit and conda install -c conda-forge cudnn. Make sure to add the right header paths to CPATH).

Quick Start

After tokenization of a dataset split (or splits), launch with:

python launch.py train.py -d configs data/<config>.yml model/<model>.yml

How do Convolutional Multi-Hybrid Architectures Work?

Multi-hybrid models combine many different classes of operators such as convolutions to balance quality and efficiency at short and long context, compared to previous generation striped hybrids. Convolutional multi-hybrids combine multiple input-dependent convolutional operators: Hyena-SE, Hyena-MR, Hyena-LI. These use the standard Hyena structure with different parametrizations for the convolutions.

Multi-hybrid architectures excel at modeling byte or character-tokenized data, such as DNA sequences.

Parametrizations

explicit_single_decay: (for Hyena-MR) Explicit parametrization of filter coefficients in time domain, followed by exponential decay.

implicit_freeform: (for Hyena-LI) Free-form parametrization of the filters via a shallow network with sine activations, followed by exponential decays of different rates.

implicit_modal: (for Hyena-LI) Real modal diagonal parametrization (poles and residues), uses a mSISO (multi-input single-output) structure

explicit: (for Hyena-SE). Not supported via config. Automatically selected with operator type hyena_se.

Kernels and Context Parallelism

gcg_two_pass: Chunked direct convolutions via two-pass GEMMs. Best used with finite impulse response convolution (windowed) parametrization (e.g., Hyena-SE, Hyena-MR).

hyena_mlp: custom kernel for short Hyena replacements to MLPs.

savanna/model/operators/hyena/distributed_a2a.py: context parallel algorithms for Hyena-LI.

savanna/model/operators/hyena/p2p_cp_conv.py: context parallel algorithm for short explicit convolutions (Hyena-SE, or input featurizer convolutions).

E2E Testing

Regression Testing

When submitting PRs, please make sure to run the regression test SLURM job ./run_regression_test.sh

The script runs two pre-defined model configs, a single-gpu and 4-gpu model parallel config. The script automatically allocates the necessary resources via SLURM.

  • Usage:
    • Run the script from within the savanna root directory.
    • Alternatively, provide the required paths to the data and model configs.
    • See ./run_regression_test.sh --help to see full list of commandline options (config paths, checkpoint, log directories, etc.)
  • Please run with the default number of iterations (2000) if making model architecture or parallelism changes.
  • For minor infrastructure changes, ok to run -- to completion -- with a smaller number of iterations
    • E.g. ./run_regression_test.sh --train-iters-1 100 --train-iters-2 100

Code Style

Install ruff, an all-in-one code linting and formatting tool. Then run:

make check

This will call ruff to check all files in savanna and enumerate formatting / linting fixes without actually making any code changes.

Run:

make style

to then fix these changes. Not all changes will be fixable, some will need be manually fixed.

Contributors

Savanna is developed by Michael Poli (Zymrael), Eric Nguyen (exnx), Jerome Ku (jeromeku), Brian Hie (brianhie), with contributions from Garyk Brixi (garykbrixi), David W. Romero (dwromero), Dave Burke (daveyburke), Brandon Yang (bclyang), Armin W. Thomas (athms), Ali Taghibakhshi (JRD971000), Greg Brockman (gdb).

Cite

If you find this project useful, consider citing the following references.