High-throughput Q&A data synthesis pipeline for training LLMs. Self-hosted, "SOTA", open-source — generate 7.2K+ validated pairs/day
  • D 83.4%
  • Python 14%
  • Shell 2.2%
  • Makefile 0.2%
  • Nix 0.1%
  • Other 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-10-21 08:55:19 +09:00
.github/workflows chore(ci): bump deprecated actions (checkout@v4→v5) 2026-07-13 23:06:37 +02:00
.serena chore: sync dataset stats, changelog, HF inference URL, serena config 2026-07-09 21:42:29 +02:00
assets docs: remove PyPI badge, add logo, fix install instructions 2026-06-29 14:31:03 +02:00
deploy feat(pi): add Raspberry Pi 3B+ OpenRouter/Ethernet/Tailnet deploy profile 2026-07-19 05:50:48 +02:00
docs feat(pi): add Raspberry Pi 3B+ OpenRouter/Ethernet/Tailnet deploy profile 2026-07-19 05:50:48 +02:00
scripts feat(pi): add Raspberry Pi 3B+ OpenRouter/Ethernet/Tailnet deploy profile 2026-07-19 05:50:48 +02:00
src/ultrawhale feat(sanctuary): add Galaxy S24 Ultra (SM-S928U1) 5-node SHOW-ME-STATE-MISSOURI{(ALEX,CHRIS)}+1~ cluster 2026-10-21 08:55:19 +09:00
tests feat(sanctuary): add Galaxy S24 Ultra (SM-S928U1) 5-node SHOW-ME-STATE-MISSOURI{(ALEX,CHRIS)}+1~ cluster 2026-10-21 08:55:19 +09:00
ultrawhale-bench backyard-ultra: auto-sync 2026-08-11 03:41 2026-08-11 04:02:45 +02:00
.gitignore housekeeping: the daemon's live state is scratch, gitignored; uv.lock settled 2026-09-19 01:48:54 -04:00
.loop-status.sh chore: add loop-status helper script 2026-07-11 04:23:15 +02:00
.pre-commit-config.yaml feat: v2.0.0 — hardened, productionized, SOTA OSS pipeline 2026-06-29 12:09:23 +02:00
AGENTS.md feat(pi): add Raspberry Pi 3B+ OpenRouter/Ethernet/Tailnet deploy profile 2026-07-19 05:50:48 +02:00
audio-edge.sh chore: update docs, add audio-edge wrapper, gitignore nested repo 2026-07-13 08:58:55 +02:00
babilon_quant_dogfood.py feat: Babilon quant-dog recursive sampling-learning pipeline 2026-08-04 02:18:17 +02:00
CHANGELOG.md chore: sync dataset stats, changelog, HF inference URL, serena config 2026-07-09 21:42:29 +02:00
CONTRIBUTING.md docs: remove PyPI badge, add logo, fix install instructions 2026-06-29 14:31:03 +02:00
docker-compose.yml feat: v2.0.0 — hardened, productionized, SOTA OSS pipeline 2026-06-29 12:09:23 +02:00
Dockerfile feat: v2.0.0 — hardened, productionized, SOTA OSS pipeline 2026-06-29 12:09:23 +02:00
flake.nix feat: v2.0.0 — hardened, productionized, SOTA OSS pipeline 2026-06-29 12:09:23 +02:00
LICENSE feat: v2.0.0 — hardened, productionized, SOTA OSS pipeline 2026-06-29 12:09:23 +02:00
llm-server.sh feat: Babilon quant-dog recursive sampling-learning pipeline 2026-08-04 02:18:17 +02:00
llms.txt docs: robots.txt, llms.txt, meta template, DR playbooks, OpenAPI 2026-06-29 14:53:25 +02:00
multidog.py feat(sanctuary): wire 4-node Memnet 0o0-SSU + SpherePop compute sanctuary into multidog and CLI 2026-10-21 08:39:52 +09:00
ntp-discipline.sh ntp-discipline.sh — the clock witness: read-only query + machine-readable offset for the telemetry rows 2026-09-19 04:22:07 -04:00
package-lock.json chore(sota): implement dynamic autoscaling and optimize inference performance 2026-06-29 10:56:40 +02:00
pi_burner.sh fix: second_burn — unmount disk again before pass 2 (macOS auto-remounts) 2026-07-01 14:34:58 +02:00
pyproject.toml backyard-ultra: auto-sync 2026-08-11 01:48 2026-08-11 02:41:04 +02:00
README.md feat(sanctuary): add Galaxy S24 Ultra (SM-S928U1) 5-node SHOW-ME-STATE-MISSOURI{(ALEX,CHRIS)}+1~ cluster 2026-10-21 08:55:19 +09:00
robots.txt docs: robots.txt, llms.txt, meta template, DR playbooks, OpenAPI 2026-06-29 14:53:25 +02:00
run.sh feat: Babilon quant-dog recursive sampling-learning pipeline 2026-08-04 02:18:17 +02:00
Taskfile.yml fix: macOS compat, real HF upload, circular sig fix, task start e2e 2026-07-01 13:17:54 +02:00
upload_local_dogfeed.py fix(upload): re-upload files that have grown since last push 2026-07-12 22:54:16 +02:00
uv.lock housekeeping: the daemon's live state is scratch, gitignored; uv.lock settled 2026-09-19 01:48:54 -04:00

Ultrawhale Dogfeed Pipeline

Ultrawhale Dogfeed Pipeline

CI Python License HF Dataset Q&A Pairs

qa-synthesis llm-judge pi-feeder ultrawhale dogfeed self-host

Industrial-grade Q&A data synthesis pipeline for training LLMs — now with Pi-powered dog feeding.

Generates high-quality, LLM-judge-validated Q&A pairs at scale — up to 7,200+ pairs/day on a single M3 MacBook. Built for self-hosters and open-source contributors who want to produce training data they can trust. Optionally burns to Raspberry Pi 3B+ for automated motor/servo feeding with real-time HuggingFace dataset upload.

Numbers

Participants Day Week Month 6 Months Year 2 Years
1 7.2K 50.4K 216K 1.3M 2.6M 5.2M
50 360K 2.5M 10.8M 65M 131M 262M
100 720K 5M 21.6M 131M 262M 524M
1,000 7.2M 50.4M 216M 1.3B 2.6B 5.2B

"The best way to predict the future is to generate it."

why? I cannot explain it honestly, I don't know, this just feels cool to me,
anon + global + fully honest public datasetgen+dataset? please copy me ^^

SUPER-ULTRA-DEVELOPMENT-MODE — this is the notice, call for action etc.
I NEVER SAID ANYWHERE THAT THIS IS PERFECT. It's not. Help make it one.

Live Stats — PeetPedro/ultrawhale-dogfood

files range pairs   view

random samples

chain-of-thought reasoning · what is a dogfeed in ML?

Quickstart

# Install (from source)
git clone https://github.com/peterlodri-sec/ultrawhale-dogfood-pipeline.git
cd ultrawhale-dogfood-pipeline
uv sync --all-extras

# Generate 100 Q&A pairs (requires a running llama.cpp server)
uv run ultrawhale generate --num 100 --category cs --host http://localhost:8080

# Check pipeline health
uv run ultrawhale status

# Upload results to HuggingFace (requires HF_TOKEN)
uv run ultrawhale upload

# --- Pi Dog Feeding ---
# Burn SD card (insert card first)
task start

# Or step by step:
task pi:check     # Check dependencies
task pi:burn      # Download Pi OS, sign dataset, burn SD card

Features

  • Parallel Generation — Multi-worker architecture with dynamic autoscaling (2-8 workers)
  • Quality Gating — Every pair scored on coherence, length, and diversity; LLM-judge validated
  • HF/OpenRouter Fallback — Low-quality local outputs fall back to HF Inference API or OpenRouter free tier
  • Difficulty Sampling — Active learning distributes questions across easy/medium/hard tiers
  • Structured Logging — JSON or human-readable, with per-component tagging
  • Async I/O — Queue-based writer ensures generation is never blocked by disk I/O
  • Resource Management — Automatic CPU/memory monitoring prevents system crashes
  • Kompress-v8 — Post-processing compression for compact dataset storage
  • Pi Dog Feeding — Burn Pi OS with signed dataset, GPIO motor/servo control, background HF upload on every cycle
  • Dual-Mode — Same code runs on Pi (GPIO feeding + dataset gen) or generic machine (dataset gen only)

The pipeline default is Qwen3.6-27B Q4_K_M, but for best throughput use a MoE model:

Model Params Active Q4_K_M Speed vs Qwen3.6 Best for
unsloth/Qwen3-30B-A3B-Instruct-GGUF:Q4_K_M 30B 3B ~18GB 3-9x faster All topics
unsloth/Qwen3.6-27B-GGUF:Q4_K_M (current) 27B 27B ~16GB 1x All topics
Qwen2.5-Coder-32B-Instruct-GGUF:Q4_K_M 32B 32B ~20GB 0.5x Technical/CS

To switch models, update your env or llm-server.sh:

LLM_MODEL=qwen3-30b-a3b
# In llm-server.sh:
# -hf unsloth/Qwen3-30B-A3B-Instruct-GGUF:Q4_K_M

Architecture

┌──────────────────────────────────────────────────────┐
│                    Orchestrator                       │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐              │
│  │Worker 1 │  │Worker 2 │  │Worker N │  ...2-8      │
│  │  (CS)   │  │(Physics)│  │(General)│              │
│  └────┬────┘  └────┬────┘  └────┬────┘              │
│       │            │            │                     │
│       ▼            ▼            ▼                     │
│  ┌─────────────────────────────────────┐             │
│  │       Quality Scoring + Curation    │             │
│  │  (coherence · length · diversity)   │             │
│  └──────────────┬──────────────────────┘             │
│                 │                                     │
│                 ▼                                     │
│  ┌─────────────────────────────────────┐             │
│  │       Async Writer (JSONL)          │             │
│  └──────────────┬──────────────────────┘             │
│                 │                                     │
│                 ▼                                     │
│  ┌─────────────────────────────────────┐             │
│  │    Kompress-v8 → HF Dataset Upload  │             │
│  └─────────────────────────────────────┘             │
│                                                       │
│  ┌─────────────────────────────────────┐             │
│  │  Pi Feeder Node (optional)          │             │
│  │  GPIO motor/servo → telemetry/      │             │
│  │  LLM gen → datasets/                │             │
│  │  bg upload thread → HF              │             │
│  └─────────────────────────────────────┘             │
└──────────────────────────────────────────────────────┘

Configuration

All settings are configured via environment variables — no config files needed.

Variable Default Description
HF_TOKEN — HuggingFace API token (required for upload + HF inference)
OPENROUTER_API_KEY — OpenRouter key (free inference fallback if HF Inference API down)
LLM_HOST http://localhost:8080 LLM server URL
LLM_MODEL qwen3.6-27b Model name (see Recommended Model)
LLAMA_SERVER_BIN /opt/homebrew/bin/llama-server llama.cpp binary path
OPENROUTER_API_KEY — OpenRouter key for inference fallback
ULTRAWHALE_MAX_WORKERS 8 Maximum parallel workers
ULTRAWHALE_MIN_WORKERS 2 Minimum parallel workers
ULTRAWHALE_MIN_SCORE 0.65 Minimum quality score threshold
ULTRAWHALE_LOG_DIR ralph_logs/ Log output directory
ULTRAWHALE_OUTPUT_DIR dogfeed_parallel/ Dataset output directory

Commands

ultrawhale generate   # Generate Q&A pairs
ultrawhale sanctuary  # Collect Memnet 0o0-SSU + SpherePop dogfood across all 4 compute nodes
ultrawhale babilon    # Generate Babilon quant-dog-v1 Xoshiro128** translation pairs
ultrawhale upload     # Upload dogfeed to HuggingFace
ultrawhale compress   # Post-process with kompress-v8
ultrawhale status     # Pipeline health check

uv run multidog.py --once  # Multi-dimensional dogfeeder (NAND oscillator + Memnet Sanctuary + zoneInn + Babilon)

task start            # E2E: insert SD → burn Pi OS → sign dataset → configure
task pi:check         # Check Pi burner dependencies
task pi:burn          # Download Pi OS, sign, dual-burn SD card

Multi-Node Compute Sanctuary (ultrawhale sanctuary + multidog.py)

ultrawhale sanctuary and multidog.py orchestrate distributed dogfood collection across all 5 available hardware nodes (42 cores / 118,481 MB RAM) using MEMNET/1.0 0o0-SSU (Kompress-Super-Ultra) + SpherePop (((( <( ? ~ ! @ > )> ))), #Commit = #ValidWitness, ~=== SHOW-ME-STATE-MISSOURI{(ALEX,CHRIS)}+1~ <3 {<3,<3,<3}):

Node ID SoC / Arch Cores RAM Sanctuary Endpoint Role
m1-max-64gb Apple M1 Max (aarch64-apple-darwin) 10 64 GB 169.254.180.64:8788 (Thunderbolt 10Gbps) Live 0o0-SSU Sanctuary Daemon + Ollama aye-qwen36:35b-16k Q&A synthesis
dev-main-m3 Apple M3 (aarch64-apple-darwin) 8 24 GB 127.0.0.1:8789 / 192.168.11.9:8789 Live 0o0-SSU Sanctuary Daemon + Ternary NAND Ring Oscillator + Babilon + Multidog
galaxy-s24-ultra Snapdragon 8 Gen 3 for Galaxy (SM8650-AC / SM-S928U1 / R5CWC3C3FXA) 8 12 GB 192.168.11.22:36347/:53601/:8788 + 127.0.0.1:8792 Flagship Adreno 750 + Hexagon NPU Sanctuary (SHOW-ME-STATE-MISSOURI{(ALEX,CHRIS)}+1~)
galaxy-tab-s9-ultra Snapdragon 8 Gen 2 for Galaxy (SM-X910 / R52X104QZWE) 8 12 GB 169.254.93.206:8790 (en15) / 192.168.11.12:8788 SpherePop 0o0-SSU edge capsule responder
blu-g75 MediaTek Helio G85 (MT6769V/CB / mt6768, Android 15) 8 4 GB 127.0.0.1:8791 (USB ADB 1010305026000644) + 192.168.11.24:8788 Native Android 15 ARM64 /data/local/tmp/memnet Sanctuary Daemon

Development

git clone https://github.com/peterlodri-sec/ultrawhale-dogfood-pipeline.git
cd ultrawhale-dogfood-pipeline
uv sync --all-extras
uv run pytest

# Test dog feeding pipeline in generic mode (no Pi needed)
HF_TOKEN=your_token python3 src/ultrawhale/dog_feeding.py

See CONTRIBUTING.md for details on setup, workflow, and PR process. See docs/ARCHITECTURE.md for deep architecture details.

vaked-audio-edge

CI Python

Cloudflare edge audio streaming + YouTube import — Python CLI + macOS menu bar app. Worker + CF Container for Opus streaming. Nested repo at audio-edge/.

License

MIT — see LICENSE.


Pi Dog Feeding Burner

** NOTICE — E2E: Once you start a Pi with it, there is no SSH, no user access, no stopping it from outside.** The Pi boots, verifies the dataset signature, and runs autonomously — feeding on schedule and uploading to HuggingFace. To stop: disconnect power, remove SD card, run /boot/erase_sd.sh on another machine to wipe it.

End-to-end dog feeding solution for Raspberry Pi 3B+. Burns a signed Pi OS image, configures GPIO motor/servo control, and uploads events + generated datasets to HuggingFace.

Components

  1. Pi Burner Script (pi_burner.sh)

    • Downloads latest Pi OS Lite 64-bit
    • Signs dataset config with SHA256
    • Dual-burn SD card for reliability
    • Writes config.txt (WiFi/BT disabled, Ethernet only)
    • Adds verify_dataset.sh + erase_sd.sh + SETUP.txt
  2. Dog Feeding Pipeline (src/ultrawhale/dog_feeding.py)

    • Auto-detects Pi vs generic mode
    • GPIO motor+servo feeding on schedule (Pi) or mock (generic)
    • Real Q&A generation via HF Inference API or OpenRouter fallback
    • Background upload thread for telemetry + datasets
    • Dataset signature verification on boot

Security

  • Dataset signed with SHA256, verified on every boot
  • No SSH, no user accounts
  • Secure erase script (/boot/erase_sd.sh)
  • Ethernet-only (WiFi/BT disabled via dtoverlay)
  • Config tampering → feed pipeline halts