- Python 72.6%
- Nix 20.9%
- Shell 6.5%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| .claude | ||
| .github | ||
| nix | ||
| receiver | ||
| worker | ||
| .editorconfig | ||
| .gitignore | ||
| CITATION.cff | ||
| CLAUDE.md | ||
| CONTRIBUTING.md | ||
| deploy.md | ||
| FINDINGS.md | ||
| flake.lock | ||
| flake.nix | ||
| LICENSE | ||
| README.md | ||
| SCHEMA.md | ||
crabcc-autoresearch
autonomous ml research loop — headless, cost-transparent, 8 hours unattended
The compute harness for the lambda-normalization-census research project. Runs Karpathy's autoresearch loop on an ephemeral GPU spot instance to train a small transformer that classifies λ-term normalization status (SN / SEP / NWN / UND). Every 5-minute cycle proposes a mutation to train.py, trains it, and reports val_bpb + cost back to a FastAPI receiver on Hetzner over Tailscale.
Each completed run archives its artifacts automatically into data/autoresearch/ in the census repo — extending the public dataset without manual intervention. Findings are curated in FINDINGS.md.
architecture
[ hetzner nixos vm ] ←── tailscale mesh ───→ [ ephemeral gpu (vast.ai) ]
receiver/ worker/
fastapi + sqlite autoresearch loop
research.crabcc.app telemetry.py hook
systemd rsync timer cloud-init.sh
Two components. One repo.
| component | runs on | purpose |
|---|---|---|
worker/ |
GPU spot instance | training loop + telemetry client |
receiver/ |
hetzner vm | telemetry ingest, SQLite store, dashboard |
nix/ |
nix-base (snippet) | systemd rsync timer for results.tsv backup |
quick start
1 — start the receiver on hetzner
cd receiver
task serve # fastapi on :8787, tailscale-only
curl localhost:8787/health # → {"ok":true}
2 — spin up a gpu instance and run
See deploy.md for the full step-by-step. Short version:
# on the gpu worker (after cloud-init)
export OPENROUTER_API_KEY=sk-or-...
export LLM_MODEL=anthropic/claude-sonnet-4-6 # optional
export CRABCC_RECEIVER_URL=http://<hetzner-tailscale-ip>:8787
export RUN_ID=crabcc-run-001
export BUDGET_USD=12
task prepare # one-time tokenizer build
task run # autonomous loop starts — gpu type and budget auto-reported
3 — monitor
Open http://<hetzner-tailscale-ip>:8787 (or research.crabcc.app if caddy is wired up).
Dashboard shows two tables:
— run sessions — gpu type, provider, budget, actual cost, best val_bpb, start/end time
— step log — every 5-minute cycle: val_bpb, status (SUCCESS / REVERTED / CRASHED), per-step api cost
Or from the cli on hetzner:
task sessions # last 10 runs
task tail RUN_ID=crabcc-run-001 # last 20 steps for a run
cost
Typical 8-hour RTX 4090 run on Vast.ai:
| line item | estimate |
|---|---|
| gpu compute (~$0.40/hr × 8h) | ~$3.20 |
| openrouter — sonnet + prompt caching | ~$3–4 |
| total | ~$6–8 |
Prompt caching via OpenRouter cuts input token cost ~60–80% on the static train.py prefix. Swap model to DeepSeek-V3 via LLM_MODEL to halve the LLM cost further.
roadmap
sponsor goal: $500/month → nightly automation
Once the lambda-normalization-census hits its $500/month sponsorship threshold, the loop runs every night unattended:
| at scale | per night | per month | per year |
|---|---|---|---|
| gpu compute | ~$3.20 | ~$96 | ~$1,150 |
| llm (sonnet + caching) | ~$3–4 | ~$90–120 | ~$1,080–1,440 |
| total | ~$6–8 | ~$186–216 | ~$2,230–2,590 |
| training experiments | ~96 | ~2,880 | ~34,560 |
why this matters:
A learned normalization classifier that generalises beyond the decidability horizon (n>16, where exact enumeration is intractable) would be a genuine research contribution — useful for type-checkers, proof assistants, and anyone reasoning about λ-calculus reduction. The loop finds the architecture and training recipe; the census provides the data; FINDINGS.md tracks what it discovers.
usefulness to the public / ai research community: 4/5
open reproducible ML experiments at this granularity are scarce — full artifact trail, cost metadata, and a task that directly advances a published open dataset. the gap to 5/5 is frontier scale; this is solo-researcher scale compute doing targeted theory-adjacent work.
stretch goal: $5,000/month → research-grade
At 10× the nightly threshold the loop shifts from solo-researcher to small-lab scale. The key change is GPU class — A100/H100 (80 GB VRAM) vs RTX 4090 (24 GB) — which unlocks model sizes that can learn structural inductive biases about λ-reduction rather than surface patterns.
| at scale | per night | per month | per year |
|---|---|---|---|
| gpu (A100 / H100 80 GB) | ~$16–40 | ~$480–1,200 | ~$5,760–14,400 |
| llm (sonnet + caching) | ~$12–16 | ~$360–480 | ~$4,320–5,760 |
| total | ~$28–56 | ~$840–1,680 | ~$10,000–20,000 |
| parallel architectures / night | 4–8 | — | — |
| training experiments / month | 5,000+ | — | — |
| max model scale | 7B+ params | — | — |
what $5k unlocks:
- parallel architecture search — run 4–8 candidate
train.pymutations simultaneously each night; the autoresearch LLM converges on the best design in weeks not years - 7B-class models — 80 GB VRAM opens transformer scales that can plausibly learn deep structural patterns in λ-term reduction sequences, not just token statistics
- decidability horizon push — a heuristic with meaningfully better accuracy above n > 20 (where exact enumeration is intractable) would be a novel result usable in proof assistants and type-checkers
- publishable dataset — ~60,000+ training experiments/year with full cost + metric provenance; sufficient for a formal-methods or ML workshop ablation study
usefulness at $5k/month: 4.5/5 — the gap to 5/5 closes when a finding clears peer review; the infrastructure and reproducibility are already there.
repo layout
.github/workflows/ci.yml ruff + pytest on push + pr
FINDINGS.md curated discoveries across nightly runs
deploy.md step-by-step vast.ai guide
worker/
telemetry.py report_start / report_run / report_end
push_dataset.py push run artifacts to lambda-normalization-census
patch_llm.py patches autoresearch → openrouter + langsmith
cloud-init.sh tailscale + uv + nvidia-smi auto-detect
Taskfile.yml prepare · run · publish · test
program.md research task: λ-term normalization classifier
receiver/
main.py post /api/telemetry, /api/runs/start|end,
get /health, /api/sessions, /api/runs/{id}, /
schema.sql runs + run_sessions tables
Taskfile.yml serve · migrate · tail · sessions
nix/
research-sync.nix NixOS module: systemd rsync timer (slots → receiver)
receiver.nix NixOS module: systemd service for the FastAPI receiver
flake.nix publishable Nix flake (modules + devShell + checks)
telemetry api
The receiver exposes five endpoints:
POST /api/runs/start → log gpu, provider, budget before loop begins
POST /api/telemetry → one call per 5-min cycle (val_bpb, status, cost)
POST /api/runs/end → seal the run with total cost
GET /api/runs/{id} → full step log for a run (JSON)
GET /api/sessions → last 50 run sessions with best val_bpb
GET /health → {"ok":true}
All writes travel over the Tailscale mesh. No public write endpoint.
dataset archive
After each run, task publish pushes three files to peterlodri-sec/lambda-normalization-census:
data/autoresearch/autoresearch-YYYYMMDD-HHMM/
results.tsv autoresearch native step log
train.py final best model code
run_meta.json {run_id, gpu_type, provider, budget_usd, total_cost_usd, best_val_bpb}
Requires GITHUB_TOKEN env var with contents: write on that repo. task run calls task publish automatically — no manual step needed.
dev
# zero-config dev shell (nix users)
nix develop
# or with uv directly
cd receiver && uv sync --dev && task test
cd worker && uv sync --dev && task test
CI runs ruff check, ruff format --check, pytest, and nix flake check on every push.
nix integration
The repo ships a publishable Nix flake with two NixOS modules.
Consume as a flake input (recommended):
# flake.nix
inputs.crabcc.url = "github:peterlodri-sec/crabcc-autoresearch";
# your NixOS host
imports = [
inputs.crabcc.nixosModules.research-sync # rsync timer
inputs.crabcc.nixosModules.receiver # FastAPI receiver service
];
services.crabcc = {
researchSync = {
enable = true;
workerHost = "vast-4090-worker"; # Tailscale hostname
};
receiver = {
enable = true;
repoPath = "/opt/crabcc-autoresearch";
};
};
Or copy directly into nix-base (backwards-compatible path still works):
imports = [ ./nix/crabcc-research-sync.nix ];
# → now an alias for ./nix/research-sync.nix
nix develop drops you into a shell with uv, task, and gh pre-loaded.
use this as a template
This repo is a GitHub template. Click Use this template to get:
- Dual-slot autoresearch loop (task-specific + general benchmark running in parallel)
- LangSmith parent trace across both slots
- Frictionless Data + JSON Schema + Sigstore-signed dataset publishing
- NixOS modules for receiver deployment and artifact sync
- Publishable Nix flake with devShell + CI checks
- Branch protection, issue/PR templates, labels, CITATION.cff, FINDINGS.md
To adapt it to a new research task: replace worker/programs/lambda_normalization.md with your task description. Everything else (telemetry, publishing, CI) works unchanged.
fast · exact · frugal