Autonomous ML research loop (autoresearch) with Tailscale telemetry to Hetzner
  • Python 72.6%
  • Nix 20.9%
  • Shell 6.5%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
peterlodri-sec 4184aebfb8 add CITATION.cff
Academic citation metadata for the harness and its dataset reference.
2026-06-11 03:53:42 +02:00
.claude feat: dual-slot parallel runs + LangSmith agent trace + launch skill 2026-06-10 23:11:27 +02:00
.github feat: publishable Nix flake + CI lint/check + GitHub template 2026-06-10 23:21:49 +02:00
nix feat: publishable Nix flake + CI lint/check + GitHub template 2026-06-10 23:21:49 +02:00
receiver feat: CI workflow (ruff + pytest), deploy.md for Vast.ai, ruff added to dev deps 2026-06-10 20:36:59 +02:00
worker feat: RUN_HOURS global config + auto-publish + auto-destroy on deadline 2026-06-11 00:29:00 +02:00
.editorconfig chore: gitignore, editorconfig, clean CLAUDE.md, project settings.json 2026-06-10 22:14:08 +02:00
.gitignore feat: dual-slot parallel runs + LangSmith agent trace + launch skill 2026-06-10 23:11:27 +02:00
CITATION.cff add CITATION.cff 2026-06-11 03:53:42 +02:00
CLAUDE.md feat: publishable Nix flake + CI lint/check + GitHub template 2026-06-10 23:21:49 +02:00
CONTRIBUTING.md docs: README and CONTRIBUTING in crabcc design language 2026-06-10 20:47:31 +02:00
deploy.md docs: rewrite deploy.md for flake modules + dual-slot + OpenRouter 2026-06-10 23:37:59 +02:00
FINDINGS.md docs: coherent research task + roadmap + FINDINGS.md 2026-06-10 22:05:58 +02:00
flake.lock feat: publishable Nix flake + CI lint/check + GitHub template 2026-06-10 23:21:49 +02:00
flake.nix feat: publishable Nix flake + CI lint/check + GitHub template 2026-06-10 23:21:49 +02:00
LICENSE chore: public release prep — license, funding, issue/pr templates, badges 2026-06-10 22:10:44 +02:00
README.md feat: publishable Nix flake + CI lint/check + GitHub template 2026-06-10 23:21:49 +02:00
SCHEMA.md docs(schema): add SCHEMA.md, run_meta JSON Schema, $schema ref in push_dataset 2026-06-10 22:41:41 +02:00

crabcc-autoresearch

License: MIT CI Sponsor

autonomous ml research loop — headless, cost-transparent, 8 hours unattended

The compute harness for the lambda-normalization-census research project. Runs Karpathy's autoresearch loop on an ephemeral GPU spot instance to train a small transformer that classifies λ-term normalization status (SN / SEP / NWN / UND). Every 5-minute cycle proposes a mutation to train.py, trains it, and reports val_bpb + cost back to a FastAPI receiver on Hetzner over Tailscale.

Each completed run archives its artifacts automatically into data/autoresearch/ in the census repo — extending the public dataset without manual intervention. Findings are curated in FINDINGS.md.


architecture

[ hetzner nixos vm ]  ←── tailscale mesh ───→  [ ephemeral gpu (vast.ai) ]
  receiver/                                        worker/
  fastapi + sqlite                                 autoresearch loop
  research.crabcc.app                              telemetry.py hook
  systemd rsync timer                              cloud-init.sh

Two components. One repo.

component runs on purpose
worker/ GPU spot instance training loop + telemetry client
receiver/ hetzner vm telemetry ingest, SQLite store, dashboard
nix/ nix-base (snippet) systemd rsync timer for results.tsv backup

quick start

1 — start the receiver on hetzner

cd receiver
task serve          # fastapi on :8787, tailscale-only
curl localhost:8787/health  # → {"ok":true}

2 — spin up a gpu instance and run

See deploy.md for the full step-by-step. Short version:

# on the gpu worker (after cloud-init)
export OPENROUTER_API_KEY=sk-or-...
export LLM_MODEL=anthropic/claude-sonnet-4-6   # optional
export CRABCC_RECEIVER_URL=http://<hetzner-tailscale-ip>:8787
export RUN_ID=crabcc-run-001
export BUDGET_USD=12

task prepare   # one-time tokenizer build
task run       # autonomous loop starts — gpu type and budget auto-reported

3 — monitor

Open http://<hetzner-tailscale-ip>:8787 (or research.crabcc.app if caddy is wired up).

Dashboard shows two tables:

— run sessions — gpu type, provider, budget, actual cost, best val_bpb, start/end time
— step log — every 5-minute cycle: val_bpb, status (SUCCESS / REVERTED / CRASHED), per-step api cost

Or from the cli on hetzner:

task sessions                              # last 10 runs
task tail RUN_ID=crabcc-run-001            # last 20 steps for a run

cost

Typical 8-hour RTX 4090 run on Vast.ai:

line item estimate
gpu compute (~$0.40/hr × 8h) ~$3.20
openrouter — sonnet + prompt caching ~$3–4
total ~$6–8

Prompt caching via OpenRouter cuts input token cost ~60–80% on the static train.py prefix. Swap model to DeepSeek-V3 via LLM_MODEL to halve the LLM cost further.


roadmap

sponsor goal: $500/month → nightly automation

Once the lambda-normalization-census hits its $500/month sponsorship threshold, the loop runs every night unattended:

at scale per night per month per year
gpu compute ~$3.20 ~$96 ~$1,150
llm (sonnet + caching) ~$3–4 ~$90–120 ~$1,080–1,440
total ~$6–8 ~$186–216 ~$2,230–2,590
training experiments ~96 ~2,880 ~34,560

why this matters:

A learned normalization classifier that generalises beyond the decidability horizon (n>16, where exact enumeration is intractable) would be a genuine research contribution — useful for type-checkers, proof assistants, and anyone reasoning about λ-calculus reduction. The loop finds the architecture and training recipe; the census provides the data; FINDINGS.md tracks what it discovers.

usefulness to the public / ai research community: 4/5
open reproducible ML experiments at this granularity are scarce — full artifact trail, cost metadata, and a task that directly advances a published open dataset. the gap to 5/5 is frontier scale; this is solo-researcher scale compute doing targeted theory-adjacent work.


stretch goal: $5,000/month → research-grade

At 10× the nightly threshold the loop shifts from solo-researcher to small-lab scale. The key change is GPU class — A100/H100 (80 GB VRAM) vs RTX 4090 (24 GB) — which unlocks model sizes that can learn structural inductive biases about λ-reduction rather than surface patterns.

at scale per night per month per year
gpu (A100 / H100 80 GB) ~$16–40 ~$480–1,200 ~$5,760–14,400
llm (sonnet + caching) ~$12–16 ~$360–480 ~$4,320–5,760
total ~$28–56 ~$840–1,680 ~$10,000–20,000
parallel architectures / night 4–8 — —
training experiments / month 5,000+ — —
max model scale 7B+ params — —

what $5k unlocks:

  • parallel architecture search — run 4–8 candidate train.py mutations simultaneously each night; the autoresearch LLM converges on the best design in weeks not years
  • 7B-class models — 80 GB VRAM opens transformer scales that can plausibly learn deep structural patterns in λ-term reduction sequences, not just token statistics
  • decidability horizon push — a heuristic with meaningfully better accuracy above n > 20 (where exact enumeration is intractable) would be a novel result usable in proof assistants and type-checkers
  • publishable dataset — ~60,000+ training experiments/year with full cost + metric provenance; sufficient for a formal-methods or ML workshop ablation study

usefulness at $5k/month: 4.5/5 — the gap to 5/5 closes when a finding clears peer review; the infrastructure and reproducibility are already there.


repo layout

.github/workflows/ci.yml   ruff + pytest on push + pr
FINDINGS.md                curated discoveries across nightly runs
deploy.md                  step-by-step vast.ai guide
worker/
  telemetry.py             report_start / report_run / report_end
  push_dataset.py          push run artifacts to lambda-normalization-census
  patch_llm.py             patches autoresearch → openrouter + langsmith
  cloud-init.sh            tailscale + uv + nvidia-smi auto-detect
  Taskfile.yml             prepare · run · publish · test
  program.md               research task: λ-term normalization classifier
receiver/
  main.py                  post /api/telemetry, /api/runs/start|end,
                           get /health, /api/sessions, /api/runs/{id}, /
  schema.sql               runs + run_sessions tables
  Taskfile.yml             serve · migrate · tail · sessions
nix/
  research-sync.nix         NixOS module: systemd rsync timer (slots → receiver)
  receiver.nix              NixOS module: systemd service for the FastAPI receiver
flake.nix                   publishable Nix flake (modules + devShell + checks)

telemetry api

The receiver exposes five endpoints:

POST /api/runs/start    → log gpu, provider, budget before loop begins
POST /api/telemetry     → one call per 5-min cycle (val_bpb, status, cost)
POST /api/runs/end      → seal the run with total cost
GET  /api/runs/{id}     → full step log for a run (JSON)
GET  /api/sessions      → last 50 run sessions with best val_bpb
GET  /health            → {"ok":true}

All writes travel over the Tailscale mesh. No public write endpoint.


dataset archive

After each run, task publish pushes three files to peterlodri-sec/lambda-normalization-census:

data/autoresearch/autoresearch-YYYYMMDD-HHMM/
  results.tsv      autoresearch native step log
  train.py         final best model code
  run_meta.json    {run_id, gpu_type, provider, budget_usd, total_cost_usd, best_val_bpb}

Requires GITHUB_TOKEN env var with contents: write on that repo. task run calls task publish automatically — no manual step needed.


dev

# zero-config dev shell (nix users)
nix develop

# or with uv directly
cd receiver && uv sync --dev && task test
cd worker   && uv sync --dev && task test

CI runs ruff check, ruff format --check, pytest, and nix flake check on every push.


nix integration

The repo ships a publishable Nix flake with two NixOS modules.

Consume as a flake input (recommended):

# flake.nix
inputs.crabcc.url = "github:peterlodri-sec/crabcc-autoresearch";

# your NixOS host
imports = [
  inputs.crabcc.nixosModules.research-sync  # rsync timer
  inputs.crabcc.nixosModules.receiver       # FastAPI receiver service
];

services.crabcc = {
  researchSync = {
    enable     = true;
    workerHost = "vast-4090-worker";   # Tailscale hostname
  };
  receiver = {
    enable   = true;
    repoPath = "/opt/crabcc-autoresearch";
  };
};

Or copy directly into nix-base (backwards-compatible path still works):

imports = [ ./nix/crabcc-research-sync.nix ];
# → now an alias for ./nix/research-sync.nix

nix develop drops you into a shell with uv, task, and gh pre-loaded.


use this as a template

This repo is a GitHub template. Click Use this template to get:

  • Dual-slot autoresearch loop (task-specific + general benchmark running in parallel)
  • LangSmith parent trace across both slots
  • Frictionless Data + JSON Schema + Sigstore-signed dataset publishing
  • NixOS modules for receiver deployment and artifact sync
  • Publishable Nix flake with devShell + CI checks
  • Branch protection, issue/PR templates, labels, CITATION.cff, FINDINGS.md

To adapt it to a new research task: replace worker/programs/lambda_normalization.md with your task description. Everything else (telemetry, publishing, CI) works unchanged.


fast · exact · frugal