Build self-improving systems
  • Python 93.5%
  • Jupyter Notebook 6.2%
  • Dockerfile 0.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Peter Lodri f2ca1bff76
Merge pull request #1 from peterlodri-sec/copilot/review-fix-issues-update-readme
fix: headroom loop metrics, README duplicates, resource leak
2026-06-26 09:03:27 +02:00
bot 🎉 LoopKit v0.1 — your loop engineering starter kit 2026-06-25 17:41:45 +02:00
concepts 🎉 LoopKit v0.1 — your loop engineering starter kit 2026-06-25 17:41:45 +02:00
docs feat: interactive docs site — single-page guide (cobusgreyling style) 2026-06-25 18:25:45 +02:00
evals fix: remove duplicate README entries and fix headroom loop metrics access 2026-06-26 07:01:41 +00:00
loops fix: remove duplicate README entries and fix headroom loop metrics access 2026-06-26 07:01:41 +00:00
notebooks 🎉 LoopKit v0.1 — your loop engineering starter kit 2026-06-25 17:41:45 +02:00
space fix(space): clean rewrite — no syntax errors, correct imports 2026-06-26 08:54:41 +02:00
.gitignore 🎉 LoopKit v0.1 — your loop engineering starter kit 2026-06-25 17:41:45 +02:00
GUIDE.md fix: replace Colab image badge with text link (CSP-safe) 2026-06-25 18:08:16 +02:00
pyproject.toml 🎉 LoopKit v0.1 — your loop engineering starter kit 2026-06-25 17:41:45 +02:00
README.md fix: remove duplicate README entries and fix headroom loop metrics access 2026-06-26 07:01:41 +00:00

🔁 LoopKit — Your Loop Engineering Starter Kit

Build self-improving systems. The same pattern that produced 15 kompress models, now in a box you can clone and extend.

git clone https://github.com/peterlodri-sec/loopkit
cd loopkit
python -m loops.hello.loop
# 5 iterations. SHIP. You just ran your first loop.

▶️ Open in Colab


What can you build with this?

You want to... Use this
Fine-tune a model iteratively loops/template/ → implement plan/execute/evaluate/decide
Have a bot run experiments for you bot/main.py — Telegram bot with council
Understand loop engineering GUIDE.md — tiered guide with patterns
See a real example loops/kompress/ — the full 15-model pipeline
Run in a notebook notebooks/loopkit_hello.ipynb — Colab-ready
Teach someone the concept concepts/ — self-labeling, evaluator-optimizer, council
Run a daily repo health check loops/daily_triage/ — CI, issues, stale branches, outdated deps
Add quality gates to any loop loops/verification/ — LangChain Level 2: grader checks, retries on failure
Test if compression breaks agents loops/headroom_swebench/ — SWE-bench style: compare with/without compression
Run CI safety gates for headroom evals/headroom_swebench.py — exit 0=pass, 1=regression

The Loop Pattern

Every loop has four phases:

  plan ──→ execute ──→ evaluate ──→ decide
    ↑                                   │
    └───────────────────────────────────┘
Phase Question Example (kompress v8)
plan What should we try? "Use Qwen2.5-7B as teacher"
execute Run the experiment Label 97 texts → train 3 epochs on RTX 4090
evaluate How did it do? Heretic exact: 0.955, agent mk_in_ref: 1.000
decide What next? Council: SHIP — best model yet

Get a Telegram Bot Running (5 min)

# 1. Create a bot with @BotFather on Telegram → get token
# 2. Install
pip install python-telegram-bot huggingface_hub

# 3. Run
export LOOPKIT_BOT_TOKEN="your_token"
python -m bot.main

# 4. Chat with your bot on Telegram:
#    /new my-experiment
#    /run my-experiment
#    /decide my-experiment

Your bot will:

  • Remember experiments across restarts (SQLite)
  • Use the council (LLM) to review results and suggest next steps
  • Fall back to heuristic rules if no LLM is available

Real Results: The Kompress Loop

This kit was extracted from a real loop that produced 15 models:

Version Heretic Keep Lesson
v2 0.975 90% Precision ceiling
v4 0.943 82% Override internalized
v8 0.955 85% Production — Qwen teacher + C3
v11 0.906 52% Larger encoder ≠ better
v14 0.882 — Council concept proven

📖 Full story on vaked.dev 🤗 All models on HuggingFace 🔧 Training repo


Structure

loopkit/
├── GUIDE.md              ← The full guide
├── README.md             ← You are here
├── bot/                  ← Telegram bot (outer loop)
│   ├── main.py           ← Entry point
│   ├── memory.py         ← SQLite persistence
│   └── council.py        ← LLM decision maker
├── loops/                ← Loop implementations
│   ├── base.py           ← Abstract Loop class
│   ├── hello/            ← Minimal example (start here!)
│   ├── template/         ← Copy this for new loops
│   ├── daily_triage/     ← Morning routine — CI, issues, deps
│   ├── verification/     ← LangChain L2 — grader + retry
│   ├── headroom_swebench/ ← Does compression break agents?
│   └── kompress/         ← Full kompress pipeline
├── concepts/             ← Reference patterns
│   ├── self_labeling.py
│   ├── evaluator_optimizer.py
│   └── council.py
├── evals/                ← Shared evaluation tools
│   └── heretic.py
└── notebooks/            ← Colab-ready notebooks
    └── loopkit_hello.ipynb

The Loop Engineering Ecosystem

LoopKit is part of a growing movement. Here's the full picture:

Resource What it offers
Addy Osmani: Loop Engineering The canonical essay — 5 building blocks + memory, practical patterns, warnings
Cobus Greyling: loop-engineering Reference implementation with npm tools (loop-audit, loop-init, loop-cost), 7 patterns, pattern picker, goal engineering
LangChain: The Art of Loop Engineering 4 stacked loops (Agent → Verification → Event-Driven → Hill Climbing), "loopcraft" concept
LoopKit (this repo) Python-native starter kit with Telegram bot, council, Colab notebook, kompress case study

Key concepts from the ecosystem

From Addy Osmani — the 5 building blocks + memory:

  • Automations: Scheduled discovery & triage on a cadence
  • Worktrees: Safe parallel execution on isolated branches
  • Skills: Persistent project knowledge (SKILL.md)
  • Plugins & Connectors: MCP-based tools that reach into your stack
  • Sub-agents: Maker/checker split — the model that wrote code is too nice grading its own work
  • Memory: A markdown file outside conversation context — "the agent forgets, the repo does not"

From LangChain — the 4 stacked loops:

  1. Agent Loop: Model calls tools until done
  2. Verification Loop: Grader checks output, retries if needed
  3. Event-Driven Loop: Webhooks/cron trigger agents automatically
  4. Hill Climbing Loop: Analysis agent reviews traces, rewrites the harness — the loop that improves the loop

From Cobus Greyling — production tools:

  • loop-audit — scores your loop's readiness
  • loop-init — scaffolds new loops from patterns
  • loop-cost — estimates token spend before you run
  • 7 battle-tested patterns with real win/failure stories

How LoopKit fits

LoopKit implements the Agent Loop (level 1) and Verification Loop (level 2) in Python, with the Hill Climbing Loop (level 4) via the Council. The Event-Driven Loop (level 3) is the Telegram bot. Combine with cobusgreyling/loop-engineering for the full stack.


Further Reading


MIT License. Build loops. Ship models.


📝 Blog post: Introducing LoopKit