Spider/crawler agent for the AgentField fleet. CloakBrowser-backed page fetch + RSS/Atom feed polling.
  • Python 92.9%
  • Shell 6.4%
  • Dockerfile 0.7%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-17 23:56:09 -05:00
.devcontainer feat(audit): HTTP/TLS connection reuse + bootstrap tooling (#17) 2026-05-19 02:44:31 +02:00
.environment feat(audit): HTTP/TLS connection reuse + bootstrap tooling (#17) 2026-05-19 02:44:31 +02:00
.githooks chore: add difit pre-commit viewer hook 2026-05-28 23:32:07 +02:00
.github/workflows fix(ci): gate Docker Hub push steps when DOCKER_USER/DOCKER_PAT are unset 2026-11-04 07:30:04 +01:00
deploy/k3s feat(agent+deploy): @app.skill('fetch','feed_add') + k3s manifests (#3) 2026-05-17 06:11:37 +02:00
docs feat(proxy): add SPIDER_PROXY_TRUST_INSECURE opt-in TLS-verify bypass 2026-05-25 05:41:47 +02:00
reasoners feat(act): POST /act + browser_act() + SSRF allowlist (VIB-20, VIB-23) 2026-05-19 03:15:09 +02:00
scripts feat(act): POST /act + browser_act() + SSRF allowlist (VIB-20, VIB-23) 2026-05-19 03:15:09 +02:00
spider feat(cloakbrowser,crawl,proxy): upgrade to CloakBrowser v0.6.5 (Chromium 155), merge VIB-24 crawl_dispatcher & SPIDER_PROXY_TRUST_INSECURE (v0.5.0) 2026-11-01 06:08:48 +01:00
tests feat(cloakbrowser,crawl,proxy): upgrade to CloakBrowser v0.6.5 (Chromium 155), merge VIB-24 crawl_dispatcher & SPIDER_PROXY_TRUST_INSECURE (v0.5.0) 2026-11-01 06:08:48 +01:00
.dockerignore Initial commit: bootstrap agentfield-spider-agent scaffold 2026-05-17 03:47:22 +02:00
.env.example feat(cloakbrowser,crawl,proxy): upgrade to CloakBrowser v0.6.5 (Chromium 155), merge VIB-24 crawl_dispatcher & SPIDER_PROXY_TRUST_INSECURE (v0.5.0) 2026-11-01 06:08:48 +01:00
.gitignore feat(sessions): profile hydrate/persist + claude-routine (#19) 2026-05-19 02:44:25 +02:00
AGENTS.md feat(crawl): spawn crawl_dispatcher on spider_server startup 2026-05-19 07:32:14 +02:00
CHANGELOG.md release(v0.4.0): bump version + CHANGELOG (#52) 2026-05-19 03:09:44 +02:00
CLAUDE.md feat(crawl): spawn crawl_dispatcher on spider_server startup 2026-05-19 07:32:14 +02:00
docker-compose.yml build(docker): switch base to Docker Hardened Images (DHI) (#44) 2026-05-19 01:43:05 +02:00
Dockerfile build(docker): switch base to Docker Hardened Images (DHI) (#44) 2026-05-19 01:43:05 +02:00
Dockerfile.cloak feat(cloakbrowser,crawl,proxy): upgrade to CloakBrowser v0.6.5 (Chromium 155), merge VIB-24 crawl_dispatcher & SPIDER_PROXY_TRUST_INSECURE (v0.5.0) 2026-11-01 06:08:48 +01:00
Dockerfile.lightpanda feat(spider): optimized browser builds + session management (#7) 2026-05-18 23:33:20 +02:00
LICENSE docs: add MIT LICENSE file 2026-08-12 13:08:24 +02:00
main.py feat(act): POST /act + browser_act() + SSRF allowlist (VIB-20, VIB-23) 2026-05-19 03:15:09 +02:00
pyproject.toml feat(cloakbrowser,crawl,proxy): upgrade to CloakBrowser v0.6.5 (Chromium 155), merge VIB-24 crawl_dispatcher & SPIDER_PROXY_TRUST_INSECURE (v0.5.0) 2026-11-01 06:08:48 +01:00
README.md docs(readme): add spider-agent.act skill and POST /act route entries (#132) 2026-05-19 06:10:18 +02:00
requirements.txt feat(cloakbrowser,crawl,proxy): upgrade to CloakBrowser v0.6.5 (Chromium 155), merge VIB-24 crawl_dispatcher & SPIDER_PROXY_TRUST_INSECURE (v0.5.0) 2026-11-01 06:08:48 +01:00
spider_server.py feat(crawl): spawn crawl_dispatcher on spider_server startup 2026-05-19 07:32:14 +02:00
uv.lock Merge pull request #15 from 8b-is/dependabot/uv/multidict-6.9.1 2026-11-17 23:56:09 -05:00

agentfield-spider-agent

Web fetch + RSS/Atom feed polling for the agent fleet. CloakBrowser-backed (source-level Chromium stealth) and Playwright-compatible. Two surfaces:

  1. AgentField skill — spider-agent.fetch(url, ...) callable from any agent via app.call(...).
  2. HTTP — :11700 matching the rag-stack crawler API so existing callers (rag-pocket pipelines etc.) can switch without code changes.

Status

v1.2 — Adds Docling for structured doc parsing (PDF/Word/PPT) + a beta lightpanda browser backend alongside CloakBrowser. Modern tooling: pyproject.toml + uv + ruff + mypy/pyright strict on new code. The dispatcher now routes 5 URL kinds (github / github-issue / hn / reddit / docling) before falling through to the browser pool.

Capability

Surface Signature Returns
AgentField spider-agent.fetch(url, sync=True) {status, id, url, fetch_url, title, text, ms, ts} on ok
AgentField spider-agent.feed_add(url, interval_s=3600, name=None) feed record
AgentField spider-agent.feed_list() {feeds: [...]}
AgentField spider-agent.feed_remove(feed_id) {ok: true} or error
AgentField spider-agent.feed_poll(feed_id) poll result dict
AgentField spider-agent.jobs_recent(limit=50) {jobs: [...]}
AgentField spider-agent.crawl(url, max_depth=2, max_pages=50, same_domain_only=True, timeout_seconds=600) {job_id, status, ...}
AgentField spider-agent.crawl_status(job_id) {job_id, status, pages_fetched, pages_queued, ...}
AgentField spider-agent.sitemap_parse(url, max_urls=1000, queue=False) {url, urls: [...], count, queued}
AgentField spider-agent.screenshot(url, full_page=True) {ok, png_b64, ms}
AgentField spider-agent.extract(url, selector, attr=None) {ok, matches: [...], ms}
AgentField spider-agent.act(task, session_id, model=None, max_steps=None, allowed_hosts=None) {status, task, session_id, model, ms, result?, urls_visited?, steps_taken?, is_done?} — browser-use Agent against an existing session's BrowserContext
HTTP POST /fetch/sync {url, use_browser?, proxy?} same dict as fetch(sync=True), enriched per URL kind
HTTP POST /fetch {url} {queued: true, url} (Redis-backed worker)
HTTP POST /feeds {url, interval_s?, name?} {id, url, ...}
HTTP GET /feeds / GET /feeds/{id} / DELETE /feeds/{id} / POST /feeds/{id}/poll feed CRUD + force-poll
HTTP POST /crawl {url, max_depth?, max_pages?, same_domain_only?, timeout_seconds?} {job_id, status, ...}
HTTP GET /crawl/{job_id} crawl status
HTTP GET /crawl/{job_id}/results {job_id, count, results: [...]}
HTTP POST /sitemap/parse {url, queue?, max_urls?} {url, urls: [...], count, queued}
HTTP POST /screenshot {url, full_page?} {ok, png_b64, ms}
HTTP POST /extract {url, selector, attr?} {ok, matches: [...], ms}
HTTP POST /act {task, session_id, model?, max_steps?, allowed_hosts?} same dict as act(...); reuses the session's existing BrowserContext (no second browser launched). Per-call allowed_hosts unions with the env-level SPIDER_AGENT_ALLOWED_HOSTS SSRF allowlist. Returns 501 without the [browser-use] extra. See AGENTS.md § Browser-use Agent.
HTTP GET /jobs?limit=50 recent fetch outcomes
HTTP GET /health snapshot

Architecture

                ┌───────────────────────────────────────────────────┐
                │  agentfield-spider-agent  (one image, two procs)  │
   any agent ──▶│  main.py        ─── AgentField skill (port 8005) │──▶ Redis
   .call(…)     │                                                   │
   external ──▶ │  spider_server.py ── aiohttp HTTP (port 11700)    │──▶ CloakBrowser
                └───────────────────────────────────────────────────┘

Both processes share spider/ (the browser pool, fetch pipeline, politeness, security) so the two surfaces never drift.

URL-kind dispatcher

/fetch/sync (and the async dispatcher) routes by URL shape before falling through to the browser:

URL pattern Fetcher Surface
github.com/{owner}/{repo} spider/fetchers/github.py REST API + base64-decoded README
github.com/{owner}/{repo}/issues/{N} spider/fetchers/github.py REST API
news.ycombinator.com/item?id={N} spider/fetchers/hn.py Firebase API; recurses into GitHub fetcher if the linked URL is a repo
reddit.com/... spider/fetchers/reddit.py redirect-follow + .json with browser UA + stripped tracking params
*.pdf / *.docx / *.pptx / etc. spider/docling_parser.py Docling-parsed markdown + tables (heavy dep, opt-in)
anything else spider/fetch.py:fetch_one CloakBrowser- or Lightpanda-rendered path

Each specialized fetcher returns the same flat-dict shape as fetch_one (with extra keys: kind, name, description, metadata), so callers that only read status / title / text keep working.

Force the browser path explicitly with use_browser: true on the request payload (or pass use_browser=True from Python).

Browser backends

Backend Module Size Use when
cloak (default) spider/browsers/cloak.py ~200 MB Chromium Stealth needed (Cloudflare, Akamai, aggressive WAFs)
lightpanda (beta) spider/browsers/lightpanda.py ~30 MB image Bulk crawls of static / lightly-dynamic sites; latency or memory matter

Pick the default via BROWSER_BACKEND=cloak|lightpanda env. Per-call override on /fetch/sync payload:

{
  "url": "https://example.com",
  "use_browser": true,
  "browser": "lightpanda"
}

The dispatcher lazy-instantiates the alternate pool on first use — the unused backend has zero memory cost until it's needed.

Lightpanda runs as a separate process:

docker compose --profile lightpanda up -d

(See Dockerfile.lightpanda — it FROM lightpanda/browser:nightly and serves CDP on :9222.)

Docling (PDF / Word / PowerPoint structured parse)

URLs ending in a document extension (.pdf, .docx, .pptx, .xlsx, .odt, ...) route to Docling instead of the browser pool — Chromium can render PDFs but only as a single unstructured canvas; Docling extracts markdown + tables + page counts.

Heavy dependency: docling pulls in transformers + torch and ~1.5 GB of ML models on first use. Not bundled in the default image; install via the [docling] extra:

pip install -e ".[docling]"

When docling isn't installed, the parser returns a clean status="error" envelope rather than crashing.

Docling runtime tuning knobs:

  • SPIDER_DOCLING_TIMEOUT_S (default 120) — parse timeout
  • SPIDER_DOCLING_MAX_TABLES (default 10) — max tables extracted
  • SPIDER_DOCLING_TABLES_PREVIEW (default 3) — preview tables in metadata
  • SPIDER_DOCLING_MAX_TEXT_CHARS (default 500000) — response text cap

On main pushes, CI also builds and pushes a Docling-enabled image (INSTALL_DOCLING=1) using the configured cloud Buildx builder and tags it as docling-latest + docling-$SHA.

Manual push helper:

DOCKER_USER=your-user scripts/build-docling-image.sh

Outbound proxy

When SPIDER_PROXY_URL is set in the environment, ALL outbound HTTP egresses through it — specialized fetchers, the redirect walker, robots.txt, and the CloakBrowser pool's Chromium process.

Phone-as-egress setup (Termux SOCKS5, Tailscale exit node, or a small Linux box on a phone hotspot) is documented in docs/phone-as-proxy.md.

Per-call override: /fetch/sync accepts a proxy field on the payload. The override applies to the HTTP fetchers (github, hn, reddit) only; the browser pool is shared across requests, so its proxy is fixed at pool launch.

Local quick start

cp .env.example .env
# .env: AGENTFIELD_SERVER, REDIS_URL, browser knobs

docker compose up --build -d
docker compose logs -f spider-http | head -30

# Smoke (post-PR 1):
curl -X POST http://localhost:11700/fetch/sync \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}'

Agent/dev bootstrap scripts

Environment bootstrap is centralized in:

  • .environment/setup.sh (shared)
  • .environment/cursor/setup.sh
  • .environment/codex/setup.sh
  • .environment/claude/setup.sh

The scripts install requirements.txt, validate with pip check, and optionally install Docling when SPIDER_INSTALL_DOCLING=1 (legacy SPIDER_DOCLING_ENABLED=1 is still accepted).

By default the same setup path also installs Docker CLI + buildx (SPIDER_INSTALL_DOCKER=1) and performs docker login when both DOCKER_USER and DOCKER_PAT are present (registry defaults to DOCKER_REGISTRY=docker.io).

If your environment lacks root/sudo, setup now warns and continues by default. Set SPIDER_INSTALL_DOCKER_STRICT=1 to force a hard failure instead.

Bootstrap validation helper:

./scripts/check-env-bootstrap.sh

For local containerized development, .devcontainer/devcontainer.json uses .devcontainer/postCreate.sh, which delegates to the same shared bootstrap script.

Engines / dependencies

Concern Library Notes
Browser (default) CloakBrowser Stealth Chromium with source-level patches. Playwright-compatible. ~200 MB binary auto-fetched on first run.
Browser (beta) Lightpanda Zig browser exposing CDP. Tiny (~10 MB binary). Connect via Playwright connect_over_cdp.
Doc parse (opt-in) Docling PDF/Word/PPT/Excel → structured markdown + tables. ~1.5 GB models on first use.
RSS/Atom feedparser Standard.
Redis redis-py (asyncio) Queue + seen-set + feeds + jobs.
HTTP aiohttp + httpx Server (aiohttp) + outbound HTTP fetchers (httpx).
Agent framework agentfield The fleet's primitive.

Tooling

Tool Config Purpose
uv pyproject.toml [tool.uv] Faster pip replacement; uv sync + uv lock
ruff [tool.ruff] Lint + format; py312 target
mypy [tool.mypy] strict Type check (per-module opt-outs for older modules)
pyright [tool.pyright] strict on new modules Editor + CI types
pytest [tool.pytest.ini_options] asyncio_mode=auto Tests

Strict-typed modules (PEP 695 syntax, no Dict/List imports): spider/browsers/**, spider/docling_parser.py, spider/dispatch.py, spider/proxy.py, spider/fetchers/**. Older modules migrate in follow-up PRs.

License + redistribution note

CloakBrowser: the Python/TS wrapper is MIT, but the compiled Chromium binary carries a non-redistribution license. We fetch it at container runtime (not baked into the image) and cache it via a persistent volume in k3s. Review the upstream license before public deployment.

Provenance

This agent extracts the useful core of two existing tools:

  • rag-stack/scripts/crawler.py — aiohttp + Scrapling crawler with feeds, politeness, robots, job log. We keep the HTTP shape and fetch pipeline, drop the topic / search-engine / rag-pocket plumbing.
  • bin/datapy/cmd/spider/ — Go binary with cookie-banner dismissal, DOM extraction, and Chrome guards. We port the cookie-banner + DOM-extraction layers to Python.

CloakBrowser replaces Scrapling as the browser backend.