8643192a71
Symmetric companion to the collection fleet (orchestrator/fleet.py)
but for *training*. Collection is embarrassingly parallel; training
is not (a model is trained at most once across the fleet), so the
receiver coordinates which worker gets which job.
Operator-control surface is etc/training_manifest.toml.example —
single canonical file declaring (a) per-host capability + per-model
allow/deny policy, (b) one [[jobs]] entry per (model, mode, hyper)
with capability constraints (require_cuda, prefer_cuda, min_vram_gib,
min_ram_gib, allowed_hosts).
Components:
capability.py — self-detection: hostname, cores, RAM, CUDA presence,
VRAM, torch version, git commit. Used by workers to filter
eligible jobs before claiming.
manifest.py — TOML loader + JobSpec/HostSpec. Job IDs are stable
sha256 of (model, mode, hyper, split_recipe, train_hosts, seed)
so manifest reload is idempotent: existing rows keep their status,
new jobs become claimable, removed jobs stay until cancelled.
queue.py — SQLite job queue (training_jobs.db) with statuses
pending|claimed|running|completed|failed|cancelled. Atomic
claim_next via single UPDATE WHERE status='pending'. Heartbeat,
complete, fail. Stale-claim sweep (stale_after_s=600s) with
max_attempts cutoff to failed.
store.py — model artifact store mirroring receiver/store.py.
Artifact ID is the sha256 of the uploaded tarball; bit-identical
re-runs deduplicate.
receiver.py — Starlette app exposing 11 endpoints:
POST /v1/job/claim (worker)
POST /v1/job/{id}/heartbeat (worker)
POST /v1/job/{id}/complete (worker)
POST /v1/job/{id}/fail (worker)
PUT /v1/model/{id} (worker — uploads tarball)
GET /v1/jobs (anyone)
GET /v1/workers (anyone)
POST /v1/job/{id}/cancel (operator: X-Operator-Token)
POST /v1/job/{id}/requeue (operator)
POST /v1/manifest/reload (operator)
GET /v1/health (anyone)
Runs as cis490-trainer-receiver.service on the Pi alongside the
existing receiver, on a separate port.
client.py — stdlib HTTP client (urllib only, no new deps).
worker.py — long-running daemon. Loop: detect capability → claim →
spawn training/trainer/run.py subprocess → heartbeat every 30s →
tar artifact, sha256, PUT /v1/model → complete. SIGTERM-safe.
Operator CLI (tools/cis490_jobs.py): status / list / show / cancel /
requeue / reload / workers. Cancel and requeue require
$CIS490_OPERATOR_TOKEN matching the receiver's configured value.
Bootstrap: scripts/install-training-worker.sh (Linux systemd) and
scripts/install-training-worker-windows.ps1 (Windows Scheduled Task)
let the operator enroll a new host with one command after cloning
the repo and setting up the venv. Worker self-tests capability
before registering.
End-to-end smoke verified on the Pi: receiver up, manifest synced,
14 jobs queued, worker registered, claimed 4 CPU-eligible jobs
(allow_jobs=["gbt","mlp"]), completed 3 (gbt-realistic, gbt-oracle,
mlp-oracle), 1 failed with the actual error visible via
cis490-jobs status, 3 artifacts uploaded to
/var/lib/cis490/models/<model>_<mode>/<sha256>/bundle.tar.zst with
proper index.jsonl row.
21 unit tests (manifest validation: 8; queue lifecycle + eligibility:
13). All pass alongside the prior 17 training tests = 38 green.
Open limitations surfaced inline:
- Hyper-key drift between manifest and run.py fails at training
time, not at manifest reload (worth tightening to argparse
introspection later).
- mTLS not yet wired through Caddy for the trainer-receiver port —
listens loopback-only until that lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
40 lines
1.2 KiB
Desktop File
40 lines
1.2 KiB
Desktop File
[Unit]
|
|
Description=CIS490 trainer-receiver (training-fleet coordinator)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
Documentation=https://maxgit.wg/spectral/CIS490
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=cis490
|
|
Group=cis490
|
|
|
|
EnvironmentFile=-/etc/cis490/trainer-receiver.env
|
|
|
|
ExecStart=/opt/cis490/.venv/bin/python -m training.fleet.receiver \
|
|
--listen-addr 127.0.0.1:8445 \
|
|
--manifest /etc/cis490/training_manifest.toml \
|
|
--db /var/lib/cis490/training_jobs.db \
|
|
--store-root /var/lib/cis490/models \
|
|
--incoming-root /var/lib/cis490/incoming-models \
|
|
--index-path /var/lib/cis490/models/index.jsonl
|
|
|
|
# Reload behavior — SIGHUP re-reads manifest into the queue without dropping
|
|
# in-flight jobs. The receiver's own /v1/manifest/reload endpoint is the
|
|
# preferred control surface; this is for systemctl reload compatibility.
|
|
ExecReload=/bin/kill -HUP $MAINPID
|
|
|
|
WorkingDirectory=/opt/cis490
|
|
Restart=on-failure
|
|
RestartSec=5s
|
|
RestartPreventExitStatus=78 # sysadmin error — don't respawn
|
|
|
|
# Hardening — same shape as cis490-receiver.service
|
|
ProtectSystem=strict
|
|
ProtectHome=true
|
|
PrivateTmp=true
|
|
NoNewPrivileges=true
|
|
ReadWritePaths=/var/lib/cis490
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|