8643192a71
Symmetric companion to the collection fleet (orchestrator/fleet.py)
but for *training*. Collection is embarrassingly parallel; training
is not (a model is trained at most once across the fleet), so the
receiver coordinates which worker gets which job.
Operator-control surface is etc/training_manifest.toml.example —
single canonical file declaring (a) per-host capability + per-model
allow/deny policy, (b) one [[jobs]] entry per (model, mode, hyper)
with capability constraints (require_cuda, prefer_cuda, min_vram_gib,
min_ram_gib, allowed_hosts).
Components:
capability.py — self-detection: hostname, cores, RAM, CUDA presence,
VRAM, torch version, git commit. Used by workers to filter
eligible jobs before claiming.
manifest.py — TOML loader + JobSpec/HostSpec. Job IDs are stable
sha256 of (model, mode, hyper, split_recipe, train_hosts, seed)
so manifest reload is idempotent: existing rows keep their status,
new jobs become claimable, removed jobs stay until cancelled.
queue.py — SQLite job queue (training_jobs.db) with statuses
pending|claimed|running|completed|failed|cancelled. Atomic
claim_next via single UPDATE WHERE status='pending'. Heartbeat,
complete, fail. Stale-claim sweep (stale_after_s=600s) with
max_attempts cutoff to failed.
store.py — model artifact store mirroring receiver/store.py.
Artifact ID is the sha256 of the uploaded tarball; bit-identical
re-runs deduplicate.
receiver.py — Starlette app exposing 11 endpoints:
POST /v1/job/claim (worker)
POST /v1/job/{id}/heartbeat (worker)
POST /v1/job/{id}/complete (worker)
POST /v1/job/{id}/fail (worker)
PUT /v1/model/{id} (worker — uploads tarball)
GET /v1/jobs (anyone)
GET /v1/workers (anyone)
POST /v1/job/{id}/cancel (operator: X-Operator-Token)
POST /v1/job/{id}/requeue (operator)
POST /v1/manifest/reload (operator)
GET /v1/health (anyone)
Runs as cis490-trainer-receiver.service on the Pi alongside the
existing receiver, on a separate port.
client.py — stdlib HTTP client (urllib only, no new deps).
worker.py — long-running daemon. Loop: detect capability → claim →
spawn training/trainer/run.py subprocess → heartbeat every 30s →
tar artifact, sha256, PUT /v1/model → complete. SIGTERM-safe.
Operator CLI (tools/cis490_jobs.py): status / list / show / cancel /
requeue / reload / workers. Cancel and requeue require
$CIS490_OPERATOR_TOKEN matching the receiver's configured value.
Bootstrap: scripts/install-training-worker.sh (Linux systemd) and
scripts/install-training-worker-windows.ps1 (Windows Scheduled Task)
let the operator enroll a new host with one command after cloning
the repo and setting up the venv. Worker self-tests capability
before registering.
End-to-end smoke verified on the Pi: receiver up, manifest synced,
14 jobs queued, worker registered, claimed 4 CPU-eligible jobs
(allow_jobs=["gbt","mlp"]), completed 3 (gbt-realistic, gbt-oracle,
mlp-oracle), 1 failed with the actual error visible via
cis490-jobs status, 3 artifacts uploaded to
/var/lib/cis490/models/<model>_<mode>/<sha256>/bundle.tar.zst with
proper index.jsonl row.
21 unit tests (manifest validation: 8; queue lifecycle + eligibility:
13). All pass alongside the prior 17 training tests = 38 green.
Open limitations surfaced inline:
- Hyper-key drift between manifest and run.py fails at training
time, not at manifest reload (worth tightening to argparse
introspection later).
- mTLS not yet wired through Caddy for the trainer-receiver port —
listens loopback-only until that lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
40 lines
1.2 KiB
Desktop File
40 lines
1.2 KiB
Desktop File
[Unit]
|
|
Description=CIS490 trainer worker (claims jobs, runs trainings, ships artifacts)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
Documentation=https://maxgit.wg/spectral/CIS490
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=cis490
|
|
Group=cis490
|
|
|
|
EnvironmentFile=-/etc/cis490/trainer-worker.env
|
|
|
|
# CIS490_TRAINER_RECEIVER_URL — set in trainer-worker.env
|
|
# FLEET_HOST_ID — override the hostname-derived host_id (optional)
|
|
|
|
ExecStart=/opt/cis490/.venv/bin/python -m training.fleet.worker \
|
|
--receiver-url ${CIS490_TRAINER_RECEIVER_URL} \
|
|
--validation /opt/cis490/data/processed/validation_v1.parquet \
|
|
--summary /opt/cis490/data/processed/features_window_v1.parquet \
|
|
--tensors /opt/cis490/data/processed/tensor_window_v1 \
|
|
--artifacts-dir artifacts \
|
|
--reports-dir reports/eval
|
|
|
|
WorkingDirectory=/opt/cis490
|
|
Restart=on-failure
|
|
RestartSec=15s
|
|
|
|
# Workers do compute-heavy training. Don't kill them just because a single
|
|
# job failed; let the daemon's own loop handle that.
|
|
TimeoutStopSec=120s
|
|
|
|
ProtectSystem=strict
|
|
ProtectHome=true
|
|
PrivateTmp=false # need /tmp for trainer scratch
|
|
NoNewPrivileges=true
|
|
ReadWritePaths=/opt/cis490 /var/lib/cis490 /tmp
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|