bolyai/CIS490
3
0
Fork 0
Code Issues 5 Pull requests Projects 1 Releases Packages Wiki Activity Actions
CIS490/etc
History
max 49eba2fd60 fleet-health: proactive alerts on the Pi + per-host doctor reports 
Two pieces of self-monitoring so the maintainer isn't the alarm:

(2) Receiver-side fleet health monitor

cis490-fleet-health.timer runs check_fleet_health.py every 5 min.
Detects three symptoms and writes them to
/var/lib/cis490/alerts.jsonl + a syslog WARNING (greppable / easy
to forward to a notifier):

  silent      — host shipped in last 24h but has been quiet >30 min
  fatal-only  — actively shipping but every PUT 4xx
  unstamped   — shipping without X-Cis490-Code-Commit header

Dedup is keyed on (host, symptom, hour-bucket) so a sustained fault
fires once per hour, not every 5 min. 15 unit tests cover the index
parser, three detectors, and dedup.

(3) Per-host doctor snapshots

Lab hosts run cis490-doctor-check.timer once a day (10 min after
boot, then daily with 30-min jitter). The timer runs
cis490_doctor.py --json and PUTs the result to a new endpoint:

  PUT /v1/host-health/<host>   →  /var/lib/cis490/host-health/<host>.json
  GET /v1/host-health          →  aggregate across all hosts

Endpoint is NOT gated by version_gate — sick hosts running stale
code MUST still be able to report sickness. 11 unit tests cover
PUT/GET, atomic-write semantics, bearer auth, and the
not-gated-by-version-gate property.

ship_health_check.py reuses the existing shipper transport (mTLS +
bearer + receiver URL from lab-host.toml) so we don't reimplement
auth.

Both timers wired into install-lab-host.sh — the loop also enables
the previously-added autoupdate + cert-fetch timers, so a single
install run gives a host all four self-healing mechanisms.

Tests: 293 pass (26 new — 15 fleet-health, 11 host-health). 2
pre-existing test_fleet.py failures from the elliott-ThinkPad
merge (667f042) are unrelated to this change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 13:48:31 -05:00
..
caddy-root.crt bootstrap: auto-issue mTLS leaves to enrolled lab hosts (closes #9, refs #3) 2026-04-30 01:30:29 -05:00
cis490-autoupdate.service lab-host: cis490-autoupdate.timer for self-healing on push 2026-05-01 16:59:31 -05:00
cis490-autoupdate.timer lab-host: cis490-autoupdate.timer for self-healing on push 2026-05-01 16:59:31 -05:00
cis490-bootstrap.service Tier-4 sample source: theZoo (no auth, no operator action) 2026-05-01 01:17:50 -05:00
cis490-cert-fetch.service lab-host: cis490-cert-fetch.timer for automatic mTLS bootstrap retry 2026-05-02 13:30:16 -05:00
cis490-cert-fetch.timer lab-host: cis490-cert-fetch.timer for automatic mTLS bootstrap retry 2026-05-02 13:30:16 -05:00
cis490-doctor-check.service fleet-health: proactive alerts on the Pi + per-host doctor reports 2026-05-02 13:48:31 -05:00
cis490-doctor-check.timer fleet-health: proactive alerts on the Pi + per-host doctor reports 2026-05-02 13:48:31 -05:00
cis490-fleet-health.service fleet-health: proactive alerts on the Pi + per-host doctor reports 2026-05-02 13:48:31 -05:00
cis490-fleet-health.timer fleet-health: proactive alerts on the Pi + per-host doctor reports 2026-05-02 13:48:31 -05:00
cis490-orchestrator.service Tier-3 bring-up: 9 bugs fixed on elliott-ThinkPad (2026-05-01) 2026-05-02 12:26:19 -06:00
cis490-receiver.service Add receiver: PUT /v1/episodes ingest with sha256 verify and idempotency 2026-04-28 23:34:04 -06:00
cis490-shipper.service shipper: systemd watchdog, quarantine cleanup; doctor surfaces ship errors 2026-05-01 12:02:59 -05:00
lab-host.toml.example etc/lab-host.toml.example: pin Caddy root, not wg-pki client CA (closes #14) 2026-04-30 17:26:36 -05:00
README.md Add receiver: PUT /v1/episodes ingest with sha256 verify and idempotency 2026-04-28 23:34:04 -06:00
receiver.toml.example docs+doctor: surface VERSION-stamp + fallback wiring 2026-05-01 11:54:36 -05:00

README.md

etc/

Templates for system-level files installed by scripts/install-*.sh:

  • cis490-receiver.service — systemd unit for the receiver
  • receiver.toml.example — config template for the receiver
  • cis490-orchestrator.service (TODO) — systemd unit for the orchestrator
  • cis490-shipper.service (TODO) — systemd unit for the shipper
  • lab-host.toml.example (TODO) — config template for the lab host

See docs/deploy.md for the install flow.