49eba2fd60
Two pieces of self-monitoring so the maintainer isn't the alarm:
(2) Receiver-side fleet health monitor
cis490-fleet-health.timer runs check_fleet_health.py every 5 min.
Detects three symptoms and writes them to
/var/lib/cis490/alerts.jsonl + a syslog WARNING (greppable / easy
to forward to a notifier):
silent — host shipped in last 24h but has been quiet >30 min
fatal-only — actively shipping but every PUT 4xx
unstamped — shipping without X-Cis490-Code-Commit header
Dedup is keyed on (host, symptom, hour-bucket) so a sustained fault
fires once per hour, not every 5 min. 15 unit tests cover the index
parser, three detectors, and dedup.
(3) Per-host doctor snapshots
Lab hosts run cis490-doctor-check.timer once a day (10 min after
boot, then daily with 30-min jitter). The timer runs
cis490_doctor.py --json and PUTs the result to a new endpoint:
PUT /v1/host-health/<host> → /var/lib/cis490/host-health/<host>.json
GET /v1/host-health → aggregate across all hosts
Endpoint is NOT gated by version_gate — sick hosts running stale
code MUST still be able to report sickness. 11 unit tests cover
PUT/GET, atomic-write semantics, bearer auth, and the
not-gated-by-version-gate property.
ship_health_check.py reuses the existing shipper transport (mTLS +
bearer + receiver URL from lab-host.toml) so we don't reimplement
auth.
Both timers wired into install-lab-host.sh — the loop also enables
the previously-added autoupdate + cert-fetch timers, so a single
install run gives a host all four self-healing mechanisms.
Tests: 293 pass (26 new — 15 fleet-health, 11 host-health). 2
pre-existing test_fleet.py failures from the elliott-ThinkPad
merge (667f042) are unrelated to this change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
332 lines
16 KiB
Bash
Executable file
332 lines
16 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Install / refresh the CIS490 lab-host role.
|
|
#
|
|
# Idempotent — safe to re-run after `git pull`. Does NOT enroll the
|
|
# host into WireGuard (that's wg-enroll's job, run separately and
|
|
# *first*) and does NOT mint TLS certs (that's wg-pki's job).
|
|
#
|
|
# Steps:
|
|
# 1. Verify prereqs (KVM, zstd, qemu, python3.11+, systemd).
|
|
# 2. Create the cis490 service user + /var/lib/cis490 layout.
|
|
# 3. Sync the repo into /opt/cis490 and build a uv-managed venv.
|
|
# 4. Install systemd units from etc/.
|
|
# 5. Drop /etc/cis490/lab-host.toml (only on first install).
|
|
#
|
|
# Operator finishes by:
|
|
# - editing /etc/cis490/lab-host.toml (host_id, receiver URL, certs)
|
|
# - placing leaf certs at /etc/cis490/certs/{lab-host.pem,key,wg-ca.pem}
|
|
# - `systemctl enable --now cis490-shipper`
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
INSTALL_ROOT="${INSTALL_ROOT:-/opt/cis490}"
|
|
DATA_ROOT="${DATA_ROOT:-/var/lib/cis490}"
|
|
ETC_ROOT="${ETC_ROOT:-/etc/cis490}"
|
|
SERVICE_USER="${SERVICE_USER:-cis490}"
|
|
|
|
log() { printf '[install-lab-host] %s\n' "$*" >&2; }
|
|
die() { log "FATAL: $*"; exit 1; }
|
|
|
|
# --- 1. prereqs --------------------------------------------------------
|
|
log "checking prereqs"
|
|
|
|
if [[ $EUID -ne 0 ]]; then
|
|
die "must run as root (writes to /opt, /etc, /var/lib, and systemd)"
|
|
fi
|
|
command -v systemctl >/dev/null || die "systemd not found"
|
|
command -v qemu-system-x86_64 >/dev/null || die "qemu-system-x86_64 not on PATH"
|
|
command -v zstd >/dev/null || die "zstd not on PATH (apt install zstd)"
|
|
[[ -e /dev/kvm ]] || die "/dev/kvm missing — KVM not available"
|
|
|
|
# uv is preferred (lockfile-driven). Fall back to system pip if absent.
|
|
USE_UV=0
|
|
if command -v uv >/dev/null; then USE_UV=1; fi
|
|
|
|
# --- 2. user + layout --------------------------------------------------
|
|
log "ensuring service user $SERVICE_USER"
|
|
if ! id -u "$SERVICE_USER" >/dev/null 2>&1; then
|
|
useradd --system --no-create-home --shell /usr/sbin/nologin \
|
|
--home-dir "$INSTALL_ROOT" "$SERVICE_USER"
|
|
fi
|
|
# kvm group lets the service spawn VMs.
|
|
if getent group kvm >/dev/null 2>&1; then
|
|
usermod -a -G kvm "$SERVICE_USER" || true
|
|
fi
|
|
|
|
install -d -o root -g root -m 0755 "$ETC_ROOT" "$ETC_ROOT/certs"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 \
|
|
"$DATA_ROOT" "$DATA_ROOT/data" \
|
|
"$DATA_ROOT/data/episodes" "$DATA_ROOT/data/outbox" \
|
|
"$DATA_ROOT/data/shipped" "$DATA_ROOT/data/queue" \
|
|
"$DATA_ROOT/samples" "$DATA_ROOT/samples/store" \
|
|
"$DATA_ROOT/vm" "$DATA_ROOT/vm/images"
|
|
|
|
# --- 3. repo + venv ----------------------------------------------------
|
|
log "syncing repo into $INSTALL_ROOT"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 "$INSTALL_ROOT"
|
|
# The expected lab-host upgrade path is `cd /opt/cis490 && git pull &&
|
|
# sudo ./scripts/install-lab-host.sh`, which means REPO_ROOT and
|
|
# INSTALL_ROOT can be the same directory. cp -aT errors out in that
|
|
# case ("are the same file") and `set -e` aborts before the systemd
|
|
# units get installed and services restart, leaving the host running
|
|
# whatever code was loaded at last service start.
|
|
if [[ "$(readlink -f "$REPO_ROOT")" == "$(readlink -f "$INSTALL_ROOT")" ]]; then
|
|
log "REPO_ROOT == INSTALL_ROOT ($INSTALL_ROOT); git pull already updated tree, skipping cp"
|
|
else
|
|
# Clean cp -aT to avoid an extra rsync dep.
|
|
cp -aT "$REPO_ROOT" "$INSTALL_ROOT"
|
|
fi
|
|
chown -R "$SERVICE_USER":"$SERVICE_USER" "$INSTALL_ROOT"
|
|
|
|
# Stamp a VERSION file at install time so episodes can record the
|
|
# code commit they were generated by. /opt/cis490 is a flat copy
|
|
# (no .git/), so we capture the source repo's HEAD here. Trainers
|
|
# read meta.json.code_version to filter out episodes from buggy
|
|
# pre-fix code.
|
|
if VC="$(cd "$REPO_ROOT" && git rev-parse HEAD 2>/dev/null)"; then
|
|
VB="$(cd "$REPO_ROOT" && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)"
|
|
VD="false"
|
|
if cd "$REPO_ROOT" && [[ -n "$(git status --porcelain 2>/dev/null)" ]]; then
|
|
VD="true"
|
|
fi
|
|
install -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0644 /dev/stdin \
|
|
"$INSTALL_ROOT/VERSION" <<EOF
|
|
{"commit": "$VC", "branch": "$VB", "dirty": $VD, "installed_at_wall": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"}
|
|
EOF
|
|
log "VERSION stamp: $VC ($VB)$([[ "$VD" == "true" ]] && echo " [dirty]")"
|
|
else
|
|
log "WARN: $REPO_ROOT not a git checkout; episodes will record code_version.commit='unknown'"
|
|
fi
|
|
|
|
log "building venv"
|
|
if [[ "$USE_UV" -eq 1 ]]; then
|
|
sudo -u "$SERVICE_USER" -- env HOME="$INSTALL_ROOT" \
|
|
uv sync --project "$INSTALL_ROOT"
|
|
else
|
|
sudo -u "$SERVICE_USER" -- python3 -m venv "$INSTALL_ROOT/.venv"
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet --upgrade pip
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet starlette 'uvicorn[standard]' httpx msgpack
|
|
fi
|
|
|
|
# --- 4. systemd --------------------------------------------------------
|
|
log "installing systemd units"
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-shipper.service" \
|
|
/etc/systemd/system/cis490-shipper.service
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-orchestrator.service" \
|
|
/etc/systemd/system/cis490-orchestrator.service
|
|
# Auto-update: a 30-min timer that does git fetch + (if behind) pull
|
|
# and re-run this script. Prevents host-falls-behind incidents when
|
|
# the receiver's allow-list rolls forward and an on-device agent
|
|
# fails to act on the 412 remediation. See AGENTS.md "Auto-update".
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-autoupdate.service" \
|
|
/etc/systemd/system/cis490-autoupdate.service
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-autoupdate.timer" \
|
|
/etc/systemd/system/cis490-autoupdate.timer
|
|
# mTLS cert-fetch retry timer: handles the "operator edited host_id
|
|
# but didn't re-run install" case AND the "bootstrap.wg was briefly
|
|
# unreachable" case. Polls every 5 min; no-op once the cert is on
|
|
# disk. See scripts/fetch-lab-host-cert.sh.
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-cert-fetch.service" \
|
|
/etc/systemd/system/cis490-cert-fetch.service
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-cert-fetch.timer" \
|
|
/etc/systemd/system/cis490-cert-fetch.timer
|
|
# Daily doctor check — runs cis490_doctor.py and PUTs JSON to the
|
|
# receiver's /v1/host-health/<host>. Lets the maintainer see fleet
|
|
# health without grepping per-host logs. See AGENTS.md "Fleet health".
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-doctor-check.service" \
|
|
/etc/systemd/system/cis490-doctor-check.service
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-doctor-check.timer" \
|
|
/etc/systemd/system/cis490-doctor-check.timer
|
|
systemctl daemon-reload
|
|
# Enable timers immediately — the operator gets self-healing on the
|
|
# next tick without an extra `systemctl enable`. Idempotent.
|
|
for t in cis490-autoupdate.timer cis490-cert-fetch.timer cis490-doctor-check.timer; do
|
|
systemctl enable --now "$t" 2>/dev/null || \
|
|
log "WARN: could not enable $t (will retry next install)"
|
|
done
|
|
|
|
# --- 5. config template (only on first install) -----------------------
|
|
if [[ ! -f "$ETC_ROOT/lab-host.toml" ]]; then
|
|
log "writing $ETC_ROOT/lab-host.toml (template)"
|
|
install -m 0640 -o root -g "$SERVICE_USER" \
|
|
"$REPO_ROOT/etc/lab-host.toml.example" "$ETC_ROOT/lab-host.toml"
|
|
NEW_INSTALL=1
|
|
else
|
|
log "$ETC_ROOT/lab-host.toml exists; leaving in place"
|
|
NEW_INSTALL=0
|
|
fi
|
|
|
|
# --- 6. orchestrator env file (read by cis490-orchestrator.service) ----
|
|
ENV_FILE="$ETC_ROOT/lab-host.env"
|
|
DEFAULT_HOST_ID="$(hostname -s)"
|
|
if [[ ! -f "$ENV_FILE" ]]; then
|
|
log "writing $ENV_FILE (host_id defaults to $DEFAULT_HOST_ID — edit if you want something else)"
|
|
install -m 0640 -o root -g "$SERVICE_USER" /dev/stdin "$ENV_FILE" <<EOF
|
|
# Read by cis490-orchestrator.service. Override per-host as needed.
|
|
FLEET_HOST_ID=$DEFAULT_HOST_ID
|
|
# BRIDGE=br-malware enables source 4 pcap capture AND unlocks the
|
|
# Tier-3 modules whose payloads need callback (reverse/bind shells).
|
|
# install-lab-host.sh provisions the bridge + tap pool below; leave
|
|
# this on unless your lab host can't run NETLINK ops.
|
|
BRIDGE=br-malware
|
|
EOF
|
|
fi
|
|
|
|
# --- 6b. host-only bridge + per-slot tap pool --------------------------
|
|
# br-malware lets pcap capture the guest traffic and lets bind/reverse
|
|
# shell payloads route between guest and host. We pre-create a small
|
|
# pool of taps so the launchers don't need sudo to attach interfaces;
|
|
# each slot uses cis490tap{SLOT,SLOT+100} (Tier-2 demo + Tier-3
|
|
# target). Idempotent: re-running on an already-set-up host is a
|
|
# no-op.
|
|
if command -v ip >/dev/null && [[ -x "$REPO_ROOT/vm/setup_bridge.sh" ]]; then
|
|
if "$REPO_ROOT/vm/setup_bridge.sh" >/dev/null 2>&1; then
|
|
log "bridge br-malware ready"
|
|
for n in 0 1 2 3 4 5 6 7; do
|
|
for prefix in cis490tap cis490target; do
|
|
tap="${prefix}${n}"
|
|
if ! ip link show "$tap" >/dev/null 2>&1; then
|
|
ip tuntap add dev "$tap" mode tap user "$SERVICE_USER" 2>/dev/null || \
|
|
ip tuntap add dev "$tap" mode tap 2>/dev/null || true
|
|
ip link set "$tap" master br-malware 2>/dev/null || true
|
|
ip link set "$tap" up 2>/dev/null || true
|
|
fi
|
|
done
|
|
done
|
|
log "tap pool: cis490tap0..7 + cis490target0..7 attached to br-malware"
|
|
else
|
|
log "WARN: setup_bridge.sh failed; BRIDGE mode will be unavailable"
|
|
# Comment out BRIDGE in the env file — fleet will still run
|
|
# Tier-2 + non-callback Tier-3 modules.
|
|
sed -i 's/^BRIDGE=br-malware/# BRIDGE=br-malware # auto-disabled: bridge setup failed/' "$ENV_FILE"
|
|
fi
|
|
fi
|
|
|
|
# --- 7. mTLS leaf cert (auto-fetch via bootstrap.wg) -------------------
|
|
# One-shot fetch via the standalone script (also wired to a 5-min
|
|
# retry timer in step 4 above, so this is just the install-time
|
|
# kick-off — the timer handles transient failures and the case
|
|
# where the operator edits host_id later).
|
|
"$INSTALL_ROOT/scripts/fetch-lab-host-cert.sh" || \
|
|
log "WARN: cert fetch failed — timer will retry"
|
|
|
|
# --- 8. baseline VM image + cidata (best-effort) -----------------------
|
|
ALPINE_IMG="$DATA_ROOT/vm/images/alpine-baseline.qcow2"
|
|
CIDATA_ISO="$DATA_ROOT/vm/images/cidata.iso"
|
|
if [[ ! -f "$ALPINE_IMG" ]]; then
|
|
if "$REPO_ROOT/scripts/fetch-alpine-baseline.sh" "$ALPINE_IMG"; then
|
|
log "fetched Alpine baseline -> $ALPINE_IMG"
|
|
else
|
|
log "WARN: Alpine baseline fetch failed; drop a qcow2 at $ALPINE_IMG manually"
|
|
fi
|
|
fi
|
|
if [[ -f "$ALPINE_IMG" && ! -f "$CIDATA_ISO" ]]; then
|
|
log "building cidata.iso (in-guest agent embedded)"
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/python" \
|
|
"$INSTALL_ROOT/tools/build_cidata.py" "$CIDATA_ISO" || \
|
|
log "WARN: cidata build failed; run tools/build_cidata.py manually"
|
|
fi
|
|
# Symlink the canonical paths the launchers look at, when missing.
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 "$INSTALL_ROOT/vm/images"
|
|
ln -sf "$ALPINE_IMG" "$INSTALL_ROOT/vm/images/alpine-baseline.qcow2" 2>/dev/null || true
|
|
ln -sf "$CIDATA_ISO" "$INSTALL_ROOT/vm/images/cidata.iso" 2>/dev/null || true
|
|
M2_IMG="$DATA_ROOT/vm/images/metasploitable2.qcow2"
|
|
[[ -f "$M2_IMG" ]] && ln -sf "$M2_IMG" "$INSTALL_ROOT/vm/images/metasploitable2.qcow2" 2>/dev/null || true
|
|
|
|
# --- 8. Tier-3 + Tier-4 deploy (auto, idempotent) ----------------------
|
|
# Bring up msfrpcd + Metasploitable2 + bridge + verify. Skipped only if
|
|
# certs aren't on disk yet (Tier-3 fire writes episodes that the
|
|
# shipper ships, so it's pointless to run before mTLS is live) or the
|
|
# operator passed --skip-tier3.
|
|
SKIP_TIER3="${SKIP_TIER3:-}"
|
|
for arg in "$@"; do
|
|
[[ "$arg" == "--skip-tier3" ]] && SKIP_TIER3=1
|
|
done
|
|
if [[ -z "$SKIP_TIER3" && -f "$ETC_ROOT/certs/lab-host.pem" ]]; then
|
|
log "deploying Tier 3 (msfrpcd + Metasploitable2 + bridge)"
|
|
if "$INSTALL_ROOT/scripts/install-tier-3-4.sh"; then
|
|
log "Tier-3 deploy ✓"
|
|
else
|
|
log "WARN: Tier-3 deploy failed — Tier 2 will keep running."
|
|
log " Re-run later: sudo $INSTALL_ROOT/scripts/install-tier-3-4.sh"
|
|
fi
|
|
elif [[ -z "$SKIP_TIER3" ]]; then
|
|
log "skipping Tier-3 deploy (no mTLS cert yet — re-run this script after"
|
|
log "host_id is set so the cert auto-fetches first)"
|
|
fi
|
|
|
|
if [[ "$NEW_INSTALL" == "1" ]]; then
|
|
log ""
|
|
log "================================================================="
|
|
log " FIRST-INSTALL NEXT STEPS "
|
|
log "================================================================="
|
|
log " 1. Edit $ETC_ROOT/lab-host.toml — set host_id and receiver URL."
|
|
log " (host_id starts as 'REPLACE_ME'; the cert auto-fetch in"
|
|
log " step 7 of this script SKIPS while that's still the value.)"
|
|
log ""
|
|
log " 2. RE-RUN THIS SCRIPT — sudo $0"
|
|
log " The second pass sees the real host_id and pulls the leaf"
|
|
log " cert from https://bootstrap.wg/v1/cert/<host_id>. There is"
|
|
log " no manual cert-minting step on this host. DO NOT openssl."
|
|
log ""
|
|
log " 3. Confirm certs landed:"
|
|
log " ls -l $ETC_ROOT/certs/"
|
|
log " Expected: lab-host.pem, lab-host.key, wg-ca.pem"
|
|
log ""
|
|
log " 4. Run the diagnostic — every red row prints the exact fix:"
|
|
log " $INSTALL_ROOT/.venv/bin/python \\"
|
|
log " $INSTALL_ROOT/tools/cis490_doctor.py --role lab-host"
|
|
log ""
|
|
log " 5. Smoke-test the pipe (returns ok=true on success):"
|
|
log " sudo -u $SERVICE_USER $INSTALL_ROOT/.venv/bin/python -m shipper \\"
|
|
log " --config $ETC_ROOT/lab-host.toml --ping"
|
|
log ""
|
|
log " 6. Turn on the services — episodes start flowing immediately:"
|
|
log " sudo systemctl enable --now cis490-shipper cis490-orchestrator"
|
|
log ""
|
|
log " If bootstrap.wg fetch fails (step 2 prints WARN), see AGENTS.md"
|
|
log " ('Securing the connection (mTLS)') — almost always a missing"
|
|
log " /etc/hosts entry or wg0 not up. Do not work around this by"
|
|
log " minting certs locally; fix the WG path instead."
|
|
log "================================================================="
|
|
fi
|
|
|
|
# --- 9. drain pre-stamp queue + restart services -----------------------
|
|
# On a re-run (not first install), the running services are still
|
|
# executing whatever code they loaded at last start, so the new module
|
|
# objects we just dropped on disk don't take effect until restart. And
|
|
# any episodes the orchestrator generated against pre-stamping code
|
|
# are missing meta.json::code_version — the receiver returns 400 for
|
|
# every PUT, the queue's fatal path didn't quarantine them in older
|
|
# code, and the shipper burns cycles re-tarring them on every pass.
|
|
# Drain them once, then restart so the new code reaches the live
|
|
# daemon.
|
|
if [[ "$NEW_INSTALL" != "1" ]]; then
|
|
PY="$INSTALL_ROOT/.venv/bin/python"
|
|
if [[ -x "$PY" && -f "$INSTALL_ROOT/tools/quarantine_unstamped.py" ]]; then
|
|
log "draining pre-stamp episodes from queue (idempotent)"
|
|
sudo -u "$SERVICE_USER" -- "$PY" \
|
|
"$INSTALL_ROOT/tools/quarantine_unstamped.py" \
|
|
--data-root "$DATA_ROOT/data" || \
|
|
log "WARN: quarantine drain returned non-zero — see output above"
|
|
fi
|
|
systemctl daemon-reload
|
|
for svc in cis490-shipper cis490-orchestrator; do
|
|
if systemctl is-enabled --quiet "$svc" 2>/dev/null; then
|
|
log "restarting $svc to pick up new code"
|
|
systemctl restart "$svc" || \
|
|
log "WARN: $svc restart failed — check 'journalctl -u $svc'"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
log "lab-host install complete."
|
|
log ""
|
|
log "Cloning this repo and running the launchers manually is NOT enough."
|
|
log "The lab-host role's data flow lives in the systemd services this"
|
|
log "script just installed. If $INSTALL_ROOT/index.jsonl on the Pi stays"
|
|
log "empty after step 5, run:"
|
|
log " $INSTALL_ROOT/.venv/bin/python $INSTALL_ROOT/tools/cis490_doctor.py"
|