7c9f9582ca
Implements the deployment loop end-to-end on the CIS490 side:
shipper/
config.py ShipperConfig (host_id, paths, receiver endpoint, mTLS)
transport.py httpx-based PUT + ping with mTLS + bearer support
queue.py scan data/episodes/, tar+zstd via system zstd, ship,
retire to data/shipped/. Idempotent across crashes per
the state machine in docs/transport.md.
__main__.py CLI: --ping (smoke test), --once (one pass), or daemon
receiver/app.py: new POST /v1/ping that requires the same auth as PUT
/v1/episodes but writes nothing. Used by `cis490-shipper --ping`
during lab-host bring-up to verify the WG/Caddy/mTLS path before
shipping any real bytes.
etc/
cis490-shipper.service systemd unit for the lab-host shipper
cis490-orchestrator.service systemd unit for the lab-host queue
(kept disabled by default until queue
mode lands)
lab-host.toml.example config template
scripts/
install-lab-host.sh idempotent installer; verifies prereqs,
creates cis490 service user, syncs repo to
/opt/cis490, builds venv, drops systemd units
and config template
install-receiver.sh same, for the receiver role on the central WG
node (Pi5 in our setup)
tests/test_shipper.py 11 end-to-end tests against a real Uvicorn
server hosting the receiver app. Exercises
ping, tar+ship, idempotent re-ship, 409
conflict, transient (receiver down), tarball
round-trip via system zstd.
AGENTS.md guidance for AI agents working on this and sibling repos.
Headline: when you hit an issue you can't fully fix in
scope, file a Forgejo issue rather than leaving a TODO.
51/51 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
112 lines
4.5 KiB
Bash
Executable file
112 lines
4.5 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Install / refresh the CIS490 lab-host role.
|
|
#
|
|
# Idempotent — safe to re-run after `git pull`. Does NOT enroll the
|
|
# host into WireGuard (that's wg-enroll's job, run separately and
|
|
# *first*) and does NOT mint TLS certs (that's wg-pki's job).
|
|
#
|
|
# Steps:
|
|
# 1. Verify prereqs (KVM, zstd, qemu, python3.11+, systemd).
|
|
# 2. Create the cis490 service user + /var/lib/cis490 layout.
|
|
# 3. Sync the repo into /opt/cis490 and build a uv-managed venv.
|
|
# 4. Install systemd units from etc/.
|
|
# 5. Drop /etc/cis490/lab-host.toml (only on first install).
|
|
#
|
|
# Operator finishes by:
|
|
# - editing /etc/cis490/lab-host.toml (host_id, receiver URL, certs)
|
|
# - placing leaf certs at /etc/cis490/certs/{lab-host.pem,key,wg-ca.pem}
|
|
# - `systemctl enable --now cis490-shipper`
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
INSTALL_ROOT="${INSTALL_ROOT:-/opt/cis490}"
|
|
DATA_ROOT="${DATA_ROOT:-/var/lib/cis490}"
|
|
ETC_ROOT="${ETC_ROOT:-/etc/cis490}"
|
|
SERVICE_USER="${SERVICE_USER:-cis490}"
|
|
|
|
log() { printf '[install-lab-host] %s\n' "$*" >&2; }
|
|
die() { log "FATAL: $*"; exit 1; }
|
|
|
|
# --- 1. prereqs --------------------------------------------------------
|
|
log "checking prereqs"
|
|
|
|
if [[ $EUID -ne 0 ]]; then
|
|
die "must run as root (writes to /opt, /etc, /var/lib, and systemd)"
|
|
fi
|
|
command -v systemctl >/dev/null || die "systemd not found"
|
|
command -v qemu-system-x86_64 >/dev/null || die "qemu-system-x86_64 not on PATH"
|
|
command -v zstd >/dev/null || die "zstd not on PATH (apt install zstd)"
|
|
[[ -e /dev/kvm ]] || die "/dev/kvm missing — KVM not available"
|
|
|
|
# uv is preferred (lockfile-driven). Fall back to system pip if absent.
|
|
USE_UV=0
|
|
if command -v uv >/dev/null; then USE_UV=1; fi
|
|
|
|
# --- 2. user + layout --------------------------------------------------
|
|
log "ensuring service user $SERVICE_USER"
|
|
if ! id -u "$SERVICE_USER" >/dev/null 2>&1; then
|
|
useradd --system --no-create-home --shell /usr/sbin/nologin \
|
|
--home-dir "$INSTALL_ROOT" "$SERVICE_USER"
|
|
fi
|
|
# kvm group lets the service spawn VMs.
|
|
if getent group kvm >/dev/null 2>&1; then
|
|
usermod -a -G kvm "$SERVICE_USER" || true
|
|
fi
|
|
|
|
install -d -o root -g root -m 0755 "$ETC_ROOT" "$ETC_ROOT/certs"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 \
|
|
"$DATA_ROOT" "$DATA_ROOT/data" \
|
|
"$DATA_ROOT/data/episodes" "$DATA_ROOT/data/outbox" \
|
|
"$DATA_ROOT/data/shipped" "$DATA_ROOT/data/queue" \
|
|
"$DATA_ROOT/samples" "$DATA_ROOT/samples/store" \
|
|
"$DATA_ROOT/vm" "$DATA_ROOT/vm/images"
|
|
|
|
# --- 3. repo + venv ----------------------------------------------------
|
|
log "syncing repo into $INSTALL_ROOT"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 "$INSTALL_ROOT"
|
|
# We use a clean cp -aT rather than rsync to avoid an extra dep.
|
|
cp -aT "$REPO_ROOT" "$INSTALL_ROOT"
|
|
chown -R "$SERVICE_USER":"$SERVICE_USER" "$INSTALL_ROOT"
|
|
|
|
log "building venv"
|
|
if [[ "$USE_UV" -eq 1 ]]; then
|
|
sudo -u "$SERVICE_USER" -- env HOME="$INSTALL_ROOT" \
|
|
uv sync --project "$INSTALL_ROOT"
|
|
else
|
|
sudo -u "$SERVICE_USER" -- python3 -m venv "$INSTALL_ROOT/.venv"
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet --upgrade pip
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet starlette 'uvicorn[standard]' httpx msgpack
|
|
fi
|
|
|
|
# --- 4. systemd --------------------------------------------------------
|
|
log "installing systemd units"
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-shipper.service" \
|
|
/etc/systemd/system/cis490-shipper.service
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-orchestrator.service" \
|
|
/etc/systemd/system/cis490-orchestrator.service
|
|
systemctl daemon-reload
|
|
|
|
# --- 5. config template (only on first install) -----------------------
|
|
if [[ ! -f "$ETC_ROOT/lab-host.toml" ]]; then
|
|
log "writing $ETC_ROOT/lab-host.toml (template)"
|
|
install -m 0640 -o root -g "$SERVICE_USER" \
|
|
"$REPO_ROOT/etc/lab-host.toml.example" "$ETC_ROOT/lab-host.toml"
|
|
log ""
|
|
log "FIRST-INSTALL NEXT STEPS:"
|
|
log " 1. Edit $ETC_ROOT/lab-host.toml — set host_id and receiver URL."
|
|
log " 2. Place TLS material at:"
|
|
log " $ETC_ROOT/certs/wg-ca.pem"
|
|
log " $ETC_ROOT/certs/lab-host.pem"
|
|
log " $ETC_ROOT/certs/lab-host.key (mode 0600, owner $SERVICE_USER)"
|
|
log " 3. Smoke-test the receiver pipe:"
|
|
log " sudo -u $SERVICE_USER $INSTALL_ROOT/.venv/bin/python -m shipper \\"
|
|
log " --config $ETC_ROOT/lab-host.toml --ping"
|
|
log " 4. systemctl enable --now cis490-shipper"
|
|
else
|
|
log "$ETC_ROOT/lab-host.toml exists; leaving in place"
|
|
fi
|
|
|
|
log "lab-host install complete."
|