7c9f9582ca
Implements the deployment loop end-to-end on the CIS490 side:
shipper/
config.py ShipperConfig (host_id, paths, receiver endpoint, mTLS)
transport.py httpx-based PUT + ping with mTLS + bearer support
queue.py scan data/episodes/, tar+zstd via system zstd, ship,
retire to data/shipped/. Idempotent across crashes per
the state machine in docs/transport.md.
__main__.py CLI: --ping (smoke test), --once (one pass), or daemon
receiver/app.py: new POST /v1/ping that requires the same auth as PUT
/v1/episodes but writes nothing. Used by `cis490-shipper --ping`
during lab-host bring-up to verify the WG/Caddy/mTLS path before
shipping any real bytes.
etc/
cis490-shipper.service systemd unit for the lab-host shipper
cis490-orchestrator.service systemd unit for the lab-host queue
(kept disabled by default until queue
mode lands)
lab-host.toml.example config template
scripts/
install-lab-host.sh idempotent installer; verifies prereqs,
creates cis490 service user, syncs repo to
/opt/cis490, builds venv, drops systemd units
and config template
install-receiver.sh same, for the receiver role on the central WG
node (Pi5 in our setup)
tests/test_shipper.py 11 end-to-end tests against a real Uvicorn
server hosting the receiver app. Exercises
ping, tar+ship, idempotent re-ship, 409
conflict, transient (receiver down), tarball
round-trip via system zstd.
AGENTS.md guidance for AI agents working on this and sibling repos.
Headline: when you hit an issue you can't fully fix in
scope, file a Forgejo issue rather than leaving a TODO.
51/51 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
106 lines
4.1 KiB
Bash
Executable file
106 lines
4.1 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Install / refresh the CIS490 receiver role on the central WG node
|
|
# (the Pi5 in our setup). Idempotent — safe to re-run.
|
|
#
|
|
# Steps:
|
|
# 1. Verify prereqs (python3.11+, systemd).
|
|
# 2. Create the cis490 service user + /var/lib/cis490 layout.
|
|
# 3. Sync the repo into /opt/cis490 and build a venv.
|
|
# 4. Install cis490-receiver.service.
|
|
# 5. Drop /etc/cis490/receiver.toml on first install.
|
|
#
|
|
# This script does NOT:
|
|
# - configure Caddy. Add a `collector.wg` block to your spectral/caddy
|
|
# config to terminate TLS and reverse-proxy to 127.0.0.1:8443.
|
|
# - issue server / client certs. wg-pki owns CA + leaf issuance.
|
|
# - open firewall ports. iptmonads owns the WG-side ruleset.
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
INSTALL_ROOT="${INSTALL_ROOT:-/opt/cis490}"
|
|
DATA_ROOT="${DATA_ROOT:-/var/lib/cis490}"
|
|
ETC_ROOT="${ETC_ROOT:-/etc/cis490}"
|
|
SERVICE_USER="${SERVICE_USER:-cis490}"
|
|
|
|
log() { printf '[install-receiver] %s\n' "$*" >&2; }
|
|
die() { log "FATAL: $*"; exit 1; }
|
|
|
|
# --- 1. prereqs --------------------------------------------------------
|
|
log "checking prereqs"
|
|
if [[ $EUID -ne 0 ]]; then
|
|
die "must run as root"
|
|
fi
|
|
command -v systemctl >/dev/null || die "systemd not found"
|
|
command -v python3 >/dev/null || die "python3 not on PATH"
|
|
|
|
PY_VER="$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
|
|
if ! python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3,11) else 1)'; then
|
|
die "python >=3.11 required, found $PY_VER"
|
|
fi
|
|
|
|
USE_UV=0
|
|
if command -v uv >/dev/null; then USE_UV=1; fi
|
|
|
|
# --- 2. user + layout --------------------------------------------------
|
|
log "ensuring service user $SERVICE_USER"
|
|
if ! id -u "$SERVICE_USER" >/dev/null 2>&1; then
|
|
useradd --system --no-create-home --shell /usr/sbin/nologin \
|
|
--home-dir "$INSTALL_ROOT" "$SERVICE_USER"
|
|
fi
|
|
|
|
install -d -o root -g root -m 0755 "$ETC_ROOT" "$ETC_ROOT/certs"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 \
|
|
"$DATA_ROOT" "$DATA_ROOT/episodes" "$DATA_ROOT/incoming"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0750 "$DATA_ROOT"
|
|
# Pre-create the index file so the first PUT doesn't race on creation.
|
|
sudo -u "$SERVICE_USER" -- touch "$DATA_ROOT/index.jsonl"
|
|
|
|
# --- 3. repo + venv ----------------------------------------------------
|
|
log "syncing repo into $INSTALL_ROOT"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 "$INSTALL_ROOT"
|
|
cp -aT "$REPO_ROOT" "$INSTALL_ROOT"
|
|
chown -R "$SERVICE_USER":"$SERVICE_USER" "$INSTALL_ROOT"
|
|
|
|
log "building venv"
|
|
if [[ "$USE_UV" -eq 1 ]]; then
|
|
sudo -u "$SERVICE_USER" -- env HOME="$INSTALL_ROOT" \
|
|
uv sync --project "$INSTALL_ROOT"
|
|
else
|
|
sudo -u "$SERVICE_USER" -- python3 -m venv "$INSTALL_ROOT/.venv"
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet --upgrade pip
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet starlette 'uvicorn[standard]'
|
|
fi
|
|
|
|
# --- 4. systemd --------------------------------------------------------
|
|
log "installing systemd unit"
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-receiver.service" \
|
|
/etc/systemd/system/cis490-receiver.service
|
|
systemctl daemon-reload
|
|
|
|
# --- 5. config template (only on first install) -----------------------
|
|
if [[ ! -f "$ETC_ROOT/receiver.toml" ]]; then
|
|
log "writing $ETC_ROOT/receiver.toml (template)"
|
|
install -m 0640 -o root -g "$SERVICE_USER" \
|
|
"$REPO_ROOT/etc/receiver.toml.example" "$ETC_ROOT/receiver.toml"
|
|
log ""
|
|
log "FIRST-INSTALL NEXT STEPS:"
|
|
log " 1. Verify $ETC_ROOT/receiver.toml paths."
|
|
log " 2. Add a collector.wg block to your spectral/caddy config."
|
|
log " Example:"
|
|
log " collector.wg {"
|
|
log " tls internal"
|
|
log " reverse_proxy 127.0.0.1:8443"
|
|
log " }"
|
|
log " (mTLS to clients is enforced by the wg-pki CA bundle on"
|
|
log " the receiver side once leaf certs are issued.)"
|
|
log " 3. Open the WG-side port via iptmonads."
|
|
log " 4. systemctl enable --now cis490-receiver"
|
|
log " 5. From a lab host: cis490-shipper --ping"
|
|
else
|
|
log "$ETC_ROOT/receiver.toml exists; leaving in place"
|
|
fi
|
|
|
|
log "receiver install complete."
|