a88ac83db0
Wraps the gaps surfaced in the "what is not implemented" audit so the
fleet really is shippable end-to-end. Verified live on the Pi:
- cis490-shipper --ping → HTTP 200 through Caddy + mTLS via the
new wg-pki client CA leaf
- real episode dir → tar+zstd → PUT → HTTP 201 stored
- re-ship same bytes → 200 (idempotent)
- re-ship different bytes under same id → 409 (conflict)
Changes:
orchestrator/episode.py
- EpisodeConfig.revert_at_start / revert_at_end (Tier 0+ snapshot/
revert per docs/architecture.md). When set + qmp_socket present,
EpisodeRunner issues loadvm <snapshot_name> and emits
snapshot_revert / snapshot_revert_failed events on the same
monotonic clock as everything else.
collectors/qmp.py
- savevm() / loadvm() helpers using human-monitor-command, plus a
test against the fake QMP server.
exploits/workloads.py
- chunked_real_binary_upload() returns a ChunkedUpload plan: 8 KiB
base64 chunks (~6 KiB binary each) so msfrpc never sees a buffer-
busting payload. Includes a finalize step that sha256-verifies on
the guest before exec.
- real_binary_workload() now wraps the chunked plan for backwards
compat with single-shot callers.
exploits/driver.py
- Tier-4 dispatch walks the chunked plan in MSFExploitDriver:
each chunk is a separate session_shell_write; finalize verifies;
exec only runs on sha-ok. New events: real_binary_upload_begin,
real_binary_verify, real_binary_aborted.
etc/cis490-orchestrator.service
- Reads /etc/cis490/lab-host.env (FLEET_HOST_ID + optional BRIDGE).
- Grants AmbientCapabilities CAP_NET_RAW (tcpdump for source 4) +
CAP_SYS_ADMIN + CAP_PERFMON (perf for source 3) so collectors
work under hardening.
scripts/install-lab-host.sh
- Writes /etc/cis490/lab-host.env on first install with FLEET_HOST_ID
defaulting to `hostname -s`.
- Best-effort: fetches the Alpine baseline qcow2 (sha512-pinned) and
builds cidata.iso with the in-guest agent embedded; symlinks both
into /opt/cis490/vm/images/ so launchers find them.
scripts/fetch-alpine-baseline.sh
- Idempotent fetcher for the Alpine 3.21 cloud-init nocloud qcow2
matching the sha512 in docs/sources.md.
tools/plot_envelope.py
- Rebuilt to render whatever telemetry the episode dir contains:
proc → QMP block ops → perf IPC/miss-rate → bridge pkts/SYNs →
guest agent load/mem. Missing sources are silently skipped.
tools/index_reader.py
- cis490-index CLI: filter receiver's index.jsonl by host / sample
/ time range, sort, count-by group. Closest thing to a query
interface until we stand up Postgres/Timescale.
samples/README.md
- Rewritten to match the new manifest schema, the kind=real vs mimic
split, the per-(host, slot, ep) selection mechanic, and the
chunked-upload safety story.
Tests: 106 pass (was 102). New cases:
- test_qmp.py — savevm + loadvm (HMP wrapper + error path)
- test_tier4.py — chunked plan splitting, sha-pinned finalize,
end-to-end driver walks all chunks + verify + exec via the fake
msfrpc client
Closes the "what is not implemented" punch list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
149 lines
6.1 KiB
Bash
Executable file
149 lines
6.1 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Install / refresh the CIS490 lab-host role.
|
|
#
|
|
# Idempotent — safe to re-run after `git pull`. Does NOT enroll the
|
|
# host into WireGuard (that's wg-enroll's job, run separately and
|
|
# *first*) and does NOT mint TLS certs (that's wg-pki's job).
|
|
#
|
|
# Steps:
|
|
# 1. Verify prereqs (KVM, zstd, qemu, python3.11+, systemd).
|
|
# 2. Create the cis490 service user + /var/lib/cis490 layout.
|
|
# 3. Sync the repo into /opt/cis490 and build a uv-managed venv.
|
|
# 4. Install systemd units from etc/.
|
|
# 5. Drop /etc/cis490/lab-host.toml (only on first install).
|
|
#
|
|
# Operator finishes by:
|
|
# - editing /etc/cis490/lab-host.toml (host_id, receiver URL, certs)
|
|
# - placing leaf certs at /etc/cis490/certs/{lab-host.pem,key,wg-ca.pem}
|
|
# - `systemctl enable --now cis490-shipper`
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
INSTALL_ROOT="${INSTALL_ROOT:-/opt/cis490}"
|
|
DATA_ROOT="${DATA_ROOT:-/var/lib/cis490}"
|
|
ETC_ROOT="${ETC_ROOT:-/etc/cis490}"
|
|
SERVICE_USER="${SERVICE_USER:-cis490}"
|
|
|
|
log() { printf '[install-lab-host] %s\n' "$*" >&2; }
|
|
die() { log "FATAL: $*"; exit 1; }
|
|
|
|
# --- 1. prereqs --------------------------------------------------------
|
|
log "checking prereqs"
|
|
|
|
if [[ $EUID -ne 0 ]]; then
|
|
die "must run as root (writes to /opt, /etc, /var/lib, and systemd)"
|
|
fi
|
|
command -v systemctl >/dev/null || die "systemd not found"
|
|
command -v qemu-system-x86_64 >/dev/null || die "qemu-system-x86_64 not on PATH"
|
|
command -v zstd >/dev/null || die "zstd not on PATH (apt install zstd)"
|
|
[[ -e /dev/kvm ]] || die "/dev/kvm missing — KVM not available"
|
|
|
|
# uv is preferred (lockfile-driven). Fall back to system pip if absent.
|
|
USE_UV=0
|
|
if command -v uv >/dev/null; then USE_UV=1; fi
|
|
|
|
# --- 2. user + layout --------------------------------------------------
|
|
log "ensuring service user $SERVICE_USER"
|
|
if ! id -u "$SERVICE_USER" >/dev/null 2>&1; then
|
|
useradd --system --no-create-home --shell /usr/sbin/nologin \
|
|
--home-dir "$INSTALL_ROOT" "$SERVICE_USER"
|
|
fi
|
|
# kvm group lets the service spawn VMs.
|
|
if getent group kvm >/dev/null 2>&1; then
|
|
usermod -a -G kvm "$SERVICE_USER" || true
|
|
fi
|
|
|
|
install -d -o root -g root -m 0755 "$ETC_ROOT" "$ETC_ROOT/certs"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 \
|
|
"$DATA_ROOT" "$DATA_ROOT/data" \
|
|
"$DATA_ROOT/data/episodes" "$DATA_ROOT/data/outbox" \
|
|
"$DATA_ROOT/data/shipped" "$DATA_ROOT/data/queue" \
|
|
"$DATA_ROOT/samples" "$DATA_ROOT/samples/store" \
|
|
"$DATA_ROOT/vm" "$DATA_ROOT/vm/images"
|
|
|
|
# --- 3. repo + venv ----------------------------------------------------
|
|
log "syncing repo into $INSTALL_ROOT"
|
|
install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0755 "$INSTALL_ROOT"
|
|
# We use a clean cp -aT rather than rsync to avoid an extra dep.
|
|
cp -aT "$REPO_ROOT" "$INSTALL_ROOT"
|
|
chown -R "$SERVICE_USER":"$SERVICE_USER" "$INSTALL_ROOT"
|
|
|
|
log "building venv"
|
|
if [[ "$USE_UV" -eq 1 ]]; then
|
|
sudo -u "$SERVICE_USER" -- env HOME="$INSTALL_ROOT" \
|
|
uv sync --project "$INSTALL_ROOT"
|
|
else
|
|
sudo -u "$SERVICE_USER" -- python3 -m venv "$INSTALL_ROOT/.venv"
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet --upgrade pip
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/pip" install \
|
|
--quiet starlette 'uvicorn[standard]' httpx msgpack
|
|
fi
|
|
|
|
# --- 4. systemd --------------------------------------------------------
|
|
log "installing systemd units"
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-shipper.service" \
|
|
/etc/systemd/system/cis490-shipper.service
|
|
install -m 0644 "$REPO_ROOT/etc/cis490-orchestrator.service" \
|
|
/etc/systemd/system/cis490-orchestrator.service
|
|
systemctl daemon-reload
|
|
|
|
# --- 5. config template (only on first install) -----------------------
|
|
if [[ ! -f "$ETC_ROOT/lab-host.toml" ]]; then
|
|
log "writing $ETC_ROOT/lab-host.toml (template)"
|
|
install -m 0640 -o root -g "$SERVICE_USER" \
|
|
"$REPO_ROOT/etc/lab-host.toml.example" "$ETC_ROOT/lab-host.toml"
|
|
NEW_INSTALL=1
|
|
else
|
|
log "$ETC_ROOT/lab-host.toml exists; leaving in place"
|
|
NEW_INSTALL=0
|
|
fi
|
|
|
|
# --- 6. orchestrator env file (read by cis490-orchestrator.service) ----
|
|
ENV_FILE="$ETC_ROOT/lab-host.env"
|
|
DEFAULT_HOST_ID="$(hostname -s)"
|
|
if [[ ! -f "$ENV_FILE" ]]; then
|
|
log "writing $ENV_FILE (host_id defaults to $DEFAULT_HOST_ID — edit if you want something else)"
|
|
install -m 0640 -o root -g "$SERVICE_USER" /dev/stdin "$ENV_FILE" <<EOF
|
|
# Read by cis490-orchestrator.service. Override per-host as needed.
|
|
FLEET_HOST_ID=$DEFAULT_HOST_ID
|
|
# BRIDGE=br-malware # uncomment to enable source 4 pcap capture
|
|
EOF
|
|
fi
|
|
|
|
# --- 7. baseline VM image + cidata (best-effort) -----------------------
|
|
ALPINE_IMG="$DATA_ROOT/vm/images/alpine-baseline.qcow2"
|
|
CIDATA_ISO="$DATA_ROOT/vm/images/cidata.iso"
|
|
if [[ ! -f "$ALPINE_IMG" ]]; then
|
|
if "$REPO_ROOT/scripts/fetch-alpine-baseline.sh" "$ALPINE_IMG"; then
|
|
log "fetched Alpine baseline -> $ALPINE_IMG"
|
|
else
|
|
log "WARN: Alpine baseline fetch failed; drop a qcow2 at $ALPINE_IMG manually"
|
|
fi
|
|
fi
|
|
if [[ -f "$ALPINE_IMG" && ! -f "$CIDATA_ISO" ]]; then
|
|
log "building cidata.iso (in-guest agent embedded)"
|
|
sudo -u "$SERVICE_USER" -- "$INSTALL_ROOT/.venv/bin/python" \
|
|
"$INSTALL_ROOT/tools/build_cidata.py" "$CIDATA_ISO" || \
|
|
log "WARN: cidata build failed; run tools/build_cidata.py manually"
|
|
fi
|
|
# Symlink the canonical paths the launchers look at, when missing.
|
|
ln -sf "$ALPINE_IMG" "$INSTALL_ROOT/vm/images/alpine-baseline.qcow2" 2>/dev/null || true
|
|
ln -sf "$CIDATA_ISO" "$INSTALL_ROOT/vm/images/cidata.iso" 2>/dev/null || true
|
|
|
|
if [[ "$NEW_INSTALL" == "1" ]]; then
|
|
log ""
|
|
log "FIRST-INSTALL NEXT STEPS:"
|
|
log " 1. Edit $ETC_ROOT/lab-host.toml — set host_id and receiver URL."
|
|
log " 2. Place TLS material at:"
|
|
log " $ETC_ROOT/certs/wg-ca.pem"
|
|
log " $ETC_ROOT/certs/lab-host.pem"
|
|
log " $ETC_ROOT/certs/lab-host.key (mode 0600, owner $SERVICE_USER)"
|
|
log " 3. Smoke-test the receiver pipe:"
|
|
log " sudo -u $SERVICE_USER $INSTALL_ROOT/.venv/bin/python -m shipper \\"
|
|
log " --config $ETC_ROOT/lab-host.toml --ping"
|
|
log " 4. systemctl enable --now cis490-shipper cis490-orchestrator"
|
|
fi
|
|
|
|
log "lab-host install complete."
|