1b6c7b2f4a
This is the chunk that makes "real data" actually flow on multiple
hosts in parallel. End-to-end pipe was up at 613c6fa / 2579683; now
the lab-host side has the diversity + concurrency it needs.
Collectors landed:
collectors/qmp.py — source 2 (oracle). Tiny synchronous QMP
client + row builder + run loop. Tolerates
older qemu without query-stats.
collectors/guest_agent.py — source 5 (deployable). Reads the
virtio-serial host-side socket, parses
agent JSON-lines, re-stamps to the host
monotonic clock, persists.
collectors/pcap.py — source 4 (deployable). tcpdump capture
+ pure-Python pcap reader + 100 ms
netflow.jsonl bucketizer. Decodes
Ethernet/IPv4/TCP/UDP enough for the
schema in docs/data-model.md.
In-guest agent:
vm/guest-agent/cis490_agent.py — stdlib-only Python agent. Reads
/proc/{stat,meminfo,loadavg,net/dev,net/tcp*}, top-N RSS procs,
thermal. Writes JSON-lines to /dev/virtio-ports/cis490.guest.agent.
tools/build_cidata.py — embeds the agent + an OpenRC service into
user-data so first boot of the Alpine cidata image auto-starts it.
Launchers:
vm/launch_demo.sh / launch_target.sh — second virtio-serial port for
the agent socket; SLOT env support so multiple VMs run without
socket / port collisions; PORT_BASE on launch_target so multiple
target VMs hostfwd different host ports.
vm/setup_bridge.sh — creates host-only br-malware (10.200.0.1/24,
no NAT). Idempotent.
Fleet:
orchestrator/fleet.py — capacity detector (cores / RAM / load
headroom) + concurrent-slot runner. Per-slot ENV selects the
sample. FleetCapacity dataclass round-trips into meta.json so
"this episode ran with 6 concurrent VMs" is auditable post-hoc.
tools/run_fleet.py — CLI: --capacity report; --waves N runs N
waves of (max_concurrent) episodes each, every slot with a
different sample.
etc/cis490-orchestrator.service — now drives the fleet runner with
Restart=always so each invocation runs one wave and respawns,
giving a continuous stream.
Samples:
samples/manifest.toml — six profiles spanning the five major
behaviour shapes. Each entry is real OR mimic (sha256 distinguishes).
samples/manifest.py — strict TOML loader (rejects dups, unknown
categories) + deterministic select(host_id, slot, episode_index)
so different hosts on the network walk the catalog in different
orders without any coordinator.
EpisodeRunner:
orchestrator/episode.py — optional qmp_socket + guest_agent_socket
fields on EpisodeConfig; when set, additional collector threads
run alongside proc_qemu. EpisodeResult now carries rows_qmp +
rows_guest counters.
Tier-3 setup automation:
scripts/install-msfrpcd.sh — installs metasploit-framework where
the package manager has it, generates a strong password into
/etc/cis490/msfrpc.env, drops a hardened systemd unit bound to
127.0.0.1:55553. After this, run_tier3_demo.py works zero-touch
once MSFRPC_PASSWORD is sourced.
scripts/fetch-metasploitable2.sh — accepts IMAGE_URL + IMAGE_SHA256
from the operator (Rapid7 download is registration-walled), pulls,
verifies, converts vmdk → qcow2, lands at vm/images/.
Tests: 82 pass (was 51). New suites:
tests/test_qmp.py — fake QMP server, capability handshake,
blockstats, async-event interleaving,
5-failure backoff
tests/test_guest_agent.py — fake virtio socket, JSON-lines read +
re-stamp, malformed-line tolerance
tests/test_pcap.py — synthetic pcap with TCP/UDP/ARP frames,
bucketize correctness across windows
tests/test_fleet.py — capacity math (8-core idle / low-RAM /
high-load / Pi5 / 1-core box), manifest
selection determinism + diversity
What's queued for the next commit (already discussed in convo):
- MSFExploitDriver v2: map sample.profile → distinct in-session
workload so Tier-3 episodes don't all produce the same yes-loop
envelope. Critical for ML to learn varied malware shapes.
- Real-sample fetch from MalwareBazaar by sha256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
69 lines
2.5 KiB
Bash
Executable file
69 lines
2.5 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Fetch + sha256-verify the Metasploitable2 disk image.
|
|
#
|
|
# Rapid7's official download is gated behind a registration form, so
|
|
# we accept the URL + sha256 from env vars (with sane defaults pointing
|
|
# at a public mirror). The user installs this once per lab host.
|
|
#
|
|
# Inputs (env):
|
|
# IMAGE_URL — direct download URL for the metasploitable2 archive
|
|
# IMAGE_SHA256 — expected sha256 of the archive
|
|
# OUT_DIR — where to drop the qcow2 (default vm/images/)
|
|
#
|
|
# Outputs:
|
|
# $OUT_DIR/metasploitable2.qcow2 — converted from the original VMDK
|
|
# if needed.
|
|
#
|
|
# We do NOT bake an image url+hash into the repo because the canonical
|
|
# distribution is a registration-walled zip on Rapid7. Operators must
|
|
# supply both; the rest is mechanical.
|
|
|
|
set -euo pipefail
|
|
|
|
IMAGE_URL="${IMAGE_URL:-}"
|
|
IMAGE_SHA256="${IMAGE_SHA256:-}"
|
|
OUT_DIR="${OUT_DIR:-$(cd "$(dirname "$0")/../vm/images" 2>/dev/null && pwd)}"
|
|
WORK_DIR="${WORK_DIR:-/tmp/cis490-metasploitable-fetch}"
|
|
|
|
log() { printf '[fetch-metasploitable2] %s\n' "$*" >&2; }
|
|
die() { log "FATAL: $*"; exit 1; }
|
|
|
|
[[ -n "$IMAGE_URL" ]] || die "set IMAGE_URL to the Metasploitable2 download URL"
|
|
[[ -n "$IMAGE_SHA256" ]] || die "set IMAGE_SHA256 to the expected sha256 of the archive"
|
|
|
|
mkdir -p "$OUT_DIR" "$WORK_DIR"
|
|
|
|
ARCHIVE="$WORK_DIR/$(basename "$IMAGE_URL")"
|
|
log "downloading $IMAGE_URL → $ARCHIVE"
|
|
if [[ -f "$ARCHIVE" ]]; then
|
|
log "archive already present; skipping download"
|
|
else
|
|
curl -fL --retry 3 --retry-delay 5 -o "$ARCHIVE.partial" "$IMAGE_URL"
|
|
mv "$ARCHIVE.partial" "$ARCHIVE"
|
|
fi
|
|
|
|
log "verifying sha256"
|
|
ACTUAL="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
|
|
if [[ "$ACTUAL" != "$IMAGE_SHA256" ]]; then
|
|
die "sha256 mismatch: expected $IMAGE_SHA256, got $ACTUAL"
|
|
fi
|
|
log "sha256 ok"
|
|
|
|
# Extract — handle either zip or 7z, since various mirrors choose one
|
|
# or the other.
|
|
case "$ARCHIVE" in
|
|
*.zip) ( cd "$WORK_DIR" && unzip -o "$ARCHIVE" ) ;;
|
|
*.7z|*.7zip) command -v 7z >/dev/null || die "7z not installed"; \
|
|
( cd "$WORK_DIR" && 7z x -y "$ARCHIVE" ) ;;
|
|
*) die "unsupported archive type: $ARCHIVE" ;;
|
|
esac
|
|
|
|
VMDK="$(find "$WORK_DIR" -name 'Metasploitable*.vmdk' -print -quit)"
|
|
[[ -n "$VMDK" ]] || die "no Metasploitable*.vmdk in extracted archive"
|
|
|
|
log "converting $VMDK → qcow2"
|
|
command -v qemu-img >/dev/null || die "qemu-img required (apt install qemu-utils)"
|
|
qemu-img convert -O qcow2 "$VMDK" "$OUT_DIR/metasploitable2.qcow2"
|
|
|
|
log "done: $OUT_DIR/metasploitable2.qcow2"
|
|
log "Tier-3 ready when msfrpcd is up. See scripts/install-msfrpcd.sh."
|