1b6c7b2f4a
This is the chunk that makes "real data" actually flow on multiple
hosts in parallel. End-to-end pipe was up at 613c6fa / 2579683; now
the lab-host side has the diversity + concurrency it needs.
Collectors landed:
collectors/qmp.py — source 2 (oracle). Tiny synchronous QMP
client + row builder + run loop. Tolerates
older qemu without query-stats.
collectors/guest_agent.py — source 5 (deployable). Reads the
virtio-serial host-side socket, parses
agent JSON-lines, re-stamps to the host
monotonic clock, persists.
collectors/pcap.py — source 4 (deployable). tcpdump capture
+ pure-Python pcap reader + 100 ms
netflow.jsonl bucketizer. Decodes
Ethernet/IPv4/TCP/UDP enough for the
schema in docs/data-model.md.
In-guest agent:
vm/guest-agent/cis490_agent.py — stdlib-only Python agent. Reads
/proc/{stat,meminfo,loadavg,net/dev,net/tcp*}, top-N RSS procs,
thermal. Writes JSON-lines to /dev/virtio-ports/cis490.guest.agent.
tools/build_cidata.py — embeds the agent + an OpenRC service into
user-data so first boot of the Alpine cidata image auto-starts it.
Launchers:
vm/launch_demo.sh / launch_target.sh — second virtio-serial port for
the agent socket; SLOT env support so multiple VMs run without
socket / port collisions; PORT_BASE on launch_target so multiple
target VMs hostfwd different host ports.
vm/setup_bridge.sh — creates host-only br-malware (10.200.0.1/24,
no NAT). Idempotent.
Fleet:
orchestrator/fleet.py — capacity detector (cores / RAM / load
headroom) + concurrent-slot runner. Per-slot ENV selects the
sample. FleetCapacity dataclass round-trips into meta.json so
"this episode ran with 6 concurrent VMs" is auditable post-hoc.
tools/run_fleet.py — CLI: --capacity report; --waves N runs N
waves of (max_concurrent) episodes each, every slot with a
different sample.
etc/cis490-orchestrator.service — now drives the fleet runner with
Restart=always so each invocation runs one wave and respawns,
giving a continuous stream.
Samples:
samples/manifest.toml — six profiles spanning the five major
behaviour shapes. Each entry is real OR mimic (sha256 distinguishes).
samples/manifest.py — strict TOML loader (rejects dups, unknown
categories) + deterministic select(host_id, slot, episode_index)
so different hosts on the network walk the catalog in different
orders without any coordinator.
EpisodeRunner:
orchestrator/episode.py — optional qmp_socket + guest_agent_socket
fields on EpisodeConfig; when set, additional collector threads
run alongside proc_qemu. EpisodeResult now carries rows_qmp +
rows_guest counters.
Tier-3 setup automation:
scripts/install-msfrpcd.sh — installs metasploit-framework where
the package manager has it, generates a strong password into
/etc/cis490/msfrpc.env, drops a hardened systemd unit bound to
127.0.0.1:55553. After this, run_tier3_demo.py works zero-touch
once MSFRPC_PASSWORD is sourced.
scripts/fetch-metasploitable2.sh — accepts IMAGE_URL + IMAGE_SHA256
from the operator (Rapid7 download is registration-walled), pulls,
verifies, converts vmdk → qcow2, lands at vm/images/.
Tests: 82 pass (was 51). New suites:
tests/test_qmp.py — fake QMP server, capability handshake,
blockstats, async-event interleaving,
5-failure backoff
tests/test_guest_agent.py — fake virtio socket, JSON-lines read +
re-stamp, malformed-line tolerance
tests/test_pcap.py — synthetic pcap with TCP/UDP/ARP frames,
bucketize correctness across windows
tests/test_fleet.py — capacity math (8-core idle / low-RAM /
high-load / Pi5 / 1-core box), manifest
selection determinism + diversity
What's queued for the next commit (already discussed in convo):
- MSFExploitDriver v2: map sample.profile → distinct in-session
workload so Tier-3 episodes don't all produce the same yes-loop
envelope. Critical for ML to learn varied malware shapes.
- Real-sample fetch from MalwareBazaar by sha256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
107 lines
3.8 KiB
Bash
Executable file
107 lines
3.8 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Boot the Tier-3 *target* VM (the intentionally-vulnerable guest the
|
|
# exploit fires against). Companion to ``launch_demo.sh``, which boots
|
|
# the *idle* Alpine guest used in Tiers 1-2.
|
|
#
|
|
# Networking note: this launcher uses SLIRP usermode networking with
|
|
# ``restrict=on`` plus an explicit ``hostfwd`` for each vulnerable port.
|
|
# That gives us:
|
|
# - the host can reach the guest's services (for msfrpcd + the
|
|
# exploit module to drive ``RHOSTS=127.0.0.1``)
|
|
# - the guest cannot reach the host or the internet (no NAT exit)
|
|
#
|
|
# The host-only ``br-malware`` bridge described in docs/architecture.md
|
|
# replaces SLIRP once the bridge-side pcap collector (source 4) lands —
|
|
# at which point payloads with ``reverse_tcp`` callbacks become viable
|
|
# too. Until then, we restrict module choices to ones that return a
|
|
# shell on the same socket they exploit (e.g. vsftpd_234_backdoor).
|
|
#
|
|
# Run-dir contract (read by run_tier3_demo.py):
|
|
# $RUN_DIR/qemu.pid
|
|
# $RUN_DIR/qmp.sock
|
|
# $RUN_DIR/monitor.sock
|
|
# $RUN_DIR/serial.sock
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
IMAGE="${IMAGE:-$REPO_ROOT/vm/images/metasploitable2.qcow2}"
|
|
SLOT="${SLOT:-0}"
|
|
RUN_DIR="${RUN_DIR:-/tmp/cis490-target-$SLOT}"
|
|
RAM_MIB="${RAM_MIB:-512}"
|
|
# Ports the host should forward to the guest. Comma-separated host:guest pairs.
|
|
# Default covers the vsftpd module's RPORT. Slot offset makes per-VM
|
|
# fleet runs collision-free (slot 0 → 21, slot 1 → 121, slot 2 → 221, ...).
|
|
PORT_BASE="${PORT_BASE:-$((21 + SLOT * 100))}"
|
|
TARGET_PORTS="${TARGET_PORTS:-${PORT_BASE}:21}"
|
|
# KVM if the host can take it; otherwise fall back to TCG. Cross-arch
|
|
# images (Metasploitable2 is x86-only) on aarch64 hosts will need TCG.
|
|
ACCEL="${ACCEL:-}"
|
|
|
|
mkdir -p "$RUN_DIR"
|
|
QMP_SOCK="$RUN_DIR/qmp.sock"
|
|
MON_SOCK="$RUN_DIR/monitor.sock"
|
|
PID_FILE="$RUN_DIR/qemu.pid"
|
|
SERIAL_SOCK="$RUN_DIR/serial.sock"
|
|
|
|
if [[ ! -f "$IMAGE" ]]; then
|
|
cat >&2 <<EOF
|
|
no target image at $IMAGE
|
|
|
|
Drop a vulnerable Linux qcow2 there. The canonical choice is
|
|
Metasploitable2 — see docs/sources.md for the download + sha256.
|
|
|
|
If the image is x86 and your host is not, set ACCEL=tcg explicitly.
|
|
EOF
|
|
exit 1
|
|
fi
|
|
|
|
# Build the netdev string with one hostfwd= per requested port pair.
|
|
NETDEV="user,id=n0,restrict=on"
|
|
IFS=',' read -ra _PAIRS <<< "$TARGET_PORTS"
|
|
for pair in "${_PAIRS[@]}"; do
|
|
host_port="${pair%%:*}"
|
|
guest_port="${pair##*:}"
|
|
NETDEV+=",hostfwd=tcp:127.0.0.1:${host_port}-:${guest_port}"
|
|
done
|
|
|
|
# Pick acceleration: explicit override wins; otherwise use KVM if the
|
|
# device is present, else TCG.
|
|
if [[ -z "$ACCEL" ]]; then
|
|
if [[ -e /dev/kvm && -r /dev/kvm && -w /dev/kvm ]]; then
|
|
ACCEL="kvm"
|
|
else
|
|
ACCEL="tcg"
|
|
fi
|
|
fi
|
|
|
|
CPU_FLAGS=()
|
|
if [[ "$ACCEL" == "kvm" ]]; then
|
|
CPU_FLAGS=(-cpu host)
|
|
fi
|
|
|
|
AGENT_SOCK="$RUN_DIR/agent.sock"
|
|
|
|
# snapshot=on so the qcow2 is never mutated — every boot is identical.
|
|
# Second virtio-serial port carries the in-guest agent's telemetry to
|
|
# the host (see vm/guest-agent/). Targets without the agent installed
|
|
# (e.g. unmodified Metasploitable2) leave the device unused — the
|
|
# host-side collector simply gets no rows. Harmless.
|
|
exec qemu-system-x86_64 \
|
|
-name cis490-target \
|
|
-machine q35,accel="$ACCEL" \
|
|
"${CPU_FLAGS[@]}" \
|
|
-smp 1,sockets=1,cores=1,threads=1 \
|
|
-m "$RAM_MIB" \
|
|
-drive file="$IMAGE",format=qcow2,if=virtio,snapshot=on \
|
|
-netdev "$NETDEV" \
|
|
-device virtio-net-pci,netdev=n0 \
|
|
-device virtio-serial-pci,id=cis490vs0 \
|
|
-chardev socket,id=cis490agent,path="$AGENT_SOCK",server=on,wait=off \
|
|
-device virtserialport,chardev=cis490agent,name=cis490.guest.agent \
|
|
-nographic \
|
|
-serial unix:"$SERIAL_SOCK",server=on,wait=off \
|
|
-monitor unix:"$MON_SOCK",server=on,wait=off \
|
|
-qmp unix:"$QMP_SOCK",server=on,wait=off \
|
|
-pidfile "$PID_FILE" \
|
|
-display none
|