1b6c7b2f4a
This is the chunk that makes "real data" actually flow on multiple
hosts in parallel. End-to-end pipe was up at 613c6fa / 2579683; now
the lab-host side has the diversity + concurrency it needs.
Collectors landed:
collectors/qmp.py — source 2 (oracle). Tiny synchronous QMP
client + row builder + run loop. Tolerates
older qemu without query-stats.
collectors/guest_agent.py — source 5 (deployable). Reads the
virtio-serial host-side socket, parses
agent JSON-lines, re-stamps to the host
monotonic clock, persists.
collectors/pcap.py — source 4 (deployable). tcpdump capture
+ pure-Python pcap reader + 100 ms
netflow.jsonl bucketizer. Decodes
Ethernet/IPv4/TCP/UDP enough for the
schema in docs/data-model.md.
In-guest agent:
vm/guest-agent/cis490_agent.py — stdlib-only Python agent. Reads
/proc/{stat,meminfo,loadavg,net/dev,net/tcp*}, top-N RSS procs,
thermal. Writes JSON-lines to /dev/virtio-ports/cis490.guest.agent.
tools/build_cidata.py — embeds the agent + an OpenRC service into
user-data so first boot of the Alpine cidata image auto-starts it.
Launchers:
vm/launch_demo.sh / launch_target.sh — second virtio-serial port for
the agent socket; SLOT env support so multiple VMs run without
socket / port collisions; PORT_BASE on launch_target so multiple
target VMs hostfwd different host ports.
vm/setup_bridge.sh — creates host-only br-malware (10.200.0.1/24,
no NAT). Idempotent.
Fleet:
orchestrator/fleet.py — capacity detector (cores / RAM / load
headroom) + concurrent-slot runner. Per-slot ENV selects the
sample. FleetCapacity dataclass round-trips into meta.json so
"this episode ran with 6 concurrent VMs" is auditable post-hoc.
tools/run_fleet.py — CLI: --capacity report; --waves N runs N
waves of (max_concurrent) episodes each, every slot with a
different sample.
etc/cis490-orchestrator.service — now drives the fleet runner with
Restart=always so each invocation runs one wave and respawns,
giving a continuous stream.
Samples:
samples/manifest.toml — six profiles spanning the five major
behaviour shapes. Each entry is real OR mimic (sha256 distinguishes).
samples/manifest.py — strict TOML loader (rejects dups, unknown
categories) + deterministic select(host_id, slot, episode_index)
so different hosts on the network walk the catalog in different
orders without any coordinator.
EpisodeRunner:
orchestrator/episode.py — optional qmp_socket + guest_agent_socket
fields on EpisodeConfig; when set, additional collector threads
run alongside proc_qemu. EpisodeResult now carries rows_qmp +
rows_guest counters.
Tier-3 setup automation:
scripts/install-msfrpcd.sh — installs metasploit-framework where
the package manager has it, generates a strong password into
/etc/cis490/msfrpc.env, drops a hardened systemd unit bound to
127.0.0.1:55553. After this, run_tier3_demo.py works zero-touch
once MSFRPC_PASSWORD is sourced.
scripts/fetch-metasploitable2.sh — accepts IMAGE_URL + IMAGE_SHA256
from the operator (Rapid7 download is registration-walled), pulls,
verifies, converts vmdk → qcow2, lands at vm/images/.
Tests: 82 pass (was 51). New suites:
tests/test_qmp.py — fake QMP server, capability handshake,
blockstats, async-event interleaving,
5-failure backoff
tests/test_guest_agent.py — fake virtio socket, JSON-lines read +
re-stamp, malformed-line tolerance
tests/test_pcap.py — synthetic pcap with TCP/UDP/ARP frames,
bucketize correctness across windows
tests/test_fleet.py — capacity math (8-core idle / low-RAM /
high-load / Pi5 / 1-core box), manifest
selection determinism + diversity
What's queued for the next commit (already discussed in convo):
- MSFExploitDriver v2: map sample.profile → distinct in-session
workload so Tier-3 episodes don't all produce the same yes-loop
envelope. Critical for ML to learn varied malware shapes.
- Real-sample fetch from MalwareBazaar by sha256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
66 lines
2.4 KiB
Bash
Executable file
66 lines
2.4 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Boot the Cirros qcow2 under KVM with QMP and a monitor socket exposed.
|
|
#
|
|
# This is the v0 VM launcher for phase 2: validate that the orchestrator
|
|
# and host /proc collector work against a real qemu-system process. No
|
|
# host-only bridge yet, no exploit driver, no payload — just boot and
|
|
# idle. We add the bridge and exploit machinery in later phases.
|
|
#
|
|
# Run dir is exported so the orchestrator can read the qemu pid:
|
|
# $RUN_DIR/qemu.pid
|
|
# $RUN_DIR/qmp.sock
|
|
# $RUN_DIR/monitor.sock
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
IMAGE="${IMAGE:-$REPO_ROOT/vm/images/alpine-baseline.qcow2}"
|
|
CIDATA="${CIDATA:-$REPO_ROOT/vm/images/cidata.iso}"
|
|
# SLOT lets the fleet runner spin up N concurrent VMs without socket /
|
|
# port collisions. Default RUN_DIR + ssh hostfwd port keep single-VM
|
|
# usage unchanged.
|
|
SLOT="${SLOT:-0}"
|
|
RUN_DIR="${RUN_DIR:-/tmp/cis490-vm-$SLOT}"
|
|
SSH_PORT="${SSH_PORT:-$((2222 + SLOT))}"
|
|
|
|
mkdir -p "$RUN_DIR"
|
|
QMP_SOCK="$RUN_DIR/qmp.sock"
|
|
MON_SOCK="$RUN_DIR/monitor.sock"
|
|
PID_FILE="$RUN_DIR/qemu.pid"
|
|
|
|
if [[ ! -f "$IMAGE" ]]; then
|
|
echo "no image at $IMAGE" >&2
|
|
exit 1
|
|
fi
|
|
if [[ ! -f "$CIDATA" ]]; then
|
|
echo "no cidata at $CIDATA — build it with: uv run python tools/build_cidata.py $CIDATA" >&2
|
|
exit 1
|
|
fi
|
|
|
|
AGENT_SOCK="$RUN_DIR/agent.sock"
|
|
|
|
# snapshot=on routes guest writes through a temporary overlay so the qcow2
|
|
# on disk is never mutated — every boot starts from the same bytes.
|
|
#
|
|
# Second virtio-serial port (cis490.guest.agent) carries telemetry
|
|
# from the in-guest agent. Surfaces inside the guest at
|
|
# /dev/virtio-ports/cis490.guest.agent and on the host at $AGENT_SOCK.
|
|
exec qemu-system-x86_64 \
|
|
-name cis490-vm \
|
|
-machine q35,accel=kvm \
|
|
-cpu host \
|
|
-smp 1,sockets=1,cores=1,threads=1 \
|
|
-m 256 \
|
|
-drive file="$IMAGE",format=qcow2,if=virtio,snapshot=on \
|
|
-drive file="$CIDATA",format=raw,if=virtio,readonly=on \
|
|
-netdev user,id=n0,hostfwd=tcp:127.0.0.1:"$SSH_PORT"-:22 \
|
|
-device virtio-net-pci,netdev=n0 \
|
|
-device virtio-serial-pci,id=cis490vs0 \
|
|
-chardev socket,id=cis490agent,path="$AGENT_SOCK",server=on,wait=off \
|
|
-device virtserialport,chardev=cis490agent,name=cis490.guest.agent \
|
|
-nographic \
|
|
-serial unix:"$RUN_DIR/serial.sock",server=on,wait=off \
|
|
-monitor unix:"$MON_SOCK",server=on,wait=off \
|
|
-qmp unix:"$QMP_SOCK",server=on,wait=off \
|
|
-pidfile "$PID_FILE" \
|
|
-display none
|