1b6c7b2f4a
This is the chunk that makes "real data" actually flow on multiple
hosts in parallel. End-to-end pipe was up at 613c6fa / 2579683; now
the lab-host side has the diversity + concurrency it needs.
Collectors landed:
collectors/qmp.py — source 2 (oracle). Tiny synchronous QMP
client + row builder + run loop. Tolerates
older qemu without query-stats.
collectors/guest_agent.py — source 5 (deployable). Reads the
virtio-serial host-side socket, parses
agent JSON-lines, re-stamps to the host
monotonic clock, persists.
collectors/pcap.py — source 4 (deployable). tcpdump capture
+ pure-Python pcap reader + 100 ms
netflow.jsonl bucketizer. Decodes
Ethernet/IPv4/TCP/UDP enough for the
schema in docs/data-model.md.
In-guest agent:
vm/guest-agent/cis490_agent.py — stdlib-only Python agent. Reads
/proc/{stat,meminfo,loadavg,net/dev,net/tcp*}, top-N RSS procs,
thermal. Writes JSON-lines to /dev/virtio-ports/cis490.guest.agent.
tools/build_cidata.py — embeds the agent + an OpenRC service into
user-data so first boot of the Alpine cidata image auto-starts it.
Launchers:
vm/launch_demo.sh / launch_target.sh — second virtio-serial port for
the agent socket; SLOT env support so multiple VMs run without
socket / port collisions; PORT_BASE on launch_target so multiple
target VMs hostfwd different host ports.
vm/setup_bridge.sh — creates host-only br-malware (10.200.0.1/24,
no NAT). Idempotent.
Fleet:
orchestrator/fleet.py — capacity detector (cores / RAM / load
headroom) + concurrent-slot runner. Per-slot ENV selects the
sample. FleetCapacity dataclass round-trips into meta.json so
"this episode ran with 6 concurrent VMs" is auditable post-hoc.
tools/run_fleet.py — CLI: --capacity report; --waves N runs N
waves of (max_concurrent) episodes each, every slot with a
different sample.
etc/cis490-orchestrator.service — now drives the fleet runner with
Restart=always so each invocation runs one wave and respawns,
giving a continuous stream.
Samples:
samples/manifest.toml — six profiles spanning the five major
behaviour shapes. Each entry is real OR mimic (sha256 distinguishes).
samples/manifest.py — strict TOML loader (rejects dups, unknown
categories) + deterministic select(host_id, slot, episode_index)
so different hosts on the network walk the catalog in different
orders without any coordinator.
EpisodeRunner:
orchestrator/episode.py — optional qmp_socket + guest_agent_socket
fields on EpisodeConfig; when set, additional collector threads
run alongside proc_qemu. EpisodeResult now carries rows_qmp +
rows_guest counters.
Tier-3 setup automation:
scripts/install-msfrpcd.sh — installs metasploit-framework where
the package manager has it, generates a strong password into
/etc/cis490/msfrpc.env, drops a hardened systemd unit bound to
127.0.0.1:55553. After this, run_tier3_demo.py works zero-touch
once MSFRPC_PASSWORD is sourced.
scripts/fetch-metasploitable2.sh — accepts IMAGE_URL + IMAGE_SHA256
from the operator (Rapid7 download is registration-walled), pulls,
verifies, converts vmdk → qcow2, lands at vm/images/.
Tests: 82 pass (was 51). New suites:
tests/test_qmp.py — fake QMP server, capability handshake,
blockstats, async-event interleaving,
5-failure backoff
tests/test_guest_agent.py — fake virtio socket, JSON-lines read +
re-stamp, malformed-line tolerance
tests/test_pcap.py — synthetic pcap with TCP/UDP/ARP frames,
bucketize correctness across windows
tests/test_fleet.py — capacity math (8-core idle / low-RAM /
high-load / Pi5 / 1-core box), manifest
selection determinism + diversity
What's queued for the next commit (already discussed in convo):
- MSFExploitDriver v2: map sample.profile → distinct in-session
workload so Tier-3 episodes don't all produce the same yes-loop
envelope. Critical for ML to learn varied malware shapes.
- Real-sample fetch from MalwareBazaar by sha256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
56 lines
1.9 KiB
Bash
Executable file
56 lines
1.9 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Create the host-only ``br-malware`` bridge for Tier-3+ episodes.
|
|
#
|
|
# Properties (from docs/architecture.md):
|
|
# - Bridge address 10.200.0.1/24 on the host side.
|
|
# - NO NAT, NO route, NO DNS — guests cannot reach the host or the
|
|
# internet. The bridge only carries traffic between the host and
|
|
# the guests on it.
|
|
# - Lab-host and target VMs both attach via tap devices created by
|
|
# the launcher.
|
|
#
|
|
# Run as root, ONCE per host. Idempotent — re-running is safe.
|
|
|
|
set -euo pipefail
|
|
|
|
BRIDGE="${BRIDGE:-br-malware}"
|
|
BRIDGE_IP="${BRIDGE_IP:-10.200.0.1/24}"
|
|
|
|
log() { printf '[setup_bridge] %s\n' "$*" >&2; }
|
|
|
|
[[ $EUID -eq 0 ]] || { log "must run as root"; exit 1; }
|
|
|
|
if ! command -v ip >/dev/null; then
|
|
log "iproute2 (`ip`) is required"
|
|
exit 1
|
|
fi
|
|
|
|
if ! ip link show "$BRIDGE" >/dev/null 2>&1; then
|
|
log "creating bridge $BRIDGE"
|
|
ip link add name "$BRIDGE" type bridge
|
|
# Disable spanning-tree on the host-only bridge — it isn't needed
|
|
# and adds startup delay.
|
|
ip link set "$BRIDGE" type bridge stp_state 0
|
|
fi
|
|
|
|
ip link set "$BRIDGE" up
|
|
|
|
# Add the host-side address if not already there.
|
|
if ! ip -4 addr show dev "$BRIDGE" | grep -q "${BRIDGE_IP%%/*}"; then
|
|
log "adding $BRIDGE_IP to $BRIDGE"
|
|
ip addr add "$BRIDGE_IP" dev "$BRIDGE"
|
|
fi
|
|
|
|
# Make sure the kernel does NOT forward between this bridge and any
|
|
# other interface. We don't want a misconfigured net.ipv4.ip_forward
|
|
# to leak the malware bridge to the LAN.
|
|
if [[ "$(cat /proc/sys/net/ipv4/ip_forward)" == "1" ]]; then
|
|
log "WARNING: net.ipv4.ip_forward=1 — make sure iptmonads / nftables"
|
|
log "blocks traffic from $BRIDGE to non-loopback devices."
|
|
fi
|
|
|
|
log "bridge ready: $(ip -4 -br addr show "$BRIDGE")"
|
|
log ""
|
|
log "Launchers can now opt into tap+bridge mode by setting:"
|
|
log " BRIDGE=$BRIDGE (tells launch_target.sh to attach a tap to this bridge)"
|
|
log "Default launcher behaviour stays SLIRP usermode for simplicity."
|