1b6c7b2f4a
This is the chunk that makes "real data" actually flow on multiple
hosts in parallel. End-to-end pipe was up at 613c6fa / 2579683; now
the lab-host side has the diversity + concurrency it needs.
Collectors landed:
collectors/qmp.py — source 2 (oracle). Tiny synchronous QMP
client + row builder + run loop. Tolerates
older qemu without query-stats.
collectors/guest_agent.py — source 5 (deployable). Reads the
virtio-serial host-side socket, parses
agent JSON-lines, re-stamps to the host
monotonic clock, persists.
collectors/pcap.py — source 4 (deployable). tcpdump capture
+ pure-Python pcap reader + 100 ms
netflow.jsonl bucketizer. Decodes
Ethernet/IPv4/TCP/UDP enough for the
schema in docs/data-model.md.
In-guest agent:
vm/guest-agent/cis490_agent.py — stdlib-only Python agent. Reads
/proc/{stat,meminfo,loadavg,net/dev,net/tcp*}, top-N RSS procs,
thermal. Writes JSON-lines to /dev/virtio-ports/cis490.guest.agent.
tools/build_cidata.py — embeds the agent + an OpenRC service into
user-data so first boot of the Alpine cidata image auto-starts it.
Launchers:
vm/launch_demo.sh / launch_target.sh — second virtio-serial port for
the agent socket; SLOT env support so multiple VMs run without
socket / port collisions; PORT_BASE on launch_target so multiple
target VMs hostfwd different host ports.
vm/setup_bridge.sh — creates host-only br-malware (10.200.0.1/24,
no NAT). Idempotent.
Fleet:
orchestrator/fleet.py — capacity detector (cores / RAM / load
headroom) + concurrent-slot runner. Per-slot ENV selects the
sample. FleetCapacity dataclass round-trips into meta.json so
"this episode ran with 6 concurrent VMs" is auditable post-hoc.
tools/run_fleet.py — CLI: --capacity report; --waves N runs N
waves of (max_concurrent) episodes each, every slot with a
different sample.
etc/cis490-orchestrator.service — now drives the fleet runner with
Restart=always so each invocation runs one wave and respawns,
giving a continuous stream.
Samples:
samples/manifest.toml — six profiles spanning the five major
behaviour shapes. Each entry is real OR mimic (sha256 distinguishes).
samples/manifest.py — strict TOML loader (rejects dups, unknown
categories) + deterministic select(host_id, slot, episode_index)
so different hosts on the network walk the catalog in different
orders without any coordinator.
EpisodeRunner:
orchestrator/episode.py — optional qmp_socket + guest_agent_socket
fields on EpisodeConfig; when set, additional collector threads
run alongside proc_qemu. EpisodeResult now carries rows_qmp +
rows_guest counters.
Tier-3 setup automation:
scripts/install-msfrpcd.sh — installs metasploit-framework where
the package manager has it, generates a strong password into
/etc/cis490/msfrpc.env, drops a hardened systemd unit bound to
127.0.0.1:55553. After this, run_tier3_demo.py works zero-touch
once MSFRPC_PASSWORD is sourced.
scripts/fetch-metasploitable2.sh — accepts IMAGE_URL + IMAGE_SHA256
from the operator (Rapid7 download is registration-walled), pulls,
verifies, converts vmdk → qcow2, lands at vm/images/.
Tests: 82 pass (was 51). New suites:
tests/test_qmp.py — fake QMP server, capability handshake,
blockstats, async-event interleaving,
5-failure backoff
tests/test_guest_agent.py — fake virtio socket, JSON-lines read +
re-stamp, malformed-line tolerance
tests/test_pcap.py — synthetic pcap with TCP/UDP/ARP frames,
bucketize correctness across windows
tests/test_fleet.py — capacity math (8-core idle / low-RAM /
high-load / Pi5 / 1-core box), manifest
selection determinism + diversity
What's queued for the next commit (already discussed in convo):
- MSFExploitDriver v2: map sample.profile → distinct in-session
workload so Tier-3 episodes don't all produce the same yes-loop
envelope. Critical for ML to learn varied malware shapes.
- Real-sample fetch from MalwareBazaar by sha256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
61 lines
2.1 KiB
TOML
61 lines
2.1 KiB
TOML
# Sample manifest — what each fleet slot picks from.
|
|
#
|
|
# Each entry has three things:
|
|
# - identity (name, family, category) for labeling
|
|
# - acquisition (source, sha256, url) for reproducibility
|
|
# - behaviour (profile) so the synthetic load mimic can run a
|
|
# reasonable proxy until the real sample lands at vm/images/
|
|
#
|
|
# When the real malware binary is present at samples/store/<sha256>,
|
|
# the orchestrator runs THAT inside the guest. When it's absent, the
|
|
# orchestrator falls back to running tools/load_mimic.py with the
|
|
# matching profile so the fleet still produces *labeled, varied* data
|
|
# while we collect the real samples. Either way, meta.json records
|
|
# which path the episode took, so trainers can filter on
|
|
# meta.sample.kind ∈ {real, mimic}.
|
|
|
|
[[sample]]
|
|
name = "xmrig-cryptominer"
|
|
family = "XMRig"
|
|
category = "cryptominer"
|
|
profile = "cpu-saturate"
|
|
# A real XMRig fetch goes here when MalwareBazaar pull is wired up:
|
|
# source = "MalwareBazaar"
|
|
# sha256 = "TBD"
|
|
# url = "https://bazaar.abuse.ch/sample/TBD/"
|
|
description = "Sustained 1-vCPU saturation, very low IO/net. Pure compute."
|
|
|
|
[[sample]]
|
|
name = "mirai-class-bot"
|
|
family = "Mirai"
|
|
category = "botnet"
|
|
profile = "scan-and-dial"
|
|
description = "SYN scans across the bridge IP space + periodic dial-home. High net, low CPU."
|
|
|
|
[[sample]]
|
|
name = "ransomware-mimic"
|
|
family = "Cryptolocker-class"
|
|
category = "ransomware"
|
|
profile = "io-walk"
|
|
description = "Heavy disk write + filesystem walk producing a per-file overwrite envelope."
|
|
|
|
[[sample]]
|
|
name = "dridex-class-trojan"
|
|
family = "Dridex"
|
|
category = "banking-trojan"
|
|
profile = "bursty-c2"
|
|
description = "Long idle, periodic short bursts of TCP egress to a fixed peer (C2 beacon shape)."
|
|
|
|
[[sample]]
|
|
name = "kovter-class-stealth"
|
|
family = "Kovter"
|
|
category = "fileless"
|
|
profile = "low-and-slow"
|
|
description = "Low CPU, periodic memory churn, no persistent on-disk artifacts. Hardest to label from /proc alone."
|
|
|
|
[[sample]]
|
|
name = "reverse-shell-resident"
|
|
family = "Reverse-Shell"
|
|
category = "rat"
|
|
profile = "shell-resident"
|
|
description = "Single TCP socket pinned to an attacker IP, occasional command bursts."
|