1b6c7b2f4a
This is the chunk that makes "real data" actually flow on multiple
hosts in parallel. End-to-end pipe was up at 613c6fa / 2579683; now
the lab-host side has the diversity + concurrency it needs.
Collectors landed:
collectors/qmp.py — source 2 (oracle). Tiny synchronous QMP
client + row builder + run loop. Tolerates
older qemu without query-stats.
collectors/guest_agent.py — source 5 (deployable). Reads the
virtio-serial host-side socket, parses
agent JSON-lines, re-stamps to the host
monotonic clock, persists.
collectors/pcap.py — source 4 (deployable). tcpdump capture
+ pure-Python pcap reader + 100 ms
netflow.jsonl bucketizer. Decodes
Ethernet/IPv4/TCP/UDP enough for the
schema in docs/data-model.md.
In-guest agent:
vm/guest-agent/cis490_agent.py — stdlib-only Python agent. Reads
/proc/{stat,meminfo,loadavg,net/dev,net/tcp*}, top-N RSS procs,
thermal. Writes JSON-lines to /dev/virtio-ports/cis490.guest.agent.
tools/build_cidata.py — embeds the agent + an OpenRC service into
user-data so first boot of the Alpine cidata image auto-starts it.
Launchers:
vm/launch_demo.sh / launch_target.sh — second virtio-serial port for
the agent socket; SLOT env support so multiple VMs run without
socket / port collisions; PORT_BASE on launch_target so multiple
target VMs hostfwd different host ports.
vm/setup_bridge.sh — creates host-only br-malware (10.200.0.1/24,
no NAT). Idempotent.
Fleet:
orchestrator/fleet.py — capacity detector (cores / RAM / load
headroom) + concurrent-slot runner. Per-slot ENV selects the
sample. FleetCapacity dataclass round-trips into meta.json so
"this episode ran with 6 concurrent VMs" is auditable post-hoc.
tools/run_fleet.py — CLI: --capacity report; --waves N runs N
waves of (max_concurrent) episodes each, every slot with a
different sample.
etc/cis490-orchestrator.service — now drives the fleet runner with
Restart=always so each invocation runs one wave and respawns,
giving a continuous stream.
Samples:
samples/manifest.toml — six profiles spanning the five major
behaviour shapes. Each entry is real OR mimic (sha256 distinguishes).
samples/manifest.py — strict TOML loader (rejects dups, unknown
categories) + deterministic select(host_id, slot, episode_index)
so different hosts on the network walk the catalog in different
orders without any coordinator.
EpisodeRunner:
orchestrator/episode.py — optional qmp_socket + guest_agent_socket
fields on EpisodeConfig; when set, additional collector threads
run alongside proc_qemu. EpisodeResult now carries rows_qmp +
rows_guest counters.
Tier-3 setup automation:
scripts/install-msfrpcd.sh — installs metasploit-framework where
the package manager has it, generates a strong password into
/etc/cis490/msfrpc.env, drops a hardened systemd unit bound to
127.0.0.1:55553. After this, run_tier3_demo.py works zero-touch
once MSFRPC_PASSWORD is sourced.
scripts/fetch-metasploitable2.sh — accepts IMAGE_URL + IMAGE_SHA256
from the operator (Rapid7 download is registration-walled), pulls,
verifies, converts vmdk → qcow2, lands at vm/images/.
Tests: 82 pass (was 51). New suites:
tests/test_qmp.py — fake QMP server, capability handshake,
blockstats, async-event interleaving,
5-failure backoff
tests/test_guest_agent.py — fake virtio socket, JSON-lines read +
re-stamp, malformed-line tolerance
tests/test_pcap.py — synthetic pcap with TCP/UDP/ARP frames,
bucketize correctness across windows
tests/test_fleet.py — capacity math (8-core idle / low-RAM /
high-load / Pi5 / 1-core box), manifest
selection determinism + diversity
What's queued for the next commit (already discussed in convo):
- MSFExploitDriver v2: map sample.profile → distinct in-session
workload so Tier-3 episodes don't all produce the same yes-loop
envelope. Critical for ML to learn varied malware shapes.
- Real-sample fetch from MalwareBazaar by sha256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
97 lines
3 KiB
Python
97 lines
3 KiB
Python
"""``cis490-fleet`` — run as many concurrent labeled episodes as the
|
|
host can handle, drawing samples from the manifest.
|
|
|
|
Modes:
|
|
|
|
--capacity Print the resource calculation and exit. No VMs spawned.
|
|
--waves N Run N waves of episodes (one wave = max_concurrent
|
|
episodes, each in its own slot). Default: 1.
|
|
--max-concurrent N
|
|
Cap concurrency below the auto-detected ceiling.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import json
|
|
import logging
|
|
import os
|
|
import signal
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
# Allow running as a script.
|
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
|
|
from orchestrator.fleet import ( # noqa: E402
|
|
FleetConfig, FleetRunner, capacity_report, detect_capacity,
|
|
)
|
|
from samples.manifest import SampleManifest # noqa: E402
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
p = argparse.ArgumentParser(prog="cis490-fleet")
|
|
p.add_argument("--capacity", action="store_true")
|
|
p.add_argument("--waves", type=int, default=1)
|
|
p.add_argument("--max-concurrent", type=int, default=None)
|
|
p.add_argument("--manifest",
|
|
default=str(Path(__file__).resolve().parent.parent / "samples" / "manifest.toml"))
|
|
p.add_argument("--data-root", default="data")
|
|
p.add_argument("--host-id", default=os.environ.get("FLEET_HOST_ID") or os.uname().nodename)
|
|
p.add_argument("--ram-per-vm-mib", type=int, default=320)
|
|
p.add_argument("--require-real-samples", action="store_true")
|
|
p.add_argument("--log-level", default="INFO")
|
|
args = p.parse_args(argv)
|
|
|
|
logging.basicConfig(
|
|
level=getattr(logging, args.log_level.upper(), logging.INFO),
|
|
format="%(asctime)s %(levelname)s %(name)s %(message)s",
|
|
)
|
|
|
|
if args.capacity:
|
|
print(capacity_report())
|
|
return 0
|
|
|
|
manifest = SampleManifest.load(args.manifest)
|
|
repo_root = Path(__file__).resolve().parent.parent
|
|
|
|
cfg = FleetConfig(
|
|
host_id=args.host_id,
|
|
repo_root=repo_root,
|
|
data_root=Path(args.data_root).resolve(),
|
|
manifest=manifest,
|
|
ram_per_vm_mib=args.ram_per_vm_mib,
|
|
max_concurrent_override=args.max_concurrent,
|
|
require_real_samples=args.require_real_samples,
|
|
)
|
|
|
|
runner = FleetRunner(cfg)
|
|
|
|
def _stop(signum, frame): # noqa: ARG001
|
|
runner.stop()
|
|
signal.signal(signal.SIGTERM, _stop)
|
|
signal.signal(signal.SIGINT, _stop)
|
|
|
|
result = runner.run(episodes=args.waves)
|
|
|
|
print(json.dumps({
|
|
"host_id": args.host_id,
|
|
"capacity": result.capacity.to_dict(),
|
|
"slots": [
|
|
{
|
|
"slot": s.slot,
|
|
"sample": s.sample_name,
|
|
"sample_kind": s.sample_kind,
|
|
"rc": s.rc,
|
|
"duration_s": s.duration_s,
|
|
"error": s.error,
|
|
} for s in result.slots
|
|
],
|
|
"total_duration_s": result.total_duration_s,
|
|
}, indent=2))
|
|
|
|
return 0 if all(s.rc == 0 for s in result.slots) else 1
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|