History

max a88ac83db0 Close out the deployment-readiness gaps Wraps the gaps surfaced in the "what is not implemented" audit so the fleet really is shippable end-to-end. Verified live on the Pi: - cis490-shipper --ping → HTTP 200 through Caddy + mTLS via the new wg-pki client CA leaf - real episode dir → tar+zstd → PUT → HTTP 201 stored - re-ship same bytes → 200 (idempotent) - re-ship different bytes under same id → 409 (conflict) Changes: orchestrator/episode.py - EpisodeConfig.revert_at_start / revert_at_end (Tier 0+ snapshot/ revert per docs/architecture.md). When set + qmp_socket present, EpisodeRunner issues loadvm <snapshot_name> and emits snapshot_revert / snapshot_revert_failed events on the same monotonic clock as everything else. collectors/qmp.py - savevm() / loadvm() helpers using human-monitor-command, plus a test against the fake QMP server. exploits/workloads.py - chunked_real_binary_upload() returns a ChunkedUpload plan: 8 KiB base64 chunks (~6 KiB binary each) so msfrpc never sees a buffer- busting payload. Includes a finalize step that sha256-verifies on the guest before exec. - real_binary_workload() now wraps the chunked plan for backwards compat with single-shot callers. exploits/driver.py - Tier-4 dispatch walks the chunked plan in MSFExploitDriver: each chunk is a separate session_shell_write; finalize verifies; exec only runs on sha-ok. New events: real_binary_upload_begin, real_binary_verify, real_binary_aborted. etc/cis490-orchestrator.service - Reads /etc/cis490/lab-host.env (FLEET_HOST_ID + optional BRIDGE). - Grants AmbientCapabilities CAP_NET_RAW (tcpdump for source 4) + CAP_SYS_ADMIN + CAP_PERFMON (perf for source 3) so collectors work under hardening. scripts/install-lab-host.sh - Writes /etc/cis490/lab-host.env on first install with FLEET_HOST_ID defaulting to `hostname -s`. - Best-effort: fetches the Alpine baseline qcow2 (sha512-pinned) and builds cidata.iso with the in-guest agent embedded; symlinks both into /opt/cis490/vm/images/ so launchers find them. scripts/fetch-alpine-baseline.sh - Idempotent fetcher for the Alpine 3.21 cloud-init nocloud qcow2 matching the sha512 in docs/sources.md. tools/plot_envelope.py - Rebuilt to render whatever telemetry the episode dir contains: proc → QMP block ops → perf IPC/miss-rate → bridge pkts/SYNs → guest agent load/mem. Missing sources are silently skipped. tools/index_reader.py - cis490-index CLI: filter receiver's index.jsonl by host / sample / time range, sort, count-by group. Closest thing to a query interface until we stand up Postgres/Timescale. samples/README.md - Rewritten to match the new manifest schema, the kind=real vs mimic split, the per-(host, slot, ep) selection mechanic, and the chunked-upload safety story. Tests: 106 pass (was 102). New cases: - test_qmp.py — savevm + loadvm (HMP wrapper + error path) - test_tier4.py — chunked plan splitting, sha-pinned finalize, end-to-end driver walks all chunks + verify + exec via the fake msfrpc client Closes the "what is not implemented" punch list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>		2026-04-30 00:31:55 -05:00
..
__init__.py	Collectors 2/4/5 + fleet runner + sample manifest + Tier-3 setup scripts	2026-04-30 00:02:27 -05:00
manifest.py	Close out the open issues: bridge pcap wiring, perf collector, Tier-4	2026-04-30 00:17:49 -05:00
manifest.toml	Collectors 2/4/5 + fleet runner + sample manifest + Tier-3 setup scripts	2026-04-30 00:02:27 -05:00
README.md	Close out the deployment-readiness gaps	2026-04-30 00:31:55 -05:00

README.md

samples/

Catalog of malware (or behaviour-matched mimics) the fleet draws from. Sample binaries are NEVER committed to this repo.

What's here

manifest.toml      schema-checked catalog (loaded by samples/manifest.py)
manifest.py        loader + per-(host_id, slot, ep) deterministic selection
store/             SHA-256-pinned binary content (gitignored — never commit)
.bazaar.token      MalwareBazaar API key (mode 0600, gitignored)

Manifest schema

Each entry in manifest.toml:

[[sample]]
name        = "xmrig-cryptominer"   # unique within manifest, DNS-safe
family      = "XMRig"               # canonical family label for ML
category    = "cryptominer"         # one of: cryptominer, botnet, ransomware,
                                    # banking-trojan, fileless, rat, worm,
                                    # loader, wiper, other
profile     = "cpu-saturate"        # behaviour profile from
                                    # exploits/workloads.py — gates the
                                    # in-session shell workload when no
                                    # real binary is staged
description = "..."

# Optional — present iff this is a real binary the fetcher should pull:
sha256 = "abc123..."
source = "MalwareBazaar"
url    = "https://bazaar.abuse.ch/sample/abc123/"

The loader rejects unknown categories and duplicate names. See tests/test_fleet.py for the property tests covering selection distribution + catalog walkability.

"real" vs "mimic"

Sample.kind is "real" when sha256 is set, otherwise "mimic".

Mimic — the orchestrator runs the matching profile-shaped shell command (cpu-saturate / scan-and-dial / io-walk / bursty-c2 / low-and-slow / shell-resident) inside the guest. No real binary needed; useful right now for testing the dataset pipeline and as the realistic-but-safe envelope class the trainer expects.
Real — the orchestrator's Tier-3+ driver chunked-uploads samples/store/<sha256> into the shell session, sha256-verifies on the guest side, and execs it. Hash mismatch fail-stops the run; a tampered binary is never executed.

meta.sample.kind lands in every episode's meta.json, so trainers can stratify on it (the realistic-model path consumes only kind == "real" episodes by default).

Fetching a real binary

# 1. Register a (free) account at https://bazaar.abuse.ch and get the API key.
echo "<your-key>" > samples/.bazaar.token
chmod 0600 samples/.bazaar.token

# 2. Add an entry with sha256+source+url to manifest.toml.

# 3. Pull the binary into samples/store/<sha256>:
uv run python tools/fetch_sample.py <sha256>

Idempotent — re-running checks the staged copy's sha256 and skips the download if it already matches.

Per-(host, slot, episode) selection

manifest.py::SampleManifest.select(host_id, slot, episode_index) hashes those three into a uniform integer and indexes the catalog. Two lab hosts on the same slot pick different samples (collision rate ~1/N). A single host walks the whole catalog within ~len(manifest) episodes. No coordinator.

Safety rules

Only download to a lab host, never to a developer workstation. samples/store/ lives only there, gitignored, on a disk that is not auto-mounted elsewhere.
The lab host's br-malware bridge is host-only by design (no NAT, no route). Real malware running in the guest cannot call out unless the operator explicitly opens egress, which we don't.
Snapshot/revert (see EpisodeConfig.revert_at_* + qmp.savevm/ loadvm) means every fresh episode starts from a known-good baseline regardless of what the previous one did to the guest.
The fetcher verifies sha256 on download; the driver verifies again in-guest before exec. Both layers must match the manifest.

Adding a sample

Pick a family + category from the closed enum above.
Pick a profile from exploits/workloads.all_profiles(). If the sample's behaviour doesn't match any of the six existing shapes, add a new factory to exploits/workloads.py first, with tests.
(Real-only) Compute sha256, fetch via tools/fetch_sample.py, verify the staged file's hash matches.
Append the entry to manifest.toml.
Run the test suite — the manifest loader's invariants catch typos.