diff --git a/.gitignore b/.gitignore index 432eda9..11c9b47 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,10 @@ vm/snapshots/ # Telemetry output data/episodes/ +data/campaign.json +data/campaign_done.marker +data/outbox/ +data/shipped/ *.pcap *.pcapng diff --git a/reports/Dev_REL1_043026.md b/reports/Dev_REL1_043026.md new file mode 100644 index 0000000..49d97b4 --- /dev/null +++ b/reports/Dev_REL1_043026.md @@ -0,0 +1,212 @@ +# Setup & Integration Report — Dev_REL1_043026 + +**Date:** 2026-04-30 +**Branch:** Dev_REL1_043026 (day branch off Dev_REL1) +**Host:** elliott-ThinkPad (Arch Linux, kernel 6.19.14-arch1-1) +**Scope:** Full lab-host bring-up from a clean `git clone` — install prereqs, +build VM image, run single episode, validate campaign loop. + +--- + +## Environment Baseline + +| Item | Expected | Found | Status | +|---|---|---|---| +| `/dev/kvm` | exists | `crw-rw-rw-` (world-writable) | OK | +| kvm group membership | required | user NOT in group | OK — world perm bypasses group | +| QEMU | ≥ 8.0 | 11.0.0 | OK | +| Python | ≥ 3.11 | 3.14.4 | OK | +| `uv` | installed | missing | Fixed | +| `perf` | installed | missing | Fixed | +| `tcpdump` | installed | missing | Fixed | +| `zstd` | installed | 1.5.7 | OK | +| `vm/images/` | populated | empty directory missing | Fixed | +| `scripts/install-lab-host.sh` | exists | missing | **Open — ISS-005** | + +--- + +## Issues Found and Resolved + +### ISS-001 — `uv` not installed + +**Symptom:** `uv: command not found` — unable to install Python deps or run +any `uv run` command. + +**Root cause:** `deploy.md` and `lab-setup.md` both reference +`scripts/install-lab-host.sh` as the mechanism to install `uv`, but that +script does not exist in the repository. + +**Fix applied:** Installed system package via pacman (after mirror sync): +``` +sudo pacman -Sy && sudo pacman -S uv +``` +Fallback (no sudo): `curl -LsSf https://astral.sh/uv/install.sh | sh` +installs to `~/.local/bin/uv`. + +**Residual:** See ISS-005 for the missing installer script. + +--- + +### ISS-002 — `perf` and `tcpdump` not installed + +**Symptom:** `perf: command not found`, `tcpdump: command not found`. +Both are listed as host prerequisites in `docs/lab-setup.md`. + +**Fix applied:** +``` +sudo pacman -S perf tcpdump +``` + +--- + +### ISS-003 — Pacman mirror 404 errors on linux-tools group + +**Symptom:** `sudo pacman -S linux-tools` failed with 404 errors across all +configured mirrors. Mirrors held linux-tools 7.0.1 while the database +referenced 7.0.2. + +**Root cause:** Local pacman databases were stale. + +**Fix applied:** `sudo pacman -Sy` refreshed the databases; subsequent +install of individual packages (`perf`, `tcpdump`, `uv`) succeeded. + +**Recommendation:** Add `pacman -Sy` as the first step in +`install-lab-host.sh`. + +--- + +### ISS-004 — VM images missing (`alpine-baseline.qcow2`, `cidata.iso`) + +**Symptom:** `launch_demo.sh` exits immediately: +``` +no image at .../vm/images/alpine-baseline.qcow2 +``` + +**Root cause:** `vm/images/` is correctly gitignored, but the bring-up +procedure for first-time hosts is undocumented outside of +`docs/lab-setup.md` (which references the non-existent install script). + +**Fix applied (manual steps):** +```bash +# 1. Create images directory +mkdir -p vm/images + +# 2. Download Alpine 3.21 cloud image +curl -L -o vm/images/alpine-baseline.qcow2 \ + https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/generic_alpine-3.21.0-x86_64-bios-cloudinit-r0.qcow2 + +# 3. Build cloud-init cidata ISO +uv run python tools/build_cidata.py vm/images/cidata.iso + +# 4. Take baseline snapshot (used as metadata label in episodes) +qemu-img snapshot -c baseline-v1 vm/images/alpine-baseline.qcow2 +``` + +**Note on snapshot mechanics:** `launch_demo.sh` uses `-drive snapshot=on`, +which creates a temporary write overlay per boot. Cloud-init runs fresh each +boot (overlay discarded between episodes), so the 35-second settle wait in +`run_real_vm_demo.py` is required and sufficient. No first-boot without +`snapshot=on` is needed. + +**Recommendation:** Add these steps to `docs/lab-setup.md` and automate in +`install-lab-host.sh`. + +--- + +### ISS-005 — `scripts/install-lab-host.sh` does not exist (open) + +**Symptom:** Both `deploy.md` and `lab-setup.md` reference +`scripts/install-lab-host.sh` as the canonical install path. The `scripts/` +directory does not exist. + +**Impact:** Every new lab host requires the manual steps from ISS-001 through +ISS-004 above. + +**Status:** Open. Requires writing the install script. Suggested content: +- `pacman -Sy` +- Install: `uv`, `perf`, `tcpdump`, `zstd`, `qemu-system-x86_64`, + `bridge-utils` +- Download Alpine cloud image if `vm/images/alpine-baseline.qcow2` absent +- Run `tools/build_cidata.py` +- Take `baseline-v1` snapshot +- Drop systemd units from `etc/` into `/etc/systemd/system/` +- Write `/etc/cis490/lab-host.toml` template + +--- + +### ISS-006 — Campaign files not promoted to Dev_REL1 + +**Symptom:** `run_campaign.py` and `shipper.py` (committed to +`Dev_REL1_042926`) were absent from `Dev_REL1_043026` (branched off +`Dev_REL1` before the merge). + +**Fix applied:** Cherry-picked commit `ce2fd66` from `Dev_REL1_042926`: +``` +git cherry-pick ce2fd66 +``` + +**Recommendation:** Merge `Dev_REL1_042926` → `Dev_REL1` before cutting +future day branches so the base is always current. + +--- + +### ISS-007 — Campaign output files not in `.gitignore` + +**Symptom:** `data/campaign.json`, `data/campaign_done.marker`, +`data/outbox/`, and `data/shipped/` would appear as untracked files after +a campaign run. + +**Fix applied:** Added to `.gitignore`: +``` +data/campaign.json +data/campaign_done.marker +data/outbox/ +data/shipped/ +``` + +--- + +## Validation Results + +All three checks passed on this host. + +### Single episode (`run_real_vm_demo.py`) +``` +episode_id = 01KQG2K4WJBE52PCD2QDCQAVKB +rows_proc = 853 +phases = [clean, armed, infecting, infected_running, + dormant, infected_running, dormant, clean] +duration = 85.26s +``` + +### 2-episode campaign (`run_campaign.py --target 2`) +``` +episode 1: id=01KQG2QRVRRTVM3TMEKCEW1E6Z rows=853 duration=85.26s ✓ +episode 2: id=01KQG2VMWM6WKEQPCZ3BKYT513 rows=853 duration=85.27s ✓ +campaign_done.marker written ✓ +campaign.json: {"completed": 2, "done": true} ✓ +``` + +### Episode directory structure (each episode) +``` +data/episodes// + meta.json ✓ + events.jsonl ✓ + labels.jsonl ✓ (8 phase transitions) + telemetry-proc.jsonl ✓ (853 rows @ 10 Hz) + done.marker ✓ +``` + +--- + +## Known Gaps (pre-existing, not introduced this session) + +| Gap | File | Status | +|---|---|---| +| `perf stat` collector | `collectors/perf_qemu.py` | Not yet implemented | +| QMP collector | `collectors/qmp.py` | Not yet implemented | +| `tcpdump` / pcap collector | `collectors/pcap.py` | Not yet implemented | +| In-guest agent collector | `collectors/guest_agent.py` | Not yet implemented | +| Metasploit exploit driver (Tier 3) | `exploits/` | Planned | +| Real malware sample (Tier 4) | `samples/` | Planned | +| `scripts/install-lab-host.sh` | `scripts/` | See ISS-005 |