fa1574a0a6
Lays down the design surface for the CIS490 behavioral-malware-detection dataset and model. No code yet — schema and topology are decided first so collection can start without rework. Docs: - README: project goal, navigation - architecture: lab topology, KVM choice, episode state machine, deployment-mirror reasoning - threat-model: train/serve parity rule, oracle-vs-deployable feature split, two-model evaluation strategy - data-model: per-episode JSONL layout, row schemas, phase enum - transport: WG-native shipper/receiver design, idempotent uploads - deploy: one-command install for lab-host and receiver roles - lab-setup: KVM prereqs, VM build, snapshot, virtio-serial wiring Skeleton: orchestrator/, collectors/, vm/, exploits/, samples/, training/ (each with a short README explaining purpose). Extended .gitignore to exclude qcow2 images, pcaps, sample binaries, secrets. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
138 lines
3.4 KiB
Markdown
138 lines
3.4 KiB
Markdown
# Deploy
|
|
|
|
Two roles. One install command each.
|
|
|
|
## Roles
|
|
|
|
| Role | Where it runs | What it does |
|
|
|---|---|---|
|
|
| `lab-host` | any KVM-capable Linux box on WG | runs episodes, ships completed episodes to the receiver |
|
|
| `receiver` | Pi5 (or any always-on WG node) | accepts ship uploads, stores tarballs + `index.jsonl` |
|
|
|
|
## Lab host install
|
|
|
|
```sh
|
|
git clone https://maxgit.wg/spectral/CIS490.git
|
|
cd CIS490
|
|
./scripts/install-lab-host.sh
|
|
```
|
|
|
|
The installer:
|
|
|
|
1. Verifies KVM (`/dev/kvm` exists, user in `kvm` group).
|
|
2. Installs system deps via the host package manager (qemu, tcpdump,
|
|
linux-tools/perf, zstd, python ≥ 3.11).
|
|
3. Bootstraps a [`uv`](https://github.com/astral-sh/uv)-managed venv at
|
|
`.venv/` and installs the pinned Python deps from `uv.lock`.
|
|
4. Drops two systemd units into `/etc/systemd/system/`:
|
|
- `cis490-orchestrator.service` — runs the episode loop on a queue
|
|
- `cis490-shipper.service` — watches `data/episodes/` and ships completed
|
|
episodes
|
|
5. Writes a config template to `/etc/cis490/lab-host.toml` (idempotent — only
|
|
on first install).
|
|
|
|
You finish by editing `/etc/cis490/lab-host.toml` to point at your receiver
|
|
and to enroll your lab host's WG-issued client cert, then:
|
|
|
|
```sh
|
|
sudo systemctl enable --now cis490-orchestrator cis490-shipper
|
|
```
|
|
|
|
### `lab-host.toml`
|
|
|
|
```toml
|
|
host_id = "lab-host-1"
|
|
|
|
[paths]
|
|
data_root = "/var/lib/cis490/data"
|
|
samples_store = "/var/lib/cis490/samples/store"
|
|
qcow_image = "/var/lib/cis490/vm/images/metasploitable2.qcow2"
|
|
|
|
[receiver]
|
|
url = "https://collector.wg"
|
|
client_cert = "/etc/cis490/certs/lab-host-1.pem"
|
|
client_key = "/etc/cis490/certs/lab-host-1.key"
|
|
ca_bundle = "/etc/cis490/certs/wg-ca.pem"
|
|
|
|
[episode]
|
|
baseline_seconds = 30
|
|
infected_seconds = 90
|
|
dormant_seconds = 60
|
|
|
|
[retention]
|
|
keep_local_for_days = 7
|
|
prune_at_disk_pct = 80
|
|
```
|
|
|
|
## Receiver install
|
|
|
|
On the Pi5 (or designated central node):
|
|
|
|
```sh
|
|
git clone https://maxgit.wg/spectral/CIS490.git
|
|
cd CIS490
|
|
./scripts/install-receiver.sh
|
|
```
|
|
|
|
The installer:
|
|
|
|
1. Installs Python ≥ 3.11 + zstd + a tiny WSGI runner (uvicorn).
|
|
2. Bootstraps the same `uv`-managed venv.
|
|
3. Drops `cis490-receiver.service` listening on `127.0.0.1:8443` (TLS
|
|
terminated by the existing Caddy in `spectral/caddy`, which already binds
|
|
`*.wg`).
|
|
4. Writes a config template to `/etc/cis490/receiver.toml`.
|
|
|
|
Caddy block (added to your `spectral/caddy` config) for the receiver:
|
|
|
|
```caddy
|
|
collector.wg {
|
|
tls internal
|
|
reverse_proxy 127.0.0.1:8443 {
|
|
transport http {
|
|
tls
|
|
tls_client_auth /etc/cis490/certs/wg-ca.pem
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
(mTLS terminates at the receiver, not Caddy — so the receiver sees the
|
|
client cert and can enforce per-host policies later.)
|
|
|
|
### `receiver.toml`
|
|
|
|
```toml
|
|
listen_addr = "127.0.0.1:8443"
|
|
store_root = "/var/lib/cis490/episodes"
|
|
incoming_root = "/var/lib/cis490/incoming"
|
|
index_path = "/var/lib/cis490/index.jsonl"
|
|
ca_bundle = "/etc/cis490/certs/wg-ca.pem"
|
|
|
|
[limits]
|
|
max_episode_bytes = 268_435_456 # 256 MiB
|
|
```
|
|
|
|
## Day-2 operations
|
|
|
|
```sh
|
|
# How many episodes have been shipped?
|
|
ssh collector.wg 'wc -l /var/lib/cis490/index.jsonl'
|
|
|
|
# What's in the outbox on a lab host? (failed/pending shipments)
|
|
ls /var/lib/cis490/data/outbox/
|
|
|
|
# Tail the orchestrator log
|
|
journalctl -u cis490-orchestrator -f
|
|
|
|
# Tail the shipper log
|
|
journalctl -u cis490-shipper -f
|
|
```
|
|
|
|
## Updating
|
|
|
|
```sh
|
|
git pull
|
|
./scripts/install-lab-host.sh # idempotent; re-syncs deps and units
|
|
sudo systemctl restart cis490-orchestrator cis490-shipper
|
|
```
|