CIS490/docs/slide_planner.md

CIS 490 — Slide Deck Planning Template (Filled)

Maps each of the assignment's 16 slides to concrete content drawn from training/, tools/, the validator output, and the trained models. Optional slides are marked [opt] and can be cut for time.

---

Slide 1 — Title Slide

• Title: Behavioral Malware Detection from Hypervisor-Layer VM Telemetry: A Cross-Architecture Comparison Under Cross-Device Generalization
• Authors / affiliation / date / advisor (Dr. Mejias).
• One subtitle line: Six architectures × two deployment modes, evaluated on held-out host.

---

Slide 2 — Motivation

Most malware doesn't look like malware in a database — it looks like a process behaving badly.

• 92 % of threats now use TLS encryption (SonicWall 2022, cited in Melvin 2025) — payload inspection is dead, behavioral detection is what's left.
• Static analysis defeated by obfuscation and packing; signature databases miss zero-days; in-guest detectors can be disabled by the malware they're trying to catch.
• The deployable answer: watch behavior from outside the VM at the hypervisor layer.

Visual: lift the dashboard's intro scene tagline.

---

Slide 3 — Problem Statement

One sentence:

Train a model that classifies a 10-second window of out-of-VM telemetry as one of {clean, armed, infecting, infected_running, dormant}, and measure whether it generalizes from the device it was trained on to a different device it has never seen.

Second sentence:

The honesty bar is cross-device test-set macro F1 with 95 % CIs, not in-distribution validation.

---

Slide 4 — Research Gaps + Questions

Gaps surfaced by the literature review:

1. Most prior work (Melvin 2025, Natsos 2025) reports in-distribution metrics; cross-device generalization is rarely measured.
2. Architecture choice for resource-utilization time-series is contested: Melvin (CNN > RNN), Natsos (Transformer > LSTM > CNN). No head-to-head with controlled training methodology and statistical significance.
3. Realistic-vs-oracle ablation (host-side /proc removed at deployment) is not reported in either paper.

Research question (single): Across six architectures, which best generalizes to a held-out host, and what does each lose when restricted to in-deployment features?

---

Slide 5 — Proposed Solution: Overview

A one-pane diagram (the dashboard's pipeline panel works):

Episodes (.tar.zst) → Validator → Feature & Tensor Builder → Held-out-by-Host Split → 6 Architectures × 2 Modes → Bootstrap-CI Eval → Comparison Report

Three sentences:

1. We collected ~73 k labeled VM episodes across two physical hosts.
2. We trained six architectures (GBT, MLP, CNN, GRU, LSTM, Transformer) twice — once with all telemetry, once with only what a deployed model would see — using a unified training loop with class-weighted loss, early stopping on val macro F1, and best-on-val checkpointing.
3. We evaluated all twelve on the unseen host with bootstrap CIs and paired-bootstrap significance.

---

Slide 6 — Model Design

Side-by-side architecture cards (all six). Visual: parameter counts + inputs:

Model	Input	Params (smoke)	Family
GBT	(230,) summary	~30 KB serialized	Tree
MLP	(230,) summary	104 K	Dense
CNN	(46, 100) tensor	101 K	Conv
GRU	(46, 100) tensor	161 K	RNN
LSTM	(46, 100) tensor	214 K	RNN
Transformer	(46, 100) tensor	76 K	Attention

Note the param counts deliberately stay within ~3× of each other — the comparison is "what does the inductive bias buy you," not "more parameters."

---

Slide 7 — Methodology

Data. 76,660 episodes shipped from 2 hosts. 95.2 % training-usable after the §4.6 acceptance gate. 10-second windows / 5-second stride → ≈9 windows per episode → ~660 k windows.

Split. Held-out-by-host (primary): train on elliott-thinkpad, val carved from train host, test on k-gamingcom. Profile-stratified; scan-and-dial flagged as untested_profiles because k-gamingcom never ran it. Held-out-by-sample (secondary) on the one profile that has ≥ 3 samples.

Standardize on train only. Per-channel for tensors, per-feature for summaries. Median imputation for NaN.

Class-weighted CE. armed weight ≈ 10.8, infecting ≈ 2.3, clean ≈ 0.4 — inverse frequency, clipped.

Training loop. AdamW, LR warmup (5 % of steps) + cosine decay, gradient clipping at 1.0, early stop on val macro F1 patience 8, mixed precision when CUDA, best-on-val checkpoint. Same loop for all five NN architectures. XGBoost uses early_stopping_rounds=30 on val mlogloss.

---

Slide 8 — Evaluation Setup

• Held-out test set: every episode from k-gamingcom (≈ 23 k windows). Never touched at train time.
• Metrics: macro F1, per-phase F1, per-profile F1, per-host F1 — every value with bootstrap 95 % CIs (1000 resamples).
• Statistical significance: paired-bootstrap of macro-F1 differences vs the top model. CI excludes 0 → significant.
• Latency: median µs per window at batch sizes {1, 8, 64, 512} — single-window timing alone is misleading because of Python overhead.
• Realistic-vs-oracle gap: every architecture trained twice, both numbers reported.

---

Slide 9 — Evaluation Results

The comparison_v2.md table from training/eval_/run.py. Smoke-set numbers (200 episodes/host, 5 epochs — full-scale numbers will replace these for the final deck):

Model	Test macro F1 (95 % CI)	Significant vs top?
gbt (oracle)	0.557 [0.543, 0.571]	— (anchor)
mlp	0.176	yes (CI excludes 0)
transformer	0.113	yes
lstm	0.112	yes
gru	0.092	yes
cnn	0.089	yes

Visualization: confusion matrix grid (one per model) from reports/eval/<model>_<mode>_confusion.png.

Headline claim (smoke-version, will be re-tuned at full scale):

At the data scale we have, GBT generalizes to the held-out host significantly better than every NN architecture — including the Transformer. The result is consistent with Natsos & Symeonidis 2025's finding that Transformer dominates only as the dataset grows past ~1k samples per family; below that, simpler inductive biases win.

---

Slide 10 — Case Study / Demonstration

The live dashboard (https://dashboard.wg):

• Scene 2 (collect): live ingest counter from index.jsonl tailing producer.
• Scene 6 (attacks): per-profile attack-envelope thumbnails from producers/profiles.py.
• Scene 7 (chunking): predictions emitted by producers/replay.py running an episode at wall-clock speed.
• Scene 8 (models): macro F1 bars from producers/metrics.py (re-published every 20 s for reconnects).
• Scene 9 (knn): PCA-2 scatter colored by phase from each model's saved projection.
• Scene 10 (perf): accuracy-vs-latency scatter from producers/perf.py, batch-size 64.

Visual: one screenshot of dashboard.wg with live data, clicked through scenes 2 → 10.

---

Slide 11 — Theoretical Contributions [opt]

• Cross-source clock alignment. Producers were inconsistent about t_mono_ns semantics (episode-relative vs system-uptime); we canonicalize on t_wall_ns. Generalizable to any multi-source telemetry pipeline.
• Held-out-host as the primary cross-device generalization claim. Most prior work reports in-distribution metrics; we report the harder number and let the reader see the gap.
• Realistic-vs-oracle ablation as the operational measure of "what the deployed model is missing."

---

Slide 12 — Practical Contributions [opt]

• Open-source training stack (training/): six architectures, schema-hashed checkpoints, validator, dashboard producers — directly reusable for any project where labeled per-window resource-utilization data is available.
• Live dashboard at dashboard.wg with both pipeline-state and trained-model events — the working example of "model running live" the assignment §10 (case study) wants.
• Validator + producer machinery that catches data-quality issues (torn writes in index.jsonl, host silently shipping without bridge pcap, scan-and-dial absent from one host) instead of training on them.

---

Slide 13 — Design Principles [opt]

• No silent downgrade — every host either ships data that meets the gate or produces nothing.
• No silent schema drift — every model checkpoint refuses to load if its training-time channel/feature schema doesn't match what _features.py produces today.
• Honest CIs over point estimates — every test number we report has bootstrap bounds.
• Held out by host, not by time slice — within-sample time splits are easy and dishonest about generalization.

---

Slide 14 — Limitations [opt]

• Two hosts. Cross-device generalization with N=2 is a single fold of leave-one-host-out CV; with more hosts the methodology becomes more rigorous.
• Twelve unique sample names total, with two profiles having 1 sample each. Held-out-by-sample is feasible only on io-walk.
• scan-and-dial not present on k-gamingcom — that profile is trained but cannot be evaluated cross-device. Reported as untested_profiles, never silently averaged.
• Producer-side bugs found during validation: receiver/store.py:130 torn write (1 occurrence in 76 k); ~24 k k-gamingcom episodes shipped without netflow (silent collector failure); cross-source clock-base inconsistency. All surfaced in training/README.md for follow-up.
• Single training corpus, single attack manifest. Generalization to new attack manifests is a stronger claim than this corpus supports.

---

Slide 15 — Conclusion and Future Work

Conclusion (1 line): The realistic-mode model can be trained, evaluated honestly cross-device, and deployed against live telemetry — but the right architecture depends on data scale, and the cross-device gap is the metric that matters.

Future work:

• Self-supervised pretrain on clean-only windows (Masked-Timestep Reconstruction + Volume-of-Hypersphere Minimization, per LogBERT). Detects novel malware without labels. Implementation in training/models/transformer_ssl.py.
• Trust-over-time scoring per IEEE 9881803 — per-window confidence accumulated over a sliding decision window, reset trigger at threshold.
• Integrated Gradients attribution per (channel, timestep) — which signals drove the call — so the writeup can show evidence, not just confidence numbers. Implementation in training/xai/integrated_gradients.py.
• More hosts in the fleet — the data generation and shipping pipeline is in place; adding hosts is a config change. With N ≥ 4 hosts, leave-one-out CV becomes meaningful.
• More distinct samples per profile — 12 samples is too few to claim novel-malware generalization; the current dataset only supports cross-device generalization.

---

Slide 16 — References / Acknowledgments

The same APA reference list as docs/project_brief.md §9. Lab acknowledgments: Dr. Mejias, Raul, the spectral lab infrastructure. Tools acknowledgments: libVMI, Drakvuf, XGBoost, PyTorch, scikit-learn, pyarrow.

---

Notes for the deck builder

• Slides 11–14 are marked optional in the assignment — keep them if you have time, drop them if you're tight. Slides 1, 3, 5, 7, 9, 10, 15 are the required spine.
• Every metric on Slide 9 should come from reports/eval/comparison_v2.md directly — copy-paste the markdown and let the deck render it.
• Slide 10 (live demo) is more memorable than any chart — bring up dashboard.wg, scroll through scenes 2–10, let it talk.
• The repo has the [opt]-marked Design Principles slide built into the README. If you cut Slide 13, the principles still live in the artifact.