CIS490/training

training/

Train a behavioral malware detector from labeled episode tarballs. Six architectures × two threat-model modes = twelve trained models, evaluated head-to-head on a held-out-by-host split.

What lives where

training/
  _episode_io.py          tarball decoder
  _features.py            channel registry + windowing (summary + tensor)
  _split.py               held-out recipes (host / sample / time)
  build_features.py       summary-stat parquet builder
  build_tensors.py        channel × time tensor shard builder
  models/                 6 architectures behind a common BaseModel interface
    gbt.py                XGBoost on summary features
    mlp.py                MLP on summary features (NN baseline parity to GBT)
    cnn.py                1D-CNN on tensor windows
    gru.py                GRU on tensor windows
    lstm.py               LSTM on tensor windows
    transformer.py        small Transformer encoder on tensor windows
    _base.py, _torch_seq.py
    _checkpoint.py        schema-hashed save/load — refuses mismatches
  trainer/
    run.py                end-to-end training driver (one model at a time)
    _loop.py              shared training loop: class-weighted CE, LR warmup +
                          cosine, early stop on val macro F1, best-on-val
    _data.py              loaders for summary parquet + tensor shards
  eval_/
    run.py                load every checkpoint, score, write comparison_v2.md
    _metrics.py           macro F1 + per-class F1 with bootstrap 95 % CIs;
                          paired-bootstrap significance for model-vs-model
    breakdown.py          per-profile, per-host metric tables
  producers/              live event sources for the dashboard
                          (replay, metrics, perf, profiles)
                          → see training/dashboard/PRODUCERS.md

The honesty rules this implements

1. Held-out by host (primary): train on elliott-thinkpad, test on k-gamingcom. Tests cross-device generalization, the claim a deployed model has to support. 5 of 6 profiles populated cross-device; scan-and-dial is untested (k-gamingcom never ran it) and explicitly reported as such, not silently averaged in.

2. Profile-stratified, sample-stratified, or time: all three split recipes are available via --split-recipe {host,sample,time}. held_out_sample excludes profiles with too few unique sample_names (would be mathematically unsound otherwise) — the dataset has 2 such profiles today (cpu-saturate, low-and-slow).

3. In-distribution val carved from train host for hyperparameter selection. Test set is never touched at training time.

4. Class-weighted cross-entropy computed from the train slice (inverse frequency, clipped). Class imbalance is real (armed/infecting rare, infected_running common) and unweighted loss under-trains on the operationally interesting phases.

5. Best-on-val checkpoint selected by macro F1 (not accuracy — accuracy hides imbalance). Early stopping with patience=8. LR warmup (5 % of steps) + cosine decay to 0.

6. Schema-hashed checkpoints. Every saved model carries a sha256 of its input schema. Loading a checkpoint against a changed _features.py registry raises ValueError instead of silently feeding mis-aligned columns to the model.

7. Bootstrap CIs on every test metric. Reporting macro_f1 = 0.873 ± 0.012 is the bar; a single point estimate from one finite test is dishonest.

8. Paired-bootstrap significance for model-vs-model gap. CI excludes 0 → significant.

9. NaN handling for the degraded set (k-gamingcom shipped without netflow): NaN fed through standardization → 0 after, but a missingness mask is kept on tensor data for the sequence models to learn to discount sparse channels. (Indicator features for summary models is a v2 enhancement.)

Pipeline

   /var/lib/cis490/episodes/                ← raw .tar.zst
   /var/lib/cis490/index.jsonl
            │
            ▼
   tools/dataset_validate.py                ← full-sweep validator
            │
            ▼
   data/processed/validation_v1.parquet     ← committed
            │
   ┌────────┴─────────┐
   ▼                  ▼                       (rsync to GPU box)
   training/build_features.py     training/build_tensors.py
            │                                  │
            ▼                                  ▼
   features_window_v1.parquet       tensor_window_v1/host=*/<id>.npz
   feature_schema_v1.json           (channel × time, ~12 GB at full scale)
            │                                  │
            └────────┬─────────────────────────┘
                     ▼
          training/trainer/run.py
            (per model × mode)
                     │
                     ▼
   artifacts/<model>_<mode>.ckpt.json   + sidecar (.pt or .xgb.json)
                     │
                     ▼
          training/eval_/run.py
                     │
                     ▼
   reports/eval/comparison_v2.md
   reports/eval/<model>_<mode>_eval.json   (full per-phase, per-profile,
                                            per-host metrics with CIs)

Quickstart on the GPU box

git clone http://maxgit.wg/spectral/CIS490.git
cd CIS490
uv sync --group training

# 1. pull raw episodes from the Pi (needs WireGuard + cis490-trainer)
PI_USER=max PI_HOST=10.100.0.1 LOCAL_DIR=./episodes \
    bash scripts/sync-training-data.sh

# 2. build features + tensors
uv run --group training python training/build_features.py \
    --validation data/processed/validation_v1.parquet \
    --store      ./episodes \
    --out-dir    data/processed

uv run --group training python training/build_tensors.py \
    --validation data/processed/validation_v1.parquet \
    --store      ./episodes \
    --out-dir    data/processed/tensor_window_v1

# 3. train all 12 (one process per model × mode)
for model in gbt mlp cnn gru lstm transformer; do
  for mode in realistic oracle; do
    uv run --group training python -m training.trainer.run \
      --model $model --mode $mode \
      --validation data/processed/validation_v1.parquet \
      --summary    data/processed/features_window_v1.parquet \
      --tensors    data/processed/tensor_window_v1 \
      --schema     data/processed/feature_schema_v1.json \
      --train-hosts elliott-thinkpad \
      --epochs 60
  done
done

# 4. evaluate, write comparison_v2.md
uv run --group training python -m training.eval_.run \
    --validation data/processed/validation_v1.parquet \
    --summary    data/processed/features_window_v1.parquet \
    --tensors    data/processed/tensor_window_v1 \
    --reports-dir reports/eval

Live dashboard

Producers under training/producers/ push events to the dashboard.wg WebSocket via the canonical training.dashboard.client.Publisher (loopback HTTP, stdlib-only). See ../dashboard/PRODUCERS.md for the event contract.

# After training, push live model_metric + model_perf bars:
uv run --group training python -m training.producers.metrics \
    --validation data/processed/validation_v1.parquet \
    --artifacts artifacts \
    --summary data/processed/features_window_v1.parquet \
    --tensors data/processed/tensor_window_v1

# Replay one episode at wall-clock speed (drives phase + prediction +
# embedding events):
uv run --group training python -m training.producers.replay \
    --episode  /var/lib/cis490/episodes/elliott-thinkpad/<id>.tar.zst \
    --host-id  elliott-thinkpad \
    --artifacts artifacts

Tests

pytest tests/test_training_split.py tests/test_training_features.py \
       tests/test_training_checkpoint.py

Guards: split coverage assertions, time-base alignment (the t_wall_ns vs t_mono_ns netflow regression), counter-to-rate correctness, schema-mismatch rejection, deterministic split.

Open data-quality issues found while building this

Surfaced for the writeup, not silently worked around:

• receiver/store.py:130 torn write — index.jsonl line 19500 has two records concatenated. The "atomic for sub-PIPE_BUF" comment isn't holding. Validator skips and warns; producer-side fix needed.
• k-gamingcom silent downgrade — ~24 k episodes shipped without netflow.jsonl. Per AGENTS.md "Do not silently downgrade a host" this is a producer hard-rule violation. We accept them as degraded and train, but the realistic model loses bridge-pcap signal on those.
• scan-and-dial absent from k-gamingcom — held-out-by-host can't evaluate that profile cross-device. Reported as untested_profiles in every metrics output rather than averaged in.
• Cross-source clock drift — _features.py aligns on t_wall_ns because netflow's t_mono_ns is system-uptime, not episode-relative. Fix is in this repo; the producer should be patched to emit episode-relative t_mono_ns consistently.
• Sample diversity is low (12 unique sample_names total across 6 profiles). held_out_sample only fits the io-walk profile. Held-out-by-host is the right primary eval until more samples are added.