CIS490/TIER3-BRINGUP.md

Tier-3 Bring-up Bug Report — elliott-ThinkPad (2026-05-01)

Bugs found and fixed during the first real-exploit fleet run on this host. All fixes are in the commits following the Dev_REL1_043026 merge of main.

---

Bug 1 — BRIDGE env var breaks Tier-3 target VM networking

Symptom: All Tier-3 slots timeout at 300 s waiting for the target service. QEMU starts with netdev tap instead of netdev user (SLIRP).

Root cause: launch_target.sh checks BRIDGE to switch between SLIRP and tap networking. The fleet runner copied the parent environment (which had BRIDGE=br-malware from the Tier-2 tap setup) into the Tier-3 subprocess. The Tier-3 target VMs don't have a tap interface configured, so all guest traffic is dropped.

Fix: fleet.py _run_slot() now calls env.pop("BRIDGE", None) before launching run_tier3_demo.py. Tier-2 idle VMs continue to use tap; Tier-3 target VMs always use SLIRP+hostfwd.

Files: orchestrator/fleet.py

---

Bug 2 — Bridge-requiring modules selected when BRIDGE is not available

Symptom: distccd_command_exec and php_cgi_arg_injection appear in usable_modules even on SLIRP-only runs. Exploit fires but the reverse-shell payload can't call back (no guest egress on restrict=on).

Root cause: usable_modules filtering was conditioned on bridge_iface being set in the environment. When BRIDGE was not set, ALL modules were considered usable. Modules that require bridge egress (reverse shells) silently fell through, fired, and timed out waiting for a session.

Fix: usable_modules now always filters requires_bridge=True modules regardless of the BRIDGE env var. The requires_bridge field in the module TOML is authoritative.

Files: orchestrator/fleet.py, exploits/modules/*.toml

---

Bug 3 — cmd/unix/interact creates no persistent session

Symptom: samba_usermap_script fires (job_id=None), no session appears in session.list after 30 s. The exploit succeeds on the wire but the driver reports session_open_timeout.

Root cause: cmd/unix/interact is a console-only payload. It attaches directly to the module's job console — it does NOT create a background Meterpreter/shell session visible via session.list. msfrpcd's module.execute returns job_id=None (no background job), and wait_for_new_session polls forever.

Fix: Changed payload to cmd/unix/bind_perl with LPORT=4444. The bind-shell payload instructs the guest to listen on LPORT; msfrpcd connects to RHOSTS:LPORT after the exploit fires, creating a proper shell session.

Files: exploits/modules/samba_usermap_script.toml

---

Bug 4 — Per-slot LPORT/hostfwd port mapping wrong

Symptom: For slots 1+, the bind-shell port is reachable on the host but msfrpcd cannot connect. ss -tlnp on the host shows port 5444 listening (QEMU) but the module tries to connect to port 4444.

Root cause: The extra hostfwd was host:5444→guest:4444 (old guest port) but FLEET_PAYLOAD_LPORT=5444 instructed the guest bind_perl to listen on 5444. Mismatch: guest binds 5444, hostfwd forwards host:5444→guest:4444. No path.

Fix: Extra hostfwd now uses extra_host_port:extra_host_port on both sides. extra_host_port = base_port + slot * 1000 is the per-slot LPORT, and the guest binds that exact port.

Files: orchestrator/fleet.py

---

Bug 5 — vsftpd module port 6200 collision across concurrent slots

Symptom: Multiple Tier-3 slots running vsftpd_234_backdoor all try to hostfwd port 6200 (the backdoor bind port). QEMU for slots 1+ fail to start because port 6200 is already bound by slot 0's QEMU.

Root cause: vsftpd's backdoor hardcodes port 6200 in both the vulnerable binary and the Metasploit module. There is no LPORT override possible. With SLIRP+hostfwd, all concurrent slots must use the same host port, which is impossible.

Fix: Marked vsftpd_234_backdoor.toml with requires_bridge = true. The fleet runner filters it from usable_modules on SLIRP runs. When a bridge is available each guest gets its own IP, and msfrpcd connects to guest_ip:6200 directly.

Files: exploits/modules/vsftpd_234_backdoor.toml

---

Bug 6 — SLIRP false-positive in _wait_for_tcp causes premature exploit fire

Symptom: Log shows "target service is up" within 0.5 s of QEMU start. The exploit fires at t=10 s (end of clean phase) but Metasploitable2 needs 30–60 s to boot Samba. Result: session_open_timeout every episode.

Root cause: SLIRP's usermode TCP stack completes the TCP three-way handshake (SYN-ACK) immediately for any port that has a hostfwd rule, regardless of whether the guest OS has booted. A bare socket.create_connection() always succeeds. Even a recv() with a short timeout (0.5 s) fires with socket.timeout because during very early boot SLIRP cannot RST the connection (the guest TCP stack is not up yet), so the connection hangs open and the recv deadline fires before SLIRP can determine the guest state.

Fix: Replaced _wait_for_tcp with _wait_for_serial_login. The new function connects to QEMU's serial console socket (serial.sock) right after the pidfile appears and streams boot output until "login:" is seen. The serial console is authoritative: it reflects actual guest OS state, not SLIRP's synthetic TCP layer.

Timing:

• serial.sock is created by QEMU at device init, before the pidfile.
• We connect immediately after the pidfile → we receive all boot output.
• Metasploitable2 prints "metasploitable login:" ≈ 50–70 s after QEMU start.
• The clean phase (10 s) runs AFTER the login prompt, so the exploit fires when Samba is reliably up.

Files: tools/run_tier3_demo.py

---

Bug 7 — Stale QEMU processes hold hostfwd ports across orchestrator restarts

Symptom: After a systemd restart of cis490-orchestrator, the new wave's QEMU processes fail to bind their hostfwd ports (e.g., 2139). The old QEMU from the previous wave is still running (QEMU is started with start_new_session=True so it survives the orchestrator's SIGTERM). The new episode detects the stale QEMU answering the port probe and proceeds as if the target is up — but the stale QEMU has different hostfwd mappings (no bind port for the current module), so the exploit never lands.

Fix: run_tier3_demo.py reads the old qemu.pid file from the run directory before recreating it. If a PID is found, os.killpg(pgid, SIGTERM) terminates the old QEMU process group, followed by a 1.5 s sleep to let QEMU exit before the port is rebound.

Files: tools/run_tier3_demo.py

---

Bug 8 — PORT_BASE default uses privileged ports (< 1024)

Symptom: launch_target.sh's default PORT_BASE was 21 + SLOT * 100. On Tier-2 hosts without Metasploitable2, standalone run_tier3_demo.py tries to bind port 21 on loopback. The cis490 service user cannot bind ports < 1024. QEMU exits immediately.

Fix: Default changed to 2021 + SLOT * 100. Port 2021 is above 1024 and reflects the scheme used by the fleet runner (base_port + 2000).

Files: vm/launch_target.sh, scripts/install-tier-3-4.sh

---

Bug 9 — msfrpc module.execute response is raw msgpack bytes, not str

Symptom: Key lookups on the module.execute response raise KeyError or fail silently because msgpack returns bin type (bytes) for all string values, even with raw=False on some Metasploit 6.x builds.

Fix: Added MSFRpcClient._str() to recursively decode bytes→str in all msgpack response dicts. Applied to module.execute and session.list.

Files: exploits/msfrpc.py

---

Bug 10 — _wait_for_tcp returns success on b'' (connection-closed-by-peer)

Symptom: Log shows "target service is up" within 0.5 s of the 65 s boot floor, but all exploit fires time out waiting for a session. FTP (port 21), Samba (139), and distccd (3632) all returned b''. The VM's services were not up; the probe was wrong.

Root cause: When recv(1) returns b'' (empty bytes), Python raises no exception. The code fell through to return, incorrectly reporting "service is up". b'' means SLIRP forwarded the connection to the guest, the guest's TCP stack RST'd (no service listening), and SLIRP converted RST→FIN → the host sees connection closed. Only socket.timeout (remote end holding the connection open, waiting for client data) and non-empty data (banner received) are genuine ready signals.

Fix: Changed recv(1) to save the return value. On socket.timeout, return immediately (genuine up). On non-empty data, return (banner). On b'', set last_err and continue (retry).

Files: tools/run_tier3_demo.py

---

Bug 11 — distccd and unreal_ircd incorrectly marked requires_bridge = true

Symptom: distcc_exec and unreal_ircd_3281_backdoor were filtered from usable_modules on every SLIRP-only run, even though their cmd/unix/bind_perl payloads create an inward-connecting bind shell (host connects to guest), which does NOT require the bridge.

Root cause: The comment in distccd_command_exec.toml said "needs bridge so the guest can reach the attacker" — correct for reverse_tcp payloads, wrong for bind_perl. bind_perl listens on the guest; msfrpcd connects to the hostfwd'd loopback port. No guest egress is needed.

Fix: Set requires_bridge = false in both modules. The fleet already adds per-slot hostfwd entries for extra_target_ports, so these modules now work on SLIRP+hostfwd runs without any other change.

Files: exploits/modules/distccd_command_exec.toml, exploits/modules/unreal_ircd_3281_backdoor.toml

---

Bug 12 — msgpack.unpackb crashes on integer session IDs

Symptom: wait_for_new_session raises ValueError: int is not allowed for map key when msfrpcd returns a session dict keyed by integer session IDs. Traceback seen in slot-0 logs on 2026-05-01.

Root cause: msgpack.unpackb(raw, raw=False) defaults to strict_map_key=True, which rejects non-string keys. Metasploit 6.x msfrpcd encodes session IDs as msgpack int64 map keys.

Fix: Added strict_map_key=False to the unpackb call in _raw_call.

Files: exploits/msfrpc.py

---

Bug 13 — samba_usermap_script never opens a session (removed from catalog)

Symptom: multi/samba/usermap_script fired, port 4444 bound in guest, but Metasploit reported Rex::Proto::SMB::Exceptions::NoReply on every run. session.list stayed empty for the full 30 s timeout.

Root cause: The SMB auth connection is disrupted when Samba's username map script executes the injected command (smbd kills the auth handler). Metasploit never received an SMB response → marked exploit "failed" → skipped calling the bind-shell handler → session never created.

Fix: Removed samba_usermap_script.toml from the catalog. The fleet now uses distccd_command_exec and unreal_ircd_3281_backdoor as SLIRP-capable modules (see Bug 11 fix). Both protocols return a proper response after the exploit fires, so Metasploit's handler is called and sessions open.

Files: exploits/modules/samba_usermap_script.toml (deleted), orchestrator/fleet.py

---

Bug 14 — QEMU launch config incompatible with Metasploitable2 (boot hang)

Symptom: Every _wait_for_tcp probe returns b'' for the full timeout (even after the Bug 10 fix). No service — FTP, Samba, distccd, IRC — ever becomes reachable. The VM consumes CPU (QEMU runs) but nothing listens.

Root cause (three compounding issues in launch_target.sh):

1. -drive if=virtio presents the disk as /dev/vda. Metasploitable2's GRUB was built for VMware SCSI (/dev/sda). Ubuntu 8.04's kernel command line says root=/dev/sda1. The kernel can't mount root on /dev/vda → kernel panic immediately after decompression. Services never start.

2. -machine q35 is a PCIe chipset (Sandy Bridge era). Old ISA-emulated devices and BIOS assumptions in Ubuntu 8.04 break under q35.

3. -cpu host exposes AVX/XSAVE and other modern CPU features. Linux 2.6.24 doesn't know how to save/restore these in context switches; the kernel freezes or mishandles the first SIMD operation during boot.

Fix: Three changes in vm/launch_target.sh:

• -machine q35 → -machine pc (i440fx, the classic PC compatible machine)
• -drive if=virtio → -drive if=ide (Ubuntu 8.04 libata presents this as /dev/sda, matching the GRUB root= line)
• -cpu host (KVM) → -cpu kvm32 (safe 32-bit KVM model, no exotic flags)
• -device virtio-net-pci → -device e1000 (Intel e1000: universally supported since Linux 2.2, in every kernel config Metasploitable2 uses)

Files: vm/launch_target.sh

---

Bug 15 — Tier-3 verify uses vsftpd (bridge-only, always fails on SLIRP)

Symptom: install-tier-3-4.sh verify step always fails. The vsftpd module's backdoor opens port 6200 (hardcoded in the binary and the MSF module). On SLIRP, all slots would need to share the same host port 6200, which QEMU refuses. The verify is killed by _wait_for_tcp or by the exploit itself never reaching a session.

Root cause: The verify step was left on vsftpd_234_backdoor after Bug 5 marked that module requires_bridge = true. The verify subprocess doesn't have a bridge configured and doesn't set up the extra hostfwd for port 6200.

Fix: Changed verify to distccd_command_exec with correct SLIRP port mappings: TARGET_PORTS="5632:3632,4444:4444" and --target-port 5632. distccd doesn't hardcode a backdoor port — the bind shell uses the fleet-assigned LPORT. No bridge needed.

Files: scripts/install-tier-3-4.sh

---

Net result after all fixes

With fixes 1–15 applied:

• Metasploitable2 boots correctly under KVM (pc machine, kvm32 CPU, ide disk, e1000 network). Services start ~60–70 s after QEMU launch.
• _wait_for_tcp correctly waits until a service is genuinely listening (returns only on socket.timeout or non-empty banner data).
• distccd_command_exec and unreal_ircd_3281_backdoor are admitted to the module catalog; both are SLIRP-compatible with cmd/unix/bind_perl.
• samba_usermap_script removed from catalog (NoReply, sessions never open).
• msgpack.unpackb accepts integer session ID keys without crashing.
• The verify step uses distccd_command_exec on SLIRP+hostfwd.
• Sessions open, workloads execute, episodes complete with session_open events.