📝 sluggish-system-transfer-checklist.mdTheme:

System Sluggishness / Slow-Transfer Diagnostic Checklist

For: "whole system feels slow" and/or "rsync/scp is crawling and it's not network." Environment: NUC (Windows 10 host) → VMware Dev VM (CentOS 6.10) ↔ Live (CentOS 6.10, 2 CPU/2GB RAM, 30GB SSD, forum.drugs-and-users.org)

Run top-to-bottom. Each block is cheap and rules things in/out fast — don't skip ahead on a hunch, but do skip a block entirely if it's already flagged "known false lead" below for this environment and nothing has changed since.

0. Scope the symptom first (30 seconds)

[ ] Is it ONE VM, or does the Windows host itself also feel slow (mouse lag, other apps stuttering)? → if host is slow too, this is a host-level resource problem, not a guest/network one — jump to §4.
[ ] Is it ALL transfers/processes, or specifically rsync/scp? → if only rsync, check §3 (competing processes) and §5 (rsync-specific) first.
[ ] Did anything change recently — disk resize, snapshot, backup job, VM reboot, host reboot, Windows update? Note it; it's usually the actual lead.

1. Per-VM resource pressure (Dev, then Live)

[ ] vmstat 1 5 — watch si/so (swap activity). Non-zero = memory pressure, real bottleneck.
[ ] free -m — check swap used, not just RAM free.
[ ] top/ps aux --sort=-%cpu — anything unexpected pegging a core.
[ ] uptime — load average vs core count (2 cores = load >2 sustained is saturated).

2. Disk I/O

[ ] iostat -x 1 5 — high %util or await = disk is the bottleneck (SSD is 12+ years old on Live — degraded/failing SSD is a known plausible failure mode here).
[ ] df -h — check for near-full disks (silently degrades MySQL/logging well before 100%).

3. Competing/leftover processes

[ ] ps aux | grep -E 'rsync|scp' on BOTH ends — a stuck/orphaned prior transfer competing for bandwidth or I/O has caused this exact symptom before on this setup. Kill and retest before going further.
[ ] Any cron jobs, backups, or the mirror.sh job overlapping in time with the slow transfer?

4. Host-layer (Windows/VMware) — only if §0 says the host itself is sluggish

[ ] Windows Task Manager — CPU/RAM/disk on the host itself, not just inside the VM.
[ ] VMware resource allocation — has the VM been given more vCPU/RAM than the host can comfortably spare alongside other running VMs/apps?
[ ] Reboot the host — clears transient kernel/driver/process state; low cost, previously used as a blunt but valid step.

5. Rsync/transfer-specific

[ ] Confirm which direction/subdir is slow — full serial rsync vs a single large new (unmatched) file behaves very differently from small already-matched files (delta-transfer can look deceptively fast).
[ ] If truly ruling out network: iperf3 between the two ends in isolation from SSH/rsync overhead entirely (raw throughput, no crypto/protocol layer).
[ ] Compare against a fresh dd-style raw copy of a large test file vs the rsync of the same file — isolates rsync/SSH overhead from raw transfer capacity.

6. Filesystem/LVM integrity (lower priority — expensive to check, rarely the cause here)

[ ] dmesg / /var/log/messages for filesystem errors, unclean-shutdown recovery, remounts.
[ ] pvs/vgs/lvs for LVM-level errors.
[ ] Full fsck only if the above shows something — do this from a live/rescue context, not from inside the running guest, since a previous in-guest fsck -f before a Clonezilla clone crashed the VM mid-check.

Known false leads in this environment (don't re-spend time here unless new evidence points back)

Filesystem fragmentation — checked exhaustively on both Dev and Live via filefrag, consistently clean.
TCP-layer causes (retransmits, cwnd, duplex, offload settings, tc/iptables shaping) — all checked clean in the prior saga.
IPv6 fallback / /etc/hosts — ruled out.
CSF/fail2ban blocking — ruled out (not installed / IP not banned).

Actual root cause found last time (for pattern-matching, not assumption)

High baseline latency + ~10% packet loss on Andrew's own connection (~390–400ms RTT) was crushing single-stream sustained transfers via slow-start/retransmit cost, while short multi-stream speedtests looked fine. This time is explicitly stated as not a network issue — so if §1–§5 all come back clean, the RTT/loss check is still worth 30 seconds of ping sanity before ruling it out entirely, since "not a network issue" was also the working assumption for months last time before it turned out to be exactly that.

Escalation trigger

If you've been through §1–§5 with no smoking gun and the same fix has been tried more than once (or the same problem that was previously called resolved has come back — as it may have here, since the prior Dev↔Live rsync slowdown was marked resolved by recreating Live's /var/www/html), stop generating more same-domain guesses. Surface the full list of what's been checked and consider a domain jump (hardware, host-layer, or a fresh model's perspective) rather than another software-layer variant.