That checkpoint covered the full build saga and the real CLIP validation success (96.65%/99.74% confidence on real photos, real video frame extraction via custom-compiled ffmpeg). This checkpoint covers what happened next: migrating that working setup to Live (the actual production forum server).
kernel-2.6.32-431.1.2.0.1.el6.x86_64 — an earlier scare about a kernel mismatch was a false alarm, resolved by checking rpm -q kernel vs uname -r), same Python 3.8.18 at the same path, same devtoolset-7. Real difference: Live has only 2GB RAM (vs Clone's 6GB), already running ~1.4GB used just serving the forum./swapfile2) alongside the existing 2GB one (Linux pools multiple swap files automatically by priority — confirmed working via swapon --summary), total 6GB swap. Set vm.swappiness=15 (down from Live's original 40) to bias the kernel toward keeping the forum's own working set in RAM, using swap only under real pressure. Made persistent: /etc/fstab entry added for /swapfile2; /etc/sysctl.conf cleaned up (found and removed a stale leftover vm.swappiness=40 line that could have silently won over the new =15 value on reboot — now only the correct line remains).Checkpoint-Collabware-Live-Migration-Master-Plan) covering the same material with a structured "Rules of Engagement" — notably: /usr/bin/python must stay untouched (yum hardcodes it), no unbounded batch jobs on Live given its 2GB RAM ceiling (embedding generation must be throttled/async when built for real), and a clean division of labor (Andrew = physical env/backups/file transfer/restarts, AI = code logic/integration wiring). Checked her version's numbers against actual session facts — the one uncertain detail (Pillow version "8.1.2") was verified correct via pip show Pillow on Clone, not a real error.On Clone, disk hit 100% full mid-tar (an interrupted first attempt to archive the full pytorch/ source tree, ~1.6GB build/ intermediate directory included, pushed it over). Recovered by:
pytorch/build/ entirely (1.6GB of intermediate compile artifacts) — reasoning at the time: already had a full VM image from right after the successful build, so this seemed safe to discardpytorch/torch/ (584MB → 119MB compressed) instead of the whole source treeTransferred three archives to Live via scp: pytorch_full.tar.gz (torch/ only, 119MB), site-packages.tar.gz (64MB), ffmpeg.tar.gz (76MB). Skipped a python3.8 archive — confirmed unnecessary since Live already has the identical custom Python 3.8.18 build.
Extraction bug (fixed): archives were tarred with paths relative to /root (leading / stripped by tar automatically), but extracted with tar xzf ... -C / instead of -C /root — landed everything at /pytorch/torch/ instead of /root/pytorch/torch/. Fixed with a simple mv.
The real gap, discovered on Live: import torch failed with
OSError: /root/pytorch/torch/lib/libtorch_global_deps.so: cannot open shared object file: No such file or directoryInvestigation (find /root/pytorch/torch/lib -xtype l) found 8 broken symlinks in torch/lib/ — libtorch_global_deps.so, libshm.so, libtorch.so, libtorch_cpu.so, libnnapi_backend.so, libc10.so, libtorch_python.so, plus one bare lib symlink. All of them point at ~/pytorch/build/lib/... — the exact directory that had just been deleted on Clone for disk space, before realizing these were load-bearing symlink targets, not disposable intermediate junk. The actual compiled shared libraries (the real 100+MB .so files that make torch functional) only ever lived in build/lib/, never as real files inside torch/lib/ itself.
Confirmed unrecoverable from the existing image: Andrew's only saved Clone VM image (Operation Clean Slate) was taken deliberately before make/pip install -e . ran that final successful time — specifically as a pre-compile rollback point. build/lib/'s .so files didn't exist yet at that snapshot's timestamp, so booting it would not have recovered anything. Correctly identified this before wasting time booting/mounting that image.
Since the PyTorch source tree itself was never deleted (only build/, the compiled-artifacts directory), and every real bug from the original build saga is already fixed in that source (submodule pollution cleaned, Python library paths pinned, the broken distutils.version patch fixed, the missing _C extension issue understood) — decided to simply rerun the exact same successful install command against the still-intact, already-fixed source tree, rather than attempting any kind of partial file recovery.
Sanity-checked before starting: confirmed /root/pytorch/torch/_C.cpython-38-x86_64-linux-gnu.so (the actual importable extension module) was still present and untouched — proof the source tree really is in the same state that produced the original successful build.
Command running (started 17:55, still running as of this checkpoint, several hours later):
cd /root/pytorch
USE_BREAKPAD=0 CMAKE_C_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/gcc CMAKE_CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++ MAX_JOBS=4 pip install -e . --no-build-isolationProgress tracking: .ninja_log line count used as a rough progress proxy (each line = one completed build step). Original build's total was 6045 steps. This rebuild's step count has now exceeded that original total (last checked: 6322 and climbing) without yet finishing — the exact reason for the discrepancy isn't confirmed (possibly a slightly different dependency graph on a from-scratch build/ vs. the original's resume-after-OOM-kill sequence), but ps aux confirms real cc1plus/gcc compiler processes are still actively running, CPU pinned at 100% across all 4 cores, memory/swap healthy (well under pressure) — genuinely still working, not stuck or hung. Confirmed via two consecutive .ninja_log count checks a short time apart, both showing the number still climbing.
import torch, open_clip; print(torch.__version__) works cleanly on Clone again (sanity check before re-transferring)torch/lib/'s REAL files, not symlinks — either tar -h (dereference symlinks, copying the actual target file contents into the archive) when creating the archive, or explicitly include build/lib/*.so in the transfer alongside torch/, preserving the symlink structure. The -h/dereference approach is cleaner — avoids the same problem recurring if build/ gets cleaned up again later.-C /root path this time (not -C / — that was the earlier extraction bug)import torch, open_clip, then a real model.encode_image() test against an actual attachment, watching htop closely given Live's 2GB RAM constraintMediaIndexer.py/MediaSearch.py's existing SimilarityRule scoring/usr/bin/python must never be touched on either box (yum hardcodes it) — all custom Python work goes through /usr/local/bin/python3.8 or PATH precedence via ~/bin