📝 PyTorch_Checkpoint-Collabware_2026-08-30-PyTorch-CLIP-Success.md

Checkpoint (2026-08-30) — PyTorch + open_clip WORKING on Clone. Saga Complete.

Status: SUCCESS. torch, torchvision, Pillow, and open_clip all import and function correctly on Clone (CentOS 6.10, Python 3.8.18, devtoolset-7). A real OpenAI CLIP ViT-B-32 checkpoint downloaded and loaded successfully via open_clip.create_model_and_transforms(). Andrew is imaging Clone NOW (full VM shutdown + disk copy) to preserve this exact working state before anything else touches it.

Current Project State

The end of a multi-session saga

This closes out a build effort that spanned multiple sessions (2026-08-23 through 2026-08-30) and three AI collaborators (Claude, Gemini, ChatGPT), starting from the original hard constraint: Live's CentOS 6 glibc 2.12 categorically cannot run torch (ImportError: GLIBC_2.14 not found), and a direct glibc upgrade on Live was ruled out as too dangerous. The entire effort moved to a disposable "Clone" VM to trial builds safely.

Multiple full build attempts failed across earlier sessions on a genuine, hard-to-diagnose multiple definition of DispatchStub linker error. The real root cause (found 2026-08-29): PyTorch's per-CPU-capability dispatch codegen writes wrapper files (Activation.cpp.DEFAULT.cpp, Activation.cpp.AVX2.cpp, etc.) directly into the source tree (aten/src/ATen/native/cpu/), not into build/. Every earlier session's rm -rf build/ never touched this pollution, so it silently compounded across restarts into absurd stacked filenames and eventually broke linking. Fixed via a three-layer git clean (Claude's targeted clean → Andrew's broader top-level git clean -fdx → Gemini's git submodule foreach --recursive git clean -ffdx, which also caught staleness hidden inside individual submodules).

Bugs found and fixed on the way to a clean build (2026-08-29/30)

  1. Python ABI mismatch: CMake's Python auto-detection silently picked up a stale system libpython3.6.3 for linking despite finding the 3.8.18 interpreter correctly. Fixed by explicitly pinning -DPYTHON_LIBRARY / -DPYTHON_INCLUDE_DIR to the real 3.8 paths.
  2. Hollowed-out submodules: the deep git submodule foreach clean was too thorough and deleted genuinely tracked files (not just build cruft) from third_party/foxi, third_party/QNNPACK, and third_party/python-enum. Fixed with git reset --hard HEAD inside each affected submodule.
  3. Missing swap / OOM risk: added a swapfile (started at 2GB, later increased to 6GB) as a safety margin rather than disabling the Linux OOM killer outright — deliberately rejected disabling it, since that risks a full unrecoverable system hang on a genuine leak instead of a clean single-process kill.
  4. cc1plus OOM kill on VariableType_2.cpp: PyTorch's huge generated autograd files need several GB per compile unit; MAX_JOBS=4 briefly pushed memory to the edge. Rebooted, increased swap to 6GB, retried — completed clean.
  5. distutils.version AttributeError: tools/setup_helpers/cmake.py used from setuptools import distutils without importing the version submodule explicitly — newer setuptools no longer auto-attaches it. Fixed with an explicit import distutils.version line.
  6. Ninja/Makefiles generator mismatch: manual builds used Unix Makefiles; pip install -e . hardcodes -GNinja internally (env var override does NOT work — it's hardcoded in the build script, confirmed by testing). Resolved by wiping build/ and letting pip install -e . do a full from-scratch Ninja build.
  7. third_party/breakpad missing directory: -e .'s default cmake invocation doesn't pass -DUSE_BREAKPAD=OFF the way the manual command did, and the actual breakpad submodule directory is gone (deleted upstream at some point). Fixed by passing USE_BREAKPAD=0 as an environment variable to pip install -e ..
  8. A broken half-finished patch to cmake.py: at some point during troubleshooting, distutils.version.LooseVersion(x) calls were incorrectly rewritten as tuple(map(int, ...))(x) — literal Python Ellipsis left in as a placeholder, called as if it were a function (TypeError: 'ellipsis' object is not iterable). Fixed with correct tuple-based version comparison: tuple(int(x) for x in line.strip().split(' ')[2].split('.')), compared against plain (3, 10, 0) tuples — no distutils dependency needed at all.
  9. Missing torch._C extension: cmake --build . --target install succeeded fully and installed all compiled .so libraries (confirmed present in torch/lib/), but the actual importable Python extension module (torch._C, built as _C.cpython-38-x86_64-linux-gnu.so directly inside the torch/ package folder) was never generated, because the packaging/metadata step that normally triggers this crashed first on the distutils bug above. Once that was fixed, running python setup.py build_ext --inplace directly (bypassing pip's packaging wrapper) built and placed _C.cpython-38-x86_64-linux-gnu.so correctly. This was the final blocker — import torch succeeded immediately after.

Confirmed working: import torch

$ python -c "import torch; print(torch.__version__)"
1.10.0a0+git71f889c

Dependency chain for open_clip_torch — Rust-toolchain wall, worked around

No Rust compiler exists on this box; building one from source was explicitly ruled out as its own multi-hour project. Two packages in the dependency chain wanted a Rust build with no available prebuilt wheel for this platform (safetensors, and huggingface-hub's newer hf-xet accelerator):

Pillow — FreeType version wall, worked around

pip install Pillow failed compiling _imagingft.c (the optional font-rendering submodule) against this CentOS 6 box's ancient system FreeType, which predates constants added in FreeType 2.5+ (FT_LOAD_COLOR, FT_PIXEL_MODE_BGRA — used for color emoji/bitmap font support, unrelated to basic image loading). Not needed at all for CLIP's actual image preprocessing pipeline. Resolved via PILLOW_DISABLE_FREETYPE=1 pip install "Pillow<10", skipping the font module during build.

Final confirmed success

import torch, open_clip, PIL
model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='openai')
tokenizer = open_clip.get_tokenizer('ViT-B-32')
# SUCCESS: OpenCLIP is fully operational!

The real OpenAI CLIP ViT-B-32 checkpoint downloaded and loaded successfully — confirming Clone's own internet access is NOT blocked the way the original sandboxed environment's egress allowlist was (that allowlist blocked huggingface.co/openaipublic.azureedge.net in an earlier, different environment months ago).

Environment summary (for reproducing this exact working state)

Immediate next step (not yet done)

Andrew is shutting Clone down cleanly and copying the full VM folder to a new sibling directory (matching the proven backup pattern from the earlier glibc-upgrade saga: plain file copy of a powered-off VM, simpler and more reliable than live snapshots) — to preserve this exact working state before anything else touches this box.

Still pending after that (the actual ISEmedia work, unblocked as of tonight)