📝 PyTorch_Checkpoint-Collabware_Team_Operation_Clean_Slate.md

Checkpoint-Collabware_Team_C

Checkpoint-Andrew_Operation_Clean_Slate

Current Project State

Checkpoint-Claude.md (2026-08-29) — Source Tree Pollution Found & Fixed; Clean Configure Achieved

Status: Genuinely clean cmake configure completed on a genuinely clean source tree, for the first time this entire saga. Andrew is shutting Clone down and imaging it NOW, before running make, to preserve this state. make -j3 has NOT yet been run against this configure.

The real root cause, finally found

Every prior multiple definition of DispatchStub linker failure (2026-08-26 and earlier tonight, 2026-08-29) was blamed on generic "build directory corruption from repeated reconfigures" — that diagnosis was incomplete. The actual mechanism, confirmed via git status --porcelain:

PyTorch's per-CPU-capability dispatch codegen generates wrapper files like Activation.cpp.DEFAULT.cpp and Activation.cpp.AVX2.cpp (each #includes the real source under a capability flag) directly inside the source tree (aten/src/ATen/native/cpu/), not inside build/. Every prior session's rm -rf build/ wipe never touched these, because they were never part of build/ to begin with. Each new build attempt re-scanned the directory, found these already-wrapped files sitting alongside real source, and wrapped them again — hence filenames compounding across the whole saga into absurd stacks like Activation.cpp.DEFAULT.cpp.DEFAULT.cpp.DEFAULT.cpp.DEFAULT.cpp.o and eventually causing the linker to find the same DispatchStub symbol defined twice.

This explains why the corruption survived every previous build/ wipe across multiple sessions — the actual pollution was never in the directory being wiped.

The fix (three of us in a row, each catching something the last one missed)

  1. Claude's first suggestion: git clean -fdx aten/src/ATen/native/cpu/ (targeted, narrow)
  2. Andrew ran a broader version himself: top-level git clean -fdx across the whole repo — caught additional stale generated headers (Config.h, CUDAConfig.h, generated_cpp.txt) and leftover third-party build directories (sleef/, foxi/, confu-deps/) that the narrow fix would have missed
  3. Gemini went deeper still: git submodule foreach --recursive git clean -ffdx — cleaned stale CMakeFiles/, generated config headers, and .pc files sitting inside every individual submodule (protobuf, gloo, ideep/mkl-dnn, tensorpipe, kineto, onnx, fbgemm) — a category of hidden staleness invisible to any top-level clean, since submodules are separate git repositories

Side effect / new bug introduced by the deep clean, found and fixed

The submodule-level clean removed third_party/foxi/CMakeLists.txt, breaking the subsequent cmake .. configure with:

CMake Error at cmake/Dependencies.cmake:1509 (add_subdirectory):
  The source directory /root/pytorch/third_party/foxi does not contain a CMakeLists.txt file.

foxi apparently ships this file as a genuinely tracked file (not generated), but the clean treated it as untracked cruft. Fixed with a targeted re-sync of just that one submodule:

git submodule update --init --recursive third_party/foxi

Confirmed restored via ls third_party/foxi/CMakeLists.txt.

Remaining git status (expected, harmless, left alone deliberately)

HEAD detached at v1.10.2
Changes to be committed:
  modified:   .gitmodules
  deleted:    third_party/breakpad

breakpad being deleted as a submodule is consistent with -DUSE_BREAKPAD=OFF being used throughout this whole build — looks like an intentional prior removal, not corruption. Andrew was advised to leave this alone rather than run git restore/git rm right before a build he wants to just work — one less variable.

Final cmake command that produced a clean configure (no CMake Errors)

cd ~/pytorch
mkdir build   # after the full clean above wiped the old build/ dir
cd build

cmake .. \
  -DCMAKE_C_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++ \
  -DPYTHON_EXECUTABLE=/usr/local/bin/python3.8 \
  -DPYTHON_LIBRARY=/usr/local/lib/libpython3.8.so \
  -DPYTHON_INCLUDE_DIR=/usr/local/include/python3.8 \
  -DUSE_BREAKPAD=OFF \
  -DBUILD_CAFFE2=0 \
  -DUSE_NUMPY=ON \
  -DUSE_QNNPACK=OFF \
  -DUSE_PYTORCH_QNNPACK=OFF \
  -DUSE_XNNPACK=OFF \
  -DUSE_FBGEMM=OFF \
  -DUSE_GLOW=OFF \
  -DBUILD_TEST=OFF \
  -DCMAKE_CXX_FLAGS="-w" \
  -Dprotobuf_DISABLE_WARNINGS=ON

Confirmed in the resulting log: Python consistently 3.8.18 across interpreter/library/includes, USE_BREAKPAD: OFF, and finishing with:

-- Configuring done
-- Generating done
-- Build files have been written to: /root/pytorch/build

No CMake Error lines anywhere in this run.

Run inside a devtoolset-7 shell (scl enable devtoolset-7 bash, confirmed gcc --version → 7.3.1) rather than relying solely on the CMake compiler-path flags.

Environment state at this checkpoint

Next step (NOT yet done as of this checkpoint)

cd ~/pytorch/build
make -j3

This has not been run yet against this specific clean configure. Andrew is shutting Clone down and taking a full VM image/snapshot first, specifically so this clean state (pristine source tree + clean submodules + working configure) is preserved as a rollback point regardless of what make does next — this is the first time all saga that a shutdown/image has been taken at exactly this point in the process.

Standing context (carried from prior sessions, unchanged)

Checkpoint-Gemini: PyTorch Build & System State

Summary

Current Build Configuration

Execution Command

cd ~/pytorch/build
make -j3