Supersedes ISE_v10_Checkpoint_merged_9_DM2S_NX_Master_Post_Chapter_4 CLOSED.md (_9, _8, _7 and _5 are not overwritten). Everything in them still stands unless stated here. Read them together.
Python 3.15 now works for the whole PDF path on both Live and Clone. The media and audio pipelines (MediaProcessor.py, AudioProcessor.py) cannot move: torch has no Python 3.15 build, so they stay on 3.8. Nothing was deployed to production this chapter; cron and #!/usr/bin/env python3 scripts still run 3.8 (/usr/local/bin/python3 is still a symlink to python3.8).
| Item | State |
|---|---|
Launcher /usr/local/bin/python3.15 |
/opt/python-3.15.0/extralib: added first in --library-path; backup python3.15.before_extralib beside it |
/opt/python-3.15.0/extralib |
libstdc++.so.6.0.19 (the C++ runtime library) taken from CentOS 7's libstdc++-4.8.5-44.el7 RPM; Clone's copy also has ~ backup files left by the transfer |
PyMuPDF |
1.25.5, ready-made wheel (MuPDF 1.25.6 inside), nothing compiled |
mysql-connector-python |
26.7.0 |
PyMySQL |
1.2.3 |
pypdf |
3.1.0, pinned (same as Live's 3.8). 6.21.1 mangles some fonts, see section 4 |
fonttools 4.67.0 |
Live only; unused by pypdf 3.1.0, harmless |
| Not installed | numpy, torch, transformers, open_clip_torch, Pillow |
Clone ~/.bashrc |
Aliases py, py3, py315, python, python3, pip, pip3, pip315, py36, py38 now present (the py line was missing and added separately). Live's ~/.bashrc lines 136-147 are the reference |
Why the library copy was needed: the PyMuPDF wheel asks for GLIBCXX_3.4.19; the system libstdc++.so.6 (6.0.13, gcc 4.4) lacks it. The launcher runs Python through glibc 2.17's own loader with an explicit --library-path, which overrides LD_LIBRARY_PATH, so the fix is the path entry in the launcher, not an environment variable.
manylinux2014 container on the NUC (not confirmed that image has 3.15). Decision: leave numpy off 3.15. Only audio_features.py and AudioProcessor.py need it, and AudioProcessor.py also needs torch.pip download --only-binary :all: torch on Live: "from versions: none". Likely no 3.15 build published yet (a fonttools cp315 glibc 2.17 wheel installed fine, so Live's pip does accept such wheels). Compiling it on CentOS 6 is not realistic.PIP_CONSTRAINT inside build environments; use --build-constraint FILE. (2) /tmp/pipc.txt held ninja<1.12 and patchelf<0.18; those versions have wheels for old glibc, which avoids building CMake. (3) Building CMake 4.4.4 from source fails: its bundled curl wants OpenSSL 1.1 headers, the system has 1.0. (4) Install packages one at a time; a failed build aborts the whole command._9 claim "only qwen_chat.py and the clap_* scripts need torch" came from a search that missed indented imports. MediaProcessor.py imports torch, open_clip and PIL inside a function (around lines 510-512, 561, 676) but does not import AudioProcessor or audio_features. AudioProcessor.py imports audio_features (numpy) at the top and torch, transformers, tokenizers lazily.mysql.connector (9 files): IndexBuilder, ise_noindex, media_diag, MediaSearch, MiscIndexer, MiscSearch, orphan_check, pdf_duplicate_spotter, PDFsearch. pymysql (4): AudioProcessor, ise_noindex (uses both), MediaIndexer, MediaProcessor. Nothing in /bin imports either. Open risk: Live's 3.8 has connector 9.0.0, 3.15 has 26.7.0; only imports were checked, no real search through 3.15 yet./bin is hand-managed and out of scope. Dev stays on Python 3.8.18 (3.6 also there) with its 32-bit PyMuPDF 1.18.17; there is no 3.15 for 32-bit, and the planned 64-bit Dev rebuild is the route (it would then follow the Clone recipe above).Dry runs (--dry-run --limit 5); per-engine "total terms found / unique-to-that-engine":
| Setup | PyMuPDF | pypdf |
|---|---|---|
| 3.8 (PyMuPDF 1.18.17, pypdf 3.1.0) | 63,265 / 1,330 | 63,444 / 1,509 |
| 3.15, pypdf 6.21.1, no fontTools | 63,237 / 866 | 63,364 / 993 |
| 3.15, pypdf 6.21.1 + fontTools | 63,237 / 2,064 | 62,842 / 1,669 |
| 3.15, pypdf 3.1.0 (adopted) | 63,237 / 1,427 | 63,444 / 1,634 |
attach-162 produced run-on junk (spaces decoded as letters, e.g. words fused into one long string). That is why pypdf is pinned at 3.1.0; an unpinned upgrade brings it back.--monitor shows "0 unique words" in a dry run (no index is kept), so it cannot compare vocabularies. --single needs a real attachment number (an earlier run used the placeholder NNN and was rejected by the argument parser).PDFIndexer.py v8.7.1 starts and runs under 3.15 on Live and Clone. The only patch needed to move PDF indexing is MegaIndexer.sh line 8 (python3.8 PDFIndexer.py --fast --monitor). Timing is the operator's call./bin is the master)cmp confirmed indexall, indeximages and mediawatchdog are identical in Clone's /bin and /var/www/html/Collabware/bin.
File (Clone /bin) |
Line(s) naming 3.8 | Can it move to 3.15? |
|---|---|---|
indexall |
33 (Check_Media_Classification.py), 40 (MediaIndexer.py), 42 (MediaProcessor.py) |
40 yes; 42 no (torch); 33 not yet checked |
indeximages |
90 (Check_Media_Classification.py), 95 (MediaIndexer.py), 96 (MediaProcessor.py) |
95 yes; 96 no; 90 not yet checked |
mediawatchdog |
37 (PY=${PY:-python3.8}) |
runs MediaProcessor, stays on 3.8 |
audio_plan (2), audio_run (1) |
call AudioProcessor.py |
no (numpy and torch) |
bin_compare_sysplex.sh |
9 (runs /root/bin_compare) |
needs a 3.15 compatibility look |
mediawatchdog.md |
26, 39, 51 | documentation wording only |
bin_deep_check.lst, sysplex_python_environment |
records, not commands | no patch |
Differences from the list in _9: chkindeximages in /bin (157 bytes, 12 Oct) no longer names 3.8, while the Collabware/bin copy (165 bytes, 6 Oct) still does; coichecks exists only in Collabware/bin (Live line 5); indeximages2 is not on Clone at all. The ISE folder's MegaIndexer.sh (4, 8, 12) and MediaClassifier.sh (9, 11) are unchanged from _9. So indexall and indeximages would become mixed (3.15 for MediaIndexer, 3.8 for MediaProcessor); whether to split them that way is the operator's decision.
crontab -l on Live showed no Python lines, jobs run through MegaIndexer.sh).Check_Media_Classification.py imports; run one real PDFsearch.py search through 3.15 to test the connector version gap./var/spool/mail/root) has new mail twice this session; probably cron output, unread._8 and _9: backup copy of Clone's VM folder after a clean shutdown; the four old-IP /bin scripts on Clone (json_md5sums_all, copies_Dev_to_Clone, copies_Clone_to_Dev, scp.text); stale http:// links (grep -n boardurl /var/www/html/Settings.php); Dev's hostname output for the indexers' IS_DEV check; swap and /etc/fstab on Clone; what "do not index all that" meant; the parked list in _8 section 5, item 10.PDFIndexer.py tokeniser is described as a plain .split() (a regex tokeniser since v7.4); the "union rebuild" cron example is not a rebuild; the blank-page numbering quirk (page number one too low after a blank page) needs a --fresh rebuild to fix./tmp/cxx, /tmp/wh, /tmp/pipc.txt, /tmp/numpy*.log, /tmp/pdf*.txt, /tmp/mu*.txt, /tmp/mu_words.py.fontTools; it made pypdf 6 worse on these fonts. Fixed by pinning pypdf 3.1.0.--only-binary "numpy<2"; wrong, numpy 1.x has no 3.15 support.NNN; give a real example value.cp asking to overwrite) swallows the rest of the paste.