Supersedes ISE_v10_Checkpoint_merged_8_DM2S_NX_Master_Post_Chapter_3_CLOSED.md (_8, _7 and _5 are not overwritten). Everything in _8, _7 and _5 still stands unless stated here. Read them together. Checkpoint_PDFviewer_Chunks_20261020_1910_2.md is reference only (the PDF viewer work was given up).
Do not deploy anything from PDFIndexer_v8.8.0.zip or ISE_PDF_indexers_v1.0.0.zip. I (Claude) misread "get rid of it" as "remove PyMuPDF from the PDF indexers" and built both zips on that basis. He meant only the failed MuPDF build. He needs PyMuPDF and the two-extractor union. The files he uploaded, PDFIndexer.py v8.7.1 and PDF-32bit-Indexer.py v8.6.1, were never changed on his side and need no edit. The manual edits in those zips (pypdf only, --fast a no-op, "Master v8") describe the wrong thing and must not be merged.
Lesson: when an instruction would remove something he depends on, say the reading back and wait for a yes before building.
python3, py, python are aliases to python3.15. Cron and #!/usr/bin/env python3 scripts do not see aliases: they use /usr/local/bin/python3, still a symlink to python3.8 (since 5 Oct). He wants everything on 3.15, nothing left on 3.8.pip 26.2.1 and PyMySQL 1.2.3. Live 3.8 has about 40 packages, including mysql-connector-python 9.0.0, numpy 1.24.4, torch 1.10.0a0, transformers 4.37.2, PyMuPDF 1.18.17, pypdf 3.1.0, Pillow, mutagen, requests, PyYAML, open_clip_torch, timm.py MediaIndexer.py works (ISE v9.9.10, 2902 items, none added). Live's clock is 3 hours behind Sydney and logs +0800./bin scripts and the ISE folder's *.py: mysql (8 files), numpy (3), torch (3), transformers (2), fitz (2). Everything else is standard library or his own modules. Not yet checked: whether the 8 mysql files use mysql.connector or pymysql (grep -ln "mysql.connector"), and what MediaProcessor.py reaches through AudioProcessor.py and audio_features.py (numpy).qwen_chat.py, audio_features.py, clap_diag.py, clap_smoketest.py, PDFIndexer.py, PDF-32bit-Indexer.py./bin masters live on Clone): indexall 33, 40, 42; indeximages 90, 95, 96; indeximages2 10, 14, 15; chkindeximages 2 (line 4 is a comment); mediawatchdog 37 (PY=${PY:-python3.8}); coichecks 5; ISE folder MegaIndexer.sh 4, 8, 12 and MediaClassifier.sh 9, 11. qwen_chat.py, audio_features_list.py and qf_Mediasearch_bridge.php also matched and were not inspected.| Package | Result |
|---|---|
mysql-connector-python |
Pure Python. Never confirmed installed (it was in the same command that aborted). Install on its own. |
numpy |
No wheel for CentOS 6. Source build died on cmake (no C++ compiler for C++11). Needs source /opt/rh/devtoolset-9/enable first; the newest numpy may still want more than gcc 9.1 offers. |
PyMuPDF 2.0 |
Source build. First failure: certificates (fixed, see below). Then -std=c++20 rejected by gcc 9 (barcode code). No switch in PyMuPDF to turn the barcode build off. |
PyMuPDF 1.26.x (<1.27) |
Source build of MuPDF 1.26.12 got further but failed; the real compiler error scrolled off and was not seen. |
torch, transformers |
Not attempted. No wheel expected for glibc 2.12 or 2.17. Largest risk. |
Fixes that worked: export SSL_CERT_FILE=$(python3.15 -c "import pip._vendor.certifi as c; print(c.where())") makes the build's downloads verify. A failed build in one pip install aborts everything in that command, so install packages one at a time.
Next step he has not yet run (read-only, builds nothing):
python3.15 -m pip debug --verbose 2>/dev/null | grep -i "manylinux2014" | head -3
pip315 install --only-binary :all: "PyMuPDF<1.26"
python3.15 -c "import fitz; print(fitz.__doc__)"Checked on PyPI at the end of this conversation (20 Oct, ~23:15): ready-made wheels DO exist. PyMuPDF 1.24.12 up to 1.26.0 ship one cp39-abi3-manylinux2014_x86_64 wheel (also tagged manylinux_2_17), and "abi3" means one file works on every Python from 3.9 up, so Python 3.15 is covered. 1.26.1 and later need manylinux_2_28 (glibc 2.28), which Live cannot meet; do not use them. Best pick: 1.25.5 (last of the 1.25 line, MuPDF inside, nothing to compile). Possible snag: pip on Live reports glibc 2.12, so it may refuse a manylinux2014 (glibc 2.17) wheel even though the 2.17 launcher is in use. If pip315 install --only-binary :all: "PyMuPDF==1.25.5" says "no matching distribution", download the .whl from PyPI on the NUC, rename its platform tag to linux_x86_64 (...-cp39-abi3-linux_x86_64.whl), copy it to Live and pip315 install that file. Second possible snag: the library inside may want a newer libstdc++ than Live's gcc 4.4 one; if import fitz then names GLIBCXX_3.4.xx, that is the fix to chase. Also note: the older 1.18.17 on Live's 3.8 is what the 32-bit/Dev notes call getText(); 1.25.x only has get_text() (the 64-bit indexer already uses that).
The original idea: an older PyMuPDF may have a ready-made wheel with MuPDF inside, so nothing is compiled. If one installs but will not import, the error will name a missing library (likely a newer C++ library) and the fix is a library path in the python3.15 launcher. If no wheel exists, the fallback is compiling an older PyMuPDF with devtoolset-9 and saving the full log (> /tmp/pym.log 2>&1, then grep -n "error").
--fast skips pypdf (PyMuPDF only). --fastest skips file(1) on already-indexed attachments and implies --fast.10 0 */2 * * ... PDFIndexer.py --monitor ("union rebuild") is not a rebuild: without --fresh it only processes PDFs not yet indexed. The 6-hourly --fast job at 00:05 indexes new PDFs first, so the 00:10 union run normally finds nothing left. His real schedule is unknown (crontab -l on Live showed no Python lines; jobs run through MegaIndexer.sh, which calls python3.8 PDFIndexer.py --fast --monitor on line 8).--fresh rebuild.PDF-32bit-Indexer.py (Dev) has no #do-not-index handling, so marked posts' PDFs get indexed on Dev.PDFIndexer.py's tokeniser is a plain .split(). The code has used a regex tokeniser since v7.4. Not corrected.MediaProcessor.py v9.30 and MediaIndexer.py v9.9.10 present. /bin/mirrorISE_Clone present (Oct 16). Clock correct (checked 20:32 and 22:06). No future-dated files under /var/www/html or /bin.cleanISE_Clone v1.0.4 is now installed in the ISE folder (/var/www/html/The_ISE_Project, owner apache) with its manual, put there at 22:10. It is not in /bin. He says it runs and cron is fine. Not confirmed whether it has been run with --apply.pip315 install pymysql, and fixing the py alias.mysql-connector-python, then numpy with devtoolset-9, then decide on torch (only qwen_chat.py and the clap_* test scripts need it).PDFsearch.py and ise_noindex.py imports for 3.15 (they probably need mysql-connector-python)./bin files from Clone. Timing is his call._8: backup copy of Clone's VM folder after a clean shutdown; the four old-IP /bin scripts on Clone (json_md5sums_all, copies_Dev_to_Clone, copies_Clone_to_Dev, scp.text); stale http:// links (grep -n boardurl /var/www/html/Settings.php); Dev's hostname output for the indexers' IS_DEV check; swap and /etc/fstab on Clone; what "do not index all that" meant; the parked list in _8 section 5, item 10.