ISE PDF edition is operational now

[smfadmin (OP)]

py3 PDFsearch.py synaptogenesis

47 match(es) — showing top 20:
  attach-3218  page  28  [1 term(s): synaptogenesis]  attach-3218
  attach-3218  page  29  [1 term(s): synaptogenesis]  attach-3218
  attach-3218  page  35  [1 term(s): synaptogenesis]  attach-3218
  attach-3218  page  74  [1 term(s): synaptogenesis]  attach-3218
  attach-3218  page  75  [1 term(s): synaptogenesis]  attach-3218
  attach-4917  page   7  [1 term(s): synaptogenesis]  attach-4917
  attach-4960  page   9  [1 term(s): synaptogenesis]  attach-4960
  attach-5045  page   1  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   2  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   4  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   5  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   6  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   7  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   8  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page   9  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page  10  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page  11  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page  12  [1 term(s): synaptogenesis]  attach-5045
  attach-5045  page  14  [1 term(s): synaptogenesis]  attach-5045
  attach-5049  page  32  [1 term(s): synaptogenesis]  attach-5049
[18:00:05]root@forum:/var/www/html/The_ISE_Project $ mysql
[/code]

Bash script:

Code: [Select]

#!/bin/bash

ATTACH_ID=$1

mysql -u root -pegerszegi smf209 -e "
SELECT a.*, m.id_msg, m.subject, m.id_topic, t.id_first_msg, t.id_last_msg
FROM smf209_attachments AS a
LEFT JOIN smf209_messages AS m ON a.id_msg = m.id_msg
LEFT JOIN smf209_topics AS t ON m.id_topic = t.id_topic
WHERE a.id_attach = ${ATTACH_ID}\G;
"


Code: [Select]

*************************** 1. row ***************************
      id_attach: 3218
       id_thumb: 0
         id_msg: 47468
      id_member: 0
      id_folder: 1
attachment_type: 0
       filename: 1.pdf
      file_hash: b8be60726af826299155c1de139b4266b77eecf7
        fileext: pdf
           size: 18433521
      downloads: 0
          width: 0
         height: 0
      mime_type:
       approved: 1
proper_rotation: 0
         id_msg: 47468
        subject: Notes backup
       id_topic: 5988
   id_first_msg: 47468
    id_last_msg: 47468


Code: [Select]

mysql> SELECT a.*, m.id_msg, m.subject, m.id_topic, t.id_first_msg, t.id_last_msg FROM smf209_attachments AS a LEFT JOIN smf209_messages AS m ON a.id_msg = m.id_msg LEFT JOIN smf209_topics AS t ON m.id_topic = t.id_topic WHERE a.id_attach = 93\G;
*************************** 1. row ***************************
      id_attach: 93
       id_thumb: 0
         id_msg: 745
      id_member: 0
      id_folder: 1
attachment_type: 0
       filename: Meth-Treatment-Options.pdf
      file_hash: 114438ca6fe72ae03d346705ee9788737c98c726
        fileext: pdf
           size: 493942
      downloads: 1373
          width: 0
         height: 0
y
      mime_type:
       approved: 1`
proper_rotation: 0
         id_msg: 745
        subject: Meth-Treatment-Options
       id_topic: 658
   id_first_msg: 745
    id_last_msg: 748
1 row in set (0.00 sec)




ISE PDF first hit was 93:

[pdf]https://forum.drugs-and-users.org/index.php?action=dlattach;topic=658;attach=93[/pdf]
Post Merged: Yesterday at 10:49:49 AM

The ISE Project — Checkpoint

Component table

File / Component	Path	Status	Use?
Search.php (SMF core)	Sources/Search.php	case 'ISE' and case 'ISEpdf' in the $_POST['custom'] switch both confirmed correct — routing, tracing, and syslog logging wired for both	✅ use as-is
index_template.php	(SMF theme)	Dropdown options value="ISE" / value="ISEpdf" confirmed correct and matching Search.php's case values. This file caused a full site outage once — edit with care, verify with a lint/syntax pass before saving	✅ use, edit carefully
qf_search_bridge.php	/The_ISE_Project/	Original post-search bridge (shell_exec → search.py → serves ISE_results.html). Confirmed working, untouched by this session's PDF work	✅ use as-is
qf_PDFsearch_bridge.php	/The_ISE_Project/	New, separate PDF bridge (kept isolated from qf_search_bridge.php by design). Shell command fixed this session — see verified facts	✅ use, but confirm current on-disk command string matches the fixed version below (2>&1 was removed then Andrew said he'd put it back — unconfirmed whether that's done)
PDFsearch.py	/The_ISE_Project/	Core PDF search script. Confirmed working standalone and via the bridge. Only real flags: positional query, --limit, --html FILE. No --json, no --source (yet)	✅ use; do not assume --json/--source exist
search.py	/The_ISE_Project/	Original post-index search engine (Tokeniser, QueryParser, IndexBuilder, IndexLookup, RankingEngine, etc.)	✅ use — separate pipeline from PDF search, no shared dependency
ise_trace.py	/The_ISE_Project/, logs to /var/www/html/ISE_data/ise_trace.log	Confirmed working end-to-end for both src=ssh and src=ui, post-search path only. PDFsearch.py does not call into this tracer yet	⚠️ works, but PDF path isn't traced through it — only via inline file_put_contents calls in Search.php's ISEpdf case
PDFIndexer.py	/The_ISE_Project/	Full run completed: 661 PDFs found, 483 indexed, 19,353 pages, 391,062 unique words. Runs ~weekly	✅ use, no known issues
word_index_pdf.json / pdf_store.json	ISE_Data/ (relative path inside PDFsearch.py)	~60MB combined, built by PDFIndexer.py	✅ use — note the relative path risk below

Verified facts — don't re-check

• Search.php has no syntax errors (php -l clean) — the outage that prompted this session was caused by index_template.php, not Search.php.
• Dropdown values (ISE, ISEpdf) and Search.php case labels match exactly — routing is not the bug source going forward.
• PDFsearch.py's argparse defines only: positional query, --limit (int, default 20), --html (takes a file path, via metavar="FILE"). No --json flag. No --source flag. Any bridge code assuming these exist will silently fail (argparse rejects unknown flags → empty stdout → bridge's fallback reads it as "no results," masking the real error).
• Correct working invocation, confirmed live:

    python3 /var/www/html/The_ISE_Project/PDFsearch.py '<query>' --html /var/www/html/The_ISE_Project/ISEpdf_results.html  

• PDF search is confirmed working end-to-end through the full chain: dropdown → Search.php (ISEpdf case) → qf_PDFsearch_bridge.php → PDFsearch.py → styled HTML results page, verified via the actual UI.
• qf_PDFsearch_bridge.php's shell_exec() does not capture stderr unless 2>&1 is appended — without it, any future PDFsearch.py error goes invisible again (this was the root cause of the "always no results" bug this session).
• WORD_INDEX_PATH / PDF_STORE_PATH in PDFsearch.py are relative paths ("ISE_Data/...") — this is the same class of bug already fixed once before in IndexLookup.py (relative paths break under shell_exec() because cwd differs from direct invocation). Not yet confirmed whether this is currently biting PDFsearch.py or just gotten lucky on cwd — flagged, not fixed.

Open items (decisions, not blockers)

• Whether to add a real --source ui|ssh flag to PDFsearch.py's argparse, and whether to route its logging through ise_trace.py for consistency with the rest of ISE (currently PDF path logs only via inline calls in Search.php, not through the shared tracer).
• Whether to add --json as a real feature to PDFsearch.py (would make the bridge's existing but currently-dead JSON fallback branch actually functional), or strip that dead fallback code out of qf_PDFsearch_bridge.php entirely.
• Whether 2>&1 has actually been re-added to qf_PDFsearch_bridge.php's shell_exec call yet (Andrew said he removed it, then said he'd put it back — last known state unconfirmed).
• Whether PDFsearch.py's relative index paths need the same __file__-relative fix already applied to IndexLookup.py.

Next actual action

Confirm 2>&1 is back in qf_PDFsearch_bridge.php's shell_exec command, and paste the current full contents of that file so its state is verified rather than assumed.