dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse


Our Discord Notification Server invitation link is https://discord.gg/jB2qmRrxyD
« previous next »
Pages: 1   Go Down

Author Topic: ISE - Intelligent Search Engine for Posts and Their Attachments  (Read 292 times)

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 596
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Yesterday at 08:44:42 PM
  • Supplied Install Member
ISE - Intelligent Search Engine for Posts and Their Attachments
« on: August 11, 2026, 11:01:38 PM »
📎 Click on me to view them list of attachments
ListAttBBC  |  v6.3  |  2026-08-17  |  Andrew.human & Claude.ai
#FileSizeDownloadsInfoDL
00Claude on PDF search not finding _listatt_ occurrences  and ISE 5.5 and PDF Infexing.pdf4.2 MB12ℹ️⬇️
01ISE_Reset_Flags_Bootstrap_v11.0.zip2.8 KB0ℹ️⬇️
02ISE_v6.0.zip6.7 MB0ℹ️⬇️
03ISEmisc_Checkpoint_for_ISE_v6.0.md1.1 KB0ℹ️⬇️
04Markdown Rendering Issues for code blocks for Andrew and ISEmisc TEXT SEARCH v6,10 final phase.pdf40.5 MB42ℹ️⬇️
05pdfjs-link.zip1.6 KB0ℹ️⬇️
06ISE_v5.5.zip4.3 MB0ℹ️⬇️
07ISE_v7.0.zip6.7 MB0ℹ️⬇️
08More ISE functionality - stopping at ISE v7.1.pdf2 MB8ℹ️⬇️
09ISE_v7.1.zip6.7 MB4ℹ️⬇️
10the rest of the ISE Functionality.pdf1.3 MB9ℹ️⬇️
11ISE_v7.2.zip7 MB2ℹ️⬇️
12ISE_v7.3.zip6.7 MB5ℹ️⬇️
13ISE_v7_4_1.zip6.9 MB2ℹ️⬇️
14checkpoint_v7_4_2.md10 KB0ℹ️⬇️
15ISE_v7.4.2.zip6.8 MB1ℹ️⬇️
16pdfjs.zip5.8 MB0ℹ️⬇️
17ISE_v8.0.zip6.9 MB0ℹ️⬇️
18checkpoint_v8.0.md7.3 KB0ℹ️⬇️
19ISEmega (ISE v8.0) - all 3 ISE methods searched in one go.pdf3.8 MB5ℹ️⬇️
20This is conversation ISE v8. 1 - Software Design Flaws Identified and Changes Pending.pdf2 MB5ℹ️⬇️
21ISE_v8.1.zip7.1 MB2ℹ️⬇️
22checkpoint_closing_v8.1.md5.5 KB0ℹ️⬇️
23changelog_v8.1.md9 KB0ℹ️⬇️
24ISE_v8.2.1.zip7.1 MB6ℹ️⬇️
25checkpoint _closing_v8.2-v8.2.1.md10.7 KB0ℹ️⬇️
26changelog_incorporating_v8.2.2).md9.9 KB0ℹ️⬇️
27ISE_v8.2.2.zip6.8 MB0ℹ️⬇️
28ISE v8.2 testing deployment plan - ONGOING.pdf1.9 MB7ℹ️⬇️
29ISE_v8.2.3.zip7.5 MB0ℹ️⬇️
ISE - Intelligent Search Engine for Posts and any/all Attachments!

NB: There is a "delete duplicate PDFs" already in progress:

https://forum.drugs-and-users.org/The_ISE_Project/pdf_dupe_review.php[/b]
Current Live version is ISE_v8.2.3, then next Test (smf20) version is currently at ISE_v8.2.3 for SMF 2.0.x only (tested on 2.0.19) -- Download using the tool below:


ISE - Intelligent Search Engine for Posts and Attachments

WHAT THIS IS

Adds a custom search engine option called "ISE" (Intelligent Search
Engine) to your forum's search-engine drop-down box, alongside the
existing Default/Google/Yandex/Nigma options.

ISE is a private, self-hosted search tool: a pure Python search engine (QueryParser -> Tokeniser -> ANDMatcher -> SortEngine -> ResultFormatter) that reads directly from your forum's own posts and returns matching results as clickable links, sorted newest first.

This mod also adds search-query logging (via syslog) for both the
native SMF search box and the ISE dropdown option, so every real search your users type gets recorded for later review.

REQUISITE MODS

1. Board-color-and-icons (Version 1.1)
2. Automatic Package Version Emulation (Version 1.6)
3. CustomSearch (Versi
on 1.0) -- Install via Emulating 2.0

Install the above, in order, before installing this package.

FEATURES

- WORD matching:   any_search_term
- PHRASE matching: "an exact quoted phrase" (whitespace-exact, no normalization -- a double space inside quotes is matched literally)
- Mixed queries:   word1 word2 "an exact phrase" -- all terms must match (AND logic) for a post to qualify
- Results sorted newest-first
- Every search (native SMF box, ISE dropdown, and the site's Google
  search widget) is logged via syslog: username/Guest, IP address,
  timestamp, and the raw query string

WHAT GETS INSTALLED

- Sources/QueryFrontend.php -- syslog logging function

- Sources/Search.php (patched)  -- adds the "ISE" case to the search-engine dropdown handler

- Sources/ManageSettings.php (patched) -- adds an admin toggle, "Use ISE search"

- Themes/default/index.template.php (patched) -- loads
  widget_write_log.js and adds the "ISE" option to the dropdown

- Themes/default/scripts/widget_write_log.js -- logs searches made
  through the site's separate Google-search widget

- Themes/default/languages/Modifications.english.php (patched)
-- adds the "ISE" and "Use ISE search" language strings
  (both languages/ and language/ directory spellings are patched,
  since this varies by SMF install)

- $boarddir/The_ISE_Project/  -- the Python search engine itself

  (QueryParser.py, Tokeniser.py, IndexBuilder.py, IndexLookup.py,
  ANDMatcher.py, SortEngine.py, ResultFormatter.py, search.py,
  qf_search_bridge.php, daily_queries.py, query_log_reader.py)

 -- protected from direct web access via its own .htaccess

- $boarddir/ISE_Data/  -- holds the generated search index
  (word_index.json, post_store.json) -- created with 0700
  permissions and its own .htaccess, since this data is a full,
  searchable copy of every post on your forum

- The full Mozilla PDF Javascript libraries at $boarddier/pdfjs

REQUIREMENTS -- READ BEFORE INSTALLING

This mod's PHP/dropdown/logging pieces install automatically through
the Package Manager. The Python search engine itself does NOT run
automatically -- it needs a few things set up on the server first,
by hand, once:

 - Python 3.6+ available on the server, with the PyMySQL package
   installed for whichever interpreter you'll use to run these
   scripts: python3 -m pip install pymysql

AFTER INSTALLING

The Python search engine itself does NOT run by default so folow this carefully:

1. cd ..whatever-your-webroot-is../The_ISE_Project

   python3 IndexBuilder.py

2. cd ..whatever-your-webroot-is../The_ISE_Project
 
   python3 PDFIndexer.py --monitor

3. Go to Admin > Configuration > Modificication Settings > CustomSearch > ISE and ISEpdf uncheck both and Save then recheck both and Save again.

4. Go to the top right of the page and now the dropdown will have ISE and ISEpdf so now you can use it but wait until the indexing has finished.

5. Test a search using the ISE option from the dropdown to confirm everything is wired up correctly before relying on it.

6. crontab -u apache -e

TBA

NOTES

- Search queries are logged via syslog (facility LOCAL0, tag smf-queries) -- typically landing in /var/log/messages depending on your syslog configuration. Two small scripts are included to help review this log:

- The_ISE_Project/daily_queries.py -- lists today's searches
- The_ISE_Project/query_log_reader.py  -- lists the full history

PRIVACY NOTE

Once enabled, every real user's search terms are logged, including guests (identified by IP address rather than username). Given the sensitive nature of a harm-reduction forum's likely search content, review who has access to your server's syslog output and how long you want to retain this history before enabling this for your full userbase.
« Last Edit: August 22, 2026, 04:03:11 PM by Chip »
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 596
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Yesterday at 08:44:42 PM
  • Supplied Install Member
(No subject)
« Reply #1 on: August 19, 2026, 06:49:48 AM »
Topic bumped
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once

Online Chip

  • Server Admin
  • Hero Member
  • *****
  • Administrator
  • *****
  • Join Date: Dec 2014
  • Location: Australia
  • Posts: 7323
  • Reputation Power: 0
  • Chip has hidden their reputation power
  • Gender: Male
  • Last Login:Today at 04:42:40 AM
  • Deeply Confused Learner
  • Profession: IT Engineer now retired
Re: ISE - Intelligent Search Engine for Posts and Their Attachments
« Reply #2 on: August 22, 2026, 02:05:08 PM »

Installing PyMuPDF on a legacy 32-bit CentOS 6.10 environment presents a significant challenge because pre-built official PyMuPDF wheels are only provided for 64-bit Linux architectures (and modern Python versions). Furthermore, CentOS 6 reached end-of-life years ago, meaning its native repositories are archived and modern Python/pip packages may not be readily available. 

To achieve this, you will need to compile both the underlying MuPDF C library and PyMuPDF from source using a local toolchain. 

Step 1: Prepare the Development Environment

CentOS 6.10's default compiler might be too old for modern source trees, but you can install development tools and ensure you have Python and a C++ compiler:
 
Code: [Select]
sudo yum groupinstall "Development Tools"
sudo yum install wget zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel

Note: Since CentOS 6 repos are archived, if yum fails to resolve, you will need to update /etc/yum.repos.d/ to point to the Vault repositories (vault.centos.org).

Step 2: Install a Compatible Python Version

CentOS 6.10 ships with Python 2.6, which is far too old for modern PyMuPDF. You should install a newer version (such as Python 3.6 or 3.7 via Software Collections - SCL, or compile Python manually).

To use Software Collections for a newer Python:

DID NOT HAVE TO DO THIS AS python 3.6.4 was already installed

Code: [Select]
sudo yum install centos-release-scl
sudo yum install python36
scl enable python36 bash
Step 3: Compile and Install MuPDF from Source

PyMuPDF requires the core MuPDF C library. Because you are on a 32-bit system, you must build MuPDF locally: 

Download a stable source archive compatible with older systems (e.g., an older stable release like MuPDF 1.14.0 or similar if newer versions throw compilation errors due to modern C requirements): 

Code: [Select]
wget https://mupdf.com/downloads/archive/mupdf-1.14.0-source.tar.gz
tar -zxvf mupdf-1.14.0-source.tar.gz
cd mupdf-1.14.0-source

Set position-independent code flags required for shared libraries on 32-bit architectures: 

Code: [Select]
export CFLAGS="-fPIC -std=gnu99"
Compile and install MuPDF to /usr/local: 

Code: [Select]
make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local
sudo make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local install
Step 4: Install PyMuPDF from Source

With MuPDF built and installed on your system, you can compile PyMuPDF: 

Code: [Select]
python3 -m pip install --upgrade pip
Install PyMuPDF by forcing a source build, pointing it to your local headers: 

Code: [Select]
pip install pymupdf --no-binary=pymupdf

[!IMPORTANT]
Given the severe security risks, lack of package support, and compilation blockers associated with running 32-bit CentOS 6.10 in modern environments, strongly consider migrating your workload to a containerized modern Linux distribution (such as a 64-bit Docker container running Ubuntu or CentOS Stream) where pre-built wheels install instantly via pip install pymupdf.

Shared conversation:

https://share.gemini.google/A3ZQwC9smv50

friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
Our Discord Server invitation link is https://discord.gg/jB2qmRrxyD

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 596
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Yesterday at 08:44:42 PM
  • Supplied Install Member
Re: ISE v9.0 - Intelligent Search Engine for Posts+ Attmnts., Includes Media !
« Reply #3 on: Yesterday at 11:23:01 AM »
📎 Click on me to view the list of attachments
ListAttBBC  |  v6.3  |  2026-08-17  |  Andrew.human & Claude.ai
#FileSizeDownloadsInfoDL
00checkpoint.md2.9 KB0ℹ️⬇️
01ISEmedia - ISE_v9.0 - AI-powered video and image search for ISE - pdf pages 1-22.pdf883.6 KB0ℹ️⬇️

Introducing AI tek in ISE v9.0 (ISEmedia) with pytorch etc.

https://claude.ai/share/25242c4b-868b-4c86-a254-2ac502cfd384


Chatty:
Code: [Select]
😂 Yep. This is the bit where ISE stops being a collection of scripts and starts becoming infrastructure.

And given that checkpoint, the VMware guest is exactly the right place to do the dangerous experiment. The Live box is CentOS 6.10 / glibc 2.12 / 2 GB RAM, and the media stack is already pushing into territory where modern ML wheels may simply refuse to cooperate.

The particularly important thing is: don't upgrade glibc on Live just to make the modern Python/ML ecosystem happy. A glibc replacement on CentOS 6 is not an ordinary package upgrade; it can turn the entire userspace into a very expensive brick.

I'd treat the VMware guest as the sacrificial glibc laboratory:

1. Clone/snapshot the known-good CentOS 6.10 guest.


2. Record the baseline:

ldd --version
rpm -q glibc
python3 --version
uname -a


3. Test the exact Torch installation first.


4. Then experiment with a newer glibc without touching the system glibc initially — ideally installed in an isolated prefix and tested with an explicit loader/library path.


5. Only if that works do we investigate whether the ISEmedia dependency chain can actually use it.


6. And meanwhile, ISEmedia itself stays decoupled: CLIP daily, Whisper weekly, checkpointed/locked, and absolutely no OOM-killer shenanigans on Live. That's already the right architecture.



In fact, I'd go one step further: the guest should become your compatibility proving ground for ISEmedia, not merely a glibc test box.

Because if we can get:

CentOS 6.10 + Python 3.6.3 + isolated newer glibc + Torch + OpenCLIP + Whisper + ffmpeg

working there, we've learned something extremely valuable without risking the actual ISE installation.

And yes — this is precisely the sort of experiment for which VMware snapshots were invented. 😈

me: feeling proud ?

Hell yes. 😎

You've gone from “custom SMF search script” to an actual search infrastructure with:

ISE → posts

ISEpdf → PDFs

ISEmisc → other attachments

ISEmedia → images/video

ISEmega → the thing tying the whole bastard together


And now you're deliberately building a sacrificial CentOS 6 VMware clone before putting the radioactive glibc experiment anywhere near production.

That is, frankly, the correct way to be a dangerous old systems programmer. 😂

The bit I particularly like is that ISEmedia isn't just “let's throw AI at it.” You've got the separation of cheap daily CLIP indexing from expensive weekly Whisper transcription, with checkpointing and locking already designed around the 2 GB Live box.

So yeah.

I'm feeling a little proud of the bastard.
And somewhat concerned about what you're about to do to glibc. 😈

me: it's only on a crash-and-burn VM !


IMAGES:
https://forum.drugs-and-users.org/index.php?action=dlattach;topic=795.0;attach=137        https://forum.drugs-and-users.org/index.php?action=dlattach;topic=795.0;attach=369  https://forum.drugs-and-users.org/index.php?action=dlattach;topic=1214.0;attach=418 https://forum.drugs-and-users.org/index.php?action=dlattach;topic=1214.0;attach=443 https://forum.drugs-and-users.org/index.php?action=dlattach;topic=1214.0;attach=586
VIDEO: 
https://forum.drugs-and-users.org/index.php?action=dlattach;topic=7353.0;attach=5805
« Last Edit: Yesterday at 09:30:00 PM by Chip »
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once
Pages: 1   Go Up
« previous next »
Tags:
 

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse




TERMS AND CONDITIONS

In no event will d&u or any person involved in creating, producing, or distributing site information be liable for any direct, indirect, incidental, punitive, special or consequential damages arising out of the use of or inability to use d&u. You agree to indemnify and hold harmless d&u, its domain founders, sponsors, maintainers, server administrators, volunteers and contributors from and against all liability, claims, damages, costs and expenses, including legal fees, that arise directly or indirectly from the use of any part of the d&u site.

TO USE THIS WEBSITE YOU MUST AGREE TO THE TERMS AND CONDITIONS ABOVE

Founded December 2014
SimplePortal 2.3.6 © 2008-2014, SimplePortal