dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse


Our Discord Notification Server invitation link is https://discord.gg/jB2qmRrxyD
« previous next »
Pages: 1   Go Down

Author Topic: ISE - Intelligent Search Engine for Posts and Their Attachments  (Read 253 times)

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 595
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Yesterday at 04:32:08 AM
  • Supplied Install Member
ISE - Intelligent Search Engine for Posts and Their Attachments
« on: August 11, 2026, 11:01:38 PM »
📎 Click on me to view them list of attachments
ListAttBBC  |  v6.3  |  2026-08-17  |  Andrew.human & Claude.ai
#FileSizeDownloadsInfoDL
00Claude on PDF search not finding _listatt_ occurrences  and ISE 5.5 and PDF Infexing.pdf4.2 MB10ℹ️⬇️
01ISE_Reset_Flags_Bootstrap_v11.0.zip2.8 KB0ℹ️⬇️
02ISE_v6.0.zip6.7 MB0ℹ️⬇️
03ISEmisc_Checkpoint_for_ISE_v6.0.md1.1 KB0ℹ️⬇️
04Markdown Rendering Issues for code blocks for Andrew and ISEmisc TEXT SEARCH v6,10 final phase.pdf40.5 MB38ℹ️⬇️
05pdfjs-link.zip1.6 KB0ℹ️⬇️
06ISE_v5.5.zip4.3 MB0ℹ️⬇️
07ISE_v7.0.zip6.7 MB0ℹ️⬇️
08More ISE functionality - stopping at ISE v7.1.pdf2 MB7ℹ️⬇️
09ISE_v7.1.zip6.7 MB4ℹ️⬇️
10the rest of the ISE Functionality.pdf1.3 MB7ℹ️⬇️
11ISE_v7.2.zip7 MB2ℹ️⬇️
12ISE_v7.3.zip6.7 MB5ℹ️⬇️
13ISE_v7_4_1.zip6.9 MB2ℹ️⬇️
14checkpoint_v7_4_2.md10 KB0ℹ️⬇️
15ISE_v7.4.2.zip6.8 MB1ℹ️⬇️
16pdfjs.zip5.8 MB0ℹ️⬇️
17ISE_v8.0.zip6.9 MB0ℹ️⬇️
18checkpoint_v8.0.md7.3 KB0ℹ️⬇️
19ISEmega (ISE v8.0) - all 3 ISE methods searched in one go.pdf3.8 MB3ℹ️⬇️
20This is conversation ISE v8. 1 - Software Design Flaws Identified and Changes Pending.pdf2 MB4ℹ️⬇️
21ISE_v8.1.zip7.1 MB2ℹ️⬇️
22checkpoint_closing_v8.1.md5.5 KB0ℹ️⬇️
23changelog_v8.1.md9 KB0ℹ️⬇️
24ISE_v8.2.1.zip7.1 MB6ℹ️⬇️
25checkpoint _closing_v8.2-v8.2.1.md10.7 KB0ℹ️⬇️
26changelog_incorporating_v8.2.2).md9.9 KB0ℹ️⬇️
27ISE_v8.2.2.zip6.8 MB0ℹ️⬇️
28ISE v8.2 testing deployment plan - ONGOING.pdf1.9 MB3ℹ️⬇️
29ISE_v8.2.3.zip7.5 MB0ℹ️⬇️
ISE - Intelligent Search Engine for Posts and any/all Attachments!

NB: There is a "delete duplicate PDFs" already in progress:

https://forum.drugs-and-users.org/The_ISE_Project/pdf_dupe_review.php[/b]
Current Live version is ISE_v8.2.3, then next Test (smf20) version is currently at ISE_v8.2.3 for SMF 2.0.x only (tested on 2.0.19) -- Download using the tool below:


ISE - Intelligent Search Engine for Posts and Attachments

WHAT THIS IS

Adds a custom search engine option called "ISE" (Intelligent Search
Engine) to your forum's search-engine drop-down box, alongside the
existing Default/Google/Yandex/Nigma options.

ISE is a private, self-hosted search tool: a pure Python search engine (QueryParser -> Tokeniser -> ANDMatcher -> SortEngine -> ResultFormatter) that reads directly from your forum's own posts and returns matching results as clickable links, sorted newest first.

This mod also adds search-query logging (via syslog) for both the
native SMF search box and the ISE dropdown option, so every real search your users type gets recorded for later review.

REQUISITE MODS

1. Board-color-and-icons (Version 1.1)
2. Automatic Package Version Emulation (Version 1.6)
3. CustomSearch (Versi
on 1.0) -- Install via Emulating 2.0

Install the above, in order, before installing this package.

FEATURES

- WORD matching:   any_search_term
- PHRASE matching: "an exact quoted phrase" (whitespace-exact, no normalization -- a double space inside quotes is matched literally)
- Mixed queries:   word1 word2 "an exact phrase" -- all terms must match (AND logic) for a post to qualify
- Results sorted newest-first
- Every search (native SMF box, ISE dropdown, and the site's Google
  search widget) is logged via syslog: username/Guest, IP address,
  timestamp, and the raw query string

WHAT GETS INSTALLED

- Sources/QueryFrontend.php -- syslog logging function

- Sources/Search.php (patched)  -- adds the "ISE" case to the search-engine dropdown handler

- Sources/ManageSettings.php (patched) -- adds an admin toggle, "Use ISE search"

- Themes/default/index.template.php (patched) -- loads
  widget_write_log.js and adds the "ISE" option to the dropdown

- Themes/default/scripts/widget_write_log.js -- logs searches made
  through the site's separate Google-search widget

- Themes/default/languages/Modifications.english.php (patched)
-- adds the "ISE" and "Use ISE search" language strings
  (both languages/ and language/ directory spellings are patched,
  since this varies by SMF install)

- $boarddir/The_ISE_Project/  -- the Python search engine itself

  (QueryParser.py, Tokeniser.py, IndexBuilder.py, IndexLookup.py,
  ANDMatcher.py, SortEngine.py, ResultFormatter.py, search.py,
  qf_search_bridge.php, daily_queries.py, query_log_reader.py)

 -- protected from direct web access via its own .htaccess

- $boarddir/ISE_Data/  -- holds the generated search index
  (word_index.json, post_store.json) -- created with 0700
  permissions and its own .htaccess, since this data is a full,
  searchable copy of every post on your forum

- The full Mozilla PDF Javascript libraries at $boarddier/pdfjs

REQUIREMENTS -- READ BEFORE INSTALLING

This mod's PHP/dropdown/logging pieces install automatically through
the Package Manager. The Python search engine itself does NOT run
automatically -- it needs a few things set up on the server first,
by hand, once:

 - Python 3.6+ available on the server, with the PyMySQL package
   installed for whichever interpreter you'll use to run these
   scripts: python3 -m pip install pymysql

AFTER INSTALLING

The Python search engine itself does NOT run by default so folow this carefully:

1. cd ..whatever-your-webroot-is../The_ISE_Project

   python3 IndexBuilder.py

2. cd ..whatever-your-webroot-is../The_ISE_Project
 
   python3 PDFIndexer.py --monitor

3. Go to Admin > Configuration > Modificication Settings > CustomSearch > ISE and ISEpdf uncheck both and Save then recheck both and Save again.

4. Go to the top right of the page and now the dropdown will have ISE and ISEpdf so now you can use it but wait until the indexing has finished.

5. Test a search using the ISE option from the dropdown to confirm everything is wired up correctly before relying on it.

6. crontab -u apache -e

TBA

NOTES

- Search queries are logged via syslog (facility LOCAL0, tag smf-queries) -- typically landing in /var/log/messages depending on your syslog configuration. Two small scripts are included to help review this log:

- The_ISE_Project/daily_queries.py -- lists today's searches
- The_ISE_Project/query_log_reader.py  -- lists the full history

PRIVACY NOTE

Once enabled, every real user's search terms are logged, including guests (identified by IP address rather than username). Given the sensitive nature of a harm-reduction forum's likely search content, review who has access to your server's syslog output and how long you want to retain this history before enabling this for your full userbase.
« Last Edit: Yesterday at 04:03:11 PM by Chip »
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 595
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Yesterday at 04:32:08 AM
  • Supplied Install Member
(No subject)
« Reply #1 on: August 19, 2026, 06:49:48 AM »
Topic bumped
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once

Offline Chip

  • Server Admin
  • Hero Member
  • *****
  • Administrator
  • *****
  • Join Date: Dec 2014
  • Location: Australia
  • Posts: 7320
  • Reputation Power: 0
  • Chip has hidden their reputation power
  • Gender: Male
  • Last Login:Yesterday at 04:03:11 PM
  • Deeply Confused Learner
  • Profession: IT Engineer now retired
Re: ISE - Intelligent Search Engine for Posts and Their Attachments
« Reply #2 on: Yesterday at 02:05:08 PM »

Installing PyMuPDF on a legacy 32-bit CentOS 6.10 environment presents a significant challenge because pre-built official PyMuPDF wheels are only provided for 64-bit Linux architectures (and modern Python versions). Furthermore, CentOS 6 reached end-of-life years ago, meaning its native repositories are archived and modern Python/pip packages may not be readily available. 

To achieve this, you will need to compile both the underlying MuPDF C library and PyMuPDF from source using a local toolchain. 

Step 1: Prepare the Development Environment

CentOS 6.10's default compiler might be too old for modern source trees, but you can install development tools and ensure you have Python and a C++ compiler:
 
Code: [Select]
sudo yum groupinstall "Development Tools"
sudo yum install wget zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel

Note: Since CentOS 6 repos are archived, if yum fails to resolve, you will need to update /etc/yum.repos.d/ to point to the Vault repositories (vault.centos.org).

Step 2: Install a Compatible Python Version

CentOS 6.10 ships with Python 2.6, which is far too old for modern PyMuPDF. You should install a newer version (such as Python 3.6 or 3.7 via Software Collections - SCL, or compile Python manually).

To use Software Collections for a newer Python:

DID NOT HAVE TO DO THIS AS python 3.6.4 was already installed

Code: [Select]
sudo yum install centos-release-scl
sudo yum install python36
scl enable python36 bash
Step 3: Compile and Install MuPDF from Source

PyMuPDF requires the core MuPDF C library. Because you are on a 32-bit system, you must build MuPDF locally: 

Download a stable source archive compatible with older systems (e.g., an older stable release like MuPDF 1.14.0 or similar if newer versions throw compilation errors due to modern C requirements): 

Code: [Select]
wget https://mupdf.com/downloads/archive/mupdf-1.14.0-source.tar.gz
tar -zxvf mupdf-1.14.0-source.tar.gz
cd mupdf-1.14.0-source

Set position-independent code flags required for shared libraries on 32-bit architectures: 

Code: [Select]
export CFLAGS="-fPIC -std=gnu99"
Compile and install MuPDF to /usr/local: 

Code: [Select]
make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local
sudo make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local install
Step 4: Install PyMuPDF from Source

With MuPDF built and installed on your system, you can compile PyMuPDF: 

Code: [Select]
python3 -m pip install --upgrade pip
Install PyMuPDF by forcing a source build, pointing it to your local headers: 

Code: [Select]
pip install pymupdf --no-binary=pymupdf

[!IMPORTANT]
Given the severe security risks, lack of package support, and compilation blockers associated with running 32-bit CentOS 6.10 in modern environments, strongly consider migrating your workload to a containerized modern Linux distribution (such as a 64-bit Docker container running Ubuntu or CentOS Stream) where pre-built wheels install instantly via pip install pymupdf.

Shared conversation:

https://share.gemini.google/A3ZQwC9smv50

friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
Our Discord Server invitation link is https://discord.gg/jB2qmRrxyD
Pages: 1   Go Up
« previous next »
Tags:
 

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse




TERMS AND CONDITIONS

In no event will d&u or any person involved in creating, producing, or distributing site information be liable for any direct, indirect, incidental, punitive, special or consequential damages arising out of the use of or inability to use d&u. You agree to indemnify and hold harmless d&u, its domain founders, sponsors, maintainers, server administrators, volunteers and contributors from and against all liability, claims, damages, costs and expenses, including legal fees, that arise directly or indirectly from the use of any part of the d&u site.

TO USE THIS WEBSITE YOU MUST AGREE TO THE TERMS AND CONDITIONS ABOVE

Founded December 2014
SimplePortal 2.3.6 © 2008-2014, SimplePortal