dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse


Our Discord Notification Server invitation link is https://discord.gg/jB2qmRrxyD
« previous next »
Pages: 1   Go Down

Author Topic: ISE - Intelligent Search Engine for Posts and Their Attachments  (Read 212 times)

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 595
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Today at 12:52:17 AM
  • Supplied Install Member
ISE - Intelligent Search Engine for Posts and Their Attachments
« on: August 11, 2026, 11:01:38 PM »
📎 Click on me to view them list of attachments
ListAttBBC  |  v6.3  |  2026-08-17  |  Andrew.human & Claude.ai
#FileSizeDownloadsInfoDL
00Claude on PDF search not finding _listatt_ occurrences  and ISE 5.5 and PDF Infexing.pdf4.2 MB8ℹ️⬇️
01ISE_Reset_Flags_Bootstrap_v11.0.zip2.8 KB0ℹ️⬇️
02ISE_v6.0.zip6.7 MB0ℹ️⬇️
03ISEmisc_Checkpoint_for_ISE_v6.0.md1.1 KB0ℹ️⬇️
04Markdown Rendering Issues for code blocks for Andrew and ISEmisc TEXT SEARCH v6,10 final phase.pdf40.5 MB32ℹ️⬇️
05pdfjs-link.zip1.6 KB0ℹ️⬇️
06ISE_v5.5.zip4.3 MB0ℹ️⬇️
07ISE_v7.0.zip6.7 MB0ℹ️⬇️
08More ISE functionality - stopping at ISE v7.1.pdf2 MB4ℹ️⬇️
09ISE_v7.1.zip6.7 MB4ℹ️⬇️
10the rest of the ISE Functionality.pdf1.3 MB4ℹ️⬇️
11ISE_v7.2.zip7 MB2ℹ️⬇️
12ISE_v7.3.zip6.7 MB5ℹ️⬇️
13ISE_v7_4_1.zip6.9 MB2ℹ️⬇️
14checkpoint_v7_4_2.md10 KB0ℹ️⬇️
15ISE_v7.4.2.zip6.8 MB1ℹ️⬇️
16pdfjs.zip5.8 MB0ℹ️⬇️
17ISE_v8.0.zip6.9 MB0ℹ️⬇️
18checkpoint_v8.0.md7.3 KB0ℹ️⬇️
19ISEmega (ISE v8.0) - all 3 ISE methods searched in one go.pdf3.8 MB1ℹ️⬇️
20This is conversation ISE v8. 1 - Software Design Flaws Identified and Changes Pending.pdf2 MB1ℹ️⬇️
21ISE_v8.1.zip7.1 MB2ℹ️⬇️
22checkpoint_closing_v8.1.md5.5 KB0ℹ️⬇️
23changelog_v8.1.md9 KB0ℹ️⬇️
24ISE_v8.2.1.zip7.1 MB0ℹ️⬇️
ISE - Intelligent Search Engine for Posts and any/all Attachments!

NB: There is a "delete duplicate PDFs" already in progress:

https://forum.drugs-and-users.org/The_ISE_Project/pdf_dupe_review.php[/b]
Current Live version is ISE_v7.4.2, then next Test (smf20) version is ISE_v8.2 (as v8.1 is not stable) for SMF 2.0.x only (tested on 2.0.19) -- Download using the tool below:


ISE - Intelligent Search Engine for Posts and Attachments

WHAT THIS IS

Adds a custom search engine option called "ISE" (Intelligent Search
Engine) to your forum's search-engine drop-down box, alongside the
existing Default/Google/Yandex/Nigma options.

ISE is a private, self-hosted search tool: a pure Python search engine (QueryParser -> Tokeniser -> ANDMatcher -> SortEngine -> ResultFormatter) that reads directly from your forum's own posts and returns matching results as clickable links, sorted newest first.

This mod also adds search-query logging (via syslog) for both the
native SMF search box and the ISE dropdown option, so every real search your users type gets recorded for later review.

REQUISITE MODS

1. Board-color-and-icons (Version 1.1)
2. Automatic Package Version Emulation (Version 1.6)
3. CustomSearch (Versi
on 1.0) -- Install via Emulating 2.0

Install the above, in order, before installing this package.

FEATURES

- WORD matching:   any_search_term
- PHRASE matching: "an exact quoted phrase" (whitespace-exact, no normalization -- a double space inside quotes is matched literally)
- Mixed queries:   word1 word2 "an exact phrase" -- all terms must match (AND logic) for a post to qualify
- Results sorted newest-first
- Every search (native SMF box, ISE dropdown, and the site's Google
  search widget) is logged via syslog: username/Guest, IP address,
  timestamp, and the raw query string

WHAT GETS INSTALLED

- Sources/QueryFrontend.php -- syslog logging function

- Sources/Search.php (patched)  -- adds the "ISE" case to the search-engine dropdown handler

- Sources/ManageSettings.php (patched) -- adds an admin toggle, "Use ISE search"

- Themes/default/index.template.php (patched) -- loads
  widget_write_log.js and adds the "ISE" option to the dropdown

- Themes/default/scripts/widget_write_log.js -- logs searches made
  through the site's separate Google-search widget

- Themes/default/languages/Modifications.english.php (patched)
-- adds the "ISE" and "Use ISE search" language strings
  (both languages/ and language/ directory spellings are patched,
  since this varies by SMF install)

- $boarddir/The_ISE_Project/  -- the Python search engine itself

  (QueryParser.py, Tokeniser.py, IndexBuilder.py, IndexLookup.py,
  ANDMatcher.py, SortEngine.py, ResultFormatter.py, search.py,
  qf_search_bridge.php, daily_queries.py, query_log_reader.py)

 -- protected from direct web access via its own .htaccess

- $boarddir/ISE_Data/  -- holds the generated search index
  (word_index.json, post_store.json) -- created with 0700
  permissions and its own .htaccess, since this data is a full,
  searchable copy of every post on your forum

- The full Mozilla PDF Javascript libraries at $boarddier/pdfjs

REQUIREMENTS -- READ BEFORE INSTALLING

This mod's PHP/dropdown/logging pieces install automatically through
the Package Manager. The Python search engine itself does NOT run
automatically -- it needs a few things set up on the server first,
by hand, once:

 - Python 3.6+ available on the server, with the PyMySQL package
   installed for whichever interpreter you'll use to run these
   scripts: python3 -m pip install pymysql

AFTER INSTALLING

The Python search engine itself does NOT run by default so folow this carefully:

1. cd ..whatever-your-webroot-is../The_ISE_Project

   python3 IndexBuilder.py

2. cd ..whatever-your-webroot-is../The_ISE_Project
 
   python3 PDFIndexer.py --monitor

3. Go to Admin > Configuration > Modificication Settings > CustomSearch > ISE and ISEpdf uncheck both and Save then recheck both and Save again.

4. Go to the top right of the page and now the dropdown will have ISE and ISEpdf so now you can use it but wait until the indexing has finished.

5. Test a search using the ISE option from the dropdown to confirm everything is wired up correctly before relying on it.

6. crontab -u apache -e

TBA

NOTES

- Search queries are logged via syslog (facility LOCAL0, tag smf-queries) -- typically landing in /var/log/messages depending on your syslog configuration. Two small scripts are included to help review this log:

- The_ISE_Project/daily_queries.py -- lists today's searches
- The_ISE_Project/query_log_reader.py  -- lists the full history

PRIVACY NOTE

Once enabled, every real user's search terms are logged, including guests (identified by IP address rather than username). Given the sensitive nature of a harm-reduction forum's likely search content, review who has access to your server's syslog output and how long you want to retain this history before enabling this for your full userbase.
« Last Edit: Yesterday at 01:24:09 PM by Chip »
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once

Offline smfadmin (OP)

  • SMF (internal) Site
  • Administrator
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 595
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Today at 12:52:17 AM
  • Supplied Install Member
(No subject)
« Reply #1 on: August 19, 2026, 06:49:48 AM »
Topic bumped
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once
Pages: 1   Go Up
« previous next »
Tags:
 

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse




TERMS AND CONDITIONS

In no event will d&u or any person involved in creating, producing, or distributing site information be liable for any direct, indirect, incidental, punitive, special or consequential damages arising out of the use of or inability to use d&u. You agree to indemnify and hold harmless d&u, its domain founders, sponsors, maintainers, server administrators, volunteers and contributors from and against all liability, claims, damages, costs and expenses, including legal fees, that arise directly or indirectly from the use of any part of the d&u site.

TO USE THIS WEBSITE YOU MUST AGREE TO THE TERMS AND CONDITIONS ABOVE

Founded December 2014
SimplePortal 2.3.6 © 2008-2014, SimplePortal