17 October 2026 (Sydney time)
#do-not-index, case-insensitive. Only the post body is checked, not the subject.id_msg, then the body text). One helper function answers "is this attachment excluded?", and every indexer calls it before processing a file.To build it, send:
PDFIndexer.py, the Misc indexer and MediaIndexer.I'll add it as one shared helper plus a call at the top of each indexer's file loop. The delivery will be a versioned zip with the release card, and it will include the purge pass.
I can zip with password too
Claude: [October 17, 2026, 05:03:23 PM]
A password on the code zip isn't needed. If you do use one, put the password in the chat message, because I can't open an encrypted archive otherwise. The indexer code isn't sensitive. The only thing to watch is that the config or database module may hold credentials, so blank those lines out before sending. I only need to see how the connection is made, not the actual password.
For the private material, a password-protected zip is a better fit than a plain PDF on the forum. Indexers can't read inside an encrypted archive, so it sidesteps indexing by design, and the contents stay protected even if someone fetches the attachment URL directly. Use AES-256 encryption (the legacy ZipCrypto scheme is weak), and use a long passphrase that you keep off the forum.
The #do-not-index rule is still worth building. It covers all the unencrypted attachments you may want hidden later, and the encrypted zip can be the extra layer for this particular material.
zip -e ?
Claude: [October 17, 2026, 05:05:24 PM]
zip -e works, but it uses the old ZipCrypto scheme, which is weak, so it won't do for this material. Use 7-Zip for AES-256:
7z a -t7z -mhe=on -p private.7z file1.pdf file2.pdf-p with nothing after it: It prompts for the password, so it stays out of your shell history.-mhe=on: It encrypts the filenames as well as the contents, so the archive doesn't reveal what's inside..zip: Use 7z a -tzip -mem=AES256 -p private.zip file1.pdf. This hides the contents, but the filenames stay visible.The p7zip package provides 7z on your Linux boxes.
[MediaIndexer.py] [PDFIndexer.py] [MiscIndexer.py] [AudioProcessor.py] [IndexBuilder.py] audio, visual, post, pdf and misc -- all apply
this is a smart addition to ISE !
even if the indexers operate in "single" mode
Claude: [October 17, 2026, 05:16:01 PM]
Single mode is covered. I'm applying the rule in every mode, and the unrequested changes are listed first so you can stop me before I touch anything.
Single mode: MediaIndexer --single, PDFIndexer --single and AudioProcessor --attachid all honour the rule. A marked attachment is not indexed, and any entry it already has is removed. MiscIndexer and IndexBuilder have no single mode; they rebuild fully each run.
Changes you didn't ask for (nothing else will change):
ise_noindex.py holding the rule once. All five indexers call it.--prune is needed.Gap: MediaProcessor.py and MediaSearch.py weren't in the upload. Frame captions and vectors already made for newly marked images or videos will stay in their store until MediaProcessor gets the same purge.
to confirm, this is all predicated on the post containing the string somewhere on lines 1-5
Confirmed, and I fixed it. My first version matched the tag anywhere in the post body. It now only counts if #do-not-index is on lines 1–5. A tag on line 6 or later is ignored. Matching is still case-insensitive, and the subject line is never checked.
SMF stores line breaks as <br />, so the line count treats <br />, <br>, <br/> and plain newlines all as line breaks.
I ran a test harness against all six files. It uses stand-in databases and PDFs, not your live SMF board or real attachments. It all passed, including these cases:
--single mode.Put the tag in the first five lines of the Master post. If it sits lower, the post and its attachments get indexed. Since a lower tag fails open, I'd keep the tag on line 1.
ise_noindex.py has to sit in the same folder as the indexers.audio_store.json to Live afterwards. Otherwise Live keeps the old entries.MediaProcessor.py or MediaSearch.py. Frame captions already generated for a newly marked image or video stay in place until MediaProcessor.py gets the same purge. Send it and I'll add it.| Item | Status |
|---|---|
| @version/@date bumped in all six files | Done |
| History entries | Done |
| Version shown at run start | Existing in the five; ise_noindex.py --version works |
| Versioned zip | ISE_DoNotIndex_v1.1.0.zip |
| xmllint | Not needed, no XML |
| Unrequested changes listed beforehand | Done |
| Legacy zipper | Not needed, these are ISE scripts rather than an SMF package |
oh, ADMIN ONLY
Claude: [October 17, 2026, 05:20:50 PM]
Keep the tag on line 1 even with the post admin-only. The two protect different things:
The tag is what closes that gap, so use both.
Direct attachment URLs aren't covered by either. A non-admin who has the exact link to an attachment may still be able to download it. If that matters, password-protect the zip as well.
or Moderator, to be safe
Claude: [October 17, 2026, 05:21:26 PM]
Admin or Moderator is fine for the post's visibility. One thing needs checking first.
A global moderator can see every board. A board-level moderator can only see the boards they moderate. So if the Master post is in an admin-only board, board moderators won't see it, but global moderators will. If you want moderators locked out too, restrict the post to Admin only. If you want them in, put it in a board they moderate.
The #do-not-index tag doesn't depend on any of this. It applies to the post whoever can view it.
no, only Moderator and above can exclude posts+their att's from indexing !
Claude: [October 17, 2026, 05:23:33 PM]
Done. #do-not-index now only works when the post's author is a Moderator or above. It still has to sit on lines 1–5. A tag in a regular member's post, or a guest's, is ignored and the post is indexed as normal.
"Moderator or above" is my reading of your rule, so check it. An author counts if they are in:
Groups 1 and 2 are SMF's default ids and are constants at the top of ise_noindex.py. If your forum uses different ids, change them there. If you don't want board moderators included, I'll drop that clause.
The check is on the post's author, not on whoever typed the tag. SMF doesn't record who edited a post's text. So if a moderator adds the tag to a member's post, it is ignored, and the moderator has to write the post themselves. Your Master post is fine, since you wrote it.
I tested this against stand-in databases, not your live board. The tests cover admin, global moderator, additional-group moderator, board moderator, regular member, guest and a moderator of a different board, all passing.
| Item | Status |
|---|---|
| @version/@date bumped in all six files | Done |
| History entries | Done |
| Version shown at run start | Existing in the five; ise_noindex.py --version works |
| Versioned zip | ISE_DoNotIndex_v1.2.0.zip |
| xmllint | Not needed, no XML |
| Unrequested changes listed beforehand | Board moderators counting as moderators and groups 1 and 2 as the cutoff, both listed above |
| Legacy zipper | Not needed, these are ISE scripts rather than an SMF package |
[ISE_Python_Scripts_Manual.md] ... and in the manual she goes too !
Claude: [October 17, 2026, 05:24:25 PM]
The manual is updated to Master v6, with a new #do-not-index section. It covers the three conditions (tag in the body, on lines 1–5, author Moderator or above), what each indexer does, and the known gaps. I also added notes to the PDFIndexer, MiscIndexer, MediaIndexer and shared-dependency sections, and a v6 history entry. The file keeps its original name.
Don't deploy the MiscIndexer.py from either zip. The manual shows MiscIndexer at v7.5.0 with --single, but the copy I patched was 7.4.1#3, which doesn't have it. Installing mine would have dropped --single. ISE_DoNotIndex_v1.2.0.zip contains that stale copy, so use ISE_DoNotIndex_v1.2.1.zip, which leaves MiscIndexer.py out. Send me your current 7.5.0 and I'll add the rule, including how --single handles a marked attachment. The manual lists MiscIndexer as pending until then.
The v1.2.1 zip holds the other four indexers plus ise_noindex.py, with no code changes from v1.2.0. Everything else in the zip is as tested earlier.
| Item | Status |
|---|---|
| @version/@date bumped | Done in the four scripts and ise_noindex.py; manual is Master v6 |
| History entries | Done |
| Version shown at run start | Existing in the indexers; ise_noindex.py --version works |
| Versioned zip | ISE_DoNotIndex_v1.2.1.zip |
| xmllint | Not needed, no XML |
| Unrequested changes listed beforehand | Withholding MiscIndexer.py, explained above |
| Legacy zipper | Not needed, these are ISE scripts rather than an SMF package |
I have my own, owner's board and the admins are not the full quid and cant see my stuff !
Claude: [October 17, 2026, 05:25:50 PM]
A board only you can see keeps the other admins out of the board itself, but not out of the indexes. The indexers read the database directly and ignore board permissions, so the tag is still what keeps the post and its attachments out of ISE. I can't confirm whether ISE's search results check board permissions, because I haven't seen the search scripts. If they don't, the tag is the only thing standing between those admins and your material.
One thing could make the tag fail without any warning. The rule honours it only if the post's author is in group 1 (Administrator) or group 2 (Global Moderator), or moderates that board. If the account you post from is the forum's owner account in group 1, you're fine. If it's a separate or custom group, the tag is silently ignored and the post gets indexed. In that case, add the account as a moderator of your board, or add its group id to PRIVILEGED_GROUPS at the top of ise_noindex.py.
Check before you rely on this: after the first run of each indexer, search ISE for a word that only appears in the Master post, as an account that can't see your board. It should return nothing.
yes but my Group 1 does not have full Admin rights -- but yeah, i will test it for sure. Time to login from the NUC as this is a very welcome mod !
Claude: [October 17, 2026, 05:28:27 PM]
Group 1 having reduced rights doesn't matter here. The rule looks only at group membership by id, not at what the group is allowed to do. As long as your account is in group 1, the tag is honoured.
Send over your current MiscIndexer.py (the 7.5.0 one) when you're back, and I'll bring it in line with the others. Until then, the four indexers in ISE_DoNotIndex_v1.2.1.zip cover posts, PDFs, media and audio. Misc attachments (.md, .txt, .json and so on) are still indexed regardless of the tag.
I sent u all of them b4, didn't I?
Claude: [October 17, 2026, 05:29:17 PM]
You did send all five, and I worked from exactly what you sent. The mismatch is between two of your own files.
MiscIndexer.py you uploaded is v7.4.1#3, dated 2026-10-05, and has no --single.--single in v7.5.0 on 2026-10-01. The v7.4.1#3 file is dated after that but carries an older version number, so one of the two is out of step.The uploaded PDFIndexer.py (8.6.1) and MediaIndexer.py match what the manual says, so those two are fine.
I can't tell from here which copy is current. To find out, run this on whichever box holds your working copy:
grep -n '^@version' MiscIndexer.py; grep -c -- '--single' MiscIndexer.py
my latest is 7.4.1#3
Claude: [October 17, 2026, 05:30:30 PM]
Then the Misc build is the right one, and ISE_DoNotIndex_v1.2.2.zip now has all six files. MiscIndexer.py is back in as 7.4.1#4, with no code changes since the v1.2.0 tests.
The manual was wrong, not your script. It describes a MiscIndexer v7.5.0 with --single, and your latest has no such flag. I left that old text in place and added a note in the MiscIndexer section saying --single isn't in the current script. The v6 history entry records the correction. The corrected manual is attached. You may want to correct the 2026-10-01 history entry yourself.
Use v1.2.2 and ignore v1.2.0 and v1.2.1. The code is the same as v1.2.1, and v1.2.2 just adds MiscIndexer.py back.
| Item | Status |
|---|---|
| @version/@date bumped | Done in all six; manual is Master v6 |
| History entries | Done |
| Version shown at run start | Existing in the indexers; ise_noindex.py --version works |
| Versioned zip | ISE_DoNotIndex_v1.2.2.zip |
| xmllint | Not needed, no XML |
| Unrequested changes listed beforehand | None this round |
| Legacy zipper | Not needed, these are ISE scripts rather than an SMF package |
where are u getting 7.5.1 from ?
Claude: [October 17, 2026, 05:30:49 PM]
I haven't used 7.5.1 for any indexer. The MiscIndexer numbers I've used are:
The only "7.5.1" anywhere in the manual or the scripts is about the search scripts, not the indexer. It's in the MegaSearch section: "Same fix as PDFsearch.py v7.5.1 and MiscSearch.py v7.5.1". It may be what you're thinking of.
what abouit the 4 indexers ?
Claude: [October 17, 2026, 05:36:44 PM]
I read your question as asking about the indexers' versions, so here is each one, before and after. There are five indexers in the zip, plus the shared helper. None of them is on 7.5.1.
| Script | You sent | In ISE_DoNotIndex_v1.2.2.zip |
|---|---|---|
IndexBuilder.py |
7.4.4 | 7.4.5 |
PDFIndexer.py |
8.6.1 | 8.7.0 |
MiscIndexer.py |
7.4.1#3 | 7.4.1#4 |
MediaIndexer.py |
ISE v9.9.9 | ISE v9.9.10 |
AudioProcessor.py |
1.2.3 | 1.3.0 |
ise_noindex.py (new) |
n/a | 1.2.0 |
The one real difference between the indexers is that MiscIndexer.py has no --single. That is why it only takes the tag on full runs.
If you meant something else by "the 4 indexers", for example the four I named when I withheld Misc, tell me which and I'll answer that instead.
oh, they are zipped
Claude: [October 17, 2026, 05:37:34 PM]
Yes, all six files are in ISE_DoNotIndex_v1.2.2.zip, flat, with no folder inside. Unzip it into the folder that holds your indexers. ise_noindex.py has to sit beside them.
all 7 files are in - now gonna check the ISE Data to see if any indexers slipped by ...
Claude: [October 17, 2026, 05:42:08 PM]
The zip holds six files, so I take your seventh to be the manual. If you counted something else, check it.
For the check, take the id_attach of one of the Master post's attachments and look for it in each store. The stores are keyed by id, so grep -c '"123456"' against pdf_store.json, misc_store.json, media_store.json and audio_store.json will do. Post text is a different test: a word that appears only in the Master post, searched against post_store.json.
Two things to expect:
IndexBuilder, MiscIndexer and MediaIndexer (plain run) drop a marked post or attachment when they rebuild. PDFIndexer and AudioProcessor purge old entries at the start of their next run.audio_store.json until you push it again.MediaProcessor already produced for a marked image or video.yes !
Claude: [October 17, 2026, 05:43:08 PM]
Good. If any store still holds the Master post after its indexer has run, send me the indexer's output and the store entry. The usual cause is the author's group id, which I covered earlier. I'll also add the same purge to MediaProcessor.py whenever you send it.
i might push them up from clone and wipe them out
Claude: [October 17, 2026, 05:43:37 PM]
Pushing the Clone's stores to Live would remove the old entries there, as long as the Clone's copies are clean first.
MediaProcessor output: pushing the stores won't clear captions or embeddings for a marked image or video that were already generated. Those live in captions_store.json and embeddings_store.json, which only MediaProcessor.py builds. Check whether those are among the files you push.-rw-r--r-- 1 apache apache 30893213 Oct 17 14:00 word_index.json
-rw-r--r-- 1 apache apache 57994111 Oct 17 14:00 post_store.json
-rw-rw-rw- 1 apache apache 259990 Oct 17 14:00 indexbuilder.log
-rw-r--r-- 1 apache apache 686760 Oct 17 14:07 media_store.json
-rw-r--r-- 1 apache apache 41728911 Oct 17 14:31 word_index_misc.json
-rw-r--r-- 1 apache apache 99669938 Oct 17 14:31 misc_store.json
drwxr-xr-x 5 apache apache 4096 Oct 17 14:31 .
-rw-rw-rw- 1 apache apache 287523 Oct 17 14:31 ise_trace.log
-rw-rw-rw- 1 apache apache 9164090 Oct 17 14:31 misc_indexer.log
Live root@forum:14:42:26:/var/www/html/ISE_Data#
Claude: [October 17, 2026, 05:43:56 PM]
Your prompt says 14:42 on Live while your clock says 17:42, so Live is three hours behind you. Read that way, only one store has been rebuilt since I delivered the zip:
misc_store.json and word_index_misc.json (14:31): that is 17:31 your time, a minute after the v1.2.2 zip arrived, so this was probably the new MiscIndexer.post_store.json and word_index.json (14:00): that is 17:00 your time, before any of my zips, so this was the old IndexBuilder.media_store.json (14:07): also before the zip, so the old MediaIndexer.Those two will still hold the Master post until you rerun them. Your listing also shows no PDF or audio stores. If that was just a filtered view, they aren't affected here.
To confirm the new code actually ran, check the logs:
grep -i 'do-not-index' misc_indexer.log should show a line like #do-not-index: N attachment(s).indexbuilder.log should end with (+N skipped: #do-not-index).A log with no mention means the old script ran, or nothing was marked.
i only just changed it at 44 min past
Claude: [October 17, 2026, 05:45:26 PM]
That corrects my read. Every store in that listing predates your change at 17:44, so all of them came from the old scripts, including Misc at 17:31. I shouldn't have credited Misc to the new code.
Nothing has been rebuilt yet, so the Master post will still be in all of them. Rerun the indexers now that the files are in and the tag is on the post. Then check each log for its skip line:
(+N skipped: #do-not-index) at the end.#do-not-index: N attachment(s).i am going to clobber post_store, word_index, word_index_misc and misc_store DOT JSON !
Claude: [October 17, 2026, 05:48:37 PM]
That's safe for those four. IndexBuilder and MiscIndexer rebuild their stores from scratch on every run, so deleting them loses nothing a rerun won't recreate.
The search for those two sources has no index to read from between the delete and the end of the rerun, so results will be empty during that window. IndexBuilder is the long one: your post_store.json is 58 MB.
Don't clobber media_store.json the same way. A plain MediaIndexer run rebuilds it, but MediaProcessor's captions and embeddings are separate stores and stay untouched. A plain run is enough for the media catalogue.
no, i am not silly
Claude: [October 17, 2026, 05:49:09 PM]
Understood. You had already left media_store.json out of the list, and my warning about it was unneeded.
done, now i am tarballing the whole ISE_Data dir then kicking off the 3 indexers
Claude: [October 17, 2026, 06:08:22 PM]
Good plan. The tarball is your rollback if anything misbehaves. If a run fails, it fails closed and writes nothing, so a store you didn't clobber is still intact.
When they finish, check the skip lines in the logs and search for a word that only appears in the Master post. Send me any log output that looks off.
#do-not-index
%supercalifragelistic%
Claude: [October 17, 2026, 06:10:10 PM]
That works as a test post. The tag is on line 1, and the word is one nothing else will contain. I take it you'll post that as your Group 1 account in your own board.
After the three indexers finish, search ISE for supercalifragelistic from an account that can't see your board. It should return nothing. If you search it as yourself and it's still missing, that confirms the post was skipped. Search with the % signs left off, since the tokeniser may treat them as part of the word.
If it does show up, send me the indexbuilder.log tail and I'll look at the author's group id first.
no, that's my real FILE !
Claude: [October 17, 2026, 06:11:27 PM]
Then that test won't work, because the rule only reads the post body. A tag inside an attached file's own text does nothing. If that file is a .txt or .md, MiscIndexer will index it and supercalifragelistic will be searchable.
For the test to work, put #do-not-index on line 1 of the post the file is attached to, and post as your Group 1 account. Then the file and the post both stay out.
If you do want a tag inside a file to hide that file, that is a different rule. I'd need to read each attachment's first lines, and it would apply to PDFs and the other types too. Tell me if you want it, and I'll list what it would change before building it.
that is in the post's body
Claude: [October 17, 2026, 06:12:14 PM]
If both lines are in the post body, my last reply doesn't apply. The tag is on line 1, so the post and its attachments should be skipped, as long as your Group 1 account wrote the post.
I'd read "my real FILE" as meaning the text was inside an attachment. Check that, because it is the one thing that would make the test fail. The text has to be in the post's own body.
ok, post is done
Claude: [October 17, 2026, 06:12:46 PM]
Good. Run the indexers now. The post only gets skipped on a run that starts after it was saved.
Afterwards, search for supercalifragelistic. It should find nothing, and each log should show its skip line.
supercal* = 0 posts, super* = lots
Claude: [October 17, 2026, 06:14:23 PM]
That result is what a working rule gives, but it can't prove it yet. A brand-new post shows 0 results whether it was skipped or the indexer simply hasn't run since you saved it. super* returning lots only shows the index itself is live.
To tell the two apart, check one of these:
tail indexbuilder.log should show (+N skipped: #do-not-index) with N of at least 1, and the log's timestamp should be after 18:12.supercal* should still be empty.If the log shows no skip line, send me its tail.
butSMF will find it if it wasn't so busy, lol !
Claude: [October 17, 2026, 06:15:40 PM]
True, and that is a useful control. SMF's own search ignores the tag and indexes every post. Once it catches up, supercalifragelistic should turn up there for you, and only you, because of your board's permissions. If it shows in SMF but not in ISE after the run, the post was skipped, which is what you want.
PDFIndexer.py v8.7.0
--fast: PyMuPDF only, pypdf skipped.
#do-not-index: 3 attachment(s) in marked posts will be skipped.
Resuming: 1236 PDFs already indexed on disk, will skip those. (Use --fresh to ignore and rebuild everything.)
Scanning 2 directories for PDFs (magic-byte detection): /var/www/html/attachments, /var/www/html/attachments2
Claude: [October 17, 2026, 06:16:02 PM]
That line shows the rule working. PDFIndexer found 3 attachments in marked posts and will skip them. It read that from the database, so the tag, the line 1–5 check and the author check all passed for those posts.
Those 3 PDFs may already be in your 1,236 indexed PDFs from earlier runs. If any were, the run prints a purged N already-indexed line before it starts scanning, and your paste doesn't show one yet. If that line doesn't appear by the end of the run, none of the 3 were in the old index. Either way, they won't be in the finished index.
...45000 posts processed
Done processing 45432 posts (+1 skipped: #do-not-index), 108593 unique words indexed.
Closing database connection...
Claude: [October 17, 2026, 06:16:33 PM]
IndexBuilder skipped 1 post for #do-not-index out of 45,433, and indexed the other 45,432. That is the post you just wrote, which has the tag on line 1. The supercal* search will now return 0 because the post was skipped. You can still check that with the control word.
Your PDFIndexer run found 3 attachments in marked posts, so there are more marked posts than this one. Those posts would have to have been marked by the same rule, with the tag on lines 1–5 and a Moderator-or-above author. Check that the other marked posts are the ones you meant, because the post count here is only 1. The extra 2 attachments could belong to this one post, since a post can carry several.
but they weren't PDFs, but nw
Claude: [October 17, 2026, 06:16:59 PM]
That explains it. The PDFIndexer line counts every attachment of marked posts, and any image thumbnail records too, not just PDFs. The figure of 3 was my wording being loose, and it wasn't a mistake in the rule. It also means the 2 extra weren't a sign of other marked posts, so my last paragraph was wrong. One marked post with its attachments and thumbnails fits what you saw.
the .csv attachs are part of misc ...
Claude: [October 17, 2026, 06:18:11 PM]
That gives the answer to the "3": your three CSVs, the Coinspot, Cake Wallet and order-history files, are the 3 attachments.
One thing doesn't fit. Your MiscIndexer.py has no mention of csv anywhere. Its list is .md, .txt, .text, .json and .html, plus the filename-only code types, and the manual's list matches. So as the script stands, MiscIndexer wouldn't pick up the CSVs at all. They'd be skipped by file type, not by the tag.
Check this in the Misc log: it should show whether the CSVs were matched. If they were matched, your running copy differs from the one you sent me. If they weren't, nothing was ever going to index them and the tag made no difference there.
i thought they would ne processed as flat files (text)
Claude: [October 17, 2026, 06:19:39 PM]
It doesn't work that way. MiscIndexer.py only takes extensions on a fixed list, MISC_EXTENSIONS, and .csv isn't on it. Nothing in the script treats unknown types as plain text. Your three CSVs have never gone into the Misc index, with or without the tag.
So they are not searchable in ISE today. They are text, so they would index fine if you add .csv to MISC_EXTENSIONS. I haven't done it because it changes what gets indexed, and you didn't ask for that. Your transaction CSVs would also become searchable in ISE for anyone who can see their post. If you want it, say so and I'll add it, but that is a decision for you, and the tag would cover them too.
there should be a catch-all otherwise all whis was for nought
Claude: [October 17, 2026, 06:20:34 PM]
It wasn't for nothing, and a catch-all wouldn't help here. A file type no indexer reads is not in any index, so nothing about it can leak through the index. The rule covers every attachment of a marked post whatever its type, so any type an indexer picks up later is covered too, as long as that indexer calls ise_noindex.
There is a gap, though, and it's in the search scripts, not the indexers. By your own manual, PDFsearch.py, MiscSearch.py and MediaSearch.py match filenames and topic subjects by querying the database live at search time, and not only from the indexes. I haven't seen those scripts, so I can't say they're leaking. But if they work as the manual describes, a marked post's attachment filenames and subject line could still turn up in results, even with its content out of every index.
That is what to test first. Search ISE for a word from the Master post's subject and from one attachment's filename, as an account that can't see your board. If either appears, those searches need the same exclusion applied at query time. Send me PDFsearch.py, MiscSearch.py and MediaSearch.py and I'll add it to all three, listing the changes before I build them.
zip files show up in binary
Claude: [October 17, 2026, 06:20:54 PM]
None of the five indexers reads a zip. PDFIndexer detects PDFs by magic bytes, MiscIndexer takes a fixed list of text types, and MediaIndexer takes images, video and audio. So a zip's contents never reach an index, and a password-protected 7-Zip or zip is unreadable anyway.
The only way a zip could show up in search is by filename, through the live database matching I described. That makes the filename and subject test the one worth running.
If you meant that you saw a zip appear as binary text in a result, paste what you saw. That would mean something is reading it that I haven't seen.
oh, that is listatt
Claude: [October 17, 2026, 06:21:22 PM]
That explains it. [listatt] shows a post's attachments, zips included, when the post is displayed. That is SMF rendering the post, so it follows your board's permissions and has nothing to do with ISE. Only people who can open the post see the list.
The ISE exposure question is still the filename and subject test from before.
[MiscIndexer.py] here, add a catch-all
Building the MiscIndexer catch-all now (7.4.1#5). Things you didn't spell out, so you can stop me:
non-binary, flat only
yeah, cool
not doc or docx tho
Claude: [October 17, 2026, 06:23:57 PM]
MiscIndexer.py 7.4.1#5 now has the catch-all, and it takes only non-binary flat text. Your three CSVs will be indexed on the next run, and doc and docx are on the never-opened list. I tested it with stand-in data: a real zip, a gzip and random bytes were rejected, and UTF-8 and cp1252 CSVs, plus a file with no extension, were indexed. It has not been run against your live board.
It indexes any regular file attachment that isn't:
doc, docx and the other Office formats);I made the catch-all files searchable in ISE like any .txt, so your transaction CSVs become searchable too, unless their post carries the tag. #do-not-index applies to them. I haven't seen MiscSearch.py, so I can't say how it displays an unknown type like csv.
Run it with --dry-run --monitor first if you want to see the new Catch-all: summary line before it writes anything.
The manual is now Master v7, with a new "Catch-all" section under MiscIndexer.
| Item | Status |
|---|---|
| @version/@date bumped | Done, 7.4.1#5; manual is Master v7 |
| History entries | Done |
| Version shown at run start | Existing |
| Versioned zip | MiscIndexer_v7.4.1.5.zip |
| xmllint | Not needed, no XML |
| Unrequested changes listed beforehand | Done, the six points sent before building |
| Legacy zipper | Not needed, this is an ISE script rather than an SMF package |
sh, py etc all good
Claude: [October 17, 2026, 06:24:13 PM]
.sh, .py and the other code types are unaffected. They are on the fixed list and still index by filename and type only, with no content read. The catch-all doesn't touch them, and none of them are on the binary list.
If you also want their contents searchable, say so. That means removing them from FILENAME_ONLY_EXTENSIONS. It would put your source code into the Misc word index.
#do-not-index: 3 attachment(s) in marked posts skipped.
901 attachments matched (.c, .conf, .cpp, .css, .go, .html, .ini, .java, .js, .json, .md, .php, .pl, .py, .rb, .rs, .sh, .sql, .text, .txt, .xml, .yaml, .yml) and exist on disk.
Claude: [October 17, 2026, 06:29:38 PM]
That output fits 7.4.1#5. Under #4 the three CSVs were never matched, so the #do-not-index line could only show 3 if three files of the fixed types were in marked posts. The "3 attachment(s)" PDFIndexer found was the whole marked set, and you said those were your CSVs. So the catch-all now sees the CSVs and the tag skips them. To confirm which version ran, look at the line above the paste: Cross-referencing against DB: listed types ... plus catch-all appears only in #5.
The second line is misleading. "901 attachments matched (.c, .conf, ...)" still prints only the fixed extension list, though the 901 now includes catch-all candidates. The count is right and only the wording is stale. I'll fix it in the next release rather than ship a new version just for a message.
The 901 is before the binary and size checks, so fewer than 901 will end up indexed. The Catch-all: line in the final summary has the real numbers.
i am happy
Claude: [October 17, 2026, 06:31:19 PM]
Good. Still open on my side:
MediaProcessor.py, to purge captions and embeddings already made for a marked image or video.PDFsearch.py, MiscSearch.py and MediaSearch.py, which match filenames and subjects live from the database. Run the filename and subject test first, and send those scripts only if it shows a leak.testing authority now, then i am done
Claude: [October 17, 2026, 06:47:37 PM]
For the authority test, post the tag as a regular member (not Group 1 or 2, not a board moderator) on line 1 with a made-up word, then rerun IndexBuilder. The post should be indexed. That makes the word searchable and the skip count unchanged. If it's skipped, the author check isn't working.
Then post the same thing as a global moderator, or as a board moderator in their own board. That one should be skipped and the count goes up by one.
Run the member test first, since a skipped member post is the failure that matters.
Done processing 45433 posts (+1 skipped: #do-not-index), 108594 unique words indexed.
Closing database connection...
Writing /var/www/html/ISE_Data/word_index.json and /var/www/html/I
Claude: [October 17, 2026, 06:47:58 PM]
That is the right result for the member test. Posts went from 45,432 to 45,433 and unique words from 108,593 to 108,594, while the skip count stayed at 1. So the member's tagged post was indexed and your original post is still skipped.
The moderator half of the test is left. Post the same thing as a global or board moderator and rerun. The skip count should go to 2.
this is "regular-users"
Claude: [October 17, 2026, 06:48:14 PM]
Then that result is a valid member test. A "regular-users" account sits outside groups 1 and 2, and its tagged post was indexed, so the authority check works for ordinary members.
That leaves the moderator side of the test, and the board-moderator case if you use it.
but it said skipped ?
Claude: [October 17, 2026, 06:49:02 PM]
"+1 skipped" is the count of all tagged posts that were skipped, and it was already 1 in the 18:16 run. That one is your original Group 1 post. Since the count didn't move, the regular-users post wasn't added to it, and the run indexed one more post and one more word.
That is only by the numbers. The decisive check is a search for the made-up word from the regular-users post. It should be found. If it isn't found, that post was skipped and the count would have gone to 2, which didn't happen. If you'd posted it before the 18:16 run, the skip count would have been 2 then as well.
ok, keyword test now
Claude: [October 17, 2026, 06:49:39 PM]
Go ahead. The made-up word from the regular-users post should come back with that post. Search supercalifragelistic as well. It should still return nothing, since that post is the Group 1 one.
all good, will get the pdf and mirror
Claude: [October 17, 2026, 06:51:29 PM]
Good, both halves of the keyword test behaved. Before you mirror to Live, confirm the Clone's stores no longer hold the Master post, so the copy that goes up is clean. PDFIndexer and AudioProcessor purge only when they run, and MediaProcessor's captions and embeddings are still untouched.
i need a markdown transcript pls - special case