dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse


Our Discord Notification Server invitation link is https://discord.gg/jB2qmRrxyD
« previous next »
Pages: 1   Go Down

Author Topic: Master Post 00: 'DMS/NX - The NextGen Document Mangagment System'  (Read 115 times)

Online Chip (OP)

  • Server Admin
  • Admin
  • Hero Member
  • ****
  • Administrator
  • *****
  • Restrictive
  • *
  • RC Discussion
  • **
  • Private
  • ***
  • Chip
  • *****
  • Join Date: Dec 2014
  • Location: Australia
  • Posts: 7388
  • Reputation Power: 0
  • Chip has hidden their reputation power
  • Gender: Male
  • Last Login:Today at 06:12:47 AM
  • Deeply Confused Learner
  • Profession: IT Engineer now retired
Master Post 00: 'DMS/NX - The NextGen Document Mangagment System'
« on: September 21, 2026, 09:51:31 AM »
📎 Click on me to view all my attachments
ListAttBBC  |  v6.12  |  2026-09-10  |  Andrew.human & Claude.ai
#FileSizeDownloadsInfoDL
00Checkpoint_CollabCore-MsgTracker_2026-09-21.md31.9 KB3ℹ️⬇️
01Conversation_Transcript_2026-09-21.md101 KB1ℹ️⬇️
02Conversation_Transcript_2026-09-21.pdf  [native]371.4 KB9ℹ️⬇️
03Checkpoint_ISEmedia-Bridge-and-Ranking_2026-09-23.md11.6 KB1ℹ️⬇️
04ISE_Python_Scripts_Manual.pdf  [native]366.1 KB4ℹ️⬇️
05ISE_Manual.pdf  [native]201.1 KB5ℹ️⬇️
06ISE_help.html22.3 KB1ℹ️⬇️
07Transcript_ISEmedia-Session_2026-09-23.pdf  [native]328.6 KB4ℹ️⬇️
08ISE_Checkpoint_2026-09-24.pdf  [native]94 KB4ℹ️⬇️
09ISE_Python_Scripts_Manual.pdf  [native]376.4 KB4ℹ️⬇️
10ISE_Manual.pdf  [native]206.2 KB4ℹ️⬇️
11Transcript_Orphan-Bug-Closeout_2026-09-24.md44.8 KB1ℹ️⬇️
12Transcript_Orphan-Bug-Closeout_2026-09-24.pdf  [native]783 KB4ℹ️⬇️
13ISE_Python_Scripts_Manual.md51.1 KB1ℹ️⬇️
14ISE_Manual.md30.4 KB1ℹ️⬇️
15changelog.md19.2 KB1ℹ️⬇️
16DMS-NX_Chapter2_Master_Post.md4.5 KB1ℹ️⬇️
17Improve the 'skip' logic - aka  'The Performance Optimization of the ISEmedia Overnight Image Indexing Job'.md34 KB1ℹ️⬇️
18Improve the 'skip' logic - aka  'The Performance Optimization of the ISEmedia Overnight Image Indexing Job'.pdf  [native]154.6 KB3ℹ️⬇️
19Classification_Progress_Report.md2.3 KB1ℹ️⬇️
20ISEmedia_Caption_Run_Transcript_2026-09-24.md17.9 KB1ℹ️⬇️
21ISEmedia_Caption_Run_Transcript_2026-09-24.pdf  [native]105.1 KB4ℹ️⬇️
22ISEmedia_Caption_Run_Transcript_2026-09-24_Part2.md38.9 KB1ℹ️⬇️
23ISEmedia_Caption_Run_Transcript_2026-09-24_ Part2.pdf  [native]105.1 KB4ℹ️⬇️
24ISEmedia_Caption_Run_Transcript_2026-09-24_Part3.pdf  [native]157.4 KB3ℹ️⬇️
25ISEmedia_Caption_Run_Transcript_2026-09-24_Part3.md30 KB1ℹ️⬇️
26mediawatchdog.md3.9 KB1ℹ️⬇️
27ISEmedia_MediaSearch_v1.4.1_Viewer-Avatars-VideoIndex.pdf  [native]124.2 KB3ℹ️⬇️
28ISEmedia_MediaSearch_v1.4.1_Viewer-Avatars-VideoIndex.md17.4 KB1ℹ️⬇️
29ISEmedia_Progress_Report_Real_Denominator.pdf  [native]114.4 KB4ℹ️⬇️
30ISEmedia_Progress_Report_Real_Denominator.md27.9 KB1ℹ️⬇️
31ISEmedia_Caption_Run_Transcript_2026-09-24.pdf  [native]174.1 KB3ℹ️⬇️
32ISE_v9.5.zip6.9 MB2ℹ️⬇️
33ISEmedia_Progress_Report_Real_Denominator.pdf  [native]178.6 KB4ℹ️⬇️
34ISEmedia_Progress_Report_Real_Denominator.md61.3 KB2ℹ️⬇️
35ISEmedia_Progress_Report_Real_Denominator.pdf  [native]114.4 KB3ℹ️⬇️
36ISEmedia_Progress_Report_Real_Denominator.md27.9 KB1ℹ️⬇️
37ISEmedia_Progress_Report_Real_Denominator_2.pdf  [native]178.6 KB3ℹ️⬇️
38ISEmedia_Progress_Report_Real_Denominator_2.md61.3 KB1ℹ️⬇️
39ISE_v9.5.zip6.9 MB2ℹ️⬇️
40DMS_NX next generation document management - Chapter 2 - Claude.pdf  [native]2 MB2ℹ️⬇️
41Chatlog webpage export issue - Claude.pdf  [native]329.6 KB2ℹ️⬇️
42Safeguard flagging issue - Claude.pdf  [native]320 KB1ℹ️⬇️
43Smokenizer Pipeline project - Claude.pdf  [native]273 KB1ℹ️⬇️
44DMS-NX__ISE_v9.6__Checkpoint_2026-09-25.md8.4 KB1ℹ️⬇️
45DevTools Snippet.txt291 B5ℹ️⬇️
46DMS-NX__ISE_v9.6__Chapter3_Checkpoint_2026-09-25.md11.9 KB1ℹ️⬇️
47audio_vocab.txt2.8 KB6ℹ️⬇️
48AudioProcessor_manual.md6 KB1ℹ️⬇️
49unclip_overflow.txt285 B5ℹ️⬇️
50Unclip_Overflow_JS_Fix.md3 KB1ℹ️⬇️
51Unclip_Overflow_JS_Fix.pdf  [native]64.1 KB1ℹ️⬇️
52psw_process_monitor.md24.7 KB1ℹ️⬇️
53psw_process_monitor.pdf  [native]162.3 KB2ℹ️⬇️
54DMS_NX next generation document management - Chapter 3 FULL - Claude.pdf  [native]4.1 MB2ℹ️⬇️
55DMS-NX__ISE_v9.6__Chapter3_Checkpoint_2026-09-25_2.md19.3 KB1ℹ️⬇️
56ISE_v9.6.zip7 MB113ℹ️⬇️
DMS/NX - The "Next Generation Document Mangagment System" Master Post

This is now LOCKED and with a new name, continues on in:
 * "Master Post 01: 'DMMS/NX - The NextGen Document and Media Mangagment System'"

Includes these SMF mods: Collaborative, ListAtt, PDFBBC, MDBBC, LMVBBC, LTVBBC and (the) ISE (Project) [stable at version 9.6]

NB: This is the pyhsical and logical extension of (the) ISE (Project) [stable at version 9.5], including these two closed (LOCKED) Master Posts:

* "Master Post for ISEpdf -- The Intelligent Search Engine for PDFs"

* "ISE - Intelligent Search Engine for Posts and Their Attachments"


This PDF attachment Orphan Report runs every Monday at 6am, logs output and only outputs when something's new

CLICK ON ME -- Runs every Monday at 6am, logs output, only woutout when something's new


1. Image and video samples classification
IMAGES (CLIP) --------------- 2564 files --  271.4 MB

VIDEO (keyframe+Whisper) --  69 files -- 2296.9 MB

1950 files --  524.2 MB  .jpg
 544 files --   723.6 MB  .png
  69 files --  2296.9 MB  .mp4

Image Classification is as good as done -- progress monitor says:

Claude: [25/09/2026--13:43:21:013 AEST]

The classifier stopped about 10h 40m ago. Both stores were last written at 01:01:35, no llama-mtmd-cli is running, and 25 items are still pending (17 images, 8 videos). The "rate: n/a" line has the same cause: the store hasn't been saved since the first sample, so there's nothing to compute a rate from. Whether and when to restart it is your call.

The self-check failure is two separate problems that pull in opposite directions. The four categories add up exactly to 351 + 349 + 223 + 2 = 925, so every one of the unexplained ids is accounted for:

CategoryCountWhich side is probably wrong
no post (avatar?)351The driver. Avatars aren't post media, so they shouldn't be classified at all.
thumbnail349The driver. SMF stores thumbnails as their own attachment rows, so each of these is a second copy of an original that's also in the store.
.webp223The inventory. These are real post images that its extension whitelist leaves out.
.3gp2The inventory. The same whitelist gap, for video.

So the true denominator isn't 1745 as the report shows, and it isn't 1745 + 925 either. If the driver drops avatars and thumbnails and the inventory adds .webp and .3gp, it comes to about 1970. That's assuming those 225 are post attachments, and the self-check didn't flag them as "no post", which suggests they are. Most of them appear to be classified already, so the 98.5% figure is roughly right for the wrong reason.

The report also doesn't reconcile keys against ids. There are 2757 keys in the store. 1720 attempted plus 925 unexplained ids is 2645, which leaves about 112 keys unaccounted for. My guess is those are extra per-frame video keys (171 frame keys stored against 61 attempted videos). If so, the report should say it rather than leaving it for you to work out. On a related point, there are 3 missing captions but only 2 cap-misses. The extra missing caption is probably one of the 925 excluded ids, and /tmp/missing_captions.txt will show which.

The one I'd check first is the 700 avatars and thumbnails. If MediaSearch only searches the store and doesn't join back to the inventory, they may already be showing up in ISEmedia results. That would look like duplicate hits (each thumbnail next to its original) and profile pictures ranking as matches for ordinary searches. A quick way to test is to search something common and see whether both copies of the same image come back.

To fix this properly I need the source files, since I've only seen the report output. Please send Classification_Progress_Report.py, the classifier driver script, and whatever builds the inventory. A zip is fine. 

I'll check the exclusion rules and the extension whitelist on both sides against the code and propose targeted fixes, with version bumps and doc-blocks per your convention.[/quote]


Live  root@forum:11:45:20:/var/www/html/The_ISE_Project# chkindeximages
================================================================
 Classification_Progress_Report.py  v2.3  (2026-09-24)
 Andrew.human & Claude.ai      2026-09-25 12:07:15
================================================================
            total    done  cap-miss  pending
 images      1676    1658         1       17
 videos        69      60         1        8
 ALL         1745    1718         2       25
----------------------------------------------------------------
 DONE      1718 of 1745  (98.5%)   attempted 98.6%
 video frame keys stored: 171
 rate: n/a yet (needs a store save at least 60s after the first sample)
----------------------------------------------------------------
 !! SELF-CHECK FAILED: 925 store ids not in the inventory, e.g. 1007, 1009, 101, 1023, 1024, 103, 1031, 1039
 !! The driver processes items this inventory excludes - the
 !! denominator is NOT trustworthy until this is resolved.
 !!     351  no post (avatar?)        e.g. 1023, 1024, 1031, 1039, 1040
 !!     349  thumbnail                e.g. 1007, 1009, 101, 103, 1048
 !!     223  ext='webp'               e.g. 3075, 3143, 3144, 3145, 3146
 !!       2  ext='3gp'                e.g. 1310, 1311
 !! full list -> /tmp/unexplained_ids.txt
 keys: 2757 embedded, 2754 captioned, 3 missing caption, 0 orphan
  embeddings_store.json  last written 2026-09-25 01:01:35 (665m 39s ago)
  captions_store.json    last written 2026-09-25 01:01:35 (665m 39s ago)
----------------------------------------------------------------
 No llama-mtmd-cli process running.
 Missing-caption keys -> /tmp/missing_captions.txt

2. Now we will classify all audio (*.wav and *.mp3 attachments:

44 files --  644.9 MB  .mp3
07 files --  173.1 MB  .wav

3. We will need to use the faster "Clone" VM system to, process the audo,so let's update/sync the Python environment to Live, and also keep the "Dev" (32-bit) VM system compatible:

Claude: [25/09/2026--15:37:43:650 AEST]

Here's what Dev's diff shows. One finding comes first, because it made part of the inventory wrong:

Something in the folder you ran the inventory from is overriding Python's own random module. On Dev, requests and mysql.connector failed with "random has no attribute randint". That's the classic sign of a file called random.py (or random.pyc) sitting in the current directory. Python finds it before its own standard random module, so every script run from that folder breaks the same way. I've made two changes as a result:

  • py_inventory.sh v1.0.1 now switches to / before testing, so this can't skew the results again.
  • dev_py_sync.sh reports that file if it's in /root, but doesn't touch it. If it isn't meant to be there, renaming it is up to you.

Where Dev stands (i686, glibc 2.12):

InterpreterState on Dev
Python 3.6 (ISE search scripts)Missing about 20 of Live's packages, including numpy, Pillow, PyYAML, regex, lxml, pikepdf and ocrmypdf. It also has the wrong mysql.connector (8.0.23 instead of Live's 2.2.9) and an older PyMuPDF (1.11.1) and cryptography (2.6.1).
Python 3.8Almost empty: pip only
Python 2.6No PIL (Live has it from the system package)
python3 and python linksAlready match Live
Dev onlyPython 2.7.14, PyPDF2, ffmpeg 2.6.8 (Live has 4.4.5)

dev_py_sync.sh (attached) makes Dev as close to Live as a 32-bit machine allows:

  • It installs Live's exact versions into Dev's own 3.6 and 3.8, one package at a time, with dependency installs turned off and a 15-minute limit per package. When an install fails, pip keeps the version that was already there.
  • Python 3.6:

  - it removes mysql-connector-python 8.0.23 and its protobuf dependency   - then it reinstalls Live's mysql-connector 2.2.9

  • It skips the heavy AI packages: torch, torchvision, open_clip, timm, transformers, tokenizers, llama and the model files. It also skips the /opt Python 3.12 and yt-dlp setup.
  • It adds Live's aliases and tries yum install python-imaging for Python 2.6.
  • It checks it's on Dev by hostname and architecture before doing anything, and has a --dry-run mode.
  • It writes a report of what worked and what didn't to /root/dev_py_sync_report.txt, with a full log for each package.

Some packages will probably fail. At these old versions, several of Live's packages have no ready-made 32-bit build, so pip will try to compile them on CentOS 6:

  • Probably won't install: cryptography 37 (building it needs Rust), lxml 5.4 and pikepdf (they need newer C libraries than CentOS 6 has), and PyMuPDF 1.18. Dev keeps its older versions of these.
  • Might compile, might not: numpy 1.24.4 and regex for Python 3.8, depending on whether CentOS 6's gcc 4.4 can build them.

So on Dev, the PDF and OCR tools may end up on older versions than Live's. The report will list exactly which ones.

Steps on Dev:

  1. bash dev_py_sync.sh --dry-run. Look at the list, and at whether it reports a stray random.py.
  2. bash dev_py_sync.sh, then send me /root/dev_py_sync_report.txt.
  3. Run py_inventory.sh v1.0.1 again and compare it with Live's inventory, as we did for Clone.

Clone's steps are unchanged (make the restore point, then clone_py_sync.sh --dry-run). Clone and Dev don't depend on each other.


4. But perform these 3 steps first:
- 1. Take an AI deep checkpoint
- 2. Build MD and PDF transcripts
- 3. run "mirror" (syncs Live webroot files+DBs to Clone and Dev)


5. Understand "venv" !

Claude: [25/09/2026--21:39:00:336 AEST]

Smoke test result

LiveClone
Peak memory1484 MB1627 MB (plenty of room in 6 GB)
One 10-second window23.4 s19.6 s
Model loadnot recorded29.4 s (once per run)

Clone is only slightly faster, which suggests torch is getting very little CPU. Check how many CPUs the VM has and how many threads torch is using:

nproc
cd / && /root/venv-audio-test/bin/python -c "import torch; print(torch.get_num_threads())"

If nproc shows 1 or 2, giving Clone more vCPUs in VMware is the single biggest speed-up available. AudioProcessor will also set torch's thread count to match nproc. The figure is from a single window, which includes one-off first-call overhead, so later windows should be somewhat faster. The "noise" line (0.008 vs 0.008) is expected: random noise isn't like either label.

What a venv is

A venv (virtual environment) is a folder that behaves like its own copy of Python, without duplicating Python itself:

  • bin/python is a link to the real interpreter. For venv-audio-test, that's /usr/local/bin/python3.8.
  • lib/python3.8/site-packages/ is the venv's own package folder. Anything you install through the venv goes here and nowhere else.
  • pyvenv.cfg holds the settings, including one that decides whether the venv can also see the main Python's packages.

Why you'd use one: isolation. The CLAP stack needs transformers 4.27.4 and tokenizers 0.13.0. Installing those into the venv means the system 3.8, which MediaProcessor, pylint and the rest use, is left untouched.

What yours does specifically. Your smoke-test output shows it: the warning came from /usr/local/lib/python3.8/site-packages/huggingface_hub, which is the system folder, not the venv's. Your venv is only ~90 MB, far too small to hold torch. So it's almost certainly set to also see the system packages. It borrows torch, numpy and huggingface_hub from the system 3.8 and adds only transformers, tokenizers and the torch stub from Chapter 2. That's also why the system 3.8 shows transformers as FAIL while the venv shows OK. You can confirm it:

cat /root/venv-audio-test/pyvenv.cfg      # expect: include-system-site-packages = true

Using it: just call its Python directly, /root/venv-audio-test/bin/python script.py. There's no need to "activate" it; activating only puts that bin/ first in your shell's path.

Things to be careful of:

  1. It can't be moved or renamed. Its absolute path is written into its scripts, and bin/activate hard-codes it too. Copying it to the same path on another machine works, which is why your tarball did. Moving it to a new path breaks it. If you ever want a better name than -test, create a new venv instead.
  2. It depends on the system 3.8. If /usr/local/bin/python3.8 is removed or replaced, the venv breaks. Because this one borrows system packages, changing torch or numpy in the system 3.8 also changes them inside the venv.
  3. Always install into it through its own Python: /root/venv-audio-test/bin/python -m pip install .... A plain pip would install into whichever Python comes first in your path.

/opt/yt-dlp-venv works the same way, built on Python 3.12, and it has its own complete set of packages.


Quote
Me: aren't u supposed to deactivate it ?

Claude: [25/09/2026--21:44:18:623 AEST]

Only if you activated it first. deactivate undoes source /root/venv-audio-test/bin/activate, and nothing else.

  • Activating changes your current shell. It puts the venv's bin/ first in your path, so plain python and pip now mean the venv's, and it adds (venv-audio-test) to the front of your prompt. deactivate puts your path and prompt back.
  • Calling /root/venv-audio-test/bin/python directly, as everything today did, doesn't touch your shell at all. The venv is used for that one command and is gone when it finishes, so there's nothing to deactivate.

To check whether a shell is currently activated:

echo $VIRTUAL_ENV     # blank = not activated

If you do activate and forget to deactivate, the only effect is on that terminal session, where python and pip keep meaning the venv's. Closing the terminal clears it too. That risk is why I use the direct path in scripts and commands: the command itself says which Python it runs, whatever state your shell is in.


« Last Edit: Today at 05:15:31 AM by Chip »
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
Our Discord Server invitation link is https://discord.gg/jB2qmRrxyD

Offline smfadmin

  • SMF (internal) Site
  • Chip
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 636
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Today at 05:24:34 AM
  • Supplied Install Member
Re: Master Post 00: 'DMS/NX - The NextGen Document Mangagment System'
« Reply #1 on: Yesterday at 09:42:46 PM »
Topic bumped
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once

Offline smfadmin

  • SMF (internal) Site
  • Chip
  • Sr. Member
  • *****
  • Join Date: Dec 2014
  • Location: Management
  • Posts: 636
  • Reputation Power: 0
  • smfadmin has hidden their reputation power
  • Last Login:Today at 05:24:34 AM
  • Supplied Install Member
Re: Master Post 00: 'DMS/NX - The NextGen Document Mangagment System'
« Reply #2 on: Yesterday at 09:44:16 PM »
Topic bumped
friendly
0
funny
0
informative
0
agree
0
disagree
0
like
0
dislike
0
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
No reactions
measure twice, cut once
Pages: 1   Go Up
« previous next »
Tags:
 

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse




TERMS AND CONDITIONS

In no event will d&u or any person involved in creating, producing, or distributing site information be liable for any direct, indirect, incidental, punitive, special or consequential damages arising out of the use of or inability to use d&u. You agree to indemnify and hold harmless d&u, its domain founders, sponsors, maintainers, server administrators, volunteers and contributors from and against all liability, claims, damages, costs and expenses, including legal fees, that arise directly or indirectly from the use of any part of the d&u site.

TO USE THIS WEBSITE YOU MUST AGREE TO THE TERMS AND CONDITIONS ABOVE

Founded December 2014
SimplePortal 2.3.6 © 2008-2014, SimplePortal