AudioProcessor.py: Manual

@version 1.0.0 · @date 2026-09-25 · Another collaboration between Andrew.human and Claude.ai

Part of DMS/NX (ISEmedia). This manual is written to be read by people and by AI assistants.

---

1. What it is

AudioProcessor turns the forum's audio attachments (mp3, wav, flac, ogg, m4a, aac, wma) into searchable tags, using the CLAP model laion/larger_clap_music. The tags are stored as plain text in ISE_Data/audio_store.json, and MediaSearch on Live matches search words against that text. Live never runs CLAP. CLAP needs about 1.6 GB of memory, so AudioProcessor runs on Clone, and only the finished store is copied to Live.

2. Files

File	Where	Role
AudioProcessor.py	/var/www/html/The_ISE_Project/ (next to MediaProcessor.py)	The program
audio_vocab.txt	same folder	The tag vocabulary. Edit freely
ISE_Data/audio_store.json	written by the program	Vectors, tags and captions. Clone is its only writer
ISE_Data/audio_checkpoint.json	written by the program	Human-readable progress
ISE_Data/audioprocessor.log	with --log	Append-only run history
ISE_Data/hf_cache/models--laion--larger_clap_music	pre-downloaded	The model, pinned to revision a0b4534a…

It depends on Collabware/core_utils.py (v1.2.0) to find attachment files, on ISE_Data/media_store.json (from MediaIndexer.py) for the list of audio items, and on ffmpeg and ffprobe being on the PATH.

3. How to run it (on Clone)

Always run it with the audio venv's Python, from /:

cd / && /root/venv-audio-test/bin/python /var/www/html/The_ISE_Project/AudioProcessor.py --plan

--plan finds and measures every pending item, prints the number of windows and the estimated time, and encodes nothing.

To run it for real in the background (Clone has no tmux):

cd / && nohup /root/venv-audio-test/bin/python /var/www/html/The_ISE_Project/AudioProcessor.py --log --monitor \
      > /dev/null 2>&1 &
tail -f /var/www/html/ISE_Data/audioprocessor.log

With no arguments it does a real run and picks up where the last one stopped (ISE CLI convention).

4. How audio is sampled

Class	Rule	Stored
Regular (≤ 15 min)	6 evenly spaced 10-second windows. Shorter tracks get fewer: one per 10 seconds of audio	One whole-track vector, its tags and caption
Long mix (> 15 min)	A 10-second window starting every 30 seconds	Every window as <id>_a<start_sec> with its own tags, plus the whole-track average

Window keys never contain _t. That suffix belongs to video frames, and MediaSearch relies on it to recognise them.

Changing --long-threshold, --stride or --samples once a store exists is refused unless you add --force, which rebuilds everything. A store never mixes two sampling schemes.

5. How tagging works

Each vector is compared with every label in audio_vocab.txt. Within each [group], the labels compete with each other: the best one is always kept, plus up to --top-k (default 2) that score at least half as well. The chosen keywords become the item's caption, e.g. techno, dark, fast, instrumental, synth, music.

Changing the vocabulary is cheap. The 512-number vectors are saved. When the vocabulary file changes, the next run re-tags every stored item from those saved vectors in seconds, and no audio is processed again.

Vocabulary format:

[genre] {} music          <- group header; {} is replaced by the keyword
techno                    <- CLAP hears "techno music"
hi-nrg | hi-nrg dance music   <- CLAP hears exactly the text after |
# comment

6. Options

Option	Default	Meaning
--plan	off	Measure and estimate only
--attachid N	n/a	Only this attachment
--chunk-size N	all	Stop after N items
--force	off	Redo items already stored. If the settings changed, rebuild everything
--threads N	all cores	torch CPU threads
--batch N	4	Windows per CLAP call
--long-threshold S	900	Seconds; anything longer is a long mix
--stride S	30	Seconds between long-mix windows
--samples N	6	Windows per regular track
--top-k N	2	Maximum tags per group
--vocab PATH	audio_vocab.txt next to the script	A different vocabulary file
--sec-per-window S	19.6	Rate used for --plan's time estimate
--log [PATH]	off	Also append output to a log file
--monitor, --verbose	off	Telemetry at the end; per-item plan and file-lookup detail

7. Safety and errors

• The store is written after every item, using a temporary file that is then swapped in, so a crash loses at most the item in progress. Resuming is based on which items are in the store, not on the checkpoint file.
• Checks before any work starts: if ffmpeg or ffprobe is missing, or the attachment folders can't be read (for example, pymysql missing or the DB down), it stops at once with one clear message.
• Errors are always shown: a missing file, a failed ffprobe (there is never a guessed duration), a window that fails to decode, or a failed encode each print immediately and are listed again at the end. If it finishes silently, the run was clean.

8. After a run: publish to Live

From Clone:

rsync --partial -a /var/www/html/ISE_Data/audio_store.json root@128.199.200.202:/var/www/html/ISE_Data/

Clone's mirror from Live must exclude ISE_Data/audio_store.json* and ISE_Data/audio_checkpoint.json. Otherwise a mirror run deletes or overwrites Clone's copy.

9. Version history

• 1.0.0 (2026-09-25): first build. Mock-tested end to end (a 40-second track, a 16-minute 40-second long mix, a 5-second wav, a corrupt file, a missing file, a non-audio ID, a vocabulary change triggering re-tagging, the settings-change refusal, and a forced redo). The real CLAP calls come unchanged from clap_smoketest.py v0.1.0, which is proven on Clone. Python 3.8-compatible (checked with vermin).