mediawatchdog — manual

Module	mediawatchdog (bash, lives in /bin)
@version	1.0
@date	2026-09-24
Credit	Another collaboration between Andrew.human and Claude.ai
Supervises	/var/www/html/The_ISE_Project/MediaProcessor.py (ISE v9.20+)

Purpose

MediaProcessor.py crashes when the Live droplet runs out of memory. The crashes are an uncaught OSError: [Errno 12] Cannot allocate memory from fork, or a SIGKILL from the kernel's OOM killer. MediaProcessor saves each caption and embedding as soon as it's made, so a restart loses nothing and carries on quickly: done items are skipped with a dictionary lookup. mediawatchdog does the restarting for you, so you don't have to watch and restart by hand.

Usage

mediawatchdog                               # start, detached (safe to log off)
mediawatchdog --caption-timeout 180         # any args are passed to MediaProcessor.py
mediawatchdog --max-caption-retries 0       # e.g. to clear the caption backlog
mediawatchdog --status                      # watchdog/processor PIDs, store counts, last log lines
mediawatchdog --stop                        # stop the watchdog AND its MediaProcessor
tail -f /var/www/html/ISE_Data/mediawatchdog.log

MediaProcessor always runs as python3.8 -u MediaProcessor.py --log --monitor <your args>.

Behaviour

MediaProcessor exit	Watchdog action
0: clean finish	Logs the missing-caption count and stops
130: SIGINT (Ctrl-C)	Treated as a deliberate stop, so it stops
143: SIGTERM (kill <pid>, --stop)	Treated as a deliberate stop, so it stops
137: SIGKILL (usually the OOM killer)	Restarts after PAUSE seconds
any other code (e.g. an ENOMEM traceback)	Restarts after PAUSE seconds

• Stall guard: progress is measured as captions + embeddings on disk, before and after each run. After MAX_STALLS crashed runs in a row with no gain, it gives up and logs GIVING UP, so it can never loop forever on a fault it can't get past.
• Adopt: if a python3.8 … MediaProcessor.py is already running (started by hand or by indeximages), it waits for that process to exit and then takes over.
• Orphans: before each run it kills any llama-mtmd-cli left behind by a crashed parent.
• Single instance: a pidfile plus flock on ISE_Data/.mediawatchdog.lock.
• Detach: setsid + nohup. It does not need tmux, which isn't installed on Live.

Tunables (environment variables)

Var	Default	Meaning
PAUSE	15	seconds between a crash and the restart (lets memory settle)
MAX_STALLS	3	no-progress crashes in a row before giving up
MAX_RESTARTS	100	hard cap on runs
ISE_DIR, DATA_DIR, PY	Live paths, python3.8	overridable for testing

Example: PAUSE=30 mediawatchdog --caption-timeout 180

Files

File	Content
ISE_Data/mediawatchdog.log	[WD …] watchdog lines, plus MediaProcessor's stdout/stderr (including tracebacks)
ISE_Data/.mediawatchdog.pid / .child	watchdog PID / current MediaProcessor PID
ISE_Data/.mediawatchdog.lock	flock single-instance lock

Limits

It restarts after crashes. It doesn't fix what causes them. The causes are MediaProcessor holding CLIP/torch while it forks, which needs memory headroom on a 2 GB droplet, and the unhandled ENOMEM in extract_frames_at_intervals(). Both need a MediaProcessor.py code fix (v9.21).

Tested

In the workspace, using a stand-in MediaProcessor, not on Live:

• crash, crash, then a clean finish: restarted twice, then stopped
• three crashes in a row with no progress: gave up
• crashes that did save progress: kept restarting until a clean finish
• --stop during a run: the child got SIGTERM and nothing was left running
• starting a second copy: refused
• a run already going: waited for it, then took over