Given that the toolchain is now operational, I would keep the ISEmedia design deliberately simple and modular. The key principle should be:
Classifier creates metadata → Indexer stores/search-enables it → Search queries it.
Do not build a giant monolithic AI pipeline yet. You have already established an incremental-development policy, and this is exactly where that pays off.
1. Proposed structure
ISEmedia/
│
├── ISEmediaClassifier.py
├── ISEmediaIndexer.py
├── ISEmediaSearch.py
│
├── lib/
│ ├── MediaDiscovery.py
│ ├── MediaMetadata.py
│ ├── ClipClassifier.py
│ ├── VideoSampler.py
│ ├── EmbeddingStore.py
│ └── SearchEngine.py
│
├── data/
│ ├── index/
│ ├── embeddings/
│ └── thumbnails/
│
└── config/
└── ISEmedia.conf
At this stage, the three primary scripts should remain independently runnable.
ISEmediaClassifier.py
Responsibility
Take a media file and produce an AI-enriched metadata record.
It should not worry about searching or database architecture.
Inputs
image
video
audio
Outputs
A structured record, initially JSON:
{
"path": "/media/example.mp4",
"type": "video",
"technical_metadata": {
"duration": 120.5,
"width": 1920,
"height": 1080,
"codec": "h264"
},
"classification": {
"labels": [
["person", 0.94],
["city", 0.82],
["night", 0.71]
]
},
"embedding": {
"model": "CLIP",
"vector_file": "..."
}
}
Classifier pseudocode
INITIALISE configuration
LOAD CLIP model
FOR each input media file:
DETERMINE media type
EXTRACT basic filesystem metadata
IF image:
LOAD image
GENERATE CLIP embedding
RUN zero-shot classification
USING configured vocabulary/prompts
CREATE metadata record
IF video:
USE ffmpeg/ffprobe
EXTRACT:
duration
resolution
codec
audio information
SELECT representative frames
simple initial algorithm:
first frame
25% frame
50% frame
75% frame
final useful frame
FOR each selected frame:
GENERATE CLIP embedding
RUN classification
COMBINE frame results
example:
aggregate labels
calculate mean/max confidence
remove duplicates
CREATE video metadata record
IF audio:
EXTRACT technical metadata with ffprobe
CREATE basic metadata record
[future AI audio analysis]
SAVE metadata JSON
SAVE embedding/vector
Video sampling
This deserves its own small component because video classification can otherwise become unnecessarily expensive.
v1 approach
Do not analyse every frame.
video
│
▼
ffprobe duration
│
▼
choose N timestamps
│
├── 0%
├── 25%
├── 50%
├── 75%
└── 95%
│
▼
ffmpeg extracts frames
│
▼
CLIP classifies frames
│
▼
aggregate results
Pseudocode:
FUNCTION sample_video(video, sample_count):
duration = ffprobe(video).duration
timestamps = evenly_spaced(
0,
duration,
sample_count
)
FOR timestamp IN timestamps:
frame = ffmpeg_extract_frame(
video,
timestamp
)
yield frame
Later versions can add:
scene detection
shot detection
motion detection
adaptive sampling
But not yet.
ClipClassifier.py
This should hide PyTorch and CLIP from the rest of the system.
The rest of ISEmedia should not care how the embedding is generated.
CLASS ClipClassifier:
INITIALISE:
load PyTorch
load CLIP model
load classification prompts
FUNCTION classify(image):
preprocess image
image_embedding =
CLIP.encode_image(image)
FOR each configured label:
text_embedding =
CLIP.encode_text(label)
similarity =
compare(
image_embedding,
text_embeddings
)
RETURN ranked labels
FUNCTION embed(image):
RETURN CLIP.encode_image(image)
This abstraction will be valuable later if you change:
CLIP
↓
OpenCLIP
↓
local multimodal LLM
↓
other embedding model
The rest of the index remains unaffected.
ISEmediaIndexer.py
Responsibility
The Indexer consumes classifier output and creates the searchable index.
Its job is not to run ffmpeg or understand CLIP.
media files
│
▼
Classifier
│
▼
metadata records + embeddings
│
▼
Indexer
│
▼
ISEmedia index
For an initial version I would strongly recommend SQLite.
Not Elasticsearch.
Not a vector database server.
Not another daemon.
SQLite gives you:
one file
portable
backup-friendly
simple SQL
full text search
easy Python integration
Suggested initial schema
media
-----
id
path
filename
media_type
size
modified_time
duration
width
height
classifier_version
indexed_time
labels
------
media_id
label
confidence
embeddings
----------
media_id
model
vector
Depending on vector size, you can initially store the vector externally:
embeddings/
000001.npy
000002.npy
000003.npy
with SQLite containing:
media_id
embedding_path
model
That keeps the initial implementation extremely simple.
Indexer pseudocode
OPEN SQLite database
CREATE tables if necessary
FOR each classifier metadata JSON:
CHECK if media path already indexed
IF existing:
compare:
modification time
file size
classifier version
IF unchanged:
SKIP
INSERT or UPDATE media record
DELETE old labels for media
INSERT new labels
REGISTER embedding location
COMMIT database
The important feature is incremental indexing.
IF file unchanged
do nothing
You absolutely do not want to re-run CLIP across a large media collection unnecessarily.
ISEmediaSearch.py
This should initially provide two completely different search modes.
1. Metadata / keyword search
Example:
ISEmediaSearch.py "city night"
Conceptually:
SELECT media.*
FROM media
JOIN labels
WHERE labels.label IN (...)
2. Semantic CLIP search
Example:
ISEmediaSearch.py --semantic "people partying at night"
Pipeline:
user text
│
▼
CLIP text encoder
│
▼
text embedding
│
▼
compare against stored media embeddings
│
▼
rank by similarity
This is where your existing CLIP infrastructure becomes particularly useful.
Search pseudocode
PARSE command line
DETERMINE search type
IF keyword search:
query SQLite labels
rank results
DISPLAY results
IF semantic search:
LOAD CLIP
text_embedding =
CLIP.encode_text(user_query)
LOAD stored embeddings
FOR each embedding:
similarity =
cosine_similarity(
text_embedding,
media_embedding
)
SORT by similarity
DISPLAY top N results
llama.cpp's role
I would not put llama.cpp in the critical indexing path initially.
Instead, use it as an optional intelligence layer.
For example:
User:
"find videos that look like old nightclub footage"
│
▼
llama.cpp
│
▼
Query interpretation
{
media_type: "video",
concepts: [
"nightclub",
"crowd",
"night",
"indoor"
]
}
│
▼
CLIP semantic search
Or:
User natural language
│
▼
llama.cpp
│
▼
structured search plan
│
├── keyword filters
│
├── metadata filters
│
└── semantic query
That should be ISEmedia v2, not the first functioning version.
Overall architecture
MEDIA LIBRARY
│
▼
ISEmediaClassifier
│
┌────────────┴────────────┐
│ │
▼ ▼
SQLite metadata CLIP vectors
│ │
└────────────┬────────────┘
│
▼
ISEmediaSearch
│
┌───────────┴───────────┐
│ │
▼ ▼
Keyword Search Semantic Search
│ │
└───────────┬───────────┘
│
▼
Results
The minimal implementation sequence
I would implement this in exactly this order:
Phase 1 — Classifier
image → CLIP labels → JSON
Prove that this works.
Phase 2 — Embeddings
image → CLIP embedding → vector file
Prove semantic similarity works.
Phase 3 — Indexer
JSON → SQLite
vector → registered
Phase 4 — Image Search
text → CLIP → similarity search
Phase 5 — Video
video
→ ffprobe
→ sample 5 frames
→ CLIP
→ aggregate
→ index
Phase 6 — Natural-language layer
user query
→ llama.cpp
→ search plan
→ keyword + semantic search
My strongest recommendation
Keep the first version constrained to this contract:
Classifier:
media → metadata + embedding
Indexer:
metadata + embedding → searchable index
Search:
query → ranked media results
That gives you three independently testable programs with clean boundaries. Once that works, we can incrementally define the actual ISEmedia metadata schema and SQLite schema, then turn this pseudocode into the first working scripts without destabilising the underlying architecture.
Home
Help
Search
Login
Register
July 21, 2015, 04:01:15 AM
Need help or a chat ?
