I like the entire premise, the one thing stopping me from trying this is not knowing the time scales that I will need to set my computer aside for the processing of large folders of video frames, or my photos library's videos, some 12,000 videos
Having built something similar with CLIP on an M1, frame sampling rate is the whole ballgame. One frame a second on 12k videos is days, keyframes only got me to an overnight run.
We sit in almost the opposite corner of this design space with the clipping pipeline we work on (VideoToReel - disclosure, it's ours): we don't semantically scan frames at all. One transcription pass over the audio, an LLM reads the whole transcript and picks the standalone moments, cuts land on sentence boundaries, and the video is only decoded around the chosen segments. The "leave the machine alone for how long" cost stays close to the ASR pass, which is nice - but the approach only works when speech carries the content. A purely visual moment is invisible to it, so for an indexed archive like this project is aiming at, per-frame embeddings really are the only way.The expensive vision work we keep is the speaker-tracking crop, and it's much cheaper than search: you're following who's talking, not indexing what's on screen. And scene detection is the right pre-filter even there - for stage talks most of the keepable stretches are single continuous shots, so cutting the candidate set at camera cuts prunes a lot before anything costly runs.
I understand why it did what it did, but I didn't want to spend the time digging in and changing its behavior.
The view options are what killed it in the end.
The thumbnails issue was just me looking at the sausage and like "ugh, I wouldn't have done that."
It's always a tradeoff between time and work. I would have done a sliding window around the current screen, but then you might have placeholders. Maybe it should do low-res first and generated hi-res thumbnails 3 pages out while scrolling. But that's a lot of work for the 10% case.
What I want is aperture back, with all the fun AI stuff and using Affinity for photo editing (which I have anyway). I'll add that to my queue of projects I guess.
In general, I hate it when software is opinionated about my directory structure, or tries to build its own “library” on top of / in addition to my files on disk. More software should be flexible about the user’s existing data and directory structure.
fwiw if you're referring to immich in this comment, it doesn't. mannyv is referring to a different type of on-disk organisation (thumb display caches).
How well do you think this would work on stock photography on m1 mac with 32GB ram? For example I'd like to be able to search a folder of ~2k photos for houses with palm trees. Or find photos of kitchens, or find photos of desert southwest landscapes.
Would require you to be locked in to the one embedding model in the camera though and cameras would need to use the same or be incompatible. Would be fine with a standard model like CLIP but would leave a lot of potential on the table compared to a good way to do your own embedding for everything.
Can you copywrite things like this now that LLMs exist? I mean, up until now if a small startup has a great idea they will get bought out by big tech which will integrate (or kill) their tech. But now with LLMs can the likes of OpenAI just tell their model to make something that works similar to X (such as this project) and then get round copying laws?
Copyrights protect the literal text of your program and binaries. Not the design and functionality. Getting Sherlocked is having your functionality duplicated, independent of the code itself.
No one thinks Apple violated Sherlock copyright. They just made it (nearly) functionally redundant.
I don't see why LLMs should make the legal part different. It's like saying if you can copyright a book considering LLMs now can copy / write a new one in just one hour. Making it faster doesn't change the legal aspect / ownership of something.
yes afaik copyright law has nothing to say about being sherlocked. for now yes go ahead and ask your llm to build you grand thefted autoz and since it wouldn't copy literal code you are good to go. it's happening now - virtually any software you can think of has some shit slop clone out there already
Because you haven't rereleased your own fork in rust yet! The "LLMs can do it" cuts both ways. If that doesn't sound worth your time then it's silly to rudely suggest it is worth someone else's.
I totally agree and I'm guilty of producing some electron crap. I think the ai explosion can be greatly improved by a new easy-for-ai but high performance and consistent ui paradigm. Would love to see a "what would html look like if it was designed for apps instead of documents from the first day" become really popular with the ai models without becoming too hard to read.
One of my deep dark hopes for LLM coding is that it means that nobody ever needs to write an Electron app again. Have a good idea and the ability to plan and structure the code in JS but don’t know a native app language? No problem! Just have the LLM write the Rust or Swift or even C++!
nullsanity got downvoted into oblivion, but they are correct. This is one of the many reasons why vibe coding produces worse software. The code that is generated and the best practice recommendations are completely separate. They both come from a distribution of "most common", and best practice is rarely common. Especially when a practice is first established, or in a specific niche.
Unless you are referencing some existing non vibe coded app, complaints the project being vibe coded may be just too generic at this point.
A hand coded electron project would have a discussion about electron vs native. Relevant in general but off topic in the context of this particular app.
And manager types in software still haven't figured out that prototypes should not be the final product. In engineering it's common knowledge and only rarely do you take a proof of concept directly to production.
Hahaha. It'll say what you bring up is absolutely right egardless of whether it's a better option. These LLMs use a training technique (RLHF) that placates the user and keeps them burning tokens. It's optimizing for human approval, not correctness.
idk about that. The big labs and their data providers are spending millions of dollars building up expert datasets. I was offered $120/hr to critique outputs and provide my own designs. Thats clean and high quality data, not the average internet word distribution
Mostly unrelated but fun thing I discovered earlier this year with Apple's vision - if you have text both correctly oriented and upside down in the same image, it likes to interpret the upside down text as a Cyrillic alphabet. I was trying to use it to read the text on camera lenses and it came up with all sorts of bizarre interpretations. If anyone's interested, I got around it by splitting the text at a point and unrolling it into a straight line before running OCR on it.
I recently tried to recover text from 8 frames of an office-shot YouTube video, where only a small, blurry portion of a computer screen was visible. After spending half a day with Astra on it, the conclusion was that it’s not possible to read.
Later that evening, I just paused the YouTube video on my phone, circled the part of the display with Google Lens, and it read the whole thing with pretty good accuracy. It was mindblowing :)
My own experiments with tesseract were very mixed. I did use just a locally runnable version from a public repository. Text from webpages. Sometimes it would do great but small variations could make it fail completely. Paddleocr on single line text for me has accuracy in the range of 95%
Not OP, but probably because they have no idea what they are doing or what CLIP even is. And the whole thing is likely just from one llm prompt, and then they dumped the whole thing onto GitHub, undoubtedly without even looking at any of the code or architecture.
jumping straight to the right moment in a video is the useful bit here. how often does the balanced sampling miss something that only appears for a second or two?
its a cool project, but I dont want to consume someones ai slop to discern whats true. if the human wrote the page in their own words, I would have considered using it.
otherwise, I can just make my own with my own ai. why consume someones slop when I can eat my own.
hahaha fair enough. If the README being AI-assisted is a dealbreaker, you definitely shouldn't use it.
plz — I bet, make your own with your own AI. Nothing would make me happier than one more "local-first AI photo search" repo. You'll love the part where ffmpeg orphans itself to PID 1 at 500% CPU after "done" and your 90min film indexes only its final 3 minutes.
Mine's over here actually working. Enjoy eating your own slop.
There are some specific point to watch out for when implementing an on-device photo processing pipeline—points I encountered firsthand and that you might want to check.
EXIF Orientation: If you decode an image without applying the orientation information, the resulting embeddings will be calculated based on the rotated image. CLIP is surprisingly sensitive to this, and OCR generally fails to work on text rotated by 90 degrees.
82 comments
[ 2.3 ms ] story [ 6.6 ms ] threadBased on my experience, sampling rate can be tricky if what you are looking for lasted less than interval period.
you are pirating first-release movies for commercial purposes?
(& by "this" I mean an approximate AI search for photos & videos - I can't account for the "every frame", nor for the comparative search quality)
Just for product aesthetics, i believe it created a thumbnail folder with every resolution, which is annoying since i have a multi-decade 8tb library.
There were other actual issues with organization and display. But with this I realized I can just get gemini to write a DAM for me in a weekend.
The view options are what killed it in the end.
The thumbnails issue was just me looking at the sausage and like "ugh, I wouldn't have done that."
It's always a tradeoff between time and work. I would have done a sliding window around the current screen, but then you might have placeholders. Maybe it should do low-res first and generated hi-res thumbnails 3 pages out while scrolling. But that's a lot of work for the 10% case.
What I want is aperture back, with all the fun AI stuff and using Affinity for photo editing (which I have anyway). I'll add that to my queue of projects I guess.
Can you copywrite things like this now that LLMs exist? I mean, up until now if a small startup has a great idea they will get bought out by big tech which will integrate (or kill) their tech. But now with LLMs can the likes of OpenAI just tell their model to make something that works similar to X (such as this project) and then get round copying laws?
1. https://news.ycombinator.com/item?id=34080326
[0]https://en.wikipedia.org/wiki/Copyright
No one thinks Apple violated Sherlock copyright. They just made it (nearly) functionally redundant.
Tall order but would love to get my cpu back!
A hand coded electron project would have a discussion about electron vs native. Relevant in general but off topic in the context of this particular app.
What’s the downside risk of having "worse software" when you’re just ideating and putting things out there to see how people like it.
- release vibe coded tool
- on HN, someone says, d'oh, you should be using Y, not X
- give that comment to your LLM
- LLM: "The commenter on Hacker News is absolutely right, my mistake. I can easily replace X with Y if you want"
- release a new, better version of vibe coded tool
Iteration, in one afternoon.
Later that evening, I just paused the YouTube video on my phone, circled the part of the display with Google Lens, and it read the whole thing with pretty good accuracy. It was mindblowing :)
It is a tool from a bygone era.
And then you end up with image metadata processing code like this, which just by a cursory glance I’m sure has edge case bugs : https://github.com/allenv0/SCM/blob/main/screenshot-probe.js
otherwise, I can just make my own with my own ai. why consume someones slop when I can eat my own.
plz — I bet, make your own with your own AI. Nothing would make me happier than one more "local-first AI photo search" repo. You'll love the part where ffmpeg orphans itself to PID 1 at 500% CPU after "done" and your 90min film indexes only its final 3 minutes.
Mine's over here actually working. Enjoy eating your own slop.
isnt it expensive tho?
EXIF Orientation: If you decode an image without applying the orientation information, the resulting embeddings will be calculated based on the rotated image. CLIP is surprisingly sensitive to this, and OCR generally fails to work on text rotated by 90 degrees.