HN Explorer

Show HN: Rem: Remember Everything (open source) (github.com)

555 points by jasonjmcghee ↗ HN
An open source approach to locally record everything you view on your Apple Silicon computer.

Note: Relies on Apple Silicon, and configured to only produce Apple Silicon builds.

I think the idea of recording everything you see has the potential to change how we interact with our computers, and believe it should be open source.

Also, from a privacy / security perspective, this is like... pretty scary stuff, and I want the code open so we know for certain that nothing is leaving your laptop. Even logging to Sentry has the potential to leak private info.

199 comments

[ 3.1 ms ] story [ 221 ms ] thread
xydac
cool concept, love the idea. Might be fun to integrate with local llama to get most privacy
jasonjmcghee
100% and local embeddings. This is the area i want to explore next.

The demo i showed with chatgpt works just as well with openhermes2.5-mistral. But is instant with chatgpt instead of 20s

ilumanty
How similar is this to rewind.ai (https://www.rewind.ai)?
jasonjmcghee
I only used rewind at alpha, so not sure how much they’ve added, but it has the value i got out of it, and doesn’t limit your searches arbitrarily.

- takes screenshots every two seconds - records all the text via ocr - builds full text search with sqlite - allows you to go back in time however far and select/copy text from there

No meeting recording / audio recognition. Kinda irks me. Easy to add though.

pmarreck
I never heard of this until now but this looks amazing

Would be even more amazing with a locally running LLM

lnxg33k1
I feel like recording everything is like recording nothing in practical terms
conception
Think of it as closed circuit tv for your computer. You don’t need to watch 24:7 but you can go back for specific incidents/information.
lnxg33k1
Yeah I understand that, it seems that it tries to classify activity in order to help finding relevant stuff seeing

  let configuration = ImageAnalyzer.Configuration([.text])
                    let nsImage = NSImage(cgImage: image, size: NSSize(width: image.width, height: image.height))
                    let analysis = try await ImageAnalyzer().analyze(nsImage, orientation: CGImagePropertyOrientation.up, configuration: configuration)
                    let textToAssociate = analysis.transcript
                    let newClipboardText = ClipboardManager.shared.getClipboardIfChanged() ?? ""
qup
It lets you query any data once you realize what is important (which might vary depending on the question you're trying to answer).

It's like law enforcement tracking everything we say. They aren't catching many people right now, but wait until the future when they start working backwards with logs.

steve1977
And when things are illegal which weren’t illegal when they were said.
nextaccountic
i record every command in .zsh_history (like everybody else does by default, but mine is configured to not have a size limit)

i often do things like

history | rg ..

it helps when you roughly know what you want to find, but want to check some detail you forgot

dotty-
For those unaware: CTRL+R in terminal will also change your prompt to search your command history. After typing, CTRL+R again to cycle through matches.
dmd
what you really want though is fzf with C-r
mraza007
Would agree 100%

fzf supercharges your shell history I can’t imagine my life without it since I spend most of my day in terminal

howmayiannoyyou
Comments:

- Insanely useful with some changes.

- Needs local llama support for privacy.

- Needs pause/record functionality, ideally w/ preset exclusions, again privacy.

- If this could evaluate in real time at some point and start intelligently adding value at that point it has the chance to change things.

My guess is that in 10 years this will seem absolutely archaic. Now, it feels a bit like magic.

jasonjmcghee
Thanks for the feedback! You can start / stop remembering whenever you want.

As far as real time stuff and local llama- absolutely, on the roadmap.

I’ve been exploring / experimenting with embedding spaces and local models a lot.

throwaway295729
Are there instructions on how to launch the app? I’m able to clone the repo but a bit lost on next steps
jasonjmcghee
Open in XCode and create an archive or run it directly.

Or you can ise the release i uploaded.

I added instructions for how to use it once it’s open in readme.

Apologies for anything unclear!

nmjohn
> Note: Relies on Apple Silicon, and configured to only produce Apple Silicon builds.

Just curious, what is relying on apple silicon?

jasonjmcghee
Full disclosure, I haven’t tested it on Intel, but I don’t think it will not be able to keep up with taking screenshots, generating ffmpeg videos, and doing OCR that often and will drain your battery very quickly.

But if you / someone can get it to be efficient enough, awesome!

kevincox
I think you underestimate computers. Taking 2fps screen recordings is a trivial task. Doing OCR may be slightly more work but at 2fps I doubt it is an issue. Worse case you could tune the OCR frequency based on the computer's abilities.
frfl
You're confusing 2fps with 1-screenshot-every-2-seconds (or 0.5fps), what the README actually says).

I wouldn't be surprised if the battery issue is problematic, likely will result in at least some kind of battery life reduction, but perhaps not 30 or 50% at 0.5fps.

I haven't looked into the code, but if you're running ffmpeg, then battery life will likely take a hit depending on what exactly you're doing. Video encoding _can be_ heavy on the CPU/GPU.

kevincox
That makes it even less work. Running ffmpeg is just video encoding, I don't think a 0.5fps video would be a huge issue.

Lots of people work plugged in most of the time. I don't see why one would want to gatekeep to keep them from using it.

beardedwizard
What gate keeping? I just see a valid correction to your misstatement and your reaction reads like a defensive Karen wrote it.
kevincox
Not supporting a platform just because it may cause it may cause battery drain which may not even matter to plugged in users seems like gatekeeping.
0xDEADFED5
it's literally an open source MIT licensed hobby project. fork it and improve it and share here. complaining about it is kinda rude.
jasonjmcghee
I don’t have an Intel Mac to test on- but you can absolutely just clone it and swap the config to Intel
frfl
I have to agree. If you're interested in supporting Intel(x86/64), it's open source, and you sound like you have the hardware to add support for and test on Intel.
beardedwizard
Not supporting? The commenter simply said it may cause battery drain. It is a discussion on the topic (both sides based purely on conjecture), and a relevant one. You disagreeing does not mean others are "gate keeping". Stop trying to weaponize trendy language and white knight this thread.
kevincox
The original README was claiming that relies on Apple Silicon and that they have configured builds to exclude other Apple platforms. I see it has been greatly softened now to "Only tested on Apple Silicon, and the release is Apple Silicon" which I think is quite reasonable.

I have no problem with not supporting a platform because you have no interest or any other reason, but previously it was quite proud to not support it which is different.

beardedwizard
Ridiculous. You are working very hard to be offended.
[deleted]
(comment deleted)
philipswood
I haven't looked this codebase yet, but a screenshot every few seconds isn't a noticeable slowdown on most machines.

At such slow rates you don't need to create video - you just keep the individual images.

OCR doesn't need to be real-time, but can be done in batch mode or when the machine is idle.

v3ss0n
I had been doing that with opensource Linksys ip cameras since 2010 and they only have like 180mhz and 32MB RAM. What are you thinking about?
pedalpete
This is also what I was wondering. The demo is showing recording a web-browser, and I'm wondering if that is all it is doing. If so, wouldn't that mean creating a browser plug-in would make this possible on any platform?

I also don't understand the chatGPT component, and what it is trying to tell him. Though I'm sure if you just threw the URL and the screenshot to chatGPT, you could ask it questions about that source.

I'm not sure how useful this is tbh, or how I would use it. I'm not saying it isn't useful, just that I'm not sure how I would use it, or why it is useful.

Bombinator
> The demo is showing recording a web-browser He said it's not recording but taking a screenshot every 2 seconds and I assume it's not just for a browser but all text on the desktop.

> I also don't understand the chatGPT component You give it context from the "recording" and it answers questions you give it with that context info.

majikandy
Loving this concept perhaps really useful for my work laptop (as in my own one but is only for work stuff) as quite often you just want to quickly backtrack and find that piece of info you looked at earlier rather than navigate to it again. I’d imagine something like a physical wheel on your desk to wind back would be amazing. I have a useless Bose one that never gets used, can imagine it would feel very “black mirror” to use that to rewind.
jasonjmcghee
I love it. The touchpad feels pretty good, but a wheel would be incredible.

I debounce the livetext analysis on history so you should be able to spin fast without issue

swah
Rewind has search - it just works better than rewinding manually.
majikandy
Can also see this being used by scammers/malware. Not saying it shouldn’t exist. It’s really cool. Just scary. Great job.
jasonjmcghee
Fwiw requires you to explicitly give it permission to record your screen. Would also require you to explicitly give it permission to use network if it needed to make any requests.

I’m super glad about this personally.

unnitle
I think op is referring to a similar attack vector used in the recently presented “triangulation exploit” wherein attackers used iOS’ stored data from its own local machine learning engine that classifies photos using object recognition and stores text from images with ocr to prioritise which photos from a victims phone had content of interest for them.

Seems a legitimate concern; unsure why op is receiving negative attention for saying so.

majikandy
Precisely, although I’ll come clean and didn’t know about that exact triangulation exploit mentioned, just the fact that nicely organised historical data of all actions is there somewhere to be found. Especially if this type of software starts getting all the modern ai magic on top, this could be used by everyone as standard tooling and perhaps be a target.
albertzeyer
Long time ago, I did sth similar, i.e. made a screenshot every few seconds, with the purpose to automatically extract information from it, e.g. how long I was using some app.

I wrote a PNG DB to split PNG images into many blocks and have each block stored in a DB. If there are several equal blocks, it is only stored once. Via a hash table, the lookup for such blocks is made fast. With this PNG DB, I have a compression rate of about 400-500%. https://github.com/albertz/png-db

Some of the scripts I used to analyze the screenshots are here, but in the end, it was not really so successful and reliable: https://github.com/albertz/screenshooting

In the end, that lead to another project, where I just was storing that information more directly, i.e. what application was in the foreground, what file was open. https://github.com/albertz/timecapture

jasonjmcghee
I used ffmpeg to try to do smart compression for me (diffing etc)- but run OCR first. Also did a poor man’s text merging to try to make use of the overlap from scrolling
philipswood
What OCR did you use?

Tesseract?

What was the performance (of the OCR) like?

kaetemi
On Windows I use a small program that grabs a frame every second through the desktop API as a DirectX texture, and compresses that straight on the GPU to h265 using AMF. I'll upload the source in case it's interesting for anyone else.
[deleted]
(comment deleted)
WirelessGigabit
I would love this!
kaetemi
Alright, here you go.

https://github.com/kaetemi/second_capture/blob/master/second...

jdthedisciple
why does CPP code always look so messy and unintelligible?

like if I see C# or Python it makes sense to me at least in some way

whereas CPP code always looks like it's powering some rocket engine?

Also thanks for sharing!

jdrek1
Well first of all, C++ is the language you'd be using to power a rocket engine. And second, that code is a terrible example because most of it isn't C++. Large parts of that are very C like or directly C because it's using the Windows API.
kaetemi
Largely because it's a melting pot of ancient and modern coding standards. Got the C Win32 API along COM style and then whatever AMF is doing. Makes things very verbose and explicit.

I've seen worse Python.

Personally, I think it's charming. :)

diggan
> like if I see C# or Python it makes sense to me at least in some way

Could it be that you're just more used to looking at C#/Python than other things, then other things are more foreign and therefor look messy?

As another anecdote, I cannot stand browsing/looking through C# code as it tends to be filled with various classes just to basically write very basic programs. The amount of over-engineering I've seen in C# surpasses everything else I've looked at. Not to mention how people seem to arbitrary chose between private/public with no real consensus on when to use what, everything seems to be encapsulated in the wrong way. And don't get me started on the infrastructure around it, csproj vs sln and dealing with dependencies/assemblies.

But then I mostly write Clojure code day-to-day, and I realize that my troubles for dealing with C# is mostly because of what I'm used to, not because the language itself is inherently shit. I only have myself to blame for this. I'm sure people who write C# day-to-day have the same feelings about Clojure as I have about C#.

theropost
Thanks, I am giving it a try - any dependency required for Windows 10? Compiles fine, but get an error about AVIFileInit - maybe to do with <vfw.h>?
vitovito
AMF is AMD's equivalent to NVENC. I think if you aren't running an AMD card, you won't have the necessary libraries?
theropost
Ah yes, that would be it. Thanks! I threwa quick C++ program together to do something similar using FFMPEG.
kaetemi
Yea, I haven't really needed an NVENC version yet. Will probably add it... eventually... for my laptop. Shouldn't be too any more difficult than just swapping out the right parts, though.
Sakos
Thanks for sharing. It's a very cool idea. I briefly tried using it but I don't have an AMD card (just Intel and NVIDIA). How difficult do you think it might be for me to implement support for something like NVENC instead? I only skimmed the code and I'm not sure yet exactly what the AMF code is doing.
eek2121
As much as I dislike the current AI hype, a local on-machine AI model that can read/interpret videos/thousands of images (basically a recording of screen time combined with video/audio/handwriting of my everyday life), store it in an indexed format, and project it back to me in an easy to understand/quickly digestible format would be a godsend I'd invest a lot of money into (provided false positives were close to zero)
jasonjmcghee
I would love this project to serve that need and personally want this to.
dlivingston
Absolutely. Combine it with real-time analysis of your current screen, and you've got a computer that knows the complete history of what you're doing and why. That kind of global analysis could be really useful.
toomuchtodo
To me, it seems obvious Apple eventually builds this into MacOS (“it’s a feature not a product”). This is like local apps or native OS features that would index your drive contents and provide a frontend to query, but on steroids. This also gets us closer to transparent computing.
aragonite
Used to do this a several years back but on a windows machine and without any of the AI stuff obviously. One use case I found is for tracking down unpredictable and seemingly randomly occurring bugs since you can rewatch the events leading up to the bug and form better hypotheses about what might reproduce it.

Eventually I had to stop because the fan was going crazy, plus I couldn't bear seeing how slow and error-prone I was at typing and at generally operating the computer (it never felt that way when I'm using the computer, but watching myself using it is a different story)

orliesaurus
I am building this exact software for exploratory bug-testing. What have you been replacing it since your last usage on Windows?? I think I tweaked the recording aspect to be super-clean and CPU/memory impact is minimal now (1%)
aragonite
Haven't found any replacement, but dashcam.io looks very promising for that use case, will definitely be checking it out!
55555
Another obvious option is to just access the browser’s History file and request and store the contents of each visited page. This prevents you from needing to do OCR and is more highly compressible. Or do your method, but throw away the screenshots after AI analyzes and OCRs them. BTW, Mistral 7B is good enough! We don’t need to rely on ChatGPT4 IMO and copy pasting context is a bit sloppy.
jasonjmcghee
Yeah that works well for browser stuff, but this works with IDEs etc too

and totally. Haven’t added direct local interaction yet, but on the roadmap.

thomasfromcdnjs
I wanted to build a similar tool that just relied on browser history. But I couldn't figure out anyway to do it (especially not through browser extensions)

If anyone has any suggestions, I'd be more than grateful.

tiborsaas
Interesting concept, however I don't get what information is pasted into the context. Also, ChatGPT's context is kinda limited, I can probably remember the recent context, what I have problem with is context from let's say a week ago which would probably be way over the LLM's context window.
jasonjmcghee
Admittedly, it might have been a mistake as a demo / feature, but haven’t built embedding support yet. Working on it!
pmarreck
Does it do inter-frame compression at all?

Also, integrating with Ollama.ai or some other local LLM with an API server would be fantastic.

jasonjmcghee
I’d love your opinion on the right way to do this! Being able to call APIs means network permissions- which i was trying to avoid. Maybe via Stdin / Stdout?
pmarreck
were you trying to avoid network permissions (I'm guessing) because this is Docker? (That's the only reason off the top of my head for wanting to avoid network access... in a non-Docker context, localhost is of course easy to hit up, but Docker and nets are a PITA)
bwghughes-fth
Jason - great work here. Your Swift code looks like mine :) on this, some folks in the UK have created Crux - an interesting abstraction layer for mobile apps using Rust. Might provide some ideas for optimisation/ipc. https://github.com/redbadger/crux
tremarley
Don’t show the VCs that invested $27.9M into https://rewind.ai this

They will be very upset

echelon
a16z deploys capital fast into AI companies. They've already funded several companies running off the shelf open source models.

Find the latest flashy thing on Twitter / GitHub, spin it up with a waitlist, then send a16z your deck.

CPLX
Guess it’s an improvement over deploying it to sociopathic felons which was their last claim to fame.
woah
Rewind.ai looks a lot more full featured (unless their site is complete BS and none of it works yet). Doesn't matter though because Apple will rebuild this themselves in 2-5 years with an on-device LLM chip that you will have to buy new hardware to get and it will be way more efficient and with way better privacy.
Oras
From the repo, OP did this in couple of days with no experience in swift. So getting to rewind stage is not that hard it seems
hruzgar
for now it's easy to catch up. But after a few months they will be so far in the sky from all the vc money that it will be tremendously harder. Like the M1 chip for example
beoberha
This has all the makings of the original “Dropbox is just rsync” comment.
tremarley
Similar to the “Loom is just OBS with Dropbox on top”
user_7832
Yes but in this case both apps are relatively new and not established ones if I’m not mistaken
ct520
Works well. Been using it since beta. I got a memory like a gold fish and this comes in handy.
congnd
Pretty interesting stuff.

I'm just wondering how you manage the limitation of context length.

jasonjmcghee
For the "copy recent context"?

The last 15 frames.

It's a terrible approach! But I had to start somewhere. Actively experimenting with properly leveraging embedding search.

But I've had a hard time finding CPU + RAM efficient vector indexing + search that meets my expectations. Been doing a lot of personal experimentation and research in this space.

Is there a known approach to be able to maintain a large embedding space that you can insert into efficiently and search accurately without needing to load / maintain the entire thing into memory?

throwup238
Have you tried using the Accessibility API instead of (or alongside) taking screenshots? It wont work with all apps but you can fall back to OCR when it doesn’t and best of all you can monitor the “DOM” for changes.
jasonjmcghee
Candidly, I don't know how to do this effectively, especially with browsers. I looked into this approach using the notification pattern, but I just couldn't see a good way to do it. I'm no expert in Mac APIs and would love to learn and / or see any specific approaches you have in mind!
NhanH
Serious question: I have a serious case of OCD where I keep trying to remember things verbatim (the verbatim part is the OCD). Naturally there are a bunch of checking and repeating in trying to do so.

I have been considering the idea of using a similar app to this (or rewind.ai), but I have the concern that it might aggravate my situation. Just imaging my checking self watching 12 hours of video footage already gave me chill.

I would appreciate if anyone with a related or similar situation can share their experience using those apps. Since this is fairly sensitive, my email is also in the profile if anyone want to contact me directly.

steve1977
You’ll end up with inception levels of watching yourself watching yourself…
ayewo
There are folks who mention that Rewind.ai has be invaluable for managing their ADHD on their Slack community. Perhaps if you join their Slack [1], you might be able to meet people in a similar situation as yours (with your type of OCD)?

1: https://rewind.ai/community

lexh
“pretty scary stuff” indeed!

This would inevitably end up ingesting secrets, right? Like say from my password manager? Or API keys in my terminal?

Lots of ways for this to go sideways even if the data stays local.

What’s the plan there?

jasonjmcghee
Come together as a community and help build the right thing. This isn’t the first implementation and I don’t have a fiduciary duty to create value to investors.
cjbprime
> Lots of ways for this to go sideways even if the data stays local.

Could you name some?

Bombinator
> Like say from my password manager? Or API keys in my terminal?
cjbprime
That's not describing a bad outcome, it's describing how the tool works.
Bombinator
Oh, well I think what he meant is that some malicious program could read and transmit this unencrypted recorded data which is normally stored in an encrypted form
cjbprime
Thanks, I think so too, but the threat model is a bit odd. On a Mac, potentially malicious programs do not normally have access to files in every location (e.g. the prompts to allow a process to access your Documents dir); there is hardware-backed crypto available for further protections; full disk encryption; and so on. It's unclear to me how to evaluate the severity of the risk.

Every security decision is a risk-reward tradeoff, and the reward of a complete memory of computing tasks seems pretty huge.

lexh
The impression I was left with is that this tool would write things to disk. It would be helpful to know how that data is stored. I wouldn’t want my password manager OCR’d and then sitting in plain text on disk for example.
koopuluri
Congrats on getting this off the ground, and thank you for putting it out there for us to learn from!

I've been curious how Rewind worked under the hood because I've been playing with an idea in my head: an AI assistant that helps you protect your attention.

You would describe the kind of content that you consider a distraction, and any other constraints you have (e.g. "Don't let me watch cat videos unless I'm on a break".

And whenever it sees you watching anything that fits your prompt, it'll pop up on the screen and start a conversation with you to try and understand whether you actually need to consume the content you're looking at.

An AI that intervenes when you're going off track (based purely on how YOU define going off track). Current website blocking approaches aren't useful because they're all-or-nothing. I don't ever want to block entire sites because often there's useful content there relevant for my work. I want to block content on a much more granular level.

And I'd love for an "attention audit" at the end of each day. Attention is our most valuable asset, and I believe protecting it is a worthwhile endeavor... I'd just like some help doing so :).

baby_souffle
Oh this seems like a wonderful idea. Loads of invasive privacy issues if you’re not doing the detection locally but I’d absolutely use something like this
koopuluri
Thanks! I agree that everything needs to happen locally, and I believe it's possible.

I'd love to better understand the problems you're facing that makes you want to use a tool like this.

Couldn't find your email, but if you're interested in chatting, you can find mine in my bio. Would appreciate it!

jasonjmcghee
I encourage you to fork this repo and build it.

Might be worth checking out Ollama and bakllava. https://ollama.ai/library/bakllava

Maybe the model is a bit too slow, but I'm sure smaller ones will come out soon. You can likely fine tune to do exactly what you need.

koopuluri
Thanks for the share! Will check it out.
shkurski_
Cool stuff. Interesting to see how these ideas evolve, now with LLMs. I made the similar thing some time ago (>2yrs): https://shkur.ski/chronocatch/ for Mac/Win (Intel, H264 for interframe compression and BM25-ranked search). Then the war started and I regret not sharing this back in time "as is" when I could.
great_psy
Maybe in 5 years apple with release a native version of this.
alwillis
It's not going to take 5 years… seems like a natural addition to the existing Time Machine feature.
keepamovin
This does look cool. It reminds me of a recent discovery I made. The other day, while trying to recover some disk space, I found a giant file on my hard disk. It turned out to be a nine-hour screen recording from almost a year ago. I had no idea it existed, so I must’ve accidentally left the screen recording on. Scrubbing through it sped up, watching the whole thing in a couple minutes, was fascinating; it was like a window into my thought process at that time. You could see how I was researching something online. It was almost like a play-by-play, akin to re-watching a sports performance – very instructive and surprisingly useful.

Also, the the sense of being back in that time seeing details that I otherwise probably would’ve forgotten was transformative.

In a similar vein to what you’ve done, but focusing specifically on web browsing, I’ve created a tool called ‘DownloadNet.’ It archives for offline use and fully indexes every page you visit. Additionally, it can be configured to archive only the pages you bookmark, offering another mode of operation. It’s an open-source tool, so feel free to check it out: https://github.com/dosyago/DownloadNet

travisjungroth
This sounds a bit obvious to me after I write it down: I think there’s some value in the fact you were unaware and it was a random time.

If you take your work very seriously, I can see it being valuable to record it like athletes do. It would be tempting to use this on the “most important” days or when you’re “really ready”. At the very least, there’s a burden of choice and memory. I don’t know about security implications, but it seems valuable to randomly record a day per month and send it to yourself a week later. Or in the case of this tool, select some period for extra review.

philipswood
There's a windows tool called Timesnapper that takes a screenshot every few seconds and let's you replay and navigate.

After reviewing a few days I learned to start focusing on one thing at a time.

It was cringeworthy to see how ineffective multitasking by switching between a few tasks was.

LeonB
Absolutely. Watching your playback in TimeSnapper gives a lot of insights into the way a small distraction can derail you for hours (or it does for me, I mean)
keepamovin
It's amazing to me the kind of vulnerable personal responsibility and insights that occur prompted by simply seeing yourself and how you act, clearly. I heartily concur with the above comments and am super happy to see other people having this similar experience.

It suggests these kind of "mirroring" self-training practices and feedback might be useful across a whole range of endeavors, which sounds awesome. A super easy way to improve -- akin to people checking their reflection in a mirror -- that a bit of technology could really help with :)

skinkestek
When allowed I use a tool called Manic Time that (in the paid version) does this.

It used to be "local by default" but now I think that might be changing to "local if you want".

They have also in the past been a perfect creator of commercial software as far as I know:

- generous free edition

- paid versions work forever with its current feature set

I typically set it to auto delete after 14 days and disallow screenshots from my ordinary browser (because meetings and passwords), Slack and Teams (meetings) etc.

bradrn
> It archives for offline use and fully indexes every page you visit.

Oh, I also made a tool to do this! Never open-sourced, since it’s an utter pain to set up and the UX is terrible, but amazingly useful all the same.

Incidentally: how does DownloadNet work? My tool uses a browser extension to send the full-text of each webpage to a server, but yours doesn’t seem to have a corresponding extension, so I can’t see how it would retrieve the text.

keepamovin
Ah, good, let me introduce you to the wonderful world of the Chrome Devtools Protocol! (fka Chrome Remote Debugging Protocol)

I love this API for almost everything browser related. I built my RBI product atop this (BrowserBox: https://dosyago.com), and I think it's a drastically underrated API.

Also, it works out of the box in Edge, Brave, Chromium, and many parts of CRDP are supported by Firefox and Safari^1

1: See for example: https://github.com/WebKit/webkit/tree/main/Source/JavaScript...

bradrn
Very interesting, thanks! I’d better add this to my list of things to look into…
leandot
I opened the page of BrowserBox but didn’t understand what it does. Can you provide an example of a real-world use case?
keepamovin
thanks for checking it out, sure: some of the ways people are using it follow below.

- reverse-proxy to protect proprietary code on your website from being inspected

- content accelerator (similar to mightyapp's original idea) where it's faster to render pages on a cloud vps with thick bandwidth than it is on a local device (in some cases at least!), and depending on the usage profile, it's even cheaper to serve that bandwidth, especially if you use additional video codecs.

- a framework to deliver web data collection and automation, agent authoring and intervention tooling on any device with no download

- cors-proxy to include and access content across domains for building design and test tools saas

- co-browsing for customer training and demonstration

and then there's the many cybersecurity and privacy ways including:

- standard remote browser isolation to isolate your device from zero day threats (an extra couple layers, requiring an even longer exploit chain, of protection, at least)

- to aid compliance and privacy by preventing insider data exfiltration in both directions when dealing with sensitive data (by blocking file transfer, copy paste, etc)

admittedly it's diverse, and hard make generalizations about customers.

one way i think is cool that i haven't seen yet (but want to get around to doing myself!) is a way to deliver "browser extensions" without needing either: 1) a compatible browser on your device, 2) the extension to come from a store, 3) any local download. In some sense it's safer as the extension does not run locally, but in other ways it's more dangerous as there's no central store. But it's very cool to explore, and what we really need for that is a great "developer API" that can expose a "browser extensions"-like layer.

One cool thing is that ad-blocking extensions built on BrowserBox will not be limited by the current restrictions that extensions developers face on existing platforms. The aim is to provide a powerful instrumentation api as simply as possible.

thank you for your question :)

btw - 42matters looks great! love your site design, really fantastic look. analytics is surely lucrative, i knew a similarly focused company also bearing the name 42 i'm sure. somewhere before anyway (but surely it had a different origin!). is 42 indicative of something special in analytics?

corndoge
Archivebox and its companion browser plugin can also accomplish the capability of archiving everything you visit and may be of interest https://archivebox.io/
chokma
DownloadNet reminds me of how I got really started with Perl programming over 20 years ago. Since I was using my parent's land line with a dial-up modem (which cost cents/minute), I wanted to speed up the process of looking for a job via the government's official job search site.

Turns out, on my slow computer it was faster to clean up a megabyte of HTML with regular expressions before giving it to Firefox than just rendering it as-is - by about 30 seconds per search result page.

Perhaps it's possible to sanitize often visited websites with DownloadNet? (currently getting aggravated by reddit hiding images via JS code to prevent download / viewing in another tab...)

paleface
> Perhaps it's possible to sanitize often visited websites with DownloadNet? (currently getting aggravated by reddit hiding images via JS code to prevent download / viewing in another tab...)

Many years ago, I remember using a utility called: privoxy, on Linux/Unix, for that very purpose. No idea if it’s still viable, but thought I’d mention it, in case you’re serious?!

keepamovin
That's a fascinating idea. I like the idea of "custom user script extensions" that folks can plug in, author and share.

If you're passionate enough you could contribute a write-up, some code sketch or even a full PR of how this works. I'm sure you're probably too busy for that, or just not interested, and that's OK. I really appreciate the contribution you've already made with this idea.

I think allowing folks to filter, or sanitize (for whatever purpose really, sanity, focus, etc), sounds very useful.

Thanks! :)

genewitch
I installed this, and nothing happened. no start menu entry, no taskbar icon, nothing. According to powershell, it put stuff in %userprofile%\.config, but that's it.

Where did everything else go? This ran and disappeared like a miner.

the localhost:22120 didn't load anything.

broscillator
Did you find significant changes and blindspots in your past behavior? I recently had a similar feeling when coming across old journals unexpectedly.
orliesaurus
This is very cool, I am building a tool [1] to record 1H of screen at a time (to help developers debug errors while doing exploratory testing) and I always thought that I could add a layer to turn my 1-hours-brain-recording into a baby Rewind.

I have tried Rewind in alpha/beta, it was cool, but it was never something I felt like I needed. That being said things change, and maybe I'll change my mind when it's part of the OS in a seamless way, but it's sketchy for as long as it's not offline: let alone the privacy consequences of running Rewind ;)

[1] https://dashcam.io