HN Explorer

Show HN: Open-source macOS AI copilot using vision and voice (github.com)

430 points by ralfelfving ↗ HN
Heeey! I built a macOS copilot that has been useful to me, so I open sourced it in case others would find it useful too.

It's pretty simple:

- Use a keyboard shortcut to take a screenshot of your active macOS window and start recording the microphone.

- Speak your question, then press the keyboard shortcut again to send your question + screenshot off to OpenAI Vision

- The Vision response is presented in-context/overlayed over the active window, and spoken to you as audio.

- The app keeps running in the background, only taking a screenshot/listening when activated by keyboard shortcut.

It's built with NodeJS/Electron, and uses OpenAI Whisper, Vision and TTS APIs under the hood (BYO API key).

There's a simple demo and a longer walk-through in the GH readme https://github.com/elfvingralf/macOSpilot-ai-assistant, and I also posted a different demo on Twitter: https://twitter.com/ralfelfving/status/1732044723630805212

185 comments

[ 0.70 ms ] story [ 240 ms ] thread
[deleted]
(comment deleted)
[dead] gmt2027
[flagged]
artursapek
Every time I see someone making that face I wanna stick a slimy cold hotdog in their mouth
ralfelfving
That's ok, I'm with you -- it's more of a joke with friends that developed after I started making tutorials. I primarily use a normal face expressions, and they said I should use one with open mouth -- so I did for funsies, and been doing it a few times over the months although I feel it's a bit cringe.
ralfelfving
There, changed it.
xNeil
hey - i know there's quite some negativity in this thread, but i just wanted to let you know that you don't have to change something if you don't want to

if that specific facial expression is getting you more views, i see that as justified. it's up to you whether or not you want to keep the thumbnail or not

ralfelfving
I appreciate the comment, I really do! Fwiw I play around with thumbnails and titles quite a bit to try to learn what seem to work, and I think the underlying sentiment in the comment rings true for most in my audience. Aaand, it may be a coincidence but the views just hockey-sticked when I changed it :D
lolinder
It depends on which audience they care about more. Do they want the HN crowd—old and cynical curmudgeons who dislike most of what is trending on social media—or the peers who cause such things to trend? If they decide they want the HN types as an audience, listening to their feedback is a good start.
[deleted]
(comment deleted)
[dead] IMTDb
[flagged]
ralfelfving
Apparently we aren't, so I changed it :D
[deleted]
(comment deleted)
faceless3
Wrote some similar scripts for my Linux setup, that I bind with XFCE keyboard shortcuts:

https://github.com/samoylenkodmitry/Linux-AI-Assistant-scrip...

F1 - ask ChatGPT API about current clipboard content F5 - same, but opens editor before asking num+ - starts/stops recording microphone, then passes to Whisper (locally installed), copies to clipboard

I find myself rarely using them however.

[dead] DeathArrow
[flagged]
ralfelfving
We can, if someone builds it. :)

I'm new to development, and this is what I went with. Don't know any better.

freedomben
You made an open source tool and shared it with the world. Use whatever you want and don't feel bad about it. GP can port it to whatever native framework they want
Turing_Machine
Sure we can. Write them.

Oh, you want someone else to write them. For free.

That's different, then. Carry on!

ProfessorZoom
e-e-e-electron... for this..
ralfelfving
I don't know man. I'm new to development, it's what I chose, probably don't know any better. Tell me what you would have chosen instead?
xNeil
electron's a really nice option, specially for people that aren't interested in porting their apps or spending too much time on development

this is a macOS specific app it seems - if you want better performance and more integration with the OS, i'd recommend using swift

ralfelfving
Time to learn learn Swift in the next project then! Thank you for the deets.
Filligree
The good news is you already have a tool to help you with inevitable XCode issues. grin
lolinder
Don't mind them—there's a certain subset of HN that is upset that web tech has taken over the world. There are some legitimate gripes about the performance of some electron apps, but with some people those have turned into compulsive shallow dismissals of any web app that they believe could have been native.

There's nothing wrong with using web tech to build things! It's often easier, the documentation is more comprehensive, and if you ever wanted to make it cross-platform election makes it trivial.

If you were working for a company it might be worth considering the trade-offs—do you need to support Macs with less RAM?—but for a side project that's for yourself and maybe some friends, just do what works for you!

ralfelfving
Thank you for the explanation! At the end of the day, I'm a newbie and I'm in it to learn something new with each project. Next time I'll probably try my hand at a different framework.
millzlane
I just watched a video about building a startup. One of the key points was to use what you know to get an MVP. Don't fret over which language or library to use (unless the goal is to learn a new framework). Just get building. I may not be a pro dev, but there is one thing I have learned over the years from hanging out amongst all of you. And that is, it doesnt matter if you are using emacs or vim, tabs vs spaces, or Java vs Python, the end product after all is what matters at the end of the day. Code can always be refactored.

Good luck in your development journey.

jdamon96
ignore the naysayers; nice job building out your idea
ralfelfving
Thank you! I got pretty thick skin, but always a bit of insecurity involved in doing something the first time -- first public GH repo and Show HN :D
airstrike
I think the parent comment is a shallow dismissal, but since you're asking, I would have built in SwiftUI
guytv
What's important is to get an product out there. Nobody cares what stack you use. just us geeks. don't get discouraged. you did well :)
programmarchy
My two cents: I think you made a good, practical choice. If you're happy with Electron, I'd say stick with it, especially if you have cross-platform plans in the future.

If you want to niche down into a more macOS specific app, you could learn AppKit and SwiftUI and build a fully native macOS app.

If you want to stay cross-platform, but you're not happy with Electron, then it might be worth checking out Tauri. It provides a JavaScript-based API to display native UI components, but without packaging a V8 runtime with your app bundle. Instead, it uses a native JavaScript host e.g. on macOS it uses WebKit, so it significantly reduces the download size of your app.

In terms of developing this into a product, on one hand it seems like deep integration with the host OS is the best way to build a "moat", but then again, Apple could release their own version and quickly blow a product like that out of the water.

atraac
Ah yes, cause what's better than building a real, working MVP? Learning Rust for half a year just so you can 'optimize' the f out of an app that does two REST calls.
wtallis
To be fair, this does sound like the kind of app that would benefit from being able to launch instantly, and potentially registering with the OS as a service in a way that cross-platform frameworks like Electron cannot easily accommodate. But Rust would not be the easiest choice to avoid those limitations.
[deleted]
(comment deleted)
[dead] mwmercury
[flagged]
kolinko
According to your definition no program that connects using rest to a webservice is open source. That is absurd.

Open Source is defined by a license, not by what a program does. Also, it's trivial to connect such a program to another image recognition model if anyone wants.

cqqxo4zV46cp
No. That is a tone-deaf and disingenuous interpretation. If I read “open-source AI copilot”, I pretty strictly read this as implying that the model is open-source. And I’m far from an open-source purist! Hell, I’ve never once made a snarky comment about OpenAI “not being open” or anything!
a2128
I just think it's a bit misleading to call something "open-source (xyz)" if it wholly depends on a proprietary service that provides (xyz). If you made an open-source implementation of the Discord client, it'd be misleading to call it "Open-source direct messaging and communities app". At least in that example it would be possible to reimplement the backend and make it truly open-source, but so far nobody has been able to reimplement GPT-4V in a way that it's nearly as useful. Hence why some people get super excited when they see "Open-source [...] AI (using vision)" and super disappointed when it's just another wrapper for OpenAI
mwmercury
This is definitely clickbait. Do you see any other GitHub url on HN that need to stick "open source" on its title?
jondwillis
So fork it and change it to use Ollama or whatever you want. It’s open source.
amelius
Why the need to fork it, though? Shouldn't it be as simple as changing the URL of the API? If it is not so simple, then perhaps time for some standardization?
jondwillis
fork it and contribute back to the main, i should have said :)
amelius
I'm contributing back not in code but in the form of architectural and UX advice :)
swells34
>Why the need to fork it, though?

Because you want it to behave differently than it does? It seems you are the one who wants things different, are complaining because they are not. Lucky for you, this is open source, and you can go ahead and fork it and change what you don't like!

[deleted]
(comment deleted)
havkom
A lot of negative comments here. However, I liked it!

Perfect Show HN and a great start of a product if the author wants to.

ralfelfving
Thank you, it's my first GH project & Show HN.. and.. yeah.. learning here :D
jonplackett
Also think this is fun.

In general I’m pretty excited about LLM as interface and what that is going to mean going forward.

I think our kids are going to think mice and keyboards are hilariously primitive.

ralfelfving
Before we know it, even voice might be obsolete when we can just think :) But maybe at that point, even thinking becomes obsolete because the AI:s are doing all the thinking for us?!
[deleted]
(comment deleted)
swiftcoder
Worth mentioning that if you are in a corporate environment, running a service that sends arbitrary desktop screenshots to a 3rd party cloud service is going to run afoul of pretty much every security and regulatory control in existence
thelittleone
The control for that is endpoints should be locked down to prevent install of non approved apps. Any org under regulatory controls would have some variation of that. Safe to assume an orgs users are stupid or nefarious and build defences accordingly.
[deleted]
(comment deleted)
ralfelfving
I assume that anyone capable of cloning the app, starting the it on their machine and obtaining + adding an OpenAI API key understands that some data is being sent offsite -- and will be aware of their corporate policies. I think that's a fair assumption.
greenie_beans
that's a fair assumption. feels like swiftcoder is just trying to gotcha
brookst
True, but also true of other screen capture utilities that send data to the cloud. Your PSA is true, but hardly unique to this little utility. And probably not surprising to the intended audience.
isoprophlex
You're telling me... the cloud... is other people's computers?!
[deleted]
(comment deleted)
jondwillis
You should add an option for streaming text as the response instead of TTS. And also maybe text in place of the voice command as well. I have been tire-kicking a similar kind of copilot for awhile, hit me up on discord @jonwilldoit
ralfelfving
There's definitely some improvements to shuttling the data between interface<->API, all that was done in a few hours on day 1 and there's a few things I decided to fix later.

I prefer speaking over typing, and I sit alone, so probably won't add a text input anytime soon. But I'll hit you up on Discord in a bit and share notes.

jondwillis
Yeah, just some features I could see adding value and not being too hard to implement :)
tomComb
> text in place of the voice command as well

That would be great for people with Mac mini who don't have a mic.

ralfelfving
Hmmm... what if I added functionality that uses the webcam to read your lips?

Just kidding. Text seem to be the most requested addition, and it wasn't on my own list :) Will see if I add it, should be fairly easy to make it configurable and render a text input window with a button instead of triggering the microphone.

Won't make any promises, but might do it.

danaris
People with a Mac mini may not have a webcam, either!
ralfelfving
Added text input instead of voice as an option today.
amelius
Please include "OpenAI-based" in the title. (Now many people here are disappointed).
[deleted]
(comment deleted)
ralfelfving
Fair point, didn't think it would matter so much. Can't edit it any more, otherwise I'd change it to add OpenAI to the title!
ukuina
This is very cool! Thank you for working on it and sharing it with us.
netika
Such a shame it uses Vision API, i.e. it can not be replaced by some random self-hosted LLM.
ralfelfving
It can be replaced with a self-hosted LLM, simply change the code where the Vision API is being called. That's true for all of the API calls in the app.
satchlj
It's not working for me, I get a "Too many requests" http error
ralfelfving
Hmm.. OpenAI bunch a few things into some error. Iirc this could be because you're out of credits / don't have a valid payment method on file, but it could also be that you're hitting rate limits. The Vision API could be the culprit, while in beta you can only call it X amount of times per day (X varies by account).

Make the console.log:s for the three API calls a bit more verbose to find out which call is causing this, and if there's more info in the error body.

I_am_tiberius
I would love to have something like this but using an open source model and without any network requests.
trenchgun
Probably in three months, approximately.
behat
Nice! Built something similar earlier to get fixes from chatgpt for error messages on screen. No voice input because I don't like speaking. My approach then was Apple Computer Vision Kit for OCR + chatgpt. This reminds me to test out OpenAI's Vision API as a replacement.

Thanks for sharing!

ralfelfving
Thanks! You could probably grab what I have, and tweak it a bit. Try checking if you can screenshot just the error message and check what the value of the window.owner is. It should be the name of the application, so you could just append `Can you help me with this error I get in ${window.owner}?` to the Vision API call.
thomashop
Just used it with the digital audio workstation Ableton Live. It is amazing! Its tips were spot-on.

I can see how much time it will save me when I'm working with a software or domain I don't know very well.

Here is the video of my interaction: https://www.youtube.com/watch?v=ikVdjom5t0E&feature=youtu.be

Weird these negative comments. Did people actually try it?

[deleted]
(comment deleted)
pelorat
I mean it does send a screenshot of your screen off to a 3rd party, and that screenshot will most likely be used in future AI training sets.

So... beware when you use it.

thomashop
Beware of it seeing a screenshot of my music set? OpenAI will start copying my song structure?

You can turn it on and off. Not necessary to turn it on when editing confidential documents.

You never enable screen-sharing in videoconferencing software?

aaronscott
I completely agree. A huge business with a singular focus isn’t going to pivot into the music business (or any of the myriad use cases the general public throws at it). And if they did use someone’s info, it’s more likely an unethical employee than a genuine business tactic.

Besides, the parent program uses the API, which allows opting out of training or retaining that data.

mecsred
Yes this makes perfect sense. As we know, businesses definitely do not treat data as a commodity and engage in selling/buying data sets on the open market as a "genuine business tactic". Therefore, since the company in question doesn't have a clear business case for data collection currently, we can be sure this data will never be used against our interests by any company.
zwily
OpenAI claims that data sent via the API (as opposed to chatGPT) will not be used in training. Whether or not you believe them is a separate question, but that's the claim.
ralfelfving
So glad when I saw this, thanks for sharing this! It was exactly music production in Ableton was the spark that lit this idea in my head the other week. I tried to explain to a friend that don't use GPT much that with Vision, you can speed up your music production and learn how to use advanced tools like Ableton more quickly. He didn't believe me. So I grabbed a Ableton screenshot off Google and used ChatGPT -- then I felt there had to be a better way, I realized that I have my own use-cases, and it all evolved into this.

I sent him your video, hopefully he'll believe me now :)

thomashop
You may be interested in two proof of concepts I've been working on. I work with generative AI and music at a company.

MidiJourney: ChatGPT integrated into Ableton Live to create MIDI clips from prompts. https://github.com/korus-labs/MIDIjourney

I have some work on a branch that makes ChatGPT a lot better at generating symbolic music (a better prompt and music notation).

LayerMosaic allows you to allow MusicGen text-to-music loops with the music library of our company. https://layermosaic.pixelynx-ai.com/

ralfelfving
Oooh. Yes, very interested in MusicGen. I played with MusicGen for the first time the other week and created a little script that uses GPT to create the prompt and params which is stored to a text file along with the output. Let it loop for a few hours to get a few 100 output files that allowed me to learn a bit more about what kind of prompts that gave reasonable output (it was all bad, lol!)
ralfelfving
My brain read midjourney until I clicked on the GH link. What a great name, MIDIjourney!
ralfelfving
Oh LayerMosaic is dope. I'm not entirely sure how it works, but the sounds coming out of it is good -- so you have me intrigued! Can I read more about it somewhere, I might have a crazy idea I'd like to use this for.
mikey_p
Is it just me or is it incredibly useless?

"Here's a list of effects. Here's a list of things that make a song. Is it good? Yes. What about my drum effects? Yes here's the name of the two effects you are using on your drum channel"

None of this is really helpful and I can't get over how much it sounds like Eliza.

thomashop
I made that video right at the start but since then I've asked it for example what kind of compression parameters would fit with a certain track and it could explain to me how to find an expert function which I would have had to consult a manual for otherwise.
urbandw311er
Yeah I thought the same. Ultra generic advice and no evidence it has actually parsed anything unique or useful from the user’s actual composition.
thomashop
I made another one: https://www.youtube.com/watch?v=zyMmurtCkHI

In the one I posted I was just so amazed how well it worked and didn't really try anything useful. In this video you can see it giving me quite good advice on how to make a bassline dubby and how to carve frequencies out of the kick drum to make space for the bass.

It also looks at spectrograms and gives feedback / takes them into account. I'm pretty amazed.

ralfelfving
Did you change the GPT Vision system prompt at all? I wonder if changing it to state getting help with specifically Ableton, and maybe some guidelines around what kind of help you want could make it better?
thomashop
No. But I found it good enough as it is
e28eta
Did you find that calling it “OSX” in the prompt worked better than macOS? Or was that just an early choice that you didn’t spend much time on?

I was skimming through the video you posted, and was curious.

https://www.youtube.com/watch?v=1IdCWqTZLyA&t=32s

code link: https://github.com/elfvingralf/macOSpilot-ai-assistant/blob/...

ralfelfving
No, this is an oversight by me. To be completely honest, up until the other day I thought it was still called OSX. So the project was literally called cOSXpilot, but at some point I double checked and realize it's been called macOS for many years. Updated the project, but apparently not the code :)

I suspect OSX vs macOS has marginal impact on the outcome :)

e28eta
Haha, makes perfect sense, thanks for the reply!
hot_gril
Heh. I remember calling it Mac OS back in the day and getting corrected that it's actually OS X, as in "OS ten," and hasn't been called Mac OS since Mac OS 9. Glad Apple finally saw it my way (except it's cased macOS).
qainsights
Great. I created `kel` for terminal users. Please check it out at https://github.com/qainsights/kel
[deleted]
(comment deleted)