This made me think: what if my little utility assistant program that I have, similar to your Stevens, had access to a mailbox?
I've got a little utility program that I can tell to get the weather or run common commands unique to my system. It's handy, and I can even cron it to run things regularly, if I'd like.
If it had its own email box, I can send it information, it could use AI to parse that info, and possibly send email back, or a new message. Now, I've got something really useful. It would parse the email, add it to whatever internal store it has, and delete the message, without screwing up my own email box.
I’ve been thinking lately that email is a good interface for certain modes of AI assistant interaction, namely “research” tasks that are asynchronous and take a relatively long time. Email is universal, asynchronous, uses open standards, supports structured metadata, etc.
I have a couple companies that force me to send them data via email. They have an email template that you have to conform to, and they can parse it. Mainly just very rudimentary line breaks and 'LineItem: content' format. But json in the body should be fine as well. Given the way email programs strip or modify html at times, I would be leery of xml.
If you want to get ahead of the curve, look into the Agent-to-Agent protocol Google just introduced. I'm currently using my own custom AI agent assistant to perform life tasks. If I could integrate a better tooling/agents into my own assistant system like your's that'd be awesome.
It's kind of like sure, I could manage my own emails, or I could offset this to someone who does it better. If you do it better and it's affordable, I'm in.
We are on that starship to the future right now and I love it.
I am still very open to this one. An email-based, artificial coworker is so obviously the right way to penetrate virtually every B2B market in existence.
I don't even really want to touch the technology aspects. Writing code that integrates with an LLM provider and a mailbox in E365 or Gmail is boring. The schema is a grand total of ten tables if we're being pedantic about things.
Working with prospects and turning them into customers is a way more interesting problem. I hunger for tangible use cases that are actually compatible with this shiny new LLM tooling. We all know they're out there, and email is probably the lowest friction way to get them applied to most businesses.
I've build adaptive agent swarms using email, mailing lists and ftp servers.
If you don't need to have the lowest possible latency for your work and you're happy to have threads die then it's better than any bespoke solution you can build without an army of engineers to keep it chugging along.
What's even better is that you can see all the context, and use the same command plane as the agents to tell them what they are doing wrong.
Email is decent for intermural communication. If it's intramural and you control both the sender and receiver, MQTT or ntfy are likely better communication channels since they increase flexibility and lower complexity, IMO.
Not if I want it able to have conversations with people, they don't.
I could see installing or implementing a custom client if there were some functionality that'd enable, but "support a conversation among two speakers" is something computers have done since well before I was born. If the wheel fits, why reinvent it?
If you're having conversations with people, then you don't control both ends and email is fine for that. Email is suboptimal for communicating between services/applications under your full control.
Consider the use case from the article: this is a family management support or "AI butler" application. So I control the end with the LLM on it, which I administer - but not necessarily the other, which is anyone in my family, not just me. So unless I want to try to make everyone use my weird custom AI messaging app like I aspire to Bay Area thought-cult leadership, I'm going to meet people where they are and SMTP's cheaper than SMS.
If I'm building myself a toy, then sure, I can implement whatever I want for a client, if that's where I get my jollies. React Native isn't hard but it is often annoying, and the fun for me in this project would be all in the conversation with the agent per se. Whatever doesn't get me to that as fast as possible is just getting in my way, you know?
And too, if this does turn out to be something that actually works well for me, then I'm going to want to integrate it with my phone's voice assistant, and at that point an app is required anyway - but if I start with a protocol and an app that that assistant already knows how to interact with, then again I have an essentially free if admittedly very imperfect prototype.
Under the hoods, is your AI butter one service or many? It would be not-great for your weather or family-event-calendar-management components to communicate with each other or the orchestrator via email.
Receiving an email from the AI-butler rescheduling or relocating a planned outdoors family event because rain is expected would be excellent, using IMAP to wire-up the subcomponents together would not.
Who suggested using email in the service layer? I mean, you're not wrong, but this feels like you handed me a banana and then said I should have picked a better hammer.
We're talking about a conversation that has a human on at least one end, so email makes sense. For conversations involving no humans, of course there are much better stores and protocols if something like an asynchronous world-writable queue is what we want.
"Number of humans in the conversation" wasn't the distinction you initially established, I believe, but I wonder if it's closer to the one you had in mind.
My one concern there would be edits: a CMS needs to support easily making edits to content (fixing typos etc) - editing existing posts via email sounds like it would be pretty fiddly.
is there a reason you went with telegram and not slack or discord? i was thinking that it could open up a broader channel for communicating with your assistant. i understand you're also just building more of a poc, but curious if you'd thought about that. great work btw :)
Mailgun (and I'm sure many other services like it) can accept emails and POST their content to an url of your choice.
I use that for journaling: I made a little system that sends me an email every day; I respond to it and the response is then sent to a page that stores it into a db.
+1 for Mailgun. My only gripe with it is that they detect and block bot activity on their frontend. So if you have end to end (e2e) integration tests built with something like Puppeteer, you can't have them log into Mailgun and check the inbox table's HTML to see that an email was sent. So you have to write some sort of plugin manually - perhaps as a testing endpoint on your website that only appears in debug mode - that interacts with their API.
This might not seem like much of a big deal. But as we transition to more of these #nocode automated tools, the idea of having to know how programming works in order to interact with an API will start to seem archaic. I'd compare it to how esoteric the terminal looked after someone saw a GUI like the one used by Apple's Macintosh back in the 1980s.
I looked forward to this day back in the early 2000s when APIs started arriving, but felt even then that something was fishy. I would have preferred that sites had a style-free request format that returned XML or even JSON generated from HTML, rather than having to use a separate API. I have this sense that the way we do it today with a split backend/frontend, distributed state, duplicated validation, etc has been a monumental waste of time.
> I use that for journaling: I made a little system that sends me an email every day; I respond to it and the response is then sent to a page that stores it into a db.
Yes. I know note taking and journaling posts are frequent on HN, but I've thought that this is the best way to go, is universal from any client, and very expandable. It's just not generically scaleable for all users, but for the HN reader-types, it'd be perfect.
This was the attack vector of a AI CTF hosted by Microsoft last year. I built an agent to assess, structure, and perform the attacks autonomously and found that even with some common guardrails in place the system was vulnerable to data exfiltration. My agent was able to successfully complete 18 of the challenges... Here is the write up after the finals.
For gmail, there's also an amazing thing where you can hook it with pubsub. So now it's push not pull. Any server will get pubsub little webhooks for any change within milliseconds (you can filter server side or client side for specific filters)
This is amazing, you can do all sorts of automations. You can feed it to an llm and have it immediately tag it (or archive it). For important emails (I have a specific label I add, where if the person responds, it's very important and I want to know immediately) you can hook into twilio and it calls me. Costs like 20 cents a month
I made an AI assistant telegram bot running on my Mac that runs commands for me. I'll tell it "Run ncdu in the root dir and tell me what's taking up all my disk space" or something and it converts that bash and runs it via os.system. It shows me the command it created, plus the output.
Extremely insecure, but kinda fun.
I turned it off because I'm not that crazy but I'm sure I could make a safer version of it.
*Update*: I tried writing a little Python code to read and write from a mailbox, reading worked great, but writing an email had the email disappear to some filter or spam or something somewhere. I've got to figure out where it went, but this is the warning that some people had about not trusting a messaging protocol (email in this case) when you can't control the servers. Messages can disappear.
Other alternatives for messages that I haven't tried. My requirement is to be able to send messages and send/receive on my mobile device. I do not want to write a mobile app.
I built up an AI Agent using n8n and email doing exactly this. Works great and was surprised I'd not seen any other place kicking the idea around.
Probably my favorite use case is I can shoot it shopping receipts and it'll roughly parse them and dump the line item and cost into a spreadsheet before uploading it to paperless-ngx.
"I can shoot it shopping receipts and it'll roughly parse them and dump the line item and cost into a spreadsheet" - very difficult to do that without using a vision LLM.
I'm building something similar and related to the other comments below! It's not production ready but it will hopefully be in a couple of weeks. You guys can sign up for free and I will upgrade you to the premium tier manually (premium cannot be bought yet anyway) in exchange for some feedback:
This is the kind of pragmatic AI hack I want to see. It feels like sometimes we are forgetting why certain tooling even exists. To simplify things! No fancy vector DBs or complex architectures, just practical integration with existing data sources. Love it.
Hmm, there's supposed to be a Tasks [reminders] feature in ChatGPT, but it's in beta (I don't have access to it). Whenever it gets released, you could make some kind of "router" that connects to different communication methods and connect that up to ChatGPT statefully, and you could just "speak"/type to ChatGPT from anywhere, and it would send you reminders. No need for all the extra logic, cron jobs, or SQLite table (ChatGPT has memory across chats).
This is fun! I think this sort of tooling is going to be very fertile ground for hackers over the next few years.
Large swathes of the stack is commoditized OSS plumbing, and hosted inference is already cheap and easy.
There are obvious security issues with plugging an agent into your email and calendar, but I think many will find it preferable to control the whole stack rather than ceding control to Apple or Google.
Lately I have been experimenting with ways to work around the "context token sweet spot" of <20k tokens (or <50k with 2.5). Essentially doing manual "context compression", where the LLM works with a database to store things permanently according to a strict schema, summarizes it's current context when it starts to get out of the sweet spot (I'm mixed on whether it is best to do this continuously like a journal, or in retrospect like a closing summary), and then passes this to a new instance with fresh context.
This works really effectively with thinking models, because the thinking eats up tons of context, but also produces very good "summary documents". So you can kind of reap the rewards of thinking without having to sacrifice that juicy sub 50k context. The database also provides a form of fallback, or RAG I suppose, for situations where the summary leaves out important details, but the model must also recognize this and go pull context from the DB.
Right now I have been trying it to make essentially an inventory management/BOM optimization agent for a database of ~10k distinct parts/materials.
I am excitedly waiting for the first company (guessing / hoping it'll be anthropic) to invest heavily in improvements to caching.
The big ones that come to mind are cheap long term caching, and innovations in compaction, differential stuff - like is there a way to only use the parts of the cached input context we need?
Isn’t a problem there that a cache would be model specific, where the cached items are only valid for exactly the same weights and inference engine? I think those are both heavily iterated on.
Prompt caches right now only last a few minutes - I believe they involve keeping a bunch of calculations in-memory, hence why for Gemini and Anthropic you get charged an initial fee for using the feature (to populate the cache), but then get a discount on prompts that use that cache.
The background tasks can call mcp servers, to connect to more data sources and services. At least you don’t have to write all the connectivities to them.
Curious, how come you decided to use a cloud solution instead of hosting this on a home server? I’ve recently bought a mini PC for small projects like this and have been loving being able to host with no cost associated to it. Albeit it’s probably still incredibly cheap to use a IaaS or PaaS but still a barrier to entry for random projects I want to work on a weekend
I'd use a hosted platform for this kind of thing myself, because then there's less for me to have to worry about. I have dozens of little systems running in GitHub Actions right now just to save me from having to maintain a machine with a crontab.
Home server AI is orders of magnitude more costly than heavily subsidized cloud based ones for this use case unless you run toy models that might hallucinate meetings.
edit: I now realize you're talking about the non-ai related functionality.
A single cloudflare durable object (sqlite db + serverless compute + cron triggers) would be enough to run this project. DOs have been added to CFs free tier recently - you could probably run a couple hundred (maybe thousands) instances of Stevens without paying a cent, aside from Claude costs ofc
"It’s very useful for personal AI tools to have access to broader context from other information sources."
How? This post shows nothing of the sort.
"I’ve written before about how the endgame for AI-driven personal software isn’t more app silos, it’s small tools operating on a shared pool of context about our lives."
Yes, probably, so now is the time to resist and refuse to open ourselves up to unprecedented degrees of vulnerability towards the state and corporations. Doing it voluntarily while it is still rather cheap is a bad idea.
This is probably naive and looking forward to a correction; isn't sending your info to Claude's API (or really any "AI API") is a violation of your safeguarded privacy data?
Only if you don't believe the AI vendors when they promise that they won't train on your data.
(Or you don't trust them not to have security breaches that grant attackers access to logged data, which remains a genuine thread, albeit one that's true of any other cloud service.)
Correct. My dusty Intel Nuc is able to run a decent 3B model(thanks to ollama) with fans spinning but does not affect any other running applications. It ks very useful for local hobby projects. Visible lags and freezes begin if I start a 5B+ model locally.
> It’s rudimentary, but already more useful to me than Siri!
For me, that is an extremely low barrier to cross.
I find Siri useful for exactly two things at the moment: setting timers and calling people while I am driving.
For these two things it is really useful, but even in these niches, when it comes to calling people, despite it having been around me for years now it insist on stupid things like telling me there is no Theresa in my contacts when I ask it to call Therese.
That said what I really want is a reliable system I can trust with calendar acccess and that is possible to discuss with, ideally voice based.
I feel like the standard Apple response is "if it isn't working correctly you just aren't using it right"
I still regularly experience a bug where my mac sends sound to speakers instead of a plugged in headphone jack after waking up from sleep. 10 years ago when I first looked into it the official Apple response was "that's not possible with the hardware" and we haven't made any progress since. Gaslighting as a service I guess.
Luckily I can just unplug and plug back in. Maybe they can bring the great Apple minds together to make my iPhone stop blasting an alarm in my ear at regular volume if I happen to be talking on the phone when it goes off (issue since my very first iPhone 3).
I've had the same issues of decay. I used to be able to say "call Mom" but now it will call some kid's mom who I have in Contacts as "[some kid's] mom". What is the underlying architecture that simple heuristic things like this can get worse? Are they gradually slipping in AI?
I went through this weird experience with Cortana on WP7, where I found it incredibly useful to begin with, and then over time it got worse. It seemed like it was created by some incredibly talented engineers. I used it to make calls while driving, set the GPS and search for information while I drove. But over time, it seemed to change behaviour and started ignoring my commands, and when it did accept them, it seemed to refer me to paid advertisers. And considering bing wasnt even as popular as it is now, 10 years ago, a paid advertiser could be 100km away.
Which I think is a path that people haven't considered with LLMs. We are expecting them to get better forever, but once we start using them, their legs will be cut out to force them to feed us advertising.
Having some experience with weaker models, you need at least 1.5B-3B to see proper prompt adherence and less hallucinations and better memory.
Also models have subtle differences, for example, I found Qwen2.5:0.5B to be more obedient(prompt respecting) and smart, compared to LLama3.2:1B. Gemma3:1B seems to be more efficient but despite heavy prompting, tends to be verbose and fails at formatted response by injecting some odd emoji or remark before/after the desired output.
In summary, Qwen2.5:1.5B and LLama3.2:3B were the weakest model which were more useful and also includes tools support(Gemma does not understand tools yet).
1. How did he tell Claude to “update” based on the notebook entries?
2. Won’t he eventually ran out of context window?
3. Won’t this be expensive when using hosted solutions? For just personal hacking, why not simply use ollama + your favorite model?
4. If one were to build this locally, can Vector DB similarity search or a hybrid combined with fulltext search be used to achieve this?
I can totally imagine using pgai for the notebook logs feature and local ollama + deepseek for the inference.
The email idea mentioned by other commenters is brilliant. But I don’t think you need a new mailbox, just pull from Gmail and grep if sender and receiver is yourself (aka the self tag).
Thank you for sharing, OP’s project is something I have been thinking for a few months now.
The "memories" table has a date column which is used to record the data when the information is relevant. The prompt can then be fed just information for today and the next few days - which will always be tiny.
It's possible to save "memories" that are always included in the prompt, but even those will add up to not a lot of tokens over time.
> Won’t this be expensive when using hosted solutions?
You may be under-estimating how absurdly cheap hosted LLMs are these days. Most prompts against most models cost a fraction of a single cent, even for tens of thousands of tokens. Play around with my LLM pricing calculator for an illustration of that: https://tools.simonwillison.net/llm-prices
> If one were to build this locally, can Vector DB similarity search or a hybrid combined with fulltext search be used to achieve this?
Geoffrey's design is so simple it doesn't even need search - all it does is dump in context that's been stamped with a date, and there are so few tokens there's no need for FTS or vector search. If you wanted to build something more sophisticated you could absolutely use those. SQLite has surprisingly capable FTS built in and there are extensions like https://github.com/asg017/sqlite-vec for doing things with vectors.
for memories (still not shown in this tutorial) I have created a pantry [0]
and a servlet for it [1] and I modified the prompt so that it would first check if a conversation existed with the given chat id, and store the result there.
The cool thing is that you can add any servlets on the registry and make your bot as capable as you want.
I argue that this kind of tools are fun to play but in the end is it really helpful? I start my day like every day and on work I just check the calendar. My private calendar has all Information i need. Where is the gap where an Assistent makes sense and where we are just complicating our lives?
Personally, this appears to be extremely helpful for me, because instead of checking several different spots every day, I can get a coherent summary in one spot, tailored to me and my family. I'm literally checking the same things every day, down to USPS Informed Delivery. This seems to simplify what's already complicated, at least for my use cases.
Is this niche? I don't know and I don't care. It looks useful to me. And the author, obviously, because they wrote it. That's enough.
I can't count the number of useful scripts and apps I've written that nobody else has used, yet I rely on them daily or nearly every day.
Now think of this at a family level. You have 2+ people with shared calendars and events.
Do you sit down as a family every morning and go through your calendars and sync up?
Or would it be better to have an automated summary posted to the family Telegram channel with "Bob has a dentist today at 1300, which overlaps with Mia's football practice, so Sara has to pick her up. Also it's going to rain so prepare accordingly."
I like the idea of parsing USPS Informed Delivery emails (a lot of people I encounter still don't know that this service exists). Maybe I'll make something to alert me when my checks are finally arriving!
This part was galling to me; somewhere in the USPS, the data about what mailpieces/packages are arriving soon exist in a very concise form, and they templatize an email and send it to me, after which I can parse the email with simple+brittle regexes or forward the emails to a relatively (environmentally-)expensive LLM or so.... but if they'd made the information available with an API or RSS feed, or attached the json payload to the email in the first place, I could get away without parsing.
It would indeed be nice to have a recipient/consumer-side API!
I don't think it'll ever happen. Really the only valid use-case would be for people to hack together something for themselves (like we are discussing)... They don't want to allow developers to create applications on top of this as a 3rd party, as informed delivery itself has to carefully navigate privacy laws and it could be disastrous.
In Finland our own USPS (Posti) has their own mobile application that does delivery notifications. They've been directing users towards the app pretty heavily and deprecating other interfaces.
You can still get the parcel ID and use a public-ish web API to get tracking information on a rough level ("in transit", "being delivered") without exact address information.
Love it, such a nice idea coupled with a flawless execution. I think the future of AI looks a lot more like this than half-cooked agent implementations that plagues LinkedIn…
I have built something similar that runs without a server. It required just a few lines in Apple shortcuts.
TL;DR I made shortcuts that work on my Apple watch directly to record my voice, transcribe it and store my daily logs on a Notion DB.
All you need are 1) a chatgpt API key and 2) a Notion account (free).
- I made one shortcut in my iPhone to record my voice, use whisper model to transcribe it (done locally using a POST request) and send this transcription to my Notion database (again a POST request on shortcuts)
- I made another shortcut that records my voice, transcribes and reads data from my Notion database to answer questions based on what exists in it. It puts all data from db into the context to answer -- costs a lot but simple and works well.
The best part is -- this workflow works without my iPhone and directly on my Apple Watch. It uses POST requests internally so no need of hosting a server. And Notion API happens to be free for this kind of a use case.
I like logging my day to day activities with just using Siri on my watch and possibly getting insights based on them. Honestly the whisper model is what makes it work because the accuracy is miles ahead of the local transcription model.
I'll plan to do it at some point -- at this moment I have hardcoded my credentials into the shortcut so it's a bit hard to share without tweaking. I didn't bother detailing it because its sort of simple. I think the idea is key here and anyone with a few hours to kill can get something working.
On second thought -- apple shortcuts is really brittle. It breaks in non obvious ways and a lot can only be learned by trial and error lol
182 comments
[ 3.1 ms ] story [ 211 ms ] threadFor others: they use Claude.
I've got a little utility program that I can tell to get the weather or run common commands unique to my system. It's handy, and I can even cron it to run things regularly, if I'd like.
If it had its own email box, I can send it information, it could use AI to parse that info, and possibly send email back, or a new message. Now, I've got something really useful. It would parse the email, add it to whatever internal store it has, and delete the message, without screwing up my own email box.
Thanks for the insight.
It is a lot cheaper to leverage existing user interfaces & tools (i.e., Outlook) than it is to build new UIs and then train users on them.
https://threadwise.app
It's kind of like sure, I could manage my own emails, or I could offset this to someone who does it better. If you do it better and it's affordable, I'm in.
We are on that starship to the future right now and I love it.
text + attachments into the system, text + attachments out
My finance guy, tax attorney, other attorneys. Send emails, get emails, occasionally a blind status update from them.
Sure, we have phone calls, sometimes get together for lunch.
But mostly it’s just emails.
I am still very open to this one. An email-based, artificial coworker is so obviously the right way to penetrate virtually every B2B market in existence.
I don't even really want to touch the technology aspects. Writing code that integrates with an LLM provider and a mailbox in E365 or Gmail is boring. The schema is a grand total of ten tables if we're being pedantic about things.
Working with prospects and turning them into customers is a way more interesting problem. I hunger for tangible use cases that are actually compatible with this shiny new LLM tooling. We all know they're out there, and email is probably the lowest friction way to get them applied to most businesses.
Agreed. That's also the hardest part, and where most value is created.
If you don't need to have the lowest possible latency for your work and you're happy to have threads die then it's better than any bespoke solution you can build without an army of engineers to keep it chugging along.
What's even better is that you can see all the context, and use the same command plane as the agents to tell them what they are doing wrong.
I could see installing or implementing a custom client if there were some functionality that'd enable, but "support a conversation among two speakers" is something computers have done since well before I was born. If the wheel fits, why reinvent it?
If I'm building myself a toy, then sure, I can implement whatever I want for a client, if that's where I get my jollies. React Native isn't hard but it is often annoying, and the fun for me in this project would be all in the conversation with the agent per se. Whatever doesn't get me to that as fast as possible is just getting in my way, you know?
And too, if this does turn out to be something that actually works well for me, then I'm going to want to integrate it with my phone's voice assistant, and at that point an app is required anyway - but if I start with a protocol and an app that that assistant already knows how to interact with, then again I have an essentially free if admittedly very imperfect prototype.
Receiving an email from the AI-butler rescheduling or relocating a planned outdoors family event because rain is expected would be excellent, using IMAP to wire-up the subcomponents together would not.
We're talking about a conversation that has a human on at least one end, so email makes sense. For conversations involving no humans, of course there are much better stores and protocols if something like an asynchronous world-writable queue is what we want.
"Number of humans in the conversation" wasn't the distinction you initially established, I believe, but I wonder if it's closer to the one you had in mind.
https://threadwise.app
- all attachments are stripped out and stored on a server in an hierarchical structure based on sender/recipient/subject line
- all discussions are archived based on similar criteria, and can be reviewed EDIT: and edited like to a wiki
https://www.val.town/x/geoffreylitt/stevensDemo/code/importe...
I think it would be pretty easy to extend to support other types of inbound email.
Also I work for Val Town, happy to answer any questions.
I use that for journaling: I made a little system that sends me an email every day; I respond to it and the response is then sent to a page that stores it into a db.
This might not seem like much of a big deal. But as we transition to more of these #nocode automated tools, the idea of having to know how programming works in order to interact with an API will start to seem archaic. I'd compare it to how esoteric the terminal looked after someone saw a GUI like the one used by Apple's Macintosh back in the 1980s.
I looked forward to this day back in the early 2000s when APIs started arriving, but felt even then that something was fishy. I would have preferred that sites had a style-free request format that returned XML or even JSON generated from HTML, rather than having to use a separate API. I have this sense that the way we do it today with a split backend/frontend, distributed state, duplicated validation, etc has been a monumental waste of time.
Yes. I know note taking and journaling posts are frequent on HN, but I've thought that this is the best way to go, is universal from any client, and very expandable. It's just not generically scaleable for all users, but for the HN reader-types, it'd be perfect.
I've found it to be very reliable with a detailed dashboard to track individual transactions, plus they give you 10,000 emails a month for free.
Not an employee, just a big fan!
[0] https://www.cloudmailin.com
https://msrc.microsoft.com/blog/2025/03/announcing-the-winne...
This is amazing, you can do all sorts of automations. You can feed it to an llm and have it immediately tag it (or archive it). For important emails (I have a specific label I add, where if the person responds, it's very important and I want to know immediately) you can hook into twilio and it calls me. Costs like 20 cents a month
Extremely insecure, but kinda fun.
I turned it off because I'm not that crazy but I'm sure I could make a safer version of it.
I read that [Mailgun](https://www.mailgun.com/) might improve this. Haven't tried it yet.
Other alternatives for messages that I haven't tried. My requirement is to be able to send messages and send/receive on my mobile device. I do not want to write a mobile app.
* [Telegram](https://telegram.org/) (OP's system) with [bots](https://core.telegram.org/bots)
* [MQTT](https://mqtt.org/) with server
* [Notify (ntfy.sh)](https://ntfy.sh/)
* Email (ubiquitous)
Also, to [simonw](https://news.ycombinator.com/user?id=simonw) point, LLM calls are cheap now, especially with something as low tokens as this.And, links don't format in HN markdown. I did the work to include them, they're staying in.
Probably my favorite use case is I can shoot it shopping receipts and it'll roughly parse them and dump the line item and cost into a spreadsheet before uploading it to paperless-ngx.
https://threadwise.app
It's about 652 tokens according to https://tools.simonwillison.net/claude-token-counter - maybe double that once you add all of the context from the database table.
1200 input tokens and 200 output tokens for Claude 3.7 Sonnet costs 0.66 cents - that's around 2/3rd of a cent.
LLM APIs are so cheap these days.
Large swathes of the stack is commoditized OSS plumbing, and hosted inference is already cheap and easy.
There are obvious security issues with plugging an agent into your email and calendar, but I think many will find it preferable to control the whole stack rather than ceding control to Apple or Google.
"There are obivious security issues with plugging and agent into your email..." Isn't this how North Korea makes all their crypto happen?
This works really effectively with thinking models, because the thinking eats up tons of context, but also produces very good "summary documents". So you can kind of reap the rewards of thinking without having to sacrifice that juicy sub 50k context. The database also provides a form of fallback, or RAG I suppose, for situations where the summary leaves out important details, but the model must also recognize this and go pull context from the DB.
Right now I have been trying it to make essentially an inventory management/BOM optimization agent for a database of ~10k distinct parts/materials.
The big ones that come to mind are cheap long term caching, and innovations in compaction, differential stuff - like is there a way to only use the parts of the cached input context we need?
The background tasks can call mcp servers, to connect to more data sources and services. At least you don’t have to write all the connectivities to them.
I'd use a hosted platform for this kind of thing myself, because then there's less for me to have to worry about. I have dozens of little systems running in GitHub Actions right now just to save me from having to maintain a machine with a crontab.
Home server AI is orders of magnitude more costly than heavily subsidized cloud based ones for this use case unless you run toy models that might hallucinate meetings.
edit: I now realize you're talking about the non-ai related functionality.
How? This post shows nothing of the sort.
"I’ve written before about how the endgame for AI-driven personal software isn’t more app silos, it’s small tools operating on a shared pool of context about our lives."
Yes, probably, so now is the time to resist and refuse to open ourselves up to unprecedented degrees of vulnerability towards the state and corporations. Doing it voluntarily while it is still rather cheap is a bad idea.
(Or you don't trust them not to have security breaches that grant attackers access to logged data, which remains a genuine thread, albeit one that's true of any other cloud service.)
Any M-series Mac Mini can run a pretty good local model with usable speed. The high-end models easily compete with dedicated GPUs.
For me, that is an extremely low barrier to cross.
I find Siri useful for exactly two things at the moment: setting timers and calling people while I am driving.
For these two things it is really useful, but even in these niches, when it comes to calling people, despite it having been around me for years now it insist on stupid things like telling me there is no Theresa in my contacts when I ask it to call Therese.
That said what I really want is a reliable system I can trust with calendar acccess and that is possible to discuss with, ideally voice based.
I still regularly experience a bug where my mac sends sound to speakers instead of a plugged in headphone jack after waking up from sleep. 10 years ago when I first looked into it the official Apple response was "that's not possible with the hardware" and we haven't made any progress since. Gaslighting as a service I guess.
Luckily I can just unplug and plug back in. Maybe they can bring the great Apple minds together to make my iPhone stop blasting an alarm in my ear at regular volume if I happen to be talking on the phone when it goes off (issue since my very first iPhone 3).
This always gets me...is there not a public bug report for this one?
Which I think is a path that people haven't considered with LLMs. We are expecting them to get better forever, but once we start using them, their legs will be cut out to force them to feed us advertising.
> cron job which makes a call to the Claude API
I am wondering, how powerful the AI model need to be to power this app?
Would a selfhosted Llama-3.2-1B, Qwen2.5-0.5B or Qwen2.5-1.5B on a phone be enough?
Also models have subtle differences, for example, I found Qwen2.5:0.5B to be more obedient(prompt respecting) and smart, compared to LLama3.2:1B. Gemma3:1B seems to be more efficient but despite heavy prompting, tends to be verbose and fails at formatted response by injecting some odd emoji or remark before/after the desired output.
In summary, Qwen2.5:1.5B and LLama3.2:3B were the weakest model which were more useful and also includes tools support(Gemma does not understand tools yet).
1. How did he tell Claude to “update” based on the notebook entries?
2. Won’t he eventually ran out of context window?
3. Won’t this be expensive when using hosted solutions? For just personal hacking, why not simply use ollama + your favorite model?
4. If one were to build this locally, can Vector DB similarity search or a hybrid combined with fulltext search be used to achieve this?
I can totally imagine using pgai for the notebook logs feature and local ollama + deepseek for the inference.
The email idea mentioned by other commenters is brilliant. But I don’t think you need a new mailbox, just pull from Gmail and grep if sender and receiver is yourself (aka the self tag).
Thank you for sharing, OP’s project is something I have been thinking for a few months now.
The "memories" table has a date column which is used to record the data when the information is relevant. The prompt can then be fed just information for today and the next few days - which will always be tiny.
It's possible to save "memories" that are always included in the prompt, but even those will add up to not a lot of tokens over time.
> Won’t this be expensive when using hosted solutions?
You may be under-estimating how absurdly cheap hosted LLMs are these days. Most prompts against most models cost a fraction of a single cent, even for tens of thousands of tokens. Play around with my LLM pricing calculator for an illustration of that: https://tools.simonwillison.net/llm-prices
> If one were to build this locally, can Vector DB similarity search or a hybrid combined with fulltext search be used to achieve this?
Geoffrey's design is so simple it doesn't even need search - all it does is dump in context that's been stamped with a date, and there are so few tokens there's no need for FTS or vector search. If you wanted to build something more sophisticated you could absolutely use those. SQLite has surprisingly capable FTS built in and there are extensions like https://github.com/asg017/sqlite-vec for doing things with vectors.
Do we even need to think of these as agents, or will the agentic frameworks move towrads being a call_llm() sql function?
- https://docs.mcp.run/tasks/tutorials/telegram-bot
for memories (still not shown in this tutorial) I have created a pantry [0] and a servlet for it [1] and I modified the prompt so that it would first check if a conversation existed with the given chat id, and store the result there.
The cool thing is that you can add any servlets on the registry and make your bot as capable as you want.
[0] https://getpantry.cloud/ [1] https://www.mcp.run/evacchi/pantry
Disclaimer: I work at Dylibso :o)
Personally, this appears to be extremely helpful for me, because instead of checking several different spots every day, I can get a coherent summary in one spot, tailored to me and my family. I'm literally checking the same things every day, down to USPS Informed Delivery. This seems to simplify what's already complicated, at least for my use cases.
Is this niche? I don't know and I don't care. It looks useful to me. And the author, obviously, because they wrote it. That's enough.
I can't count the number of useful scripts and apps I've written that nobody else has used, yet I rely on them daily or nearly every day.
Do you sit down as a family every morning and go through your calendars and sync up?
Or would it be better to have an automated summary posted to the family Telegram channel with "Bob has a dentist today at 1300, which overlaps with Mia's football practice, so Sara has to pick her up. Also it's going to rain so prepare accordingly."
Which is a nice time with the family we are sitting drinking coffee also with other family members.
When you Lack so much organization skills the AI would never get the data to write you this.
I don't think it'll ever happen. Really the only valid use-case would be for people to hack together something for themselves (like we are discussing)... They don't want to allow developers to create applications on top of this as a 3rd party, as informed delivery itself has to carefully navigate privacy laws and it could be disastrous.
You can still get the parcel ID and use a public-ish web API to get tracking information on a rough level ("in transit", "being delivered") without exact address information.
TL;DR I made shortcuts that work on my Apple watch directly to record my voice, transcribe it and store my daily logs on a Notion DB.
All you need are 1) a chatgpt API key and 2) a Notion account (free).
- I made one shortcut in my iPhone to record my voice, use whisper model to transcribe it (done locally using a POST request) and send this transcription to my Notion database (again a POST request on shortcuts)
- I made another shortcut that records my voice, transcribes and reads data from my Notion database to answer questions based on what exists in it. It puts all data from db into the context to answer -- costs a lot but simple and works well.
The best part is -- this workflow works without my iPhone and directly on my Apple Watch. It uses POST requests internally so no need of hosting a server. And Notion API happens to be free for this kind of a use case.
I like logging my day to day activities with just using Siri on my watch and possibly getting insights based on them. Honestly the whisper model is what makes it work because the accuracy is miles ahead of the local transcription model.
On second thought -- apple shortcuts is really brittle. It breaks in non obvious ways and a lot can only be learned by trial and error lol
Edit: I just wrote up something quick https://simianwords.bearblog.dev/how-i-use-my-apple-watch-to...