Very cool setup. I went down a similar path and while it is definitely cool to see the extent of how agentic AI can be, I agree the ROI isn't _quite_ there yet.
I don't live in a world where I need to be constantly reading and replying to emails, so 95% of my inbox is just subscription spam. With development I need to be at the helm to design the planning requirements and actively make decisions before the agent goes off and executes the plan. But I don't need a personal agent for that, I work directly out of codex/claude-code.
being able to talk to each of the agents via dm (but also in group chats) sounds interesting
does that mean that you have 1 chat per domain specific agent? can you also start multiple sessions/threads or is that not part of the way you interact with them currently?
Not pointed at the author, but at the current state of affairs: this is fucking exhausting. We went and made a trillions-dollar market out of the bikeshedding maximization machine.
I've been building a triage agent for my inbox and whatsapp (it's product shaped), which has ironically left me not building one of these. So even while productizing, I'm getting fomo on the full monty.
I've also been building a harness that maintains my apps which I'm hoping to open source.
Hard agree that these things don't have personal ROI, and are actually quite hard to build reliably.
But it's really fun! And having a bot fix a live error is pretty exciting.
I actually went down a different path and set it up so my friend and I can use the same agent with approval-gated turns. He's more technical than I am, but I believe I'm better at marketing. We can see both of our prompts and be on the same page. There is also a mode where we can branch off with our own agents.
Watching his prompts made me better at producing my own and using the agent effectively.
tried similar setups in the past, but messages is just not my jam, so built a html wrapper on top of claude code and open code that runs on a small-ish instance. looks like this https://i.imgur.com/lj9Fgco.png (screenshot anonymized with chatgpt) and allows multiple conversation per project, and has a few conveniences like scheduled tasks. one of the project in the list is the project itself, so I can add features whenever.
Love the post and have a similar setup. But I have questions about how you handle sensitive personal info with Codex and Claude with this setup given Hermes has no boundaries on its own with its integrations with Codex for example. It can leak memory and tool calls and context through.
At home I have Hermes on a VPS with matrix and mnemosyne and it is largely for personal use. While I was excited to try out the seamless Codex wrapper, this leakage was unacceptable compared to a ZDR OpenRouter provider like fireworks.
Where I struggle is now in architecting a clean way to have Hermes separate out personal stuff from coding stuff, and then orchestrate coding through Codex as that seems to be the only way to use it headlessly on my phone since they have no way of doing that in the ChatGPT app right now on Linux.
Does Buzz let you neatly silo that?
I'm struggling between:
"I want to use Hermes with frontier models with a seamless wrapper on my $20 ChatGPT pro subscription so I can go through my Hermes setup for everything including coding"
And
"OpenAI and Anthropic are not ZDR and I am not comfortable sending sensitive personal and family data to them. But I at least want a mobile-friendly remote coding experience with them on my phone and VPS"
It depends on what you mean by sensitive personal data. I actually don't give any agents access to my main email boxes instead they access a business google workspace that is spinning up.
I do separate the agents duties somewhat to try and give them the least privileges they need to do a job, and talk to all of them through buzz, and they talk to each other (occasionally) when they need something they don't have access to.
As mentioned in the post I still do a majority of my coding at my local terminal with just bits and pieces sent to my "dev-agent" through Buzz. This is a next experiment for me - (that's a hyphen!) to offload more coding to agents.
This is awesome! I have a similar setup, with multiple Nanobot Docker instances across three different VMs, along with Matrix and Gitea. I also have a few connected with Obsidian.
I’d love to understand more about your Obsidian setup and Buzz.
The Matrix app on iOS isn’t as good as Telegram.
I have 16 Nanobot instances—4 for my parents and 12 for me and my spouse.
With OpenWebUI and its Knowledge feature, I was thinking of moving some of them there.
I have only one small VPS for ntfy and Matrix. Everything else is self-hosted.
A few years ago, I bought way too much RAM for my Dell server and added more storage just to experiment. Now I’m sitting on a gold mine, with prices having nearly tripled!
I also have a Synology for VM snapshots and backups to Google Drive and B2.
Obsidian is like the shared knowledge base that everyone has access to and that I can easily read/review (although I try not to too often). It has architecture stuff, go-to-market section, howtos (setting up new agents, dns backups, etc.), info on my main projects, writing, etc. It gets synced between my machine and the VPS. I also store in github as a backup occasionally (thx for reminder!).
I use the actual Buzz ios app which I was building on my own but at first glance it appears is now in the iOS store.
29 comments
[ 1.4 ms ] story [ 24.3 ms ] thread> Has it been worth it? For the journey, yes, for the ROI, nope.
It's also nice seeing someone experiment without succumbing to AI psychosis.
I don't live in a world where I need to be constantly reading and replying to emails, so 95% of my inbox is just subscription spam. With development I need to be at the helm to design the planning requirements and actively make decisions before the agent goes off and executes the plan. But I don't need a personal agent for that, I work directly out of codex/claude-code.
It does not cost too much effort to maintain MCP servers. No port forwarding or VPNs required thanks to OpenAI tunnels.
And security wise its quite nice, since you have to activate MCP or give permission sometimes. So each chat is kind of isolated from each-other.
being able to talk to each of the agents via dm (but also in group chats) sounds interesting
does that mean that you have 1 chat per domain specific agent? can you also start multiple sessions/threads or is that not part of the way you interact with them currently?
I've also been building a harness that maintains my apps which I'm hoping to open source.
Hard agree that these things don't have personal ROI, and are actually quite hard to build reliably.
But it's really fun! And having a bot fix a live error is pretty exciting.
Great write up!
How agents know other's capabilities? Shout out and call for help? Is it handled by Buzz?
At home I have Hermes on a VPS with matrix and mnemosyne and it is largely for personal use. While I was excited to try out the seamless Codex wrapper, this leakage was unacceptable compared to a ZDR OpenRouter provider like fireworks.
Where I struggle is now in architecting a clean way to have Hermes separate out personal stuff from coding stuff, and then orchestrate coding through Codex as that seems to be the only way to use it headlessly on my phone since they have no way of doing that in the ChatGPT app right now on Linux.
Does Buzz let you neatly silo that?
I'm struggling between: "I want to use Hermes with frontier models with a seamless wrapper on my $20 ChatGPT pro subscription so I can go through my Hermes setup for everything including coding"
And
"OpenAI and Anthropic are not ZDR and I am not comfortable sending sensitive personal and family data to them. But I at least want a mobile-friendly remote coding experience with them on my phone and VPS"
I do separate the agents duties somewhat to try and give them the least privileges they need to do a job, and talk to all of them through buzz, and they talk to each other (occasionally) when they need something they don't have access to.
As mentioned in the post I still do a majority of my coding at my local terminal with just bits and pieces sent to my "dev-agent" through Buzz. This is a next experiment for me - (that's a hyphen!) to offload more coding to agents.
I’d love to understand more about your Obsidian setup and Buzz.
The Matrix app on iOS isn’t as good as Telegram.
I have 16 Nanobot instances—4 for my parents and 12 for me and my spouse.
With OpenWebUI and its Knowledge feature, I was thinking of moving some of them there.
I have only one small VPS for ntfy and Matrix. Everything else is self-hosted.
A few years ago, I bought way too much RAM for my Dell server and added more storage just to experiment. Now I’m sitting on a gold mine, with prices having nearly tripled!
I also have a Synology for VM snapshots and backups to Google Drive and B2.
I use the actual Buzz ios app which I was building on my own but at first glance it appears is now in the iOS store.