Hi I am Sagar, We just open-sourced a complete framework to build an AI-powered telephony agent that can handle both inbound and outbound calls—using Python, SIP, and cloud LLMs like OpenAI or Gemini.
You can use it to create smart appointment bots, voice feedback collectors, or even enterprise IVR systems. It’s modular (plug in your SIP provider or AI model), production-ready, and extensible for real-time workflows.
Features include:
SIP & VoIP call handling (Twilio, Plivo, etc)
LLM-integrated AI agent (customizable prompt & tools)
I'm still getting to grips with all this AI stuff but I do know SIP n RTP rather well. Your project looks ideal for me to play with.
My initial project will be to replace my aging parent's and my home FreePBX IVR with something a bit more useful. At the moment it requests you press 1 to make the phones ring and that drops pretty much all unwanted calls but that will soon be automatically defeated.
I do also have some rather more commercial ideas but baby steps. Mind you a RPi based drop in box with a decent auto attendant for say £100 hardware, open source software and some fettling time seems a pretty good opportunity. You could go for somewhat cheaper hardware. The UK is being migrated away from copper A and B wires to what is called SOGEA and/or FTTP. Basically its going to be all VoIP for telephony from pretty much now on. The good old days of an analogue handset that is powered by the line will mostly go away within a few years. On the bright side our signalling was bloody weird and SIP is nearly eternal!
That is a fantastic idea. We’ve tested it on low-resource hardware. It’s SIP-agnostic and modular, making it ideal for home or SMB setups. We would love to highlight your RPi implementation if you publish it.
GitHub: https://github.com/videosdk-live/agents
The question is totally fair, but it's unreasonable (imho) to expect owners of this project to have to answer. We are looking at < 500 lines of python, mostly just gluing together SIP and agents.
My reaction was slightly different: how many companies selling this (meager) service at a high premium will go out of business now that it's free/open?
That is one of our goals. If a solution is under 500 lines of Python, you should not need to pay $499 per month for it. We want to lower the barrier for developers and businesses to build their own voice agents.
We just hook up generators to bikes so that the former phone workers can now power the AI thats replaced them. This will eventually be a cheaper alternative to the current power grid as ai electricity consumption increases.
A huge number of these jobs will go yes. A call centre supervisor may be safe taking escalation calls that need a human and so on but the masses will be replaced by this kind of tech.
However what will actually happen is society will use these people to brick lay for houses, care for the elderly or something else. That's honestly a good thing for society as we have massive shortages there, and not a bad thing for the individuals as a whole.
The same technology can also enable businesses that never had live phone support to offer it affordably. The goal is augmentation and access, not mass replacement.
I'll take an alternative view of someone else's comment. No offense, but how much shittier will my customer service experience with my power company or ISP be because of this?
Thanks for the mention. Curious—what challenges are you finding with Pipecat that you're hoping something else (like https://github.com/videosdk-live/agents) might fix?
Always looking to improve based on real gaps devs are facing.
Valid concern. That’s why we focused on responsible use cases such as appointment bots, IVRs, or call reminders, not spam. Our project is open-source, modular, and designed for ethical, contextual automation.
we tried to build something similar lately for outbound calls (for simple reminders to partners) and faced massive issues using gpt-4o-realtime-audio. Noise detection, turn detection, random telephony issues (we were using Twilio too), prompt not holding together, and more.
We dropped the project because it would have resulted in a terrible experience for the person on the other side of the phone. Building these things is non trivial.
The plan would have been to A/B test and see what the response would have been (watching NPS and business metrics uplift). Human handoff was always the plan in case things got too tricky for the LLM to handle.
I see some hostility here towards this project and while I share many concerns, it is very naive to think that these services won’t be massively leveraged going forward. An AI agent can handle things as well as humans (not in our case but there are good services out there, i.e. Parloa) and the key elements are the same as all the other agentic based workflows:
- narrow use cases
- human in the loop ready to pick up/steer/correct
we will see a lot more of
this and as LLM capabilities improve, it will only get better - it is inevitable at this point and might (_might_) result in a better experience for customers in some cases.
Nevertheless I also see the possibility that we will go full circle and we will always reach for a human, maybe showing up in person in a physical office to make sure cases or requests are handled well… or not :-)
17 comments
[ 5.2 ms ] story [ 202 ms ] threadYou can use it to create smart appointment bots, voice feedback collectors, or even enterprise IVR systems. It’s modular (plug in your SIP provider or AI model), production-ready, and extensible for real-time workflows.
Features include:
SIP & VoIP call handling (Twilio, Plivo, etc)
LLM-integrated AI agent (customizable prompt & tools)
FastAPI-based server for routing and control
Plugins for STT, TTS, sentiment analysis
Support for Agent2Agent and MCP protocols
GitHub Repo:https://github.com/videosdk-live/agents Full Blog: https://www.videosdk.live/blog/ai-telephony-agent-inbound-ou...
Would love feedback from anyone working with telephony, LLMs, or real-time automation!
My initial project will be to replace my aging parent's and my home FreePBX IVR with something a bit more useful. At the moment it requests you press 1 to make the phones ring and that drops pretty much all unwanted calls but that will soon be automatically defeated.
I do also have some rather more commercial ideas but baby steps. Mind you a RPi based drop in box with a decent auto attendant for say £100 hardware, open source software and some fettling time seems a pretty good opportunity. You could go for somewhat cheaper hardware. The UK is being migrated away from copper A and B wires to what is called SOGEA and/or FTTP. Basically its going to be all VoIP for telephony from pretty much now on. The good old days of an analogue handset that is powered by the line will mostly go away within a few years. On the bright side our signalling was bloody weird and SIP is nearly eternal!
My reaction was slightly different: how many companies selling this (meager) service at a high premium will go out of business now that it's free/open?
GitHub: https://github.com/videosdk-live/agents
https://docs.videosdk.live/ai_agents/voice-agent-quick-start
https://docs.videosdk.live/ai_agents/sip
However what will actually happen is society will use these people to brick lay for houses, care for the elderly or something else. That's honestly a good thing for society as we have massive shortages there, and not a bad thing for the individuals as a whole.
Always looking to improve based on real gaps devs are facing.
GitHub: https://github.com/videosdk-live/agents
We dropped the project because it would have resulted in a terrible experience for the person on the other side of the phone. Building these things is non trivial.
The plan would have been to A/B test and see what the response would have been (watching NPS and business metrics uplift). Human handoff was always the plan in case things got too tricky for the LLM to handle.
I see some hostility here towards this project and while I share many concerns, it is very naive to think that these services won’t be massively leveraged going forward. An AI agent can handle things as well as humans (not in our case but there are good services out there, i.e. Parloa) and the key elements are the same as all the other agentic based workflows:
- narrow use cases
- human in the loop ready to pick up/steer/correct
we will see a lot more of this and as LLM capabilities improve, it will only get better - it is inevitable at this point and might (_might_) result in a better experience for customers in some cases.
Nevertheless I also see the possibility that we will go full circle and we will always reach for a human, maybe showing up in person in a physical office to make sure cases or requests are handled well… or not :-)