Show HN: I built an ISP infrastructure emulator from scratch with a custom vBNG (aether.saphal.me)
Aether is a multi-BNG (Broadband Network Gateway) ISP infrastructure lab built almost from scratch that emulates IPoE IPv4 subscriber management end-to-end. It supports IPoE/Ipv4 networks and runs a python-based vBNG with RADIUS AAA, per-subscriber traffic shaping, and traffic simulation emulated on Containerlab. It is also my first personal networking project, built roughly over a month.
Motivations behind the project
I'm a CS sophomore. About three years ago, I was assigned, as an intern, to build a OSS/BSS platform for a regional ISP by myself without mentoring. Referencing demo.splynx.com , I developed most of the BSS side ( bookkeeping, accounting, inventory management ), but, in terms of networking, I managed to install and setup RADIUS and that was about it. I didn't have anyone to mentor me or ask questions to, so I had given up then.
Three years later, I decided to try cracking it again. This project is meant to serve as a learning reference for anyone who's been in that same position i.e staring at closed-source vendor stacks without proper guidance. This is absolutely not production-grade, but I hope it gives someone a place to start.
Architecture overview
The core component, the BNG, runs on an event-driven architecture where state changes are passed around as messages to avoid handling mutexes and locks. The session manager is the sole owner of the session state. To keep it clean and predictable, the direBNG never accepts external inputctly. The one exception is the Go RADIUS CoA daemon, which passes CoA messages in via IPC sockets. Everything the BNG produces(events, session snapshots) gets pushed to Redis Streams, where the bng-ingestor picks them up, processes them, and persists them.
Simulation and meta-configs
I am generating traffic through a simulator node that mounts the host's docker socket and runs docker exec commands on selected hosts. The topology.yaml used by Containerlab to define the network topology grows bigger as more BNG's and access nodes are added. So aether.config.yaml, a simpler configuration, is consumed by the configuration pipeline to generate the topology.yaml and other files (nginx.conf, kea-dhcp.conf, RADIUS clients.conf etc.)
Known Limitations
- Multiple veth hops through the emulated topology add significant overhead. Profiling with iperf3 (-P 10 -t 10, 9500 MTU, 24 vCPUs) shows BNG→upstream at ~24 Gbit/s, but host→BNG→upstream drops to ~3.5 Gbit/s. The 9500 MTU also isn't representative of real ISP deployments. This gets worse when the actual network is reintroduced capping my throughput to 1.6 Gbits/sec in local. - The circuit ID format (1/0/X) is non-standard. I simplified it for clarity. - No iBGP or VLAN support. - No Ipv6 support. I wanted to target IPv4 networks from the start to avoid getting too much breadth without a lot of depth.
Nearly everything I know about networking (except some sections from AWS) I learned building this. A lot was figured out on the fly, so engineers will likely spot questionable decisions in the codebase. I'd genuinely appreciate that feedback.
Questions
- Currently, the circuit where the user connects is arbitrarily decided by the demo user. In a real system with thousands of circuits, it'd be very difficult to properly assess which circuit the customer might connect to. When adding a new customer to a service, how does the operator decide, based on customer's location, which circuit to provide the service to ?
15 comments
[ 2.7 ms ] story [ 30.4 ms ] threadA better and UX-friendly implementation would have been Netbox + aether.config.yaml -> configuration pipeline -> topology.yaml + <other generated files>.
On the other hand, building even a tiny subset but doing it yourself from scratch is a great way to learn. I made a very poor man's VM image builder for HyperV years back because Packer didn't have a builder for it at the time and that was a pretty interesting experience. Finally grokked the Windows object model and even though I still don't use it, I at least no longer jeer at PowerShell.
I'm interested in the answer to your question, too, but as a customer of an ISP. I don't work for one. I was the first owner of my house and when they hooked me into their network, whoever did messed up my neighbors badly, putting them on the wrong circuit and bleeding noise into adjacent neighborhoods. For three years, complaint calls would get our network cut by third-party contractors with no warning, then we'd have to call and get it reconnected. I don't know how they're supposed to do it, but know it can cause quite a mess when they do it wrong.
Mininet did help me a lot during the initial phases. The main reason I made the switch to containernet(mininet-fork with containers ) and then to containerlab was because I wanted to run an actual NOS image as part of my topology. That was really what pushed me to try and switch to other options.
Yup, its a different experience. Sometimes, you end up learning something you never even intended to.
As for the circuit planning, I'd guess that they have circuits map on something like Netbox and using a intermediary system that maps a customer's location to the nearest circuit. Though, I don't know how they handle the optimization side of it to prevent cases like what happened to your neighbour.
(or whichever operators group best fits your area. i only subscribe to NANOG, so cant speak to the activity/friendliness of the other groups. you can find a pretty comprehensive list here: https://nanog.org/resources/organizations-our-community/)
I plan to take all the feedback I can this week, and work on them on spring break.
I'm not exactly sure what you're asking, but port allocation is, depending on the ISP's deployment model, either going to be fixed at the time the infrastructure was built, or whoever is doing the last metre install will choose a random available port on the switch. The subscriber will be assigned to that port in the RADIUS or equivalent database, and the BNG will query the subscriber based on DHCP Option 82 port information added by the switch. You could also map the subscriber based on MAC address, but this doesn't really work unless you don't support customer provided equipment on their end.
https://github.com/codelaboratoryltd/bng-edge-infra
https://github.com/veesix-networks/osvbng
Meanwhile, I also found another project that works with eBPF-accelerated BNGs for k8s edge deployments. https://github.com/codelaboratoryltd/bng (BUSL-1.1 license though)
Thank you for listing me the projects. Really helped me out!!
Good job!
<sniff>
I am surprised the author did not mention or uses Software Defined Networking (SDN), Openflow or P4 (programming language for programmable switches) or the mininet simulator. He must have skipped reading the scientific literature even though he is a computer science sophomore?
I programmed and build one of the very first ISP hardware and software systems in 1987-1997 when we connected the first submarine link between the US and Europe in Amsterdam.
Google switched 50% of the internet that they owned in 2012 to SDN and Openflow [1]. I'm sure they progressed to P4 and more recent SDN controllers since then. They build the Google Fiber ISP[5] with SDN. Cloudflare also uses SDN when last I checked. A majority of the internet has moved to SDN (there are many versions.
The author built his simulation on legacy systems mostly from the Telecom world, an alternate reality distinct from the real internet and acces providers we call ISPs. Telecom systems are about surveillance and monetizing the free internet.
You can query the US ISPs on the Nanog mailing list, there are similar social media for the European, Asian and other ISPs on other continents. Beware that those are biased to Telecom as well as Tier 1 network operators and less to ISP access providers.
I do not think we should continue with the current implementation of the internet. I think we should start deploying the true internet (decentralized, peer to peer) standard and expand it to the Enernet standards of the near future: every building a router (switch) and fiber optic and electricity cables to their peers; their closest neighbors. If every building has peer connections than you are connected all the way to the internet exchanges without need for Tech Bros, Government, Telecom, ISP or Tier 1 network oligopolies. True internet [3], true Enernet [4].
[1] OpenFlow @ Google - Urs Hoelzle, https://www.youtube.com/watch?v=VLHJUfgxEO4
[2] The Future of Networking, and the Past of Protocols - Scott Shenker https://www.youtube.com/watch?v=YHeyuD89n1Y
[3] Fiberhood White Paper https://www.researchgate.net/profile/Merik-Voswinkel/publica...
[4] Enernet - Bob Metcalfe https://www.youtube.com/watch?v=axfsqdpHVFU
[5] Google Fiber build "Fiberhoods" but my own Enernet ISP Fiberhood had trademarked that name before in 2011.
P4 was a great idea, but there's not much hardware that supports it.
fd.io / vpp is an impressive stack for software-only routing. Like all SW-only solutions, it suffers from high power consumption and packet rate variability. At today's packet rates, you always have to ask 'how many CPU instructions / cycles are required to perform this or that function per packet'.