I build my own NAT instances from Debian Trixie with Packer on AWS. AWS built-in NAT Gateways use an absurdly outdated and end-of-life version of Amazon Linux and are ridiculously expensive (especially traffic).
The bash configuration is literally a few lines:
cat <<'EOF' | sudo tee /etc/sysctl.d/99-ip-forwarding.conf > /dev/null
net.ipv4.ip_forward=1
EOF
sudo sysctl --system
sudo iptables -t nat -A POSTROUTING -o ens5 -j MASQUERADE
sudo iptables -F FORWARD
sudo iptables -A FORWARD -i ens5 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -o ens5 -j ACCEPT
sudo iptables-save | sudo tee /etc/iptables/rules.v4 > /dev/null
Change ens5 with your instance network interface name. Also, VERY IMPORTANT you must set source_dest_check = false on the EC2 NAT instances.
Also, don’t assign a EIP to your EC2 NAT instances (unless you absolutely must persist a given public IP) as that counterintuitively routes through public traffic. Just use a auto-assigned public IP (no EIP).
NAT instance with EIP
- AWS routes it through the public AWS network infrastructure (hairpinning).
- You get charged $0.01/GB regional data transfer, even if in the same AZ.
That's what you did before AWS had the "NAT Gateway" managed service. It's literally called "NAT Instance" in current AWS documentation, and you can implement it in any way you wish. Of course, you don't have to limit yourself to iptables/nftables etc. OPNsense is a great way to do a NAT instance.
> Also, don’t assign a EIP to your EC2 NAT instances (unless you absolutely must persist a given public IP) as that counterintuitively routes through public traffic. Just use a auto-assigned public IP (no EIP).
Could you point me to somewhere I can read more about this? I didn't know there was an extra charge for using an EIP (other than for the EIP itself).
For company hosting cloud solutions gets you the various compliance stuff for free which can be worth it if you're not too large, and of course faster turnaround if you need to get a product out.
For personal a cheap vps will end up costing around the same as something you can do on your own, without the risk of messing up your machine/network from a vulnerable endpont
I think AI coding is another part of why this is seeing a resurgence. It’s a lot quicker to build quick and dirty scripts or debug the random issues that come up self hosting.
You’re misunderstanding the value of cloud computing, which should be obvious at this point. But I’ll go ahead and re-explain it for you.
Businesses need to maintain focus and allocate resources toward delivering their core product.
Software is highly profitable and even with inflated cloud computing costs, it makes some level of sense to not over-optimize and spread teams thin reinventing the wheel.
If I can deliver my product or feature to the market 20% faster that’s going to make more money than if I optimize my cloud infrastructure costs to save 50%.
As a business owner I don’t want to have to hire high-paid specialists who understand the deep intricacies of data center infrastructure, I want to be able to pay people with highly common skills who can quickly translate my business logic to working software.
I've worked at four tech companies and never saw a hint of IPv6 (except for some tests that verified that third-party networking code accepted that address family).
Instead I played with IPv6 at home to make sure I understood it well enough should it ever come up at work. We'll see!
I can't believe people are paying these crazy amounts for what is basically a fleet of firewalls. What is the difficulty in running VMs with nftables rules?
As an OG networking person, developer, and Linux user, the state of modern dev culture just makes me sad.
Modern devs are helpless in the face of things I taught myself to do in a day or two when I was fourteen, and they’re paralyzed with terror at the thought of running something.
It’s “hard” goes the cliche. Networking is “hard.” Sys admin is “hard.” Everything is “hard” so you’d better pay an expert to do it.
Where do we get these experts? Ever wonder that?
It’s just depressing. Why even bother.
It really makes me worry about who will keep all this stuff running or build anything new in the future if we are losing not only skills but spine and curiosity. Maybe AI.
Yeah, I just use a VPS box I pay $20/year for. Only the most basic config goes on this machine. Basically load is 0.1 , and has no data.
Then I run my stuff locally.
And then I use ssh tunneling to forward the port to localhost of the remote machine. Its a unit file, and will reconstruct the tunnel every 30s if broken. So at most 30s downtime.
I think it might be as simple as ipv4 is just nicer to look at…maybe we should have just done “ipv5” and added another block. Eg 1.1.1.1.1. I know its stupid, but ipv6 addresses are just so hard to remember and look at that I think its just human nature to gravitate towards the simplicity of ipv4.
Eh. Judging by the large negative response I think the point might still have a point. Can you quickly rattle out ipv6 cidr blocks when you are setting up network configs? Most can grok and crank on ipv4 no problem. But for ipv6…its to the calculator. Not saying its a valid reason. Just saying ppl are lazy sooo if you want something adopted you might have to lean into lazy.
We did this at OpsLevel a few years back. Went from AWS managed NAT gateway to fck-nat (Option 1 in the article).
It’s a (small) moving part we now have to maintain. But it’s very much worth the massive cost savings in NATGateway-Bytes.
A big part of OpsLevel is we receive all kinds of event and payload data from prod systems, so as we grew, so did our network costs. fck-nat turned that growing variable cost into an adorably small fixed one.
For anyone else who is super confused as to wtf this is about: 1) it's not "NAT Gateway " but rather "The AWS service called NAT Gateway" and 2) it's not "self-hosting" but "hosting in EC2", in the same sense that "running postgresql on an EC2 instance" wouldn't be "self hosting aurora".
The article seems to perpetuate one of those age old myths that NAT has something to do with protection.
Yes, in a very superficial sense, you can't literally route a packet over the internet backwards to a host behind NAT without matching a state entry or explicit port forwarding. But implementing NAT on it's own says nothing about the behavior of your router firewall with regards to receiving Martians, or with regards to whether the router firewall itself accepts connections and if the router firewall itself isn't running some service which causes exposure.
To actually protect things behind NAT you still need firewall rules and you can keep those rules even when you are not using NAT. Thus those rules, and by extension the protection, are separable from the concept of NAT.
This is the kind of weird argument that has caused a lot of people who hadn't ever used IPv6 to avoid trying it.
Fwiw, the solutions mentioned here don't seem to properly secure the kernel's network stack against common attacks (rp_filter, accept_redirects, accept_source_route, syncookies, netfilter rules, etc). Ask your local security guru to harden the instance before deploying.
I'm not to much into networks, although I've been sysadmin my vps for years.
why I would need a NAT Gateway? it's not enough with a good set of rules of ufw or similar software?
> Society would have you believe that self hosting a NAT Gateway is “crazy”, “irresponsible” and potentially even “dangerous”.
I just can't take articles seriously when they lead with these kind of claims and then don't back them up. Typically to give their articles some sort of justification and/or weight. Did not bother to read the rest.
Am I the only one who read this and thought, "doesn't everyone self host a NAT gateway?"
Mine's in the living room, it says TP Link.
More seriously, NAT is fun and all but it can introduce unexpected behaviors that wouldn't exist in a firewall that doesn't do translation. Less is more.
32 comments
[ 24.1 ms ] story [ 1028 ms ] threadThe bash configuration is literally a few lines:
Change ens5 with your instance network interface name. Also, VERY IMPORTANT you must set source_dest_check = false on the EC2 NAT instances.Also, don’t assign a EIP to your EC2 NAT instances (unless you absolutely must persist a given public IP) as that counterintuitively routes through public traffic. Just use a auto-assigned public IP (no EIP).
That's what you did before AWS had the "NAT Gateway" managed service. It's literally called "NAT Instance" in current AWS documentation, and you can implement it in any way you wish. Of course, you don't have to limit yourself to iptables/nftables etc. OPNsense is a great way to do a NAT instance.
Could you point me to somewhere I can read more about this? I didn't know there was an extra charge for using an EIP (other than for the EIP itself).
No your service does not need the extra .099% availability for 100x the price...
Make your own VPN while you are at it, wireguard is basically the same config.
For personal a cheap vps will end up costing around the same as something you can do on your own, without the risk of messing up your machine/network from a vulnerable endpont
Businesses need to maintain focus and allocate resources toward delivering their core product.
Software is highly profitable and even with inflated cloud computing costs, it makes some level of sense to not over-optimize and spread teams thin reinventing the wheel.
If I can deliver my product or feature to the market 20% faster that’s going to make more money than if I optimize my cloud infrastructure costs to save 50%.
As a business owner I don’t want to have to hire high-paid specialists who understand the deep intricacies of data center infrastructure, I want to be able to pay people with highly common skills who can quickly translate my business logic to working software.
Instead I played with IPv6 at home to make sure I understood it well enough should it ever come up at work. We'll see!
For someone just getting started with networking and learning things, this seems rhe best way to go forward.
It is a damn service, which is defined as "you pay someone to do it".
Why state this as absolute fact? Seems a bit lacking in epistemic humility.
Modern devs are helpless in the face of things I taught myself to do in a day or two when I was fourteen, and they’re paralyzed with terror at the thought of running something.
It’s “hard” goes the cliche. Networking is “hard.” Sys admin is “hard.” Everything is “hard” so you’d better pay an expert to do it.
Where do we get these experts? Ever wonder that?
It’s just depressing. Why even bother.
It really makes me worry about who will keep all this stuff running or build anything new in the future if we are losing not only skills but spine and curiosity. Maybe AI.
Then I run my stuff locally.
And then I use ssh tunneling to forward the port to localhost of the remote machine. Its a unit file, and will reconstruct the tunnel every 30s if broken. So at most 30s downtime.
Then nginx picks it up.
Repeat after me: NAT is not a firewall. And we need to stop pretending it is.
Death , taxes and transfer fees
It’s a (small) moving part we now have to maintain. But it’s very much worth the massive cost savings in NATGateway-Bytes.
A big part of OpsLevel is we receive all kinds of event and payload data from prod systems, so as we grew, so did our network costs. fck-nat turned that growing variable cost into an adorably small fixed one.
Yes, in a very superficial sense, you can't literally route a packet over the internet backwards to a host behind NAT without matching a state entry or explicit port forwarding. But implementing NAT on it's own says nothing about the behavior of your router firewall with regards to receiving Martians, or with regards to whether the router firewall itself accepts connections and if the router firewall itself isn't running some service which causes exposure.
To actually protect things behind NAT you still need firewall rules and you can keep those rules even when you are not using NAT. Thus those rules, and by extension the protection, are separable from the concept of NAT.
This is the kind of weird argument that has caused a lot of people who hadn't ever used IPv6 to avoid trying it.
I just can't take articles seriously when they lead with these kind of claims and then don't back them up. Typically to give their articles some sort of justification and/or weight. Did not bother to read the rest.
Mine's in the living room, it says TP Link.
More seriously, NAT is fun and all but it can introduce unexpected behaviors that wouldn't exist in a firewall that doesn't do translation. Less is more.