> Don’t expose the proxy to the world or attackers will be able to interact with your home devices
Or use it to get stuff on the internet that they don't want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn't have a NAT router. Just modem straight to Linux box.
One day I checked my Apache logs to see if my extremely low traffic web site had any visitors.
First impression: "What the hell is all this traffic? This is orders of magnitude more than I should have!".
Second impression: "Why the hell is someone trying to get horse_fucking.mov from my server?".
Third impression: "That didn't give a 404. Why is my server actually providing horse_fucking.mov?".
(No, the orders of magnitude extra traffic were not all people getting horse_fucking.mov. There was a whole plethora of bestiality titles, plus a lot of other porn that you would not want to fetch directly from your own IP address either out of embarrassment if you got caught or because it was almost certainly illegal).
> Or use it to get stuff on the internet that they don't want to get directly themselves
That's a good point, another one is people using your proxy to attack other servers.
I find the proxy area to be interesting but something I'm unlikely to use for business, you are quoting some examples that made me laugh, and I'd certainly laugh if it happens to me, but I know I will remove the public access, if I were involved in business with this, the story would be different.
I once accidentally left privoxy open on one of my Linux servers and misconfigured ACL. A couple hours later it started getting hit with loads of traffic. A Google search quickly revealed that my ip:port ended up on several “free proxy” lists. They probably found it on Shodan or something. I quickly fixed acl to drop the traffic, but IIRC my server persisted on the free proxy lists for a couple days afterwards.
That was the first hand experience that confirmed my suspicions of what those “free proxies” really are.
Just using non-default settings for your proxy would be enough to thwart 99.9% of automated attacks, and you wouldn't have to rely on a hand-rolled proxy.
I evaluated Squid which was my first option, due to the existing code on the project I integrated the proxy, it wasn't that straightforward to interact with Squid.
I'm still thinking to use several Pies and it's simpler to evolve the custom-made proxy to support that, if you see the Github repo, it's pretty simple.
Last, getting a Scala project to run on the old Pi was a nice experiment worth trying.
I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you respect the websites in terms of not hitting them multiple times a minute (or only once an hour which I do) then you’ll very seldom get banned.
It's interesting how simple is to get small services out of the cloud now.
I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel.
Certainly it's something worth implementing if I ever start getting more traffic.
I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine.
For this article, does the OP feel they need the proxy due to scraping rapidly?
The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi.
The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
In the post I detailed that I run a service (https://cazadescuentos.net) which is a browser extension to look for discounts.
Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go.
So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
Don't forget to set gpu_mem in config.txt. It looks like you are running with the default, which is 64MB for the GPU. I'm guessing you are not using the GPU in this application and should reduce this value to the bare minimum (which I don't recall, sorry)
They all do, in fact the GPU is required to boot the CPU. In your free output it is clear that 64MB is assigned to the GPU, as per default, because it says you only have 432M total (available to the CPU).
You can gain most (not all) of that 64MB back for free, simply by running `echo gpu_mem=1 | sudo tee -a /boot/config.txt && sudo reboot`
(Pretty sure that the minimum is actually more than 1, but also that it will accept that and set it to the minimum... check the docs if you need to be sure)
Have you looked at running wireguard on your server and connecting to it from the pi. Then you have a vpn between server and the pi where you can send data in any protocol
I do this to expose services to the internet in a limited way. Wireguard link between a pi running, say, octoprint, and a vps. Then, the vps is running nginx or caddy as a reverse proxy over that wireguard link, giving me https access and even letting me add basic auth if I want another layer of authentication.
It's a shame that client-certificates is implemented in such a clunky way in most browsers and operating systems, because that could also be an easy way to achieve this even without installing anything.
The first time I hit an issue, autossh was the first proposed solution but I preferred to not include more dependencies. Also, I believed it should be possible to handle this with plain ssh + systemd.
Do you have any experiences with autossh/ssh to share?
I setup an Openvpn server on my LAN, with username/password credentials so I can access my LAN from anywhere. I have multiple vpn configurations for my cell phone/laptop to either map only 192.x through the VPN or all traffic through it. It's very very useful, and seems like a much more secure option in my opinion.
Is there a solid solution to the SD card problem? Every Rpi I've tried to use as a server eventually kills its SD card, probably as running an os on SD is destined for failure with write wear. I'm guessing using SD for boot only and then using a home NAS would be an option, but I've not come across a dummies guide to making a Rpi reliable long term.
Years ago a friend mentioned her company would spring for very expensive industrial grade SD cards. My company up and refuses to have anything to do with SD cards.
I used to run the boot partition from the SD card and everything else over a mounted USB flash drive. That seemed to work quite ok for a long time, but maybe I just got lucky.
Spending a little extra on good quality SD cards helps both durability and performance.
Also making a backup image of the SD card once configured and using a NAS or external disk for important info helps mitigate the data loss risk, and gives you a quick path to restore onto a fresh SD.
With that said, I have a few raspberry pis that have been running for years without SD failures.
46 comments
[ 3.1 ms ] story [ 92.8 ms ] threadOr use it to get stuff on the internet that they don't want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn't have a NAT router. Just modem straight to Linux box.
One day I checked my Apache logs to see if my extremely low traffic web site had any visitors.
First impression: "What the hell is all this traffic? This is orders of magnitude more than I should have!".
Second impression: "Why the hell is someone trying to get horse_fucking.mov from my server?".
Third impression: "That didn't give a 404. Why is my server actually providing horse_fucking.mov?".
(No, the orders of magnitude extra traffic were not all people getting horse_fucking.mov. There was a whole plethora of bestiality titles, plus a lot of other porn that you would not want to fetch directly from your own IP address either out of embarrassment if you got caught or because it was almost certainly illegal).
That's a good point, another one is people using your proxy to attack other servers.
I find the proxy area to be interesting but something I'm unlikely to use for business, you are quoting some examples that made me laugh, and I'd certainly laugh if it happens to me, but I know I will remove the public access, if I were involved in business with this, the story would be different.
Thanks for sharing.
That was the first hand experience that confirmed my suspicions of what those “free proxies” really are.
That's when I realized from the scrolling log messages that even residental ADSL was already under heavy probing from around the world.
I decided I didn't have the time or the stomach to try to deal with that at home, and there wasn't much I could do, so I disabled the logging.
I'm still thinking to use several Pies and it's simpler to evolve the custom-made proxy to support that, if you see the Github repo, it's pretty simple.
Last, getting a Scala project to run on the old Pi was a nice experiment worth trying.
I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel.
Certainly it's something worth implementing if I ever start getting more traffic.
Thanks for sharing.
For this article, does the OP feel they need the proxy due to scraping rapidly?
The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
In the post I detailed that I run a service (https://cazadescuentos.net) which is a browser extension to look for discounts.
Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go.
So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
Just ssh -D 12345 raspberry and then just use localhost:12345 as the scrapers SOCKS5 proxy. No need to involve a http proxy.
EDIT: AAh, ssh port is not accessible from outside and the raspi does a reverse connect.
A reason to writing the custom-made proxy is mostly to evolve it supporting several pies, the tunnel being the simpler workaround for now.
Also, I'm a coder and wanted to try getting Scala running on my old pi.
What is total RSS of all your proxy processes?
You can gain most (not all) of that 64MB back for free, simply by running `echo gpu_mem=1 | sudo tee -a /boot/config.txt && sudo reboot`
(Pretty sure that the minimum is actually more than 1, but also that it will accept that and set it to the minimum... check the docs if you need to be sure)
Mainly the custom-made proxy allows scaling the app to several pies if I ever need to.
https://www.harding.motd.ca/autossh/
Do you have any experiences with autossh/ssh to share?
Also making a backup image of the SD card once configured and using a NAS or external disk for important info helps mitigate the data loss risk, and gives you a quick path to restore onto a fresh SD.
With that said, I have a few raspberry pis that have been running for years without SD failures.
Also, make sure swap is not enabled on the SD