Pixeldrain is a project which I started working on during the lunch breaks at school around 6 years ago. Originally it was intended as a screenshotting tool for Linux. Eventually I realized I had more fun working on the backend than the desktop app, so I abandoned the app and opened up the site for all file types instead of just images. Both the backend and frontend are optimized to be as fast as I can possibly make it.
Recently I also responded to a question on HN asking about tech stacks, you can find that one here: https://news.ycombinator.com/item?id=28300657. Following the success of that post I thought I should show my project to everyone on HN.
It's not really meant as cloud storage, but more as a catchnet to prevent abuse.
If you're not paying for pixeldrain your files will always expire 30 days after the last time they're viewed. So even if someone uploads 500 GB the chance is very slim that it will still be there next month. As long as the files stay up they're generating ad revenue and I don't need to remove them.
Bandwidth costs 10 times as much as storage, so I'm more concerned about that :-)
If it becomes a problem I'll do something about it. The storage limit is a recent thing anyway. The site operated 6 years without any limit on storage space.
Nice project. I like the text file and hotlinking features! Unfortunately supporting hotlinking makes end-to-end encryption impossible. Also, the inclusion of third-party tracker/ad scripts and the discussion of possibly adding “more shady ads” is concerning.
For an E2EE alternative with no third party scripts, see https://wormhole.app (Disclosure: I built this)
Oh yea it's probably not a good idea to threaten users with "more shady ads" on the home page.. I'll remove that.
Pixeldrain focuses more on distibuting files to large audiences than one-to-one file transfer. That's why it's not focused on encryption / privacy as much, everything is supposed to be public anyway.
I like Wormhole a lot. If I hadn't built my own alternative I would probably be using it regularly :-)
My hosting provider just decided to turn my server off for no apparent reason. I have no idea why yet, waiting on an answer from support. The timing for this is the absolute worst possible...
Needless to say I'm going to be looking for alternatives again. If anyone knows of good hosting companies with unmetered 10 Gbps let me know.
I did notice one potential flaw in the way your API/website works, though, which is that raw access to the uploaded files is enabled regardless of content type. This allows people to use your website to host scripts and other malicious files that they can reuse to execute XSS attacks against other websites. This is mostly a problem for the other website (which would lack proper security headers) but it could lead to Google blacklisting your website for "hosting malware". Some phone scammers leverage websites like yours that try to provide a simple, cheap service to host their fake virus warnings and such.
I'd personally recommend against allowing text content to be directly accessible with a single GET request from another source.
You've set up some protections against this (rate limiting mode) but that could be circumvented by pointing a browser at the file viewer and placing something heavy on the F5 key. I'd personally also check the origin/referrer headers for files that aren't images or video and block their direct inclusion in some way.
Views are unique per IP address (/48 subnet for IPv6) so continuously refreshing the page wouldn't work. I had problems like this in the past, but rate limiting mode seems to have solved it. I have had little abuse complaints in that regard the last few years.
I'm curious what was the biggest difficulty you encountered when building this?
If I had to guess I would imagine it to be the economics behind finding suitable hosting providers and pricing rather than something purely technical, but having never built anything like it this could be completely off-base. Maybe moderation of user content will become the big one.
Scaling has been tricky. Finding a database system which can handle the load was the hardest part. A single Postgres instance buckled quite early on. Later I tried Cockroachdb but it also was not up for the task. Now I'm running ScyllaDB and it's handling tens of thousands of queries per seconds like a champ.
Content moderation is already a big problem. It's what I spend most of my time on nowadays. There is a lot of crazy shit getting uploaded. Abuse reports are flying in from all corners of the internet. I'm working on automated systems for processing them. The most annoying part is that a lot of copyright management firms don't sign their e-mails with DKIM, so I can't verify that the sender address was not spoofed. This prevents me from fully automating the e-mail responses.
Do you mean the buttons for sharing files? Those are disabled until you have uploaded something. After uploading they become active and the contrast becomes much higher.
I still have a Sia daemon running but it's not doing much. When the site still ran on Sia I had a lot of issues with the scalability of the Sia daemon. While it's still getting better, it's still not as scalable as I'd like. The team has shifted their focus to building a new decentralized internet and the original Sia software has not seen much development since. I might try their new Skynet daemon in the future to see if it scales any better.
I am still upholding your tradition of benchmarking new Sia releases from time to time. If you're curious, here are the latest results: https://siastats.info/benchmarking
Oh, interesting. I knew about Skynet, but I figured they'd still need to develop Sia. Looking at the repo now, I see what you mean. It looks like development on the repo stopped earlier this year.[0]
I didn't realize the benchmarks were still going. That's cool! Thanks for carrying the torch. It's interesting to see how Sia's metrics have evolved over time.
21 comments
[ 1.9 ms ] story [ 55.5 ms ] threadRecently I also responded to a question on HN asking about tech stacks, you can find that one here: https://news.ycombinator.com/item?id=28300657. Following the success of that post I thought I should show my project to everyone on HN.
If you're not paying for pixeldrain your files will always expire 30 days after the last time they're viewed. So even if someone uploads 500 GB the chance is very slim that it will still be there next month. As long as the files stay up they're generating ad revenue and I don't need to remove them.
If it becomes a problem I'll do something about it. The storage limit is a recent thing anyway. The site operated 6 years without any limit on storage space.
For an E2EE alternative with no third party scripts, see https://wormhole.app (Disclosure: I built this)
Pixeldrain focuses more on distibuting files to large audiences than one-to-one file transfer. That's why it's not focused on encryption / privacy as much, everything is supposed to be public anyway.
I like Wormhole a lot. If I hadn't built my own alternative I would probably be using it regularly :-)
An archived version of a non-functional version of the home page is here though: https://web.archive.org/web/20210911035325/https://pixeldrai...
Needless to say I'm going to be looking for alternatives again. If anyone knows of good hosting companies with unmetered 10 Gbps let me know.
I did notice one potential flaw in the way your API/website works, though, which is that raw access to the uploaded files is enabled regardless of content type. This allows people to use your website to host scripts and other malicious files that they can reuse to execute XSS attacks against other websites. This is mostly a problem for the other website (which would lack proper security headers) but it could lead to Google blacklisting your website for "hosting malware". Some phone scammers leverage websites like yours that try to provide a simple, cheap service to host their fake virus warnings and such.
I'd personally recommend against allowing text content to be directly accessible with a single GET request from another source.
You've set up some protections against this (rate limiting mode) but that could be circumvented by pointing a browser at the file viewer and placing something heavy on the F5 key. I'd personally also check the origin/referrer headers for files that aren't images or video and block their direct inclusion in some way.
But thanks for the suggestion, I'll look into it.
If I had to guess I would imagine it to be the economics behind finding suitable hosting providers and pricing rather than something purely technical, but having never built anything like it this could be completely off-base. Maybe moderation of user content will become the big one.
Content moderation is already a big problem. It's what I spend most of my time on nowadays. There is a lot of crazy shit getting uploaded. Abuse reports are flying in from all corners of the internet. I'm working on automated systems for processing them. The most annoying part is that a lot of copyright management firms don't sign their e-mails with DKIM, so I can't verify that the sender address was not spoofed. This prevents me from fully automating the e-mail responses.
Are you still using any kind of decentralized storage technology on the backend?
I still have a Sia daemon running but it's not doing much. When the site still ran on Sia I had a lot of issues with the scalability of the Sia daemon. While it's still getting better, it's still not as scalable as I'd like. The team has shifted their focus to building a new decentralized internet and the original Sia software has not seen much development since. I might try their new Skynet daemon in the future to see if it scales any better.
I am still upholding your tradition of benchmarking new Sia releases from time to time. If you're curious, here are the latest results: https://siastats.info/benchmarking
I didn't realize the benchmarks were still going. That's cool! Thanks for carrying the torch. It's interesting to see how Sia's metrics have evolved over time.
[0] https://gitlab.com/NebulousLabs/Sia
And Skynet was forked into a different repo: https://gitlab.com/SkynetLabs/skyd