Ask HN: How to handle user file uploads?
hey, i work as an SRE for a company where we allow users to upload media files (e.g. profile picture, attach docs or videos to tasks..the usual). We currently just take a S3 pre-signed URL and let the user upload stuff. Occasionally, limits are set on the <input/> element for file types.
I don't feel this is safe enough. I also feel we could do better by optimizing images on the BE, or creating thumbnails for videos. But then there is the question of cost on AWS side.
Anybody have experience with any of this? I imagine having a big team and dedicated services for media processing could work, but what about small teams?
All thoughts/discussions are welcome.
186 comments
[ 4.3 ms ] story [ 232 ms ] threadI’m honestly surprise this isn’t a value-added capability offered by AWS S3 because it’s such a common need and undifferentiated work.
https://learn.microsoft.com/en-us/azure/defender-for-cloud/d...
Mind you, last time I designed something using this we just used ClamAV - which is pretty easy to develop against even if it is a slight pain to manage on an ongoing basis.
[1] - https://docs.virustotal.com/reference/overview
https://www.cisa.gov/resources-tools/services/malware-next-g...
Do you also handle large files e.g. videos with lambda? Don't you struggle with Lambda limits then?
> a ton of additional complexity with a proven track record of CVEs
become:
> adding it in couldn’t hurt
?
We ultimately scrapped it, if anyone else has any better experience I'd love to hear it.
I've used ClamAV before in web apps, and it was fairly easy to get working - but it has a few big flaws: it's very slow, it uses a huge amount of memory, and detection rates are... honestly, pretty terrible.
I've used Verisys Antivirus API for 2 recently projects, and it works great, a huge improvement over self-hosted ClamAV. There are a couple of other virus scan APIs available, including Scanii[1] which I think has been mentioned on HN before, and I Trend Micro has something too - but their pricing is secret, so I have to assume it's expensive.
[0] https://www.ionxsolutions.com/products/verisys-virus-api [1] https://scanii.com
ClamAV needs lots of memory to work - we needed at least 8GB in one project I worked on (where we also used commercial signatures), 4GB was just about enough for another. The more signatures you have, the more memory it uses.
ClamAV chews through a lot of CPU, and it's so very slow. For example, scanning a 25MB MSI file from my test corpus takes 65 seconds - and that's on a fairly beefy machine!
I've used both of the services linked above, and both seem excellent - detection rates and performance are far superior to ClamAV, and they Just Work(TM).
What is the business case for making the necessary changes?
Good luck.
You will usually want the profile image of user A to be shown to user B. Same for videos and other files - users likely upload them so that others can download them / consume them in some way, right?
I think where you need to start is doing some threat analysis, and proceed from there. Hosting user content can be built out from "very small" to "very big", depending on the particular threat scenario/use cases/your particular userbase. With the description that you are giving, I would say you are more at risk of building an overcomplicated ("oversecured") solution which might compromise UX for the sake of some protection that is not necessarily needed.
If you are a small team, likely you could use an image resizing / video thumbnailing proxy server such as https://www.imgix.com/ https://imgproxy.net/ etc. You generate a signed URL to it and then the service picks up the file from S3 and does $thing to it. https://www.thumbor.org/ is another such tool. There are quite a few.
Re. uploads and downloads - you have quite some options with S3. You can generate the presigned upload URL on the server (in fact: you should do just that), make it time limited and add the Content-Length of the upload to the signed headers - this way the server may restrict the size of the upload. Similarly, access to display the images can be done via a CDN or using low-TTL signed URLs... plenty of things to do.
You can do the math on the ingress to your service (let's say it's EC2), and then the upload from EC2 to S3.
It appears that AWS doesn't charge for ingress [0]. "There is no charge for inbound data transfer across all services in all Regions".
S3 is half a cent per 1000 PUT requests [1], but you were already going to pay that anyway. Storage costs would also be paid anyway with a presigned URL.
You'll have more overhead on your server, but how often do people upload files? It'll depend on your application. Generally, I'd lean towards sending it to your backend until it becomes an issue, but I doubt it ever will. Having the ability to run all the post processing you want and validate it is a big win. It's also just so much easier to test when everything is right there in your application.
[0] https://aws.amazon.com/blogs/architecture/overview-of-data-t...
[1] https://aws.amazon.com/s3/pricing/
The function ensures images aren’t bigger than 10mb, the image is compressed to our sizing, and put into s3.
In my case we allow up to 10mb file upload. For example just testing something right now from my iPhone I selected a 3.7mb image which was uploaded and resized to the "frame" it will go into in the UI and its now 288kb in S3. We have a few "frame" sizes which are consistent throughout our application. Now, my cloudfront is serving 288kb file instead of 3.7mb, which is good for me because I want to avoid the bandwidth costs and users honestly can't tell the difference, and then also we aren't a photo gallery app.
So in my case I need the lambda to resize the image to our "frame" sizes. I wonder if I didn't explain right. I'm ESL so just want to make sure since I'm confused about the your last sentence.
But if you do resizing as well, then ofc you need the Lambda (or some other compute engine).
Images are easy to display but for other media files you will probably need some streaming solution, a player, etc.
We're in the audio hosting and processing space. We still don't have an api though.
For video maybe look into Wistia, Cloudflare, BunnyCdn, etc.
1. Re-encoding the image is a good idea to make it harder to distribute exploits. For example imaging the recent WebP vulnerability. A malicious user could upload a compromised image as their profile picture and pwn anyone who saw that image in the app. There is a chance that the image survives the re-encoding but it is much less likely and at the very least makes your app not the easiest channel to distribute it.
2. It gives a good place to strip metadata. For example you should almost certianlly be stripping geo location. But in general I would recommend stripping everything non-essential.
3. Generating different sizes as you mentioned can be useful.
4. Allows accepting a variety of formats without requiring consumers to support them all. As you just transcode in one place.
I don't know much about the cost on the AWS side, but it seems like you are always at some sort of risk given that if the user knows the bucket name they can create infinite billable requests. Can you create a size limit on the pre-signed URL? That would be a basic line of defence. But you probably also want to validate once the URL expires the data uploaded and decide if it conforms to your expectations (and delete it if you aren't interested in preserving the original data).
Best to sandbox/jail/etc as tightly as possible, and limit the codecs to only what you need. You can configure the ffmpeg builds pretty granularly... default will include too much.
So not encoding is probably the safer way to go for the business.
What if the other user getting attacked is you or another admin on your team?
Now the attacker has admin access and can compromise your servers and “leak everyone’s data” just fine.
I don’t think you’ve thought this through.
If websites get blamed when users reuse passwords, they are definitely getting blamed if you distribute malicious files.
In fact re-encoding probably would have solved this as the server could enforce the expected dimensions and rescale or reject the image.
For anyone wondering:
https://blog.cloudflare.com/uncovering-the-hidden-webp-vulne...
> The vulnerability allows an attacker to create a malformed WebP image file that makes libwebp write data beyond the buffer memory allocated to the image decoder. By writing past the legal bounds of the buffer, it is possible to modify sensitive data in memory, eventually leading to execution of the attacker's code.
Here is a canonical old article on one type of buffer overflow:
http://www.phrack.org/issues/49/14.html#article
(The webp one, iirc, is not a stack buffer overflow, but rather a heap, but that'll give you a general sense of what's going on, even if that article is a bit ... dated)
For years now I use nginx's image filter [1] which handles file resizing quite nicely. Resized images are cached by nginx. For some usecases it works very vell and I no longer need to specify sizes beforehand, you just ask for the size by crafting your url properly.
[1] https://nginx.org/en/docs/http/ngx_http_image_filter_module....
You can hang an event off S3 and have a lambda that does the work / warns you of a bad upload.
Videos will add even more to your AWS bill if you're not careful. Re-encode that 4K cat video as soon as it comes in, or wire up a CDN to do it for you.
And now your server which is doing re-encoding is pwned. Gotta segment the server doing that somehow, or use pledge, capsicum, or seccomp I guess.
Not that OpenBSD is actually unhackable or anything, but I doubt many attackers would guess you're running imagemagick on OpenBSD in your image pipeline.
I rather like it for such use cases; it has the added benefit that it never, ever seems to die. I found a 6.0 machine I setup doing some kind of risky Kafka processing that had an uptime of 6 years the other week (since migrated).
It is running imagemagic to optimize images, create different resolutions and reencode them. It's only open for us to upload manually from our customers though, they can't upload themselves. Input anything, output jpg, very easy to use.
In my mind the chain of events looks like: - Alice uploads an image file to Bob's file server - Charlie's image converter server invokes a local browser with the address of the file on Bob's server, resulting in a .bmp (or .png? a little out of my element here) saved to Charlie's server. - Doris can now pull images from Charlie's server knowing they've been vetted by a major browser.
Does that make sense?
False security, if anything.
A 1080p image with 8 bits per channel would be 6 MB. Real mobile-friendly...
Someone might still be able to get an RCE on them and burn a bit of money, but they certainly wouldn't be able to move laterally.
Including animation in most cases. Otherwise somebody can use a single frame with very long duration that will be reviewed by moderators, then follow that frame with a different frame containing objectionable material which will eventually be shown to people.
Yes, pre-signed URL can have the `Content-Length` set and amazon S3 checks it. However, note that this is true for Amazon S3 but not for others like BackBlaze or R2. Last time I tried, BackBlaze didn't support it.
Famously, the Dangerous Kitten hacking toolset was distributed with the classic zip-in-a-jpeg technique, because imageboards used to not re-encode images.
https://web.archive.org/web/20120322025258/http://partyvan.i...
Yes, if you use the POST method you can set the content-length-range property in your presigned URL form inputs to limit min and max bytes. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authen...
Which immediately makes you vulnerable to zip-bomb attacks.
If you want to be more safe, you first have to do all kinds of checks of the image headers.
Like you stated, an async process using a function would suffice. Previously used ClamAV for this in a private cloud solution, I’ve also used the built in anti-virus support on Azure Blob Storage if you don’t mind multi-cloud, plus an Azure Function has the ability to support blob triggers, which is a nice feature.
The file types scan is relatively simple. You just need a list of known “magic string” header values to do a comparison, and for that you only need a max of 40 bytes of the beginning of the file to do the check (from memory). Depending on your stack, there are usually some libraries already available to perform the matching.
And it goes without saying, but never trust the client, and always generate your own filenames.
https://en.m.wikipedia.org/wiki/List_of_file_signatures
What you instead want is a whitelist: Only allow properly formatted images and videos and ruthlessly reject anything else. I wrote about how I implemented this for my service in another response.
It’s not particularly cheap. But it is fast and flexible and safe.
One of the most important, yet oft ignored on HN, principles of architecting solutions is having an unbiased view of products and then choosing what will fit within an existing organisation’s architecture.
Given the specifications shared, utilising S3 further is the correct advice.
One advantage is that detecting composed files. Take pdf+exe file for example, the library will report something like 70% pdf and 30% pebin file.
I'd suggest existing tooling like `exiftool` to do mimetype detection and metadata stripping.
- upload bucket
- processed bucket
upload bucket has an event triggered on new file upload which triggers a lambda, the lambda will re-encode and do wtv you deem fit and upload to new bucket
your app will use the processed bucket
if you want to get into the nitty grit of filetype abuse, to learn how to possibly detect that good. ange albertini's work on polymorphic filetypes and the ezine poc||gtfo as well as lots of articles by AV code devs are available. its really hard problem and also depends a lot on what program is interpreting the files submitted. if its some custom tool.there might even be unique vectors to take into account. fuzzing and penteting the upload form and any tools handing these files can shed light on those issues potentially.
(edit: fat fingers)
https://developer.massive.io/js-uploader/
Just point it to your local files, as big as you want, and it does the rest. It handles the complexities of S3 juggling and browser constraints. Your users pay nothing to upload, you pay for egress.
Full disclosure: I'm MASV's developer advocate.
Do not serve content from S3 directly.
ISPs often deprioritize traffic from S3, so downloading assets can be very slow. I've seen kbytes/s on a connection that Speedtest.net says has a download speed of 850 mbit. Putting Cloudfront in front of S3 solves that.
I wonder why
I don't do videos yet, but I'm kinda terrified of the idea of putting user-uploaded files through ffmpeg if/when I'll support them.
[1] https://github.com/grishka/Smithereen
[2] https://github.com/imgproxy/imgproxy
I have seen a team struggle for over a month to eliminate NSFW content - avatars - uploaded by a user that lead to their site being demonetised.
*: act as if successful, including processing time, but throw away the upload
The raw RGBA output is then received and converted back into PNG or similar. It was a bit tricky to get everything working without additional allocation and using syscalls triggered by glibc somewhere, but works pretty well now and is fast enough for my use case (around 20ms/item).
I’m a huge fan of building minimal self-contained tools, so all of the C programs statically link in the required parser libs (libavcodec/wuffs/freetype) so the resulting binaries don’t require additional dependencies on the target machine. The python wrapping code is rather straightforward as well and is only like 300 lines of code.
Yeah definitely. Even optimizing the vids. I just spend time writing scripts to convert, in parallel, a massive amount of JPG, PNG, PDFs, mp4 videos and even some HEIC files customers sent of their ID (identity card or passport, basically). I did shrink them all to reasonable size.
The issue is: if you let user do anything, you'll have that one user, once in a while, that shall send a 30 MB JPG of his ID. Recto. Than Verso.
Then the signed contracts: imagine a user printing a 15 pages contract, signing/paraphing every single page, then not scanning it but taking a 30 MB picture, with his phone, in diagonal, in perspective. And sending all the files individually.
After a decade, this represented a massive amount of data.
It was beautiful to crush that data to anywhere from 1/4th to 1/10th of its size and see all the cores working at full speed, compressing everything to reasonable sizes.
Many sites and 3rd party identity verification services (whatever these are called) do put limit on the allowed size per document, which already helps.
In my case I simply used ImageMagick (mogrify), ffmpeg (to convert to x265) and... GhostScript (good old gs command). PDFs didn't have to be searchable for text so there's that too (and often already weren't at least not easily, due to users taking pictures then creating a PDF out of the picture).
This was not in Amazon S3 but basically all in Google Workspace: it was for a SME to make everything leaner, snapper, quicker, smaller. Cheaper too (no need to buy additional storage).
Backups of all the originals, full size, files were of course made too but these shall probably never be needed.
In my case I downloaded everything. Both to create backups (offsite, offline) and to crush everything locally (simply on an AMD 7700X: powerful enough as long as you don't have months of videos to encode).
> Anybody have experience with any of this? I imagine having a big team and dedicated services for media processing could work, but what about small teams?
I did it as a one-person job. Putting limits in place or automatically resizing, right after upload, a 30 MB JPG file which you know if of an ID card to a 3 MB JPG file doesn't require a team.
Same for invoking the following to downsize vids:
My script's logic were quite simple: files above a certain size were candidates for downsizing then downsizing then if the output was successful and took less than a certain amount of time, use that, otherwise keep the original.I didn't bother verifying that the files visually matched (once again: all the originals are available on offline backups in case something went south and some file is really badly needed) but I could have done that too. There was a blog post posted here a few years ago where a book author would visually compare thumbnails of different revisions of his book, to make sure that nothing changed too much between two minor revisions. I considered doing something similar but didn't bother.
Needless to say my client is very happy with the results and the savings.
YMMV but worked for me and worked for my client.
If you can spare the CPU cycles (depending on how you optimize, they can actually be expensive) and if your images will be downloaded frequently, your users will thank you.
I vaguely remember flattening images in some manner so imagemagick wouldn't choke. Something about setting key tones? It's been a while.
One of the major lessons I pulled out of that time is that most library authors will happily fix bugs or add features if you give them money. You can also just ask them to sell you a different license if FOSS ruins your plan.
Users can't upload to your S3 storage because they lack credentials. (It would be dangerous to make it public.) But you can give them access with a specially-generated URL (generated for each time they want to upload). So your server makes a special URL, "signed" with its own authorization. That lets them upload one file, to that specific URL.
(I dunno about anybody else, but I find working with AWS always involves cramming in a lot of security-related concepts that I had never had to think about before. It puts a lot of overhead on simple operations, but presumably is mandatory if you want an application accessible to the world.)
but for certain applications, s3 is simpler to use, especially when you need to scale
Not that I'm asking for an explanation. Just illustrating how much stuff there is to learn for the basic operation of "run this on your computer", even for an experienced developer. (I've been doing this for nearly 40 years.)
I am curious, so I took the opportunity to look up the strange sounding term "pre signed url". But that's just being a dilettante.
It's nost likely more efficient, faster, and cheaper (no need to handle the traffic and hardware to proxy).
Honestly though if this is an authenticated function and you have a small user base… who cares? Is there a reasonable chance at this disrupting any end user services? Maybe it’s not the best way to spend hundreds of hours and thousands of dollars.
Granted you’re an SRE so it’s your job to ideas this. I’d just push back on defaulting to dropping serious resources on a process that might be entirely superfluous for your use case.