It gets a lot of binary file formats wrong for me out-of-the-box. I think it needs to be a bit more effective before we can truly determine the effectiveness of such exploits.
It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.
Well, what they used this for at Google was apparently scanning their user's files for things they shouldn't store in the cloud. Probably they don't care much about MIDI.
As somebody who's dealt with the ambiguity of attempting to use file signatures in order to identify file type, this seems like a pretty useful library. Especially since it seems to be able to distinguish between different types of text files based on their format/content e.g. CSV, markdown, etc.
Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?
Nothing is ever simple. Even for the most basic .txt files it’s still useful to know what the character encoding is (utf? 8/16? Latin-whatever? etc.) and what the line format is (\n,\cr\lf,\n\lf) as well as determining if some maniac removed all the indentation characters and replaced them with a mystery number of spaces.
Then there are all the container formats that have different kinds of formats embedded in them (mov,mkv,pdf etc.)
Sometimes a file has no extension. Other times the extension is a lie. Still other times, you may be dealing with an unnamed bytestring and wish to know what kind of content it is.
This last case happens quite a lot in Nosey Parker [1], a detector of secrets in textual data. There, it is possible to come across unnamed files in Git history, and it would be useful to the user to still indicate what type of file it seems to be.
I added file type detection based on libmagic to Nosey Parker a while back, but it's not compiled in by default because libmagic is slow and complicates the build process. Also, libmagic is implemented as a large C library whose primary job is parsing, which makes the security side of me jittery.
I will likely add enabled-by-default filetype detection to Nosey Parker using Magika's ONNX model.
At multiple points in my career I've been responsible for apis that accept PDFs. Many non-tech savvy people seeing this, will just change the extension of the file they're uploading to `.pdf`.
To make matters worse, there is some business software out there that will actually bastardize the PDF format and put garbage before the PDF file header. So for some things you end up writing custom validation and cleanup logic anyway.
Content sniffing can be disabled by the server (X-Content-Type-Options: nosniff), but it’s still used by default. Web browsers have to assume that servers are stupid, and that for relatively harmless cases, it’s fine to e.g. render a PNG loaded by an <img> even if it’s served as text/plain.
Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless).
I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797. My patch got rejected for needing to look at the first 3KB bytes of the file to figure out the type. They had a hard limit that they wouldn't see past the first 256 bytes. Now in 2024 we're doing this with deep learning! It'd be cool if google released some speed performance benchmarks here against the old-fashioned implementations. Obviously it'd be slower, but is it 1000x or 10^6x?
> At inference time Magika uses Onnx as an inference engine to ensure files are identified in a matter of milliseconds, almost as fast as a non-AI tool even on CPU.
Co-author of Magika here (Elie) so we didn't include the measurements in the blog post to avoid making it too long but we did those measurements.
Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples.
So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeated detection. This is why we said it is not that much slower.
We will have more performance measurements in the upcoming research paper. Hope that answer the question
Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.
Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.
Of course, it's arguably unlikely a virus scanner would opt for an ML-based approach, as they specifically need to be robust against adversarial inputs.
You might be surprised. Rename your Photo.JPG as Photo.PNG and you'll still get a perfectly fine thumbnail. The extension is a hint, but it isn't definitive, especially when you start downloading from the web.
In general you're right, but I can't think of a single local use for identifying file types by a human on a laptop - at least, one with scale where this matters. It's all going to be SaaS services where people upload stuff.
We are building a data analysis tool with great UX, where users select data files, which are then parsed and uploaded to S3 directly, on their client machines. The server only takes over after this step.
Since the data files can be large, this approach bypasses having to trnasfer the file twice, first to the server, and then to S3 after parsing.
I mean if you care about that you shouldn't be running anything that isn't highly optimized. Don't open webpages that might be CPU or GPU intensive. Don't run Electron apps, or really anything that isn't built in a compiled language.
Certainly you should do an audit of all the Android and iOS apps as well, to make sure they've been made in a efficient manner.
Block ads as well, they waste power.
This file identification is SUCH a small aspect of everything that is burning power in your laptop or phone as to be laughable.
Whilst energy usage is indeed a small aspect this early on when using bespoke models, we do have to consider that this is a model for simply identifying a file type.
What happens when we introduce more bespoke models for manipulating the data in that file?
This feels like it could slowly boil to the point of programs using magnitudes higher power, at which point it'll be hard to claw it back.
That's a slippery slope argument, which is a common logical fallacy[0]. This model being inefficient compared to the best possible implementation does not mean that future additions will also be inefficient.
It's the equivalent to saying many people programming in Ruby is causing all future programs to be less efficient. Which is not true. In fact, many people programming in Ruby has caused Ruby to become more efficient because it gets optimised as it gets used more (or Python for that matter).
It's not as energy efficient as C, but it hasn't caused it to get worse and worse, and spiral out of control.
Likewise smart contracts are incredibly inefficient mechanisms of computation. The result is mostly that people don't use them for any meaningful amounts of computation, that all gets done "Off Chain".
Generative AI is definitely less efficient, but it's likely to improve over time, and indeed things like quantization has allowed models that would normally to require much more substantial hardware resources (and therefore, more energy intensive) to be run on smaller systems.
The slippery slope fallacy is: "this is a slope. you will slip down it." and is always fallacious. Always. The valid form of such an argument is: "this is a slope, and it is a slippery one, therefore, you will slip down it."
Is that single-threaded libmagic vs Magika using every core on the system? What are the numbers like if you run multiple libmagic instances in parallel for multiple files, or limit both libmagic and magika to a single core?
Testing it on my own system, magika seems to use a lot more CPU-time:
file /usr/lib/* 0,34s user 0,54s system 43% cpu 2,010 total
./file-parallel.sh 0,85s user 1,91s system 580% cpu 0,477 total
bin/magika /usr/lib/* 92,73s user 1,11s system 393% cpu 23,869 total
Looks about 50x slower to me. There's 5k files in my lib folder. It's definitely still impressively fast given how the identification is done, but the difference is far from negligible.
I've ended up implementing a layer on top of "magic" which, if magic detects application/zip, reads the zip file manifest and checks for telltale file names to reliably detect Office files.
The "magic" library does not seem to be equipped with the capabilities needed to be robust against the zip manifest being ordered in a different way than expected.
But this deep learning approach... I don't know. It might be hard to shoehorn in to many applications where the traditional methods have negligible memory and compute costs and the accuracy is basically 100% for cases that matter (detecting particular file types of interest). But when looking at a large random collection of unknown blobs, yeah, I can see how this could be great.
magic is the core detection logic of file that was extracted out to be available as a library. So these days file is just a higher level wrapper around magic
If you're curious, here's how I solved it for ruby back in the day. Still used magic bytes, but added an overlay on top of the freedesktop.org DB: https://github.com/mimemagicrb/mimemagic/pull/20
> They had a hard limit that they wouldn't see past the first 256 bytes.
Then they could never detect zip files with certainty, given that to do that you need to read up to 65KB (+ 22) at the END of the file. The reason is that the zip archive format allows "gargabe" bytes both in the beginning of the file and in between local file headers.... and it's actually not uncommon to prepend a program that self-extracts the archive, for example. The only way to know if a file is a valid zip archive is to look for the End of Central Directory Entry, which is always at the end of the file AND allows for a comment of unknown length at the end (and as the comment length field takes 2 bytes, the comment can be up to 65K long).
That's why the whole question is ill formed. A file does not have exactly one type. It may be a valid input in various contexts. A zip archive may also very well be something else.
That's not what the performance table in the article is implying, with a precision and recall higher on Magika hovering around 99%, while magic is at 92% prec and 72% recall.
One can doubt the representativity of their dataset, but if what is in the article is correct, Magika is clearly way more capable and predictable
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
This group of Linux users used to brag Linux will identify files even if you change the extension, Windoze needs to police you about changing extension, nearly 20 years back.
I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on precision or recall.
The `file` command checks only the first few bytes, and doesn’t parse the structure of the file. APK files are indeed reported as Zip archives by the latest version of `file`.
This is false in every sense for https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version.
FILE_45:
./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk
../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing Block
Interesting! I checked with file 5.44 from Ubuntu 23.10 and 5.45 on macOS using homebrew, and in both cases, I got “Zip archive data, at least v2.0 to extract” for the file here[1]. I don’t have an Android phone to check and I’m also not familiar with Android tooling, so is this a corrupt APK?
I also tried this with the sources of file from the homepage you linked above, and I still get the same results.
You could try this for yourself using the same APKPure file which I uploaded at the following alternative link[1]. Further, while this could be a corrupt APK, I can’t see any signs of that from a cursory inspection as both the `classes.dex` and `META-INF` directory are present, and this is APKPure’s own APK, instead of an APK contributed for an app contributed by a third-party.
This feels like old school google. I like that it's just a static webpage that basically can't be shut down or sunsetted. It reminds of when Google just made useful stuff and gave them away for free on a webpage like translate and google books. Obviously less life changing than the above but still a great option to have when I need this.
But file(2) is already like that - my data files without headers are reported randomly as disk images, compressed archives or even executables for never-heard-of machines.
Other methods use heuristics to guess many filetypes and in the benchmark they show worse performance (in terms of precision). Assuming benchmarks are not biased, the fact that this approach uses AI heuristics instead of hard-coded heuristics shouldn't make it strictly worse.
I wonder how big of a deal it is that you'd have to retrain the model to support a new or changed file type? It doesn't seem like the repo contains training code, but I could be missing it...
As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space.
Though I have to say when looking at the Node module, I don't understand why they released it.
By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale.
Which implies a production-ready release for general usage, as well as usage by security researchers.
hyperfine ./magika.bash ./file.bash
Benchmark 1: ./magika.bash
Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms]
Range (min … max): 684.0 ms … 738.9 ms 10 runs
Benchmark 2: ./file.bash
Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms]
Range (min … max): 22.4 ms … 29.0 ms 111 runs
Summary
'./file.bash' ran
29.88 ± 1.65 times faster than './magika.bash'
Realistically, either you're identifying one file interactively and you don't care about latency differences in the 10s of ms, or you're identifying in bulk (batch command line or online in response to requests), in which case you should measure the marginal cost and exclude Python startup and model loading times.
My little script is trying to identify in bulk, at least by passing 165 file paths to `magika`, and `file`.
Though, I absolutely agree with you. I think realistically it's better to do this kind of thing in a library rather than shell out to it at all. I was just trying to get an idea on how it generally compares.
Another note, I was trying to be generous to `magicka` here because when it's single file identification, it's about 160-180ms on my machine vs <1ms for `file`. I realize that's going to be quite a bit of python startup in that number, which is why I didn't go with it when pushing that benchmark up earlier. I'll probably push an update to that gist to include the single file benchmark as well.
Going by those number it's taking almost a second to run, not 10s of ms. And going by those numbers, it's doing something massively parallel in that time. So basically all your cores will spike to 100% for almost a second during those one-shot identifications. It looks like GP has a 12-16 threads CPU, and it is using those while still being 30 times slower than single-threaded libmagic.
That tool needs 100x more CPU time just to figure out some filetypes than vim needs to open a file from a cold start (which presumably includes using libmagic to check the type).
If I had to wait a second just to open something during which that thing uses every resource available on my computer to the fullest, I'd probably break my keyboard. Try using that thing as a drop-in file replacement, open some folder in your favorite file manager, and watch your computer slow to a crawl as your file manager tries to figure out what thumbnails to render.
It's utterly unsuitable for "interactive" identifications.
We did release the npm package because indeed we create a web demo and thought people might want to also use it. We know it is not as fast as the python version or a C++ version -- which why we did mark it as experimental.
The release include the python package and the cli which are quite fast and is the main way we did expect people to use -- sorry if that hasn't be clear in the post.
The goal of the release is to offer a tool that is far more accurate that other tools and works on the major file types as we hope it to be useful to the community.
Thank you for the release! I understand you're just getting it out the door. I just hope to see it delivered as a native library or something more reusable.
I did try the python cli, but it seems to be about 30x slower than `file` for the random bag of files I checked.
I'll probably take some time this weekend to make a couple of issues around misidentified files.
Hello!
We wrote the Node library as a first functional version.
Its API is already stable, but it's a bit slower than the Python library for two reasons: it loads the model at runtime, and it doesn't do batch lookups, meaning it calls the model for each file.
Other than that, it's just as fast for single file lookups, which is the most common usecase.
> The model appears to only detect 116 file types [...] Where libmagic detects... a lot. Over 1600 last time I checked
As I'm sure you know, in a lot of applications, you're preparing things for a downstream process which supports far fewer than 1600 file types.
For example, a printer driver might call on file to check if an input is postscript or PDF, to choose the appropriate converter - and for any other format, just reject the input.
Or someone training an ML model to generate Python code might have a load of files they've scraped from the web, but might want to discard anything that isn't Python.
Okay, but your one file type is more likely to be included in the 1600 that libmagic supports rather than Magika's 116?
For that matter, the file types I care about are unfortunately misdetected by Magika (which is also an important point - the `file` command at least gives up and says "data" when it doesn't know, whereas the Magika demo gives a confidently wrong answer).
I don't want to criticize the release because it's not meant to be a production-ready piece of software, and I'm sure the current 116 types isn't a hard limit, but I do understand the parent comment's contention.
Surely identifying just one file type (or two, as in your example) is a much simpler task that shouldn’t rely on horribly inefficient and imprecise “AI” tools?
Absolutely, and honestly in a non-interactive ingestion workflow you're probably doing multiple checks anyway. I've worked with systems that call multiple libraries and hand-coded validation for each incoming file.
Maybe it's my general malaise, or disillusionment with the software industry, but when I wrote that I was really just expecting more.
> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format.
This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand.
Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".
the rules aren't based on the formats, but on a small portion of them (their magic numbers). this makes them inaccurate (think docx vs zip) and heuristic.
What are use-cases for this? I mean, obviously detecting the filetype is useful, but we kinda already have plenty of tools to do that, and I cannot imagine, why we need some "smart" way of doing this. If you are not a human, and you are not sure what is it (like, an unknown file being uploaded to a server) you would be better off just rejecting it completely, right? After all, there's absolutely no way an "AI powered" tool can be more reliable than some dumb, err-on-safer-side heuristic, and you wouldn't want to trust that thing to protect you from malicious payloads.
Virus detection is mentioned in the article. Code editors need to find the programming language for syntax highlighting of code before you give it a name. Your desktop OS needs to know which program to open files with. Or, recovering files from a corrupted drive. Etc
It's easy to distinguish, say, a PNG from a JPG file (or anything else that has well-defined magic bytes). But some files look virtually identical (eg. .jar files are really just .zip files). Also see polyglot files [1].
If you allow an `unknown` label or human intervention, then yes, magic bytes might be enough, but sometimes you'd rather have a 99% chance to be right about 95% of files vs. a 100% chance to be right about 50% of files.
I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.
Indeed but as pointed out in the blog post -- file is significantly less accurate that Magika. There are also some file type that we support and file doesn't as reported in the table.
I can't immediately find the dataset used for benchmarking. Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?
> Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?
That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.
I still don't see how this is useful. The only time I want to answer the question "what type of file is this" is if it is an opaque blob of binary data. If it's a plain text file like Python, HTML, or JSON, I can figure that out by just catting the file.
> What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.
I use the file command all the time. The value is when you get this:
... $ file somefile.xyz
somefile.xyz: data
AIUI from reading TFA, magika can determine more filetypes than what the file command can detect.
It'd actually be very easy to determine if there's any value in magika: run file on every file on your filesystem and then for every file where the file command returns "data", run magika and see if magika is right.
If it's right, there's your value.
P.S: it may also be easier to run on Windows than the file command? But then I can't do much to help people who are on Windows.
From elsewhere in this thread, it appears that Magika detects far fewer file types than file (116 vs ~1600), which makes sense. For file, you just need to drop in a few rules to add a new, somewhat obscure type. An AI approach like Magika will need lots of training and test data for each new file type. Where Magika might have a leg up is with distinguishing different textual data files (i.e., source code), but I don't see that as a particularly big use case honestly.
It's not always deterministic, sometimes it's fuzzy depending on the file type. Example of this is a one-line CSV file. I tested one case of that, libmagic detects it as a text file while magika correctly detects it as a CSV (and gives a confidence score, which is killer).
But even with determinism, it's not always right. It's not too rare to find a text file with a byte order mark indicating UTF16 (0xFE 0xFF) but then actually containing utf-8. But what "format" does it have then? Is it UTF-8 or UTF-16? Same with e.g. a jar file missing a manifest. That's just a zip, even though I'm sure some runtime might eat it.
But the question is when you have the issue of having to guess the format of a file? Is it when reverse engineering? Last time I did something like this was in the 90's when trying to pick apart some texture from a directory of files called asset0001.k and it turns out it was a bitmap or whatever. Fun times.
Argh, the risks of browsing the web without JavaScript and/or third party scripts enabled, you miss content, because rendering text and images on the modern web can't be done without them, apparently. (Sarcasm).
You are of course correct. I can see the images showing the comparison. Apologies.
261 comments
[ 3.6 ms ] story [ 166 ms ] threadI wonder how susceptible it is to adversarial binaries or, hah, prompt-injected binaries.
> Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data".
Seems like this is genuinely useless for anybody but AI researchers.
[1] https://news.ycombinator.com/item?id=39395677
Then there are all the container formats that have different kinds of formats embedded in them (mov,mkv,pdf etc.)
Sometimes a file has no extension. Other times the extension is a lie. Still other times, you may be dealing with an unnamed bytestring and wish to know what kind of content it is.
This last case happens quite a lot in Nosey Parker [1], a detector of secrets in textual data. There, it is possible to come across unnamed files in Git history, and it would be useful to the user to still indicate what type of file it seems to be.
I added file type detection based on libmagic to Nosey Parker a while back, but it's not compiled in by default because libmagic is slow and complicates the build process. Also, libmagic is implemented as a large C library whose primary job is parsing, which makes the security side of me jittery.
I will likely add enabled-by-default filetype detection to Nosey Parker using Magika's ONNX model.
[1] https://github.com/praetorian-inc/noseyparker
To make matters worse, there is some business software out there that will actually bastardize the PDF format and put garbage before the PDF file header. So for some things you end up writing custom validation and cleanup logic anyway.
"web browsers"? Odd to see this coming from Google itself. https://en.wikipedia.org/wiki/Content_sniffing was widely criticised for being problematic for security.
I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797. My patch got rejected for needing to look at the first 3KB bytes of the file to figure out the type. They had a hard limit that they wouldn't see past the first 256 bytes. Now in 2024 we're doing this with deep learning! It'd be cool if google released some speed performance benchmarks here against the old-fashioned implementations. Obviously it'd be slower, but is it 1000x or 10^6x?
> enabling precise file identification within milliseconds, even when running on a CPU.
Maybe your old-fashioned implementations were detecting in microseconds?
Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples.
So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeated detection. This is why we said it is not that much slower.
We will have more performance measurements in the upcoming research paper. Hope that answer the question
Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.
This entire use case seems to be one suited for servers handling user media.
Of course, it's arguably unlikely a virus scanner would opt for an ML-based approach, as they specifically need to be robust against adversarial inputs.
https://twitter.com/taviso/status/732365178872856577
Several major players such as Norton, McAfee, and Symantec all at least claim to use AI/ML in their antivirus products.
Of course on Windows the convention is to use the file extension, but on other platforms the convention is to look at the file contents
MacOS (that is, Finder) also looks at the extension. That has also been the case with any file manager I've used on Linux distros that I can recall.
Since the data files can be large, this approach bypasses having to trnasfer the file twice, first to the server, and then to S3 after parsing.
Certainly you should do an audit of all the Android and iOS apps as well, to make sure they've been made in a efficient manner.
Block ads as well, they waste power.
This file identification is SUCH a small aspect of everything that is burning power in your laptop or phone as to be laughable.
What happens when we introduce more bespoke models for manipulating the data in that file?
This feels like it could slowly boil to the point of programs using magnitudes higher power, at which point it'll be hard to claw it back.
It's the equivalent to saying many people programming in Ruby is causing all future programs to be less efficient. Which is not true. In fact, many people programming in Ruby has caused Ruby to become more efficient because it gets optimised as it gets used more (or Python for that matter).
It's not as energy efficient as C, but it hasn't caused it to get worse and worse, and spiral out of control.
Likewise smart contracts are incredibly inefficient mechanisms of computation. The result is mostly that people don't use them for any meaningful amounts of computation, that all gets done "Off Chain".
Generative AI is definitely less efficient, but it's likely to improve over time, and indeed things like quantization has allowed models that would normally to require much more substantial hardware resources (and therefore, more energy intensive) to be run on smaller systems.
[0]: https://en.wikipedia.org/wiki/Slippery_slope
We're already there. Modern software is, by and large, profoundly inefficient.
Testing it on my own system, magika seems to use a lot more CPU-time:
Looks about 50x slower to me. There's 5k files in my lib folder. It's definitely still impressively fast given how the identification is done, but the difference is far from negligible.The "magic" library does not seem to be equipped with the capabilities needed to be robust against the zip manifest being ordered in a different way than expected.
But this deep learning approach... I don't know. It might be hard to shoehorn in to many applications where the traditional methods have negligible memory and compute costs and the accuracy is basically 100% for cases that matter (detecting particular file types of interest). But when looking at a large random collection of unknown blobs, yeah, I can see how this could be great.
Then they could never detect zip files with certainty, given that to do that you need to read up to 65KB (+ 22) at the END of the file. The reason is that the zip archive format allows "gargabe" bytes both in the beginning of the file and in between local file headers.... and it's actually not uncommon to prepend a program that self-extracts the archive, for example. The only way to know if a file is a valid zip archive is to look for the End of Central Directory Entry, which is always at the end of the file AND allows for a comment of unknown length at the end (and as the comment length field takes 2 bytes, the comment can be up to 65K long).
RTFA.
One can doubt the representativity of their dataset, but if what is in the article is correct, Magika is clearly way more capable and predictable
I passionately dislike Surveillance Capitalism but bringing it up when it’s completely irrelevant only weakens the argument.
RTFA.
From the HN guidelines
I shortened it to RTFA.
FILE_45:
[1] https://download.apkpure.net/custom/com.apkpure.aegon-319781...
You could try this for yourself using the same APKPure file which I uploaded at the following alternative link[1]. Further, while this could be a corrupt APK, I can’t see any signs of that from a cursory inspection as both the `classes.dex` and `META-INF` directory are present, and this is APKPure’s own APK, instead of an APK contributed for an app contributed by a third-party.
[1] https://wormhole.app/Mebmy#CDv86juV9H4aRCL2DSJeDw
The tooling even supports it. https://docs.oracle.com/en/java/javase/21/docs/specs/man/jar...:
Works but occasionally having bugs and weird collisions when working with billions of files.
Happy to see new contributions in the space.
Though I have to say when looking at the Node module, I don't understand why they released it.
Their docs say it's slow:
https://github.com/google/magika/blob/120205323e260dad4e5877...
It loads the model an runtime:
https://github.com/google/magika/blob/120205323e260dad4e5877...
They mark it as Experimental in the documentation, but it seems like it was just made for the web demo.
Also as others have mentioned. The model appears to only detect 116 file types:
https://github.com/google/magika/blob/120205323e260dad4e5877...
Where libmagic detects... a lot. Over 1600 last time I checked:
https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c...
I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.
Though, I absolutely agree with you. I think realistically it's better to do this kind of thing in a library rather than shell out to it at all. I was just trying to get an idea on how it generally compares.
Another note, I was trying to be generous to `magicka` here because when it's single file identification, it's about 160-180ms on my machine vs <1ms for `file`. I realize that's going to be quite a bit of python startup in that number, which is why I didn't go with it when pushing that benchmark up earlier. I'll probably push an update to that gist to include the single file benchmark as well.
That tool needs 100x more CPU time just to figure out some filetypes than vim needs to open a file from a cold start (which presumably includes using libmagic to check the type).
If I had to wait a second just to open something during which that thing uses every resource available on my computer to the fullest, I'd probably break my keyboard. Try using that thing as a drop-in file replacement, open some folder in your favorite file manager, and watch your computer slow to a crawl as your file manager tries to figure out what thumbnails to render.
It's utterly unsuitable for "interactive" identifications.
The release include the python package and the cli which are quite fast and is the main way we did expect people to use -- sorry if that hasn't be clear in the post.
The goal of the release is to offer a tool that is far more accurate that other tools and works on the major file types as we hope it to be useful to the community.
Glad to hear it worked on your files
I did try the python cli, but it seems to be about 30x slower than `file` for the random bag of files I checked.
I'll probably take some time this weekend to make a couple of issues around misidentified files.
I'll definitely be adding this to my toolset!
I also appreciate the use of ONNX here, as I'm already thinking about using another version of the runtime.
Do you think you'll open source your F1 benchmark?
As I'm sure you know, in a lot of applications, you're preparing things for a downstream process which supports far fewer than 1600 file types.
For example, a printer driver might call on file to check if an input is postscript or PDF, to choose the appropriate converter - and for any other format, just reject the input.
Or someone training an ML model to generate Python code might have a load of files they've scraped from the web, but might want to discard anything that isn't Python.
For that matter, the file types I care about are unfortunately misdetected by Magika (which is also an important point - the `file` command at least gives up and says "data" when it doesn't know, whereas the Magika demo gives a confidently wrong answer).
I don't want to criticize the release because it's not meant to be a production-ready piece of software, and I'm sure the current 116 types isn't a hard limit, but I do understand the parent comment's contention.
Maybe it's my general malaise, or disillusionment with the software industry, but when I wrote that I was really just expecting more.
This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand.
Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".
Those facts are clearly reflected in the table.
The article provides accuracy benchmarks.
> you would be better off just rejecting it completely
They mention using it in gmail and Drive, neither of which have the luxury of rejecting files willy-nilly.
It's easy to distinguish, say, a PNG from a JPG file (or anything else that has well-defined magic bytes). But some files look virtually identical (eg. .jar files are really just .zip files). Also see polyglot files [1].
If you allow an `unknown` label or human intervention, then yes, magic bytes might be enough, but sometimes you'd rather have a 99% chance to be right about 95% of files vs. a 100% chance to be right about 50% of files.
[1] https://en.wikipedia.org/wiki/Polyglot_(computing)
That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.
Knowing is better than guessing.
And there are also a billion formats that are not uniquely determined by magic bytes. You don't have to go further than text files.
I use the file command all the time. The value is when you get this:
AIUI from reading TFA, magika can determine more filetypes than what the file command can detect.It'd actually be very easy to determine if there's any value in magika: run file on every file on your filesystem and then for every file where the file command returns "data", run magika and see if magika is right.
If it's right, there's your value.
P.S: it may also be easier to run on Windows than the file command? But then I can't do much to help people who are on Windows.
But the question is when you have the issue of having to guess the format of a file? Is it when reverse engineering? Last time I did something like this was in the 90's when trying to pick apart some texture from a directory of files called asset0001.k and it turns out it was a bitmap or whatever. Fun times.
[0] https://mark0.net/soft-trid-e.html
You are of course correct. I can see the images showing the comparison. Apologies.
* https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/src/main...
There's a much larger list of file signatures at:
* https://github.com/veniware/Space-Maker/blob/master/FileSign...