Show HN: Musefs – organize and tag music without touching the original files (github.com)

2 points by sohex ↗ HN
musefs is a virtual filesystem for organizing and tagging your music without altering or duplicating the backing audio files. It scans your library to build up a SQLite db with all the original tags and embedded art and makes note of exactly where in the file the actual audio is. That db becomes the source of truth for the FUSE filesystem. At playback it generates new metadata for the files on the fly from the tags in the db and splices it onto the original backing audio.

I have a music collection that's grown rather large over the years and I've always had to choose between the lesser of three evils. I could either preserve the original files as they are, actually correct all the messy tags and organize them nicely, or keep the former and make a copy for the latter. Keeping both has worked, but now that my library is over 1TB the storage cost is definitely noticable. So, inspired by a long since defunct project called beetsfs; I built this, musefs.

Right now it supports MP3, FLAC, M4A/M4B, and Ogg (Opus, Vorbis, and FLAC, did you know you can put FLAC in Ogg? I didn't until I started working on this project). Implementing MP3 and FLAC were both straightforward, I'd call them the mobs. Making M4A the mini-boss, because it's a lot more loosey-goosey with how it's structured. Which leaves Ogg as the Final Boss. Ogg is a lovely container format, however, it was absolutely not designed for this. The way art has to be handled is unique to it, but more importantly each file is composed of individually checksummed pages. Now a key focus for me throughout development has been performance, so reading each page in full to rechecksum it was a non-starter. I'm honestly delighted by the solution I came to instead, it's by far the most technically interesting part of this project.

Let me give you the short version. Every Ogg page has a header and a payload. The header includes a page number and the aforementioned checksum. Since we're dynamically generating the preceding metadata every subsequent page (including the actual audio pages) needs to be renumbered. Which breaks the checksum. But! Due to the particular nature of CRC-32 implementation that Ogg uses, with a bit of math the actual payload drops right out of the equation. Then by applying some matrix algebra it's possible to compute the new CRC for the whole page with just the bytes of the header. I think that's really neat. For all the details you can look in docs/OGG.md in the repo.

In terms of support, right now it's 64-bit only on AMD64 and AARCH64, with first-class support for Linux and FreeBSD. macOS is theoretically supported, but I don't have the hardware to actually test that on, so caveat emptor. Caveat downloador? It's MIT licensed, so. At any rate, I decided that if I put off posting about it any longer I'd just keep polishing it forever, so I cut v1.0.0 and here we are. I've given it some pretty deep performance improvement passes and it's tested thoroughly, unit tests, prop tests, mutation testing, interop testing, we've got it all. It also comes with in tree plugins for beets, Picard, and Lidarr, as well as the common python package backing them if you want to build a plugin for something else. There are glibc and musl packages and containers available for both supported architectures too along with a sample systemd user service file for running it directly on a host.

It's near the start of the README but I'll reiterate it here as well for the sake of transparency. I built this using AI, Opus wrote specs and plans, MiMo v2.5 implemented them (along with some other models along the way). I put a lot of effort into making this not slop though and I think it paid off.

I'd love to hear any feedback that anyone has, especially if you run it against a large library!

3 comments

[ 2.6 ms ] story [ 15.6 ms ] thread
Nice approach. I'm the developer of https://www.blisshq.com/ and other projects. The key insight there is that managing large collections is easier with rules applied continuously, rather than thoughts-in-the-moment "I'll run a batch command with a tagger".

You could have the rules encoded in your config (or the DB) and apply those. For example, someone may want a "portable mirror" of their gold library, but that means transcoding AND chopping the embedded album art to size (or maybe their player has max resolutions).

I guess having the DB means you could potentially version control the metadata changes?

Oh very nice, bliss looks pretty slick. I see musefs as being kind of an unglamorous middle man layer that ultimately just has one job, separate the metadata handling from the backing files.

Since the DB is the source of truth you can absolutely version your metadata. The portable mirror use case is definitely one that people run into, but I see musefs more enabling that than handling it directly. It’d be really easy to handle transcoding and art handling just pointing at a musefs mount or by having a plugin for whatever you’re using handle it.

I definitely agree that continuous automation is the way to go, personally I set it up differently though. Whenever I have a new album in my music folder it gets detected and then beets imports it and museFS autoscans it. That handles 99% of cases silently in the background and the other 1% are the ones I’d want surfaced to make a judgement call on anyway.

That's what I mean by continuous automation, using rules (constraints) to dictate how the library is managed (in your case, what beets does, e.g. auto download art of the given size and format, constrain the genre etc etc).