True, RPM is pretty messed up as far as complexity.
But, let's consider that I have never had an RPM-using system have its package database corrupted beyond repair (you can rebuild it with rpmdb --rebuilddb), and unlike dpkg/deb, rpm fully supports downgrading packages, which is something I have missed on Debian multiple times.
RPM is too complex, but it's also a good format from a reliability standpoint.
I've had a Debian system break because I couldn't fix the package database, yes. I might well have just missed a trick, but I did google around for fixes and got nowhere.
> Huh? Downgrades are fully supported by dpkg and apt.
No, they aren't. You need to remove the newer version prior to installing the old one. RPM based systems allow you to downgrade to an entirely different distro release with one command.
How are these qualities descriptive of the file format and not of the particular implementation of the package manager used by the distribution in question?
The file format and the quality of the package manager implementation seem to be rather orthogonal, so I don't understand how your top comment is relevant to the article.
there was a chance to unify linux packaging in the last year around creating universal packages - what ended up happening instead was yet another war between snap and flatpak.
Everyone flaunts some superiority or another - https://github.com/AppImage/AppImageKit/wiki/Similar-project...
however, it would be incredible to have a single format
If you want a universal format, you have a compressed source tar-ball. Anything else will run into the whims (and on the ground circumstances) of maintainers and admins.
Fun story: I once had to exchange data with a Solaris box, and discovered that its tar had no "z" flag. Ended up just exchanging uncompressed tar files because I didn't want to figure out .tar.Z files.
Those formats aren't comparable to RPM/deb. In any case even if we did unify RPM and dpkg, it would make no difference because distros would still have their own standards for naming, dependencies, etc. SLES and RHEL both use RPMs but you cannot copy RPMs from one to the other.
This person tries inspecting the format with a hex debugger, implying he hasn't done any research. Rpm2cpio then extract the RPM file. It's a binary and a text .spec file. The spec file format is divided into sections to build, make etc the app and while ancient (like dpkg) not hard to learn.
That's even worse. Why would you analyse a binary RPM to inspect it, when present that as a normal way to analyse or build RPMs?
> the author was analysing trying to build a binary package directly.
To analyse, there's two approaches that have been well documented for around 20 years now:
- open the .src.rpm for a package
- Run the rpm binary over the .rpm, which can dump spec file contents too
To build:
- Make a spec file
Again, there's lots to dislike about RPM (my main issue is how the sections are macros and so are the autoconf macros), which confused a lot of new people) but this article in incredibly unresearched.
In fairness, the author was trying to write his own RPM encoder, so not using existing tools was... less of a bad idea than usual. That said, I agree that this was under-researched.
It makes it seem like RPM is an incredibly difficult packaging system, when in reality the author has deliberately chosen an obscure and unusual way of working with it.
This is nothing to do with the packaging system or the ecosystem, the author decided to dive into the file format specifically. They even posted comments from the rpm source that echo the confusion and disbelief reflected in the article. Is the tone the author took the problem you have with it? Otherwise I don’t get why you are so against someone trying to learn how a file format works and documenting the experience.
I've built tools that automatically create RPM files in less time than the author tool to write this article. I take umbrage in the bizarre implication that building a tool to make RPMs needs you to dive into the format with a hex editor, when it does not.
ASN.1 and XML are easy to work with using the right existing tools, but that doesn't mean someone can't post their thoughts about them being frustrating formats.
OP made it clear what they were focusing on. If a passerby mistakes it for a complaint about using RPM as a sysadmin, it's their own problem.
Keep in mind that the author tried to build an independent implementation, so your complaint is like "Why are Gecko developers wasting time on reverse-engineering websites, instead of installing Chrome?"
For this purpose it's fitting to do a deep analysis of files actually produced, rather than trust the documentation to be entirely complete and sufficient.
Such low-level dump is also needed to debug interoperability issues when files produced by the new implementation are rejected by existing tools.
I'm developing cargo-deb, which builds Debian packages without need for spec files. It doesn't use Perl, and can even build Debian packages from macOS. I had to learn and reproduce the structure of the deb files, which was relatively easy. I would need to go through exactly the same horror as in the article if I wanted to add RPM support the same NIH way (so RPM tooling may be good and easy, but RPM as a format is a failure.)
> I ship configuration as system packages. Every distribution has their own tooling and process for building these packages, but I eventually grew tired of the ceremony involved in it, and wrote my own system package compiler.
Which to most engineers would mean making a tool that builds.rpm files, not reimplements the rpm build tools.
Read a bit further: I stubbornly refused to add dependencies and use existing tooling (i.e., the rpm-build(1) command). I wanted to serialize the format directly from my own code, like I did for Pacman and Debian packages.
I've authored many RPM packages, it's dead simple. Have a spec file, tar it, and run rpmbuild on it.
I don't really know why this guy was freaking out.
I also like the fact that transient dependency and repo handling is separated from the package handler itself. We have rpm to handle a single package, and we have dnf (yum) to handle trees of packages.
> I don't really know why this guy was freaking out.
That's clearly explained in the article. This is not about authoring from a specfile, so your comment is off. He wants to serialise a package into RPM without the traditional toolchain; it turns out to be much more difficult than the other serialisers he already has because of the rpm format's bad design decisions and accretions.
I don't think they're bad decisions; I think he has no idea why it's that way and so is assuming that it's terrible. The RPM format was designed when computers were smaller and slower, and is extremely flexible while forcing you to declare enough that it can validate the file contents as it goes. It's over-engineered, and there are certainly things I would change, but it's not just random stupid decisions. If you can't think of any good reason why it would be written the way it is, the main options are that you're missing something or the original spec authors were stupid. Of course, sometimes they were stupid, but maybe don't immediately assume that after your very first investigation into the format.
Debian under-engineered, Redhat over-engineered. In perfect hindsight, Debian probably made better tradeoffs, but that's only with decades of hindsight.
I suppose "under" is perhaps a stronger term than I really want, since it did in fact work out. I agree that Debian, indeed, fulfilled exactly what it needed, but only what it definitely needed right then; a minimum viable product that turned out to be sufficent. RH built for everything that they could imagine ever needing, and left options to extend for anything they didn't think of in advance. Perhaps "just enough"-engineered vs over-engineered?
I disagree, RH did not in fact built it for everything; if they had, they would not need headers-inside-headers or fixed "nonsense" values. What they build is an over-engineered yet incomplete format, but also inflexible, so they had then to pile hacks upon it.
Yes, the RPM format is crappy. A normal person would have seen that quickly (or known it already), gotten their business done, and moved on. What's the point of dissecting every little piece to criticize, or of criticizing even the oversights or practices normal for their time as much as the things that are truly insane? It's just rage-bait.
You also need to understand how it got that way. In this case, a lot of the things the author takes issue with are a combination of eternal backwards compatibility, really wide portability, and having originally been devised when memory was tiny and processors were slow. It's one of those things where if you rebuilt it yourself, by the time you covered all features of the original, you would be just as ugly.
I agree. For instance, one of the things the author complains about is the rpmlib(...) dependencies. To me, they are clearly a way to make older versions of rpm (which do not know anything about these features) fail cleanly with a "missing dependency" error.
Almost no one who uses RPMs needs to know the internal format, and if they do they should use rpmlib (or rpm2archive if they just want to unpack the files). Does anyone not use JPEG because they think the internals of JPEG binary format are a bit complicated?
The point of the exercise was to not use rpmlib. The comparison to JPEG is inappropriate because it has dozens of interoperable implementations.
Formats defined by a single implementation are sickly and much less viable (remember the lessons of http://libmng.com/ ?); I applaud the reverse engineering effort from the article.
> Does anyone not use JPEG because they think the internals of JPEG binary format are a bit complicated?
Yes! There are lots of examples like this.
• Plenty of small tools write BMP or Targa instead of JPEG or PNG, because these formats are dead simple to reinvent.
• JPEG as we know it is JFIF, which has clearly won over alternative JPEG data containers, such as more complicated TIFF (mode 6). BTW JPEG, for what it does, is brilliantly simple.
• MNG never took off, but dumber APNG did.
Formats that are intended for interoperability need multiple implementations, and it does happen that authors say "no, fuck that" and choose a different format instead.
That's funny, I've always thought that the various Linux distribution using different packaging format was a huge waste of resources and wanted everyone to standardize on RPM.
After reading this article, I still thinks it is a waste of resources but I'd prefer that everyone standardize on .deb not RPM!
57 comments
[ 3.2 ms ] story [ 122 ms ] threadThere are times i wonder if IT has some innate attraction to masochism...
But, let's consider that I have never had an RPM-using system have its package database corrupted beyond repair (you can rebuild it with rpmdb --rebuilddb), and unlike dpkg/deb, rpm fully supports downgrading packages, which is something I have missed on Debian multiple times.
RPM is too complex, but it's also a good format from a reliability standpoint.
you've had that on Debian?
> rpm fully supports downgrading packages, which is something I have missed on Debian multiple times
Huh? Downgrades are fully supported by dpkg and apt*.
No, they aren't. You need to remove the newer version prior to installing the old one. RPM based systems allow you to downgrade to an entirely different distro release with one command.
> you've had that on Debian?
More than once, sadly.
however, it would be incredible to have a single format
> the author was analysing trying to build a binary package directly.
To analyse, there's two approaches that have been well documented for around 20 years now:
- open the .src.rpm for a package
- Run the rpm binary over the .rpm, which can dump spec file contents too
To build:
- Make a spec file
Again, there's lots to dislike about RPM (my main issue is how the sections are macros and so are the autoconf macros), which confused a lot of new people) but this article in incredibly unresearched.
OP made it clear what they were focusing on. If a passerby mistakes it for a complaint about using RPM as a sysadmin, it's their own problem.
For this purpose it's fitting to do a deep analysis of files actually produced, rather than trust the documentation to be entirely complete and sufficient. Such low-level dump is also needed to debug interoperability issues when files produced by the new implementation are rejected by existing tools.
I'm developing cargo-deb, which builds Debian packages without need for spec files. It doesn't use Perl, and can even build Debian packages from macOS. I had to learn and reproduce the structure of the deb files, which was relatively easy. I would need to go through exactly the same horror as in the article if I wanted to add RPM support the same NIH way (so RPM tooling may be good and easy, but RPM as a format is a failure.)
But even if I wanted to recreate RPM which the author does (although he doesn't state this) I'd start with the source code.
Yes, he does, right in the first paragraphs.
I'd start with the source code.
He started with the formal specification of the RPM format, which includes relevant pieces of the source; how is that not a good place to start?
> I ship configuration as system packages. Every distribution has their own tooling and process for building these packages, but I eventually grew tired of the ceremony involved in it, and wrote my own system package compiler.
Which to most engineers would mean making a tool that builds.rpm files, not reimplements the rpm build tools.
There is no competition between package managers, there is only competition between distributions.
https://github.com/ctron/rpm-builder
I don't really know why this guy was freaking out.
I also like the fact that transient dependency and repo handling is separated from the package handler itself. We have rpm to handle a single package, and we have dnf (yum) to handle trees of packages.
That's clearly explained in the article. This is not about authoring from a specfile, so your comment is off. He wants to serialise a package into RPM without the traditional toolchain; it turns out to be much more difficult than the other serialisers he already has because of the rpm format's bad design decisions and accretions.
So was DEB. That's not a very good excuse.
Plus, RH could have decided to make a proper v2 rather than build this lasagna of a format.
Formats defined by a single implementation are sickly and much less viable (remember the lessons of http://libmng.com/ ?); I applaud the reverse engineering effort from the article.
Yes! There are lots of examples like this.
• Plenty of small tools write BMP or Targa instead of JPEG or PNG, because these formats are dead simple to reinvent.
• JPEG as we know it is JFIF, which has clearly won over alternative JPEG data containers, such as more complicated TIFF (mode 6). BTW JPEG, for what it does, is brilliantly simple.
• MNG never took off, but dumber APNG did.
Formats that are intended for interoperability need multiple implementations, and it does happen that authors say "no, fuck that" and choose a different format instead.
> The <rpm/rpmtag.h> header has an enum with all acceptable values, but please consider the environment before trying to print it.