59 comments

[ 3.2 ms ] story [ 120 ms ] thread
Aah xml-rpc; don't know why it is found interesting these days but still ..,

My thinking is: if you employ a relative high-ceremony meta-language such as XML as service payload format, then you'll at least want to use its features to model free yet strictly validated information exchanges (such as ASN-based protocols have been doing). But xml-rpc doesn't give you that and imposes param=values logic known from simple URL-encoded invocation forms instead, improving little over those.

Then SOAP was introduced as the big unified payload serialization format covering document- and RPC-oriented uses. We all know just how it sucked.

But then came the even worse end result/eternal September of "REST" APIs - a misuse of HTTP and blatant and painful misappropriation of Fielding's concepts, whose proponents used SOAP's flaws as an excuse for their anti-engineering practices.

If REST APIs are the worst, how does that account for their immense utility and popularity?

I wrangled XML-RPC and SOAP back in the day. It was bad enough when you had exactly the same stacks talking to each other. When you had to interop between systems, ie .NET talking to Java, which was half the point of it all, it was a whole new circle of hell.

Trying to get Python SOAP stacks to talk to Java/.NET banking services was another thing too.

I still get mild PTSD from thinking about XMLsec and XML c14n.

I have to disagree.

I used XML-RPC exactly twice, but it proved to be a quick solution to nasty integration problems and we were very happy with it.

Case #1 - we had to implement an interface to book flights on Amadeus (https://amadeus.com/en). In order to guarantee the caller identity they provided a C library (binaries that you had to link with your stuff) that would generate tokens that you would then add to your own calls to them to guarantee your identity. We were trying to use it from a Solaris machine, and the library would bomb at each call. But their Windows binary module worked fine, so we basically put up an XML-RPC connection between our Solaris hosted main app and a little Windows service which would simply provide the token for us to embed in the subsequent call. (This was the only way we could find to hit our release date in time, and it worked fine for 5 years serving hundred of thousands of calls every year).

Case #2 - less "business critical", but still fine: I was on sick leave from the office recovering from minor trauma to my knee and here is what I suggested to a guy trying to use a PERL library as part of non-PERL stack: https://stackoverflow.com/a/2635719/54504

(it was part of his final exam for a degree in CS, I provided more assistance outside of StackOverflow and he was very happy with the results).

Yes SOAP interoperability sucked, no arguing about that. The fact alone that you needed interoperability guidelines (WS-I) on top speaks volumes about W3C's derailed standardization effort - a similar problem is hunting today's OAuth standard btw.

But just that SOAP sucks doesn't mean "REST" is ideal. Fallacy of the excluded middle and all.

REST was at least spiritually trying to adhere to the ideas of HTTP verbs.

It not being a rigid protocol was a boon for web development although it generally meant "REST" was so nebulous people would describe their usage as "REST-like" or "REST-ish" if they broke conventions (like just using POST for most payloads)

But I'd say most of all it took off because it was simple and people like simple, particularly after jumping through hoops for a decade prior.

> REST was at least spiritually trying to adhere to the ideas of HTTP verbs

That's kindof the problem:

Except in the case where a thin browser front end is rendering a single response payload transferred by necessity over HTTP a la XSLT (which we're not doing anymore for better or worse) there's absolutely no rational reason to stick to "REST principles" in quasi-religious manner and exegetic interpretation of Fielding's thesis that coined that term. In fact, I'd say it's counterproductive, because a simplistic "REST" facade doesn't even begin to describe the actual interaction between backends or rich frontends and backends, those interactions often being stateful and much more involved and granular than sending back-end-forth idealized full "representations" or flawed over-exposing ideas of the nature of a backend concept. You might think that SOAP-like UpdateOrderLineItemBilling granular ops suck, but these are much more representative and telling than coarse pretentious REST APIs suggesting you could update anything at any point in time during the course of a process, when in reality backend logic just doesn't work that way (eg in the ecommerce example, you'll have to honor the delivery and billing status of items to cancel, and compensate or retour items accordingly, etc).

Really, "spiritual programming" isn't an engineering discipline; you can model your backend-to-backend interactions freely without having to appeal to concepts from HTTP taken completely out of context.

> My thinking is: if you employ a relative high-ceremony meta-language such as XML as service payload format, then you'll at least want to use its features to model free yet strictly validated information exchanges

No? There is no relationship between the two. XMLRPC is a straightforward application of XML, its intended niche was similar to JSON(RPC): point the client at an endpoint and go to town.

And if you were using Ruby or Python it worked nicely, and still does really.

Well what you seem to be arguing more about the misrepresentation than the final product.

Given the alternatives it really seemed like a lesser of the three/four evils.

I've had the great misfortune to be writing code against an XML-RPC service recently. It's really a miserable design, even compared to contemporaries like Sun RPC.

* Incredibly verbose. Even a trivial array value like [1, 2, 3] encodes to over a hundred bytes. I'm convinced the author was not aware of attributes, or for some reason held a grudge against them.

* There's only like five primitive types, so implementations that want to transmit wild and unusual values such as 64-bit integers need to define their own ad-hoc extensions.

* It's written in XML but there's no namespace, and implementations may or may not use namespaces for their extensions. Is an int64 represented as `i8` or `{http://ws.apache.org/xmlrpc/namespaces/extensions}i8`? Depends on which library you're talking to!

* There's a native "date/time" type, but the syntax is unspecified other than being ISO8601-ish. Don't even get me started on timezones, which will probably be in the local time of the server, but might be in UTC or BST or who knows what.

* The XML-RPC spec claims that <string> elements can be used to transmit any character except '<' and '&', including binary data, including NUL. I want you to imagine what that looks like on the decoding end. Your XML parser is just humming along, decoding UTF-8 and lexing some tags, and all the sudden it comes across a big blob of fucking binary data in the middle of the document. Why was this allowed when the spec also defines a <base64> element??

The XML-RPC spec ought to come with a Surgeon General's warning that reading it might give you brain worms.

> I'm convinced the author was not aware of attributes, or for some reason held a grudge against them.

That is completely unsurprising, attributes suck ass for data, what would XML-RPC even have used them for? Replacing <int> by <integer width=“32”>?

Other serialisation formats make the same choice (e.g. plists) because it’s so much simpler.

> There's only like five primitive types, so implementations that want to transmit wild and unusual values such as 64-bit integers need to define their own ad-hoc extensions.

An issue which, which very much unfortunate, is not exactly shocking. Json would have the same if it had integers in the first place.

> including binary data, including NUL

Nul is a perfectly valid UTF8 character.

  > That is completely unsurprising, attributes suck ass for data,
  > what would XML-RPC even have used them for?
Instead of this:

  <struct>
    <member>
      <name>faultCode</name>
      <value><int>-123</int></value>
    </member>
    <member>
      <name>faultString</name>
      <value><string>some fault</string></value>
    </member>
  </struct>
It could have been:

  <struct>
    <member name="faultCode"><int>-123</int></member>
    <member name="faultString"><string>some fault</string></member>
  </struct>
or even:

  <struct>
    <int name="faultCode">-123</int>
    <string name="faultString">some fault</string>
  </struct>
--

  > An issue which, which very much unfortunate, is not exactly shocking.
  > Json would have the same if it had integers in the first place.
XDR (used in Sun RPC) was defined in the '80s and had 64-bit integers. The DEC Alpha line had been out for 5 years or so when XML-RPC was published. It's not like 64-bit integers were unusual at the time.

  > Nul is a perfectly valid UTF8 character.
NUL is not permitted in XML documents. The XML-RPC spec's explicit directive to allow NUL puts it at odds with standards-conforming XML parsers.
This reminds me that Apple's Plist[1] is similar too:

    <array>
        <dict>
            <key>CFBundleTypeName</key>
            <string>ShapeEditDocument</string>
            <key>LSHandlerRank</key>
            <string>Owner</string>
            <key>LSItemContentTypes</key>
            <array>
                <string>com.example.shape-doc</string>
            </array>
        </dict>
    </array>
( Taken from[2] )

Maybe it's common to design XML schema like this at that time?

[1] https://en.wikipedia.org/wiki/Property_list

[2] https://github.com/robovm/apple-ios-samples/blob/1f6b14ef6e2...

You’re absolutely correct. That’s how XML was written in the late 1990s.
Might be useful to look at what we did for the VWRAP abstract type system in the late 2000's. We generally eschewed attributes in XML and defined a subset of XML that could be automagically converted between JSON and a previously defined legacy binary format.

I'm certainly not saying this is how you should do things, but it's interesting to look at and has the feature that messages in one format (like XML) could be automagically transmogrified into another (like JSON.)

https://datatracker.ietf.org/doc/html/draft-ietf-vwrap-type-...

I feel that the complaints about "old tech", like comparing XML-RPC to XDR or the DEC Alpha at the time isn't very helpful, and maybe lacks some context. The specification came out in 1999, which means that people were discussing the spec for months or even years beforehand. Was any Microsoft-associated engineer using a 64-bit computer during that time? Did they even care? It's a guess on my part, but I wouldn't be surprised they couldn't put 64-bit in the spec since their competitors may have had patents in that space, and some tech companies weren't being friendly with each other at the time.

Also, messages that look like a HTML web page at the time also could have made it seemingly easier for developers to work with, and be attracted to use a new platform.

NT ran on Alpha and MIPS R4000, both of which were 64b architectures, though Windows only ran in 32b until XP64 (there was a 64b 2K prototype targeted at Itanium, but since Itanium ended up delayed it was shelved without release).
According to a post on xml.com[0], development of the spec that would become XML-RPC started in 1998. Microsoft was selling (32-bit) NT for Alpha at that point, and according to Raymond Chen[1] they had an internal 64-bit build of it.

I may be dating myself here, but a lot of folks new to the industry seem to think of anything pre-2000 as a dark time full of greyscale CRTs and dot-matrix printers. While those did exist in the home market, academic and business computing (where most of the advanced tech was being developed) was pretty advanced and a modern engineer would have felt right at home on a mid-90s UNIX workstation.

If you go back further you'll start encountering things like CORBA, which were pretty bad, but had the legitimate excuse of being very early. Remember that the '90s moved fast by modern standards -- CORBA was contemporary with MS-DOS, while XML-RPC was after IE had dethroned Netscape.

[0] https://www.xml.com/pub/a/ws/2001/04/04/soap.html

[1] https://docs.microsoft.com/en-us/previous-versions/technet-m...

Windows on the DEC Alpha was still in support at the time, I think, so someone at Microsoft must have used it.
Attributes suck when you're trying to write that data into a database. That's the main reason why lots of implementations enforced no attributes, because it made writing that data into a database table very hard.
JSON was originally a transfer syntax, not a serialization format. But then when people wanted a definition, it got documented with type semantics. So there's all sort of mistakes that were made along the way...
> I'm convinced the author was not aware of attributes, or for some reason held a grudge against them.

The XML heyday is a little before my time, but I think attributes were considered improper design compared to only using tags by purists.

It is arguably "good design" because tags are infinitely extensible whereas attributes are not.

For example in the case the author supplied imagine if you wanted to add a compression type to a member. Your only choice would be an attribute which means that it can only be a string.

  <struct>
    <member name="faultCode"><int>-123</int></member>
    <member name="faultString"><string>some fault</string></member>
  </struct>
If in the original you could add a new element inside the member like

  <compression>
    <algorithm>zstd</algorithm>
    <dictionary>dict7</dictionary>
  </compression>
Not the best example but the point stands. Of course the complaint is valid because while extensibility is valuable you need to weigh it against the cost of the verbosity (both in use and for developers).
Note that in the specific case of XML-RPC, extensibility is forbidden by the spec:

  A <fault> struct may not contain members other than those specified.
  This is true for all other structures. We believe the specification
  is flexible enough so that all reasonable data-transfer needs can be
  accomodated within the specified structures. If you believe strongly
  that this is not true, please post a message on the discussion group.
At one point the author did intend to allow elements to contain user-defined children[0], but it seems like their position changed between that mailing list post and the spec update 7 days later.

[0] https://web.archive.org/web/19991010205056/http://discuss.us...

My recollection of the personalities involved is that the author is absolutely not a purist. He's a pragmatist, and this was simple, understandable, and worked well enough at the time. He was doing this because he needed the functionality, not because it was the best design possible.

I also think that this design was also reasonably well aligned with the internal XML capabilities of Frontier.

> My recollection of the personalities involved is that the author is absolutely not a purist.

The opposite really, half-assed "good enough for me" is also very much visible in RSS.

At the time, the point was simple, non-binary RPC calls. I'm pretty sure that the spec came from a hacked up client and server, then frozen, with all warts intact. I'd be surprised if the initial (and final) spec had more than a handful of weeks of actual work in it. At the time, it did mostly work, and it did enable an explosion of connectivity in the early weblog adjacent world.

SOAP was basically XML-RPC++, with namespaces, 4 part harmony, and 8x10 color glossy photographs. It's the codification of XML-RPC adding all of those XML features, data types, and interface definitions. It's also a miracle if there's any interop between different stacks.

Compared to SOAP, this was pretty minimalistic and a pragmatic way to talk to a server from a browser. And of course people actually used SOAP as a really convoluted way to do RPC. This was the lightweight alternative. Json rpc also briefly was a thing.

I had the "pleasure" of dealing with a few SOAP web services a few times. That stuff was just completely horrible and unusable.

HTML5 became a thing when W3C began insisting that XHTML should be a thing, complete with namespaces and other nonsense. That was peak XML basically. Developers rejected it. Browser vendors and content creators basically stepped in and fixed HTML and CSS properly because w3c could not be bothered to step over the semantics and actually define how that stuff was supposed to behave in the real world (i.e. specifiying the semantics of their semantic HTML).

Come to think about it, there was a whole bunch of things that came out of w3c that essentially was about using namespaced XML: web service architecture, RDF, semantic web, XHTML, etc. All that stuff seems to have faded away into obscurity.

XML namespaces sound like a good idea until you realize that it just means endless verbosity that doesn't really serve much purpose and that really complicates things like parsing. Thankfully, people simply refused to apply that to json. Most json continues to be free of that nonsense. Same with yaml and similar formats. Works great without a lot of namespace urls.

It wasn’t “people” who saved JSON from this nonsense. It was Douglas Crockford, alone. I have been frequently annoyed at him for not allowing comments in JSON, but the explicit reason was that people were using json comments to encode all the metadata and crap from XML like namespaces to make fully reversible JSON/XML converters. And Crockford very much did not want JSON to be compatible with XML - he wanted it to be a simpler alternative. So good on him.
I hadn’t heard that explanation for not having comments, but if true, I want to buy him a drink.
> the explicit reason was that people were using json comments to encode all the metadata and crap from XML like namespaces to make fully reversible JSON/XML converters

Oh is that why. I had heard that comments were taken out because people were using it for "parser directives" or something vague. This makes a lot more sense (although I'm sure there were all manners of hacks in the comments). Definitely a bullet dodged there.

Oh many people have tried adding namespaces to json. It's just that these people and their solutions are consistently ignored by most developers. If you consider that Json came out of the javascript world where people tend to prefer to not specify a lot of types, it's perfectly obvious that they would probably not want or need any namespaces. Never was a thing in Json. And good on Douglas for keeping Json simple. The comment thing is annoying indeed. Using a good parser, you can usually configure it to not freak out over stuff like that. But sadly that is not very common.

Elasticsearch is a positive exception. It will happily accept comments.

I don't understand why is it hard to build a JSON/XML converter? XML is represented by a DOM and DOM should be easy to convert to JSON. Actually I did just that. I made a service which uses XML for inputs and outputs and those JavaScript people kept asking me about JSON. Not sure if they were happy in the end, JSON is too verbose to my taste, but whatever.

So some simple XML like

    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
        <soap:Body>
            <soap:Fault>
                <faultcode>soap:Server</faultcode>
                <detail>
                    <ns2:SendMessageFault1_SendMessageFault xmlns:ns2="http://bip.bee.kz/SyncChannel/v10/Types">
                        <errorCode>SCE013</errorCode>
                    </ns2:SendMessageFault1_SendMessageFault>
                </detail>
            </soap:Fault>
        </soap:Body>
    </soap:Envelope>
is represented by the following JSON monstrosity:

    {
      "name": "soap:Envelope",
      "attributes": [
        {
          "name": "xmlns:soap",
          "value": "http://schemas.xmlsoap.org/soap/envelope/"
        }
      ],
      "children": [
        "\n    ",
        {
          "name": "soap:Body",
          "children": [
            "\n        ",
            {
              "name": "soap:Fault",
              "children": [
                "\n            ",
                {
                  "name": "faultcode",
                  "children": [
                    "soap:Server"
                  ]
                },
                "\n            ",
                {
                  "name": "detail",
                  "children": [
                    "\n                ",
                    {
                      "name": "ns2:SendMessageFault1_SendMessageFault",
                      "attributes": [
                        {
                          "name": "xmlns:ns2",
                          "value": "http://bip.bee.kz/SyncChannel/v10/Types"
                        }
                      ],
                      "children": [
                        "\n                    ",
                        {
                          "name": "errorCode",
                          "children": [
                            "SCE013"
                          ]
                        },
                        "\n                "
                      ]
                    },
                    "\n            "
                  ]
                },
                "\n        "
              ]
            },
            "\n    "
          ]
        },
        "\n"
      ]
    }
Actually, that's what an automated translation would look like. A human would likely represent this as follows:

  { "errorCode "SCE013" }
That SOAP envelope is 99% just telling you it's a SOAP envelope, which technically is completely redundant because presumably you'd be aware of the fact that you are talking to a SOAP API. And apparently they had no schema for actually telling you anything meaningful about the error. Which is why it is represented as a string with an error code. That string is the only useful bit of information in the message.

Basically the SOAP equivalent of Internal Server Error.

From a parsing point of view it's horrible because all those namespaces need to be resolved and the schemas need to be processed. And of course a lot of those namespace urls don't actually have servers behind them that serve those schemas. Most of those urls never resolved to anything useful at all. The servers that were supposed to host those schemas never existed.

Alternatively you just ignore all that crap and pick the message apart with XPath or a regular expression. That's what I used to do when dealing with SOAP APIs. Most of those APIs were just stupendously convoluted RPC wrappers anyway from some bit of enterprise crap ware. I never could be bothered to treat those systems with the respect they demanded from me. Way faster to just sidestep all that nonsense. Send a message, pick apart what comes back to determine success / failure and move on with your life.

That's why SOAP died. Because as soon as people caught up to this being the optimal way to deal with SOAP, the next logical thing was to get rid of it entirely.

Not a super fan of JSON, not least because it has no comments, but that was totally the right call on his part. We'd have a ton of semi-de-facto crap to parse out of comments by now.
This brings back bad memories. I once worked on a mobile app that used XML-RPC for communications, in the late 2000's. I remember looking at packet captures of this stuff. Ugh.
20 years ago or so I wrote a LiveJournal client that used the XML-RPC interface. At the time the choices were that or an older custom text interface. The XML-RPC interface was more understandable to a very beginning developer and even then there were libraries I could use in Visual Basic that took care of all the hard parts. The worst part was that not everything was available via the XML interface so you still had to drop back to the old text interface for some things.

While yeah it’s pretty dated by modern standards at the time it was a revelation that you could build standard-based APIs that anyone could use with a simple library and without having to write a TON of custom code.

And regardless, it was far easier to implement than SOAP, which was just a mistake unless you are fully immersed in the Java ecosystem.

Still using JSON-RPC 1.0 over HTTP here:

  - specs so simple it's incredibly easy to implement client and server in most languages
  - good enough for most use case, supports whatever that can be expressed in JSON
  - easy enough to inspect, it's just JSON
  - not too heavy compared to XML-RPC
  - as performant as your JSON serializer/deserializer
  - None of the ReSTish philosophical questionings
  - understood by developers of any generation: call methods with parameters
also:

> - None of the ReSTish philosophical questionings

Adding to this,

- If you ever -do- need to drop to something else (i.e. GRPC, websockets) you're not having to re-translate all of your restishness into commands.

I used it just the other day to talk to a Flectra server.
Pretty sure the WordPress API is still XML-RPC. Like many things in tech, it’s a lot more common than you might think, even after three or four “replacement” technologies.
Just to repeat what everyone else is saying: it was awful, but a whole lot better than SOAP.

I wrote a service to allow BSD servers running Python apps to run SQL queries on a Windows server running Visual FoxPro, and everything at the time said SOAP is what you use for such things. That was overengineering writ large. After a while I replaced it with XML-RPC and life got much easier.

If JSON had widely existed at the time, and someone would have shown it to me, they would’ve been my friend for life.

Why did it take so long for people to come up with something like JSON? Why did we start out with such overly complicated formats?
Tech is driven by fads and developers perpetually chasing the next shiny object. XML was the flavor of the day back in the late 90's.
It's important we publish things like this so the next generation will know what not to do.
I worked with Edd Dumbill[1] to bring HTTPS and certificate support to the PHP XML-RPC bindings. It was one of my very first open source contributions and interactions, and it was super empowering. I was running the tech at Voxel.net at the time, an early web hosting provider to many open source projects. We were using XML-RPC to write the beginnings of Ubersmith[2], which was our billing, hosting, and support management platform.

Later, those bindings made it into very early Drupal core[3] and onto thousands of websites. In this era, you could make desktop apps talk to websites using an XML-RPC gateway — for content management or many other tasks.

Yes, XML and related tech is fairly horrible, but context is everything.

If you were running servers, there was enormous pressure to use Microsoft.

If you were by chance running open source (LAMP stack), making applications work together was a challenge. Interoperability was not the norm, despite a pretty rich internet. Formats and standards were the problem.

You would email code patches around. There was no GitHub, and SourceForge was only starting to gain traction.

If were using open source version control in this era, you were likely on CVS, which was an "improvement" over RCS, but still nothing like the promised future tech of Subversion, which wasn't text-backed. Text-backed! Versions of files were concatenated, and instead of force pushing, you opened this concatenation abomination in a text editor to hack the repo history (if you were a bad, bad person, but needed to get the job done).

As mentioned many other places in this thread, if you were doing open source interop, the heavyweight option was SOAP. XML-RPC was as much a breath of fresh air as JSON is to XML.

Fairly-literal text was bloated, slow, and XML even more so, but it was all pretty cutting edge for the time.

[1] https://www.xml.com/pub/au/11

[2] https://ubersmith.com

[3] https://git.drupalcode.org/project/drupal/-/blob/4.0.x/inclu...

Anyone who touched SOAP or CORBA for any amount of time larger than an hour have fond memories of XML RPC, despite all the flaws. I know I do.
SOAP was also incredibly verbose, but being able to generate both client code and server stub code off of the spec (WSDL) was useful. There was no tedious hand coding of clients like you'd often see with REST.
Have to agree with this - context is everything and there are an awful lot of naive comments here who don’t understand that XML-RPC was a breath of fresh air compared to SOAP, especially if like me you were using the LAMP stack and PHP. SOAP was abysmal- I really don’t recal a single project where communication between two systems using SOAP wasn’t without serious issues, incompatibility issues and just plain broken.

Many developers here might not recall serious compatibility issues with Microsoft- the most obvious one I recall was WebDAV ; pretty much strangled at birth by MS terrible broken implementation.

I used to use the PHP XML-RPC implementation you worked on for so many projects, so thanks for that- helped me to integrate so many projects, so cheers!

To be honest, I am very fond of XML-RPC. It is a very easy protocol to understand and implement. It is my first choice for my own small personal projects.
XML: The greatest productivity destroyer of the 20th century.
Only when developers insist in using stuff like vi to write it by hand.
I always expected the great boon of XML to be a proliferation of tools leveraging XSD to provide GUIs for managing the XML files. XML seemed like a great way to manage config files, but even auto-complete wasn't enough IMO. I'm thinking specifically about Apache Jakarta config files.

Not sure if any ever existed. Of course, I'm unsure if XSDs were much used outside of big Java shops.

They did exist, on the domains where development tools actually get paid for.
"Fun" Fact: ROS (the "Robot operating system") has used XML-RPC under the hood for many years. While it's finally transitioning away with the move to ROS2, I expect that it will still be in use in this space for many years to come, for better or worse.

http://wiki.ros.org/ROS/Technical%20Overview