Nope, The author's view of glue code is that most of it is to make the data output by function A a usable input for function B. This varies from the trivial (I'm moving an integer from A to B. Piece of cake as long as A & B both have the same sized integer with the same endianness and we've defined what to do if A is outputting a 64-bit value while B is inputting a 32-bit value and A sends something that doesn't fit in 32 bits) to the nightmarish (a large, poorly documented binary blob with inconsistent protocols) to lots of stuff in-between. Programming is easy. Data is hard.
> as long as A & B both have the same sized integer with the same endianness and we've defined what to do if A is outputting a 64-bit value while B is inputting a 32-bit value and A sends something that doesn't fit in 32 bits
Well put. I still remember the time I helped a junior colleague debug an error (in C) simply by suggesting he look at the size of the types he was computing. Lightbulbs went off.
In Haskell, monads are all about composition of types. While the glue code doesn't go away, it stays neatly in the bind operator's definition for a given monad, which makes the rest of the code compose very easily. So no, it doesn't eliminate it, but I think it makes it a lot more manageable.
I also view glue code as going up with the square (n^2) of the number of connections (n) grows between two conceptually connected but implementation-disconnected systems.
But that is hardly the worst of it.
The two system model generalizes to much worse complexity (n^m) relative to the number of connections (n) that involve tight linkage of a multiple number of systems (m >= 2).
In practice, we all avoid m>2 wherever we can, and simply slog along less and less efficiently with m=2 as we try to add new value (or maintain value in the presence of necessary changes).
So the general law is mostly apparent in the broad absence of conceptually simple m>2 solutions, as apposed to their existence as quagmires.
Simple thought experiment to illustrate m>2 solutions:
“Captain! If we just reprogram the nanobot solar system explorer cannon to spread the bots on a tight bezier trajectory matching our enemies detected firing pattern, adjust their transmission frequencies to pass through Romulan dark hull steel and resonate with their traditional diptholamite crystal bathroom fixtures, we can apply a combination frequency comb filter and inverse fractal hash to completely uncloak their vessels in real time!”
“But Chief Engineer, there is but one problem!! While conceptually simple, your elegant solution will take one hundred years to implement! None of those systems were designed together!”
Kaboom. Hissss… Silence
I call the problem of rapidly integrating foreign systems into conceptually simple novel solutions “The Star Trek Rapid Integration Problem.”
Apparently this problem has been solved by the Federation of Planets in our future.
It seems like there most be some group theory formulation of this problem, where each "gluing" can be seen as different symmetries of a software system.
In the future, Star Trek equips its starships with casks of Phlebotium, which is then applied generously to both systems in such situations, allowing them to interact seamlessly:
> Glue isn’t some kind of computational waste; it’s what holds our systems together. Glue development is software development.
My take is: glue code is just “obvious” code, as opposed to the less obvious code.
The problem is that it’s not really obvious. That’s how you get “write-only” code. What’s obvious to one person isn’t to another, or even to that same person later on. This is why Java, a very verbose (“full of glue”) language, is so popular.
People say “an ideal programming language would have no glue code, it would all be inferred by the compiler”. And it’s true that a good language and libraries can remove a lot of trivial boilerplate. But not all of it, because some of that “boilerplate” isn’t so trivial.
Like, casting floats to integers. The cast is glue code. C and other languages truncate the float, so it gets floored if positive and ceiled if negative. But do you want that, or do you want to round the float or ceil it always? Or is casting the float to an integer an indicator that you’re using the wrong variable?
—-
I think in the future, we will create AI to write a lot of ”glue” code fast, like GitHub copilot. Also, proven-equivalent (or heuristically 99% predicted equivalent) large-scale refactors. But even in the ideal scenario where we eliminate low-level glue code, we’ll just consider higher-level code “glue code”. Because, all code is glue code, to convert “what I want” to “what computer does”.
The last few sentences of the article capture how I feel about the whole thing:
> Programming is ultimately about gluing things together, whether they’re microservices or programming libraries. Glue isn’t some kind of computational waste; it’s what holds our systems together. Glue development is software development.
Unless you're programming in assembly, aren't you always essentially writing glue code? Your language has some number of predefined assembly instructions which you can call in different ways and connect to fulfill your needs.
I think there's a real meaningful difference between glue code and business logic, even if your business logic is created by encapsulating low level operations that are hidden by your PL, the total reductiviness to "glue" is not helpful
Someone in this thread gave an example of casting floats to integers being glue code. But how exactly you want to cast - using round, floor or ceil is a business logic. Where is the meaningful difference in this example?
I'm not sure about that. But business logic is what you have to do de Novo to make your product. I work for an invoice processing company. Connecting to a/many Saas is glue code (this is most of our codebase). The code that decides the rules for how an invoice is "approved" by reviewers is business logic.
If you're intentionally rounding a number because it's in some algorithm, or maybe your tax law, or to give discounts to customers, then it's not glue code. Using a cast to round down will obscure the intention, but I'd argue it's not really glue code.
However, if you cast the float because you have two systems and the second one only accepts integers, then you are using glue code to make the two incompatible things work together.
The lines a bit blurry sometimes.
But the distinction is still helpful, because less code can potentially (but not necessarily) mean less bugs, less time needed to understand the system, less cost, more performance, etc. And glue code is code that you can potentially (but not necessarily) remove.
Is the choice meaningful? Or just something you have to type out?
In the casting example, it's meaningful: there are several ways to "round" a float, each having different consequences. As a programmer, you have to make a choice driven by some business requirements, and encode it (preferably only once for a given business concept).
Sometimes, however, you can modify the source of the float - push the decision upstream, so that your code always gets an int the way you like. This is one way of reducing glue code, but it's not always available.
Anyway, a different type of glue code - I think the most common one, one I call "code bureaucracy" - is reconciling concepts between systems/modules. One system produces data that's almost, but not quite, like what the other system wants to consume. Sometimes it's literally the same data, but named as two separate concepts with separate ownership. This kind of glue code can grow in size rapidly as a consequence of following seemingly good design practices, and often feels pointlessly wasteful.
Imagine you're writing a 3D video game. You decide to use a library to load 3D models. You give it a data stream, and get back a list of (x, y, z) points. Unfortunately, despite containing exactly the same data, perhaps in exactly the same memory layout, that list of points is not a proper "list of points" as the rest of your game logic understands it. You're faced with 3 options:
1. Change your game to use library's format for representing 3D data. You're reluctant, because it would tie your system to the API of a third-party component. It would also be difficult if you used another library (perhaps for physics handling) that used its own "list of points" representation.
2. Change the library to use your game's format. Not the work you wanted to do, and the library author won't be interested in it anyway.
3. Write glue code to convert between the formats. Can be as simple as switching out typesystem labels, or as complex as translating each point in a loop.
For obvious reasons, everyone will choose option 3. What's curious is, though, that the reasons are more social than technological. You don't want 1. because it will make you dependent on a third-party system. That system's author won't accept 2., because it'll make them dependent on you. Hence, option 3.
The exact same thing happens internally, within subsystems developed by different people, and even with code of a single author, across abstraction boundaries. It's an ironic example of Conway's law: system design reflects social structures designers live in. After all, there is another option - one that is rarely thought about, one that's sometimes considered a violation of good design - one that's socially expensive:
4. Get together with the other system's author (and possibly with other users) and agree on a common representation you'll both use.
It breaks abstraction layers. It breaks ownership separation. It breaks responsibility division.
4. Get together with the other system's author (and possibly with other users) and agree on a common representation you'll both use.
Good standards are a wonderful thing, and IMHO we as an industry tend to invest far too little in establishing them now.
I believe programming languages can help a lot here, if you take the view that their standard library should be not just a ready-made toolbox but also a library of standards, in the sense of establishing concepts and (depending on the language) terminology, interfaces, types, symbolic constants and other common foundations for implementations in a specific field.
An example of not doing this is JavaScript, where the paucity of standard data structures and algorithms out of the box leads to huge trees of almost-trivial dependencies and to a culture where almost any data we need to move around just gets shoehorned into objects or transmitted as JSON.
A more positive example is Haskell, where the standard typeclasses capture several broadly useful programming patterns and given them a widely recognisable name and interface. Haskell libraries (standard or otherwise) can then provide algorithms with generic interfaces defined in those terms that are not tied to specific implementations or specific data structures. This results in an ecosystem of data structures and algorithms that are relatively easy to compose even when coming from different sources, which often allows Haskell programmers to express ideas very concisely. A related example is the use of generic programming within the standard library of C++, particularly with the arrival of concepts in C++20 that should serve a similar role to typeclasses in Haskell.
Both of those languages also learned some of these lessons the hard way. Each of them failed to establish a single, standard way to represent something as “simple” as a text string, with the result that many popular libraries did things their own way, and passing text around between things like databases, UIs and remote communications protocols can be a chore to this day.
I think it’s interesting that in some cases, the community around a language has evolved a kind of de facto standard interface for common tasks like accessing a SQL database, so that for example whether you’re talking to Postgres or Maria or SQLite or whatever, the corresponding drivers all provide very similar interfaces and are (mostly) interchangeable as far as calling code is concerned. I think it would be helpful if we did a lot more of that as an industry, within and perhaps even across languages, so we don’t keep reinventing the wheel every time we need to access a relational database or to draw a simple GUI or transmit data using typical Internet protocols or represent mathematical structures beyond the very basic ones.
float to int destroys information... unless the first program uses a float to store integers, so that that information is not destroyed. Similarly, the values could all be something-point-5 (1.5, 2.5, etx). That is, the literal type may be a float, but the application type is narrower than that. Like non-significant white-space, or a set whose elements are listed in an order, but that order is not significant.
A similar view is that, even if the application type for the first program really is a float, and the non-integer part is signicant to that program, that the type for this specific transfer is an integer. The specific way it's done depends on what's needed.
Is this "business logic"? Maybe. I would tend to say business logic requires some computation, other than a lossless transformation. Putting it into a different form, that is exactly equivalent, I would call glue.
I don't believe there is any authoritative definition; all we can do is note distinctions.
Perhaps I'm just playing semantics with a "transfer type"... but I think there is something important and useful in this distinction.
> the real problems stem more from schemas than data formats
Oh, how the turntables turn. Ontologies are hard, and always fail in some way[1]. There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thing, but they are expensive and time-consuming and even when they work, only approximate.
Anyone has ever done a data migration project of any complexity knows that there must be provisions for dealing with data quality issues, especially if effort includes reconciling data from multiple sources.
> There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thing
The more code I have written, the more convinced I become that "general" solutions almost never pay for themselves when you compare the implementation and/or performance complexity required to make them work.
It's almost always better to "just write the damn code" and solve the exact problem you actually have.
If every time you do it is a one-off, yes. If your business is essentially nothing but aggregating datasets (which may themselves be aggregates of still more datasets), then by the third time you've solved the exact problem, you're going to see commonalities. You abstract those out so that the fourth, fifth, and sixth problems don't have to re-write that part. After a while, you've built out enough abstractions that you're programming the migration process itself, not each migration.
Add in the fact that your business depends on doing it at scale, ingesting live data as it is generated, and developing an entity resolution system is almost a no-brainer.
We did a great many of them over the decades; I find the mongoldb ones we encountered the most horrible (even compared to vague 80s + 90s plain text formats as they usually at least had some documentation as to what the developer was storing and how), but in general, we found the most reliable and, when you are used to it, the fastest (in the end) is creating types for everything and convertors from one to the other type. We did try other (more sloppy) ways before it and they always kind of result in bad quality. We have noticed that making (very exact) types for everything when doing complex conversions works well. Most notably, we try to avoid stringy types and add a lot of validations in the deserialisers. You can run conversions 10 years later on new batches and still get understandable errors.
> try to avoid stringy types and add a lot of validations
Right there you've stated a key practice that may, by itself, be 80% of the solution. Of course there's no "right" set of of types, but just putting the work into thinking about the types and the essential constraints will drive towards exposing and eliminating assumptions that will come back to bite later.
Another huge chunk of problems can come from over-reliance on language primitive types. Is that numeric field really an integer, meaning that any signed value representable in 32, 64, or however many bits, including zero, is valid? Probably not. To give an example: A person's age is not an signed integer, it's a duration, we don't expect it to be negative. Maybe we should flag values above 90 or 100. If you're doing a one-off, that's the sort of thing you can code once and forget. If you dataset migration is your business, your Age type needs to be more robust, and perhaps there's even a need for the behavior of Age to vary with the domain.
We work in C# mainly; unfortunately; I have been in favour of F# for a long time, but it's just not popular enough so it would make our (already large, with decades of work) codebase less maintainable. However it does not really matter as long as you can make the types (classes in the case of C#) precise enough. You can indeed (see the other commenter) make (an) Age classe(s) which will give a nice error message if you do something illegal, for instance;
int i = -2;
Console.WriteLine((Age)i);
Console.WriteLine((Age)200);
Console.WriteLine((AdultAge<UnitedStates>)17);
will throw three Exceptions. We define every single piece of data like that.
One thing that evolved over the years is swagger. It is a defined way to document APIs - the endpoint, return type, returning data structure, arguments, behavior, etc.
Microsoft took this as a way to connect different APIs in their logic app (think of workflow definition like an advanced version of ifttt, n8n, ...).
It certainly doesn't remove the need for glue code but it reduces the amount of it!
On the flip side you have up the control of the program flow by using their framework that handles all the connections between APIs.
All swagger really is is a tool for formalization of HTTP/JSON interfaces. And that's great! Formalized interfaces are really the best way to reduce the need for glue-code and post-processing, since the parties on both sides can agree on the exact target their shooting for.
The simplest solution to having N data types and N^2 conversions between them is to have a canonical data type and 2N conversions between that one type and the rest.
The real problem is that all that glue code is proprietary, and companies cannot cooperate on a large scale to pool their resources and share their software.
I’m not sure I buy the N² argument being made here.
If we’re talking about communications between components within a software system then yes, for N components, there are roughly N² possible connections between them. But isn’t managing that complexity so we don’t connect everything to everything Software Design 101? If you design your system using structure like layers and pipelines then you’re effectively reducing that O(N²) connectedness to something more like O(N).
If we’re talking about having multiple choices for how to implement each component, such as when we have several libraries available plus the option of writing something from scratch, then given N choices for the first component and M choices for the second, there are NM possible connections we might have to implement. However, we probably only need one of those connections at any given time. If we need to modify or replace either implementation later, the most important thing for flexibility and future-proofing is that the data models and communications protocols are well specified, ideally following established standards that are likely to be followed by the other alternatives as well. Again, though, separating interface from implementation is Software Design 101.
So it seems to me that if either of those interpretations was intended, we’re talking about a problem that has largely been solved for a long time. Perhaps part of the reason it doesn’t always feel that way today is that there is so much emphasis on doing everything in “agile” fashion now. Sometimes that agility comes at the expense of establishing a well-considered software architecture and long-lived standards for interoperability. The “glue code” in software systems is where we pay any interest on those forms of technical debt, so while there will always be some need to marshal data around a large system with many components, there will be much, much more code needed if those forms of complexity aren’t well managed.
came in here to say the much the same. I also see much more that we get "pre glued" things that we then glue a few more things to. After 40ish years of coding, I think the amount of glue we use is about the same. It goes up and down a bit but there seems to be a natural limit to the complexity we can deal with.
I also see much more that we get "pre glued" things that we then glue a few more things to.
Do you mean other trends like increasing use of frameworks, scaffolding tools and “convention over configuration”? If so then I agree. One of the trade-offs being made there is immediate convenience against long-term flexibility. You get more functionality out-of-the-box with these kinds of tools, but the price is that they often don’t compose well.
I don’t think either article is talking about every component of the system, just the things that do need to be aware of one another. Marcel Weihers original article says:
> Glue is quadratic. If you have N features that interact with each other, you have O(N²) pieces of glue to get them to talk to each other.
And this one says:
> it’s “quadratic”: the glue code is proportional to the square of the number of things you need to glue
Even the most well-designed architecture will not let you only glue things pairwise, and you don’t need a very high N to run into challenges with N^2.
Sorry, I still don’t buy it. How often do you really have a set of features that all need to interact directly with every other feature in a well-designed system, other than for small N like, say, an MVC triangle?
Very often, a combination of better software design and standardisation can reduce the actual complexity of an implementation below the apparent complexity. For example, suppose you have a system where N components can produce a certain type of data and another M components can then consume that data. This might appear to be a system with N×M connections between components. However, these components don’t necessarily need to be directly glued together in the implementation. If you define a standard, canonical representation for this type of data, you only need each supplier to convert into that format and each consumer to convert from it, and now you have an N+M glue problem instead of the apparently N×M problem you started with.
Of course you can do better than just blindly gluing everything onto everything. But as a general strategy you’re not really escaping N×M. The data format needs to support everything, and all of the consumers need to be adapted to any change in any producer. That’s your glue!
If your design is data-driven, and the data specification in turn comes from your requirements, why would you be updating your common, standardised data format just because any particular supplier or consumer of that data changed? The dependencies appear to be the wrong way around in that situation.
If there is a material change in requirements, you update your data interchange standard accordingly, and then you still only need to adapt N+M producers and consumers to work with the updated standard.
Either way, I still don’t see how we end up back with N×M. The consumers and suppliers are still communicating via the intermediary format and not all talking directly to each other.
But this is a “let them eat cake” argument. It doesn’t matter that there are strategies to mitigate the problem. Most parts of most systems don’t communicate through a standardized data format, but an ad-hoc interface that addresses the immediate needs of the current requirements.
Most parts of most systems don’t communicate through a standardized data format, but an ad-hoc interface that addresses the immediate needs of the current requirements.
That’s actually a very big claim, and I don’t know how we could possibly test it.
Counterexamples are easy to find, however. Consider that you’re reading this in a web browser that is communicating with a web server hosting HN. That communication, as indeed the article itself observes, happens via several standard protocols layered atop one another. I didn’t really understand the argument the article made at that point that this doesn’t solve the N² problem because it doesn’t take into account the application data at the top layer, as in practice almost everything sent over the Internet is also using standard application protocols like HTTP or SMTP as well. This has enabled literally billions of devices to communicate many types of data effectively on a global scale with a wide variety of software implementations on both ends of each connection, so IMHO it’s a pretty convincing example of the benefits of effective standardisation.
Of course it’s still true that some software systems have connections all over the place using ad-hoc data formats and protocols, but often that is simply bad design. Software design is a skill and like any other skill you don’t get good at it until you’ve worked on it for a while. My argument here, in a nutshell, is that if you learn good design skills and make the effort to establish good software architecture and useful standards for a project, very often the N² problem described in the article isn’t such a problem after all.
Sorry, I’m not following your argument. What is also big-O? And why is suggesting that the relevant scenario rarely if ever happens in practice suggesting agreement with anything?
I've heard this kind of expression differently formulated (I think often by Steven Sinofsky while he was the head of Windows) as the complexity of the software unavoidably increasing quadratically with either time or with feature-count.
So maybe it's less about the # of systems here specifically, more about just the # of "things it does", where most of what it does is connect systems together.
About forty years ago, I did a weekend consulting gig at a lab in San Diego where they had a massive N^2 connection problem of data processing code vs. data formats. I wrote a FORTRAN program to enumerate all combinations and generate a lot of very baroque code. My customer was not happy with the solution until I showed them how simple it was to modify the generating code and regenerate the glue code. Then they liked the approach. If I had written this a few years later, I would have used Lisp, and things would have been even simpler.
45 comments
[ 2.9 ms ] story [ 90.2 ms ] threadWell put. I still remember the time I helped a junior colleague debug an error (in C) simply by suggesting he look at the size of the types he was computing. Lightbulbs went off.
But that is hardly the worst of it.
The two system model generalizes to much worse complexity (n^m) relative to the number of connections (n) that involve tight linkage of a multiple number of systems (m >= 2).
In practice, we all avoid m>2 wherever we can, and simply slog along less and less efficiently with m=2 as we try to add new value (or maintain value in the presence of necessary changes).
So the general law is mostly apparent in the broad absence of conceptually simple m>2 solutions, as apposed to their existence as quagmires.
Simple thought experiment to illustrate m>2 solutions:
“Captain! If we just reprogram the nanobot solar system explorer cannon to spread the bots on a tight bezier trajectory matching our enemies detected firing pattern, adjust their transmission frequencies to pass through Romulan dark hull steel and resonate with their traditional diptholamite crystal bathroom fixtures, we can apply a combination frequency comb filter and inverse fractal hash to completely uncloak their vessels in real time!”
“But Chief Engineer, there is but one problem!! While conceptually simple, your elegant solution will take one hundred years to implement! None of those systems were designed together!”
Kaboom. Hissss… Silence
I call the problem of rapidly integrating foreign systems into conceptually simple novel solutions “The Star Trek Rapid Integration Problem.”
Apparently this problem has been solved by the Federation of Planets in our future.
https://tvtropes.org/pmwiki/pmwiki.php/Main/AppliedPhlebotin...
My take is: glue code is just “obvious” code, as opposed to the less obvious code.
The problem is that it’s not really obvious. That’s how you get “write-only” code. What’s obvious to one person isn’t to another, or even to that same person later on. This is why Java, a very verbose (“full of glue”) language, is so popular.
People say “an ideal programming language would have no glue code, it would all be inferred by the compiler”. And it’s true that a good language and libraries can remove a lot of trivial boilerplate. But not all of it, because some of that “boilerplate” isn’t so trivial.
Like, casting floats to integers. The cast is glue code. C and other languages truncate the float, so it gets floored if positive and ceiled if negative. But do you want that, or do you want to round the float or ceil it always? Or is casting the float to an integer an indicator that you’re using the wrong variable?
—-
I think in the future, we will create AI to write a lot of ”glue” code fast, like GitHub copilot. Also, proven-equivalent (or heuristically 99% predicted equivalent) large-scale refactors. But even in the ideal scenario where we eliminate low-level glue code, we’ll just consider higher-level code “glue code”. Because, all code is glue code, to convert “what I want” to “what computer does”.
> Programming is ultimately about gluing things together, whether they’re microservices or programming libraries. Glue isn’t some kind of computational waste; it’s what holds our systems together. Glue development is software development.
Unless you're programming in assembly, aren't you always essentially writing glue code? Your language has some number of predefined assembly instructions which you can call in different ways and connect to fulfill your needs.
If you're intentionally rounding a number because it's in some algorithm, or maybe your tax law, or to give discounts to customers, then it's not glue code. Using a cast to round down will obscure the intention, but I'd argue it's not really glue code.
However, if you cast the float because you have two systems and the second one only accepts integers, then you are using glue code to make the two incompatible things work together.
The lines a bit blurry sometimes.
But the distinction is still helpful, because less code can potentially (but not necessarily) mean less bugs, less time needed to understand the system, less cost, more performance, etc. And glue code is code that you can potentially (but not necessarily) remove.
In the casting example, it's meaningful: there are several ways to "round" a float, each having different consequences. As a programmer, you have to make a choice driven by some business requirements, and encode it (preferably only once for a given business concept).
Sometimes, however, you can modify the source of the float - push the decision upstream, so that your code always gets an int the way you like. This is one way of reducing glue code, but it's not always available.
Anyway, a different type of glue code - I think the most common one, one I call "code bureaucracy" - is reconciling concepts between systems/modules. One system produces data that's almost, but not quite, like what the other system wants to consume. Sometimes it's literally the same data, but named as two separate concepts with separate ownership. This kind of glue code can grow in size rapidly as a consequence of following seemingly good design practices, and often feels pointlessly wasteful.
Imagine you're writing a 3D video game. You decide to use a library to load 3D models. You give it a data stream, and get back a list of (x, y, z) points. Unfortunately, despite containing exactly the same data, perhaps in exactly the same memory layout, that list of points is not a proper "list of points" as the rest of your game logic understands it. You're faced with 3 options:
1. Change your game to use library's format for representing 3D data. You're reluctant, because it would tie your system to the API of a third-party component. It would also be difficult if you used another library (perhaps for physics handling) that used its own "list of points" representation.
2. Change the library to use your game's format. Not the work you wanted to do, and the library author won't be interested in it anyway.
3. Write glue code to convert between the formats. Can be as simple as switching out typesystem labels, or as complex as translating each point in a loop.
For obvious reasons, everyone will choose option 3. What's curious is, though, that the reasons are more social than technological. You don't want 1. because it will make you dependent on a third-party system. That system's author won't accept 2., because it'll make them dependent on you. Hence, option 3.
The exact same thing happens internally, within subsystems developed by different people, and even with code of a single author, across abstraction boundaries. It's an ironic example of Conway's law: system design reflects social structures designers live in. After all, there is another option - one that is rarely thought about, one that's sometimes considered a violation of good design - one that's socially expensive:
4. Get together with the other system's author (and possibly with other users) and agree on a common representation you'll both use.
It breaks abstraction layers. It breaks ownership separation. It breaks responsibility division.
It also eliminates hell of a lot of glue code.
Good standards are a wonderful thing, and IMHO we as an industry tend to invest far too little in establishing them now.
I believe programming languages can help a lot here, if you take the view that their standard library should be not just a ready-made toolbox but also a library of standards, in the sense of establishing concepts and (depending on the language) terminology, interfaces, types, symbolic constants and other common foundations for implementations in a specific field.
An example of not doing this is JavaScript, where the paucity of standard data structures and algorithms out of the box leads to huge trees of almost-trivial dependencies and to a culture where almost any data we need to move around just gets shoehorned into objects or transmitted as JSON.
A more positive example is Haskell, where the standard typeclasses capture several broadly useful programming patterns and given them a widely recognisable name and interface. Haskell libraries (standard or otherwise) can then provide algorithms with generic interfaces defined in those terms that are not tied to specific implementations or specific data structures. This results in an ecosystem of data structures and algorithms that are relatively easy to compose even when coming from different sources, which often allows Haskell programmers to express ideas very concisely. A related example is the use of generic programming within the standard library of C++, particularly with the arrival of concepts in C++20 that should serve a similar role to typeclasses in Haskell.
Both of those languages also learned some of these lessons the hard way. Each of them failed to establish a single, standard way to represent something as “simple” as a text string, with the result that many popular libraries did things their own way, and passing text around between things like databases, UIs and remote communications protocols can be a chore to this day.
I think it’s interesting that in some cases, the community around a language has evolved a kind of de facto standard interface for common tasks like accessing a SQL database, so that for example whether you’re talking to Postgres or Maria or SQLite or whatever, the corresponding drivers all provide very similar interfaces and are (mostly) interchangeable as far as calling code is concerned. I think it would be helpful if we did a lot more of that as an industry, within and perhaps even across languages, so we don’t keep reinventing the wheel every time we need to access a relational database or to draw a simple GUI or transmit data using typical Internet protocols or represent mathematical structures beyond the very basic ones.
float to int destroys information... unless the first program uses a float to store integers, so that that information is not destroyed. Similarly, the values could all be something-point-5 (1.5, 2.5, etx). That is, the literal type may be a float, but the application type is narrower than that. Like non-significant white-space, or a set whose elements are listed in an order, but that order is not significant.
A similar view is that, even if the application type for the first program really is a float, and the non-integer part is signicant to that program, that the type for this specific transfer is an integer. The specific way it's done depends on what's needed.
Is this "business logic"? Maybe. I would tend to say business logic requires some computation, other than a lossless transformation. Putting it into a different form, that is exactly equivalent, I would call glue.
I don't believe there is any authoritative definition; all we can do is note distinctions.
Perhaps I'm just playing semantics with a "transfer type"... but I think there is something important and useful in this distinction.
Oh, how the turntables turn. Ontologies are hard, and always fail in some way[1]. There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thing, but they are expensive and time-consuming and even when they work, only approximate.
Anyone has ever done a data migration project of any complexity knows that there must be provisions for dealing with data quality issues, especially if effort includes reconciling data from multiple sources.
1 https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Over... 2 https://doi.org/10.1145/3418896
The more code I have written, the more convinced I become that "general" solutions almost never pay for themselves when you compare the implementation and/or performance complexity required to make them work.
It's almost always better to "just write the damn code" and solve the exact problem you actually have.
Add in the fact that your business depends on doing it at scale, ingesting live data as it is generated, and developing an entity resolution system is almost a no-brainer.
They don't pay if you try to generilize before you have a problem.
If you have a frequent use case that follows the same patterns, then you probably can benefit from an abstraction layer.
But trying to build an abstraction layer before even having a use case then yes, it usually doesn't pay off.
Right there you've stated a key practice that may, by itself, be 80% of the solution. Of course there's no "right" set of of types, but just putting the work into thinking about the types and the essential constraints will drive towards exposing and eliminating assumptions that will come back to bite later.
Another huge chunk of problems can come from over-reliance on language primitive types. Is that numeric field really an integer, meaning that any signed value representable in 32, 64, or however many bits, including zero, is valid? Probably not. To give an example: A person's age is not an signed integer, it's a duration, we don't expect it to be negative. Maybe we should flag values above 90 or 100. If you're doing a one-off, that's the sort of thing you can code once and forget. If you dataset migration is your business, your Age type needs to be more robust, and perhaps there's even a need for the behavior of Age to vary with the domain.
The real problem is that all that glue code is proprietary, and companies cannot cooperate on a large scale to pool their resources and share their software.
This is the underlying problem: the perception that programs need to become more all encompassing. It's the ultimate feature creep.
Your software is just a tiny amount of code (relative to the whole system) making up the control plane between all the pieces.
I think the author probably meant that to personally write less code you need to leverage more external code (libraries and network API's).
Write less code -> Leverage other peoples code -> More integration with different systems
If we’re talking about communications between components within a software system then yes, for N components, there are roughly N² possible connections between them. But isn’t managing that complexity so we don’t connect everything to everything Software Design 101? If you design your system using structure like layers and pipelines then you’re effectively reducing that O(N²) connectedness to something more like O(N).
If we’re talking about having multiple choices for how to implement each component, such as when we have several libraries available plus the option of writing something from scratch, then given N choices for the first component and M choices for the second, there are NM possible connections we might have to implement. However, we probably only need one of those connections at any given time. If we need to modify or replace either implementation later, the most important thing for flexibility and future-proofing is that the data models and communications protocols are well specified, ideally following established standards that are likely to be followed by the other alternatives as well. Again, though, separating interface from implementation is Software Design 101.
So it seems to me that if either of those interpretations was intended, we’re talking about a problem that has largely been solved for a long time. Perhaps part of the reason it doesn’t always feel that way today is that there is so much emphasis on doing everything in “agile” fashion now. Sometimes that agility comes at the expense of establishing a well-considered software architecture and long-lived standards for interoperability. The “glue code” in software systems is where we pay any interest on those forms of technical debt, so while there will always be some need to marshal data around a large system with many components, there will be much, much more code needed if those forms of complexity aren’t well managed.
Do you mean other trends like increasing use of frameworks, scaffolding tools and “convention over configuration”? If so then I agree. One of the trade-offs being made there is immediate convenience against long-term flexibility. You get more functionality out-of-the-box with these kinds of tools, but the price is that they often don’t compose well.
> Glue is quadratic. If you have N features that interact with each other, you have O(N²) pieces of glue to get them to talk to each other.
And this one says:
> it’s “quadratic”: the glue code is proportional to the square of the number of things you need to glue
Even the most well-designed architecture will not let you only glue things pairwise, and you don’t need a very high N to run into challenges with N^2.
Very often, a combination of better software design and standardisation can reduce the actual complexity of an implementation below the apparent complexity. For example, suppose you have a system where N components can produce a certain type of data and another M components can then consume that data. This might appear to be a system with N×M connections between components. However, these components don’t necessarily need to be directly glued together in the implementation. If you define a standard, canonical representation for this type of data, you only need each supplier to convert into that format and each consumer to convert from it, and now you have an N+M glue problem instead of the apparently N×M problem you started with.
If there is a material change in requirements, you update your data interchange standard accordingly, and then you still only need to adapt N+M producers and consumers to work with the updated standard.
Either way, I still don’t see how we end up back with N×M. The consumers and suppliers are still communicating via the intermediary format and not all talking directly to each other.
That’s actually a very big claim, and I don’t know how we could possibly test it.
Counterexamples are easy to find, however. Consider that you’re reading this in a web browser that is communicating with a web server hosting HN. That communication, as indeed the article itself observes, happens via several standard protocols layered atop one another. I didn’t really understand the argument the article made at that point that this doesn’t solve the N² problem because it doesn’t take into account the application data at the top layer, as in practice almost everything sent over the Internet is also using standard application protocols like HTTP or SMTP as well. This has enabled literally billions of devices to communicate many types of data effectively on a global scale with a wide variety of software implementations on both ends of each connection, so IMHO it’s a pretty convincing example of the benefits of effective standardisation.
Of course it’s still true that some software systems have connections all over the place using ad-hoc data formats and protocols, but often that is simply bad design. Software design is a skill and like any other skill you don’t get good at it until you’ve worked on it for a while. My argument here, in a nutshell, is that if you learn good design skills and make the effort to establish good software architecture and useful standards for a project, very often the N² problem described in the article isn’t such a problem after all.
So maybe it's less about the # of systems here specifically, more about just the # of "things it does", where most of what it does is connect systems together.
- gRPC, protocol bufffers
- OpenAPI, Swagger, JSON Schema
- Apache Thrift
- Apache Avro
- Cap'n Proto
- IDL, Microsoft IDL, Web IDL
- etc