The city metaphor is evocative and new to me. I like it.
> Conversely, if we built our cities the way we build our software, you would need to enter the shop through the special garage, and exit through the roof to walk a wire to get to another custom made building from scrapped containers to do the checkout. And some of the windows are just painted on because they’re an MVP.
That does actually sound like real cities to me. You have to walk 1 km to get to a point 50 m away because they never put an underpass in when they built the railway line. That flagship building stands half finished for 4 years because the people involved got bought out by another company that never got around to finishing it.
And then you have that one section left over from the original release that all the engineers agree desperately needs to be refactored and upgraded, but due to cost and politics they never get to do it. And anyway you have a couple of power users who insist that the backwards and broken way that part is implemented is actually perfect and shout very loudly anytime you suggest changing it.
Thought the same, but I also think we already have software-as-a-city if you take operating systems, cloud platforms or for example message queue systems on top of which you build components (e.g. microservices at larger scale). Message queues are the streets and as such they are probably not the most efficient but at least they are flexible enough to allow you to experiment with the buildings: rebuild, renovate and repurpose them while leaving the streets more or less intact.
The problem with this industry seems to be that once something is solved, i.e. we already have reliable, battle-tested "streets", there is a big pressure to push everything further, build even more complex systems, faster. The pressure is rather natural: you will have a competitive advantage if you can push the limits and build something that can't be done based on the previous architectures, within a limited time frame.
For example, building desktop apps is a solved problem. These are you streets, these are your building blocks. But because it's a solved problem there's little money to be made out of it. The money lies somewhere on the edges of the map (e.g. SaaS) where there are still no roads and no general urbanization plan, and it's where the businesses tend to flock to. Hence the chaos, uncertainty and quality problems in most of innovative software.
Large software projects are almost as complex as cities and yet we have almost no one working on them. 4 developers can be enough to build a product used by millions. That’s kind of incredible IMO. 4 builders/architects/engineers don’t go nearly as far.
4 architects are also enough to design a house that can be built millions of times. If you include the builders into the city projects, it would be fair to include the tens of thousands of people working in data centers and network exchanges as well.
Also like a sibling comment states, 4 developers is not even close to a "large" software project.
I think Stewart Brand’s concept of Pace Layers fits well with the city metaphor of how software changes over time at different rates https://sketchplanations.com/pace-layers
> Conversely, if we built our cities the way we build our software, you would need to enter the shop through the special garage, and exit through the roof to walk a wire to get to another custom made building from scrapped containers to do the checkout.
I live 1.6km from my office. I drive 9.2km to the office. Cities aren't as straightforward as you might think.
I know this comes with experience but imagining how to scale or how to actually implement it is daunting. Obviously not everyone is going to have to need, but there's just so many tools and so many things to keep in mind to scale. What would you reckon is the best way to learn how to scale? Possibly one step at a time.
The way I learnt a lot myself was to run applications with little budget, e.g. running them on anaemic servers with few resources. When you have no option but to improve performance, because the alternative is too expensive, it gives you a good prod to design with it in mind.
Pick the most expensive servers/databases/storage and increase the number of users. Every 1-2 changes in orders of magnitude of the latter will require you to think thoroughly about how to best use resources and not explode your budget.
Also read from the highscalability blog, there's a lot of experience recorded there. See for example this transcript of an AWS presentation about a step-by-step guide to scaling: http://highscalability.com/blog/2016/1/11/a-beginners-guide-.... Of course it's done with AWS bricks but the ideas are universal
If you mean large software projects, yes, if you include slipped deadlines.
The real killers are:
- no owner, or fake (insincere) owners
- data cleanup
- schema re-mapping (on 10,000 tables)
- no usable requirements
- disinterest.
I've done several enterprise-scale migration projects, and have a 100% track record, because I was a "real owner." But when I look through old jiras, I always see failed versions of the identical project by employees who refused to step up and be an owner.
Favorite compliment/curse, "You must have really wanted it."
Most SAP projects fail, as well as most new billing projects, some on a colossally expensive scale - think $100 million and up.
"Brooks insists that there is no one silver bullet -- "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity."
The argument relies on the distinction between accidental complexity and essential complexity, similar to the way Amdahl's law relies on the distinction between "strictly serial" and "parallelizable"."
To add/elaborate systems get complex because the underlying reality is not simple, as much as we’d like it to be. No matter what paradigm (OO, strong/weak type etc.,) we choose or methodology (waterfall, agile etc) we adopt to build the underlying reality can’t be changed or mounded to fit the software, it is always the other way around.
Just a few days ago there was this thread about date calculation. One would think how complex can it be? But as one digs deeper they keep discovering layers and layers of special cases.
Take Uber as another example. Push a button get a car was their aim. We see people commenting here that they will be able to build it over a weekend. But the real world of cab hailing has thousands of unwritten, implicit rules that are in people’s head. All of them have to be codified and need to work seamlessly, and on top you add constraints of distributed systems and physics etc.
Or take tax laws, it’s a labyrinth of rules large enough that people have built DSLs.
So, any software that’s solving real world need will be complex. We just have to bite the bullet and deal with the reality.
I have yet to find an organization that has figured out how to effectively transfer learned lessons to other projects/employees without retaining those specific people for long periods.
Add in high levels of turnover and unless the specific person who learned the lesson is also on the project, there isn't a clear wrong way to prevent a wrong turn.
I am part of a new team at a startup and everyone on the new team is a relatively new hire. As far as I know, we haven't taken any meaningful learnings. Despite the company existing for years, we have done everything from scratch.
I help often customers understanding technical debt, and code quality. Most people focus of code metrics, but they tend to forget organizational metrics, and how that impact the software. There are several studies on this. I particularly like one from Microsoft [1].
One of the metrics that I use address your point:
Number of Ex-Engineers (NOEE): This is the total number of unique engineers who have touched a code and have left the company as of the release date of the software system
Implications: This measure deals with knowledge transfer. If
the employee(s) who worked on a piece of code leaves the
company then there is a likelihood that the new person taking
over might not be familiar with the design rationale, the
reasoning behind certain bug fixes, and information about
other stake holders in the code.
A large loss of team members affects the knowledge retention and thus quality.
> Number of Ex-Engineers (NOEE): This is the total number of unique engineers who have touched a code and have left the company as of the release date of the software system
I have been on a project where I arrived after the first guy's replacement was replaced. So three people had used my desk before for that project.
High levels of turnover typical suggest high dysfunction of some kind.
Also, the way to disseminate knowledge is by talking about problems. As in, if you manage to create culture where people do that, you will occasionally hear "team x tried that and had peoblems, lets ask them".
I agree with the post that software at scale is a big open question.
What gives me hope is that there were successful mega projects though not about software. Für example, I read about the Manhattan Project over Project Atlas to the Apollo Project. Those were all mega projects with high innovation and uncertainty like software development. I have no good theory what makes it successful though.
The tooling is simply not there, so every software project keeps pushing the boundary of what is possible in its own unique fragile way.
People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.
By tooling I mean programming languages, frameworks, libraries and operating systems. All of whuch have been designed for a single machine operation with random access memory model.
This no longer holds true. In order to scale software today you need multi machine operations and no such OS exists for it yet. Only immature attempts such as Kubernetes, whose ergonomics are far from the simolicity that you'd expect from a unix like system.
And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
I don't think we'll push the boundary further if we keep piling crap on top of existing crap. The foundations have to change and for that we'd better go back to the 70s when these ideas were being explored.
I watch Erlang / Elixir. But I really like strong static typing and type safety. So for now, I'm not learning anything beyond basic beam theory, lest I will have to unlearn it later.
Gleam lang at least is definitely something I watch closely and may (may) be the BEAMs breakout moment.
Building up the tooling is how we advanced over the last decades. It is a very slow process though because one tool has to become stable before tools on top of it can progress instead of adapting to changes.
Large software projects do not have decades. Only few years.
Distributed operating systems are nothing new though. Plan9 for example.
> Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
Why would you need a cluster of machines to work on mere gigabytes? You can get single-processor machines with terabytes of ram. Even laptops come with 64GB now.
I'd say the problem is way different and it's a people problem. A small percentage of people invest enougu resources to be able to have the big picture and write higher quality solutions. There is a huuge percentage of people with minimal education learning along the way and by learning along the way they reinvent things as a part of their learning process. This also incentovizes rising popularity of low quality but simpler tooling
The number of developers (and developer managers) who want to only complete tasks where someone (else) has defined the inputs and the outputs is extraordinary.
It is my firm belief that the vast majority of value in Software Development is in understanding the problem and the ways that Software can help. Crucially this includes knowing when it can't.
When asked what my job was once I said, "to think and write code, the hard part is doing it in that order."
This is a pet peeve of mine - I've worked with folks who were/are good developers but wanted to constrain their role to only development tasks. No effort to get exposure to build automation/deployment pipelines or understand how Docker/Kubernetes work at a high level, no desire to understand the bigger business domain or how our customers use the product. Pretty much just "give me a fully groomed ticket and leave me alone until I tell you it's done."
On the one hand, I get it - sometimes it actually is nice to have a ticket that you don't have to think too much about the bigger context for, and can just crank out code.
On the other hand, I don't think it's particularly good for your career security if you're limiting your own role to "minimally communicative code monkey" - if that's all you want to do, there's an awful lot of competition out there.
I've made an effort the past couple of years, across multiple jobs now, to get some real exposure to the ops aspects of my roles (most recently, prototyping and then helping implement CI/CD pipelines for Azure Durable Functions) as well as making an effort to understand the business domain, where the gaps in our product offerings are, and what the competitors in that sector look like. It's really helpful in terms of looking for a more efficient way to solve a business problem, not to mention being able to say things like "hey, the market we're in is moving pretty heavily to Kubernetes, so it's really important that we decide how we're going to support that."
I'm not saying you need to be (or can be) an expert in all of those things, but I think having the high level exposure is really important, at least when you get to a senior IC level. A minor bonus is it helps you develop a Spidey sense for when the business is pursuing strategies that don't seem likely to work, giving you a chance to either try to offer feedback/seek clarification on why, or to pull the plug and find a new gig if it seems bad enough.
It's very frustrating and makes it easy for management to outsource your role. Sometimes that's the right thing, but given how little most "ordinary" companies leverage technology and how utterly most non-tech understand what can (and can't) be done it's usually the wrong call.
> Noone will pay you for invoking existing solutions. They pay you to push the boundary.
I wouldn't put it like that. 90% of DevOps jobs is to invoke the existing solutions quickly. Sure, there will be a little bit of pushing the boundaries here and there was needed. But largely I'd describe those jobs as putting the Lego blocks together every day - and there's lots of money in it.
DevOps Engineer here, former Lead (moved from startup to Big Corp). That's literally my job day in day out. Put the Lego bricks together. Everybody around me tells me I'm doing a good job and I'm sitting here thinking, "but all I did was Google these things and insert some variables in some Helm charts?"
The other part of my job is more difficult, which is encouraging teams to de-silo and automate wherever possible but so many of the employees are threatened by this idea at Big Corp that I'm encountering a LOT of resistance.
Same here. It's easy to forget the difference between a master LEGO builder and the average person just piecing together bricks though. Knowing what to build and when/how is a lot of the job too. I try and remind myself that we're also often there to help other people build better things, perhaps analogous to automatically sorting LEGO bricks so other people building can move faster with fewer compromises.
This applies to most high-value knowledge work. Figuring out which problem to solve and how to define success is much more difficult than actually turning the crank to implement whatever solution.
Quickly and cheaply. People who only do development tend to look at every problem as some code they need to write.
We saved multiple customers a lot of time and money by just doing minor load balancer or web server configurations, rather than letting their developers doing a custom application to handle things like proxying or URL rewrites. Similarly we also frequently have to ask customers why they aren't offloading work to their databases and let it do sorting, selection, aggregation or search.
Having just a small amount of knowledge about what your existing tools can actually do for you, and knowing how to "Lego" them together solve a large number of everyday problems.
> People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions.
I am not exactly sure what you mean by this, but taken literally; most companies pay well for this and it's also the most common work programmers do all over the world. But maybe I misunderstand your meaning.
Heck, aren't Google devs complaining about "proto-to-proto" work ("protobuff to protobuff", not "prototype to prototype"), where they're basically just CRUDing together pre-existing services? Google pays a lot and has huge scale plus is considered quite innovative (I think they publish a lot of papers each year), yet most of their work is the typical enterprise middleware development job.
> Noone will pay you for invoking existing solutions. They pay you to push the boundary.
Most businesses pay developers to solve problems, not to push boundaries. They typically prefer developers using "boring" tried-and-true approaches.
In my experience it is developers who push for "boundary pushing" solutions because it is more exciting. And of course the vendors selling those solutions.
If it was true that the only way to get "paid" is to push boundaries, 99% of everyone working in IT would be out of a job. Kurt Vonnegut once said that "the problem with the world is that everyone wants to build, and noone wants to do maintenance". I'm reminded of that quote daily working in the software industry -- the people "pushing boundaries" are a tiny minority and if they were the only ones working our modern world would come crashing down in an afternoon.
Maintenance is seen by business as a cost, innovation is seen as potentially bringing more money.
The incentives are all wrong and that translate to engineers (and teams, and companies) chasing features and not fixing bugs.
A rewrite is often the only way of getting enough resources to fix up things.
They should value both but just don’t in my experience. Building a system can get you promoted. Performing maintenance to keep it running generally will not. Doesn’t matter if the maintenance project demonstrates a higher level of proficiency than the original development. Doesn’t matter that it’s extending the life of a system providing known value, as opposed to building something with theoretical value.
Of course they should, but it just doesn't seem to work out like that.
I was basically pushed out of my previous job because I was the bug-fix guy and liked it.
When I was the 'new stuff' guy, I got big raises every year. When I moved to being the bug-fix guy, and other people were the 'new stuff' people, suddenly I stopped getting good raises and barely met inflation. Even my coworkers were heaping praise on someone who I knew was over-complicating things because he was doing new stuff.
As I was leaving, a system that I had rewritten twice was up for another rewrite for more functionality. I would have loved to do it, and I knew I could. They gave it to someone else. I heard, 6 months after I left, that they failed and gave up on the rewrite. I am absolutely certain I could have done it again.
Getting a new job got me a 40% pay raise. It's not that I wasn't still worth a lot more every year. It's that they couldn't see it because it was all maintenance, and not shiny new stuff.
I still prefer bugfixes, and I still rock at them. But I end up doing mostly new stuff and don't say anything because I know how that goes.
Why? Maintenance can easily be quantified. Fixed x number of issues, implement y number of feature request/changes. Limited downtime to z amount of time by doing following actions...
...why in the world wouldn’t you talk about the ways you made an application better, more resilient, or heck, just able to continue operating??
No, I don’t want my local government official to build a new damn road. I would clap and love them for fixing the one I drive on everyday! Even better if they do it in a way that’s going to last for 100 years instead of 10, and that’s the way you brag about maintenance work.
agreed. ML [1] is short for meta langauge. Because it is not very C-like syntax by default, it can feel exotic and alien to most of today's unix graybeards, 90s enterprise, aughts startup folks, and even the JS crowd.
see also its 'newer incarnations' Scala, F#, Nemerle...(don't slay me)Rust(ish)
I think Nix will take a long while to increase adoption, because it is hard to learn. The language is new, the concept is new, the mental model is new. But you need to master all of those to use the tool effectively.
Same goes for the other items in your list. Git had enough time and force behind it, and I believe the other tools will succeed as well. But it will take time.
For your typical scaling org, I think data layers are often the main issue. Moving from a single postgres/mysql primary to something that isn't that represents the biggest hurdle.
Some companies are "lucky" and have either natural sharding keys for their primary business line, are an easily cacheable product, or can just scale reads on replicas. Others aren't, and that's where things get complicated.
Tbh, That's why for our new projects we've completely ignored relational databases. They're a pain in the ass to manage and scale poorly.
DynamoDB, on the other hand, trivially scales to thousands (and more!) of TPS and doesn't come with footguns. If it works, then it'll continue to work forever.
This is funny to me since modern relational databases can get thousands and more TPS in a single node. My dev machine reports 16k TPS on a table with 100M rows with 100 clients.
> pgbench -c 100 -s 100 -T 20 -n -U postgres
> number of transactions actually processed: 321524
> latency average = 6.253 ms
> tps = 15991.775957
Yep. And 98% of software written today will never need to scale beyond 10k TPS at the database layer. Most software is small. And of the software that does need to go faster than that, most of the time you can get away with read replicas. Or there are obvious sharding keys.
Even when thats not the case, it usually ends up being a minority of the tables and collections that need additional speed.
If you don't believe me, look through the HN hiring thread sometime and notice how few product names you recognise.
Most products will never need to scale like GMail or Facebook.
ML was first developed in 1973. Ocaml in 1996 and SML in 1997. Great tools which haven't been popular in 20-40 years probably have something beyond fear of early adoption inhibiting them.
I'd say herd behavior and network effects are the main issues.
Do a quick search on YouTube on "what programming language to learn". You'll find video after video using popularity as a primary or even the primary factor to base the decision on.
Non-technical management tends to do the same, based out of the belief that languages are all pretty much equivalent and a desire to hire easily swappable "resources".
Yaron Minsky joined Jane Street in 2003. The success of Jane Street and the use of OCaml there is well known in finance circles. Finance is full of people who don’t exhibit herd behavior - many firms have a culture of studying fundamental causes and taking contrarian positions. That begs the question - why did OCaml not get widely adopted in finance?
Jet was purchased by Walmart in 2016. Mark Lore from Jet was appointed as CEO of Walmart to turn the company around, particular its technology operations (to compete effectively with Amazon). Jet’s tech was mostly developed in F#. Yet, Lore did not push for its adoption widely within Walmart.
IMO explaining away the failure of ML languages to gain market share over multiple decades as “herd behavior and network effects” is lazy.
Agreed. The stuff "sitting on the shelf", as your parent comment said, have problems too (eg tooling). They might solve some problems, but are far from the silver bullets we are looking for.
>People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.
You'd be surprised. Most software production re-builds and re-invokes existing solutions that just aren't systematized yet. It certainly doesn't push any boundaries...
> By tooling I mean programming languages, frameworks, libraries and operating systems. All of whuch have been designed for a single machine operation with random access memory model.
You should learn about Elixir (or anything on Erlang VM).
Agreed completely. Today there's such a large disconnect between how you think about software (APIs, services, systems, data flows, replication, data storage, access patterns, and so on) and how you actually develop software, with plain text files on disk that don't at all resemble our mental model.
I've been working on a new type of framework that is about making how we develop software match this mental model. Where the higher-level primitives we think in terms of are first-class citizens at the language level.
I think only then is it possible to reason about, and scale, software in a much more productive and effective way.
> Today there's such a large disconnect between how you think about software (APIs, services, systems, data flows, replication, data storage, access patterns, and so on) and how you actually develop software, with plain text files on disk
I view this a little differently. People have tried again and again to move the abstraction layer for coding above text files. This was happening back in the 90's (in the guise of so-called "4GLs" [0]) and still today (now rebranded as "no-code" [1]). I myself spent a good deal of effort trying to code "diagramatically" through top down UML (with an amazing product for it's time, Together). So the ambition to shift programming up the abstraction food chain has been tried continuously for 30 years and continued to fail every time. Eventually I changed my view and decided that there were fundamental reasons why higher level abstractions don't work - software is just too complex, the abstractions are too leaky. The details matter, they are critically important. We are in a long arc of figuring out the right abstractions that may take a century or more and in the meantime, we simply have to have the flexibility of text based formats to let us express the complexity we need to manage in the meantime.
> People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.
People are paid to work on products which provide value to the business. There is always a Minimum Viable Product, MVP. Meet that. Exceeding that is the boundary that needs to be pushed, not grabbing the latest, possibly untested, tools off the shelf.
> And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
This is why constantly re-engineering (for the purpose of engineering) is not the most useful method. I used to work in FoxPro with databases that were GBs of data. If today those GBs are difficult to handle, which they aren't, then there is a problem with how the stack was put together. GBs are trivial.
To put is cynically, everyone knows that yesterday's tools don't work. So it's better to try today's tools, which you don't yet know don't work. This allows one poorly thought out idea to replace another - especially when selling new stuff is profitable.
That said, it's implausible that the main problem is scaling since mainframes scaled considerable data quite a while back.
Very often, even tough I only ever write very short programs (100 to 1000 lines) in Python, C++ (and learning Rust), I look at my finished program and feel like there is a million things to improve and a billion other ways to do it. Each variable could have a better name, each abstraction could be abstracted more, the whole thing could be rewritten in a different style or language.
Whereas if I design something in CAD, I send it to the printer, try it, improve it once, print again and never ever again think of it as long as it does it's job.
When you start drawing diagrams of non-trivial software, for instance Blender (an open source tool for building 3D models and animations), you start to understand how complex and complicated software is.
I'm working on driver assistance stuff. In terms of raw functionality the software is not that complex. The complexity comes from other aspects: Safety-critical, hardware-software-codesign, commercial, resource constrained. Then, of course, organizational disfunction creates accidental complexity as it does everywhere. All of that together, and a software developer barely achieves a thousand lines of code per year.
Regarding comments about manufacturing plants and the LCH, sure both of these examples aren’t software directly (but they also wouldn’t be possible without software), but they are also always under constant fixes, feature upgrades and optimisations.
I think under market conditions you would be continuously tweaking the model, and manufacturing processes, based on feedback eg wrt robustness, comfort etc and also taking into account changing usage parameters/expectations, changes in material supply costs, regulatory requirements etc
Of course these concerns matter more if you’re making a lot of these “widgets” but even in the case of a one off you’re going to have servicing costs and the occasional redesign/replacement/upgrade ...
It’s not that far out of line with software - widely used products go through tight highly iterative development cycles whereas one off solutions tend to be just “good enough” with bug fixing and the occasional feature request.
IMO, technical leadership isn’t allowed to build at scale or the leadership doesn’t exist.
The race to the bottom incentivises profit over principle. Boards don’t care if something is built to scale if the competitor gets to market with a cheaper solution. Scaling is a day two problem.
I’m bearish on my future employment as someone who questions the motivations for profit driven development.
I naively thought the market would reward the best product, instead I see the cheapest product being rewarded. What happened to innovation and doing what’s best for society?
I think it's fairly common for even established companies to prioritize short-term costs over long-term costs, even when the long-term costs of NOT investing in good tooling are astronomical. So absolute cost isn't really the deciding factor.
> reward the best product, instead I see the cheapest product being rewarded.
That's because you are looking at best from one angle and the market from another.
To the "market" the best may well be the cheapest thing that mostly does it's job/most of the time.
When I was young I used to think users cared about bugs and they do but they only care when the bugs are sufficient in number and degree to cross a threshold where they outweigh the utility of the software, That sort of 1 in 100 times it loses an hours work, well if the 99 times it saves me an hours work, shrug.
What we (as techies and I'm generalising wildly here) want to build is heirloom quality carpentry, what we get to actually build is franken-furniture from ikea packs that we hope has some some instructions and won't fall apart next week.
Yesterday I watched a video about how Netflix scaled its API with GraphQL Federation.[0] The video contains some neat visualizations that helps you see how complex data accees problems at scale can get. And this is just the services level they talk about.
No mentioning of the underlying infra with all its complexities needed to acheive the goals of flexibility, reliability, speed and cost cutting.
You don't have to be of Netflix size - when you start getting tens of thousands of users, complexity hits you real fast.
Torvalds once said [1] that the only scalable software development methodology is open source, and I tend to agree for two reasons:
1. The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. In Coasian terms, the transaction costs around development are minimized.
2. It enforces hard interfaces and it clearly separates development from operations. Lack of discipline around these issues is a source of much accidental complexity.
[1] I can’t seem to find the quote, I read it in an interview a few years ago and stuck with me since.
Yes, the linux kernel is a big codebase but so are a lot of others [1]. And you will find lots of private companies not mentioned that have long-standing projects with >10 mil LOC.
If you took the 100 biggest code bases in the world. I would be surprised if more than 10% of them were open source.
I know a medium sized French-American multinational. You've probably not even heard of it. A decade ago they had multiple products that had several million lines of code. Their entire platforms probably had 100 million. I can't even imagine how many contributors they had, the oldest project was started in the 80's and it probably had 1000 contributors over time, at least.
And again, that's for a middle size company you haven't even heard of. FAANG, Microsoft, Oracle, IBM for sure will have stuff dwarfing that.
Is it actually less appealing? My understanding is that DBAs consider Oracle very good, just very (very) expensive. This also lines up with my experience tuning queries against Oracle vs Postgres backends. The folk wisdom seems, to me, to be Postgres is 99% as good in the common cases and 90-95% as good in the difficult cases.
Oracle DBAs consider Oracle clearly superior. SQL Server DBAs that know all the ways to optimize SQL Servers consider SQL Server clearly superior. I don't know of any PostgreSQL DBA, just generalists that work with Postgres, so I don't know about them.
The truth is that Oracle is ridden with coherence bugs, and have a much worse performance picture out of the box than Postgres. But while improving the Postgres performance requires deep digging into the DBMS itself, Oracle has a lot you can optimize just on the SQL interface. But there are plenty of cases where Oracle just can not become as fast as out of the box Postgres, and many where it could in theory, but a bug prevents you from doing things right.
Overall, three is no definitive ordering on the speed of the "good 3" DBMS. It always depends on what you are doing.
SAP told me their framework is a billion LoC of Java and 30k tables in the database. That was years ago so it has probably grown further. It is only the framework, so no useful application yet.
How many of those 20k contributors worked on drivers, and how many - on the actual core (~150kloc)? Every driver is like a separate subproject, having 20k people work on hundreds/thousands of drivers (unrelated to each other) wouldn't sound as impressive.
That's the scary thing. How many real, core contributors does even something like the Linux kernel have? People who have actually written more than a couple patches that landed and stayed in the kernel. I'd be astounded if it's much more than a hundred.
Most open-source projects have one to maybe five real contributors, outside the drive-by pull requests to fix some bug.
I guess (and I have no way to verify this, it's really just a guess) that the number of developers that worked on the Windows code base in its 35 year history is in the same order, if not higher.
Bonus chatter: In a discussion of Windows source control, one person argued that git works great on large projects and gave an example of a large git repo: “For example, the linux kernel repository at roughly eight years old is 800–900MB in size, has about 45,000 files, and is considered to have heavy churn, with 400,000 commits.”
I found that adorable. You have 45,000 files. Yeah, call me when your repo starts to get big. The Windows repo has over three million files.
Four hundred thousand commits over eight years averages to around [130] commits per day. This is heavy churn? That’s so cute.
You know what we call a day with [130] commits? “Catastrophic network outage.”
To get closer to an apples-to-apples comparison, it'd be necessary to know whether the commit counts in each case include all development branches for all development groups. By design, git-based development can be highly distributed.
Also, even if we normalized both cases for 'code files only' and/or 'kernel code only', there could still be architectural, code style, and development process differences that lead to different metrics for each project.
> the only scalable software development methodology is open source
“open source” isn’t a development methodology, or even a distinct set of methodologies.
> The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale.
Plenty of open source projects don’t have that, nor is there anything restricting those things to open source projects.
It is true that some open source projects, because they see the value in new developers jumping in quickly, prioritize having structure, documentation, and tooling that supports that. It’s also true that some proprietary software projects do, too, because the project owner sees value in that.
> There are books written about open source development
There are books written about different authors idealizations of how to do open source development. The fact that many people have written about different methodologies for approaching a particular challenge doesn’t make the challenge a methodology.
And, yes, I am aware that the very approach known as “the bazaar” (from the essay “The Cathedral and the Bazaar”) is sometimes referred to erroneously as “open source development”, which is a particularly glaring error since all of the examples of both “cathedral” and “bazaar” development were open source projects.
To go further I think it's not exactly open source but remote-first that made software development scalable.
If you grossly simplify it to its innermost core, making development scalable means that if you have 10 times more people you can have 10 times more features/bugfixes/speed improvements/... The only way to do that is to make sure that an additional developer doesn't depend on other developers to work, and that can only happen if everything is properly documented, the build instructions are up-to-date, the processes are clear, basically anyone can start from scratch and get up to speed without the help of anyone else.
That kind of organization traditionally doesn't happen in on-site companies, where newcomers are followed by senior people, they have to follow some introduction course to familiarize themselves with the processes, they have to ask many questions every day, they need to be synchronized with other persons which brings some inefficiency beacuse everyone works at their own pace, etc... This all disappears when everything is properly documentend and every contributor can work in the middle of the night if they wish. I think the Gitlab Handbook goes over this quite well and describes a framework to implement that kind of organization but the rules are retrospectively obvious for people already used to open-source (https://about.gitlab.com/company/culture/all-remote/guide/):
- write down everything
- discussion should happen asynchronously. Any synchronous discussion (by text or call) should be only very small points. Whatever the type, write down the conclusions of that discussions
- Everything is public (to the organization), including decisions taken, issues, processes
Yeah, remote-first is important, but it's not only it.
Another very relevant factor is that people can just clone stuff, create new projects, and everything moves independently. So the open source development model has a pile of solutions for dependency management that team based development doesn't adopt.
But the one thing I don't get is why team based development doesn't adopt those solution. They are not expensive. Yet, even when I was able to dictate into everybody's requirements, I wasn't able to force teams into adopting them. Instead they insisted on synchronizing themselves by much more expensive and less reliable means. My guess is that most developers never dug any deep into open source and have no idea how it's done.
But cities ain't pretty. Dig the ground and nothing looks like what's on the map. To create a new building, you need to cut a lot of plumbing and re-route it somehow.
Stuff gets old and breaks all the time, and maintenance teams are on the ground 24/7 to fix it. NYT subway is the mainframe of the city. Look at the central steam heating that's still pumping water into high rises.
Sure, you can sell a shop, and people will be able to repurpose that space to suit their needs very efficiently. But isn't that what Docker does? Provide a cookie cutter space to run an app?
But in cities, there are places where trucks can't go, where you can't park, where unplanned work happens, where the sewer floods. That's when the city 'bugs', and people need to manually intervene to fix it...
Trying to find an ideal way of running software at scale is just as utopian as building the perfect city using 'super structures'. It's a dream of the 60s that's never going to happen.
I mean in the sense of what perceived chaos or deterioration is “allowed” in a city or it’s infrastructure. E.g whether a plot can be left unbuilt or half torn down for example, and the standards which are expected of public infrastructure like roads or train stations.
Cities like Tokyo might be more comparabe, and much nicer than NYC in many aspects, especially the public transportation system. For a major city NY looks pretty shabby and not that nice in average.
Maybe it's time to finally differentiate the tools for software as in functions, algorithms, classes, libraries, etc. and software as in e-shops, databases, sso services, streaming services, etc.
We have good science and steady (albeit slow) progress in the former while it feels like the latter is more or less subject to stagnation. For instance, when I need to setup a DB cluster, why is there no tool that takes my requirements and generates deployment scripts, monitoring, migration tools, etc.?
>How is it that we’ve found ways to organize the work around so many other creative disciplines but writing software is still hard?
I think OP heavily overestimates the organisational praxis in other disciplines. Nearly every creative discipline I was ever in was largely purely ad-hoc with very little explicitly stated organisational approaches to the craft. Academia, movie and music production, and creative writing for instance have much less readily-available principles than software.
Software is probably the most thought about creative industry I can think off in modern history.
I think the software industry could learn a lot from the collaborative attitudes and approaches employed in the more typically creative industries. Those industries you mentioned may not have as many formalised principles, yet collaboration is so imbued into the process that it naturally lends itself to large scale creations.
> There is something peculiar about software that makes it different from other crafts.
What distinguishes software from other techne is a lack of physics. There is no 'solid ground of reality'. All other forms of making involve discovering and then applying the governing laws.
Hardware architectures, operating system, and programming languages, to an extent, do furnish a phenomenal context and where these characteristics are stable and well established (e.g. Memory Models), science appears [1].
But clearly, this ground of reality is indeed 'soft' and the practitioners usually re-invent not just the wheel, but the ground that it rolls on as well.
The dilemma -- which nearly means facing a choice of a multiplicity of lemmas -- that confronts the designers of hardware, OS, and languages is precisely that tension between generality (and corresponding weak "world" semantics) and specificity (with robust but constrained semantics).
Conversely, writing software for a small scale is absurdly inefficient.
I consult for government departments that often have legally mandated requirements for making something available online. That could be a form submission process, some geospatial information presented on a map, or whatever.
The problem is that when you might have 500-10K user transactions annually, it becomes crazy expensive to write bespoke software, even with the most agile process and the lowest overhead tooling.
Take cloud deployments, for instance. Sure, you can just "click through some wizard" pressing next-next-next-finish and be up and running, but the security team won't allow your infrequently-maintained web server on the Internet without a web application firewall. Setting one of those up is days of tweaking to avoid false positives.
Need to send mail? Azure bans outbound port 25 connections, you have to use Sendgrid, or something like it. Time to read up on yet another unique and special API!
Collecting fees and penalties or making payments to citizens? Woah there... there's a massive API surface you have to learn. Security on top of security is needed. Whitelisted IP addresses. Client certificates. That have to be rotated, manually!
You'll forget some essential maintenance, of course, and then you'll have to set up triggers and alarms so you don't get burned the second time. Which entails mailing lists that change dynamically because the team of contractors has a turnover rate faster than the typical certificate expiry time. Send too many alarms and all recipients will configure an Outlook rule to ignore them. Not enough alarms and you'll miss issues. Just setting this up semi-reliably is an exercise in itself.
Really basic stuff becomes difficult, when you realise that 99% of the alerting and monitoring features in Azure and AWS are designed for systems at their scale. It's all about analysing beautifully smooth curves of graphs aggregating millions of points of data, where deviations are glaringly obvious. These tools are utterly useless when you get one real transaction per day, swamped by a thousand bots. The load balancer health check is 99.99% of the traffic for some of these sites!
Then there's the human element:
Have you tried justifying the time to set alarms for a system where a week-long outage might only affect a dozen customers?
How about the budget to upgrade to a newer operating system for something that is not technically broken -- merely hard to support now?
Or have you tried doing any sort of maintenance on a system that was built by contractors hired for a fixed term, all of whom are now gone?
Meanwhile departments are renamed every couple of years to suit the whims of the latest batch of politicians, so everything has to rebranded. Even tiny little sites use by practically nobody.
I've seen sites up for 10 years, where I estimated that they cost $2,000 per citizen that actually used the site! Madness.
It is easier than ever to deploy software on a small scale. It sounds like your security team is failing to provide a suitable platform and you chose the wrong cloud provider for sending email.
>Everyone is still terrible at creating software at scale
Terrible is a relative term. Terrible compared to what?
Who said/showed (much less proved) that there's a better way and we just don't follow it to achieve it some optimum?
There's also a semantic confusion here. Compared to e.g. the car industry we're infinitely better at "creating software at scale". We can create a billion copies of a software and distribute it our the world, with marginal cost close to zero.
But the author doesn't mean "creating software at scale" like when they say "car production at scale". They mean production of "large software".
Well, let's see the car industry make a car with the scope, flux requirements, shifting environments, etc that large scale software has...
A lot of guessing and hunching and try to find one answer to a couple of problems where the question isn’t clear. So I don‘t know what point this article wants to make.
Maybe this happens because the inefficiency is resilience/anti-fragility?
The implication is that highly efficient businesses exist for a short time but they get wiped out quickly when their environment changes.
Sometimes a CEO thinks up a radical crazy idea. While it distributes through middle management it gets twisted. Once it reaches the front line workers nothing really changes. One might consider this inefficient change management but maybe the company protected itself from a stupid CEO decision.
There is at least one massively scaled system that relies heavily on software, that doesn't often get a mention because it just works.
When was the last time you picked up a land-line phone and it didn't work because of a fault on the system?
Reading about Neuralink yesterday makes me wonder if in 50 years it will be possible to connect a bunch of minds together to achieve a task none could do on their own. Could be programming or some other large scale task with a lot of interdependence.
The whole point of the article is that communication on a very large scale is very hard. There's a point that adding more people makes things worse. What I mean is, one day, maybe more people will be able to collaborate by becoming one big brain.
393 comments
[ 3.0 ms ] story [ 298 ms ] thread> Conversely, if we built our cities the way we build our software, you would need to enter the shop through the special garage, and exit through the roof to walk a wire to get to another custom made building from scrapped containers to do the checkout. And some of the windows are just painted on because they’re an MVP.
And then you have that one section left over from the original release that all the engineers agree desperately needs to be refactored and upgraded, but due to cost and politics they never get to do it. And anyway you have a couple of power users who insist that the backwards and broken way that part is implemented is actually perfect and shout very loudly anytime you suggest changing it.
The problem with this industry seems to be that once something is solved, i.e. we already have reliable, battle-tested "streets", there is a big pressure to push everything further, build even more complex systems, faster. The pressure is rather natural: you will have a competitive advantage if you can push the limits and build something that can't be done based on the previous architectures, within a limited time frame.
For example, building desktop apps is a solved problem. These are you streets, these are your building blocks. But because it's a solved problem there's little money to be made out of it. The money lies somewhere on the edges of the map (e.g. SaaS) where there are still no roads and no general urbanization plan, and it's where the businesses tend to flock to. Hence the chaos, uncertainty and quality problems in most of innovative software.
Also like a sibling comment states, 4 developers is not even close to a "large" software project.
I live 1.6km from my office. I drive 9.2km to the office. Cities aren't as straightforward as you might think.
Also read from the highscalability blog, there's a lot of experience recorded there. See for example this transcript of an AWS presentation about a step-by-step guide to scaling: http://highscalability.com/blog/2016/1/11/a-beginners-guide-.... Of course it's done with AWS bricks but the ideas are universal
I found so many parallels in what I've done while scaling up. Only thing I did differently was starting with MongoDB and changing to SQL later; huh.
The real killers are:
- no owner, or fake (insincere) owners
- data cleanup
- schema re-mapping (on 10,000 tables)
- no usable requirements
- disinterest.
I've done several enterprise-scale migration projects, and have a 100% track record, because I was a "real owner." But when I look through old jiras, I always see failed versions of the identical project by employees who refused to step up and be an owner.
Favorite compliment/curse, "You must have really wanted it."
Most SAP projects fail, as well as most new billing projects, some on a colossally expensive scale - think $100 million and up.
"Brooks insists that there is no one silver bullet -- "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity."
The argument relies on the distinction between accidental complexity and essential complexity, similar to the way Amdahl's law relies on the distinction between "strictly serial" and "parallelizable"."
https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month
Just a few days ago there was this thread about date calculation. One would think how complex can it be? But as one digs deeper they keep discovering layers and layers of special cases.
Take Uber as another example. Push a button get a car was their aim. We see people commenting here that they will be able to build it over a weekend. But the real world of cab hailing has thousands of unwritten, implicit rules that are in people’s head. All of them have to be codified and need to work seamlessly, and on top you add constraints of distributed systems and physics etc.
Or take tax laws, it’s a labyrinth of rules large enough that people have built DSLs.
So, any software that’s solving real world need will be complex. We just have to bite the bullet and deal with the reality.
This observation seems both true and important.
Add in high levels of turnover and unless the specific person who learned the lesson is also on the project, there isn't a clear wrong way to prevent a wrong turn.
I am part of a new team at a startup and everyone on the new team is a relatively new hire. As far as I know, we haven't taken any meaningful learnings. Despite the company existing for years, we have done everything from scratch.
One of the metrics that I use address your point:
Number of Ex-Engineers (NOEE): This is the total number of unique engineers who have touched a code and have left the company as of the release date of the software system
Implications: This measure deals with knowledge transfer. If the employee(s) who worked on a piece of code leaves the company then there is a likelihood that the new person taking over might not be familiar with the design rationale, the reasoning behind certain bug fixes, and information about other stake holders in the code.
A large loss of team members affects the knowledge retention and thus quality.
[1] https://www.microsoft.com/en-us/research/wp-content/uploads/...
I have been on a project where I arrived after the first guy's replacement was replaced. So three people had used my desk before for that project.
So much knowledge was lost.
Also, the way to disseminate knowledge is by talking about problems. As in, if you manage to create culture where people do that, you will occasionally hear "team x tried that and had peoblems, lets ask them".
What gives me hope is that there were successful mega projects though not about software. Für example, I read about the Manhattan Project over Project Atlas to the Apollo Project. Those were all mega projects with high innovation and uncertainty like software development. I have no good theory what makes it successful though.
People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.
By tooling I mean programming languages, frameworks, libraries and operating systems. All of whuch have been designed for a single machine operation with random access memory model.
This no longer holds true. In order to scale software today you need multi machine operations and no such OS exists for it yet. Only immature attempts such as Kubernetes, whose ergonomics are far from the simolicity that you'd expect from a unix like system.
And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
I don't think we'll push the boundary further if we keep piling crap on top of existing crap. The foundations have to change and for that we'd better go back to the 70s when these ideas were being explored.
Gleam lang at least is definitely something I watch closely and may (may) be the BEAMs breakout moment.
Large software projects do not have decades. Only few years.
Distributed operating systems are nothing new though. Plan9 for example.
Why would you need a cluster of machines to work on mere gigabytes? You can get single-processor machines with terabytes of ram. Even laptops come with 64GB now.
It is my firm belief that the vast majority of value in Software Development is in understanding the problem and the ways that Software can help. Crucially this includes knowing when it can't.
When asked what my job was once I said, "to think and write code, the hard part is doing it in that order."
On the one hand, I get it - sometimes it actually is nice to have a ticket that you don't have to think too much about the bigger context for, and can just crank out code.
On the other hand, I don't think it's particularly good for your career security if you're limiting your own role to "minimally communicative code monkey" - if that's all you want to do, there's an awful lot of competition out there.
I've made an effort the past couple of years, across multiple jobs now, to get some real exposure to the ops aspects of my roles (most recently, prototyping and then helping implement CI/CD pipelines for Azure Durable Functions) as well as making an effort to understand the business domain, where the gaps in our product offerings are, and what the competitors in that sector look like. It's really helpful in terms of looking for a more efficient way to solve a business problem, not to mention being able to say things like "hey, the market we're in is moving pretty heavily to Kubernetes, so it's really important that we decide how we're going to support that."
I'm not saying you need to be (or can be) an expert in all of those things, but I think having the high level exposure is really important, at least when you get to a senior IC level. A minor bonus is it helps you develop a Spidey sense for when the business is pursuing strategies that don't seem likely to work, giving you a chance to either try to offer feedback/seek clarification on why, or to pull the plug and find a new gig if it seems bad enough.
I wouldn't put it like that. 90% of DevOps jobs is to invoke the existing solutions quickly. Sure, there will be a little bit of pushing the boundaries here and there was needed. But largely I'd describe those jobs as putting the Lego blocks together every day - and there's lots of money in it.
The other part of my job is more difficult, which is encouraging teams to de-silo and automate wherever possible but so many of the employees are threatened by this idea at Big Corp that I'm encountering a LOT of resistance.
We saved multiple customers a lot of time and money by just doing minor load balancer or web server configurations, rather than letting their developers doing a custom application to handle things like proxying or URL rewrites. Similarly we also frequently have to ask customers why they aren't offloading work to their databases and let it do sorting, selection, aggregation or search.
Having just a small amount of knowledge about what your existing tools can actually do for you, and knowing how to "Lego" them together solve a large number of everyday problems.
I am not exactly sure what you mean by this, but taken literally; most companies pay well for this and it's also the most common work programmers do all over the world. But maybe I misunderstand your meaning.
Most businesses pay developers to solve problems, not to push boundaries. They typically prefer developers using "boring" tried-and-true approaches.
In my experience it is developers who push for "boundary pushing" solutions because it is more exciting. And of course the vendors selling those solutions.
Work should be balanced between creating new code and maintaining existing code. And managers should value both.
Maintenance is seen by business as a cost, innovation is seen as potentially bringing more money. The incentives are all wrong and that translate to engineers (and teams, and companies) chasing features and not fixing bugs.
A rewrite is often the only way of getting enough resources to fix up things.
I was basically pushed out of my previous job because I was the bug-fix guy and liked it.
When I was the 'new stuff' guy, I got big raises every year. When I moved to being the bug-fix guy, and other people were the 'new stuff' people, suddenly I stopped getting good raises and barely met inflation. Even my coworkers were heaping praise on someone who I knew was over-complicating things because he was doing new stuff.
As I was leaving, a system that I had rewritten twice was up for another rewrite for more functionality. I would have loved to do it, and I knew I could. They gave it to someone else. I heard, 6 months after I left, that they failed and gave up on the rewrite. I am absolutely certain I could have done it again.
Getting a new job got me a 40% pay raise. It's not that I wasn't still worth a lot more every year. It's that they couldn't see it because it was all maintenance, and not shiny new stuff.
I still prefer bugfixes, and I still rock at them. But I end up doing mostly new stuff and don't say anything because I know how that goes.
No, I don’t want my local government official to build a new damn road. I would clap and love them for fixing the one I drive on everyday! Even better if they do it in a way that’s going to last for 100 years instead of 10, and that’s the way you brag about maintenance work.
Meanwhile, there are things sitting on the shelf that solve these problems in a principled way and make things simpler:
- ML languages
- Nix
- Bazel
- Erlang
- Rust
Some tools that are great and have large adoption:
- Git
- Terraform
What are those?
see also its 'newer incarnations' Scala, F#, Nemerle...(don't slay me)Rust(ish)
[1] https://en.m.wikipedia.org/wiki/ML_(programming_language)
[1]http://erights.org/elang/index.html
Same goes for the other items in your list. Git had enough time and force behind it, and I believe the other tools will succeed as well. But it will take time.
Some companies are "lucky" and have either natural sharding keys for their primary business line, are an easily cacheable product, or can just scale reads on replicas. Others aren't, and that's where things get complicated.
DynamoDB, on the other hand, trivially scales to thousands (and more!) of TPS and doesn't come with footguns. If it works, then it'll continue to work forever.
Banking app is a classic example.
> pgbench -c 100 -s 100 -T 20 -n -U postgres > number of transactions actually processed: 321524 > latency average = 6.253 ms > tps = 15991.775957
Even when thats not the case, it usually ends up being a minority of the tables and collections that need additional speed.
If you don't believe me, look through the HN hiring thread sometime and notice how few product names you recognise.
Most products will never need to scale like GMail or Facebook.
ML was first developed in 1973. Ocaml in 1996 and SML in 1997. Great tools which haven't been popular in 20-40 years probably have something beyond fear of early adoption inhibiting them.
Do a quick search on YouTube on "what programming language to learn". You'll find video after video using popularity as a primary or even the primary factor to base the decision on.
Non-technical management tends to do the same, based out of the belief that languages are all pretty much equivalent and a desire to hire easily swappable "resources".
Jet was purchased by Walmart in 2016. Mark Lore from Jet was appointed as CEO of Walmart to turn the company around, particular its technology operations (to compete effectively with Amazon). Jet’s tech was mostly developed in F#. Yet, Lore did not push for its adoption widely within Walmart.
IMO explaining away the failure of ML languages to gain market share over multiple decades as “herd behavior and network effects” is lazy.
I think it has been pretty successful. The bigger success story in finance is F# though, which takes a bunch of ideas from OCaml.
You'd be surprised. Most software production re-builds and re-invokes existing solutions that just aren't systematized yet. It certainly doesn't push any boundaries...
You should learn about Elixir (or anything on Erlang VM).
I've been working on a new type of framework that is about making how we develop software match this mental model. Where the higher-level primitives we think in terms of are first-class citizens at the language level.
I think only then is it possible to reason about, and scale, software in a much more productive and effective way.
I view this a little differently. People have tried again and again to move the abstraction layer for coding above text files. This was happening back in the 90's (in the guise of so-called "4GLs" [0]) and still today (now rebranded as "no-code" [1]). I myself spent a good deal of effort trying to code "diagramatically" through top down UML (with an amazing product for it's time, Together). So the ambition to shift programming up the abstraction food chain has been tried continuously for 30 years and continued to fail every time. Eventually I changed my view and decided that there were fundamental reasons why higher level abstractions don't work - software is just too complex, the abstractions are too leaky. The details matter, they are critically important. We are in a long arc of figuring out the right abstractions that may take a century or more and in the meantime, we simply have to have the flexibility of text based formats to let us express the complexity we need to manage in the meantime.
[0] https://en.wikipedia.org/wiki/Fourth-generation_programming_...
[1] https://techcrunch.com/2020/10/26/the-no-code-generation-is-...
People are paid to work on products which provide value to the business. There is always a Minimum Viable Product, MVP. Meet that. Exceeding that is the boundary that needs to be pushed, not grabbing the latest, possibly untested, tools off the shelf.
> And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
This is why constantly re-engineering (for the purpose of engineering) is not the most useful method. I used to work in FoxPro with databases that were GBs of data. If today those GBs are difficult to handle, which they aren't, then there is a problem with how the stack was put together. GBs are trivial.
That said, it's implausible that the main problem is scaling since mainframes scaled considerable data quite a while back.
Whereas if I design something in CAD, I send it to the printer, try it, improve it once, print again and never ever again think of it as long as it does it's job.
No other machine built by humans is that complex.
I used Blender as an example because it's open source.
Of course these concerns matter more if you’re making a lot of these “widgets” but even in the case of a one off you’re going to have servicing costs and the occasional redesign/replacement/upgrade ...
It’s not that far out of line with software - widely used products go through tight highly iterative development cycles whereas one off solutions tend to be just “good enough” with bug fixing and the occasional feature request.
The race to the bottom incentivises profit over principle. Boards don’t care if something is built to scale if the competitor gets to market with a cheaper solution. Scaling is a day two problem.
I’m bearish on my future employment as someone who questions the motivations for profit driven development.
I naively thought the market would reward the best product, instead I see the cheapest product being rewarded. What happened to innovation and doing what’s best for society?
That's because you are looking at best from one angle and the market from another.
To the "market" the best may well be the cheapest thing that mostly does it's job/most of the time.
When I was young I used to think users cared about bugs and they do but they only care when the bugs are sufficient in number and degree to cross a threshold where they outweigh the utility of the software, That sort of 1 in 100 times it loses an hours work, well if the 99 times it saves me an hours work, shrug.
What we (as techies and I'm generalising wildly here) want to build is heirloom quality carpentry, what we get to actually build is franken-furniture from ikea packs that we hope has some some instructions and won't fall apart next week.
In this case that would be a high degree bug.
No mentioning of the underlying infra with all its complexities needed to acheive the goals of flexibility, reliability, speed and cost cutting.
You don't have to be of Netflix size - when you start getting tens of thousands of users, complexity hits you real fast.
[0] https://youtu.be/QrEOvHdH2Cg
1. The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. In Coasian terms, the transaction costs around development are minimized.
2. It enforces hard interfaces and it clearly separates development from operations. Lack of discipline around these issues is a source of much accidental complexity.
[1] I can’t seem to find the quote, I read it in an interview a few years ago and stuck with me since.
Some companies such as Microsoft, Google, Amazon would disagree.
If you took the 100 biggest code bases in the world. I would be surprised if more than 10% of them were open source.
[1] https://www.informationisbeautiful.net/visualizations/millio...
And again, that's for a middle size company you haven't even heard of. FAANG, Microsoft, Oracle, IBM for sure will have stuff dwarfing that.
An explosion on the number of lines of code is one of the way development teams fail.
The truth is that Oracle is ridden with coherence bugs, and have a much worse performance picture out of the box than Postgres. But while improving the Postgres performance requires deep digging into the DBMS itself, Oracle has a lot you can optimize just on the SQL interface. But there are plenty of cases where Oracle just can not become as fast as out of the box Postgres, and many where it could in theory, but a bug prevents you from doing things right.
Overall, three is no definitive ordering on the speed of the "good 3" DBMS. It always depends on what you are doing.
Most open-source projects have one to maybe five real contributors, outside the drive-by pull requests to fix some bug.
https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=98...
Bonus chatter: In a discussion of Windows source control, one person argued that git works great on large projects and gave an example of a large git repo: “For example, the linux kernel repository at roughly eight years old is 800–900MB in size, has about 45,000 files, and is considered to have heavy churn, with 400,000 commits.”
I found that adorable. You have 45,000 files. Yeah, call me when your repo starts to get big. The Windows repo has over three million files.
Four hundred thousand commits over eight years averages to around [130] commits per day. This is heavy churn? That’s so cute.
You know what we call a day with [130] commits? “Catastrophic network outage.”
Also, even if we normalized both cases for 'code files only' and/or 'kernel code only', there could still be architectural, code style, and development process differences that lead to different metrics for each project.
“open source” isn’t a development methodology, or even a distinct set of methodologies.
> The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale.
Plenty of open source projects don’t have that, nor is there anything restricting those things to open source projects.
It is true that some open source projects, because they see the value in new developers jumping in quickly, prioritize having structure, documentation, and tooling that supports that. It’s also true that some proprietary software projects do, too, because the project owner sees value in that.
Yet every large project behaves in a similar way, with only a few larger variations.
There are books written about different authors idealizations of how to do open source development. The fact that many people have written about different methodologies for approaching a particular challenge doesn’t make the challenge a methodology.
And, yes, I am aware that the very approach known as “the bazaar” (from the essay “The Cathedral and the Bazaar”) is sometimes referred to erroneously as “open source development”, which is a particularly glaring error since all of the examples of both “cathedral” and “bazaar” development were open source projects.
If you grossly simplify it to its innermost core, making development scalable means that if you have 10 times more people you can have 10 times more features/bugfixes/speed improvements/... The only way to do that is to make sure that an additional developer doesn't depend on other developers to work, and that can only happen if everything is properly documented, the build instructions are up-to-date, the processes are clear, basically anyone can start from scratch and get up to speed without the help of anyone else.
That kind of organization traditionally doesn't happen in on-site companies, where newcomers are followed by senior people, they have to follow some introduction course to familiarize themselves with the processes, they have to ask many questions every day, they need to be synchronized with other persons which brings some inefficiency beacuse everyone works at their own pace, etc... This all disappears when everything is properly documentend and every contributor can work in the middle of the night if they wish. I think the Gitlab Handbook goes over this quite well and describes a framework to implement that kind of organization but the rules are retrospectively obvious for people already used to open-source (https://about.gitlab.com/company/culture/all-remote/guide/):
- write down everything
- discussion should happen asynchronously. Any synchronous discussion (by text or call) should be only very small points. Whatever the type, write down the conclusions of that discussions
- Everything is public (to the organization), including decisions taken, issues, processes
Another very relevant factor is that people can just clone stuff, create new projects, and everything moves independently. So the open source development model has a pile of solutions for dependency management that team based development doesn't adopt.
But the one thing I don't get is why team based development doesn't adopt those solution. They are not expensive. Yet, even when I was able to dictate into everybody's requirements, I wasn't able to force teams into adopting them. Instead they insisted on synchronizing themselves by much more expensive and less reliable means. My guess is that most developers never dug any deep into open source and have no idea how it's done.
But cities ain't pretty. Dig the ground and nothing looks like what's on the map. To create a new building, you need to cut a lot of plumbing and re-route it somehow.
Stuff gets old and breaks all the time, and maintenance teams are on the ground 24/7 to fix it. NYT subway is the mainframe of the city. Look at the central steam heating that's still pumping water into high rises.
Sure, you can sell a shop, and people will be able to repurpose that space to suit their needs very efficiently. But isn't that what Docker does? Provide a cookie cutter space to run an app?
But in cities, there are places where trucks can't go, where you can't park, where unplanned work happens, where the sewer floods. That's when the city 'bugs', and people need to manually intervene to fix it...
Trying to find an ideal way of running software at scale is just as utopian as building the perfect city using 'super structures'. It's a dream of the 60s that's never going to happen.
We have good science and steady (albeit slow) progress in the former while it feels like the latter is more or less subject to stagnation. For instance, when I need to setup a DB cluster, why is there no tool that takes my requirements and generates deployment scripts, monitoring, migration tools, etc.?
I think OP heavily overestimates the organisational praxis in other disciplines. Nearly every creative discipline I was ever in was largely purely ad-hoc with very little explicitly stated organisational approaches to the craft. Academia, movie and music production, and creative writing for instance have much less readily-available principles than software.
Software is probably the most thought about creative industry I can think off in modern history.
What distinguishes software from other techne is a lack of physics. There is no 'solid ground of reality'. All other forms of making involve discovering and then applying the governing laws.
Hardware architectures, operating system, and programming languages, to an extent, do furnish a phenomenal context and where these characteristics are stable and well established (e.g. Memory Models), science appears [1].
But clearly, this ground of reality is indeed 'soft' and the practitioners usually re-invent not just the wheel, but the ground that it rolls on as well.
The dilemma -- which nearly means facing a choice of a multiplicity of lemmas -- that confronts the designers of hardware, OS, and languages is precisely that tension between generality (and corresponding weak "world" semantics) and specificity (with robust but constrained semantics).
[1]: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
I consult for government departments that often have legally mandated requirements for making something available online. That could be a form submission process, some geospatial information presented on a map, or whatever.
The problem is that when you might have 500-10K user transactions annually, it becomes crazy expensive to write bespoke software, even with the most agile process and the lowest overhead tooling.
Take cloud deployments, for instance. Sure, you can just "click through some wizard" pressing next-next-next-finish and be up and running, but the security team won't allow your infrequently-maintained web server on the Internet without a web application firewall. Setting one of those up is days of tweaking to avoid false positives.
Need to send mail? Azure bans outbound port 25 connections, you have to use Sendgrid, or something like it. Time to read up on yet another unique and special API!
Collecting fees and penalties or making payments to citizens? Woah there... there's a massive API surface you have to learn. Security on top of security is needed. Whitelisted IP addresses. Client certificates. That have to be rotated, manually!
You'll forget some essential maintenance, of course, and then you'll have to set up triggers and alarms so you don't get burned the second time. Which entails mailing lists that change dynamically because the team of contractors has a turnover rate faster than the typical certificate expiry time. Send too many alarms and all recipients will configure an Outlook rule to ignore them. Not enough alarms and you'll miss issues. Just setting this up semi-reliably is an exercise in itself.
Really basic stuff becomes difficult, when you realise that 99% of the alerting and monitoring features in Azure and AWS are designed for systems at their scale. It's all about analysing beautifully smooth curves of graphs aggregating millions of points of data, where deviations are glaringly obvious. These tools are utterly useless when you get one real transaction per day, swamped by a thousand bots. The load balancer health check is 99.99% of the traffic for some of these sites!
Then there's the human element:
Have you tried justifying the time to set alarms for a system where a week-long outage might only affect a dozen customers?
How about the budget to upgrade to a newer operating system for something that is not technically broken -- merely hard to support now?
Or have you tried doing any sort of maintenance on a system that was built by contractors hired for a fixed term, all of whom are now gone?
Meanwhile departments are renamed every couple of years to suit the whims of the latest batch of politicians, so everything has to rebranded. Even tiny little sites use by practically nobody.
I've seen sites up for 10 years, where I estimated that they cost $2,000 per citizen that actually used the site! Madness.
Terrible is a relative term. Terrible compared to what?
Who said/showed (much less proved) that there's a better way and we just don't follow it to achieve it some optimum?
There's also a semantic confusion here. Compared to e.g. the car industry we're infinitely better at "creating software at scale". We can create a billion copies of a software and distribute it our the world, with marginal cost close to zero.
But the author doesn't mean "creating software at scale" like when they say "car production at scale". They mean production of "large software".
Well, let's see the car industry make a car with the scope, flux requirements, shifting environments, etc that large scale software has...
Sometimes a CEO thinks up a radical crazy idea. While it distributes through middle management it gets twisted. Once it reaches the front line workers nothing really changes. One might consider this inefficient change management but maybe the company protected itself from a stupid CEO decision.