I feel like a lot of boilerplate is a choice, that often comes from frameworks being used.
I disagree with the text criticism as well. The reason we spend time thinking and reasoning about code, is not because its text format. Its because that's the job. We aren't scribes just copying books. The hard part of programming is figuring out what to do, not writing it out. Well i am sure better visualizations or something could offer minor improvements to comprehending code bases, i doubt it will change things significantly. We will always spend most of our time trying to figure out what we need to do not doing it.
> I feel like a lot of boilerplate is a choice, that often comes from frameworks being used. [...] I disagree with the text criticism as well.
The representation as text files is also part of what's causing the boilerplate.
As an example: How would you reduce the devops and configuration boilerplate in a monorepo?
In your typical JS microservice, a lot of code is not actually JS but yaml, json, terraform, etc. And it is very hard to abstract these away since a lot of tools rely on the existence of actual files.
Of course you can use code generation and other macros to manage the units in your monorepo (nx.js does this). But this is very instable since you might need to tweak some file resulting in your macros not working any more.
My take would be that textual representation is a local maximum. And our needs have outgrown this maximum. We're spending 90% of our time to innovate to push the boulder up the remaining 10% of this very hill. But there are most likely other hills with a much higher peak.
>In your typical JS microservice, a lot of code is not actually JS but yaml, json, terraform, etc. And it is very hard to abstract these away since a lot of tools rely on the existence of actual files.
This is a problem of your own making, though. If you don't do a microservice architecture in JS and do a modularized monolith in Go, as an example, you don't have to write any of this code.
Anything that feels repetitive, like writing boiler plate code, is something that can, should, and probably will be automated eventually. This is a constant if you are a programmer, whatever you are doing today that is repetitive, you'll likely be using some better way of doing that in the future. And as Alan Kay says, the best way to invent the future is to invent it. I'm always on the lookout for doing the same things with less repetition.
Boiler plate usually results from people using something that was not originally designed to do that thing or not designed that well. It's indicative of some kind of design friction or feature creep. When it works, people just do more of it without really thinking about what they are doing or why. If you see people copy pasting the same blobs of code over and over again, that's a good sign something is wrong.
At some point that becomes the way things are done and people start nit picking each other about doing it properly and then inevitably somebody comes along and does a thing that is vastly simpler and accomplishes the same thing. Happens over and over again. Some people then inevitably resist that new way of doing things because they are really invested in the old way. But the way our industry works is kind of Darwinist; so those things tend to die out quickly once something better comes along.
The two mistakes people in this industry make over and over again is assuming that 1) they know it all and 2) things don't change. Because things actually do change, and usually for good reasons, the former requires work to stay true. Some skills last longer than others and not all change is great. If you are just coasting and writing the same stupid code over and over again like it's ground hog day, it's going to eventually run out on you.
> Maybe you could write tests as queries that would test a whole set of possible programs, not only the current version of your program at the moment.
I think that the future of programming is more sophisticated static analysis. Programmers will write statements like, "every code path that writes to the Payments database must have called validate_user()." Then, the tooling will confirm that rule with every commit.
We kind of have this already (for example, Facebook's Infer tool [0]), but I think it will become much more important in the coming decade.
To set up an example, Azure has some API management stuff that could let you do this before you even got to your code. Writing a tool to make sure that API management rule exists would be different than static analysis.
I'd agree with you, but not all of the business logic like that is going to live in code in the future.
What we need before all this is a better way of specifying program behavior, both at a high and a low level.
In mobile there are multiple tools that you can use to prototype UI behavior. They allow you to specify navigation, transitions, clickable areas, behaviors when clicked, swipe actions, etc. They're this way so that the UI behaviors can be seen before coded (to an extent - not every prototyping tool does everything).
For code, well, only formal specifications get to that level. I haven't actually seen one of those outside of software engineering books, but I presume they exist in more critical fields. Commercial software doesn't really use them because they're too much work.
I would suggest to author to look at functional languages, like Haskell. These are based on principles of function composition and referential transparency.
The move from testing to static analysis is happening there, in the expressiveness of the type system, and its inferences. Functional languages are on the forefront of experimentation with new type systems.
There’s a lot of complaining and not a lot of suggesting going on in this article so I don’t take it for very much.
I feel like the author got a job and realized that it almost exclusively involves writing glue code and tests and looking at text diffs, so how convenient to imagine a world where they didn’t have to do the boring parts of their job.
a Shannon-inspired taxonomy (note the indefinite object; ymmv):
- glue code: given the outputs, you can reconstruct the inputs. this code translates between formats, between systems, etc.
- parsley code: given the outputs, you could reconstruct the inputs. this code "gussies up" its inputs: it takes table rows to JSON, or JSON to framework objects, PDF, etc. so in principle the original data is still there, but once the outputs are sufficiently complex the inputs become extremely awkward to reconstitute.
- crunch code: given the outputs, you have no hope of reconstructing the inputs. this code does real computation, and forgets stuff. Example: You can get a source program given only an executable, but you won't get the source that it was compiled from (nor will you, without a lot of work, even get a source you're willing to read)
I think you're being unnecessarily harsh. I get your point about OP presenting problems, not solutions, but on the other hand, they are a programmer & they are clearly trying to automate the repetitive parts of the job. Isn't that basically the essence of programming: what good programmers should be doing is automating as much of their job as possible.
Software reuse-in-the-small is a solved problem, but reuse-in-the-large seems unsolvable. That’s been the state of things for decades.
Testing is hard because the requirement for correctness is often absolute and inflexible. When software is allowed to be a little incorrect, then yeah testing gets much easier.
Yep. And your "reuse-in-the-large" I'd argue is also a side effect of reality itself. Take any market sector or industry, not just software - I suspect the "easyness" of directly reusing and copying any complex system is vanishingly small. If that were not the case, we would have infinite choices and brands of any complex product ever made!
Eh, unfortunately the one programming breakthrough the world actually needs is one that would drastically change, and perhaps harm, most of the people around here.
We need more "Excels." More and better tools that let "regular" people program.
Or more realistically the rapid application development tools need to be better so we're not bogged down dealing with latest fad or resume driven development
There's not much reason to improve things. No one's out here saying "you don't get this job because you chose new technologies when they weren't needed".
Even if you can equate glue code with trash, we seem to have unlimited space or a black hole to throw that trash.
We can't blame spreadsheets for the cited errors, because they're errors in arithmetic rather than Excel bugs. All businesses should get rid of arithmetic.
The question is not just how much Excel costs, it is how much revenue it helps bring in. And in many places, it brings more than it costs. Nothing is worse than a poorly designed webapp that was supposed to solve the "Excel problem" in a process.
Many more such cases [0], but Excel and spreadsheets in general also save businesses $$$$$, which explains their pervasiveness. Calculations take a long time and are easier for a human to mess up, which is why the initial spreadsheets were so popular as to drive the adoption of the personal computer in general [1]. The gulf between Excel and Python is pretty obvious, but there recently there have been few with their sights set on it, as most programmers looking to improve programming do so from their own experienced vantage point.
It is not a fantasy for small businesses, who rely on internal tools and processes to operate. These tools solve real pain points for them and that is why they exist and why it is a fast growing market.
It's a lot of hype right now, not necessarily indicative of staying power. We've been through multiple bouts of "no-code/low-code will replace X" and it has never materialized in any meaningful way.
I'm not going to argue whether the 'world actually needs' that, or if people mostly, actually want that, or if, since excel is already there if you want it, you probably don't have to write, sell or build any more low code tools.
...because, although I could argue about those points, it's fundamentally unrelated to the OP, and issue that programming is hard, and, for a very long time, no one has really had any idea how to solve it.
Now, however, AI generated code (like copilot) is, for the first time in a long time, a potential avenue to actually change how software is created at all levels.
I think there's pretty interesting, because it opens up a lot of new opportunities.
Is the future dynamic languages / high level specifications that are AI-transformed into typed verbose languages like C/Rust/whatever and then compiled?
The ancient tools like Rational Rose tried to do this, but the tech was never actually technically good enough. Maybe... we'll see a real change in this space as the sophistication of the models improves... or maybe, like self-driving cars, it's always going to be 'nearly good enough'.
Hard to say.
...but, hot damn. More excels? Please no. Excel already exists. Don't rebuild that stuff again. Build new, different interesting tools please.
>...but, hot damn. More excels? Please no. Excel already exists. Don't rebuild that stuff again.
I don't think they literally meant another Excel. Just something that is easily accessible and usable by non-programmers to do very 'programmy' things, etc.
This argument also just assumes, 'anyone can sit down and use excel'. Which is not true, excel has a learning curve like anything else, most people learn 'just enough' to do their job, maybe there is a guy in their office that knows a little bit more and they can learn from them. But excel proficiency acquisition is very similar to a programming proficiency acquisition, maybe the track runs out faster (at some point you reach real diminishing returns and limits to a spreadsheet's capabilities but I've seen some insane shit written in excel, raytracers for instance, but no-one who has done that thinks it's a good idea.)
>But excel proficiency acquisition is very similar to a programming proficiency acquisition
Strong disagree. Maybe if you only consider those extreme cases you referenced (ray-tracing, heavy VBA, etc.), sure. But in general? No shot.
Excel gives immediate feedback with a visual interface. There's no need to learn about variables or syntax or memory or compiling. There's no dependencies and no package managers. It abstracts nearly everything away. Click the place you want the thing to go, type the thing. Want a chart? Click the picture of the chart you like and click and drag over what you want in the chart. Want to change chart colors? Click the color you want. Want to change how the data is displayed visually? Press the bold button, or the color button, or the border button, whatever your heart desires is a click away with the same interface your used to with Word.
A great comparison is a geographic heatmap. In Excel, you need 2 columns. One with some States/Provinces and the other with some numbers. Then you click the big map button and instantly have your map. I can write the instructions for it in the space of a napkin and someone who has never touched Excel can be making all the maps they want. Now think about how you would explain making the same geographic heatmap in Python (or whatever your 'easy' language of choice is) to someone who has never learned any programming concepts.
The amazing thing about Excel is that many (most?) people start using it without EVEN knowing how to make a simple formula.
They literally start using it as a way to lay out tables (usually of data).
Then they make graphs.
And add more tables.
And learn how to use it like a basic pocket calculator (half of the formula I see that should be sums are actually =B21+B22+B23+B24+B25…)
Then comes SUM. Then VLOOOKUP.
Perhaps some IFs or COUNTIFs.
This process can take years for someone to go from first use to anything even slightly programming-like, if you squint at it.
A tiny fraction of these people ever do anything complex enough it looks even much like low-code or no-code "development". Even less write macros.
But you know what? In the first half hour, when all they wanted was to email round a list of team members and their lunch preferences, they already achieved something with Excel that they found useful, and they felt productive ever since. Most Excel users spend almost NO time learning the tool compared to how much time they spend getting (what they see as, and what probably is) value out of it, even when we are sat in sheer terror as we witness the horror they have unleashed on the world.
That is the bar you have to clear to make a "better Excel".
Honestly I tried to do some simple shit in excel today for the first time in years and holy shit, who thought this ribbon interface was a good idea. I'm going back to apple's numbers.
They mean no-code or low-code tools, but those come with a large hidden layer. These tools are known for making the 80% easy and the other 20% impossible. Excel can also become a mess.
>Is the future dynamic languages / high level specifications that are AI-transformed into typed verbose languages like C/Rust/whatever and then compiled?
How is that different from the present? Why are these "AI-transformations" different from what a compiler can do?
It’s fundamentally more sophisticated. This is like asking what is the difference between modern ML translation and the previous 20 years of research on language translation; the former actually works.
The latter basically doesn’t except in very specific circumstances.
Compilers can turn language into instructions only in a limited extremely specific set of circumstances.
We need more interesting tools, as you said. As long as the horizon of innovation remains on "something that generates good old code" I'm not very excited about the future. Copilot is an amazing tool, sometimes gimmicky, but a tool of the present nonetheless. As a compiler, it takes instructions and generates instructions.
We need a future where AI is involved in debugging, and program comprehension, and correctness. More interesting tools than a better RAD.
Yeah, I was using Excel as shorthand for "simple enough yet powerful enough tools such that someone who is perhaps an expert in something else can build themselves something useful."
I'm a firm believer in empowering end user automation. The atrocities that folks cobble together with the Excels of the world are a marvel, and more power to them.
But in the end, we all talk about leaky abstractions, and the stark horror is that the fact that all of these wonders run on a computer, the worlds leakiest abstraction, the worlds most stubborn, pig headed, cantankerous contraption out there.
I think it goes beyond automation and into customization. Giving end users the power to "program the program" just makes it possible to make the thing do what they want in a way that it wasn't set out to do. Without having to pay/find/wait for a developer to do it. Sometimes it's about automation. Sometimes it isn't.
Why would it harm us? We are problem solvers, we can use our intellect somewhere else. Programming is just a tool.
A lot of my colleagues are hybrid of traders + programmers. They do a lot of Excel. Sure, we can automate text programming, let everyone do Excel, and us problem solvers still won’t find difficulties finding high paying job.
I was agreeing with you, but as I thought about it, I remembered MS Access. It is used, often ships with Office, but still isn't close to as popular as Excel. Either the model is wrong, or the need isn't there.
Access used to be pretty standard for a small business trying to improve their situation, but we in the software development world spent so much time trying to work around limitations of Access and upgrading growing businesses out of Access that we took it personally and shouted to everyone not to use Access.
However, it is still taught in IT classes as a useful tool for small businesses, and it is.
My very first paid web gig was connecting a small business' Access based product catalog to their website. I was 14 or 15 at the time. I was able to do it with out any real coding knowledge except for some basic HTML I learned from MySpace and some copy-paste VBScript, haha. The site was a hit and I made $1,200 and immediately spent it all on an electric guitar.
In my experience, the main problem is that while Excel "distributed code versioning" with a whole bunch of copies around is awful, Access databases are not only much more prone to corruption and data loss in general (even in single user, non-simultaneous ideal use cases, never mind people attempting to multiplayer edit it over network shares...), it's often very hard for non-programmers to get any insights on any data you threw on it without using Excel in the first place.
So if the data is coming from Excel, and being extracted back to Excel, why use anything besides Excel?
You can already have "regular" people as software engineers if you follow basic concepts like the SOLID principles. Unfortunately many engineers are incredibly smart and they don't see the difference between writing spaghetti code with cryptic names and writing clear, legible code that normal people can follow. I've worked with people that could figure out a program written in binary if they needed to, but that meant that they never learned to write structured, sensible, readable programs.
>We need more "Excels." More and better tools that let "regular" people program.
The only reason why more people don't program is because Windows is antithetical to programming _anything_. The second you remove people from a windows environment is the moment they start coding, even by accident.
True, but just using Excel does not lead to decent data models, though. IMHO, what we would need is for "regular" folks to grasp the basics of relational database first (at least 1:n and n:1) and then build an easy to use excel-like tool around it.
>More and better tools that let "regular" people program.
There's one catch with Excel, the more complicated your spreadsheet gets, the closer it gets to programming. And at some point, it's more cost effective to write in a proper programming language than continue to maintain mess.
However, until this complexity level, Excel is great, and empowers a lot of people who barely know the basic features of Excel to solve their computational problems.
Maybe future programming systems can use an assertive pattern, where software developers only define what the software is supposed to do. And the programming systems generates a program that fulfills all assertions.
Then when the software designer finds that the software does something wrong, he can simply add assertions. Its a little like TDD but without coding.
For GUI applications it would be cool to define assertions via natural language and the programming system drawing pictures to explain what it understood.
My intuition is that making all the assertions to write correct code with decent performance and scaling is at least as complicated as writing the code.
Main reason to make the distinction is that most systems that parade themselves as "low-code" today are IME indistinguishable to high-code development to your average joe.
Even taking this example, the level of value totally depends on the degree of abstraction you can trust the computer to perform for you. There is a tremendous amount of nuance to human language and while we've come a very long way in that field of study, it's still a hard nut to crack.
> Maybe future programming systems can use an assertive pattern, where software developers only define what the software is supposed to do. And the programming systems generates a program that fulfills all assertions.
Isn't this just a programming, but harder and worse?
We've invented programming languages to be less ambiguous about what we want done, that's a feature and not a bug. It allows us to avoid the struggle of lawmakers (who are programming in spoken language) and lets us tell the computer exactly and unambiguously what to do.
Natural language does not.
"Bang on the floor" may mean to slam the floor or to have sex on the floor.
There's a programming joke that goes like a programmer's partner says "Go to the store and buy a loaf of bread, and if they have eggs, buy twelve". The programmer comes back with twelve loaves of bread.
Time flies like an arrow, fruit flies like a banana.
When I say "what" but not "how", I am trusting the system to be reasonable in the "how" it generates. But "reasonable" depends on my situation. Unless the system knows and understands that, it may easily do what I said, but do something that is anywhere from non-optimal to disastrous.
And if I have to say enough to prevent that - to prevent all the ways that could happen - is that more efficient, or less, compared to just writing the code myself?
If an AI can now draw a rembrandt, then it's not far off that an AI can read this:
Regular users are allowed to edit their own books.
Editors are allowed to edit any books with the same account id.
The public can read about any book that is not in draft status.
I need a domain model where users can write books with the following properties: userId, accountId, title, author, status.
status is an enum supporting draft, published, no longer available
I know this seems like a completely different paradigm, but I think we were wrong about General AI needed to replace programmers - we just didn't know how "normal AI" would go without being General AI.
I can also see this extended into the front-end at least for rudimentary forms. I can imaging someone doing a transfer AI to make front-ends have a consistent and nice looking feel that can change pretty easily - perhaps to match the user making the request.
Assuming you mean that to be a rough draft for the spec of a book management app, you wouldn't need an AI to parse that and generate code. A simple DSL is enough and indeed the "spec" you wrote could be implemented in about ten minutes with Rails, devise and pundit. (I imagine most other popular backend webdev languages have similar libraries available)
I wonder if the author of the blogpost ever tried Rails’ convention-over-configuration approach. It might be exactly what he means when he says he gets to write less boilerplate.
> I think we were wrong about General AI needed to replace programmers
So, the hard part of software is figuring out what you actually need done.
You still need the person writing the system requirements to be precise, accurate and not ambiguous, or you need General AI to figure out if the supplied system requirements meets those requirements.
And when things inevitably goes wrong, or requirements change as they always do, "you just" have to review and rephrase your entire requirement list, or you need General AI to do that for you.
I’ve been programming for over 30 years, and never been as productive as before. Need to load and decode a jpeg/png, I grab stb_image.h. Need to decode an ogg file, libogg. Need to decompress, libz. Need to decode video, libavformat. Need physics, libbullet. Need truetype fonts, freetype. Need a GUI, Qt. Need SSL, libssl.
My day becomes selecting libraries, integrating them, and testing integration. My business code is < 20% of our entire codebase. And I’m much more productive in 2022 than in 1992.
What is your alternative then? Vendored-in dependencies with their ossified security vulnerabilities? Or figuring out homegrown code of dubious quality for the functionality that is not core business of the company/product?
Nothing so dramatic. The alternative is judicious inclusion of dependencies rooted in thoughtful, experienced engineering.
Robust, comprehensive libraries from reliable vendors can provide a lot of value and are slow to rot. These can be anticipated and added early so that they’re made good use of and can become the first tool to reach for before adding other dependencies. Think React, lodash, QT, boost, etc.
Meanwhile, I acknowledge that most other packages and repos are of far more dubious quality than anything my team would write, have no accountability to my team or stakeholders, receive few/no code reviews when added or updated, introduce conflicting style/semantic conventions, and generally expand the surface area for bugs and vulnerabilities by including many lines of code that have no relevance to the project.
There are no strict rules, but these are the sort of considerations that weigh in.
My experience in engineering tells me that vendors and dependencies come and go and get acquired and all of them develop their own set of problems over the years. I get what you're saying, but you picked projects that are easy to see as popular in hindsight, but not so much in early adoption. Why react over angular? There is a problem with picking vendors outside of just "picking the best" that has multiple elements:
- Age of the company
- Complexity of integration with the vendor
- Stability in early adoption
If you adopt the tech too early, you risk instability during a time where you don't have the size as a business to afford that. Nobody that's a customer of Google is going to leave their contract because it was down for an hour. However your new 20 employee startup goes down for an hour? And you just signed your largest customer yet? And it's because you decided to early adopt some tech because it simply "looks good and efficient?". No thanks, that customer will be looking at alternatives when renewal time comes, if they don't just pay out the contract and leave on you already. And good luck getting funding with that blemish that you blew it by adopting cool new trends.
Okay, so being early adopter at a startup is risky. So maybe you go with something that is "kind of" new, and is being used by other large companies. Great, so you start building, and the feature system grows, and the integration grows, and now some new system comes along that is way better. Think about life before React/AngularJS and people who built systems with plain jQuery. There was a time where that was the hottest thing and nothing like React/Angular even existed. So you pick that and then 3 years later you have piles of jQuery and AngularJS/React/whatever else comes out, but remember the maturity thing mentioned above. So now you watch it grow over 2 years and now you have 5 years worth of jQuery and the amount of work it will take to switch to React/Angular is a year long project. You can't just stop everything else to work on this, so it's 20% of your time, nevermind that it's a moving target as you fix bugs/release new things in jQuery over that year. So now you're maintaining both for a year, and then 3 years later it becomes clear that React is the winner over Angular, and you picked the wrong one! At the time you picked they both seemed pretty mature (Google w/Angular and FB with React). Oh jeeze, do we migrate everything again? And from the business perspective how do you even begin to sell this to the people in charge when it results in no benefits to the end user. And by the time you finish it will these even be the tech you should be using? Or will something else rise up to replace that?
The complexity grows with the integration as the age of the company increases. And adding headcount and expecting to solve the problem just introduces more problems. Now your team size is 10 instead of 5 and standup is taking an hour everyday, do you break it up into other teams? What are those teams called? What if nobody wants to work on the upkeep of the legacy? Why would they as an IC? So now how do you split the work up fairly?
I'm not saying "don't use vendors", "don't adopt early" or "don't vet your vendors", but what I'm saying is the problem isn't the tech. It's the fact that the "obvious correct solution" is a changing answer over time. And the integration and added complexity combined with team dynamics eventually becomes the hardest problem a company has to solve.
The real answer is to cultivate a team that can work on the project long term using the old tech for significant gain. Your turnover just needs to be slow enough that new developers can be ramped up properly.
I did a gig at a place that brought me (along with like 50 other people) in to transition from a server rendered PHP app to a React front-end. They'd bootstrapped to something like $300M and had just brought in a new CEO who wanted to 10x the company.
It was fucking pandemonium. Every day I saw another message on their slack along the lines of "It's my last day, I had a lot of fun working with you all the last 8 years". My team was completely useless, as far as I could tell our mandate was to basically be nazis and tell all the supposed mouthbreathing pre-historic over the hill PHP greybeards how to write "good" React code (aka unmaintainable garbage).
I have no idea how they're doing now. I don't see them mentioned much anymore. They're probably worth $30M.
If you had to bet on who could 10x your company, would you go with the guys on the ground floor that have literally bootstrapped a company to $300M off the back of their PHP tech? Or would you go with the army of hip Javascript consultants, half of them off shore?
I say this as a specialised React developer, who has been doing solely React for 2015.
There's absolutely no reason why you have to roll with the industry to every new fad. I have no intention of going anywhere when the next thing comes along. In fact, I'm waiting for all the noobs to fuck the hell off so I'm free to actually write good React code again. I bet that's probably how the PHP developers felt back when that was the fad of the decade.
If that company had backed their PHP developers in and given them enough shares to stick around, I bet they'd probably be a $3B company today. 90% of that is going to come down to the business side of things, not whether you throw away your entire engineering culture to score 20% better on some fucking render metric on a page transition or whatever.
> Robust, comprehensive libraries from reliable vendors
Hah! In all my years the biggest, most difficult vulnerabilities/problems to remediate all came from vendor-made (90% of the time closed source) software. With the free, open source stuff the problems are usually very minor but even if they're BIG we'll pretty much always get fixes and deploy them faster than if we had to wait on a vendor.
The more popular the OSS the safer it is but even niche OSS tools/libraries are easier to work with by definition because we can look at and modify the source to correct the problem even if the original maintainer hasn't gotten around to it yet. This is actually a big reason why Python is preferred where I work: The code is inherently visible and fixable.
What language code is written in has little to do with how "flexible" it is. Or what did you mean? Refactorable? Readable? I don't immediately see what the choice of Python specifically has over other languages.
The alternative is shipping your own, which most of the time is a) a bad idea since you don't have domain specific knowledge and b) the customer is not paying for you to reimplement the 10th jpeg parsing function.
> My day becomes selecting libraries, integrating them, and testing integration.
Funny, being an old, cranky, gray-no-beard, if I wanted to spend my days selecting libraries, integrating them, and testing integration, I would have become a digital electronics engineer.
Your point is well taken, it's just our modern truth. It's just not was attracted me to this world in the first place.
> Writing tests is time-consuming, usually doesn't scale, and it easily creates tight coupling with implementations
After thousands of years, humans still use double accounting in finance. This is because it's effective. Unsurprisingly, it's tightly coupled to the original transaction recording.
The constant assertion that tight coupling is bad, because it creates more work which can be error prone. This is a fundamental problem with how people discuss software testing. It's not a panacea, but a guardrail. Guardrails aren't indestructible. That isn't the point. Developers have an obsession with making things as easy as possible (be that simplifying or codifying) and this is not a domain where that makes for a better solution, all other qualities being equal.
I think "double accounting" is a good analogy, and if you look at it that way, some tests that look like the same thing written twice in a different way start to make more sense.
The problem is just with effectivity - it'd be fine if the time spent on test would be the same as time on code, but very often the effort on test is much bigger while not even covering all important cases. I think we need to look for methods to get a good case coverage without spending majority of our time writing tests.
Hit the nail on the head. For same reason requirements writing is so difficult, to explain every edge case you end up writing the whole software again, using human language.
Writing software is the process of explaining a set of conditions and conversions from input to output. We should assume the programmer does this in as short code as possible (unless we are using a low level language like C which adds a lot of accidental complexity like memory management). To repeat all these conditions in test-code, would be to write the same thing twice, in a different way. Not saying this is wrong, it's just important to acknowledge it.
Testing often becomes more like sampling, with hard coded or generated known set of inputs/outputs, often taken from a scenario derived from a common user journey. This verifies that this particular journey works, but it would never be a guarantee that other edge cases of it does, to do that you do end up in the write-twice situation.
And still you're typing repetitive text. My dad was punching cards and able to read code from punched tape. I grew up writing code on a home computer's display. And now, 35 years later, it's long overdue that some visual NoCode tool should be the default for mostly repetitive programming jobs. If I want to build a Finder Quick Action to resize an image, for example, I certainly won't open a C editor, I'll just use Automator.
> My day becomes selecting libraries, integrating them, and testing integration.
This is why I hate modern programming.
When was the last time you had a job where you could actually learn how to actually program something? The prolifiration of libraries means that all your bosses and managers would extremely frown upon anyone handrolling a solution instead of using an existing library.
Which means the more time you spend at a regular job, the worse your brain will rot.
Just talk to any programmer at any company and you will notice that almost none of them know how to program anything. They just know how to stitch together libraries with some glue code.
> The prolifiration of libraries means that all your bosses and managers would extremely frown upon anyone handrolling a solution instead of using an existing library.
They are right, even if you do not agree. Handrolling is incredibly cost ineffective, yet another source of bugs and can be seen as a bad idea when it comes to security.
Huge third party libraries like LibOgg are maintained by gigantic industry players. Chances are, they are more knowledgeable in that specific domain than you are and they can focus more developer time on what is just a small part in your business application.
How many of the people who are working on any given library actually are domain expert on the subject matter as you imagine?
This might have been true 20 years ago. But now the cultural shift affects even the people who work on the lower levels.
A sort of famous example is how slow Visual Studio has gotten compared to 20 years ago (when it was running on slower hardware even!) where it was nearly instant.
The refterm saga has demonstrated that people employed by Microsoft and paid very high salaries to work on a terminal emulator have no idea how to make a half decent terminal renderer.
> handrolling is incredibly cost ineffective, yet another source of bugs and can be seen as a bad idea when it comes to security.
Without context this statement is blatantly false.
Many libraries are full of bugs and security problems.
I find this comment amusing because most tasks the GP mentions are so specialized that you wouldn't actually have time to learn them. If your complaint is left pad and similar libraries, that's not what GP comment is talking about.
> usually the biggest problem is an enormous amount of CRUD boilerplate that looks very similar in each project, but it's nevertheless different in important details.
Once upon a time, a lot of microprocessor code was written in assembly. Then C came along, and many of the assembly people said: "whoah, hold on, I use different calling conventions in different places for good reason; what do you mean every C function is going to use a whole stack frame even if it's never re-entered?" But now 99+% of programmers don't care about sub-kb stack frames, and probably most of them even consider C too low-level[0] to touch.
I've said elsewhere[1] that much of the ceremony of cloud sw dev reminds me of 1960's mainframes — so what is the equivalent of a "structured programming" toolchain for the cloud, that takes some higher level description of inputs, outputs, and logic[2], and then generates not-entirely-ludicrous boilerplate to mash it all together?
[0] in hindsight, C's big advantage back in the day was that it mixed much better with assembly than purer HLLs.
[2] compare the "environment division" of COBOL with Terraform, and the "data division" with Swagger. I mean, we've definitely progressed on many fronts since then, but certainly not on the axis of "lack of verbosity"[3].
[3] OTOH, Conway's law suggests that multiple-team projects will always wind up with multiple configuration in multiple places.
The interesting part is that no, function calls do not all use an entire stack frame, unless you compile your code for debugging. Compilers are smart enough to reencode the calling into just registry shifting, or even inline the entire thing.
Well, the developer saying that some times had a real point in 1978. A lot of developers migrated, some didn't, often for very good reasons. The 99% of the developers being ok with C only happened after optimizing compilers was a thing.
Of course, none of that means that the entire endeavor was worthless before that. It just means that whatever improvement you create, it won't appease to a lot of developers, and it's ok. One can only get unanimity if the change has no drawbacks at all, under no context.
Great post - the one word missing, "maintenance." Most programming work is done maintaining/enhancing existing code. The greenfield work is a piece of cake by comparison.
You want to do the hard stuff? Maintain existing code you're not familiar with.
The problems around maintaining unfamiliar code are huge, largely unsolved, expensive and risky. There's a little branch of computer science called Program Comprehension and no one pays any attention. Though most programming money is spent on maintenance.
I always tell my clients - the difference between writing code and maintaining it, is the difference between raising your hands and keeping them raised indefinitely.
I always smile when a green field project starts and then they claim its “Clean Code”. No, you won’t known if it was clean code until years down and the system will need updates. Then and only then you can reflect and see how hard it was to changes things in it.
Fully agreed. No matter how "clean code", the next person or team is immediately going to label it "legacy" and complain endlessly about all the choices made by the original author(s).
Much as we denigrate COBOL, that is still its greatest advantage. Yes, it's wordy, yes it's old. Yes, it needs to be really updated. But it's still easier for a new hire to understand the COBOL old code than any other old code.
yes. I wish I could somehow get to work on a blend between: a decompiler, debugger, emulator, static analyzer, memory profiler, and so on.
The idea being some kind of a runtime for assembly code which does not actually execute the program but allows one to understand it in different sematinc levels, or dunno.. this is a very raw idea. needs a lot of work (and a lot more knowldedge) to set down.
too bad none of the professors that I was able to meet were really interested in this kind of thing
I think dynamic analysis is incredibly powerful and criminally underused in IDEs and other dev tools.
I have thought of an idea about 6 months ago that has been fermenting in my mind since then : what if (e.g.) a Python VM had a mode where it records all type info of all identifiers as it executed the code and persisted this info into a standard format, later when you open a .py file in an IDE, all type info of the objects defined or named in the file are pulled from the persistent record and crunched by the IDE and used to present a top-notch dev experience?
The traditional achilles's heel of static analysis and type systems is Turing Completeness, traditional answers range from trying and giving up (Java's Object or Kotlin's Any?), not bothering to try in the first place (Ruby, Python, etc...), very cleverly restricting what you can say so that you never or rarely run into the embarrassing problems (Haskell, Rust,...), and whatever the fuck C++'s type system is. The type-profiling approach suggests another answer entirely : what if we just execute the damn thing without regard for types, like we already do now for Python and the like, but record everything that happens so that later static analysis can do a whole ton of things it can't do from program text alone. You can have Turing-Complete types that way, you just can't have them immediately (as soon as you write the code) or completely (as there are always execution paths that aren't visited, which can change types of things you think you know, e.g. x = 1 ; if VERY_SPECIFIC_RARE_CONDITION : x = "Hello" ).
You can have incredibly specific and fine-grained types, like "Dict[String->int] WHERE 'foo' in Dict and Dict['bar'] == 42", which is peculiar subset of all string-int dictionaries that satisfy the WHERE clause. All of this would be "profiled" automatically from the runtime, you're already executing the code for free anyway. Essentially, type- checking and inference becomes a never-halting computation amortized over all executions of a program, producing incremental results along the way.
I have ahead of me some opportunity to at least have a go at this idea, but I'm not completely free to pursue it (others can veto the whole thing) and I'm not sure I have all the angles or the prerequisite knowledge necessary to dive in and make something that matters. If anyone of the good folks at JetBrains or VisualStudio or similar orgs are reading this : please steal this idea and make it far better than I can, or at least pass it to others if you don't have the time.
I developed a dynamic analysis/IDE tool for old IBM systems. I was told by VC's that IT management never spends significant money on programmer productivity. I think they're mostly right. Maybe because no one knows how to actually measure it.
This is how JavaScript intellisense in Visual Studio used to work, except that the program was executed "behind the scenes" using a trimmed-down VM that could execute without side effects or infinite loops. It was eventually abandoned due to poor performance, predictability, and stability.
The problem is this dilemma: If you have to wait for a "real" execution of a program, then very reasonable expectations like "I can see a local variable I just declared" doesn't work. If you try to fake-execute a program, you have problems like trying to figure out what to do with side-effecting calls, loops, and other control flow problems.
Trying to reconcile a previous type snapshot with an arbitrary set of program edits was tried by an early version of TypeScript and wholly abandoned because it's extremely difficult to get right, and any wrongness quickly propagates and corrupts the entire state. The flow team is still trying this approach and is having a very hard time with it, from what I can tell.
There has been attempts as you describe before. I can specifically point to work done in Ruby by my PhD advisor using the exact profiling approach, and then static typing from that: http://www.cs.tufts.edu/~jfoster/papers/cs-tr-4935.pdf
> you're already executing the code for free anyway
Based on my experience of working on similar domain of type systems for Ruby (though not the exact approach you describe), this turns out to be the ultimate bottleneck. If you are instrumenting everything, the code execution is very slow. A practical approach here is to abstract values in the interpreter (like represent all whole numbers are Int). However, this would eliminate the specific cases where you can track "Dict[String->int] WHERE 'foo' in Dict and Dict['bar'] == 42". You could get some mileage out of singleton types, but there are still limitations on running arbitrary queries: how do you record a profile and run queries on open file or network handles later? How do you reconcile side effects between two program execution profiles? It is a tradeoff between how much information can you record in a profile vs cost of recording.
There is definitely some scope here that can be undertaken with longer term studies that I have not seen yet. Does recording type information (or other facts from profiling) over the longer term enough to cover all paths through the program? If so, as this discussion is about maintaining code long term, does it help developers refactor and maintain code as a code base undergoes bitrot and then gets minor updates? There is a gap between industry who faces this problem but usually doesn't invest in such studies and academia who usually invests in such studies but doesn't have the same changing requirements as an industrial codebase.
https://github.com/instagram/MonkeyType can perform the call logging, and can export a static typing file which is used by mypy, but also e.g. PyCharm. It doesn't expose such fine grained types, but you could build that based on the logged data.
This is only feasible if the program takes no input, or a very limited set. Once you open it up to arbitrary input, no single run (or even a large set of runs) can capture everything the program might be expected to handle.
How does the type profiler know that the variable that only contained values like "123" or "456" was handling identifiers, not numbers?
It would seem like that's getting the types too late to be useful, but in practice most code goes into production in stages, so you could start getting this probabilistic typing data from code that has rolled out in a limited way, for example features that are hidden to most users.
>This is only feasible if the program takes no input, or a very limited set.
One of the insights that people making tools for dynamic languages discover over and over again is that most uses of dynamic features is highly static and constrained. In general, yes, a python program can just do eval(input("enter some python code to execute :) \n>>")), but people mostly don't do this. People use extremly dynamic and extremly flexible constructs and features in very constrained ways. This is like the observation that most utterances of human languages are highly constrained and highly specific, even within syntactically-valid and semantically-meaningful sentences, not all utterances are equally probable, and some are so improbable as to be essentially irrelevant. People who try to memorize the dictionary never learn the language, because the vast majority of the dictionary is useless and mostly unused, and even the words that are used are only used in a subset of their possible meanings.
>Once you open it up to arbitrary input, no single run (or even a large set of runs) can capture everything the program might be expected to handle.
Anything is better than nothing, right ? if your program keeps executing with some set of types over and over again (and it will, because no program is infinitely-generic, the human brains that wrote the code can't reason over infinity in general), wouldn't it be better to record this and make it avilable at static write-time ?
Human brains are finite, how do we reason over the "infinite" types that every Python program theoretically deals with ? We don't! like I said, most dynamic features are an illusion, there is a very finite set of uses that we have in mind for them. Here is an experiment you might try, the next time you write in a dynamic language, try to observe yourself thinking about the code. In the vast majority of cases, you will find that your brain already has a very specific type in mind for each variable (or else how can you do anything ? even printing the thing requires assuming it has a __repr__ method that doesn't fail.).
>How does the type profiler know that the variable that only contained values like "123" or "456" was handling identifiers, not numbers?
It doesn't. I think you misunderstood the idea a little, the type profiler makes no attempt whatsoever at discerning the "meaning" of the data pointed to by variables, it will only record that your variable held strings during runtime. If the number of string values the variable held was small enough, it might attempt to list them like this "Str WHERE Str in ["123","456"]". If the number of values the variable held was larger than some threshold but some predicate held for it consistently it can also use that, i.e. "Str WHERE is_numeric(Str)". If a string variable was always tested against a regex before every use, it will notice that and include the regex into the type. No additional "smart pants" than this is attempted, just the info your VM or interpreter already knows, just recorded instead of thrown away after each execution.
The profiler will not and can not attempt to understand any "meaning" behind the data nor it needs to in order to be useful, it's just a (dynamic) type system. No current type system, static or otherwise, attempts to say "'123' is a numeric, would you like to make it an int ?", that would be painful, absurd in most cases I can think of and misguided in general.
> No current type system, static or otherwise, attempts to say "'123' is a numeric, would you like to make it an int ?"
SQLite's column type affinity will in fact do this, if you tell SQLite that a column is an INTEGER it will turn '123' into 123 but will happily take a string like "widget" and just store it.
I also wanted to add that your thoughts on this subject are well-stated and align with some work I've been patiently chipping away at, in the intersection of gradual typing and unit testing. I'll have more to say on that subject someday...
> wouldn't it be better to record this and make it available at static write-time
I think you misunderstand my position. It's better for the creator to simply specify it when they write the code - i.e. static typing.
> the type profiler makes no attempt whatsoever at discerning the "meaning" of the data
Your examples are waaay beyond what I need or expect. What I need is for the program to recognize that, for a number, 123 and 456 are valid, but "abc" will never be. Conversely, for an identifier, no matter how many runs use values like 123, someday someone might provide the value "abc" and that's ok. Also, any code that attempts to sum up a collection of identifiers should not be runnable, even if the identifiers in question all happen to be numbers.
This is something that static typing provides, and no amount of profiling will ever be able to divine.
> In the vast majority of cases, you will find that your brain already has a very specific type in mind for each variable
Sure, for the code I write. But I've seen plenty of code written by juniors where, upon inspection, it was completely inscrutable whether a given parameter expects an integer, a string, a brick wall or a banana.
Which is all to say that my default position is unchanged: dynamic typing is unhelpful for anything larger than small, single-purpose scripts.
Something very close to this can be done and it's in fact done already by static analysis. Static analysis doesn't have any Turing complete problem. Static analysis has a limitation of providing complete answers because of the halting problem, but it can provide instead sound answers. That is, static analysis can provide all possible runtime types for any given (e.g. python) expression. This can be accomplished by doing Abstract Interpretation, or Constraint Analysis and Dataflow Analysis (kCFA), which is in way similar to what you suggest of running the program in a profiler, but instead it runs the program in an abstract value domain. With static analysis you will get some imprecisions (false positives) but no false negatives, so it can effectively be very useful for a developer in an IDE. The precision (amount of FPs) and performance are mutually dependent, but good performance and reasonable (read useful) precision can be achieved, although it's a non-trivial engineering problem.
Additionally, some programs cannot be easily and/or quickly executed to cover all possible paths, so parts of the program will remain uncovered by the profiler. That is one place where static analysis becomes very powerful, because you can cover all the program much faster (in linear time making largish precision tradeoffs, but analysis still yielding a usable result).
Indeed, Abstract Interpretation is powerful and beautiful as heck. Type Systems can be seen as just a special case of general Abstract Interpretation, where the types are the abstract values computed from each expression. The problem is that it requires ingenuity to devise abstract value systems that don't degrade into useless generality as unknown branches in the code 'execute'. Even types only succeed as much as they do because they require extensive and invasive (language-design-changing as well as program-changing) cooperation from both the language designer(s)\implementer(s), as well as the language developer(s), without this cooperation you will leave open very wide gaps. Symbolic Execution is another much much more powerful technique but, predictably, also very inefficient in general.
My way of looking at this is just frustration at the misallocation of resources. Dynamic programs already have tons of typing information available, just not in the right time and place. Your brain spends an aweful lot of time "executing" the code of a dynamic language while it's writing the code, just like the language runtime only far more slowly and with a sky high probability of error. So all what I'm saying is, why this immense suffering, when the information that your brain is in dire need of is already right there in the interpreter's data structures, just in opaque and execution-temporary forms ? It strikes me as incredibly obvious to try to bring in those typing information from the runtime into persistent transparent records available at dev-time.
Abstract Interpretation or Symbolic Executation are (fancy) tools in programming language theory's toolbox that can help us, but the simplest possible thing to try first is to make use of the already-available information that are just lying around elsewhere. To make a somewhat forced analogy : if we have a food crisis at hand, we could try fancy solutions like genetic engineering of super crops or hydroponic farming, but the dumbest possible solution to try first would be to simply bring in food from other places where its plentiful. Typing info in dynamic languages is very plentiful, it's just not there when we need it.
The problem is what I mentioned at the end of my other comment -- complex programs will get inputs from web APIs, console, database, so it's not possible to have complete runtime coverage for all paths in the interpreter on the general case.
"Principles of Program Analysis" covers the subject very well. It's not an easy or beginner book because it's very formal, but very complete and with plenty of examples.
> The traditional achilles's heel of static analysis and type systems is Turing Completeness, traditional answers range from trying and giving up (Java's Object or Kotlin's Any?), not bothering to try in the first place (Ruby, Python, etc...), very cleverly restricting what you can say so that you never or rarely run into the embarrassing problems (Haskell, Rust,...), and whatever the fuck C++'s type system is.
This stanza was maybe not your primary point but it was beatifully written and multiple programmer friends of every variety mentioned have been cackling at it.
I think if you have to run a whole program to understand a small part of it, you've already lost. The most valuable tools are a REPL that can execute units of your code, and a language that enforces purity and immutability so that local reasoning is more likely to be sufficient.
The idea of runtime type feedback was originally explored by the SELF group for optimization [0], and is currently used by JS runtimes like V8. The obstacle your vision faces today is that program editors are almost all completely separate to the languages they are used for, and thus such communication with the runtime is enormously painful and complex. Time to return to the truly integrated development environments of Smalltalk/Lisp with modern niceties like gradual types and multicore processors, methinks!
There have been interesting progress in this space from a different angle of attack. o11y (Akita, honeycomb, ebpf, prodfiler, ...) and "Learning from incidents" are two of these angle that have really built on this idea that "understanding what the system does in prod" matters far more than "writing it right the first time".
It is also the thinking we can see supporting a lot of the early devops movement.
Working on old but popular software is where legends are made. You need to be methodical about changes. How do you make changes in a million line+ codebase without breaking anything for millions of existing users? This challenge is reserved for the finest engineers on the planet. These are the people you want your desk near when you start your professional career as a programmer.
this is simpler than it sounds - the process trumps any engineering stardom. the process is king. the process is love, the process is life, quite literally. getting any change in there isn't so much fine engineering as it is wrestling with layers and layers of process, where every layer has been added due to a monumental f-up in the past. it's an environment where getting any change committed into a repository usually takes weeks, unless one of processes for sidestepping the process is invoked.
the problem is that a large amount of code produced is to support the accidental complexity of the software (let's call it infrastructure code) while the value generated by it comes from its essential complexity, that is, the application domain, as cited by the article .
while Domain Driven Design and Clean Architecture are a first step in the right direction, languages and frameworks are still limited in supporting these ideas.
It's almost insane to think that once you have a modeled domain you need to replicate them in resolvers, types, queries, etc (for graphql), resources, responses, endpoints (for rest), etc, etc.
to reinforce the point of the article and the one brought by @SKILNER, I believe that the great transformation that is to come is in maintainability through a focus on the domain
I think, as OP alludes, ultimately explorability is the heart of the problem that stops people from writing code this way.
If you don't care about other people being able to read and explore your code without a lot of preperation, you can go full hog creating layers of DSLs and metaprogramming, and with enough dedication you can end up with all your real domain level business rules in one place separate from the "infrastructure"
But if you do this, you end up with a codebase that is hard for a newcomer to ask simple maintenance questions about like "What are all the places XYZ is called from?" or "What are all the places to write to ABC" etc
So an experienced developer learns to limit their metaprogramming so their code retains easy explorability at the expense of long term sustainability. Golang is kind of an epitome of this kind of thinking. Lisps are kind of the epitome of the opposite I guess.
This is what's behind the paradox where a good dev working alone or maybe with one very like minded person can produce a level of productivity you can't match again as you add developers, till you have way more developers. The dip represents the loss due to having to communicate a common understanding of the codebase, that doesn't get adequately compensated for till you've added a lot more people.
> This is what's behind the paradox where a good dev working alone or maybe with one very like minded person can produce a level of productivity you can't match again as you add developers, till you have way more developers. The dip represents the loss due to having to communicate a common understanding of the codebase, that doesn't get adequately compensated for till you've added a lot more people.
Wow I couldn't agree more with this point. You put it perfectly. I worked alone and later with one other developer on a project and it felt way more productive than my current team of 5 developers!
I find other people's code difficult to follow due to all the abstractions that I wouldn't have inserted.
When I write Clojure code for example, the code just works. I somewhat understand the code I write.
But when I read other people's Clojure, I find it difficult to understand. I think it's my maturity in understanding Clojure, the mental model isn't there yet. Which is funny because I've been on two projects where we used Clojure.
I don't have the same problem with less metaprogramming languages such as C, Java or Python.
Also possibly including: install very old OS version, get libs and dependencies to some old version, and maybe update to a slightly newer compatible version, but not the latest. Create mock services of complex back end systems for test cases. Convert https connections to http for testing, or use unsupported TLS versions and protocols for some connections. Then you can get started!
I sincerely don't know why uni don't make more classes on that only.
- pick any software
- try to change something
- give a precise impact analysis
- generate a few potential implementation paths
- measure how fast you did all that and what failed / worked
A fundamental problem with how we teach programming is that we focus on writing, instead of reading, software. To borrow terminology from the language arts; we don't focus enough on reading comprehension.
At the University I'm associated with, there was a discussion about why more 'practical' classes weren't taught. The answer was 'We're not a trade school.'
I've also wondered if it's time to back off on agile a bit. Or at least "agile" as it is implemented generally, which means we get to make up new requirements every two weeks. In my experience, the hardest maintenance problems occur because we're trying to re-shape code into something that the developers never knew would be coming down the line. Spending lots more time up-front deciding requirements would go a long ways towards more maintainable code.
The part of Agile a lot of people ignore is the phase when, after your code works, you spend as much time as it takes to make it well structured and readable to others.
If you think of code writing as a process of organizing your thoughts, exploring/understanding a domain, and articulating it iteratively (for code/run/debug loop), then your pithy comment seems perfectly rational.
It's interesting to me that people are so into Agile when it simply doesn't work all by itself. How many books, conferences, certifications, practitioners, evangelists, etc. does it take to make a paradigm, supposedly the paradigm, of working actually work? Every company I have worked for that used Agile basically had broken processes and were not productive. The one job where we did not explicitly use Agile was actually a place where I produced the most useful work.
If the number one answer to we're using Agile but it's not working is "you're doing Agile wrong", then maybe Agile isn't the solution?
The way I worked at the place that did not have an explicit process was one of switching between agile and waterfall methods. Early on in the projects, the process was primarily waterfall, defining things up front, building prototypes, laying out the project, etc. Then once passed that stage, projects would enter into a more agile or iterative process. Then as new big features came in, back to waterfall and then switching to agile once that stabilized. This worked quite well.
I think engineering effectiveness comes down to a people and communication problem.
Writing software well is difficult by itself and people are all different in understanding, experience and skill level. Throwing a group of people together and trying to build something that works for every scenario is a miracle given the complexity of software and the complexity of interpersonal relationships and organisations (see Conway's law). I'm impressed by every multiplatform language or tool or software. It's a lot of work!
If everybody did things the same way i.e the agile way and if agile was proven to work and everyone followed it the way it was intended and designed, then maybe we could all be interoperable and easily work together and produce projects that don't fail. That's the fantasy.
To be fair I've been 6 years worth of agile projects and I was never on a project that failed.
Maybe I'm just an old curmudgeon, but it seems to me there's way too much emphasis on speed of development in general. Time to market seems to be more important than quality, robustness, security, performance, or any other concern.
Another thing that rubs me wrong is the recurring notion that we need to get rid of the text as a representation of code. I've yet to meet a mathematician who wants to get completely rid of formulas because coming up with a proof is slow and cumbersome.
I understand that the incentives are drastically different between academia and business, but perhaps we've gone too far in this particular direction. Perhaps it's okay to admit that programming isn't as easy as we all want it to be and it's okay to take the time to change things carefully instead of moving fast and breaking stuff.
> Another thing that rubs me wrong is the recurring notion that we need to get rid of the text as a representation of code.
I completely agree with this notion. However, I feel like we're sorely missing out on some form of visual exploration. I feel like the majority of my time is spent trying to understand the flow of execution of a program I'm trying to maintain that was written by other teams that are long gone.
It would be so amazing to be able to "zoom out" from the text and get a graph view of execution flow through different files. And then being able to highlight a particular execution flow that you're studying would be great. I know we already have some flavors of this with "find all references", or something like Visual Studio's profiling tools that highlight flows of execution, but none of these have ever felt like they improve the exploration of the codebase very much.
It would be very interesting to see some tools that allow a more fluid exploration of an unknown codebase. More akin to zooming out of a Google map view and tracing flow from point to point, instead of diving headfirst into a million files looking for the correct information.
Yes, like crocodile clips for a circuit board, I can insert a probe into a circuit and see what's on the wire.
When your callstack is 50 methods deep and there's 20 layers involved and marshalling and it's difficult to see what is going on.
I want to mark two pieces of code and see the data structures passing through them, similar to a debugger but more like a log file or trace like Jaegar or Kibana. But the actual POJO or JSON objects themselves.
This is a great analogy and captures what I meant very clearly! It would be awesome to pinpoint two pieces of code and ask for the execution flow visualizer for that path :)
I've felt like this for a very long time, and it's disappointing that there aren't really any tools to do this yet. It would make it so much faster to come up to speed on a new codebase, and also to see where some potential problems might lie.
> Another thing that rubs me wrong is the recurring notion that we need to get rid of the text as a representation of code. I've yet to meet a mathematician who wants to get completely rid of formulas because coming up with a proof is slow and cumbersome.
I see what you're saying, if I understand correctly, in terms of the hype around "low or no code" environments, if that's what you mean. However, I do disagree with the notion of equating code with text. It seems myopic in the same fashion of equating tape or punch cards with code.
Also, I think the mathematics example isn't quite apt. Mathematicians are very willing to use visual representations to both illustrate and prove ideas. The way mathematicians work is actually a great example of moving between visual and symbolic representations and also prose.
In many significant ways, text is a highly limiting form of expression. In my opinion, we have nearly tapped out the available expression that can come from purely text-based programming languages. One example of this is the limitations of the innovations in syntax. There's been basically no significant innovation in this area aside from small iterative improvements, and I think that's a fundamental limit we've hit. I feel the future of programming is likely to hinge on a highly hybrid environment. In some sense, Smalltalk and its ilk, Emacs and Common Lisp, TouchDesigner, LabVIEW, and vvvv are precursors for what could come.
A huge opportunity here: ways to assist developers to figure out how existing code base works by generating code flow, class hierarchy, etc.
yes there are a few tools on the market, nothing really standout though, maybe it's for AI/ML to innovate in this field.
Linux kernel is well designed, in that I can add code relatively easy into its subsystems, it's those Object oriented or FP code base that bothers me the most, especially when they're large, often times they just made me feel hopeless, good tools desperately needed.
I would argue that a huge or maybe even the primary reason why maintenance is so hard, though, is because a lot of software was originally not written "correctly", that is without maintenance and longevity in mind.
The hardest part of maintenance, in my experience, is that programs were developed under needless or incorrect constraints and then expected to be magically maintained. There is a huge downstream effect of decisions made early on in a software system's life.
The story of "just get it working" that evolves into "now that's it's working, don't change it but add these new features" repeats itself over, and over, and over. It's not surprising why maintenance in systems developed like that is hard.
A similar concept is the maintenance of physical world machinery, it's also often overlooked, but a huge industry even compared to manufacturing the machinery it self
Alot of popular languages make it really easy to write code, but hard to maintain it .
It's one of the reasons I like Rust so much,even for relatively high level code. It has one of the srictest type systems and std/library ecosystems of any language in existence, which tends to make code very robust and easy to refactor.
I often wish for a language that has a similarly strict type type/ecosystem but is higher level. Swift is quite close, but very Mac OS centric. Haskell has exceptions, which are often used for error handling, making code less robust (plus the ecosystem is small). Other languages like Ada/Spark or Idris are way too fringe...
>Alot of popular languages make it really easy to write code, but hard to maintain it .
The language you're writing in has next to no meaningful impact on long term maintenance. When you're trying to do software archeology on why a function even exists the types it has will not help you figure out it's there because between 1996 and 2002 there was a tax rebate on capex for rocket development.
You need human understandable documentation, not fancy language features. This is no where near as sexy so no one does documentation and we're in the middle of a digital dark age.
I disagree. Language does matter for long term maintenance. For instance with a tax rebate.
Static typing - a fancy language feature - can immediately tell you the function deals with money. +1 for maintenance. Static typing with a solid IDE will immediately reveal everywhere the function is used. +1.
Languages shepherd you into certain designs or mistakes by making some things easier than others, such as when class inheritance can be easily abused in OO languages, cryptic one liners are easy in Perl, point-free cleverness in Haskell, monkey patching in dynamic languages, etc. +/-1 depending.
Per documentation, languages come with tools and best practices: Javadoc, Godoc, Rustdoc; C# even has a dedicated comment syntax for documentation - a fancy language feature. +1. No one does documentation, until the language tooling forces them to.
That said, BG (Before Git) and widespread use of source control, as in '96 to '02, was a dark age. We know more now.
> The language you're writing in has next to no meaningful impact on long term maintenance.
I disagree.
If you abandon a large block of code for a year, then come back to it, depending on the language, it may be difficult to grok what the code was doing and how it worked.
If it's Java, the tendency to create 100 levels of abstraction make it hard to reason about. If it's Python without type annotations, then your IDE may not be able to determine the types of variables, so it's difficult to determine what can be done with an object.
If it's Perl, then God help you. May He have mercy on your soul.
My 2 cents: There are thousands of projects trapped into formatting hell because nobody wants to touch git history. We need a standard to fix formatting programming without messing up history.
Some code formatters such as Spotless (https://github.com/diffplug/spotless/tree/main/plugin-gradle...) allow you to format code only in files that have changes against some designated branch such as `master`. So, you check out your feature branch, make changes, do some commits, and run spotless. Only the files which have some changes between your workspace and the master branch will be formatted. This allows you to gradually format the project as and when files would be changed anyways.
Until these ideas go into mainstream, treesitter has resulted in some interesting improvements to e.g Neovim. I use it extensively for AST based selection/navigation and along with conceal I can hide/transform a lot of noise characters in text source code.
I've thought about this issue so very many times, and I feel that until someone creates a system capable of understanding the underlying purpose of the code, we will not be able to automate this stuff away. I think the issue is that all of the things the author wants a breakthrough for is exactly what makes programming hard. We don't have systems intelligent enough to reason about a contract between two systems so as to create a general-purpose solution for things like glue code. The subtle differences that are there and that require us to do this work over and over again with small variations are precisely why programming requires intelligence.
Part of the "structured programming" revolution was that the compilers said "any color you want, as long as it's black": they were general-purpose glue code generators by declaring everything to use the standard contract.
(later, optimizing, compilers would spend a lot of effort to reason about different contracts, and in recent times, "undefined behavior" would come to surprise people who were used to the rather laxer approach to contracts in earlier days, but these are epicycles.)
There will always be work doing things over and over again with small variations, but I for one am glad to no longer be making variations on decisions like "will this parameter be coming in a register (which?) or on the stack (and if so, with what alignment)?"
Charles Simyoni's Intentional Software attempted systems "capable of understanding the underlying purpose" targeted at both programmers and business people. It was (hidden) parameterization all the way down. They invested several engineer decades and got no where. Microsoft bought them out for their patents.
We can't even automate testing of code (no matter how much "test automation" you have, you have manual testers too). That's orders of magnitude easier than automating the creation of the code in the first place.
A paradigm that Dave Farley has mentioned could be on the same level as OOP or structured programming is a completely asynchronous programming language. Every operation would have no garunteee of executing immediately. Such a thing may unlock more performance for cheaper on modern multicore cpus.
When we tried async-everything in the 1980's we discovered that there is such a thing as too-fine grained parallelism: the sweet spot (problem dependent!) is where communication and computation are balanced.
mostly just a glimpse of that talk, perhaps better suited is the one about programming he gave at Dropbox. In my opinion quite more in-line with what's being said.
> It feels like there's an unexplored dimension of abstraction here. Something where generics, interfaces and higher order functions are too static and low level primitives. Something where it's really difficult to pinpoint what exactly is the repeated pattern here and how to exploit it. Instead of generic framework that can do everything, I'd like an efficient way do something specific.
This bit reminds me a lot of a concept Aaron Hsu mentions in his fantastic talk on anti-patterns in the Iversonian languages[0], a paradigm he terms "Idioms over Libraries". The idea being that because of the expressive power of APL's primitives programmers often opt for utilizing easily recalled snippets of symbols over the importing of library code to implement common functionality. This facet of array-lang culture can be seen with sites like APL Cart[1], which serve as indexes of these frequently used snippets.
I think a lot of people miss that this is one of the major advantages induced by the terseness of these languages (it's not just about code-golfing for the implicit joy of laconicism). By allowing for the direct expression of algorithms in the same context they are being used, APL and its kin enable one to tweak and optimize these idioms on a case-by-case basis: precisely the kind of specificity the author of this article is asking for.
1. Less code. Operating systems have become obscenely bloated and restrictive. There is an enormous amount of power in the hardware we run our stuff on these days but you can't even touch it as a programmer anymore.
2. Powerful proof assistants and code synthesis. I love low-level programming but even the most hardcore, brilliant people I've met can easily introduce show-stopping security flaws into software without realizing it. It turns out that programming is hard and without rigorous models and proofs humans are really poor at understanding programs. We naturally hand-wave away details which get us most of the way there but these are discrete systems and details matter a lot.
As for testing... I agree, unit tests are often insufficient but probably the lowest bar you could convince typical industry software developers to hurdle to prove the correctness of their programs. Even then, as the author notes, they will come kicking and screaming with their opinions. There are far better options and strategies but as soon as you mention property testing or formalization you'll lose about 90% of the room so you have to pick your battles.
The trouble is that most startups are motivated to accidentally stumble upon a market niche and in this day and age that often means, "something with computers." They're not in it to write reliable, robust software that is efficient and performs well on a target platform. They're writing code that they can cash out on and never see again. Very different world.
The final breakthrough we need is to fund research labs again. Technology isn't inevitable and civilizations in the past have lost the ability to build and maintain it. We shouldn't let that happen again if we can help it.
> To be more specific - for most web projects I work on, I have a very similar yaml file for CI, Dockerfile,
Very similar, but not actually the same - of course there's always a tradeoff between control and convenience. Maybe you don't like where kubernetes, $CI_TOOL_OF_CHOICE, etc puts that tradeoff, but that's why there's a zillion tools and languages out there, because someone wanted a slightly different tradeoff
> Changing the program is the most common thing we need to do in our work, it's the reason our job even exist. And yet, most of programmer's time is spent reading or planning how to change the code.
I think I fundamentally disagree with the idea that "reading or planning" are second-class activities here. I think that one of the highest purposes of a programming language is is to communicate to other humans, unambiguously, what the program is intended to do.
I think most languages have a sweet spot for "good at communicating for $TYPE_OF_PROBLEM" - it'd be a nightmare to read the assembly code for say a AAA videogame and try to figure out if it's a FPS or RPG game, but if the intent of a program is to do a small thing in a super-architecture-specific way, probably assembly is a great choice - exactly because some other human will see _exactly_ what you mean.
If I have to work in a big complex codebase, it's great to use a very structured language like java/rust/etc where all the many, many pieces are (or at least can be) clearly named. I'll have no idea which CPU registers are twiddled when, but I don't care, and neither did the author of that code (presumably why they chose a high-level language).
On the extreme end - a tool like excel which is AWESOME for getting computations done and presenting the results, but is (IMO) not the best for communicating _how_ to do those computations. Do every single one of the cells in this column have the same formula? If not why? Is that intentional?
I thing programming languages are awesome exactly because of the extreme, explicit precision and repeatability that they allow us to communicate with, but certainly not all problems [that are currently solved with programming languages] require that kind of rigor.
Agree with most of the points in the article. I’ve tried to attack this party by generating code and leaving all other maintenance to developer without any framework constraints: https://github.com/sashabaranov/pike
I think, the problem is: the production of languages (and libraries especially) is way too much market-driven, there is too little cooperation, too little time spent to polish and to model at least some major use cases.
Also the quality of education is too poor, too focused on time-to-market - it makes harder for non-seniors to understand e.g. complex type systems (which are required for more usable languages anyway).
A language is released too prematurely -> some companies start to use it -> design problems become clear, but the language is in a compatibility trap at this stage -> e.g. it becomes a swamp or it slowly becomes a monstrosity of design afterthoughts and crutches.
It probably even wouldn't require much more spending from corporations, only some initiative. They anyway already invest considerably in language&toolchain development, but often with wasteful goals.
FOAM is a modelling framework that generates cross-language boilerplate for you, but it takes a much broader view of what constitutes boilerplate than most systems. Typically, it can generate between 95-98% of a working cross-language cross-tier system.
FOAM helps you create features for modelled data. Features include things like a Java/Javascript/Swift classes to hold your modelled data, code to marshall to/from JSON/XML/CSV/etc., various GUI Views, and support for storing your data in various databases or file formats. However, FOAM models are themselves modelled, meaning they're afforded all of the above benefits as well. This lets you apply the MVC technique of having multiple views work against the same underlying data-model concurrently (say a grid and a pie-chart in a spreadsheet), so that you can choose the best view or views for your current need. When treated this way, your code is no longer text (but it can be, if that's one of your views), and you can easily view and store it in many different ways and more easily programmatically manipulate it.
There's a touch of "let them eat cake" in the author's expectation that these difficult problems could be easily solved, if someone just put a little thought into it.
This is perhaps most clearly revealed here: "Changing the program is the most common thing we need to do in our work, it's the reason our job even exist. And yet, most of programmer's time is spent reading or planning how to change the code." Combined with the disdain for testing those changes, it seems the author wants a magic programming wand.
This. The author lost me there, I just scrolled past the rest to the conclusion.
This guys seems to believe produtivity equals amount of information produced, and fails to grasp the basics of creative work, which is exactly thinking.
507 comments
[ 1.2 ms ] story [ 105 ms ] threadI disagree with the text criticism as well. The reason we spend time thinking and reasoning about code, is not because its text format. Its because that's the job. We aren't scribes just copying books. The hard part of programming is figuring out what to do, not writing it out. Well i am sure better visualizations or something could offer minor improvements to comprehending code bases, i doubt it will change things significantly. We will always spend most of our time trying to figure out what we need to do not doing it.
The representation as text files is also part of what's causing the boilerplate.
As an example: How would you reduce the devops and configuration boilerplate in a monorepo?
In your typical JS microservice, a lot of code is not actually JS but yaml, json, terraform, etc. And it is very hard to abstract these away since a lot of tools rely on the existence of actual files.
Of course you can use code generation and other macros to manage the units in your monorepo (nx.js does this). But this is very instable since you might need to tweak some file resulting in your macros not working any more.
My take would be that textual representation is a local maximum. And our needs have outgrown this maximum. We're spending 90% of our time to innovate to push the boulder up the remaining 10% of this very hill. But there are most likely other hills with a much higher peak.
This is a problem of your own making, though. If you don't do a microservice architecture in JS and do a modularized monolith in Go, as an example, you don't have to write any of this code.
Boiler plate usually results from people using something that was not originally designed to do that thing or not designed that well. It's indicative of some kind of design friction or feature creep. When it works, people just do more of it without really thinking about what they are doing or why. If you see people copy pasting the same blobs of code over and over again, that's a good sign something is wrong.
At some point that becomes the way things are done and people start nit picking each other about doing it properly and then inevitably somebody comes along and does a thing that is vastly simpler and accomplishes the same thing. Happens over and over again. Some people then inevitably resist that new way of doing things because they are really invested in the old way. But the way our industry works is kind of Darwinist; so those things tend to die out quickly once something better comes along.
The two mistakes people in this industry make over and over again is assuming that 1) they know it all and 2) things don't change. Because things actually do change, and usually for good reasons, the former requires work to stay true. Some skills last longer than others and not all change is great. If you are just coasting and writing the same stupid code over and over again like it's ground hog day, it's going to eventually run out on you.
I think that the future of programming is more sophisticated static analysis. Programmers will write statements like, "every code path that writes to the Payments database must have called validate_user()." Then, the tooling will confirm that rule with every commit.
We kind of have this already (for example, Facebook's Infer tool [0]), but I think it will become much more important in the coming decade.
0: https://github.com/facebook/infer
https://youtu.be/OyfBQmvr2Hc
It’s exploiting logic programming to query for a program that satisfies some condition.
I'd agree with you, but not all of the business logic like that is going to live in code in the future.
In mobile there are multiple tools that you can use to prototype UI behavior. They allow you to specify navigation, transitions, clickable areas, behaviors when clicked, swipe actions, etc. They're this way so that the UI behaviors can be seen before coded (to an extent - not every prototyping tool does everything).
For code, well, only formal specifications get to that level. I haven't actually seen one of those outside of software engineering books, but I presume they exist in more critical fields. Commercial software doesn't really use them because they're too much work.
It's not just a UI prototype; you build a model of the underlying state of the system as you transition from screen to screen.
0: https://www.hillelwayne.com/formally-specifying-uis/
1: https://sketch.systems/
https://en.wikipedia.org/wiki/Fifth-generation_programming_l...
The move from testing to static analysis is happening there, in the expressiveness of the type system, and its inferences. Functional languages are on the forefront of experimentation with new type systems.
Are you claiming that static analysis of code can replace the concept of writing tests completely? If so, I'm very skeptical.
But if we are talking about unit test on the level of functions, yes, I think they can be replaced.
- glue code: given the outputs, you can reconstruct the inputs. this code translates between formats, between systems, etc.
- parsley code: given the outputs, you could reconstruct the inputs. this code "gussies up" its inputs: it takes table rows to JSON, or JSON to framework objects, PDF, etc. so in principle the original data is still there, but once the outputs are sufficiently complex the inputs become extremely awkward to reconstitute.
- crunch code: given the outputs, you have no hope of reconstructing the inputs. this code does real computation, and forgets stuff. Example: You can get a source program given only an executable, but you won't get the source that it was compiled from (nor will you, without a lot of work, even get a source you're willing to read)
Testing is hard because the requirement for correctness is often absolute and inflexible. When software is allowed to be a little incorrect, then yeah testing gets much easier.
We need more "Excels." More and better tools that let "regular" people program.
Even if you can equate glue code with trash, we seem to have unlimited space or a black hole to throw that trash.
[0] http://www.eusprig.org/horror-stories.htm
[1] https://www.wired.com/2014/10/a-spreadsheet-way-of-knowledge...
I'm all into PowerApps and all that, but no-code for anything the least bit sizable or complex is a fantasy from citizen developers.
...because, although I could argue about those points, it's fundamentally unrelated to the OP, and issue that programming is hard, and, for a very long time, no one has really had any idea how to solve it.
Now, however, AI generated code (like copilot) is, for the first time in a long time, a potential avenue to actually change how software is created at all levels.
I think there's pretty interesting, because it opens up a lot of new opportunities.
Is the future dynamic languages / high level specifications that are AI-transformed into typed verbose languages like C/Rust/whatever and then compiled?
The ancient tools like Rational Rose tried to do this, but the tech was never actually technically good enough. Maybe... we'll see a real change in this space as the sophistication of the models improves... or maybe, like self-driving cars, it's always going to be 'nearly good enough'.
Hard to say.
...but, hot damn. More excels? Please no. Excel already exists. Don't rebuild that stuff again. Build new, different interesting tools please.
I don't think they literally meant another Excel. Just something that is easily accessible and usable by non-programmers to do very 'programmy' things, etc.
Agreed.
>But excel proficiency acquisition is very similar to a programming proficiency acquisition
Strong disagree. Maybe if you only consider those extreme cases you referenced (ray-tracing, heavy VBA, etc.), sure. But in general? No shot.
Excel gives immediate feedback with a visual interface. There's no need to learn about variables or syntax or memory or compiling. There's no dependencies and no package managers. It abstracts nearly everything away. Click the place you want the thing to go, type the thing. Want a chart? Click the picture of the chart you like and click and drag over what you want in the chart. Want to change chart colors? Click the color you want. Want to change how the data is displayed visually? Press the bold button, or the color button, or the border button, whatever your heart desires is a click away with the same interface your used to with Word.
A great comparison is a geographic heatmap. In Excel, you need 2 columns. One with some States/Provinces and the other with some numbers. Then you click the big map button and instantly have your map. I can write the instructions for it in the space of a napkin and someone who has never touched Excel can be making all the maps they want. Now think about how you would explain making the same geographic heatmap in Python (or whatever your 'easy' language of choice is) to someone who has never learned any programming concepts.
They literally start using it as a way to lay out tables (usually of data). Then they make graphs. And add more tables. And learn how to use it like a basic pocket calculator (half of the formula I see that should be sums are actually =B21+B22+B23+B24+B25…) Then comes SUM. Then VLOOOKUP. Perhaps some IFs or COUNTIFs.
This process can take years for someone to go from first use to anything even slightly programming-like, if you squint at it.
A tiny fraction of these people ever do anything complex enough it looks even much like low-code or no-code "development". Even less write macros.
But you know what? In the first half hour, when all they wanted was to email round a list of team members and their lunch preferences, they already achieved something with Excel that they found useful, and they felt productive ever since. Most Excel users spend almost NO time learning the tool compared to how much time they spend getting (what they see as, and what probably is) value out of it, even when we are sat in sheer terror as we witness the horror they have unleashed on the world.
That is the bar you have to clear to make a "better Excel".
How is that different from the present? Why are these "AI-transformations" different from what a compiler can do?
You’ve seen copilot right? (https://github.com/features/copilot)
It’s fundamentally more sophisticated. This is like asking what is the difference between modern ML translation and the previous 20 years of research on language translation; the former actually works.
The latter basically doesn’t except in very specific circumstances.
Compilers can turn language into instructions only in a limited extremely specific set of circumstances.
But in the end, we all talk about leaky abstractions, and the stark horror is that the fact that all of these wonders run on a computer, the worlds leakiest abstraction, the worlds most stubborn, pig headed, cantankerous contraption out there.
And all the lipstick in the world can't hide it.
A lot of my colleagues are hybrid of traders + programmers. They do a lot of Excel. Sure, we can automate text programming, let everyone do Excel, and us problem solvers still won’t find difficulties finding high paying job.
The main problem is that MOST "programming" languages/environments are not made for business apps, (or for other niches, btw).
Python? Ruby? Julia? Delphi? all fail to work to make(or solve even small cases of) business apps.
By a long mile.
-----
What Excel solves, is have a barebones, half-working, unscalable, ill-suited... BUT MADE FOR help people get a "solution" for this.
Things that are more like this are Django, Jupyter notebooks, a little of Julia, FoxPro. But still are more pro-developer than pro-power-user.
I'd say the great weakness of No Code solutions isn't so much they are bad, but that they have no FOSS support. At all.
However, it is still taught in IT classes as a useful tool for small businesses, and it is.
If MS prioritised working on it it may have helped lift some people out of Excel and into DBA-lite territory but not in its current state.
So if the data is coming from Excel, and being extracted back to Excel, why use anything besides Excel?
The only reason why more people don't program is because Windows is antithetical to programming _anything_. The second you remove people from a windows environment is the moment they start coding, even by accident.
There's one catch with Excel, the more complicated your spreadsheet gets, the closer it gets to programming. And at some point, it's more cost effective to write in a proper programming language than continue to maintain mess.
However, until this complexity level, Excel is great, and empowers a lot of people who barely know the basic features of Excel to solve their computational problems.
https://www.microsoft.com/en-us/research/podcast/advancing-e...
User defined functions:
https://www.microsoft.com/en-us/research/blog/lambda-the-ult...
Then when the software designer finds that the software does something wrong, he can simply add assertions. Its a little like TDD but without coding.
For GUI applications it would be cool to define assertions via natural language and the programming system drawing pictures to explain what it understood.
Main reason to make the distinction is that most systems that parade themselves as "low-code" today are IME indistinguishable to high-code development to your average joe.
Even taking this example, the level of value totally depends on the degree of abstraction you can trust the computer to perform for you. There is a tremendous amount of nuance to human language and while we've come a very long way in that field of study, it's still a hard nut to crack.
I agree with you: assertions in natural language would be a killer app.
[0]: https://en.wikipedia.org/wiki/Declarative_programming
Isn't this just a programming, but harder and worse?
We've invented programming languages to be less ambiguous about what we want done, that's a feature and not a bug. It allows us to avoid the struggle of lawmakers (who are programming in spoken language) and lets us tell the computer exactly and unambiguously what to do.
Natural language does not.
"Bang on the floor" may mean to slam the floor or to have sex on the floor.
There's a programming joke that goes like a programmer's partner says "Go to the store and buy a loaf of bread, and if they have eggs, buy twelve". The programmer comes back with twelve loaves of bread.
Time flies like an arrow, fruit flies like a banana.
When I say "what" but not "how", I am trusting the system to be reasonable in the "how" it generates. But "reasonable" depends on my situation. Unless the system knows and understands that, it may easily do what I said, but do something that is anywhere from non-optimal to disastrous.
And if I have to say enough to prevent that - to prevent all the ways that could happen - is that more efficient, or less, compared to just writing the code myself?
Regular users are allowed to edit their own books.
Editors are allowed to edit any books with the same account id.
The public can read about any book that is not in draft status.
I need a domain model where users can write books with the following properties: userId, accountId, title, author, status.
status is an enum supporting draft, published, no longer available
I know this seems like a completely different paradigm, but I think we were wrong about General AI needed to replace programmers - we just didn't know how "normal AI" would go without being General AI.
I can also see this extended into the front-end at least for rudimentary forms. I can imaging someone doing a transfer AI to make front-ends have a consistent and nice looking feel that can change pretty easily - perhaps to match the user making the request.
So, the hard part of software is figuring out what you actually need done.
You still need the person writing the system requirements to be precise, accurate and not ambiguous, or you need General AI to figure out if the supplied system requirements meets those requirements.
And when things inevitably goes wrong, or requirements change as they always do, "you just" have to review and rephrase your entire requirement list, or you need General AI to do that for you.
My day becomes selecting libraries, integrating them, and testing integration. My business code is < 20% of our entire codebase. And I’m much more productive in 2022 than in 1992.
It feels anything but productive.
Robust, comprehensive libraries from reliable vendors can provide a lot of value and are slow to rot. These can be anticipated and added early so that they’re made good use of and can become the first tool to reach for before adding other dependencies. Think React, lodash, QT, boost, etc.
Meanwhile, I acknowledge that most other packages and repos are of far more dubious quality than anything my team would write, have no accountability to my team or stakeholders, receive few/no code reviews when added or updated, introduce conflicting style/semantic conventions, and generally expand the surface area for bugs and vulnerabilities by including many lines of code that have no relevance to the project.
There are no strict rules, but these are the sort of considerations that weigh in.
- Age of the company
- Complexity of integration with the vendor
- Stability in early adoption
If you adopt the tech too early, you risk instability during a time where you don't have the size as a business to afford that. Nobody that's a customer of Google is going to leave their contract because it was down for an hour. However your new 20 employee startup goes down for an hour? And you just signed your largest customer yet? And it's because you decided to early adopt some tech because it simply "looks good and efficient?". No thanks, that customer will be looking at alternatives when renewal time comes, if they don't just pay out the contract and leave on you already. And good luck getting funding with that blemish that you blew it by adopting cool new trends.
Okay, so being early adopter at a startup is risky. So maybe you go with something that is "kind of" new, and is being used by other large companies. Great, so you start building, and the feature system grows, and the integration grows, and now some new system comes along that is way better. Think about life before React/AngularJS and people who built systems with plain jQuery. There was a time where that was the hottest thing and nothing like React/Angular even existed. So you pick that and then 3 years later you have piles of jQuery and AngularJS/React/whatever else comes out, but remember the maturity thing mentioned above. So now you watch it grow over 2 years and now you have 5 years worth of jQuery and the amount of work it will take to switch to React/Angular is a year long project. You can't just stop everything else to work on this, so it's 20% of your time, nevermind that it's a moving target as you fix bugs/release new things in jQuery over that year. So now you're maintaining both for a year, and then 3 years later it becomes clear that React is the winner over Angular, and you picked the wrong one! At the time you picked they both seemed pretty mature (Google w/Angular and FB with React). Oh jeeze, do we migrate everything again? And from the business perspective how do you even begin to sell this to the people in charge when it results in no benefits to the end user. And by the time you finish it will these even be the tech you should be using? Or will something else rise up to replace that?
The complexity grows with the integration as the age of the company increases. And adding headcount and expecting to solve the problem just introduces more problems. Now your team size is 10 instead of 5 and standup is taking an hour everyday, do you break it up into other teams? What are those teams called? What if nobody wants to work on the upkeep of the legacy? Why would they as an IC? So now how do you split the work up fairly?
I'm not saying "don't use vendors", "don't adopt early" or "don't vet your vendors", but what I'm saying is the problem isn't the tech. It's the fact that the "obvious correct solution" is a changing answer over time. And the integration and added complexity combined with team dynamics eventually becomes the hardest problem a company has to solve.
I did a gig at a place that brought me (along with like 50 other people) in to transition from a server rendered PHP app to a React front-end. They'd bootstrapped to something like $300M and had just brought in a new CEO who wanted to 10x the company.
It was fucking pandemonium. Every day I saw another message on their slack along the lines of "It's my last day, I had a lot of fun working with you all the last 8 years". My team was completely useless, as far as I could tell our mandate was to basically be nazis and tell all the supposed mouthbreathing pre-historic over the hill PHP greybeards how to write "good" React code (aka unmaintainable garbage).
I have no idea how they're doing now. I don't see them mentioned much anymore. They're probably worth $30M.
If you had to bet on who could 10x your company, would you go with the guys on the ground floor that have literally bootstrapped a company to $300M off the back of their PHP tech? Or would you go with the army of hip Javascript consultants, half of them off shore?
I say this as a specialised React developer, who has been doing solely React for 2015.
There's absolutely no reason why you have to roll with the industry to every new fad. I have no intention of going anywhere when the next thing comes along. In fact, I'm waiting for all the noobs to fuck the hell off so I'm free to actually write good React code again. I bet that's probably how the PHP developers felt back when that was the fad of the decade.
If that company had backed their PHP developers in and given them enough shares to stick around, I bet they'd probably be a $3B company today. 90% of that is going to come down to the business side of things, not whether you throw away your entire engineering culture to score 20% better on some fucking render metric on a page transition or whatever.
Hah! In all my years the biggest, most difficult vulnerabilities/problems to remediate all came from vendor-made (90% of the time closed source) software. With the free, open source stuff the problems are usually very minor but even if they're BIG we'll pretty much always get fixes and deploy them faster than if we had to wait on a vendor.
The more popular the OSS the safer it is but even niche OSS tools/libraries are easier to work with by definition because we can look at and modify the source to correct the problem even if the original maintainer hasn't gotten around to it yet. This is actually a big reason why Python is preferred where I work: The code is inherently visible and fixable.
Funny, being an old, cranky, gray-no-beard, if I wanted to spend my days selecting libraries, integrating them, and testing integration, I would have become a digital electronics engineer.
Your point is well taken, it's just our modern truth. It's just not was attracted me to this world in the first place.
After thousands of years, humans still use double accounting in finance. This is because it's effective. Unsurprisingly, it's tightly coupled to the original transaction recording.
The constant assertion that tight coupling is bad, because it creates more work which can be error prone. This is a fundamental problem with how people discuss software testing. It's not a panacea, but a guardrail. Guardrails aren't indestructible. That isn't the point. Developers have an obsession with making things as easy as possible (be that simplifying or codifying) and this is not a domain where that makes for a better solution, all other qualities being equal.
The problem is just with effectivity - it'd be fine if the time spent on test would be the same as time on code, but very often the effort on test is much bigger while not even covering all important cases. I think we need to look for methods to get a good case coverage without spending majority of our time writing tests.
Writing software is the process of explaining a set of conditions and conversions from input to output. We should assume the programmer does this in as short code as possible (unless we are using a low level language like C which adds a lot of accidental complexity like memory management). To repeat all these conditions in test-code, would be to write the same thing twice, in a different way. Not saying this is wrong, it's just important to acknowledge it.
Testing often becomes more like sampling, with hard coded or generated known set of inputs/outputs, often taken from a scenario derived from a common user journey. This verifies that this particular journey works, but it would never be a guarantee that other edge cases of it does, to do that you do end up in the write-twice situation.
This is why I hate modern programming.
When was the last time you had a job where you could actually learn how to actually program something? The prolifiration of libraries means that all your bosses and managers would extremely frown upon anyone handrolling a solution instead of using an existing library.
Which means the more time you spend at a regular job, the worse your brain will rot.
Just talk to any programmer at any company and you will notice that almost none of them know how to program anything. They just know how to stitch together libraries with some glue code.
They are right, even if you do not agree. Handrolling is incredibly cost ineffective, yet another source of bugs and can be seen as a bad idea when it comes to security.
Huge third party libraries like LibOgg are maintained by gigantic industry players. Chances are, they are more knowledgeable in that specific domain than you are and they can focus more developer time on what is just a small part in your business application.
This might have been true 20 years ago. But now the cultural shift affects even the people who work on the lower levels.
A sort of famous example is how slow Visual Studio has gotten compared to 20 years ago (when it was running on slower hardware even!) where it was nearly instant.
The refterm saga has demonstrated that people employed by Microsoft and paid very high salaries to work on a terminal emulator have no idea how to make a half decent terminal renderer.
> handrolling is incredibly cost ineffective, yet another source of bugs and can be seen as a bad idea when it comes to security.
Without context this statement is blatantly false.
Many libraries are full of bugs and security problems.
Once upon a time, a lot of microprocessor code was written in assembly. Then C came along, and many of the assembly people said: "whoah, hold on, I use different calling conventions in different places for good reason; what do you mean every C function is going to use a whole stack frame even if it's never re-entered?" But now 99+% of programmers don't care about sub-kb stack frames, and probably most of them even consider C too low-level[0] to touch.
I've said elsewhere[1] that much of the ceremony of cloud sw dev reminds me of 1960's mainframes — so what is the equivalent of a "structured programming" toolchain for the cloud, that takes some higher level description of inputs, outputs, and logic[2], and then generates not-entirely-ludicrous boilerplate to mash it all together?
[0] in hindsight, C's big advantage back in the day was that it mixed much better with assembly than purer HLLs.
[1] https://news.ycombinator.com/item?id=32467919
[2] compare the "environment division" of COBOL with Terraform, and the "data division" with Swagger. I mean, we've definitely progressed on many fronts since then, but certainly not on the axis of "lack of verbosity"[3].
[3] OTOH, Conway's law suggests that multiple-team projects will always wind up with multiple configuration in multiple places.
The machine I learned C with maxed out at 64K data. Needless to say, the compiler was not doing much beyond peephole optimization.
Of course, none of that means that the entire endeavor was worthless before that. It just means that whatever improvement you create, it won't appease to a lot of developers, and it's ok. One can only get unanimity if the change has no drawbacks at all, under no context.
You want to do the hard stuff? Maintain existing code you're not familiar with.
The problems around maintaining unfamiliar code are huge, largely unsolved, expensive and risky. There's a little branch of computer science called Program Comprehension and no one pays any attention. Though most programming money is spent on maintenance.
Few dependencies? Too much reinventing the wheel, delete all that code and add dependencies!
There is no perfect code, can always make different trade-offs and move things around.
Even magic stuff like `@attributes` is easily searchable.
Maintaining and cleaning up a mess someone left behind are two different things.
The idea being some kind of a runtime for assembly code which does not actually execute the program but allows one to understand it in different sematinc levels, or dunno.. this is a very raw idea. needs a lot of work (and a lot more knowldedge) to set down.
too bad none of the professors that I was able to meet were really interested in this kind of thing
I have thought of an idea about 6 months ago that has been fermenting in my mind since then : what if (e.g.) a Python VM had a mode where it records all type info of all identifiers as it executed the code and persisted this info into a standard format, later when you open a .py file in an IDE, all type info of the objects defined or named in the file are pulled from the persistent record and crunched by the IDE and used to present a top-notch dev experience?
The traditional achilles's heel of static analysis and type systems is Turing Completeness, traditional answers range from trying and giving up (Java's Object or Kotlin's Any?), not bothering to try in the first place (Ruby, Python, etc...), very cleverly restricting what you can say so that you never or rarely run into the embarrassing problems (Haskell, Rust,...), and whatever the fuck C++'s type system is. The type-profiling approach suggests another answer entirely : what if we just execute the damn thing without regard for types, like we already do now for Python and the like, but record everything that happens so that later static analysis can do a whole ton of things it can't do from program text alone. You can have Turing-Complete types that way, you just can't have them immediately (as soon as you write the code) or completely (as there are always execution paths that aren't visited, which can change types of things you think you know, e.g. x = 1 ; if VERY_SPECIFIC_RARE_CONDITION : x = "Hello" ).
You can have incredibly specific and fine-grained types, like "Dict[String->int] WHERE 'foo' in Dict and Dict['bar'] == 42", which is peculiar subset of all string-int dictionaries that satisfy the WHERE clause. All of this would be "profiled" automatically from the runtime, you're already executing the code for free anyway. Essentially, type- checking and inference becomes a never-halting computation amortized over all executions of a program, producing incremental results along the way.
I have ahead of me some opportunity to at least have a go at this idea, but I'm not completely free to pursue it (others can veto the whole thing) and I'm not sure I have all the angles or the prerequisite knowledge necessary to dive in and make something that matters. If anyone of the good folks at JetBrains or VisualStudio or similar orgs are reading this : please steal this idea and make it far better than I can, or at least pass it to others if you don't have the time.
The problem is this dilemma: If you have to wait for a "real" execution of a program, then very reasonable expectations like "I can see a local variable I just declared" doesn't work. If you try to fake-execute a program, you have problems like trying to figure out what to do with side-effecting calls, loops, and other control flow problems.
Trying to reconcile a previous type snapshot with an arbitrary set of program edits was tried by an early version of TypeScript and wholly abandoned because it's extremely difficult to get right, and any wrongness quickly propagates and corrupts the entire state. The flow team is still trying this approach and is having a very hard time with it, from what I can tell.
> you're already executing the code for free anyway
Based on my experience of working on similar domain of type systems for Ruby (though not the exact approach you describe), this turns out to be the ultimate bottleneck. If you are instrumenting everything, the code execution is very slow. A practical approach here is to abstract values in the interpreter (like represent all whole numbers are Int). However, this would eliminate the specific cases where you can track "Dict[String->int] WHERE 'foo' in Dict and Dict['bar'] == 42". You could get some mileage out of singleton types, but there are still limitations on running arbitrary queries: how do you record a profile and run queries on open file or network handles later? How do you reconcile side effects between two program execution profiles? It is a tradeoff between how much information can you record in a profile vs cost of recording.
There is definitely some scope here that can be undertaken with longer term studies that I have not seen yet. Does recording type information (or other facts from profiling) over the longer term enough to cover all paths through the program? If so, as this discussion is about maintaining code long term, does it help developers refactor and maintain code as a code base undergoes bitrot and then gets minor updates? There is a gap between industry who faces this problem but usually doesn't invest in such studies and academia who usually invests in such studies but doesn't have the same changing requirements as an industrial codebase.
How does the type profiler know that the variable that only contained values like "123" or "456" was handling identifiers, not numbers?
One of the insights that people making tools for dynamic languages discover over and over again is that most uses of dynamic features is highly static and constrained. In general, yes, a python program can just do eval(input("enter some python code to execute :) \n>>")), but people mostly don't do this. People use extremly dynamic and extremly flexible constructs and features in very constrained ways. This is like the observation that most utterances of human languages are highly constrained and highly specific, even within syntactically-valid and semantically-meaningful sentences, not all utterances are equally probable, and some are so improbable as to be essentially irrelevant. People who try to memorize the dictionary never learn the language, because the vast majority of the dictionary is useless and mostly unused, and even the words that are used are only used in a subset of their possible meanings.
>Once you open it up to arbitrary input, no single run (or even a large set of runs) can capture everything the program might be expected to handle.
Anything is better than nothing, right ? if your program keeps executing with some set of types over and over again (and it will, because no program is infinitely-generic, the human brains that wrote the code can't reason over infinity in general), wouldn't it be better to record this and make it avilable at static write-time ?
Human brains are finite, how do we reason over the "infinite" types that every Python program theoretically deals with ? We don't! like I said, most dynamic features are an illusion, there is a very finite set of uses that we have in mind for them. Here is an experiment you might try, the next time you write in a dynamic language, try to observe yourself thinking about the code. In the vast majority of cases, you will find that your brain already has a very specific type in mind for each variable (or else how can you do anything ? even printing the thing requires assuming it has a __repr__ method that doesn't fail.).
>How does the type profiler know that the variable that only contained values like "123" or "456" was handling identifiers, not numbers?
It doesn't. I think you misunderstood the idea a little, the type profiler makes no attempt whatsoever at discerning the "meaning" of the data pointed to by variables, it will only record that your variable held strings during runtime. If the number of string values the variable held was small enough, it might attempt to list them like this "Str WHERE Str in ["123","456"]". If the number of values the variable held was larger than some threshold but some predicate held for it consistently it can also use that, i.e. "Str WHERE is_numeric(Str)". If a string variable was always tested against a regex before every use, it will notice that and include the regex into the type. No additional "smart pants" than this is attempted, just the info your VM or interpreter already knows, just recorded instead of thrown away after each execution.
The profiler will not and can not attempt to understand any "meaning" behind the data nor it needs to in order to be useful, it's just a (dynamic) type system. No current type system, static or otherwise, attempts to say "'123' is a numeric, would you like to make it an int ?", that would be painful, absurd in most cases I can think of and misguided in general.
SQLite's column type affinity will in fact do this, if you tell SQLite that a column is an INTEGER it will turn '123' into 123 but will happily take a string like "widget" and just store it.
I also wanted to add that your thoughts on this subject are well-stated and align with some work I've been patiently chipping away at, in the intersection of gradual typing and unit testing. I'll have more to say on that subject someday...
I think you misunderstand my position. It's better for the creator to simply specify it when they write the code - i.e. static typing.
> the type profiler makes no attempt whatsoever at discerning the "meaning" of the data
Your examples are waaay beyond what I need or expect. What I need is for the program to recognize that, for a number, 123 and 456 are valid, but "abc" will never be. Conversely, for an identifier, no matter how many runs use values like 123, someday someone might provide the value "abc" and that's ok. Also, any code that attempts to sum up a collection of identifiers should not be runnable, even if the identifiers in question all happen to be numbers.
This is something that static typing provides, and no amount of profiling will ever be able to divine.
> In the vast majority of cases, you will find that your brain already has a very specific type in mind for each variable
Sure, for the code I write. But I've seen plenty of code written by juniors where, upon inspection, it was completely inscrutable whether a given parameter expects an integer, a string, a brick wall or a banana.
Which is all to say that my default position is unchanged: dynamic typing is unhelpful for anything larger than small, single-purpose scripts.
Additionally, some programs cannot be easily and/or quickly executed to cover all possible paths, so parts of the program will remain uncovered by the profiler. That is one place where static analysis becomes very powerful, because you can cover all the program much faster (in linear time making largish precision tradeoffs, but analysis still yielding a usable result).
source: I work on a flagship static analyzer.
My way of looking at this is just frustration at the misallocation of resources. Dynamic programs already have tons of typing information available, just not in the right time and place. Your brain spends an aweful lot of time "executing" the code of a dynamic language while it's writing the code, just like the language runtime only far more slowly and with a sky high probability of error. So all what I'm saying is, why this immense suffering, when the information that your brain is in dire need of is already right there in the interpreter's data structures, just in opaque and execution-temporary forms ? It strikes me as incredibly obvious to try to bring in those typing information from the runtime into persistent transparent records available at dev-time.
Abstract Interpretation or Symbolic Executation are (fancy) tools in programming language theory's toolbox that can help us, but the simplest possible thing to try first is to make use of the already-available information that are just lying around elsewhere. To make a somewhat forced analogy : if we have a food crisis at hand, we could try fancy solutions like genetic engineering of super crops or hydroponic farming, but the dumbest possible solution to try first would be to simply bring in food from other places where its plentiful. Typing info in dynamic languages is very plentiful, it's just not there when we need it.
This stanza was maybe not your primary point but it was beatifully written and multiple programmer friends of every variety mentioned have been cackling at it.
Urbit "infers" types from functions by executing them with type objects so the relevant bindings themselves return type objects.
If you can skip over both the syntax and the creator, there are some intesting things to be found in there.
[0] https://dl.acm.org/doi/pdf/10.1145/178243.178478
It is also the thinking we can see supporting a lot of the early devops movement.
while Domain Driven Design and Clean Architecture are a first step in the right direction, languages and frameworks are still limited in supporting these ideas.
It's almost insane to think that once you have a modeled domain you need to replicate them in resolvers, types, queries, etc (for graphql), resources, responses, endpoints (for rest), etc, etc.
to reinforce the point of the article and the one brought by @SKILNER, I believe that the great transformation that is to come is in maintainability through a focus on the domain
If you don't care about other people being able to read and explore your code without a lot of preperation, you can go full hog creating layers of DSLs and metaprogramming, and with enough dedication you can end up with all your real domain level business rules in one place separate from the "infrastructure"
But if you do this, you end up with a codebase that is hard for a newcomer to ask simple maintenance questions about like "What are all the places XYZ is called from?" or "What are all the places to write to ABC" etc
So an experienced developer learns to limit their metaprogramming so their code retains easy explorability at the expense of long term sustainability. Golang is kind of an epitome of this kind of thinking. Lisps are kind of the epitome of the opposite I guess.
This is what's behind the paradox where a good dev working alone or maybe with one very like minded person can produce a level of productivity you can't match again as you add developers, till you have way more developers. The dip represents the loss due to having to communicate a common understanding of the codebase, that doesn't get adequately compensated for till you've added a lot more people.
Wow I couldn't agree more with this point. You put it perfectly. I worked alone and later with one other developer on a project and it felt way more productive than my current team of 5 developers!
His response: wow, that's a lot of crap just to run a sql query.
He was fully right.
I find other people's code difficult to follow due to all the abstractions that I wouldn't have inserted.
When I write Clojure code for example, the code just works. I somewhat understand the code I write.
But when I read other people's Clojure, I find it difficult to understand. I think it's my maturity in understanding Clojure, the mental model isn't there yet. Which is funny because I've been on two projects where we used Clojure.
I don't have the same problem with less metaprogramming languages such as C, Java or Python.
Not to mention massively underestimated and unappreciated by non-programmers.
I'm not sure if we are respected for this capability. Do people want to pay for what they do not understand why it is so hard?
A fundamental problem with how we teach programming is that we focus on writing, instead of reading, software. To borrow terminology from the language arts; we don't focus enough on reading comprehension.
You first write the code, then you design it. Genius!
If the number one answer to we're using Agile but it's not working is "you're doing Agile wrong", then maybe Agile isn't the solution?
The way I worked at the place that did not have an explicit process was one of switching between agile and waterfall methods. Early on in the projects, the process was primarily waterfall, defining things up front, building prototypes, laying out the project, etc. Then once passed that stage, projects would enter into a more agile or iterative process. Then as new big features came in, back to waterfall and then switching to agile once that stabilized. This worked quite well.
Writing software well is difficult by itself and people are all different in understanding, experience and skill level. Throwing a group of people together and trying to build something that works for every scenario is a miracle given the complexity of software and the complexity of interpersonal relationships and organisations (see Conway's law). I'm impressed by every multiplatform language or tool or software. It's a lot of work!
If everybody did things the same way i.e the agile way and if agile was proven to work and everyone followed it the way it was intended and designed, then maybe we could all be interoperable and easily work together and produce projects that don't fail. That's the fantasy.
To be fair I've been 6 years worth of agile projects and I was never on a project that failed.
Another thing that rubs me wrong is the recurring notion that we need to get rid of the text as a representation of code. I've yet to meet a mathematician who wants to get completely rid of formulas because coming up with a proof is slow and cumbersome.
I understand that the incentives are drastically different between academia and business, but perhaps we've gone too far in this particular direction. Perhaps it's okay to admit that programming isn't as easy as we all want it to be and it's okay to take the time to change things carefully instead of moving fast and breaking stuff.
I completely agree with this notion. However, I feel like we're sorely missing out on some form of visual exploration. I feel like the majority of my time is spent trying to understand the flow of execution of a program I'm trying to maintain that was written by other teams that are long gone.
It would be so amazing to be able to "zoom out" from the text and get a graph view of execution flow through different files. And then being able to highlight a particular execution flow that you're studying would be great. I know we already have some flavors of this with "find all references", or something like Visual Studio's profiling tools that highlight flows of execution, but none of these have ever felt like they improve the exploration of the codebase very much.
It would be very interesting to see some tools that allow a more fluid exploration of an unknown codebase. More akin to zooming out of a Google map view and tracing flow from point to point, instead of diving headfirst into a million files looking for the correct information.
When your callstack is 50 methods deep and there's 20 layers involved and marshalling and it's difficult to see what is going on.
I want to mark two pieces of code and see the data structures passing through them, similar to a debugger but more like a log file or trace like Jaegar or Kibana. But the actual POJO or JSON objects themselves.
Sadly, they retired the entire project a while back.
I see what you're saying, if I understand correctly, in terms of the hype around "low or no code" environments, if that's what you mean. However, I do disagree with the notion of equating code with text. It seems myopic in the same fashion of equating tape or punch cards with code.
Also, I think the mathematics example isn't quite apt. Mathematicians are very willing to use visual representations to both illustrate and prove ideas. The way mathematicians work is actually a great example of moving between visual and symbolic representations and also prose.
In many significant ways, text is a highly limiting form of expression. In my opinion, we have nearly tapped out the available expression that can come from purely text-based programming languages. One example of this is the limitations of the innovations in syntax. There's been basically no significant innovation in this area aside from small iterative improvements, and I think that's a fundamental limit we've hit. I feel the future of programming is likely to hinge on a highly hybrid environment. In some sense, Smalltalk and its ilk, Emacs and Common Lisp, TouchDesigner, LabVIEW, and vvvv are precursors for what could come.
yes there are a few tools on the market, nothing really standout though, maybe it's for AI/ML to innovate in this field.
Linux kernel is well designed, in that I can add code relatively easy into its subsystems, it's those Object oriented or FP code base that bothers me the most, especially when they're large, often times they just made me feel hopeless, good tools desperately needed.
The hardest part of maintenance, in my experience, is that programs were developed under needless or incorrect constraints and then expected to be magically maintained. There is a huge downstream effect of decisions made early on in a software system's life.
The story of "just get it working" that evolves into "now that's it's working, don't change it but add these new features" repeats itself over, and over, and over. It's not surprising why maintenance in systems developed like that is hard.
Alot of popular languages make it really easy to write code, but hard to maintain it .
It's one of the reasons I like Rust so much,even for relatively high level code. It has one of the srictest type systems and std/library ecosystems of any language in existence, which tends to make code very robust and easy to refactor.
I often wish for a language that has a similarly strict type type/ecosystem but is higher level. Swift is quite close, but very Mac OS centric. Haskell has exceptions, which are often used for error handling, making code less robust (plus the ecosystem is small). Other languages like Ada/Spark or Idris are way too fringe...
The language you're writing in has next to no meaningful impact on long term maintenance. When you're trying to do software archeology on why a function even exists the types it has will not help you figure out it's there because between 1996 and 2002 there was a tax rebate on capex for rocket development.
You need human understandable documentation, not fancy language features. This is no where near as sexy so no one does documentation and we're in the middle of a digital dark age.
Static typing - a fancy language feature - can immediately tell you the function deals with money. +1 for maintenance. Static typing with a solid IDE will immediately reveal everywhere the function is used. +1.
Languages shepherd you into certain designs or mistakes by making some things easier than others, such as when class inheritance can be easily abused in OO languages, cryptic one liners are easy in Perl, point-free cleverness in Haskell, monkey patching in dynamic languages, etc. +/-1 depending.
Per documentation, languages come with tools and best practices: Javadoc, Godoc, Rustdoc; C# even has a dedicated comment syntax for documentation - a fancy language feature. +1. No one does documentation, until the language tooling forces them to.
That said, BG (Before Git) and widespread use of source control, as in '96 to '02, was a dark age. We know more now.
I disagree.
If you abandon a large block of code for a year, then come back to it, depending on the language, it may be difficult to grok what the code was doing and how it worked.
If it's Java, the tendency to create 100 levels of abstraction make it hard to reason about. If it's Python without type annotations, then your IDE may not be able to determine the types of variables, so it's difficult to determine what can be done with an object.
If it's Perl, then God help you. May He have mercy on your soul.
An example: https://www.moxio.com/blog/43/ignoring-bulk-change-commits-w...
https://www.unison-lang.org/learn/the-big-idea/#richer-codeb...
Until these ideas go into mainstream, treesitter has resulted in some interesting improvements to e.g Neovim. I use it extensively for AST based selection/navigation and along with conceal I can hide/transform a lot of noise characters in text source code.
(later, optimizing, compilers would spend a lot of effort to reason about different contracts, and in recent times, "undefined behavior" would come to surprise people who were used to the rather laxer approach to contracts in earlier days, but these are epicycles.)
There will always be work doing things over and over again with small variations, but I for one am glad to no longer be making variations on decisions like "will this parameter be coming in a register (which?) or on the stack (and if so, with what alignment)?"
We can't even automate testing of code (no matter how much "test automation" you have, you have manual testers too). That's orders of magnitude easier than automating the creation of the code in the first place.
[0] https://youtu.be/EGqwXt90ZqA
This bit reminds me a lot of a concept Aaron Hsu mentions in his fantastic talk on anti-patterns in the Iversonian languages[0], a paradigm he terms "Idioms over Libraries". The idea being that because of the expressive power of APL's primitives programmers often opt for utilizing easily recalled snippets of symbols over the importing of library code to implement common functionality. This facet of array-lang culture can be seen with sites like APL Cart[1], which serve as indexes of these frequently used snippets.
I think a lot of people miss that this is one of the major advantages induced by the terseness of these languages (it's not just about code-golfing for the implicit joy of laconicism). By allowing for the direct expression of algorithms in the same context they are being used, APL and its kin enable one to tweak and optimize these idioms on a case-by-case basis: precisely the kind of specificity the author of this article is asking for.
[0]: https://youtu.be/v7Mt0GYHU9A?t=1027
[1]: https://aplcart.info/
1. Less code. Operating systems have become obscenely bloated and restrictive. There is an enormous amount of power in the hardware we run our stuff on these days but you can't even touch it as a programmer anymore.
2. Powerful proof assistants and code synthesis. I love low-level programming but even the most hardcore, brilliant people I've met can easily introduce show-stopping security flaws into software without realizing it. It turns out that programming is hard and without rigorous models and proofs humans are really poor at understanding programs. We naturally hand-wave away details which get us most of the way there but these are discrete systems and details matter a lot.
As for testing... I agree, unit tests are often insufficient but probably the lowest bar you could convince typical industry software developers to hurdle to prove the correctness of their programs. Even then, as the author notes, they will come kicking and screaming with their opinions. There are far better options and strategies but as soon as you mention property testing or formalization you'll lose about 90% of the room so you have to pick your battles.
The trouble is that most startups are motivated to accidentally stumble upon a market niche and in this day and age that often means, "something with computers." They're not in it to write reliable, robust software that is efficient and performs well on a target platform. They're writing code that they can cash out on and never see again. Very different world.
The final breakthrough we need is to fund research labs again. Technology isn't inevitable and civilizations in the past have lost the ability to build and maintain it. We shouldn't let that happen again if we can help it.
Amen.
Even more amen.
> To be more specific - for most web projects I work on, I have a very similar yaml file for CI, Dockerfile,
Very similar, but not actually the same - of course there's always a tradeoff between control and convenience. Maybe you don't like where kubernetes, $CI_TOOL_OF_CHOICE, etc puts that tradeoff, but that's why there's a zillion tools and languages out there, because someone wanted a slightly different tradeoff
> Changing the program is the most common thing we need to do in our work, it's the reason our job even exist. And yet, most of programmer's time is spent reading or planning how to change the code.
I think I fundamentally disagree with the idea that "reading or planning" are second-class activities here. I think that one of the highest purposes of a programming language is is to communicate to other humans, unambiguously, what the program is intended to do.
I think most languages have a sweet spot for "good at communicating for $TYPE_OF_PROBLEM" - it'd be a nightmare to read the assembly code for say a AAA videogame and try to figure out if it's a FPS or RPG game, but if the intent of a program is to do a small thing in a super-architecture-specific way, probably assembly is a great choice - exactly because some other human will see _exactly_ what you mean.
If I have to work in a big complex codebase, it's great to use a very structured language like java/rust/etc where all the many, many pieces are (or at least can be) clearly named. I'll have no idea which CPU registers are twiddled when, but I don't care, and neither did the author of that code (presumably why they chose a high-level language).
On the extreme end - a tool like excel which is AWESOME for getting computations done and presenting the results, but is (IMO) not the best for communicating _how_ to do those computations. Do every single one of the cells in this column have the same formula? If not why? Is that intentional?
I thing programming languages are awesome exactly because of the extreme, explicit precision and repeatability that they allow us to communicate with, but certainly not all problems [that are currently solved with programming languages] require that kind of rigor.
A language is released too prematurely -> some companies start to use it -> design problems become clear, but the language is in a compatibility trap at this stage -> e.g. it becomes a swamp or it slowly becomes a monstrosity of design afterthoughts and crutches.
It probably even wouldn't require much more spending from corporations, only some initiative. They anyway already invest considerably in language&toolchain development, but often with wasteful goals.
A quick explanation is provided in this video: https://www.youtube.com/watch?v=S4LbUv5FsGQ
And more details can be found at: https://github.com/kgrgreer/foam3#videos
FOAM is a modelling framework that generates cross-language boilerplate for you, but it takes a much broader view of what constitutes boilerplate than most systems. Typically, it can generate between 95-98% of a working cross-language cross-tier system.
FOAM helps you create features for modelled data. Features include things like a Java/Javascript/Swift classes to hold your modelled data, code to marshall to/from JSON/XML/CSV/etc., various GUI Views, and support for storing your data in various databases or file formats. However, FOAM models are themselves modelled, meaning they're afforded all of the above benefits as well. This lets you apply the MVC technique of having multiple views work against the same underlying data-model concurrently (say a grid and a pie-chart in a spreadsheet), so that you can choose the best view or views for your current need. When treated this way, your code is no longer text (but it can be, if that's one of your views), and you can easily view and store it in many different ways and more easily programmatically manipulate it.
This is perhaps most clearly revealed here: "Changing the program is the most common thing we need to do in our work, it's the reason our job even exist. And yet, most of programmer's time is spent reading or planning how to change the code." Combined with the disdain for testing those changes, it seems the author wants a magic programming wand.
This guys seems to believe produtivity equals amount of information produced, and fails to grasp the basics of creative work, which is exactly thinking.