Greg Young described a good way to deal with requirements volatility: optimize from the beginning to be able to delete your code, and structure your code so that any part of it is no bigger than 1 week's worth of coding. So that any part can be re-written in 1 week.
> Requirements change. Every software engineering project will face this hard problem at some point.
> There is really only one answer to this: you need to find a way to match the requirements-development-delivery cycle to the rate at which the requirements change.
Honestly, I believe that much of the problem is with the word “requirements” and I so wish we could kill this use of the word. Bottom line, for systems built for humans, ploughing through backlogs of requirements is the path to mediocracy (or worse) – as the OP discovered.
A less misleading framing: you need to find a way to match the delivery cycle to the rate at which understanding changes. In other words, your development process is a learning process.
100% agree learning is the core problem people in large organizations really don't know how to learn and the typical corporate environment is the perfect quasi academic environment geared towards pursuing metrics that amount to chasing grades
This absolutely. As understanding evolves, requirements evolve. Some domains are well-understood enough that this process can be done up-front a-la waterfall, but more often these domains are so dynamical that the only way to understand them is incrementally, one dynamic (or metadynamic) at a time.
Each software development methodology seems to be about mitigating particular risks.
Agile seems to be mitigating the customer not knowing what they want the software to do; tiny little software cycles, little steps, making the customer look at it frequently and saying what they think.
There are software projects with high-quality customers (sadly so rare) for which that risk is much lower, and as such agile is a bad fit. There are certainly software projects where waterfall is a better fit, mitigating against different risks.
That's when done in a factory feature farm where you've already sold your control for the safety of a steady paycheck
Try selling a really agile methodology to a client of your own before you jump on the turnip truck
When done properly it gives the client immediate results with little up front commitment or risk and gives you all the control you could want as long as you don't mind being untethered from the communal trough and putting your reputation on the line at every iteration
I would say that agile has helped create the conditions for the proliferation of tech debt mills, because it empowers product people while disempowering engineers.
"Rapid and flexible response to change" is really just bruteforcing ideas leaving a trail of tech debt that makes developers miserable.
What you describe makes me think that you pivoted from an engineering role into a product role in order to make agile to work for you. This only reaffirms what I said, because if agile really empowered you as an engineer that transition would not have happened.
By the way the word pivot is largely misunderstood
It means to change the approach while maintaining the vision and values
My values have always been on teamwork and knowledge creation and I pivoted in the sense of changing the approach but not in the sense of changing the vision
This has less to do with agile and more to do with the company culture, processes, and probably the people involved. Agile isn't a cure to these problems, it's a tool used by people who "get it" to communicate effectively and to empower everyone to do their jobs.
Thanks to agile I am directly involved in evolving the product I work on and providing information back to the product owner and anyone else involved. I regularly make suggestions that are listened to and I get feedback on my work several times a week. The changes I make are released to users rapidly and regularly and we gain insight from that to make our next decisions.
I also openly discuss the compromises I make in terms of technical debt and I'm given the freedom to clean it up when it makes sense for the business.
The business would not be able to innovate as quickly if it weren't for agile. I would be less empowered and performing suboptimal work if it weren't for agile.
Tech debt "just get it done" style management with egos too sensitive for candid feedback will always exist and the way the work is defined will have little effect. It's unfortunate that you work for such managers, but please don't blame it on agile.
Yes mostly people are engaged in agile theater and then blaming agile as a concept add to that the whole industry of meddlers middlers and muddlers that has sprung up and it's no wonder most of it has devolved into nothing more than a wholly ineffectual circlejerk to put it mildly
If agile is done in such a way to amplify the voices of invested parties then I'm all for it -- starting with the team
And the way engineering culture contributes to this toxic ecosystem is by perpetrating the myth of casual professionalism thereby selling ownership by avoiding responsibility and taking perks over substance in the bargain
I believe the agile manifesto was written by smart people that knew what they're talking about. But what is known today as agile has nothing to do with what they really meant.
The agile manifesto says "working software over comprehensive documentation", but in practice, all agile projects I have seen put more emphasis on project documentation than working software: kanban boards, sprint burndown charts, the backlog, project planning documents... that's documentation.
The manifesto also mentions "individuals and interactions over processes and tools". Standup meetings, sprint planning, sprint retrospectives, etc. are processes. And agile practitioners I have met put more emphasis on those rather than doing something in the spirit of the manifesto, which is using reason and logic to collaboratively find solutions.
I think a lot of people are conflating Agility with SCRUM. If you look at the official SCRUM guide, you'll find ZERO references to agility. And if you look at the original Agile Manifesto, you'll find ZERO references to stories, sprints, story points, velocity, standups or any other thing that we consider agile.
For me, the essence of agility is: break down large projects into smaller and smaller projects until the "mini-project" becomes simple enough for you to execute. But each such mini-project must delivery something that works. Also, don't show documents about the software, show the software.
It is funny to compare that to traditional product design. Let's say your goal is to design a chair. The most important tools in your toolbox are variants and iterations. The worst you could do as a designer is constantly working on one (final) chair. Because every change that goes aginst the initial concept will have you saw pieces away and glue them to other spots till you end up with a completely irrational whole.
The problem is of course, that a low effort design prototype still looks and feels like a low effort design prototype. But the customer sees 5 variants of the low effort prototype and can decide which one is iterated on further.
If you program a low effort software prototype, it should still kinda work and this is where the customer would decide to run with it.
Also software (like graphic design) is a field where people without any skill tend to have strong opinions about how things should look or work, which makes manipulazing the customer into taking a decent solution part of the job.
> Also software (like graphic design) is a field where people without any skill tend to have strong opinions about how things should look or work
Business too. You do not have to be a good strategist to “succeed” as a VP in a medium or larger sized organization. In fact, you can be downright terrible.
It is why I ultimately stopped taking on much graphics design work and did more film sound mixing jobs — there is so much less people who think they have the experience to tell you precisely what they want and much more we trust you and do your thing.
I am slowly starting to get the idea behind medival guilds..
But when you're writing business software to solve business problems, many more people have a "right" to an opinion. Usually the business people know a lot more about the problem domain than the programmer. They are also the ones who will use the solution.
I've been thinking that this is one of the key differences between software and bridges. When you design a bridge, very few people have a "right" to an opinion on the bolt sizing. But when you design an order entry system, the business users (and users of the downstream data) have a valid right to an opinion.
Agreed, if it is their profession they know it better. The thing I was initially refering to is stuff like people who do a horrible layout in word and ask to to literally do the same thing without having any qualification on the matter. Or people who want you to put a whole page of text nobody will ever be able to read onto a roadside sign.
This feels like if one would ask a electrician to use wires without insulation because they like them better — you can totally have your taste, but some stuff is just plain stupid and won't work as intended.
Graphics e.g. is about communicating certain things, and a graphic designers job is to be aware of what this communication does and how effective it does it. And so just like with a bridge (but arguably less dangerous) there can be design that does the job and design that does the polar oposite of what your customer intended.
But chairs are made in large runs, 99% of the work is in the factory. So everyone understands that the work of making a prototype is a different thing to the final product.
A better engineering example needs to be completely one-off, like building a movie set, a stadium, or maybe an airport. These also have endless problems with the requirements changing during design & construction.
You are right, movie sets are a way better analogy. Things on movie sets only work if all people involved adopt nearly military discipline, if you talk to people within the industry they always prepare for the worst and still try to do so on budget.
As somebody who worked as a freelance DOP as well, I think the software world could learn a lot by looking into other areas, especially when it comes to how requirements, changes in said requirements and the impact of these changes on the whole are communicated. IMO one of the most important abilities of an professional regardless of field is to be able to estimate how long something might take and how much work it might be. And this means also beeing able to say how much impact a change in requirements would have on the quality and deadline for the final product.
I know from experience that this estimation is harder in software than in nearly any other field, because problems that look simple on first glance might turn out to be nearly unsolvable, while other problems that look hard might have fairly straightforward solutions — and without putting time into it, it might be hard to tell beforehand.
My problem with that is that it seems to be expected that even a developer without any experience should somehow be able to do this kind of estimation?
I work as a freelancer mostly in film postproduction and there are a lot of times when I have take a wild guess on how long it is going to take. It helps to mentally outline all tasks involved and just take a pessimistic guess on how long each of those will take you. Then just sum them and round up to a full day.
If the topic at hand is really something completely foreign to you, be honest about it and tell them the time in which you will get it done for sure. If they don't agree to it, you wouldn't have made them happy anyways.
However: if you are not just a hobby developer, knowing at least roughly how long things take is key — you have to live from it after all: if you earn 1k for a job that takes you two months you cannot pay your rent. And even if you just do it as a side job, being able to give estimates still helps you to use your time wisely.
I don't think an dev without experience is expected to be able to do that. But a professional who has to live of their profession certainly should, if they want to survive.
But the code is the design. The actual building is done by the compiler (or equivalent). So the important lesson is to budget for iterations of the design and not putting too much effort into the first sketch presented to the customer.
Yes, that is very true. If changing and complex requirements are a problem, you are doing it wrong. This is the main reason why code quality must be as high as possible. It is the best chance there is to be able to accommodate new and changing requirements in a reasonable time frame. It is also the reason why automated tests are a necessity. Otherwise it is way to easy to satisfy one requirement while not noticing that at the same time one is destroying three others.
If the subject is Software Engineering and you said "If changing and complex requirements are a problem, you are doing it wrong" the it in that sentence should be Software Engineering - but the changing and complex requirements seldomly come from the Engineers.
No, they do not come from the software engineers. It is like being a DJ who is taking requests. You are not playing music for yourself but for other people. It the DJs task to make them sound good. On the other hand, one sometimes also needs to clarify what is wanted because the customers will ask for a rock song with a slight edge of metal in it and as the DJ you will need to know what song fits that description.
It seems that when I say that requirements seldomly come from the software engineers you have somehow misunderstood me as meaning that requirements do come from the software engineers?
No, it is just that your post sounded to me like you were saying that it is a problem that the requirements are not coming from the software engineers. My post attempts to say that this is not a problem and that it actually is how it should be.
How could you possibly interpret what I wrote in that manner?
I quoted your statement "If changing and complex requirements are a problem, you are doing it wrong" and pointed out that it was not reasonable to say someone is doing Software Engineering wrong due to changing requirements, as the engineers are seldomly in charge of requirements.
Ah, now I understand what you are trying to say. Yes, the 'it' in my post is indeed referring to software engineering.
It is not so much that they are doing software engineering wrong due to changing requirements as that the changing requirements make clear that the quality of the engineering is not that great. Another engineer who is working with a higher quality perhaps could have handled the changing and complex requirements without much problems. And in another project the requirements are perhaps quite simple and there is not much requests for changes and both engineers would have done equally well because their skill was not tested much.
Of course, there also has to be some limit in changing and complex requirements beyond which it is no longer reasonable for anyone to be able to keep up...... but we do call it SOFTware as opposed to HARDware because it is supposed to be changeable.... and if that which is supposed to be changeable actually is not, there must be something wrong.....
100% agree. I realised many years ago that to try to restrict/fix requirements was futile - or, you can try, but to what end. It’s like swimming upstream.
The hard part is that being adaptive to change increases the complexity exponentially with size (of project, of people, etc...).
Therefore the only way out of this formula is to reduce ‘size’, and iterate with (/alongside) the customer (even if that is an internal _customer_).
Yes, it's indeed a property, not a problem. The problem is the accidental complexity which has the tendency to increase and requires constant effort/energy from us to keep it low.
This is somewhat triggering for me right now as I am just getting done with a project that is quite a few degrees hotter on the Requirements volatility scale than is currently assumed.
The article talks about the one contract that had the requirements fixed -- but then fails to talk about how contracts should be laid out to embrace changing requirements.
People have known for a long time that volatile requirements are a core problem, but that typically hasn't really affected the way software and software development services are procured.
This might be different in other parts of the world, but my impression is that even custom development contracts are procured the same way that you would a big shipment of physical goods, which is just insane.
Once you work as a developer in a project that was procured in such a way, it's already too late. The knowledge that requirements will change and will totally disrupt your development process is there, and it doesn't really help you.
The article mentions how one part of change is that regulations may have changed.
I work in a heavily regulated industry, and am dealing with a set of regulators nearly all who depend on just one particular auditing company to ... effectively write their regulations for them. To the point where a new jurisdiction happily refers to their draft regulations as documents prefixed with the auditing firm's name on them. (This is not a joke or hyperbole.)
That particular company has trained their own workforce to evaluate acceptance criteria according to their internal workflows and practices. This in turn means that almost all industry regulations are geared not towards functionality, but to make sure the auditing company has the least amount of training to do. As far as I am concerned, this is a particularly perverse form of regulatory capture: it's not even the industry at large writing their regulations, but an already entrenched gatekeeper acting on their own interest, crippling both the regulators and the companies in the industry.
Now, I have no problem with regulations as such, because for most parts the rules tend to make good sense. But I do have a problem with regulations shaped after the desires of this particular auditing company. Instead of focusing on results, quality, transparency, accuracy, recoverability and functionality, they have a fixated view on particularly dubious change management processes.
I am convinced that at some point in history, someone has done extensive research into 70 years of software engineering best practices - and gone to extreme lengths to expressly reject as many of them as possible.
Two examples: they insist that even unit tests MUST NOT be written by the engineers who work on the code; and they are mortally afraid of the idea that engineers who work on a problem could ever understand and evaluate the problem. (Not to mention that the very concept of a workflow tool is completely alien to them.)
The end result is that you can either try to do software well, or you can try to do it according to regulations.
It's definitely part of the job. If you don't understand what the problem is that you're trying to solve, how can you verify that the constraints you are working with actually make sense?
Developers who believe they can forsee the future are more dangerous than missing business requirements.
The problem with missing business requirements are that developers often are either to focused on writing the code instead of actually understanding the business.
You cannot build something you do not understand.
Understand small and expand your understanding. Code is not the goal, it never is. It's a means to a goal.
Another thing about software is, that a lot of it should never ever be written. It's the last thing you do, is write code for overcoming a challenge, because rarely you really need it.
Nicely put. In my nearly three decades of software development, the most successful of my various colleagues have been the ones motivated by understanding the problem and its wider context, rather than the tech/process/language _de jour_.
This is a multiplier because it allows them to detect and ignore non-problems and work on problems whose solutions _will_ deliver value.
I disagree with the title. I've used waterfall and still found engineering hard.
Even you have a bulletproof requirements document, implementation and engineering still produces unforeseen eventualities and unexpected consequences.
I think software engineering is hard primarily because it's the coalface between abstract and concrete.
We're wired to be great at concrete, but capable at abstract. With software it's like we can get ourselves into a perfect storm of intractable-for-us problems.
Other forms of engineering, civil, mechanical, etc are still really hard, but I think we're better wired for them because they're less abstract, more concrete.
But in software, abstraction is part of your bread-and-butter daily-grind tool-kit.
Well rewrites have two conflicting goals - simplifying the code/improving performance and consistency with the old version. Even if the requirements are nominally the same "unwritten" ones pop up
The only software project I ever worked on in which the customer didn't report so much as a single bug had the requirements done properly, up front, and agreed all round. Of the thousands of identified requirements, I think less than ten changed over the lifetime of the project, and each change underwent significant examination for consequences.
The project was waterfall all the way.
The quality of software that can be built if you know before you start what it's meant to do, and that doesn't change, is astounding. Truly astounding.
The funny thing is that most people never even attempt to do that, and then still manage to complain about it being impossible.
Yes in some projects that can be harder, but I'm not buying the ambiant: "in most projects". I actually think in that in most projects, a good part of requirement analysis can be put upfront, and of course a tiny bit will still evolve, but not enough to prevent doing things properly.
That does not impose to have too long projects, btw. Just cut them in phases and/or pieces. Feedback cycles are important, but this is still an equilibrium, and in some projects I don't see how 2 weeks feedback cycles without proper requirement engineering would be any good.
Maybe one problem of software development is that it is an ever growing industry, so the mean programmer is always young and inexperienced. Maybe things will be less impulsive with the eventual end of that growth.
I have worked on huge government projects that has been specced and planned for years before arriving at my team's desks. It takes a few days before we will have questions. Things not defined good enough, inconsistencies, glaring holes etc.
I think having all reqs upfront is a utopia that will not happen. Those projects I was involved in would have been better if the reqs and scope was done in conjunction with us implementing it.
We had some good requirements people. Really good. They didn't just write things down. There was a lot of examination and analysis and plotting and so on, and they were good at it. They'd all begun their careers as software engineers and gradually specialised into being really good at requirements.
I think having all reqs upfront is a utopia that will not happen.
What can I say? It happened to me. Only with one company in seven or so worked for so far. I would not be surprised if people worked two dozen jobs and never experienced it.
You do need high-quality customers, though, and they are hard to come by. I've worked for a company that has turned down customers on the grounds that the potential customer just isn't good enough; they worded it differently, but that's what it was.
The fact that you immediately had questions of the requirements means they were not properly thought out and/or not well-defined (regardless of how many years they spent in the planning/gathering stage). Both of those (well defined, thought-out requirements), along with reasonable people evaluating them, would have yielded a much better outcome overall.
Just like spaghetti programming, you get spaghetti requirements analysis & process-engineering. And even then, they hardly consider some of the things that developers do.
Almost as if the developers that encode the rules in a "hard format" such as a program/system need to be the ones dictating the processes/requirements. I would definitely see that as separate to "reqs and scope done in conjunction with devs developing them".
"Take a government contract I worked on years ago. It is undoubtedly the most successful project I’ve ever worked on, at least from the standpoint of the usual project management metrics: it was completed early, it was completed under budget, and it completed a scheduled month-long acceptance test in three days.
This project operated under some unusual constraints: the contract was denominated and paid in a foreign currency and was absolutely firm fixed-price, with no change management process in the contract at all. In fact, as part of the contract, the acceptance test was laid out as a series of observable, do-this and this-follows tests that could be checked off, yes or no, with very little room for dispute. Because of the terms of the contract, all the risk of any variation in requirements or in foreign exchange rates were on my company.
The process was absolutely, firmly, the classical waterfall, and we proceeded through the steps with confidence, until the final system was completed, delivered, and the acceptance test was, well, accepted.
After which I spend another 18 months with the system, modifying it until it actually satisfied the customers needs."
> As a consequence of this, software is never finished, only abandoned.
Ditty of mine from around 1988 or so. The hardware is old and obsolete but the software never is complete.
One thing I realized a long time ago was that you need enough resources to keep up with the requirements drift or you will never finish. That's what's behind YAGNI and Minimum Viable Product.
There is this other thing which is a successful product 'creates it's own weather' as far as requirements go.
Perhaps that is the core problem, but that problem is caused by not knowing the requirements to begin with. Which again is usually caused by not knowing what your customer wants (which you should be able to do, even if they don’t know themselves what they want), or not knowing who your customers are. Contract work is the best environment for fostering a culture of not knowing what your customers want, because it is the ultimate exercise in fence-throwing (often to somebody who’s just going to throw it over yet another fence).
Problems tend not to change so dramatically over the course of a project. What usually happens is some stakeholder realizes at some point that they didn’t actually understand the problem the first time around, and thus need to change the requirements. Even significant regulatory changes won’t usually cause that. In almost all cases, regulatory changes have a significant amount of time between announcement and enforcement.
Making your development practices more responsive to change is a good goal, but it doesn’t solve this problem. Because this problem is typically created before the first line of code is written. If the defined requirements don’t actually solve the customers problem, then they’re going to be delivered a non-functional solution, whether it takes one year to deliver, or one day.
This quote is old and explain very well why early software was often high quality:
> “Walking on water and developing software from a specification are easy if both are frozen.”
― Edward V. Berard
The more you plan ahead and the less you change goals while writing software, the less problems you have.
Software is the only field of engineering where you can change things at any point in time. But it doesn't mean you HAVE to change things. Software should really take lesson from other fields of engineering. Companies always want to cut costs, and with software, they really can.
I often hear people say that software engineering is a joke compared to civil engineering or other kinds. But they forget that when a civil engineer designs a bridge he doesn't start with: "We don't know exactly what weight it should support yet, but just start building the bridge already". And he doesn't finish with a: "Ah, turns out we wanted a tunnel instead of a bridge, can we change that next sprint?".
My view is software is literacy not engineering or science.
It tends to make more sense for me. A book that starts out as a romance is just as hard to convert to a spy thriller half way through as a bridge and a tunnel (Ok not total effort but in terms of percentage of rework)
Engineering means using math to model your solution so the first time you build something it works to spec. Or at least it's close enough to only require minor changes, not a fresh redesign.
It's essentially applied physics.
Software is more like nailing things together in the hope they might work. ML has some modelling, and formal methods are available for mission critical projects.
But the rest is mostly nail guns and glue.
Engineers have never done themselves any favours with the "engineering" label. In many countries, especially English-speaking ones, the job doesn't get the respect it deserves.
Most people have no idea what the job involves. They literally think it means someone with an oily rag who fixes cars and/or computers and/or drives big machines around to make bridges and tunnels and such - like a house builder, but on a bigger scale.
But how many mech/civil engineers actually do any math beyond napkin calculations in their day-to-day work? It's way too complicated and all the math is now done by computers. In fact I'd argue that outside of research more computer scientists are doing mechanical engineering math than actual mechanical engineers, because they have to write the CAD software that the latter use. At the end of the day the vast majority of mechanical, civil, electrical engineers and computer scientists work on stuff that doesn't require math at all.
* "Engineering is the use of scientific principles to design and build machines, structures, and other items"
* Mathematics isn't only about calculation and applied physics. When you write a simple algorithm and convince yourself with a logical argument about its correctness, time complexity, etc. that is math too. You also do calculations when you design various systems (latency, throughput, etc.)
* I know a dozen mechanical, structural, and electrical engineers. They almost never model anything from first mathematical principles, they use well known tools and methods they learned. Their "nail guns and glue". Actually I think most of them don't understand math/physics behind it very deeply. So even if those methods are underpinned with mathematics and physics, it's abstracted away most of the time which is good for practical purposes.
Even in "user-interface programming", I often had to use math, algorithms, etc. In other parts of software engineering I used it much more.
Even most mechanical/civil engineers use very simplified models than is used by physicists. Engineers usually use Newtonian mechanics in its elementary formulation, and is not usual for them to be familiar with Lagrangian/Hamilton/Variational formulations.
"Developer" is appropriately diffuse in its meaning. It includes all the steps of making software happen, requirements gathering, critical thinking thereon, etc. "Programmer" can sound like just "guy what taps on the keyboard". It can even be faintly perjorative in some contexts.
I'd use "software engineer" if I could in good conscience, but "software developer" is pretty neutral while still being accurate.
To clarify, I use "programmer" when speaking to very non-technical people, usually elderly. Many other languages have a similar word but it might have different connotations, for example, be less negative (pejorative).
Real engineers in north America must write technical and ethical exams, have at least 4 years of continuous and proved experience before they can call themselves engineers.
Is indeed sad that people with no knowledge of thermodynamics or strength of materials can call themselves engineers as well...
I guess you're making the distinction between professional and non-professional engineers?
There's not really the class divide in engineering fields that you're suggesting. I have a brother who is an ME, designs power plants (the building sized ones) and is not "professional".
I asked him about it and he said it wouldn't change his role or his salary, so he never bothered going through the process. IIRC there's one person on the team who is, for whatever formalities are required.
I always thought engineering was simply applied math and science. You're not researching and testing new theory explaining the nature behind how phenomena work, or developing new mathematical techniques, that's for scientists and mathematicians. Rather engineers take their work, and apply it to invent things and solve problems.
As so, I'd consider software development can qualify as engineering. If you're taking the work of computer scientists and mathematicians (or any other science fields) and applying it to software, it fits the definition. If you're just laying code to a given spec without much more thought, then you're just a programmer/developer...
I think that the engineering mindset (adherence to the scientific method, evidence based decisions, planning for the future etc..) is one that is definitely shared by both groups at least.
But you can absolutely get away with building software without proper engineering standards. It's not so easy to build bridges that way.
I'd wager that it is but the stakes for failure are too high to wing it (hopefully).
Imagine:
We want to build a bridge that spans N meters. We don't know how to do it but we can go look at an existing bridge and copy it. We might not understand metallurgy / physics that well, but we talk to suppliers and ask for "steel of this shape and size". Ditto cables, concrete and everything else.
Probably, provided we're not scaling up or changing the design too much, it would work just fine.
For most non-engineers (either kind), engineering is just a means to an end, i.e. don't care about the finer details as long as it works.
When software goes wrong, the effects are more often than not, intangible. When mechanical engineering or electrical engineering goes wrong, people can die.
Software can obviously kill people too, it just happens less. Then industry practices revert to the mean due to cost cutting.
Sometimes, that happens. But there's a mutual agreement that such change comes with significant cost, and it's this part that is missing in the software world.
I think it is possible to convert a bridge into a tunnel, to follow with your extreme example.
It's totally possible. The only thing is that when the cost estimation arrives to the hands of decision makers, they open their eyes wide and decide that for that money they better go ahead with a bridge.
So IMHO it's just a matter of cost. Say you go with a bridge but there are some initial wooden stairs, and sometime along the middle of placing them they decide it is acceptable to pay the cost of stopping and replacing them with metallic ones. Could happen.
Biggest change I experienced in the rate of unexpected last-time feature changes in software, was having a project manager that first of all knew how to say "No" (or discuss the sense of the requests), and secondly, was able to communicate the real cost that each change would have (in terms of developer time, which at the end of the day means money). Then it would be the company's boss himself who would decide to discard the most crazy changes that were "mandatory" and coming from the sales people.
Also a civil engineer usually understands exactly what the bridge is supposed to do and how it is going to be used.
In a large corporation, the software engineer expects to be told that there are going to be cars on this bridge, and they are moving at a certain speed from one side to the other and that they will be of a weight between x and y, etc...
Understanding the domain goes a long way to get software done.
As someone who has worked in both worlds, it is just a different sort of nonsense that you have to deal with. Typically the physical engineering thing is "we want you to work out why this weird thing is happening, then fix it, without us having to change anything operationally or spend any money." just a different flavor of the same crap.
So much this. I want a civil engineer try and build a bridge over 6 tectonic plates with pretty much all physical constants constantly changing. That would be the equivalent of working with an ever moving tech stack.
Even the language and best practise change constantly. Looking at possibly the most stable language C, we have entirely different best practise pattern from community to community.
This is hyperbolic. Plenty of completely reasonable tech stacks are over ten years old. You could have started writing a web application in TypeScript, my current language of choice, over five years ago. I can go write a web application in React and Spring Boot (which I also like) that's almost completely recognizable, and could have been carried forward from the state of the art of, 2014. Much older, in terms of the backend; Spring Boot is relatively new but it still uses the Spring that somebody might know from 2010 and builds very incrementally from there. Even that noophile bug zapper, Golang, is most of a decade old now.
New stuff shows up, but that doesn't mean you are compelled to leap upon it.
And software engineers don't have to deal with half their tensile bolts having 90% of the tensile strength built they don't know which half, or the bridge collapsing because they mixed brass and zinc bolts[1], or hurricanes...
Turns out that every field of engineering is hard!
> I often hear people say that software engineering is a joke compared to civil engineering ...
I think your example points out exactly why some think software engineering is a joke: they rush ass first to implement something (and they implement the wrong thing) instead of sitting down and figuring out the actual requirements first like any real engineering job would have you do.
And it looks like everyone is pushing it more into this ass-first direction with agile. Let's not even try to plan ahead! You know what you want next week, so this short-sighted focus leads to chaotic development and once the real requirements are figured out 9 months into the project, well too bad it's too late to redesign and rewrite it all properly. So it's forever going to be a bumpy and crooked bridge on the verge of collapse, held together by more and more duck tape, hot glue and occult ceremony. And there's a plastic tunnel hanging underneath it. Only the brave will crawl through.
I have been in a position where I have said no. I was able to stop the feature factory for a few months. But after that we were right back to cranking on features because several end users "validated our roadmap".
Santa Claus isn't going to fly down your chimney and deliver a fully formed business case. Developers share responsibility with other roles for figuring it out. Sometimes it takes multiple iterations of exploration and experimentation. Expecting to be told what to do is just lazy and unprofessional.
> instead of sitting down and figuring out the actual requirements first like any real engineering job
If it were so easy, then we would have done it already that way for the last seven decades. The simile topples over upon further contemplation:
civil engineers:
• client is not a domain expert
• to an overwhelming part, client needs are easy to transport into the mind of c.eng.
• can employ a wealth of standard solutions refined over the course of milleniums
software authors:
• client is a domain expert for the subject matter that the software is supposed to model
• to an overwhelming part, client needs are difficult to express and transport into the mind of s.auth.
• if a standard solution exists, the client would have already bought it off the shelf, so coming to a s.auth. always means customised development
• the entire sector is still in its proverbial baby shoes
We have to come to grips that we unfortunately can't apply what happens to work well for a different sector. The supposed value of non-traditional software methodology lies in that at least the client notices that the requirements are off the needs rather early.
I agree it's not easy, but instead of trying harder to become good at it, it seems like this industry in its proverbial baby shoes threw a tantrum and decided that it doesn't want to learn at all.
So instead of pushing for more effort & skill for up-front planning and modelling of the requirements and design (and trying to employ modelling tools & formal methods), we're just like fuckit nah let's just implement something and see where that leads us.
It doesn't help that so many of the up-front design failures were not examples of engineers screwing up but sales people selling shit and then having engineers cook it.
We'll be stuck in baby shoes for a long time with this approach.
Companies are more than happy to churn through staff, losing millions in accumulated human knowledge, just to keep it out of the hands of employees.
My company just shut down an entire office, they offered three out of four critical employees the opportunity to move to Atlanta. They didn’t want to move, opting to retire instead. Company just shrugged their shoulders rather than letting them work remote, gave them like a month to transfer knowledge and that’s that.
Didn't we start by trying to plan ahead? There's a reason we have moved to agile.
In software if you know what you are doing ahead of time you are not really innovating and you might as well be using an existing system. Since everything that is standardised you can just use as libraries.
When building bridges, you can't just use a library that's .buildBridge(...parameters), even though similar things have been done thousands of times before. With code, if something is done enough times that you know how to plan for it, it is likely already a library.
You can plan building a bridge since you have done that and have experience of it thousands times over.
> Didn't we start by trying to plan ahead? There's a reason we have moved to agile.
"Everything was waterfall and then came agile" is myth, but yes, it looks like it's fashionable to no longer try to plan ahead. I'm not convinced the reasons are good, and I'm not convinced the craft of planning ahead was ever perfected to the point that anyone could say it doesn't work.
Scrum, XP, Agile all date back to the 90s when one server running PHP or Perl (with SQL injections) passed for software and others were still re-inventing basic data structures in C (and then gasp C++ with templates!) from first principles... The methodology, tooling, rigor, and collective experience in software development has taken leaps in the past two to three decades. Likewise, understanding systems and planning ahead has become so much easier. If only people took it seriously and spent as much effort on it as they spend on fad-of-the-year frameworks.
People just stopped trying, and when I look in software projects now, there's hardly ever anything resembling a model of the software that you could use to test new ideas "on paper" to see how they fit the given problem and current application.
> In software if you know what you are doing ahead of time you are not really innovating and you might as well be using an existing system.
I think the vast majority of software is boring and not even meant to be innovative and indeed would be best written using an existing system, but customizing these to the specific needs is just another can of worms. As with bridges, you can't just copy & paste an old bridge to a new location and use it as-it is, you need to design it given the locale and other constraints. It's still probably going to be yet another relatively boring bridge with no innovation.
And indeed a lot of software work is all about using existing systems, but perhaps at a lower level than you posited. Customizing complete off-the-self packages does happen, but it's more common to glue something "new" (but not innovative) using existing frameworks, libraries, services and glue languages.
You still need to design to do it right. But even here people seem to rush ass-first into whatever kit looks fashionable now.
"Not trying" is not new. The problems with both Waterfall and Agile is that nobody really ever did them the way they were intended. Waterfall was supposed to include prototyping during requirements gathering, specifically to aid in discovering constraints. Agile was supposed to give equal footing to all team members and not let the project manager spring new constraints on the team without planning for the change. Neither of those things happened very frequently, because our modern business culture treats knowledge workers as interchangeable minions, and management as unassailably perfect.
Yes, people don't do their jobs. That's not the fault of the process that they are failing to adhere to. Indeed, no process of any type could ever save them. It's the culture that is wrong.
So, in a roundabout way, I agree that Agile isn't the solution. No bottom-up process will ever succeed, because at the end of the day, the failure of the team is always the fault of the leader.
Agile as a named methodology may date to 90s, but the concept of rapid iterations, incremental improvement, and constant feedback cycles has been on the books since at least 1950s.
The last time[0] I vented on the subject, I had to dig out the research papers I keep on my desk, because there was more than passing curiosity. Oh yes. I keep these on my desk so I can quote them when needed.
Check out this research paper from your nearest sci-hub entry point:
- DOI: 10.1109/MC.2003.1204375 ; "Iterative and Incremental Development: A Brief History"
The binary thinking in this thread strikes me as odd. Is it bad to try and plan everything 6 months ahead of time? Yes. Is it also bad to fly by the seat of your pants and only plan 1 week ahead? Also yes. Instead, why don't we plan out by some intermediate time period. Say, 3, 4 weeks ahead of time. Or whatever time horizon ends up being the most efficient.
Yes there is no single correct planning horizon. The optimal horizon is proportional to requirements stability. At the extreme when requirements flux is extremely high then the best approach can be a Kanban process, planning no more than a day in advance.
There is a reason we moved to Agile, yes, but it was the wrong solution to the right problem. Scrum identified the problem as a lack of accountability for management, and then imposed new processes on employees without any teeth to hold management accountable. That's why I hate Scrum's too-cutesy-by-half zoomorphization of the problem with the analogy of the chickens and pigs (https://en.wikipedia.org/wiki/The_Chicken_and_the_Pig). In a real barnyard, the pigs and chickens are on equal footing: they are both slaves to the farmer. But in most software dev projects, the developers are not on equal footing with the project manager.
We tried all that. Formal requirements gathering, modeling tools, the works. It was the SDLC/Six Sigma/ISO9000/vendor-certs-are-substitutes-for-degrees era of the late-90s/early-2000s. And it also didn't work.
Often times, it's very easy to determine the functional requirements: "Process this billing statement", "Spit out that report." The complexity doesn't come from the requirements. It comes from the constraints: "We only use this ancient version of this database", "We didn't tell you at the start that the users expect to interface with this system via email".
To quote Mike Tyson, "everyone has a plan, until they get punched in the face." The point of Agile is to get punched in the face earlier and more frequently.
> The complexity doesn't come from the requirements. It comes from the constraints: "We only use this ancient version of this database", "We didn't tell you at the start that the users expect to interface with this system via email".
Those constraints sound like requirements: we require that only this ancient version of this database is used. We require that users can interface with this system via email. Different term, same thing.
That's in the bucket with the kind of stuff that I'm advocating people should pay more attention to up-front. Gathering these requirements before you start designing anything is a big deal. If you failed to do so, then you failed to do a proper job at the part I called sitting down and figuring out the actual requirements.
A lot of places will never give you access to the people who can answer those questions and will expect you to start work right away and "quit wasting time" or they "put you on a Performance Improvement Plan"/fire you. The problem is not developers doing a bad job, it's management preventing them from doing good job.
There's another problem not really addressed here; when the requirements you get from the customer are wrong/make no sense/could be done better another way/are for things they won't actually use/are a part of some internal political struggle.
This is something that crops up quite frequently in my work. We'll get a requirement like this: "the app should use HTTP like a web browser, not a different protocol". On inquiry as to why it's so important for something that's not a web browser to so strongly resemble a browser, the answer will be something like, "so we don't have to ask IT to open a firewall port". The person deploying the software could just fill out the paperwork, but they find it easier to "require" that the software sneak past their own corporate security controls. This is the sort of requirement that is real in some sense and not in another sense, depending on whether you define the customer as whoever you're talking to on the phone right now or the business as a whole.
You're missing the point. In the real world some of those requirements are unknowable on any reasonable cost or schedule basis regardless of your analysis process. From an economic standpoint sometimes the best choice is to accept some uncertainty and get moving.
A large part of the problem is that grand majority of software (at least if it's facing actual human clients) is constantly trying to search or validate market-fit. If you don't get it out first, you're going to lose to a competitor who does.
And even when you do have a solid market-fit figured out, you still don't know what your customers will actually want. (And don't get me started on customer research. People lie all the time, and their accuracy on what features are honest-to-$deity blockers.. Nope.) This means that for most parts the incentives are misaligned and work actively against delivering high-quality software.
Underlying the agile manifesto is the need for engineering to be properly aligned with business incentives. Continuous delivery, fast iteration cycles, incremental improvements and rapid deployments are all facets of the same fact: you don't actually know what your customers want (because your customers don't know what they want!), and the only true way to figure it out is to iterate as fast as you can.
It's only when you have an established moat and a practically guaranteed, net-profitable income stream, when you can even imagine at having the opportunity of doing things the way you would assume you want to.
Oddly enough.. the companies who have such moats and guaranteed profits tend to be the same ones that have been the slowest to modernise themselves. Banks - until the modern online-only challengers came along. Insurance, where the regulatory moats are even higher and is only now being "disrupted". Loan brokering. Energy. Water. Public transport (where monopolies and/or charters prevent competition). Healthcare. On and on and on...
And even then, every one of these established, guaranteed-profit machines has the same problems: the moment one good new player enters the field and starts stealing customers, the old guard needs to find a way to either adapt - which is the best-case scenario - or rely on their power structures with politicians and quickly require new regulations to strangle the usurpers.
Software is expensive to make. It's only cheap to deliver. And time spent making the wrong thing is really expensive.
EDIT: I forgot to say this the first time around. Engineering career incentives are also misaligned. Maintenance is not appreciated, only shipping new stuff is. So the developers who would want to improve what exists and make it better will be looked down upon. The promotions and raises go to those who ship something new, even if it ends up setting the world on fire a year later.
Manager and Director are the people who are pushing "agile". They want quick results, and SE can only do what they want. You simply really can't "gather" requirement and finish a feature and do proper testing in a 2-week sprint.
I do agree there are a lot of bad practices in SE as well, but let's not blame everything on them.
Yeah I'm not really blaming individual software engineers (I'm in the same boat, fighting a slow trickle of feature after feature without really being given a chance to step back and consider the overall design & its future evolution), but the industry as a whole. As I said in another comment in this thread, all too often SEs are just tasked with cooking sh*t to vague spec (as laid out by sales & other non-engineers) instead of being active at the earlier stages where engineering should already begin.
It's like sales people together with the customer decided the shape and colour of the bridge, as well as the number of pillars it has, and how many lanes it has, and what material it's made of -- here's the stack of things you build with -- and what other fancy features it has, and now finally we can call the engineer so they can hop on a bulldozer and start engineering with their hammer for the next two weeks and show the result so we can adjust the shape if it's not as pretty as we imagined. Actual engineering didn't happen.
My understanding of Scrum is that requirement elicitation must happen before the sprint. Otherwise the story does not meet the Definition of Ready and the team should reject it.
It is completely possible to deliver a user story within a 2 week sprint provided that the acceptance criteria are reasonably clear. Many agile teams so this successfully.
Features (or epics or whatever) are larger and contain multiple user stories. In most cases features extend over multiple sprints.
Thats because we have tried it the other way. Its called waterfall. The problem is, the process is too slow in such a fast moving industry.
If you are building a bridge, you dont have the problem of being half way done and all your customers say, "nah, nvm there is tunnel that just went up, that can kind of solve my needs. I will use that one instead."
People built a lot of ugly, useless, dangerous bridges before it was possible to plan and build a good one reliably. Even then, bridges still open with huge flaws. I think software engineering will get there eventually. Someone has to beat their head against the unknown a few times to map out its form.
Having worked on government projects, agile is definitely better than waterfall. The specs will change, better accept that early on and work with it, instead of trying to nail down the wrong thing.
> figuring out the actual requirements first like any real engineering job would have you do
There are no "actual" requirements for most software projects. Software is generally co-evolved with the audience.
Bridge designers can work like they do because bridges are hard to change and fundamentally pretty similar. But because software infinitely changeable and infinitely copyable, our field is different. If you're going to build 500 highway overpass bridges, you'll need engineers looking at each one, because local conditions and needs vary. But in software if we need to serve 500 users, we build one product, give each user a copy, and just keep adding features to solve the local problems.
A big reason we think software can be like bridges is that in the early history of our profession, we were tackling problems that almost worked in a waterfall context. That's because we were taking existing paper processes and automating them. But the hard part of requirements discovery via process coevolution had been done for us by people tinkering with paper and human working relationships.
That era is past. Today if the requirements for something are perfectly knowable, that's because it's either trivial or it's copying an existing piece of software. Any real product is discovering what's possible. It's growing along with its users' needs. It's solving new problems created by other people and groups.
I know of no significant piece of software that, like a bridge, hit 1.0 and just stayed there for a decade while getting plenty of use because the requirements were perfectly foreseen. Not one. To me that's a sign that perfect foreknowledge is, if not impossible, at least incredibly rare. So I think the thing to do is to figure out how to build excellent software with the expectation that needs will change.
Is that because that code and hardware is especially good? Because the initial design and research process was so perfect that people are still entirely happy with it? I'm guessing not.
There is no secret. You develop a process, enforce it, and refine it. It becomes bureaucratic, expensive, and boring. Just like the civil engineering people compare against here.
The process is nothing surprising. Write tests, document your code, follow the style guide, etc.
The problem is that you cannot figure out the actual requirements. In every non-trivial application project I had so far, the requirements (and there were lots of them) were simply not correct. The requirements the actual users created, that is. They _think_ they know the workflow. They don’t.
If you don’t even ask the users, it’s even worse.
But that’s okay. It is simply not possible to accurately judge how something will work until you get to use it. That goes for the creators, too. That’s why we iterate.
I think there is more to it than process. I'm not an engineer, but I know in the US you usually have to pass the PE exam to be a real certified engineer, plus there is the reality that if your bridge collapses, you are liable, might go to jail, etc. I'm unaware of any similar licensing process in software engineering or liability realities. Even when software kills people, it is usually not prosecuted and the devs are not held to the same standards as licensed engineers. For this reason, I just tell people I'm a dev, not an engineer.
I hope this myth would die. Most engineers don’t ever take a PE exam.
The PE exam is only needed for a handful of disciplines, like civil and HVAC engineering. Even then, you only need the PE certification if you are going to be signing legal documents. You can be a civil engineer without a PE working in a team producing designs and plans that are signed by a single certified PE.
By your logic almost no one that graduated from an engineering discipline would ever be able to call themselves engineer.
I get your point about how people end up doing engineering jobs sans PE certificate, but I would still stick to the position that people without certifications are in fact, not engineers. Maybe they are doing the work, but paralegals who failed the bar aren't lawyers, even if they went to law school, they're paralegals (or whatever lesser title applies). You don't get to be an engineer because you passed a degree program, or got someone to hire you with such a title. You have to demonstrate an objective mastery of a large body of knowledge, etc. This is what the PE actually tests for. This is different than in tech, where getting a degree or a job actually does make you a software engineer. There is no standard exam to pass or anything. Just get any random hiring manager to hire you and you're a software engineer.
Well, you can certainly stick to your convictions, no problem with that :) but that doesn’t mean you are right.
You can’t compare against paralegals/lawyers because calling yourself a “lawyer” has legal implications, as in, you must be certified by the bar or you can get in trouble. It doesn’t matter which legal discipline you practice, you must be certified by the bar.
For engineers no such thing as a bar association exists for ALL disciplines. The PE certification only covers a handful of engineering disciplines therefore it can’t be a gatekeeper for all engineers. Moreover, note that certified engineers are called Professional Engineers, not just Engineers! Even the name distinguishes between certified/non-certified, it doesn’t try to take over every meaning of Engineer.
The first step in civil engineering is getting data including hydraulic, geo, and traffic data. Deciding between a bridge and a culvert is part of the design process.
It's not that software engineering is a joke it's that you don't understand the difference between computer work and moving earth/pouring ckncrete. Of course people are going to plan the project differently in civil engineering. The consequences to building any part of a bridge wrong are pretty dire and redoing work costs much more than it did initially.
Maybe don't try to draw parallels with other forms of engineering just because it has the same name in university. The practical differences are large.
They don't forget that, that's why they say it's a joke to compare it to actual engineering. Try treating an civil engineer that way, say "we actually wanted a tunnel". If they have the means to, they might sue you into the ground, and if they had the clout and publicity, no other civil engineer, ever, will work for you. Not in 5 years and not in 20, not until you made super clear you changed. In software dev, you just wrap the stupid or dishonest thing in a new euphemism and go harvest a new batch of suckers, with zero real repercussions.
I saw this yesterday and didn't know if I should laugh or cry, starting from the jingle at the very start of the video: https://www.youtube.com/watch?v=YJGCZCaIZkQ&t=20m06s Imagine an architect conference where a vendor proudly expains how in a building the average tenant lifespan increased a lot -- after their automatic turret was moved from lobby to a less frequented room in the basement. I know that's the opposite of a hardcore SW engineering talk, but still. Everybody wants to be nice, nobody wants to get blacklisted, kool-aid makes the dissonance go away. The main issue was that Linus sometimes called people idiots, and that's solved now.
Maybe other professions also started out this way, as cliques of fools making policy based on what they dreamed last night or how the birds flew, with the main driver being greed -- but being hundreds of even many thousands years more developed, they are on a totally different level than software engineering is today, and just calling it something noble and serious doesn't make it so.
As a web developer with 20 years experience I would argue that one web SPA project is more similar to the next than comparing two bridges and developers still can’t figure it out.
One difference is that a license is required to practice civil engineering. Most web developers, on the other hand, require large tools to do theirs jobs for them and recoil in irrational fear and disgust when standard DOM methods are used. I still have people argue with me that the DOM is slow even though all the evidence and browser vendors claim updates to the DOM is simply switching a bit in memory. In other words many web developers have utterly no idea how their platform works.
Another difference is that many developers cannot write. Everything about a bridge is planned in extreme detail before work on the ground ever starts. This level of planning is absent in software. Documentation is also largely absent. In most cases everything is an ad hoc mess. In any other industry this would be a cause of numerous lawsuits.
Another difference is that construction is treated as a business project with a dedicated project manager and project budget. Most business software, on the other hand, is treated as an operation. If you cannot discern the difference between a project (temporary) and an operation (living) your software will never be released as a product to a specified level of quality by a specific date.
> One difference is that a license is required to practice civil engineering.
I've often wondered about this. Who exactly has to be licensed? Are there management layers in civil engineering firms where the license isn't required? I think my point is pretty easy to follow. I'd love for all of us to be licensed to practice software engineering, but I don't hold any hope that it would fix all of the other compounding problems that our industry has as it relates to product/project management and management-born issues.
For example, would being licensed stop executives at a triple-A video game company from prematurely announcing games / release dates, or accepting pre-orders for games that won't be finished on time, and forcing all of the people under then to scramble and crunch and do things that would generally be considered anti-thetical to whatever being licensed in software engineering could mean?
My uncle is a senior partner in a major civil engineering firm and my wife’s best friend is a civil engineer. Everybody who practices civil engineering in Texas must be licensed to be employable. Maybe you could intern without a license, but you wouldn’t be involved with any project related work.
You've just changed the word without actually defining anything. What constitutes practicing? What stops one licensed engineer approve a design nominally created by thousands of unlicensed lower engineers?
There's a reason we have hard standards for being a PE or an MD. If you just let anyone do it you risk many problems.
Most software simply isn't like this. The valley loves talking about disruption and all and how world changing everything they do is, but the fact of the matter is social media isn't killing anyone like a failed bridge would or a person who isn't qualified as a doctor. Most of the money made in the valley is off of advertising or data. Who cares if the product is shitty? people don't die. rich people make a little less money.
That said, social media and targeted advertising is used to cause far greater (and more subtle) damage, like polarising opinion and influencing elections. Not to mention common patterns of social media usage (which we have developed and incentivised) are proven to strongly correlate with lower quality of life. (I vaguely remember a causal link too, but I won't say that for sure without a reference.)
A bridge hosts what? 400 people at worst (i.e. when there äre passenger trains on it.) Social media easily hosts 400 thousand or million people.
The single bridge-collapsing event is very dramatic and visceral, but digital platforms allow us to cause damage and death by a thousand cuts.
You missed my point. Pay the licensed engineer enough money and he'll become an approver. He can do whatever quality checks he wants as long as he signs off on it. Then I can recoup savings by having tons of unlicensed engineers doing the actual work.
There are checkpoints in civil engineering projects that can't be passed without a professional engineer affixing their official seal to a document. If you are a professional engineer and you are found to have done this improperly you will lose your license, which is way worse than being fired because you won't be able to get a job anywhere else either. And civil engineers are indoctrinated into their field's ethics culture with lots of reminders of how many people died when current practices were not followed.
If a firm is found to have inappropriately (for example, pressuring a professional engineer to affix their seal inappropriately) they will lose their certificate of authorization and then not be able to do any engineering work.
> And civil engineers are indoctrinated into their field's ethics culture with lots of reminders of how many people died when current practices were not followed.
> The Iron Ring is a ring worn by many Canadian-trained engineers, as a symbol and reminder of the obligations and ethics associated with their profession. The ring is presented to engineering graduates in a closed ceremony known as the Ritual of the Calling of an Engineer. The concept of the ritual and its Iron Rings originated from H. E. T. Haultain in 1922, with assistance from Rudyard Kipling, who crafted the ritual at Haultain's request.
> The ring symbolizes the pride which engineers have in their profession, while simultaneously reminding them of their humility. The ring serves as a reminder to the engineer and others of the engineer's obligation to live by a high standard of professional conduct.
I didn't see it mentioned in the Wikipedia article, but i was under the impression that the first of these rings were made from the iron of a bridge that failed.
One huge issue with introducing licensing in software engineering is the fact that there are a great many very competent self taught software engineers. This breaks the conventional model of how licensing works with its close coupling to university degree programs.
The reason you see so many autodidacts is that its one of the few engineering fields where tinkering is both cheap and safe. Civil engineers basically can't tinker beyond maybe building models or using physically realistic simulations, and neither of those is a substitute for really building things at scale. Anyone with a decent computer can tinker endlessly with software with little or no consequence to anyone else when it breaks.
The only other engineering field like this I can think of is electronics, where you also have a lot of autodidacts because components are cheap and unless you are messing with super high voltages it's pretty safe to play around.
There is always a huge pushback against licensing in this field because it would be (accurately I think) seen as a way for universities to erect a mandatory toll booth.
The second huge issue is that we don't have a good theory of good programming practice. There was an attempt to systematize things in the 90s "design patterns" era and all it got us was a ton of monstrous overengineering satirized by the many "hello world, enterprise edition" joke code bases on GitHub. More academic efforts around provable correctness have never managed to cross the practicality gap for widespread adoption.
There is a third issue to licensing that dates back to ancient "law of strongmen" without even lofty claimed principles like divine right, let alone anything worth folllwing of free will; introducing a law that cannot be enforced only breeds contempt for the law. It is hard, expensive, and rare to covertly build infastructure. It isn't the case for software. Just blindly aping it would be draconian and silly.
At best it could be a license for specific safety critical applications but that would be dependent upon the licensed ones actually doing a better job at being safer which is harder for the first two reasons you listed.
Hell certifications are infamous for generally not being that career useful.
> One huge issue with introducing licensing in software engineering is the fact that there are a great many very competent self taught software engineers. This breaks the conventional model of how licensing works with its close coupling to university degree programs.
Uncle Bob argues that this is why we need to start self-policing and create certification structures that take this into account.
If we don't, the next software-related disaster will force governments to set up regulatory structures the only way they know how -- tied to university degrees. They won't understand the value of a self-taught developer.
We do. We should be the ones to create structures in which such people can be officially recognised for their abilities.
>Most business software, on the other hand, is treated as an operation. If you cannot discern the difference between a project (temporary) and an operation (living) your software will never be released as a product to a specified level of quality by a specific date.
I never really thought about that. There is so much truth to it and why so much Saas is a never ending feature factory! I assume Basecamp thinks of their Saas as a project and that's how they are able to escape the never ending Big Ball of Mud software craziness.
>Most web developers, on the other hand, require large tools to do theirs jobs for them and recoil in irrational fear and disgust when standard DOM methods are used
I would recoil in disgust if I saw a civil engineer building a suspension bridge by hand as well.
Most software projects are nowhere near the scope of a suspension bridge, and the DOM is certainly a lot higher level than "building by hand". Your analogy is terrible.
Abstraction can be good. But this reliance on abstraction that does nothing but obscure problems is a weakness of modern software development. And now we're left with no one who can solve them.
I agree with you in principal. When I worked in web development many of my projects were clearly defined and similar.
However when I got into more creative development I found myself faced with unique problems to which the solution was unknown. The development process in this case is less like an implementation and more a exploration of the problem space.
This process is more like sculpting or writing a book or solving a puzzle. As you sculpt the next steps reveal themselves. As you write a story the plot emerges. In order to solve a puzzle box you probe the box.
Solving the unknown in bridge building is working through sets of equations and to solve for variables. It involves doing math.
Solving the unknown in software involves a similar working through. "Doing the math" for code involves executing code. It's faster and easier. Ignoring this is like restricting your bridge planning by avoiding CAD or doing the math by hand.
Yes, once a solution is mapped out the ideal would be a full rewrite. The benefit of software is that it can be modified and finalized directly like baking a sculpt or revising a manuscript.
If a button allowed the working schematic of your bridge to magically materialize, and changing the schematic affected the real world bridge instantly with negligible implementation cost then civil engineering would evolve in the direction of software development. It's not the other way around.
>One difference is that a license is required to practice civil engineering
The world would be a better place if developers (or the company they work for) were as much on the hook for bugs in software as en engineer is for a bug in a bridge. How programming can still be the wild west in 2020 is beyond me. I've read some good thoughts about this topic by Poul-Henning Kamp (Varnish, FreeBSD) but can't seem to find it at the moment.
Also, physical engineering is constrained by a set of available materials, technologies, and physics. In software you could build a pretty accurate simulation of an entire bridge, generalize it so that it can represent all possible bridges, model new materials, and rewrite the rules of physics, the complexity space is infinite.
> Requirements change. Every software engineering project will face this hard problem at some point.
Rich Hickey calls this a 'situated' program in his talk from Clojure/Conj 2017, where the program has to deal with information and real-world irregularities for extended periods.
While agile disciplines help you from one perspective, your tools (e.g. program language) might also help you solve some of the problems. In this framing, Clojure certainly tries to help where the it matters most.
Highly recommended to read the transcript or watch the video, even if you are not interested in the language itself.
Just to nitpick, there are counter examples of finished but not abandoned software. As usual, Tex comes to mind.
Or they just could. E.g. we could probably take the current Python 3 and freeze it and debug it for 20 years, at which point it would be basically finished, and maintenance could still continue (but way much less would be needed)
IMO a good part of requirement volatility is a self inflicted wound. Some are essential, of course, but some changes are for the sake of changing and giving PM, UI, and software engineers jobs, and I even don't know lot of people who appreciate too many gratuitous changes in e.g. software GUI.
My general advice is to slow down. And fucking finish things before starting new ones. We don't let half-constructed bridges around just because we start to build new ones with fancier materials.
Probably that plays a role. However, not everything is a startup. Accumulating technical debt during even just a few years like there is no tomorrow is also a non negligible risk. You can also go from billion to zero in a few years...
1) Poorly communicated goals. Sometimes not communicated at all. If you don't know where you're head and _why_, it's damn near impossible to get there.
2) Wants being confused with actual business needs.
3) Clients / stakeholders not able or willing to put in the time for essential discussions and critical decisions.
There is a whole industry of people who are in the business of telling programmers what to do but not how to successfully complete a project because they do not know anything about how to actually finish the things they tell people to start. "Just use agile everything will be OK!"
Oh and lets not forget the project managers themselves that blame the client/users instead of their own inability to meet the requirements; http://owensoft.net/v4/item/2368/
Buildings certainly do change. I just finished 3 months of work on a structure that was "finished" 20 years ago. We had the client coming to us with changes to our changes several times a week.
The problem with software is that we don't build it in a way that makes changes easy (or at least not all the types of changes that clients frequently want to do), and we do a terrible job of communicating the costs of changes.
Give me any program, and I can change a couple words in its requirements which would cause more than half of its lines to need to be changed. That's just how we write software today.
(We diligently made 'time_t' a typedef, and then we embedded the physical size of it in every file format and API and protocol so we can't actually change it without breaking everything.)
Maybe we need a new type of specification system, to encode the distinction between "this is a decision which is immensely expensive to change and will necessitate rewriting half the program" and "this is just part of the facade and we can change it cheaply at any time".
By now, I'm rather convinced that the "core problem" of software engineering is that it's too easy. Yes, despite the myriad failures in delivery, once you actually get some working software onto the market, its value immediately multiplies as a sole function of your marketing (and, incidentally, I think this is one of the reasons why US based companies are more successful than e.g. European ones: they can start out with a big single market). And even when you're not profitable, VC money is being pumped into the most ridiculous of companies right now.
Of course, when faced with this kind of situation, why bother getting things right and professionalising? I've seen big companies with the most insane development cultures where things would constantly break, and yet it didn't really have any big negative impact on profits. Maybe the well will dry up at some point, but so far it doesn't look like it.
Requirements volatility does not fully explain the frequency with which basic security flaws keep reappearing, long after they have been recognized and documented.
250 comments
[ 4.2 ms ] story [ 245 ms ] threadhttps://vimeo.com/108441214
> There is really only one answer to this: you need to find a way to match the requirements-development-delivery cycle to the rate at which the requirements change.
Honestly, I believe that much of the problem is with the word “requirements” and I so wish we could kill this use of the word. Bottom line, for systems built for humans, ploughing through backlogs of requirements is the path to mediocracy (or worse) – as the OP discovered.
A less misleading framing: you need to find a way to match the delivery cycle to the rate at which understanding changes. In other words, your development process is a learning process.
Agile seems to be mitigating the customer not knowing what they want the software to do; tiny little software cycles, little steps, making the customer look at it frequently and saying what they think.
There are software projects with high-quality customers (sadly so rare) for which that risk is much lower, and as such agile is a bad fit. There are certainly software projects where waterfall is a better fit, mitigating against different risks.
Try selling a really agile methodology to a client of your own before you jump on the turnip truck
When done properly it gives the client immediate results with little up front commitment or risk and gives you all the control you could want as long as you don't mind being untethered from the communal trough and putting your reputation on the line at every iteration
I would say that agile has helped create the conditions for the proliferation of tech debt mills, because it empowers product people while disempowering engineers.
"Rapid and flexible response to change" is really just bruteforcing ideas leaving a trail of tech debt that makes developers miserable.
Why was I contributing to this madness when I knew better?
So I took the show directly to the client and never looked back
I'd love to elaborate but it sounds like you have a journey ahead of you before you can appreciate what I have to say
The difference is that the emphasis is on knowledge creation and relationship building
Believe me that there is a way to do the right thing by everyone but most people set themselves up for failure at the outset
It's both way simpler and much more complex than most people can truly appreciate
It means to change the approach while maintaining the vision and values
My values have always been on teamwork and knowledge creation and I pivoted in the sense of changing the approach but not in the sense of changing the vision
Thanks to agile I am directly involved in evolving the product I work on and providing information back to the product owner and anyone else involved. I regularly make suggestions that are listened to and I get feedback on my work several times a week. The changes I make are released to users rapidly and regularly and we gain insight from that to make our next decisions.
I also openly discuss the compromises I make in terms of technical debt and I'm given the freedom to clean it up when it makes sense for the business.
The business would not be able to innovate as quickly if it weren't for agile. I would be less empowered and performing suboptimal work if it weren't for agile.
Tech debt "just get it done" style management with egos too sensitive for candid feedback will always exist and the way the work is defined will have little effect. It's unfortunate that you work for such managers, but please don't blame it on agile.
If agile is done in such a way to amplify the voices of invested parties then I'm all for it -- starting with the team
Occasionally you'll come across somebody with a healthy dose of skepticism.
https://news.ycombinator.com/item?id=5406384
I believe the agile manifesto was written by smart people that knew what they're talking about. But what is known today as agile has nothing to do with what they really meant.
The agile manifesto says "working software over comprehensive documentation", but in practice, all agile projects I have seen put more emphasis on project documentation than working software: kanban boards, sprint burndown charts, the backlog, project planning documents... that's documentation.
The manifesto also mentions "individuals and interactions over processes and tools". Standup meetings, sprint planning, sprint retrospectives, etc. are processes. And agile practitioners I have met put more emphasis on those rather than doing something in the spirit of the manifesto, which is using reason and logic to collaboratively find solutions.
For me, the essence of agility is: break down large projects into smaller and smaller projects until the "mini-project" becomes simple enough for you to execute. But each such mini-project must delivery something that works. Also, don't show documents about the software, show the software.
The problem is of course, that a low effort design prototype still looks and feels like a low effort design prototype. But the customer sees 5 variants of the low effort prototype and can decide which one is iterated on further.
If you program a low effort software prototype, it should still kinda work and this is where the customer would decide to run with it.
Also software (like graphic design) is a field where people without any skill tend to have strong opinions about how things should look or work, which makes manipulazing the customer into taking a decent solution part of the job.
Business too. You do not have to be a good strategist to “succeed” as a VP in a medium or larger sized organization. In fact, you can be downright terrible.
I am slowly starting to get the idea behind medival guilds..
I've been thinking that this is one of the key differences between software and bridges. When you design a bridge, very few people have a "right" to an opinion on the bolt sizing. But when you design an order entry system, the business users (and users of the downstream data) have a valid right to an opinion.
This feels like if one would ask a electrician to use wires without insulation because they like them better — you can totally have your taste, but some stuff is just plain stupid and won't work as intended.
Graphics e.g. is about communicating certain things, and a graphic designers job is to be aware of what this communication does and how effective it does it. And so just like with a bridge (but arguably less dangerous) there can be design that does the job and design that does the polar oposite of what your customer intended.
A better engineering example needs to be completely one-off, like building a movie set, a stadium, or maybe an airport. These also have endless problems with the requirements changing during design & construction.
As somebody who worked as a freelance DOP as well, I think the software world could learn a lot by looking into other areas, especially when it comes to how requirements, changes in said requirements and the impact of these changes on the whole are communicated. IMO one of the most important abilities of an professional regardless of field is to be able to estimate how long something might take and how much work it might be. And this means also beeing able to say how much impact a change in requirements would have on the quality and deadline for the final product.
I know from experience that this estimation is harder in software than in nearly any other field, because problems that look simple on first glance might turn out to be nearly unsolvable, while other problems that look hard might have fairly straightforward solutions — and without putting time into it, it might be hard to tell beforehand.
If the topic at hand is really something completely foreign to you, be honest about it and tell them the time in which you will get it done for sure. If they don't agree to it, you wouldn't have made them happy anyways.
However: if you are not just a hobby developer, knowing at least roughly how long things take is key — you have to live from it after all: if you earn 1k for a job that takes you two months you cannot pay your rent. And even if you just do it as a side job, being able to give estimates still helps you to use your time wisely.
I don't think an dev without experience is expected to be able to do that. But a professional who has to live of their profession certainly should, if they want to survive.
I quoted your statement "If changing and complex requirements are a problem, you are doing it wrong" and pointed out that it was not reasonable to say someone is doing Software Engineering wrong due to changing requirements, as the engineers are seldomly in charge of requirements.
It is not so much that they are doing software engineering wrong due to changing requirements as that the changing requirements make clear that the quality of the engineering is not that great. Another engineer who is working with a higher quality perhaps could have handled the changing and complex requirements without much problems. And in another project the requirements are perhaps quite simple and there is not much requests for changes and both engineers would have done equally well because their skill was not tested much.
Of course, there also has to be some limit in changing and complex requirements beyond which it is no longer reasonable for anyone to be able to keep up...... but we do call it SOFTware as opposed to HARDware because it is supposed to be changeable.... and if that which is supposed to be changeable actually is not, there must be something wrong.....
Many times I had a clean elegant abstraction break with a change in requirements.
The hard part is that being adaptive to change increases the complexity exponentially with size (of project, of people, etc...).
Therefore the only way out of this formula is to reduce ‘size’, and iterate with (/alongside) the customer (even if that is an internal _customer_).
It is basic math(s).
People have known for a long time that volatile requirements are a core problem, but that typically hasn't really affected the way software and software development services are procured.
This might be different in other parts of the world, but my impression is that even custom development contracts are procured the same way that you would a big shipment of physical goods, which is just insane.
Once you work as a developer in a project that was procured in such a way, it's already too late. The knowledge that requirements will change and will totally disrupt your development process is there, and it doesn't really help you.
I work in a heavily regulated industry, and am dealing with a set of regulators nearly all who depend on just one particular auditing company to ... effectively write their regulations for them. To the point where a new jurisdiction happily refers to their draft regulations as documents prefixed with the auditing firm's name on them. (This is not a joke or hyperbole.)
That particular company has trained their own workforce to evaluate acceptance criteria according to their internal workflows and practices. This in turn means that almost all industry regulations are geared not towards functionality, but to make sure the auditing company has the least amount of training to do. As far as I am concerned, this is a particularly perverse form of regulatory capture: it's not even the industry at large writing their regulations, but an already entrenched gatekeeper acting on their own interest, crippling both the regulators and the companies in the industry.
Now, I have no problem with regulations as such, because for most parts the rules tend to make good sense. But I do have a problem with regulations shaped after the desires of this particular auditing company. Instead of focusing on results, quality, transparency, accuracy, recoverability and functionality, they have a fixated view on particularly dubious change management processes.
I am convinced that at some point in history, someone has done extensive research into 70 years of software engineering best practices - and gone to extreme lengths to expressly reject as many of them as possible.
Two examples: they insist that even unit tests MUST NOT be written by the engineers who work on the code; and they are mortally afraid of the idea that engineers who work on a problem could ever understand and evaluate the problem. (Not to mention that the very concept of a workflow tool is completely alien to them.)
The end result is that you can either try to do software well, or you can try to do it according to regulations.
The problem with missing business requirements are that developers often are either to focused on writing the code instead of actually understanding the business.
You cannot build something you do not understand.
Understand small and expand your understanding. Code is not the goal, it never is. It's a means to a goal.
Another thing about software is, that a lot of it should never ever be written. It's the last thing you do, is write code for overcoming a challenge, because rarely you really need it.
This is a multiplier because it allows them to detect and ignore non-problems and work on problems whose solutions _will_ deliver value.
Even you have a bulletproof requirements document, implementation and engineering still produces unforeseen eventualities and unexpected consequences.
I think software engineering is hard primarily because it's the coalface between abstract and concrete.
We're wired to be great at concrete, but capable at abstract. With software it's like we can get ourselves into a perfect storm of intractable-for-us problems.
Other forms of engineering, civil, mechanical, etc are still really hard, but I think we're better wired for them because they're less abstract, more concrete.
But in software, abstraction is part of your bread-and-butter daily-grind tool-kit.
Rewrites would mostly be successful by the articles premise.
The project was waterfall all the way.
The quality of software that can be built if you know before you start what it's meant to do, and that doesn't change, is astounding. Truly astounding.
Yes in some projects that can be harder, but I'm not buying the ambiant: "in most projects". I actually think in that in most projects, a good part of requirement analysis can be put upfront, and of course a tiny bit will still evolve, but not enough to prevent doing things properly.
That does not impose to have too long projects, btw. Just cut them in phases and/or pieces. Feedback cycles are important, but this is still an equilibrium, and in some projects I don't see how 2 weeks feedback cycles without proper requirement engineering would be any good.
Maybe one problem of software development is that it is an ever growing industry, so the mean programmer is always young and inexperienced. Maybe things will be less impulsive with the eventual end of that growth.
And it's hard do do a proper waterfall when you hardly have an idea of how to even approach the problem.
I think having all reqs upfront is a utopia that will not happen. Those projects I was involved in would have been better if the reqs and scope was done in conjunction with us implementing it.
I think having all reqs upfront is a utopia that will not happen.
What can I say? It happened to me. Only with one company in seven or so worked for so far. I would not be surprised if people worked two dozen jobs and never experienced it.
You do need high-quality customers, though, and they are hard to come by. I've worked for a company that has turned down customers on the grounds that the potential customer just isn't good enough; they worded it differently, but that's what it was.
Just like spaghetti programming, you get spaghetti requirements analysis & process-engineering. And even then, they hardly consider some of the things that developers do.
Almost as if the developers that encode the rules in a "hard format" such as a program/system need to be the ones dictating the processes/requirements. I would definitely see that as separate to "reqs and scope done in conjunction with devs developing them".
This project operated under some unusual constraints: the contract was denominated and paid in a foreign currency and was absolutely firm fixed-price, with no change management process in the contract at all. In fact, as part of the contract, the acceptance test was laid out as a series of observable, do-this and this-follows tests that could be checked off, yes or no, with very little room for dispute. Because of the terms of the contract, all the risk of any variation in requirements or in foreign exchange rates were on my company.
The process was absolutely, firmly, the classical waterfall, and we proceeded through the steps with confidence, until the final system was completed, delivered, and the acceptance test was, well, accepted.
After which I spend another 18 months with the system, modifying it until it actually satisfied the customers needs."
Ditty of mine from around 1988 or so. The hardware is old and obsolete but the software never is complete.
One thing I realized a long time ago was that you need enough resources to keep up with the requirements drift or you will never finish. That's what's behind YAGNI and Minimum Viable Product.
There is this other thing which is a successful product 'creates it's own weather' as far as requirements go.
Problems tend not to change so dramatically over the course of a project. What usually happens is some stakeholder realizes at some point that they didn’t actually understand the problem the first time around, and thus need to change the requirements. Even significant regulatory changes won’t usually cause that. In almost all cases, regulatory changes have a significant amount of time between announcement and enforcement.
Making your development practices more responsive to change is a good goal, but it doesn’t solve this problem. Because this problem is typically created before the first line of code is written. If the defined requirements don’t actually solve the customers problem, then they’re going to be delivered a non-functional solution, whether it takes one year to deliver, or one day.
> “Walking on water and developing software from a specification are easy if both are frozen.”
― Edward V. Berard
The more you plan ahead and the less you change goals while writing software, the less problems you have.
Software is the only field of engineering where you can change things at any point in time. But it doesn't mean you HAVE to change things. Software should really take lesson from other fields of engineering. Companies always want to cut costs, and with software, they really can.
I’ve always thought it is a little bit of a stretch.
It tends to make more sense for me. A book that starts out as a romance is just as hard to convert to a spy thriller half way through as a bridge and a tunnel (Ok not total effort but in terms of percentage of rework)
It's essentially applied physics.
Software is more like nailing things together in the hope they might work. ML has some modelling, and formal methods are available for mission critical projects.
But the rest is mostly nail guns and glue.
Engineers have never done themselves any favours with the "engineering" label. In many countries, especially English-speaking ones, the job doesn't get the respect it deserves.
Most people have no idea what the job involves. They literally think it means someone with an oily rag who fixes cars and/or computers and/or drives big machines around to make bridges and tunnels and such - like a house builder, but on a bigger scale.
* Mathematics isn't only about calculation and applied physics. When you write a simple algorithm and convince yourself with a logical argument about its correctness, time complexity, etc. that is math too. You also do calculations when you design various systems (latency, throughput, etc.)
* I know a dozen mechanical, structural, and electrical engineers. They almost never model anything from first mathematical principles, they use well known tools and methods they learned. Their "nail guns and glue". Actually I think most of them don't understand math/physics behind it very deeply. So even if those methods are underpinned with mathematics and physics, it's abstracted away most of the time which is good for practical purposes.
Even in "user-interface programming", I often had to use math, algorithms, etc. In other parts of software engineering I used it much more.
Depending who I talk to, I use "programmer" or "software engineer" most often.
I'd use "software engineer" if I could in good conscience, but "software developer" is pretty neutral while still being accurate.
Is indeed sad that people with no knowledge of thermodynamics or strength of materials can call themselves engineers as well...
There's not really the class divide in engineering fields that you're suggesting. I have a brother who is an ME, designs power plants (the building sized ones) and is not "professional".
I asked him about it and he said it wouldn't change his role or his salary, so he never bothered going through the process. IIRC there's one person on the team who is, for whatever formalities are required.
Is that person making more money? Because I've heard stories that these people do, because they are especially valuable to the company.
As so, I'd consider software development can qualify as engineering. If you're taking the work of computer scientists and mathematicians (or any other science fields) and applying it to software, it fits the definition. If you're just laying code to a given spec without much more thought, then you're just a programmer/developer...
I think that the engineering mindset (adherence to the scientific method, evidence based decisions, planning for the future etc..) is one that is definitely shared by both groups at least.
But you can absolutely get away with building software without proper engineering standards. It's not so easy to build bridges that way.
I'd wager that it is but the stakes for failure are too high to wing it (hopefully).
Imagine:
We want to build a bridge that spans N meters. We don't know how to do it but we can go look at an existing bridge and copy it. We might not understand metallurgy / physics that well, but we talk to suppliers and ask for "steel of this shape and size". Ditto cables, concrete and everything else.
Probably, provided we're not scaling up or changing the design too much, it would work just fine.
When software goes wrong, the effects are more often than not, intangible. When mechanical engineering or electrical engineering goes wrong, people can die.
Software can obviously kill people too, it just happens less. Then industry practices revert to the mean due to cost cutting.
It's totally possible. The only thing is that when the cost estimation arrives to the hands of decision makers, they open their eyes wide and decide that for that money they better go ahead with a bridge.
So IMHO it's just a matter of cost. Say you go with a bridge but there are some initial wooden stairs, and sometime along the middle of placing them they decide it is acceptable to pay the cost of stopping and replacing them with metallic ones. Could happen.
Biggest change I experienced in the rate of unexpected last-time feature changes in software, was having a project manager that first of all knew how to say "No" (or discuss the sense of the requests), and secondly, was able to communicate the real cost that each change would have (in terms of developer time, which at the end of the day means money). Then it would be the company's boss himself who would decide to discard the most crazy changes that were "mandatory" and coming from the sales people.
In a large corporation, the software engineer expects to be told that there are going to be cars on this bridge, and they are moving at a certain speed from one side to the other and that they will be of a weight between x and y, etc...
Understanding the domain goes a long way to get software done.
Or where it should be exactly: "It should be easy to move once it is build, right?"
Or what type of vehicles it should handle: "Just build a generic bridge. How difficult can it be."
But the time and monetary budget was already negotiated and written in stone: "Oh btw, you got 2 weeks and $537.25 to finish the bridge."
Even the language and best practise change constantly. Looking at possibly the most stable language C, we have entirely different best practise pattern from community to community.
New stuff shows up, but that doesn't mean you are compelled to leap upon it.
Turns out that every field of engineering is hard!
[1]: https://www.fastenal.com/en/70/corrosion
I think your example points out exactly why some think software engineering is a joke: they rush ass first to implement something (and they implement the wrong thing) instead of sitting down and figuring out the actual requirements first like any real engineering job would have you do.
And it looks like everyone is pushing it more into this ass-first direction with agile. Let's not even try to plan ahead! You know what you want next week, so this short-sighted focus leads to chaotic development and once the real requirements are figured out 9 months into the project, well too bad it's too late to redesign and rewrite it all properly. So it's forever going to be a bumpy and crooked bridge on the verge of collapse, held together by more and more duck tape, hot glue and occult ceremony. And there's a plastic tunnel hanging underneath it. Only the brave will crawl through.
Because it's not about bridges or tunnels, common sense is thrown out of the window, people start building something they conjure out of thin air.
Few people say 'No!', we don't do anything. Anything. Until we have written a clear business case / usecase, a raison d'être for this project.
Agile is often abused to not think, not design and just follow the whims of the customer. Nice for billable hours, but long term?
Do you produce anything of value?
If it were so easy, then we would have done it already that way for the last seven decades. The simile topples over upon further contemplation:
civil engineers:
• client is not a domain expert
• to an overwhelming part, client needs are easy to transport into the mind of c.eng.
• can employ a wealth of standard solutions refined over the course of milleniums
software authors:
• client is a domain expert for the subject matter that the software is supposed to model
• to an overwhelming part, client needs are difficult to express and transport into the mind of s.auth.
• if a standard solution exists, the client would have already bought it off the shelf, so coming to a s.auth. always means customised development
• the entire sector is still in its proverbial baby shoes
We have to come to grips that we unfortunately can't apply what happens to work well for a different sector. The supposed value of non-traditional software methodology lies in that at least the client notices that the requirements are off the needs rather early.
So instead of pushing for more effort & skill for up-front planning and modelling of the requirements and design (and trying to employ modelling tools & formal methods), we're just like fuckit nah let's just implement something and see where that leads us.
It doesn't help that so many of the up-front design failures were not examples of engineers screwing up but sales people selling shit and then having engineers cook it.
We'll be stuck in baby shoes for a long time with this approach.
My company just shut down an entire office, they offered three out of four critical employees the opportunity to move to Atlanta. They didn’t want to move, opting to retire instead. Company just shrugged their shoulders rather than letting them work remote, gave them like a month to transfer knowledge and that’s that.
In software if you know what you are doing ahead of time you are not really innovating and you might as well be using an existing system. Since everything that is standardised you can just use as libraries.
When building bridges, you can't just use a library that's .buildBridge(...parameters), even though similar things have been done thousands of times before. With code, if something is done enough times that you know how to plan for it, it is likely already a library.
You can plan building a bridge since you have done that and have experience of it thousands times over.
"Everything was waterfall and then came agile" is myth, but yes, it looks like it's fashionable to no longer try to plan ahead. I'm not convinced the reasons are good, and I'm not convinced the craft of planning ahead was ever perfected to the point that anyone could say it doesn't work.
Scrum, XP, Agile all date back to the 90s when one server running PHP or Perl (with SQL injections) passed for software and others were still re-inventing basic data structures in C (and then gasp C++ with templates!) from first principles... The methodology, tooling, rigor, and collective experience in software development has taken leaps in the past two to three decades. Likewise, understanding systems and planning ahead has become so much easier. If only people took it seriously and spent as much effort on it as they spend on fad-of-the-year frameworks.
People just stopped trying, and when I look in software projects now, there's hardly ever anything resembling a model of the software that you could use to test new ideas "on paper" to see how they fit the given problem and current application.
People aren't even trying, just like they aren't even trying to achieve correctness: https://news.ycombinator.com/item?id=22224054
The less you try, the harder it will be.
> In software if you know what you are doing ahead of time you are not really innovating and you might as well be using an existing system.
I think the vast majority of software is boring and not even meant to be innovative and indeed would be best written using an existing system, but customizing these to the specific needs is just another can of worms. As with bridges, you can't just copy & paste an old bridge to a new location and use it as-it is, you need to design it given the locale and other constraints. It's still probably going to be yet another relatively boring bridge with no innovation.
And indeed a lot of software work is all about using existing systems, but perhaps at a lower level than you posited. Customizing complete off-the-self packages does happen, but it's more common to glue something "new" (but not innovative) using existing frameworks, libraries, services and glue languages.
You still need to design to do it right. But even here people seem to rush ass-first into whatever kit looks fashionable now.
Yes, people don't do their jobs. That's not the fault of the process that they are failing to adhere to. Indeed, no process of any type could ever save them. It's the culture that is wrong.
So, in a roundabout way, I agree that Agile isn't the solution. No bottom-up process will ever succeed, because at the end of the day, the failure of the team is always the fault of the leader.
The last time[0] I vented on the subject, I had to dig out the research papers I keep on my desk, because there was more than passing curiosity. Oh yes. I keep these on my desk so I can quote them when needed.
Check out this research paper from your nearest sci-hub entry point:
- DOI: 10.1109/MC.2003.1204375 ; "Iterative and Incremental Development: A Brief History"
---
0: https://news.ycombinator.com/item?id=20563125
Often times, it's very easy to determine the functional requirements: "Process this billing statement", "Spit out that report." The complexity doesn't come from the requirements. It comes from the constraints: "We only use this ancient version of this database", "We didn't tell you at the start that the users expect to interface with this system via email".
To quote Mike Tyson, "everyone has a plan, until they get punched in the face." The point of Agile is to get punched in the face earlier and more frequently.
Those constraints sound like requirements: we require that only this ancient version of this database is used. We require that users can interface with this system via email. Different term, same thing.
That's in the bucket with the kind of stuff that I'm advocating people should pay more attention to up-front. Gathering these requirements before you start designing anything is a big deal. If you failed to do so, then you failed to do a proper job at the part I called sitting down and figuring out the actual requirements.
This is something that crops up quite frequently in my work. We'll get a requirement like this: "the app should use HTTP like a web browser, not a different protocol". On inquiry as to why it's so important for something that's not a web browser to so strongly resemble a browser, the answer will be something like, "so we don't have to ask IT to open a firewall port". The person deploying the software could just fill out the paperwork, but they find it easier to "require" that the software sneak past their own corporate security controls. This is the sort of requirement that is real in some sense and not in another sense, depending on whether you define the customer as whoever you're talking to on the phone right now or the business as a whole.
And even when you do have a solid market-fit figured out, you still don't know what your customers will actually want. (And don't get me started on customer research. People lie all the time, and their accuracy on what features are honest-to-$deity blockers.. Nope.) This means that for most parts the incentives are misaligned and work actively against delivering high-quality software.
Underlying the agile manifesto is the need for engineering to be properly aligned with business incentives. Continuous delivery, fast iteration cycles, incremental improvements and rapid deployments are all facets of the same fact: you don't actually know what your customers want (because your customers don't know what they want!), and the only true way to figure it out is to iterate as fast as you can.
It's only when you have an established moat and a practically guaranteed, net-profitable income stream, when you can even imagine at having the opportunity of doing things the way you would assume you want to.
Oddly enough.. the companies who have such moats and guaranteed profits tend to be the same ones that have been the slowest to modernise themselves. Banks - until the modern online-only challengers came along. Insurance, where the regulatory moats are even higher and is only now being "disrupted". Loan brokering. Energy. Water. Public transport (where monopolies and/or charters prevent competition). Healthcare. On and on and on...
And even then, every one of these established, guaranteed-profit machines has the same problems: the moment one good new player enters the field and starts stealing customers, the old guard needs to find a way to either adapt - which is the best-case scenario - or rely on their power structures with politicians and quickly require new regulations to strangle the usurpers.
Software is expensive to make. It's only cheap to deliver. And time spent making the wrong thing is really expensive.
EDIT: I forgot to say this the first time around. Engineering career incentives are also misaligned. Maintenance is not appreciated, only shipping new stuff is. So the developers who would want to improve what exists and make it better will be looked down upon. The promotions and raises go to those who ship something new, even if it ends up setting the world on fire a year later.
I do agree there are a lot of bad practices in SE as well, but let's not blame everything on them.
It's like sales people together with the customer decided the shape and colour of the bridge, as well as the number of pillars it has, and how many lanes it has, and what material it's made of -- here's the stack of things you build with -- and what other fancy features it has, and now finally we can call the engineer so they can hop on a bulldozer and start engineering with their hammer for the next two weeks and show the result so we can adjust the shape if it's not as pretty as we imagined. Actual engineering didn't happen.
https://www.scruminc.com/definition-of-ready/
Features (or epics or whatever) are larger and contain multiple user stories. In most cases features extend over multiple sprints.
There are no "actual" requirements for most software projects. Software is generally co-evolved with the audience.
Bridge designers can work like they do because bridges are hard to change and fundamentally pretty similar. But because software infinitely changeable and infinitely copyable, our field is different. If you're going to build 500 highway overpass bridges, you'll need engineers looking at each one, because local conditions and needs vary. But in software if we need to serve 500 users, we build one product, give each user a copy, and just keep adding features to solve the local problems.
A big reason we think software can be like bridges is that in the early history of our profession, we were tackling problems that almost worked in a waterfall context. That's because we were taking existing paper processes and automating them. But the hard part of requirements discovery via process coevolution had been done for us by people tinkering with paper and human working relationships.
That era is past. Today if the requirements for something are perfectly knowable, that's because it's either trivial or it's copying an existing piece of software. Any real product is discovering what's possible. It's growing along with its users' needs. It's solving new problems created by other people and groups.
I know of no significant piece of software that, like a bridge, hit 1.0 and just stayed there for a decade while getting plenty of use because the requirements were perfectly foreseen. Not one. To me that's a sign that perfect foreknowledge is, if not impossible, at least incredibly rare. So I think the thing to do is to figure out how to build excellent software with the expectation that needs will change.
As an example, the IRS is still running 60-year-old code: https://www.nextgov.com/it-modernization/2018/04/irs-60-year...
Is that because that code and hardware is especially good? Because the initial design and research process was so perfect that people are still entirely happy with it? I'm guessing not.
The process is nothing surprising. Write tests, document your code, follow the style guide, etc.
If you don’t even ask the users, it’s even worse.
But that’s okay. It is simply not possible to accurately judge how something will work until you get to use it. That goes for the creators, too. That’s why we iterate.
The PE exam is only needed for a handful of disciplines, like civil and HVAC engineering. Even then, you only need the PE certification if you are going to be signing legal documents. You can be a civil engineer without a PE working in a team producing designs and plans that are signed by a single certified PE.
By your logic almost no one that graduated from an engineering discipline would ever be able to call themselves engineer.
You can’t compare against paralegals/lawyers because calling yourself a “lawyer” has legal implications, as in, you must be certified by the bar or you can get in trouble. It doesn’t matter which legal discipline you practice, you must be certified by the bar.
For engineers no such thing as a bar association exists for ALL disciplines. The PE certification only covers a handful of engineering disciplines therefore it can’t be a gatekeeper for all engineers. Moreover, note that certified engineers are called Professional Engineers, not just Engineers! Even the name distinguishes between certified/non-certified, it doesn’t try to take over every meaning of Engineer.
It's not that software engineering is a joke it's that you don't understand the difference between computer work and moving earth/pouring ckncrete. Of course people are going to plan the project differently in civil engineering. The consequences to building any part of a bridge wrong are pretty dire and redoing work costs much more than it did initially.
Maybe don't try to draw parallels with other forms of engineering just because it has the same name in university. The practical differences are large.
I saw this yesterday and didn't know if I should laugh or cry, starting from the jingle at the very start of the video: https://www.youtube.com/watch?v=YJGCZCaIZkQ&t=20m06s Imagine an architect conference where a vendor proudly expains how in a building the average tenant lifespan increased a lot -- after their automatic turret was moved from lobby to a less frequented room in the basement. I know that's the opposite of a hardcore SW engineering talk, but still. Everybody wants to be nice, nobody wants to get blacklisted, kool-aid makes the dissonance go away. The main issue was that Linus sometimes called people idiots, and that's solved now.
Maybe other professions also started out this way, as cliques of fools making policy based on what they dreamed last night or how the birds flew, with the main driver being greed -- but being hundreds of even many thousands years more developed, they are on a totally different level than software engineering is today, and just calling it something noble and serious doesn't make it so.
One difference is that a license is required to practice civil engineering. Most web developers, on the other hand, require large tools to do theirs jobs for them and recoil in irrational fear and disgust when standard DOM methods are used. I still have people argue with me that the DOM is slow even though all the evidence and browser vendors claim updates to the DOM is simply switching a bit in memory. In other words many web developers have utterly no idea how their platform works.
Another difference is that many developers cannot write. Everything about a bridge is planned in extreme detail before work on the ground ever starts. This level of planning is absent in software. Documentation is also largely absent. In most cases everything is an ad hoc mess. In any other industry this would be a cause of numerous lawsuits.
Another difference is that construction is treated as a business project with a dedicated project manager and project budget. Most business software, on the other hand, is treated as an operation. If you cannot discern the difference between a project (temporary) and an operation (living) your software will never be released as a product to a specified level of quality by a specific date.
I've often wondered about this. Who exactly has to be licensed? Are there management layers in civil engineering firms where the license isn't required? I think my point is pretty easy to follow. I'd love for all of us to be licensed to practice software engineering, but I don't hold any hope that it would fix all of the other compounding problems that our industry has as it relates to product/project management and management-born issues.
For example, would being licensed stop executives at a triple-A video game company from prematurely announcing games / release dates, or accepting pre-orders for games that won't be finished on time, and forcing all of the people under then to scramble and crunch and do things that would generally be considered anti-thetical to whatever being licensed in software engineering could mean?
Work for compensation.
There's a reason we have hard standards for being a PE or an MD. If you just let anyone do it you risk many problems.
Most software simply isn't like this. The valley loves talking about disruption and all and how world changing everything they do is, but the fact of the matter is social media isn't killing anyone like a failed bridge would or a person who isn't qualified as a doctor. Most of the money made in the valley is off of advertising or data. Who cares if the product is shitty? people don't die. rich people make a little less money.
A bridge hosts what? 400 people at worst (i.e. when there äre passenger trains on it.) Social media easily hosts 400 thousand or million people.
The single bridge-collapsing event is very dramatic and visceral, but digital platforms allow us to cause damage and death by a thousand cuts.
If a firm is found to have inappropriately (for example, pressuring a professional engineer to affix their seal inappropriately) they will lose their certificate of authorization and then not be able to do any engineering work.
E.g. the Iron Ring: https://en.wikipedia.org/wiki/Iron_Ring
> The Iron Ring is a ring worn by many Canadian-trained engineers, as a symbol and reminder of the obligations and ethics associated with their profession. The ring is presented to engineering graduates in a closed ceremony known as the Ritual of the Calling of an Engineer. The concept of the ritual and its Iron Rings originated from H. E. T. Haultain in 1922, with assistance from Rudyard Kipling, who crafted the ritual at Haultain's request.
> The ring symbolizes the pride which engineers have in their profession, while simultaneously reminding them of their humility. The ring serves as a reminder to the engineer and others of the engineer's obligation to live by a high standard of professional conduct.
I didn't see it mentioned in the Wikipedia article, but i was under the impression that the first of these rings were made from the iron of a bridge that failed.
The reason you see so many autodidacts is that its one of the few engineering fields where tinkering is both cheap and safe. Civil engineers basically can't tinker beyond maybe building models or using physically realistic simulations, and neither of those is a substitute for really building things at scale. Anyone with a decent computer can tinker endlessly with software with little or no consequence to anyone else when it breaks.
The only other engineering field like this I can think of is electronics, where you also have a lot of autodidacts because components are cheap and unless you are messing with super high voltages it's pretty safe to play around.
There is always a huge pushback against licensing in this field because it would be (accurately I think) seen as a way for universities to erect a mandatory toll booth.
The second huge issue is that we don't have a good theory of good programming practice. There was an attempt to systematize things in the 90s "design patterns" era and all it got us was a ton of monstrous overengineering satirized by the many "hello world, enterprise edition" joke code bases on GitHub. More academic efforts around provable correctness have never managed to cross the practicality gap for widespread adoption.
https://www.isc2.org/Certifications/CISSP/experience-require...
At best it could be a license for specific safety critical applications but that would be dependent upon the licensed ones actually doing a better job at being safer which is harder for the first two reasons you listed.
Hell certifications are infamous for generally not being that career useful.
Uncle Bob argues that this is why we need to start self-policing and create certification structures that take this into account.
If we don't, the next software-related disaster will force governments to set up regulatory structures the only way they know how -- tied to university degrees. They won't understand the value of a self-taught developer.
We do. We should be the ones to create structures in which such people can be officially recognised for their abilities.
I never really thought about that. There is so much truth to it and why so much Saas is a never ending feature factory! I assume Basecamp thinks of their Saas as a project and that's how they are able to escape the never ending Big Ball of Mud software craziness.
I would recoil in disgust if I saw a civil engineer building a suspension bridge by hand as well.
However when I got into more creative development I found myself faced with unique problems to which the solution was unknown. The development process in this case is less like an implementation and more a exploration of the problem space.
This process is more like sculpting or writing a book or solving a puzzle. As you sculpt the next steps reveal themselves. As you write a story the plot emerges. In order to solve a puzzle box you probe the box.
Solving the unknown in bridge building is working through sets of equations and to solve for variables. It involves doing math.
Solving the unknown in software involves a similar working through. "Doing the math" for code involves executing code. It's faster and easier. Ignoring this is like restricting your bridge planning by avoiding CAD or doing the math by hand.
Yes, once a solution is mapped out the ideal would be a full rewrite. The benefit of software is that it can be modified and finalized directly like baking a sculpt or revising a manuscript.
If a button allowed the working schematic of your bridge to magically materialize, and changing the schematic affected the real world bridge instantly with negligible implementation cost then civil engineering would evolve in the direction of software development. It's not the other way around.
The world would be a better place if developers (or the company they work for) were as much on the hook for bugs in software as en engineer is for a bug in a bridge. How programming can still be the wild west in 2020 is beyond me. I've read some good thoughts about this topic by Poul-Henning Kamp (Varnish, FreeBSD) but can't seem to find it at the moment.
Rich Hickey calls this a 'situated' program in his talk from Clojure/Conj 2017, where the program has to deal with information and real-world irregularities for extended periods.
While agile disciplines help you from one perspective, your tools (e.g. program language) might also help you solve some of the problems. In this framing, Clojure certainly tries to help where the it matters most.
Highly recommended to read the transcript or watch the video, even if you are not interested in the language itself.
Video: https://www.youtube.com/watch?v=2V1FtfBDsLU Transcript: https://github.com/matthiasn/talk-transcripts/blob/master/Hi...
Or they just could. E.g. we could probably take the current Python 3 and freeze it and debug it for 20 years, at which point it would be basically finished, and maintenance could still continue (but way much less would be needed)
IMO a good part of requirement volatility is a self inflicted wound. Some are essential, of course, but some changes are for the sake of changing and giving PM, UI, and software engineers jobs, and I even don't know lot of people who appreciate too many gratuitous changes in e.g. software GUI.
My general advice is to slow down. And fucking finish things before starting new ones. We don't let half-constructed bridges around just because we start to build new ones with fancier materials.
1) Poorly communicated goals. Sometimes not communicated at all. If you don't know where you're head and _why_, it's damn near impossible to get there.
2) Wants being confused with actual business needs.
3) Clients / stakeholders not able or willing to put in the time for essential discussions and critical decisions.
If bridges could be replaced in minutes for pennies, we’d build them iteratively too.
The problem with software is that we don't build it in a way that makes changes easy (or at least not all the types of changes that clients frequently want to do), and we do a terrible job of communicating the costs of changes.
Give me any program, and I can change a couple words in its requirements which would cause more than half of its lines to need to be changed. That's just how we write software today.
(We diligently made 'time_t' a typedef, and then we embedded the physical size of it in every file format and API and protocol so we can't actually change it without breaking everything.)
Maybe we need a new type of specification system, to encode the distinction between "this is a decision which is immensely expensive to change and will necessitate rewriting half the program" and "this is just part of the facade and we can change it cheaply at any time".
Of course, when faced with this kind of situation, why bother getting things right and professionalising? I've seen big companies with the most insane development cultures where things would constantly break, and yet it didn't really have any big negative impact on profits. Maybe the well will dry up at some point, but so far it doesn't look like it.