This http://i.imgur.com/UW5rM.png I truly believe that most software developers want to solve problems, and establishing a database error is a problem that people are willing to pay for...thus, there is a class of people that are willing to do the work!
EDIT: I'm sure the db connection will come back up, but this error is the root of this discussion, in my opinion. No one wants to be yelled at, so to prevent this, we revert to what "just works" and your idea of that concept is different than mine, thus the more we can agree on standards the better off we will be for the general use case.
To use the plumber analogy, we can't agree on how big the common pipes should be. They have physical limitations set by properties of water and waste. Software handles a liquid of constantly changing viscosity.
> Plus, in the history of the world, he said, is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex?
It is a testimony to the innovation in any field that leads to tools become outdated so quickly. Easy commoditisation is directly related to the tools one has at their disposal. Creative fields like photography or movie-making also face the same issues.
Interestingly, if we are given a choice to constantly upgrade our tools then we would always prefer to. For eg: I don't find most people using the cork screw on a swiss-army knife. I would rather have a lighter multi-tool. But redesigning, testing and manufacturing a new multi-tool takes a long time.
Software has relatively speaking a much shorter lifecycle. Most of the clients recognise this and sometimes have higher expectations of the turnaround times. This leads to unintentionally short timelines or bug-ridden software.
As in any uncertain venture, it is often better to go with iterative and agile methodologies rather than a big bang approach.
Wow, thought-provoking post! I had three reactions to this.
1) Software is in it's early days (in the grand scheme of things.) Though things may seem "hand-made" now, it won't always be the case. In fact I'd say OO, design patterns, frameworks, cloud technologies and more are the early equivalents of regulations, well-accepted standards, etc.
2) What other profession is as egalitarian? What other profession has as much upside? Entire industries can be changed by a few smart people. That kind of opportunity makes the "hassle" worth it.
3) The hassle of having stakeholders/customers constantly wanting things faster and cheaper can easily be mitigated by being picky about what company your work, for, it's business model and it's culture. Not all software companies face that issue.
I'm not sure I understand the author's definition of "handmade." My Rails web app runs on my handmade code with a bunch of handmade Ruby libraries, on the handmade Ruby language, on physical hardware that was made by handmade machines (or machines made by handmade machines, etc.). How is software any more handmade than any man-made technology, like a cellphone or automobile? The distinction sounds necessarily fuzzy.
I'm equally bewildered by that. Everything mass-produced is originally designed in CAD, by a person. Those things are more like software than the items made one at a time by craftsmen. It's a false dichotomy.
My dad would start yelling at the screen if he read this article. He runs power tool manufacturing business, that requires each product to be tested and tested to perfection. If there is a fatal flaw the entire line may need to be recalled.
The fact I can push a software update to a million users in minutes makes him green with envy.
So yes software is hard... but cost free replication is an enormous upside.
Not every piece of software is like that. My updates take time measured in months from the moment developers declare them "done" till the first time a line of code is executed in production. And this is assuming all forks on this path go positive.
Not that I'm complaining, just giving some perspective. Maybe this will make your dad less jealous :)
Sure I can completely see that being the case in many situations (for example all Aircraft software engineers have my utmost thanks :-)), but even then the cost of prototyping changes or testing fixes is made much easier by the malleable nature of code.
Except that software is written by machines these days. I stopped writing in assembly language after it stopped being cool in the 90's. Even still, you'd have to be hand-crafting machine code in a hex editor to really get all the machines out of the loop.
These days I write in an interpreted language using complex libraries that handle a multitude of protocol choices for me. Even if you argue that each of those tools were at one point hand-crafted, even down to the compiler, there are now development tools that help me write code, from basic code completion and tooltips to the most complex static and runtime analysis. That is machines writing software, with a human in the loop.
Exactly - this article makes the very common (still) (1) mistake of thinking construction of software == writing code. Construction of software is compiler/interpreter/all the machinery that kicks in to take some text or instructions and makes it do something. To correct the analogy with building physical things - you could compare the architect designing a bridge (and arguing with engineers about materials and budgets) with writing code - that would be a fair comparison. The construction of software has been automated for some time.
This works just fine for construction of physical things because the cost sunk into to the "non construction" bit is tiny compared to the whole project - so no one spends too much time thinking about methodologies and automation of an architect coming up with the concept of a building.
(1) I note in the article the person mention spent time as a consultant in a big consulting company - this view is still held by them at least with the last brush I had with organisations like that (and they are incredibly frustrated by it).
Or alternatively you could say writing assembler is the software equivalent of building physical things by putting together individual molecules. Physical builders work at a higher level of abstraction too.
I think the analogy is becoming a little stretched though.
In the plumbing example in the article, the plumber isn't first making pipes from raw materials. They get them pre built in standardized lengths, diameters, thicknesses and materials. Same can be said for the compiler providing a standard, generally agreed upon abstraction.
Wow, the pipe-building factory (generating standardized parts to be assembled by the plumber) to compiler building (generating standardized machine code to be assembled by the high level language programmer) comparison is really cool!
I feel like the ISO/ANSI etc standards-making bodies are where the analogy breaks into time and space.
In one sense there _is_ this part of "constructing software", and _largely_ it can be done by the software equivalent of stereotypical "construction workers". (This is what a lot of people who've tried outsourcing to India are trying to do.)
The problem is, while you can collect a pickup full of Mexicans who can lay bricks / hang sheet rock / tar roofs on most street corners in the south of The Mission, and they'll do a great job of it if you give them good directions - you don't expect those guys to be making architectural or structural decisions, or zoning or permitting or code decisions.
"Code writers" have to make those sorts of decisions every day - a current high-profile example is Marius Milner and "his" decision about what data Netstumbler should collect from the Streetview cars. One of the biggest software companies ever, having ethical/legal/policy decisions made by the coder-on-the-spot (at least if you believe Google's representations on the topic). Or Apple with Lion debug-logging clear text passwords for FileVault, and having it escape "into the wild".
The "architect" and the "civil engineer" and the "structural engineer" who have important roles in the world of building physical things, the guys who sign off on bridges or tunnels or even just-repaired airliners, the guys who put their careers on the line when they sign the paperwork, the guys with qualifications and certifications and often indemnity insurance to satisfy society that they understand the risks - for the vast majority of software/websites discussed here that's reasonably likely to be a 22year old college dropout aiming to be "the next Zuck". Even in small and medium enterprise sized businesses, those roles are largely thrust upon whichever developer seems to be good (and doesn't duck their head quickly enough). And if the shit hits the fan, they say "sorry boss, it seemed like the right answer at the time" (and hopefully doesn't get hung out in the press like it seems Milner has been…) (And the "big consulting companies" mentioned in the post I'm responding too, in my limited experience they often seem to want to make all the architectural/engineering/policy/ethical/legal decisions, then leave with their paycheck before the "codemonkeys" implement it all, and not be contactable when their "solutions" turn out to be incomplete/contradictory/impossible)
I _hope_ government regulation of "software construction" isn't the answer (at least not for software that'll just cost investors money when it fails, as opposed to bridges or airliners that'll kill people), but I think lines of responsibility and authority need to be more explicitly identified in many software projects, with appropriate authority conferred on the people burdened with the responsibility. Holding developers to deadlines without giving them the authority to adjudicate on them or be involved in the determination of them, is a startlingly common way to have your developers cut corners - and worse, feel entirely justified in cutting corners and convincing themselves they're "doing the right thing".
The grandparent comment explained clearly why this is. It's because the work that can be done by "codemonkeys" who will hang sheetrock if you give them good directions, but can't make architectural or structural decisions, is already being done by the compiler and other such software, so the only work that's left for humans is the stuff that involves making those decisions.
You could probably regulate "software construction" and mandate specific methodologies, but we can see already what the result would be: just look at the SEI CMM Level 5 Certified software development teams that already exist: Wipro, R Systems, and so on — technically inept companies that only exist to rip off clients who don't know any better.
Any country that mandates that kind of development for all software will be rapidly left behind by the countries that don't as software becomes an increasingly important part of the 21st-century economy. They'll still have human beings laying their literal bricks and tarring their literal roofs, while the rest of us are living in robot-built houses full of fountains and sculptures, or dynamically-reconfigurable programmable houses.
The excellent book "Dreaming in Code" (1) follows a software project that takes way longer than expected, keeps changing goals, and is generally a mess. The author says "If the subject of software’s flaws is discussed for more than a few minutes at a time, it is a certainty that someone will eventually pound a fist on the table and say, “Why can’t we build software the way we build bridges?”". Then he shows how the San Francisco Bay Bridge project ended up going very much like a software project, complete with major changes in specs years into building.
And of course, since I'm in Boston I'm required to mention the Big Dig (2), which was a tunnel and bridge project that cost over $14 billion. Oh, and a ceiling panel fell, killing a woman in her car. And the guardrails in the tunnels tend to kill motorcyclists who would otherwise suffer only minor injuries. Plus the all 25,000 of the 120 pound (55 kg) light fixtures in the tunnel ceilings have to be replaced lest more of them fall, maybe killing more people.
But yeah, let's keep trying to make software engineering just like civil engineering.
>>These days I write in an interpreted language using complex libraries that handle a multitude of protocol choices for me.
Those days are gone too. In fact currently you just have to know how to press ctrl+space at the right time and wait for intellisense to do the magic. Nearly 99.99% of the Java world works like that.
In other words, these days we learn how to learn tools that write programs.
I think wanting a product cheaper and faster is a pretty universal sentiment. I think it can also be said that aside from cooking a meal or buying IKEA furniture, the typical consumer purchases a product that's ready to go with no assembly or lead time required. I like to think that if my (ordinary non-commercial) client assumes something is easily done, that's a compliment to my work. He or she can ask for a new button with a simple function to be added to a page, but they have no comprehension of the complexity behind adding what they see to be a simple element. To the ordinary or non-technical person, there is a lot of "magic" behind what we do (it's just a button, why can't you just add it in?). That lack of understanding combined with how easily we've solved other (seemingly similar) problems gives them no way to properly gauge how long a task should take or what it should cost.
The analogy the OP establishes - building a house vs making software - is good (I've actually used it too in debates), but things are not as tightly regulated by the government as the OP claims:
* While there are certain constraints a house has to satisfy, architects do have freedom to be creative. They MUST plan such that it will resist certain standard incidents, and they have to plan it such that the floors can carry ~10x the weight you would ever expect there. These regulations vary depending on the region you live in. E.g., in Austria there is the possibility that meters of snow lie on roofs, and architects (or their structural designers) have to keep that in mind when calculating the max weight. You won't find this constraint in Portugal.
* Keeping to these constraints does not guarantee a building won't break. Constraints (like requirements) are prone to grow outdated, or be incomplete. In Austria, roofs will sometimes break down when it snows for some days in a row (4m of snow are very heavy).
* As incidents in e.g. Turkey and China show, plumbers or other construction workers can do such a bad job ("botching") that buildings just break down after some years. Governmental constraints don't assure quality. Some countries like Dubai won't let local workers unguarded near any expensive building for this exact reason - most of the times, German and other Western workforces are hired to oversee the construction process.
I think one could compare governmental constraints on construction work with requirements. A building's floor must be able to carry a weight of 100 tons, and a software system should be able to serve 1000 requests/second - I think that's comparable, and software constraints might even be easier to test.
What I think this really boils down to is that software requirements evolve much more quickly, and due to this it's not feasible to establish legal requirements - government would have to issue new requirements every week, adjusting them for technologies like nodejs. Also, how do you establish best practices for bleeding-edge technologies within weeks? Not at all. We are in a constant state of learning and experimenting.
To date, software engineering has failed to deliver on one critical goal, which is why we are in this mess.
The world of software development needs to be divided in two -- the component creators, and the component assemblers.
To stick with the given analogy, component creators are the people inventing new kinds of plumbing: easier ways of connecting pipes, taps that don't ever drip. Component assemblers are the people fitting out bathrooms.
Creating new components is high-end engineering. It needs to happen far away from the day-to-day challenges of making a client happy.
To a large extent, this division is already present, but it doesn't go far enough. There has been amazing progress - nowadays we work on top of an incredible stack of technology that we don't have to re-invent, but so far we've not achieved the "last mile".
We'll know when we're there because component assembly (i.e. making something for a client) will start to look more like a trade.
Today, just getting a regular been-done-a-million-times-before database-driven website (or whatever), requires FAR too much low-level code. This, I think is what is meant here by "hand made". We should be snapping things together, and often we are, but suddenly you get to an awkward bit and you're back to forging a new kind of pipe joint that never existed before.
I think you're spot on. Just as the personal computer and GUI brought "computing" to the masses for their use and massive scale return on investment, we need to see the equivalent innovation in programming components. In other words just as it's now easier than ever to edit movies, write a paper, polish a photo: we need to see this ease of interface brought to programming.
I know hard core CS types will say that making programming easy for the masses will mean it won't be fast, it won't allow for the optimal algorithms. However computing speed and bandwidth is rising exponentially and countless applications for software do not need to be as optimal as possible.
In the future people who are not programmers but who are one step removed (e.g. okay with spreadsheets, SQL, some basic scripting when needed, visual programming like Labview, and who understand software architecture) will need to be able to do more complex things with computers, things that today only programmers can do.
We need to see the separation between programmer and the technical creative masses disappear a bit. Or at least as tablatom says, there has to be room for two types: those who build the tools for easy programming and those who do easy programming.
This is one of those classic ideas that's been around forever. It was more or less implied when the idea of "code reuse" became popular alongside object-oriented programming, I think. But I don't think it's ever really going to happen, because one of two things will happen. One is that you'll run into a set of requirements that no one has run into before, not because any of the requirements is unusual but because the space of possible requirements is very big if not infinite in size, and once you combine requirements that space increases exponentially. The other is that you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it. Both of these more or less happen already; the real question is what would surmount these problems.
> You'll run into a set of requirements that no one has run into before.
Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project. I don't see this as a fundamental reason why component-assembly can never become viable. As we get better at creating flexible components, these situations will get less common, but they will never go away.
> you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it
100% this. For me this pretty much sums up why component-assembly isn't viable today. For all but the simplest components this turns out to be the case. (e.g. date-picker, file-uploader, or maybe something a bit bigger with /very/ fixed requirements, like a disqus comment trail). But jumping from this to "I don't think it's every going to happen" is overly pessimistic. The glue code is too hard to write? We need a better way to write glue code. That could be a fundamentally different type of language, or a fundamentally different conception of what we mean by "component".
(Aside: In my foolishness I am working on such things).
> Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project.
It's not just that you'll run into one unique requirement; you might run into a unique combination of requirements, each of which already has proven solutions, but with no good way to glue it all together. That's the reason C programmers still sometimes write their own string handling or memory allocation code despite that stuff being literally in the standard library.
Writing general purpose software components is hard. If you're creating a product, you know what kind of component you need, and you don't really care about anyone else. If you're making a general purpose component, you have almost literally no possible way of even comprehending, much less fulfilling, all the requirements of every product that could potentially use your component.
You're always going to notice a difference between something that's been cobbled together out of spare parts and something that's been designed to fit an integral product vision. There's a reason we've been hearing about reusing program components for literally decades. I'm sure some chunk of the problem will be broken off and solved, some kind of standard solution to the CRUD app or something, but there are still going to be products out there that need real engineering, not just component assembly.
I'm just reading the book Flow-Based Programming (2nd Edition) by Paul Morrison[1].
I would say the main idea is to shift focus from control-flow to how data ("information packets") flow through a network of (black box) components and only use traditional control-flow style programming for the most simple, atomic components. The book presents some convincing examples from business programming but I think the idea should work very well for other areas than text processing, certainly for image manipulation or sound processing.
The split between component creators and component assemblers (application programmers) is highlighted in the book, I hate it how often I tend to slip from one role to the other in regular programming, maybe such an explicit split would help a lot (well, or certainly a lot of experience will ...)
Programming has delivered spectacularly on that critical goal, although software engineering did indeed fail, which is why the "software crisis" ended decades ago.
The division you propose already exists.
The "component creators" are called "programmers".
The "component assemblers" are called "users".
That's why you can use, say, Google Docs Forms to slap together a signup form in ten minutes that would have taken you a couple of hours in PHP with MySQL in 2000.
I think he's got the hand-made idea wrong. Programs are machines. Writing software is the act of specifying how those machines work. I don't know any industry in which this is done by other machines. It's always manual work.
The work of a financial analyst is also entirely manual by the way.
How is the code hand-made? The millions of gmail users are using the same piece of code. Same thing with Windows users. That's what you call mass production.
I think this is looking backwards. Who uses an item is not related to it's producing methods. Say a handmade sword can be used by different swordsmen in generations. And this would not make it mass produced.
Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done.
I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied customer at the end. Overwhelmingy, some (or most) of the design gets "made up as we go along". Even on projects with several small forests worth of up-front documentation, there's almost always large areas of vagueness or outright contradictory requirements, which need decision making on the developers part halfway into the job.
There are _very_ few websites that couldn't be coded from the ground up in a few weeks with a few good programmers - _IF_ you had already thought through all of the things the site needs and all of the consequences and the contradictions in those consequences. Even the big ugly complex projects, like Google's search or Facebook or Wikipedia or Twitter - if you had a spec that answered all the details about how you wanted them, a small team of experienced web guys could get it up and running in a month or so, and let you now how much cloud/hardware/sysadmin/support you'd need to budget for as your customer acquisition kicked in.
It doesn't take very long to "build" the software or the website (obvious "large scale" projects like OSes excepted). What takes most of the time is identifying and solving the problem. (And, I suspect a lot of the reason software/websites have a reputation for taking "so long" is that they're _way_ too often rushed into the "building" phase way before all the things that need designing are even identified, then all the new solutions take longer 'cause we try to fit them around all the work that's already been done.)
I think you might be underestimating how much custom code has been written by companies like twitter and Facebook, think php to c++ compilers, databases, rpc servers, log frameworks etc etc. even if all of the designs of all of that software was given upfront I doubt any small team could build it within a month or so.
If the design and spec job was done with full knowledge of current state-of-the-art "web frameworks, databases, rpc servers, log frameworks, etc etc." - and requirements adjusted to suit "off the shelf" tested and reliable code, only speccing "custom code" where absolutely necessary, I think my claim is still supportable. I don't want a spec that says "you need to write this in php - therefore you need to write a php compiler to scale it". I want those problems solved by the spec. The "coders" just want APIs and datastructures and wireframes and finalised graphic design and content/content-inventory. I reckon I know people who could do it.
And the reason that custom code gets written is because it's specific to that environment. The wide majority of the software available is focused on either a very generic target or a very specific one. If you use something that is very generic, you need to customize it because one size does not fit all. And you can't use the very specific one because it doesn't match your use case. Or what's available is designed for a single webserver and a single database server, so it's really tempting to use that when you're small then you'll most likely need to replace that at scale. And by then, you're writing something that is specific for your use case, which you've discovered has grown dynamically and organically based on changing business and technical requirements.
It's difficult to take technology or software from one place and transplant it into another. The best we can do is take what we learn on past projects and past companies and keep those lessons in mind when making new decisions.
> I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied customer at the end. Overwhelmingy, some (or most) of the design gets "made up as we go along". Even on projects with several small forests worth of up-front documentation, there's almost always large areas of vagueness or outright contradictory requirements, which need decision making on the developers part halfway into the job.
So the underlying development method, nowadays denounced as 'waterfall', wasn't sufficient? I am not an Agile zealot but change and 'knowing better' needs to be embraced during the development process and not ruled out.
And that is what actually differentiates software engineering from other engineering disciplines.
You cant't simply embrace change when building a house or you'd have to tear it down multiple times during building...
I've worked in construction, and that's actually not the case at all. What happens is very similar to what happens in software development: someone realizes that the spec is borked; the subcontractor and general contractor get together to look it over, and come up with a kludge that fixes the problem as easily as possible.
If the change is at the customer's request, they get charged a nice premium and the necessary changes are made. At the end of the day, everyone still gets paid regardless of how often the customer changes their mind.
This, then, is perhaps what the trades do better than us: they're very upfront with the customer about added costs. "Yes, we can make your bathtub a jacuzzi. It'll cost you $2500 more and the job will take a couple days more. Do you want to go ahead with your change?" Because there is a clear and complete plan that the customer signs off on before construction begins, it's very easy to show them what a change in requirements entails.
Everyone can envision what knocking a wall down means, and can see it will cost money and time. Many decide, upon reflection, that they really don't care enough to pay that price. Others go nuts and end up with expensive projects that take forever. Either way, the contractors get paid.
Right, I've worked on many projects where it wasn't really possible to gather all requirements up front. This is hugely frustrating to everyone involved but the fact remains that until you have at the very least a working prototype you will often not be able to generate a final requirements document.
On the other hand when requirements are nailed from the very start, the development proccess has been swift, well executed and largely bug free.
While developers love discussing the pros and cons of different languages, libraries and frameworks I often think that more work should be put into developing ways of better capturing requirements, as this has such a huge effect on the amount of time a project will take.
Agreed. My experience is that there is way too much emphasis on languages, libraries, and frameworks. Nobody really talks about the methodologies of building software. For any given day on HN, how many articles are about how such-and-such extension for this-and-that language is the best thing since the great pyramids and how many are about how code review can help reduce bugs? Or how to more effectively gather requirements? Or anything about comprehensive testing practices (unit tests are just a small part of effective testing.)
There is also a gap in discussion of software architecture. Everyone seems happy to use whatever their framework forces them into, or add more machinery to make up for it. Concepts like coupling and cohesion don't get much airtime, particularly with the prevalence of dynamic languages.
I think that's largely because languages, libraries, and frameworks are advancing, and methodologies are not. Better methodologies are becoming more widespread, but there's very little that's being invented that's actually new.
E.g. use Agile and smart guys with good knowledge of application domain for first version, then drop it and rewrite from scratch using Waterfall and outsource professionals.
It is ok if you don't change anything (or nothing significant) between version 1 and 2. But then, why would you rewrite from scratch ?
In any case, if you use a different team to rewrite your application, you just lost the biggest value of the previous team: experience solving that problem the first time. No amount of documentation can really make up for the difference . The source (with doc) of a software is a compiled version of business/technical knowledge of the team that developed it and not really that good as a guide to the next team.
Why not? Professionals knowns their tools much better, so they will made better choices in every aspect of project. If you place restrictions on their tools, they will work less efficient, unless you are already professional developer.
> The source (with doc) of a software is a compiled version of business/technical knowledge of the team that developed it and not really that good as a guide to the next team.
Of course. They will ask questions, so somebody from first team should respond to them.
"It doesn't take very long to 'build' the software or the website (obvious 'large scale' projects like OSes excepted)."
A lot of assumptions have to be met for that to be true. It shouldn't take very long provided:
1. The programmers are very familiar with the tools.
2. The programmers are very familiar with all necessary 3rd party components.
3. The 3rd party components account for 99% of functionality.
4. The programmers have built that exact project before, preferably more than once.
I usually work in a best-case scenario for development: I'm building projects with limited scope, solo and with no other decision makers involved in the project. I'm also experienced enough to know how complex things will generally be from a high level standpoint. And I have never once met my own estimate for complexity and the amount of time something will take. Exactly 100% of the time silly, seemingly inconsequential things add up to an extra 30% or more.
It doesn't matter if you can write 1000 shippable lines of code in a day because tomorrow you'll spend most of your time hunting a bug in an external library. Suddenly your machine-like pace has been cut in half. This happens on every project I've worked on and I would assume all software projects.
I agree that poor planning adds to the amount of time needed. Of course it does. But there really is no "best case" scenario where you and a couple of buddies could hammer out Facebook in three weeks.
"It doesn't matter if you can write 1000 shippable lines of code in a day because tomorrow you'll spend most of your time hunting a bug in an external library. Suddenly your machine-like pace has been cut in half. "
I'm there right now - why doesn't this Concrete 5 ecommerece plugin's Paypal addon work this week, when nothing's changed since last week???
I was thinking a bit lower level than that but yeah, I guess you get my point.
I spent a day with a bug that I found in OpenCV (a computer vision/machine learning library) that spooks me to this day. How it works internally is that for images to be processed you create a header (with usual image header info) which contains a pointer to the actual image data. There are two ways to do it: one you create the header and then create the image data separately and the other you create the image all in one go. I ended up in a situation where the image (it was going through a couple of different filtering algorithms) would get processed and be displayed regardless of whether or not I had set the pointer to the image data. That image data was already in memory since it was getting read from disk in a separate step, but there was really no way for the library to know its location in memory. I spent a day trying to sort out what was going on -- and threw my own schedule -- because I had to assume something was seriously wrong and would lead to other unintended consequences. I never did figure out what it was but after a lot of testing I decided it wouldn't cause any other problems (it never did, the library I was writing is still the core of a product I'm selling today).
If you don't know it OpenCV is about as mature as a software project can get. It was most likely the best tool for the job I had to tackle. But even with the library's stability and widespread use there was just plain voodoo that set a project I was working on back. Other 3rd party tools can be much worse. I don't do any web stuff but I have had to integrate Facebook Connect into products. Interfacing with Facebook can be a nightmare. Their API changes fairly regularly and the documentation can be a mess. Even using a 4th party, regularly updated, helper library I've seen bugs that took ages to sort out.
And that's just talking about integrating 3rd party, hopefully mature, components. I don't care how good you are, writing new code means a lot of testing and debugging. I've never seen new code that didn't introduce some voodoo of its own into a project.
I wouldn't call OpenCV 'about as mature as a software project can get,' as it's actively developed library with lots of contributors in yet experimental domain (more about mid-level pattern recognition than low-level image filtering). And the documentation was not up-to-date last I checked (~2 months ago).
Bugs in really mature software (i.e. libtiff, libxml, sqlite) are very rare.
That's probably more accurate. I was conflating a long history with maturity/stability. I would guess most ~13 year old libraries are broadly more stable and bug-free than something a few years old but I guess anything that is being actively adapted to new problems will have inherent instability.
I seriously doubt the code for reading images into an OpenCV-native format is being actively developed though. Anything being done in pattern recognition would likely not effect the library's basic IO stuff. What I was actually using it for (their RDP implementation, image thresholding) were likely pretty stable as well.
I tend to agree that the actual code writing phase is easy if the spec is very thorough. But the spec can never specify everything and tons of little decisions are left up to the developer. There are always people on the team who will create little messes. Those messes start to pile up unless there is a very strong leader who refuses to let any inconsistent code through (and is given the time to do so)
I was working as a freelancer for a long time and I had a quite simple solution for lack of documentation. Before the first meeting I'd send a customer 12-page form to fill which covered most of the "what and why" of the project. It quickly turned out that most of people have no idea whatsoever about the product they wanted to buy. I know that it's partly our job to guide people through rough path, but if all customer knows is "I want a website" it's better to quit early.
Sometimes I waited for a filled form for few months or even more (not paying much attention), and some of these projects were very successful. If I'd have started building right away, I'd spent these few months struggling and waiting for decisions to be made. What is even worse, I'd probably estimate my time and salary based on wrong assumptions, so I would be mad and underpaid. The project would take way longer than I estimated, so customer would be angry too.
Do yourself a favor and never start coding before final documentation arrives.
It is in Polish, so I guess it won't be very useful. I may translate it and post somewhere if I have some free time. For now, try googling "website brief template", it's how I made my first version, then improved it over time. It's basically a compilation of most of the questions I had to ask during my previous projects. If you like to look like a funny guy try adding some "would you like a pictures of your cat on the website". It releases some tension associated with such formal document (which I usually attach to signed contract, just in case), but be prepared to hear "yes, I actually would like to have cats on my website".
It's probably worth mentioning that this approach is correct only for small and medium projects, like things possible to deploy with Wordpress ninja in team.
The projects I do have never had final documentation before development started, and its not necessary.
I got this process from another Ruby consultant and it works great
- Get wireframes of main page views (usually there are about 6 of these. You can help them create this
- Get them to write out, or describe the requirements and take notes
- Break the wireframes and requirements into user stories and load it into pivotal tracker with estimates
- Tell them "we are going to build all these user stories PLUS 30 'freebie' points of stores. Any changes you want you can spend your freebie points on, and after that you can substitute out currently scheduled points if you really want those changes
That sounds like a good idea. Maybe not the best for me, as I always had trouble with drawing a line "our contract does not cover that" and I'd probably lose negotiations about how many points something is worth. It's still better than having no safe margin at all.
As I've written in my unknown blog, I think there is another misunderstanding here: software development is not the analogy to building something according to a plan. It is making the plan. http://blinker.net/2009/08/31/the-architecture-analogy-for-p...
Jack Reeves is the man here though - writing source code is designing the product. Before code can be written you need to take all the trade offs, the decisions and the research (or just make wild assumptions).
Once you have source code, it has been designed.
Which is why instead of sending out a 12 page form telling a client they need to think harder about something they don't understand, put out version 0.1, and ask them what they want changed.
Look at your own experience, the building phase in the design phase. We will (almost) never get a complete spec before hand because people are in charge and people change their minds. The sad reality is that building something with developers is often a faster, cheaper way to discover what is actually needed. Most people cannot tell you what they want until it is right in front of them and even then the response is usually, "Not this".
I'm a civil engineer (five years) and have never been on a project that avoided the problems of a custom poorly specificied software build.
Civil projects are a constant dialogue between client and engineer - but we get further monkey wrenched by one additional stakeholder that software doesn't have: local and at times federal government review. If you think software dev is difficult now, wait til your code is regulated.
My code is regulated. I develop retail point of sale technology which requires certification with various state agencies such as WIC. Each state likes to do things just a bit differently, which keeps things interesting to say the least.
Software for banking and insurance is like this. The product has to accommodate different rules for all 50 states and federal oversight. Changes to just about anything need to be cleared by the relevant state and federal agencies.
Another big difference between physical engineering and information engineering is that when an overpass is completed, the project is done. If you're lucky, someone will come by and look it over or maintain it over the years. There's no such thing as "done" software. When a program wraps up, everybody rushes back in and starts monkeying with it for the next iteration.
In Alberta, Canada, software developers cannot be have "Engineer" in their title, for this very reason, unless they fit a very specific set of requirements [1]. The Canadian Council of Professional Engineers defends the title, via The EGP Act, in order to preserve the credibility of the profession.
Nothing is made entirely by machines. At some point it's made by hand. The thing is software is a bit like doing clay pottery when it's not properly done. Once you add a bit of a systematic approach to it and you test is, review it, it becomes a bit closer to an industrial product.
The thing is the rest of the world is also in a state of flux. Plumbers changed materials quite a few times in the past decade which changed the fixtures a bit. So the concept is the same (ie. design patterns) but the actual material and its characteristics changed (ie. framework, programming languages) and there are different ways to put them together: welding, glue, etc (ie. APIs).
But changing some material, fixtures and binding material doesn't even compare to the kind of flexibility we have with software!
The solution for plumbers is that they always use what's current. Which makes their field as much of a fashion-driven field as ours (think about it).
But give them an existing house with multi-layer pipes and tell them to fix it and they will also comment that it's old tech and you should switch to copper or whatever.
In conclusion, I don't see how software is any different than other fields except that we are much more flexible and go with a much greater speed.
Software is at its infancy because science in general is at its infancy.
I think of software development more as an art than a science. If those questions were asked of a professional painter I think we would hear similar responses. It's all by hand, colors shapes and textures can be amazingly complex in their combinations, standards would only inhibit a true artist, and a painter usually has no idea at all when a masterpiece might come forth. So my answer to "Why is is so hard?" would be, "because it's art." And "why do we keep doing it?" "Cause we're artists, and people enjoy art."
That's outrageous self-glorification and utterly misleading. Software development qualifies as much as an art as other forms of engineering and most crafts do -- somewhat, at the highest levels, but generally not. Don't justify the shortcomings of current software engineering by nebulously labeling it a form of art.
I don't consider any of those things as shortcomings, simply aspects of anything involving human minds using a creative process. You're right that it's the same as all other forms of engineering, but I would also consider most of that art as well. With that I mostly just mean; despite the laws/science/logic involved, creating something new is usually a creative process.
Was it hard to create this blog entry? Obviously he has a point but, in general, the abundance of predefined frameworks and libraries makes it easier to create "web sites and software". Fortunately for us (developers) at the same time user demands grow higher and higher. Web-Sites 'need' to become more dynamic, styled, responsive, ... simple CRUD 'needs' to turn into a 3-trier distributed middleware architecture.
tl;dr software complexity is a constant. Progress in easing development is compensated by increasing demands.
In my professional experience (I started programming 20 years ago and worked on my first 'enterprise' Java project 12 or so ago), software complexity has become much greater. If you look at just the different areas you need to be proficient in now to build 'enterprise' software compared to 20 years ago, it's insane.
What I think has happened is that experienced developers have constantly had to keep up with learning to account for this. I don't think complexity has stayed constant - we have expanded our skill set and knowledge to keep up.
I'm not sure if your memory serves you well. Anyway, the human mind is limited wrt complexity. It cannot grow infinitely. Also, complexity shifted from languages (e.g. C++) to frameworks and tools (e.g. JEE).
I'm not sure if it is just my programmers mentality but I haven't been able to even run a company blog on Wordpress without coming across a few things I can improve. I have seen people put together a Wordpress with nothing more than addons and some basic HTML knowledge and it is pretty impressive what can be done, I think it is far from a tool anyone can pickup and make whatever blog type site they need without knowing about a ton of technologies.
Building software is often compared to building houses. If civil engineers can design and create a very complex bridge or dam or whatever, and it keeps working for 150 years without breaking. Why can't us software engineers make software that will work without bugs for more than 10 minutes?
Problem is, the two aren't really comparable.
A while ago someone wrote an article that we are Software Gardeners not engineers[1]. And I think that comparison is much more apt.
Software is maleable, it can change, the layers of complexity build up so high it's impossible for any one person, or team of persons, to fully understand. Even the mathematical theory of software isn't fully understood yet, and yet we are piling those little lacks of understanding on every layer of the architecture ... they add up and things become monstrous.
BUT! This is a good thing. Software isn't a physical object. I want to be able to change the specs half way through a project rather than having to wait another 50 years to design a better bridge. This makes progress quicker.
As always, when progress is quick (and easy), it is also messy.
For example: it took builders thousands of years to invent The Arch and trully revolutionize the industry. It took software only a couple of decades to go from machine code to python.
Oh and never forget, the Turing Machine was "invented" in 1948. Our lives revolve around an industry the theoretic principles of which were first defined less than 70 years ago.
> To boot, when constructing buildings, the methods are so well known that one person can design (the architect) and another person can build (the carpenter, plumber, electrician)
I was toying with the analogy of "building things" vs. "building software" the other day.
I've always felt that comparing software with architecture, building a car etc. is what people naturally reach for since it's something they're familiar with in the physical world and much of the same terminology is used.
I don't think that software is at that point, though. Physical things "play nice" together because they are part of the physical world. Materials have characteristics that are inherent and don't need to be conceived of where as the way things behave and interact in software needs to be defined and constructed entirely by humans.
I guess one could argue that happens to a certain extent in the world of materials science but ... I don't really think it holds up.
I arrived at a point where I started to think of building software as writing a novel. Once you start to use that as an analogy, it doesn't seem so weird that it's hard and takes ages, because so does writing a novel.
In a novel, one must define the entire world, and one can make the choice of using/re-using story lines or doing something original. People that trot out formulaic drivel make better money than the tortured geniuses on average, but the few tortured geniuses that manage to hit, hit it big and serve as an example to all the others.
I kind of stopped thinking about it at this point and got back to work, but I think that using that analogy, things really start to make more sense ... what do you reckon?
I wonder what changes we might see if we insisted on teaching programming in the large upfront. For instance, instead of teaching 'public static void main,' we taught people about component oriented programming. This way, people are equipped with an understanding of the complexity involved.
I also wonder whether teaching practitioners to diagnose and identify what type of project one is dealing with might avoid blind spots when estimating projects. For instance, an LDAP project should be identified as an integration project, where one might have to deal with unexpected schema.
The real problem with software projects at the moment is we lack a proper framework to think about the nature of each project and therefore don't understand where the risks are, and properly advise clients.
158 comments
[ 4.6 ms ] story [ 240 ms ] threadEDIT: I'm sure the db connection will come back up, but this error is the root of this discussion, in my opinion. No one wants to be yelled at, so to prevent this, we revert to what "just works" and your idea of that concept is different than mine, thus the more we can agree on standards the better off we will be for the general use case.
it's late where i am, so i'm gonna go to bed. hopefully the site will survive until morning.
It is a testimony to the innovation in any field that leads to tools become outdated so quickly. Easy commoditisation is directly related to the tools one has at their disposal. Creative fields like photography or movie-making also face the same issues.
Interestingly, if we are given a choice to constantly upgrade our tools then we would always prefer to. For eg: I don't find most people using the cork screw on a swiss-army knife. I would rather have a lighter multi-tool. But redesigning, testing and manufacturing a new multi-tool takes a long time.
Software has relatively speaking a much shorter lifecycle. Most of the clients recognise this and sometimes have higher expectations of the turnaround times. This leads to unintentionally short timelines or bug-ridden software.
As in any uncertain venture, it is often better to go with iterative and agile methodologies rather than a big bang approach.
1) Software is in it's early days (in the grand scheme of things.) Though things may seem "hand-made" now, it won't always be the case. In fact I'd say OO, design patterns, frameworks, cloud technologies and more are the early equivalents of regulations, well-accepted standards, etc.
2) What other profession is as egalitarian? What other profession has as much upside? Entire industries can be changed by a few smart people. That kind of opportunity makes the "hassle" worth it.
3) The hassle of having stakeholders/customers constantly wanting things faster and cheaper can easily be mitigated by being picky about what company your work, for, it's business model and it's culture. Not all software companies face that issue.
If you have an aversion to using Google cache and would like to see a clean version.
The fact I can push a software update to a million users in minutes makes him green with envy.
So yes software is hard... but cost free replication is an enormous upside.
Not that I'm complaining, just giving some perspective. Maybe this will make your dad less jealous :)
These days I write in an interpreted language using complex libraries that handle a multitude of protocol choices for me. Even if you argue that each of those tools were at one point hand-crafted, even down to the compiler, there are now development tools that help me write code, from basic code completion and tooltips to the most complex static and runtime analysis. That is machines writing software, with a human in the loop.
This works just fine for construction of physical things because the cost sunk into to the "non construction" bit is tiny compared to the whole project - so no one spends too much time thinking about methodologies and automation of an architect coming up with the concept of a building.
(1) I note in the article the person mention spent time as a consultant in a big consulting company - this view is still held by them at least with the last brush I had with organisations like that (and they are incredibly frustrated by it).
I think the analogy is becoming a little stretched though.
I feel like the ISO/ANSI etc standards-making bodies are where the analogy breaks into time and space.
http://www.developerdotstar.com/mag/articles/reeves_design_m...
In one sense there _is_ this part of "constructing software", and _largely_ it can be done by the software equivalent of stereotypical "construction workers". (This is what a lot of people who've tried outsourcing to India are trying to do.)
The problem is, while you can collect a pickup full of Mexicans who can lay bricks / hang sheet rock / tar roofs on most street corners in the south of The Mission, and they'll do a great job of it if you give them good directions - you don't expect those guys to be making architectural or structural decisions, or zoning or permitting or code decisions.
"Code writers" have to make those sorts of decisions every day - a current high-profile example is Marius Milner and "his" decision about what data Netstumbler should collect from the Streetview cars. One of the biggest software companies ever, having ethical/legal/policy decisions made by the coder-on-the-spot (at least if you believe Google's representations on the topic). Or Apple with Lion debug-logging clear text passwords for FileVault, and having it escape "into the wild".
The "architect" and the "civil engineer" and the "structural engineer" who have important roles in the world of building physical things, the guys who sign off on bridges or tunnels or even just-repaired airliners, the guys who put their careers on the line when they sign the paperwork, the guys with qualifications and certifications and often indemnity insurance to satisfy society that they understand the risks - for the vast majority of software/websites discussed here that's reasonably likely to be a 22year old college dropout aiming to be "the next Zuck". Even in small and medium enterprise sized businesses, those roles are largely thrust upon whichever developer seems to be good (and doesn't duck their head quickly enough). And if the shit hits the fan, they say "sorry boss, it seemed like the right answer at the time" (and hopefully doesn't get hung out in the press like it seems Milner has been…) (And the "big consulting companies" mentioned in the post I'm responding too, in my limited experience they often seem to want to make all the architectural/engineering/policy/ethical/legal decisions, then leave with their paycheck before the "codemonkeys" implement it all, and not be contactable when their "solutions" turn out to be incomplete/contradictory/impossible)
I _hope_ government regulation of "software construction" isn't the answer (at least not for software that'll just cost investors money when it fails, as opposed to bridges or airliners that'll kill people), but I think lines of responsibility and authority need to be more explicitly identified in many software projects, with appropriate authority conferred on the people burdened with the responsibility. Holding developers to deadlines without giving them the authority to adjudicate on them or be involved in the determination of them, is a startlingly common way to have your developers cut corners - and worse, feel entirely justified in cutting corners and convincing themselves they're "doing the right thing".
You could probably regulate "software construction" and mandate specific methodologies, but we can see already what the result would be: just look at the SEI CMM Level 5 Certified software development teams that already exist: Wipro, R Systems, and so on — technically inept companies that only exist to rip off clients who don't know any better.
Any country that mandates that kind of development for all software will be rapidly left behind by the countries that don't as software becomes an increasingly important part of the 21st-century economy. They'll still have human beings laying their literal bricks and tarring their literal roofs, while the rest of us are living in robot-built houses full of fountains and sculptures, or dynamically-reconfigurable programmable houses.
And of course, since I'm in Boston I'm required to mention the Big Dig (2), which was a tunnel and bridge project that cost over $14 billion. Oh, and a ceiling panel fell, killing a woman in her car. And the guardrails in the tunnels tend to kill motorcyclists who would otherwise suffer only minor injuries. Plus the all 25,000 of the 120 pound (55 kg) light fixtures in the tunnel ceilings have to be replaced lest more of them fall, maybe killing more people.
But yeah, let's keep trying to make software engineering just like civil engineering.
1. http://www.dreamingincode.com/ 2. http://en.wikipedia.org/wiki/Big_Dig
Those days are gone too. In fact currently you just have to know how to press ctrl+space at the right time and wait for intellisense to do the magic. Nearly 99.99% of the Java world works like that.
In other words, these days we learn how to learn tools that write programs.
Hand made items - essentially think back to before the industrial age. Most things were hand made. Art, tools, food etc
Now we have machines to make things easier. Or we have off shore labour.
If it is a precise trade, well sorry but you still need to struggle by hand. You should therefore new charging a premium to do it.
* While there are certain constraints a house has to satisfy, architects do have freedom to be creative. They MUST plan such that it will resist certain standard incidents, and they have to plan it such that the floors can carry ~10x the weight you would ever expect there. These regulations vary depending on the region you live in. E.g., in Austria there is the possibility that meters of snow lie on roofs, and architects (or their structural designers) have to keep that in mind when calculating the max weight. You won't find this constraint in Portugal.
* Keeping to these constraints does not guarantee a building won't break. Constraints (like requirements) are prone to grow outdated, or be incomplete. In Austria, roofs will sometimes break down when it snows for some days in a row (4m of snow are very heavy).
* As incidents in e.g. Turkey and China show, plumbers or other construction workers can do such a bad job ("botching") that buildings just break down after some years. Governmental constraints don't assure quality. Some countries like Dubai won't let local workers unguarded near any expensive building for this exact reason - most of the times, German and other Western workforces are hired to oversee the construction process.
I think one could compare governmental constraints on construction work with requirements. A building's floor must be able to carry a weight of 100 tons, and a software system should be able to serve 1000 requests/second - I think that's comparable, and software constraints might even be easier to test.
What I think this really boils down to is that software requirements evolve much more quickly, and due to this it's not feasible to establish legal requirements - government would have to issue new requirements every week, adjusting them for technologies like nodejs. Also, how do you establish best practices for bleeding-edge technologies within weeks? Not at all. We are in a constant state of learning and experimenting.
The world of software development needs to be divided in two -- the component creators, and the component assemblers.
To stick with the given analogy, component creators are the people inventing new kinds of plumbing: easier ways of connecting pipes, taps that don't ever drip. Component assemblers are the people fitting out bathrooms.
Creating new components is high-end engineering. It needs to happen far away from the day-to-day challenges of making a client happy.
To a large extent, this division is already present, but it doesn't go far enough. There has been amazing progress - nowadays we work on top of an incredible stack of technology that we don't have to re-invent, but so far we've not achieved the "last mile".
We'll know when we're there because component assembly (i.e. making something for a client) will start to look more like a trade.
Today, just getting a regular been-done-a-million-times-before database-driven website (or whatever), requires FAR too much low-level code. This, I think is what is meant here by "hand made". We should be snapping things together, and often we are, but suddenly you get to an awkward bit and you're back to forging a new kind of pipe joint that never existed before.
I know hard core CS types will say that making programming easy for the masses will mean it won't be fast, it won't allow for the optimal algorithms. However computing speed and bandwidth is rising exponentially and countless applications for software do not need to be as optimal as possible.
In the future people who are not programmers but who are one step removed (e.g. okay with spreadsheets, SQL, some basic scripting when needed, visual programming like Labview, and who understand software architecture) will need to be able to do more complex things with computers, things that today only programmers can do.
We need to see the separation between programmer and the technical creative masses disappear a bit. Or at least as tablatom says, there has to be room for two types: those who build the tools for easy programming and those who do easy programming.
Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project. I don't see this as a fundamental reason why component-assembly can never become viable. As we get better at creating flexible components, these situations will get less common, but they will never go away.
> you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it
100% this. For me this pretty much sums up why component-assembly isn't viable today. For all but the simplest components this turns out to be the case. (e.g. date-picker, file-uploader, or maybe something a bit bigger with /very/ fixed requirements, like a disqus comment trail). But jumping from this to "I don't think it's every going to happen" is overly pessimistic. The glue code is too hard to write? We need a better way to write glue code. That could be a fundamentally different type of language, or a fundamentally different conception of what we mean by "component".
(Aside: In my foolishness I am working on such things).
It's not just that you'll run into one unique requirement; you might run into a unique combination of requirements, each of which already has proven solutions, but with no good way to glue it all together. That's the reason C programmers still sometimes write their own string handling or memory allocation code despite that stuff being literally in the standard library.
Writing general purpose software components is hard. If you're creating a product, you know what kind of component you need, and you don't really care about anyone else. If you're making a general purpose component, you have almost literally no possible way of even comprehending, much less fulfilling, all the requirements of every product that could potentially use your component.
You're always going to notice a difference between something that's been cobbled together out of spare parts and something that's been designed to fit an integral product vision. There's a reason we've been hearing about reusing program components for literally decades. I'm sure some chunk of the problem will be broken off and solved, some kind of standard solution to the CRUD app or something, but there are still going to be products out there that need real engineering, not just component assembly.
I would say the main idea is to shift focus from control-flow to how data ("information packets") flow through a network of (black box) components and only use traditional control-flow style programming for the most simple, atomic components. The book presents some convincing examples from business programming but I think the idea should work very well for other areas than text processing, certainly for image manipulation or sound processing.
The split between component creators and component assemblers (application programmers) is highlighted in the book, I hate it how often I tend to slip from one role to the other in regular programming, maybe such an explicit split would help a lot (well, or certainly a lot of experience will ...)
[1] http://www.jpaulmorrison.com/fbp/
The division you propose already exists.
The "component creators" are called "programmers".
The "component assemblers" are called "users".
That's why you can use, say, Google Docs Forms to slap together a signup form in ten minutes that would have taken you a couple of hours in PHP with MySQL in 2000.
The work of a financial analyst is also entirely manual by the way.
Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done.
I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied customer at the end. Overwhelmingy, some (or most) of the design gets "made up as we go along". Even on projects with several small forests worth of up-front documentation, there's almost always large areas of vagueness or outright contradictory requirements, which need decision making on the developers part halfway into the job.
There are _very_ few websites that couldn't be coded from the ground up in a few weeks with a few good programmers - _IF_ you had already thought through all of the things the site needs and all of the consequences and the contradictions in those consequences. Even the big ugly complex projects, like Google's search or Facebook or Wikipedia or Twitter - if you had a spec that answered all the details about how you wanted them, a small team of experienced web guys could get it up and running in a month or so, and let you now how much cloud/hardware/sysadmin/support you'd need to budget for as your customer acquisition kicked in.
It doesn't take very long to "build" the software or the website (obvious "large scale" projects like OSes excepted). What takes most of the time is identifying and solving the problem. (And, I suspect a lot of the reason software/websites have a reputation for taking "so long" is that they're _way_ too often rushed into the "building" phase way before all the things that need designing are even identified, then all the new solutions take longer 'cause we try to fit them around all the work that's already been done.)
If the design and spec job was done with full knowledge of current state-of-the-art "web frameworks, databases, rpc servers, log frameworks, etc etc." - and requirements adjusted to suit "off the shelf" tested and reliable code, only speccing "custom code" where absolutely necessary, I think my claim is still supportable. I don't want a spec that says "you need to write this in php - therefore you need to write a php compiler to scale it". I want those problems solved by the spec. The "coders" just want APIs and datastructures and wireframes and finalised graphic design and content/content-inventory. I reckon I know people who could do it.
It's difficult to take technology or software from one place and transplant it into another. The best we can do is take what we learn on past projects and past companies and keep those lessons in mind when making new decisions.
So the underlying development method, nowadays denounced as 'waterfall', wasn't sufficient? I am not an Agile zealot but change and 'knowing better' needs to be embraced during the development process and not ruled out.
If the change is at the customer's request, they get charged a nice premium and the necessary changes are made. At the end of the day, everyone still gets paid regardless of how often the customer changes their mind.
This, then, is perhaps what the trades do better than us: they're very upfront with the customer about added costs. "Yes, we can make your bathtub a jacuzzi. It'll cost you $2500 more and the job will take a couple days more. Do you want to go ahead with your change?" Because there is a clear and complete plan that the customer signs off on before construction begins, it's very easy to show them what a change in requirements entails.
Everyone can envision what knocking a wall down means, and can see it will cost money and time. Many decide, upon reflection, that they really don't care enough to pay that price. Others go nuts and end up with expensive projects that take forever. Either way, the contractors get paid.
On the other hand when requirements are nailed from the very start, the development proccess has been swift, well executed and largely bug free.
While developers love discussing the pros and cons of different languages, libraries and frameworks I often think that more work should be put into developing ways of better capturing requirements, as this has such a huge effect on the amount of time a project will take.
There is also a gap in discussion of software architecture. Everyone seems happy to use whatever their framework forces them into, or add more machinery to make up for it. Concepts like coupling and cohesion don't get much airtime, particularly with the prevalence of dynamic languages.
E.g. use Agile and smart guys with good knowledge of application domain for first version, then drop it and rewrite from scratch using Waterfall and outsource professionals.
Why not? Professionals knowns their tools much better, so they will made better choices in every aspect of project. If you place restrictions on their tools, they will work less efficient, unless you are already professional developer.
> The source (with doc) of a software is a compiled version of business/technical knowledge of the team that developed it and not really that good as a guide to the next team.
Of course. They will ask questions, so somebody from first team should respond to them.
A lot of assumptions have to be met for that to be true. It shouldn't take very long provided:
1. The programmers are very familiar with the tools.
2. The programmers are very familiar with all necessary 3rd party components.
3. The 3rd party components account for 99% of functionality.
4. The programmers have built that exact project before, preferably more than once.
I usually work in a best-case scenario for development: I'm building projects with limited scope, solo and with no other decision makers involved in the project. I'm also experienced enough to know how complex things will generally be from a high level standpoint. And I have never once met my own estimate for complexity and the amount of time something will take. Exactly 100% of the time silly, seemingly inconsequential things add up to an extra 30% or more.
It doesn't matter if you can write 1000 shippable lines of code in a day because tomorrow you'll spend most of your time hunting a bug in an external library. Suddenly your machine-like pace has been cut in half. This happens on every project I've worked on and I would assume all software projects.
I agree that poor planning adds to the amount of time needed. Of course it does. But there really is no "best case" scenario where you and a couple of buddies could hammer out Facebook in three weeks.
I'm there right now - why doesn't this Concrete 5 ecommerece plugin's Paypal addon work this week, when nothing's changed since last week???
It's not just you…
I spent a day with a bug that I found in OpenCV (a computer vision/machine learning library) that spooks me to this day. How it works internally is that for images to be processed you create a header (with usual image header info) which contains a pointer to the actual image data. There are two ways to do it: one you create the header and then create the image data separately and the other you create the image all in one go. I ended up in a situation where the image (it was going through a couple of different filtering algorithms) would get processed and be displayed regardless of whether or not I had set the pointer to the image data. That image data was already in memory since it was getting read from disk in a separate step, but there was really no way for the library to know its location in memory. I spent a day trying to sort out what was going on -- and threw my own schedule -- because I had to assume something was seriously wrong and would lead to other unintended consequences. I never did figure out what it was but after a lot of testing I decided it wouldn't cause any other problems (it never did, the library I was writing is still the core of a product I'm selling today).
If you don't know it OpenCV is about as mature as a software project can get. It was most likely the best tool for the job I had to tackle. But even with the library's stability and widespread use there was just plain voodoo that set a project I was working on back. Other 3rd party tools can be much worse. I don't do any web stuff but I have had to integrate Facebook Connect into products. Interfacing with Facebook can be a nightmare. Their API changes fairly regularly and the documentation can be a mess. Even using a 4th party, regularly updated, helper library I've seen bugs that took ages to sort out.
And that's just talking about integrating 3rd party, hopefully mature, components. I don't care how good you are, writing new code means a lot of testing and debugging. I've never seen new code that didn't introduce some voodoo of its own into a project.
Bugs in really mature software (i.e. libtiff, libxml, sqlite) are very rare.
I seriously doubt the code for reading images into an OpenCV-native format is being actively developed though. Anything being done in pattern recognition would likely not effect the library's basic IO stuff. What I was actually using it for (their RDP implementation, image thresholding) were likely pretty stable as well.
Sometimes I waited for a filled form for few months or even more (not paying much attention), and some of these projects were very successful. If I'd have started building right away, I'd spent these few months struggling and waiting for decisions to be made. What is even worse, I'd probably estimate my time and salary based on wrong assumptions, so I would be mad and underpaid. The project would take way longer than I estimated, so customer would be angry too.
Do yourself a favor and never start coding before final documentation arrives.
It's probably worth mentioning that this approach is correct only for small and medium projects, like things possible to deploy with Wordpress ninja in team.
This may be a good place to start: http://www.methodandclass.com/article/write-a-web-site-brief...
I got this process from another Ruby consultant and it works great - Get wireframes of main page views (usually there are about 6 of these. You can help them create this - Get them to write out, or describe the requirements and take notes - Break the wireframes and requirements into user stories and load it into pivotal tracker with estimates - Tell them "we are going to build all these user stories PLUS 30 'freebie' points of stores. Any changes you want you can spend your freebie points on, and after that you can substitute out currently scheduled points if you really want those changes
It works great so far.
http://ciocookbook.com/philosophy/thecodeisthedesign.html
Jack Reeves is the man here though - writing source code is designing the product. Before code can be written you need to take all the trade offs, the decisions and the research (or just make wild assumptions).
Once you have source code, it has been designed.
Which is why instead of sending out a 12 page form telling a client they need to think harder about something they don't understand, put out version 0.1, and ask them what they want changed.
Civil projects are a constant dialogue between client and engineer - but we get further monkey wrenched by one additional stakeholder that software doesn't have: local and at times federal government review. If you think software dev is difficult now, wait til your code is regulated.
Another big difference between physical engineering and information engineering is that when an overpass is completed, the project is done. If you're lucky, someone will come by and look it over or maintain it over the years. There's no such thing as "done" software. When a program wraps up, everybody rushes back in and starts monkeying with it for the next iteration.
[1] http://www.apega.ca/About/ACT/preface.htm
The thing is the rest of the world is also in a state of flux. Plumbers changed materials quite a few times in the past decade which changed the fixtures a bit. So the concept is the same (ie. design patterns) but the actual material and its characteristics changed (ie. framework, programming languages) and there are different ways to put them together: welding, glue, etc (ie. APIs).
But changing some material, fixtures and binding material doesn't even compare to the kind of flexibility we have with software!
The solution for plumbers is that they always use what's current. Which makes their field as much of a fashion-driven field as ours (think about it).
But give them an existing house with multi-layer pipes and tell them to fix it and they will also comment that it's old tech and you should switch to copper or whatever.
In conclusion, I don't see how software is any different than other fields except that we are much more flexible and go with a much greater speed.
Software is at its infancy because science in general is at its infancy.
> meta name="generator" content="WordPress 3.0.1"
Was it hard to create this blog entry? Obviously he has a point but, in general, the abundance of predefined frameworks and libraries makes it easier to create "web sites and software". Fortunately for us (developers) at the same time user demands grow higher and higher. Web-Sites 'need' to become more dynamic, styled, responsive, ... simple CRUD 'needs' to turn into a 3-trier distributed middleware architecture.
tl;dr software complexity is a constant. Progress in easing development is compensated by increasing demands.
In my professional experience (I started programming 20 years ago and worked on my first 'enterprise' Java project 12 or so ago), software complexity has become much greater. If you look at just the different areas you need to be proficient in now to build 'enterprise' software compared to 20 years ago, it's insane.
What I think has happened is that experienced developers have constantly had to keep up with learning to account for this. I don't think complexity has stayed constant - we have expanded our skill set and knowledge to keep up.
I'm not sure if your memory serves you well. Anyway, the human mind is limited wrt complexity. It cannot grow infinitely. Also, complexity shifted from languages (e.g. C++) to frameworks and tools (e.g. JEE).
Problem is, the two aren't really comparable.
A while ago someone wrote an article that we are Software Gardeners not engineers[1]. And I think that comparison is much more apt.
Software is maleable, it can change, the layers of complexity build up so high it's impossible for any one person, or team of persons, to fully understand. Even the mathematical theory of software isn't fully understood yet, and yet we are piling those little lacks of understanding on every layer of the architecture ... they add up and things become monstrous.
BUT! This is a good thing. Software isn't a physical object. I want to be able to change the specs half way through a project rather than having to wait another 50 years to design a better bridge. This makes progress quicker.
As always, when progress is quick (and easy), it is also messy.
For example: it took builders thousands of years to invent The Arch and trully revolutionize the industry. It took software only a couple of decades to go from machine code to python.
Oh and never forget, the Turing Machine was "invented" in 1948. Our lives revolve around an industry the theoretic principles of which were first defined less than 70 years ago.
[1] http://chrisaitchison.com/2011/05/03/you-are-not-a-software-...
http://www.artima.com/weblogs/viewpost.jsp?thread=255898
As an architect... easier said than done.
I've always felt that comparing software with architecture, building a car etc. is what people naturally reach for since it's something they're familiar with in the physical world and much of the same terminology is used.
I don't think that software is at that point, though. Physical things "play nice" together because they are part of the physical world. Materials have characteristics that are inherent and don't need to be conceived of where as the way things behave and interact in software needs to be defined and constructed entirely by humans.
I guess one could argue that happens to a certain extent in the world of materials science but ... I don't really think it holds up.
I arrived at a point where I started to think of building software as writing a novel. Once you start to use that as an analogy, it doesn't seem so weird that it's hard and takes ages, because so does writing a novel.
In a novel, one must define the entire world, and one can make the choice of using/re-using story lines or doing something original. People that trot out formulaic drivel make better money than the tortured geniuses on average, but the few tortured geniuses that manage to hit, hit it big and serve as an example to all the others.
I kind of stopped thinking about it at this point and got back to work, but I think that using that analogy, things really start to make more sense ... what do you reckon?
I also wonder whether teaching practitioners to diagnose and identify what type of project one is dealing with might avoid blind spots when estimating projects. For instance, an LDAP project should be identified as an integration project, where one might have to deal with unexpected schema.
The real problem with software projects at the moment is we lack a proper framework to think about the nature of each project and therefore don't understand where the risks are, and properly advise clients.