> They might realize Debra has been sitting in that corner, staring at uptime graphs of the internal server farm for 10 years, despite the fact that the company moved to AWS five years ago.
Ouch. The tone of the article is a little harsh, and I’m not sure if snippets like the above are intentionally hyperbolic, but there is a fair amount of truth in it.
Most of my career has involved convincing peers to do less, and solve simpler problems with simpler solutions.
The author hits the nail on the head with his claim that imaginary problems are more fun than real ones.
As developers and smart folks in general, we like complicated problems that are big and far away. How many times have I heard in a meeting, "Yeah, but when we have 1M users..."
It's great fun to think your product will get to 1M users. It's also very unlikely. It's not nearly as fun to finish and ship and market and monetize the half-broken thing the team is working on now. Yet that's the only way out and the only way anyone gets to 1M users to begin with.
Reminds me of a PM I used to work with. "Will this work for 1000 simultaneous users?" After almost 2 months, we have less than 100 users total, maybe 5 of them log in a day, and maybe 1 will actually do anything of interest.
There is no technical problem. The problem is nobody worked on actually marketing the product. Build it and nobody shows up is the norm.
I’ve worked somewhere like that. Our baseline requirements for concurrent users were based on the numbers required for the product launch team to maximise their bonus.
We never saw anywhere near those numbers in production, but I don’t really blame them - it was a big company and you do what you can to get ahead. A lot of money was spent on infrastructure that wasn’t needed but nobody seemed to care.
Whenever someone asks me this question, I reply with a question "How many simultaneous users do you/we have today and what is our projection say for 12-18 months from now ?". If the answer is not clear, I tell them not to worry yet. If the answer is very clear but numbers are much smaller today (say 5-10), then I challenge them on where they think they/we could be in 12-18 months. A lot of times, it helps the other side see that they are mostly asking "How long is a piece of string".
I was interviewing with a company that had barely any customers and they were asking scaling questions with Spark, etc. The salaries they paid could barely hire a team capable of dealing with the complexities of Spark, so they asked, "what would you do."
I told them I'd buy another stick of RAM and scale vertically until I had more customers, and save money on staff in the meantime. The interviewer went cold, I didnt get the job.
About 10 years ago, I worked on a project where I had to develop some sort of elaborate chain of map/reduce jobs because "big data" and "Hadoop." We were processing about 10 megabytes for each run. Most of the processing was consumed in job scheduling / overhead.
And people underestimate how well some solid, dumb solutions can scale. Boring spring boot with a decent data model and a bit effort to stay stateless scales to the moon. Or, we're having a grand "data export" system customers use to collect data from us for their own DWHs. It has survived 2 attempts at replacement so far. At it's core, it's psql + rsync, or was recently migrated to psql + s3 when we decomissioned our FTP servers. And it's easy to extend and customers are happy because it integrates well.
HipHop (later HHVM) was around 2010, so they scaled from 2004-2010 before that became needed. MyRocks was 2015. Wikipedia says FB was around 300 million users in 2009, then 400 million users in 2010.
Yes, good point. But you have to wonder what kind of engineering effort went into scaling PHP and MySQL up to the point where they decided to build a compiler and a storage engine.
The trick was there was enough growth that the savings from the compiler were massive. (I worked there at the time.) The inefficiency of the PHP interpreter was a great problem to have, because it came from the success it enabled.
So I think the interesting question is whether the rest of us can learn anything from what happened there.
I believe Mark Zuckerberg simply used the technology he knew and took it from there. That's fine. I probably would have done the same thing.
But many people are making an ideology out of this, arguing that not giving a shit about performance is always the right choice initially because that's how you grow fast enough to be able to retool later.
I think this is based assumptions that are no longer true.
In the early 2000s, mainstream programming languages and runtimes were either fast and low productivity or slow and high productivity (Exceptions such as Pascal/Delphi did exist but they were not mainstream). And the cost of scaling up was prohibitive compared to scaling out.
Today, you can choose any fast high productivity language/runtime and go very far mostly scaling up.
Learn what? That you should use the language that you’re more comfortable with and then scale? Or that languages have become more efficient? Php 8, for example, is many times faster than the php 4 and 5 that Facebook was using.
I think using what you already know remains a choice that is very hard to criticise. But we didn't have to learn that, did we?
Beyond that I think there is more to unlearn than to learn from the history of the Y2K batch of startups. The economics of essentially everything related to writing, running and distributing software have changed completely.
In part the reason that PHP8 (and it was 7 that had the quantum leap in perf) are now so fast is precisely because of Hack - it was easy to accept the status quo on performance until Hack showed there really was a lot of performance left on the table.
For me the biggest win was the changes they made to how arrays are stored in memory, I saw some systems drop by half in memory usage and had to change basically nothing - those kinds of wins are rare.
I take away two lessons from it, which are in a kind of essential tension that can only be mediated by wisdom and experience:
1) Pick technologies that you can optimize.
2) Don't over-optimize.
Also, the concept of "optimization" here has very little to do with the language itself. It's far more about the overall stack, and it definitely includes people and processes (like hiring). It's not like FB invested $0 toward performance before swapping out PHP interpreters! Its massive caching layer, for example, was already taking shape well before HPHP (the C++ transpiler which preceded HipHop), not to mention the effort and tooling behind the MySQL sharding and multi-region that still exists in some form today. Many backend FB services were already written in C++ by 2010. But they had already gone very, very far—farther than most businesses ever will—on "just" PHP. Heroics like HPHP only happened after enormous pipelines of money were already flowing into the company.
They succeeded because of php. It was easy to use for them. So it enabled them to materialize their ideas. It was the right tool for them. Anything else would have been fine either way, if it was the language they were the most comfortable with. In their case, it happened to be php.
Well, OK. But by that logic, if the language they had been most familiar with was Fortran, should they have used Fortran for Facebook? I tend to think that there are actually material differences between languages and technologies, and it's worth knowing more than one language and not using terrible ones.
“ But by that logic, if the language they had been most familiar with was Fortran, should they have used Fortran for Facebook”
Absolutely. Otherwise they wouldn’t have been able to release the actual product and keep adding features to it the way they did with Facebook. They’d spend half the time learning the “right” language & environment. That would have slowed them down to the point they wouldn’t have been able to work on the actual product as much as they did.
And feature-wise, Facebook evolved really quickly.
No, they totally succeeded because the used php. I think zuckerberg said it himself that php allowed them to add new features easily. I think he mentioned that it was easy for new people to pick it up. I’m pretty sure Facebook wouldn’t exist today if it had been written in the more corporate/esoteric languages available at the time.
Its ease of use allowed him to launch the site from his dorm room. Iirc, YouTube was also written in php (it had .php urls), before google bought it and rewrote it using python, so you could probably thank php for that site too.
I don't think there was anything similar to PHP that wasn't proprietary (Cold Fusion etc.), and FB engineering culture was to avoid vendor lock-in.
In any case, in the 2000s a PHP programmer/designer was analogous to a JavaScript developer today. Lots of talent out there, and it only took a few weeks of orientation and familiarizing for new hires to be productive.
Depends on what you consider "far down the road" and what they had to do before writing a compiler and a storage engine.
How long did it take until Facebook engineers realised that their technology stack was not the best tool for the job? It definitely wasn't the day when they decided to build a compiler and a storage engine.
I'm not sure there was really a best tool for the job in 2003-2004 that would have been high-level enough to be productive, and scalable enough to stay mostly as-is. Java, maybe.
I agree, and I'm not criticising the choices that Mark Zuckerberg made at the time. But we are no longer facing the same situation he did. We do now have high productivity, high performance language runtimes. And scaling up has become much cheaper (relative to the number of users you can serve).
That's why I think it can't hurt to remind people of the great lengths to which Facebook had to go in order to deal with the limitations of their chosen platform.
Yeah, I kinda don’t agree with the dichotomy of “you either optimize or you build features”. They’re not exclusive. If you understand the tools and their trade offs you should be able to use the right tools for the job which won’t hinder you in the future.
Of course if all you know is JavaScript, then that requires going a bit outside your comfort zone.
>> And people underestimate how well some solid, dumb solutions can scale.
I think this started with the old Apache web server. When the www got started, that server did the job so everyone used it. The problem was it didn't scale, so all kinds of cool solutions (load balancers and such) were developed and everyone building something bigger than a personal blog used that stuff. For most the root problem was that Apache had terrible performance. Nginx has solved that now, and we also have faster hardware and networks, so anything less than HN can probably be hosted on an rpi on your home network. OK, I'm exaggerating, but only a little. Bottom line is that scaling is still treated like a big fundamental problem for everyone, but it doesn't need to be.
> And people underestimate how well some solid, dumb solutions can scale.
I'd say they underestimate how long "just throwing money" (servers) at the problem can work.
If you earn decent money now, scaling number of app servers 10x times to serve 10x times the traffic will still earn decent money. Doesn't matter that "PHP is slow", deal with it when your infrastructure cost warrant hiring more/better developers to fix it.
Especially now. Pair of fat servers on some NVMes and 1TB of RAM gonna cost you less than few dev-months and that can serve plenty of users in most use cases even before any extra caching will be needed.
Even then, you don't have to throw money at the problem right away. If you feel you can save time and money by using Rust instead of PHP (just using the two languages as examples, not a specific indication of Rust or PHP's resource drain), go ahead. Making that decision early on costs nothing.
It's only after a project is off the ground that caring about these decisions winds up wasting everyone's time, that's when you wind up slowing momentum tremendously due to dangling a potential new toy in front of your team.
> And people underestimate how well some solid, dumb solutions can scale.
And overestimate how expensive it is to just add another server, or underestimate how expensive a rebuild is. But then, part of that is also that IT departments want the budget; if they don't spend their annual budgets on innovation, their budget gets cut.
Or in my neck of the woods, the EU subsidies will stop if they don't have anything they can file as "innovation".
As a solo dev on a project, I constantly re-evaluate whether the thing I am working on is beneficial to my users or if it's an "imaginary problem" as this post describes. In a large software project you always have a laundry list of things to do from: "fix the date format on this email template" to "implement a better feature flag system". While I'm tempted to always work on the "fun stuff" (the feature flag system), I make myself work on the "boring stuff" (fixing the date format on an email template) because I know that's what users need. Occasionally you get an intersection where the cool fun an interesting problem is also the most pressing one but I found those times are few and far between and most times I have to decide between the "cool fun [imaginary] problem" and "the boring [real] problem".
I recently added a couple of constants to some project. One of my teammates said it wasn't a good idea, because we could have hundreds of similar constants eventually.
Those constants represent the markets supported by that app. By the time the app supports even a few dozen markets, every engineer involved will have exercised their stock options and left.
> The author hits the nail on the head with his claim that imaginary problems are more fun than real ones.
Not necessarily. It's just that most developers have never worked in a setting where they got to work on problems properly.
Solving real problems for real people is very addictive. There is a reason some people like working for startups. It's because you live very close to your users and when you make them happy you know.
The second interesting fact is that if you just plow ahead and solve many real problems fast you will eventually run into problems that are both real and interesting.
After having tried that there has been no going back for me. I am allergic to imaginary problems. It feels as pointless as watching people who are famous for being famous argue on TV.
I think we are all victims of our feedback loops. (University) Education sublely teaches us that the only important problems are those that are very difficult and preferably have never been solved before. Those same problems also make for better blogposts. In the real world the incentives are mostly opposite. Problems with no known solutions (or only really difficult solutions) are generally bad. They can be worth it, but you should stay away from them until you know that are worth it. Software engineers seem to almost pride themselves on not knowing what their users want.
It takes a while to scrub all that bad learning out and replace it with something better. Unfortunately some people are stuck.
> Solving real problems for real people is very addictive.
This was the feedback loop that worked best for me. imaginary problems and needless complexity go hand in hand, ruthless editing at the planning stage is necessary to combat it.
> (University) Education sublely teaches us that the only important problems are those that are very difficult and preferably have never been solved before. Those same problems also make for better blogposts. In the real world the incentives are mostly opposite. Problems with no known solutions (or only really difficult solutions) are generally bad.
This is a great insight. One can add value to other people's lives by applying known solutions in relatively novel contexts (e.g. building a CRUD form at XYZ employer), whereas it's very hard to add value to other people's lives by trying to develop entirely novel solutions (because the probability of success is so low). Most of our training however, focuses on the methodology used to develop these novel solutions, rather than on the application of the solutions themselves.
I am dealing with that today. Talking about scaling out to hundreds if not thousands of aws accounts and I'm like, "we've added 6? in two years?" Why are we wasting time on this?
I remember having a similar argument with someone saying that your C code has to compile and work on every platform that exists, including weird CPUs with known bugs.
Unless you're working on something like the Linux kernel, that's an imaginary problem.
"Newer versions of the compiler build your code with security vulnerabilities" is a very real problem in C. E.g. since x86_64 has no aligned memory access instructions, a lot of programmers assume there's nothing wrong with doing unaligned memory accesses, but actually recent gcc/clang will happily compile those into RCE vulnerabilities.
I worked with a project manager that went too far in the opposite direction, though. Their pushback against premature optimization manifested in wanting to start with a functionalish "proof of concept" without any real design phase so they can say we cranked out an MVP in the first sprint... and before you know it, like most non-blocking technical debt, the "migrate functionality to final codebase" kanban card moves to the "long term goals" column (aka trash) and you're stuck with a shitty, fragile producion codebase. The opposite side— trying to get everything into a final state right off the bat— it is like trying to play an entire 8 measure song in one measure.
At the beginning of a project, before I write a line of code, I try to: a) if it's user-facing software, get some UI designer input to shape functionality/interactions, not styling. To end users, the UI is the software, not just the shell, so mediocre UI = mediocre software. It can also illuminate needs you didn't consider that affect architecture, etc.; then b) block out the broad stroke architecture, usually on paper, c) intentionally choose languages/environments/tooling/etc rather than reflexively going with whatever we've been using recently, and d) spend some design time on a reasonably sane and extensible, but not overly detailed data model.
There's no perfect solution, but at least in my cases, it seems like a good middle ground.
The trouble is, an 'MVP' is often far from 'minimum' in those sorts of situations.
The reality is that MVP should be missing most planned functionality and should really just be a few core features and functions that the rest of the application builds off of, the trunk of the dependency tree so to speak.
That idea is, unfortunately lost on the majority of PM's, and ultimately it costs more time/money to get to a finished v1 because of it.
It was actually the appropriate scope for an mvp, it was just an unreasonable time frame to make a solid codebase for anything other than a demo for the complexity of the project. That's fine for a genuine proof of concept/rapid prototype you're going to be disciplined enough to trash, but letting that slip into the role of your core codebase is like pouring a sloppy scaled-down concrete foundation as a test for a house, and then trying to just expand it into what you need.
Now you have a whole bunch of New SQL databases that will scale past whatever number of users you can imagine. So your good old Django or Rails app can scale to 1M or 10M users without you doing anything exotic. That's not "fun" though.
Not the first time the issue has been pointed out:
> Simplify the problem you've got or rather don't complexify it. I've done it myself, it's fun to do. You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had.
I think this is an attitude shift that a lot of developers need to get over. They like writing code, they like working with computers, and they pick that to do as their day job.
But their day job actually isn't writing code, it's solving a problem for an end-user; the programming language is just a tool.
Rethink your job from a coder to a problem solver and you should be able to get over the compulsion to overcomplicate things for your own gratification.
The hard part here is telling the difference between a pure phantasm of an imaginary problem and innovation. Things that have never been done (or never really been done well) might look as far off as hypotheticals.
I do think that “imaginary problem” is the answer most of the time though. Most things that look like unnecessary complexity or hobby horses really are.
And here am I struggling not to solve memory leaks or improve up the build time of or speed of our app because I am stuck implementing the most boring of the features already digested by a team of businesses analyst and a design team.
It's amazing how many otherwise brilliant people dive headlong into project without considering these basic principals or even intentionally brush them aside.
I believe Saying simple > complex doesn’t actually mean anything because it’s effectively impossible to pin down definitions. They are totally in the eye of the beholder. Solutions that are simple in one axis almost always trade of complexity in other axes.
As a good illustration, consider FEniCS[1], where you can write a few lines of Python code which looks almost exactly like the math you're trying to solve, and have it compute the answer. Very simple!
Except to make that work there's a lot of infrastructure, including runtime-generated-and-compiled C++ code that gets dynamically loaded by said Python code to perform the actual calculations. Quite complex!
The true skill comes in finding the right balance between simplicity and complexity for a given situation.
In the case of FEniCS, the complexity is worth it because it allows the system to be used by less skilled programmers (who might know more about the math and physics), and the complexity handled by experienced programmers.
For our codebase we've got junior programmers who might need to read and understand my code if I'm on vacation and shit hits the fan, so I err on the side of making it easy to read and reason about. Which might not be the "simplest" for some measures of simplicity (like fewer lines of code).
I’d argue that FEniCs was a prime example of this in some ways, at least earlier in it’s history.
I started using it in about 2014. It was not exactly what I would call a simple project. It used to be an ordeal to build, could only easily be used via Docker, ported to Python 3 a long long time after most of it’s dependencies did, had an unstable C++ API that changes were not documented for but nonetheless you were required to use if you wanted for reasonable performance for some calculations, etc. The national supercomputing centre in my country managed to only get one version to build because it was so poorly specified at the time and basically only supported Ubuntu!
FEniCsX is considerably better usability wise, but that’s the result of hard lessons learnt.
That's why I prefer the form "Do the simplest thing that could possibly work."
There's always going to be a minimum level of complexity. Sometimes that minimal level calls for throwing a PHP script up on some shared hosting provider somewhere. Sometimes that minimal level calls for an Enterprise-level design with ten layers of abstraction because you need to be able to individually unit test every aspect.
Being aware of where and when to introduce complexity is half the battle.
And here's the deal, with more years in this industry, I've always found that the optimal performance almost always comes from a simple architecture.
And that performance optimization done prematurely often achieve complete opposite result. Meanwhile performance optimization done at necessity rarely have the same problem.
People need to stop throwing around 'Premature optimization is the root of all evil'. Simple > Complex <--- Look at that. It's optimization! Complex != optimized
It need to be thrown around more. I cannot count on one hand the designs I came across that were needlessly complicated that rather than solving a pain point became a problem in themselves.
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."
this is definitely one of the advice i give to senior dev i work with: if you're proud of how smart your solution is, there's a high chance you overengineered and made a mess of a simple problem.
i now take great pride when my code looks boringly obvious.
Related: my favorite pull requests are the ones that remove more lines of code than they add. People think you need to hoard old code that’s not used anymore like it’s made of gold. It’s not. You aren’t gonna need it, and if you do, you can find it in the git history.
I push that so hard. Put the removal in an isolated, clearly named commit that will be easy to search later, tag that commit so it never gets garbage collected, then take a deep breath and say goodbye.
You'll be better off without it and 99.9% of the time you won't have to retrieve it later anyway.
At work we have this terrible “Enterprise Architecture” team comprising of highly paid people who haven’t written a single line of code and who don’t know the intricacies of our business but keep proposing complicated “Event Driven Architectures” and “Micro this and Micro that” and just reciting the latest buzz words just to keep appearing cool.
It’s insane how much total cost they add to the organization, both directly and indirectly.
I always find it bizarre how people like this can operate. After almost 20 years of software development, I've considered seeking some kind of an architect role, but I cannot, for the life of me, imagine operating as one without working closely and collaboratively with the development team on a solution, rather than just dictating how things should be done "from on high". But that may just be a personality thing, I don't know.
"real architects" write code and simply hop from one team to the other so that they have a reasonable picture of the overall system and can try to guide all the teams to make harmonious choices and possibly even reach some goal.
This still results in a lot of compromises and problems. Anyhow they should be there talking to the developers in a 2-way fashion so that then end result is not entirely "from on high".
IME architects tend to be people who tell your team that you should be using Azure Cosmos after a Microsoft salesman takes them out to lunch. They last coded 5 years ago.
They exist - I am one - but it doesn't work unless your leadership wants someone who doesn't fit in existing hierarchies. I tend to consult and if I go somewhere where I have prior relationships with leadership, it works. If not, I get treated as part of someone's narrow reporting chain and all the incentives are wrong.
I've seen this even without the code - them just bouncing between teams and sitting in on many meetings and sort of being the common note-taker who then knows where almost every team is going, what they need, and what cross-team work could be done to improve everyone's life.
> I always find it bizarre how people like this can operate.
They are acting rationally within their belief system of making money.
Enterprise software is exactly that, software used by enterprise -- it doesn't signify any good qualities (far from it) only that it provides a cost-effective software solution to a defined business need.
The issue is when corps get bailed out, overfunded, or have revenue mostly outside of software (e.g. gov't contracts) as it eliminates cost-effective from the equation... so you just end up with buggy messes (code as a cost center).
You'd have to work in the tiny niches where tech is the true product to find good development ...and even then...
I consult as a software architect and my job is mostly the opposite of this: asking people what problem they are trying to solve and why they haven't considered $EXISTING_SOLUTION
Sort of related, but this is one reason I moved into “Frontend“ about 10 years ago. I started seeing over and over that when our (SaaS) projects didn’t start from what the customer sees, teams usually got distracted with imaginary or hypothetical design issues. It was a lot more effective to iterate from the visible features, and then let that drive much of the backend design, APIs, development timeline, etc.
This meant I needed to deal with more JavaScript than I originally intended with my career, and eyerolls from backend architect types, but projects go much smoother than in my past, that's for sure.
In our agency 10 years ago we build such websites, it would cost 3000-5000 dollars max. Just with PHP and a simple self build CMS. Including a simple responsive design. We also hosted around 250 of such websites on a single dedicated server. It was very very fast.
I totally agree with this, normally it should be the cost of any content website for clients. The site is fast, and we just need to deploy on one dedicated server or for free with simple maintenance cost per month these days.
We still do the same for small clients, with application building with us and want a content page for their blogs.
It could be both too high or too low depending on the communicated expectations of the client. Which is I think the real issue here anyway.
But let’s say 15k for two months of a team of contractors. That’s 7.5k/mo. A team says at least two, but I think given they mention sales etc, that’s 3-5 people. Netting 2.5k/person before you take out built in profit margins, healthcare (because it’s dollars, I assume America) and more. That’s roughly 1.5k/person per month. (Of course they could have multiple projects but even then that’s a low amount imho for a contractor with sales)
If they’re expecting an off the shelf solution, then they’re getting fleeced on the costs and 15k is too high, but if they’re going to a company that does big solutions than they spent too little.
In the world of entrepreneurship, you mostly find the extremes.
Either you have the bootstrap founders that will trawl through fiverr to find the cheapest labor (the same types that will scoff and get insulted at anything over $10k), or you have the well-funded founders that will pay whatever it takes.
From experience: Cheapskates and low-ballers will never be happy. They always want something free, more haggle room, more discounts, and always have high demands and expectations. The best thing one can do is to price yourself away from them.
i think you might be trying to solve the author's imaginary problems.
all the author needs is a wordpress site with a couple plugins and a few weeks of back and forth on the design work. if you can't make a good profit on that with a $15k invoice, you're doing something very wrong.
The problem is that their brief is too vague and the real issue is communication.
At no point is it clear that they’ve discussed off the shelf solutions, or infrastructure (storage, server hiding, CDN etc).
If they had, then they either messed up because 15k over two months is too high or too low. It’s squarely in the: “nobody has actually defined the project territory” for me.
Yes, a Wordpress solution would work, but then why even budget 2 months for it, unless bespoke design is involved. Again that becomes a: this is too low or too high to be realistic
>At no point is it clear that they’ve discussed off the shelf solutions, or infrastructure (storage, server hiding, CDN etc).
why should they? the requirements are listed in the article. the job is to meet the requirements. "where should we store the files" is a job the client hires you to answer, not something requiring communication.
The requirements are insufficient if you’re going to be so particular as to stick exactly to them. You can’t guarantee the uptime’s requested without talking about infrastructure.
This resonates and one way to describe it is an incentive problem. Someone whose incentives are tightly aligned with the business is going to solve the actual problem and simply and effectively as possible. Someone who is incentivized to build career capital and experience other than via impact (e.g. so they can get uplevelled, pass an external interview loop, etc) is much more likely to focus on unimportant hard problems and/or over engineer.
That's been a thing for 30 years. Entrepreneurship is HARD, and tech salaries are fat right now. I think we'll see a lot more software entrepreneurship when there's another recession.
Makes you wonder what the actual state of the industry is right now with thousands of layoffs, but then comments like this one. Probably it's a bifurcation and an uneven distribution of reality.
There were layoffs in the big tech companies, but the sector itself is strong. Still very low unemployment. They over-hired. It happens. It's been a relative minor correction.
You hit the nail on the head. There are different motivations for different roles within the same company, sometimes those motivations clash internally, all the while each individual IS acting completely logically from their own unique perspectives.
This is absolutely a thing, but I'd say there's a related option which is "Job Listing Driven Development". The more niche, dated, or specific your platform is, the harder it is to hire people onto the team who don't need months of on-the-job practice and training to be useful.
You see the most extreme versions of dangers of this in stories about governments and older companies having to pay insane salaries to bring FORTRAN or COBOL developers out of retirement to keep systems running. If you keep doing the simple solutions within the existing system, you risk creating a system so inbred that only the folks who built it can maintain it effectively.
For less extreme setups, it's still a balancing act to consider how much your unique and specific solution that is the simple option for you company starts closing you off of the larger hiring pools in more common technologies and patterns.
What's kind of funny is that MUMPS is equally as archaic and idiosyncratic as Fortran or Cobol, yet there are companies willing to put new hires through a bootcamp to make them productive. Are all the Fortran and Cobol companies too small to afford a month or three of training time on new devs?
My guess would be that the entities short-sighted enough to still be using those languages in 2023 are also ones short sighed enough to not invest in the training to preemptively hire juniors without the skillset to train them up.
As someone who maintains a large Fortran codebase actively maintained from the 50s, I can say with 100% confidence that syntax, compiler, and other tools aren't even 10% of getting up to speed. It's some of the worst code you will ever see. A lot of it predates "GOTO considered harmful." It also comes from an era where different common blocks and subroutines were moved into and out of memory using a custom virtual memory system.
The demand for Fortran/Cobol experience has nothing to do with training. We need to make sure you are masochistic enough to trudge through the sludge.
> Someone whose incentives are tightly aligned with the business is going to solve the actual problem and simply and effectively as possible.
Equity is the entirely the answer for cutting through all the bullshit. At least in my head. I don't know how it plays in other people's minds but mine sounds like: "If we ship and go live, I get x% of all profit moving forward in my personal scrooge mcduck money bin". Pretty big carrot. It's kind of like time share in my own personal business, but I don't have much of the headache that goes along with running my own 100%.
This has some caveats, namely that equity in a 10k person org is often times not nearly as meaningful as equity in a 10 person org. Shipping your code 2 weeks early at Ford or Dell means what, exactly? If the code you are shipping is the business, then things are different. It also really helps if you care about the problem you are solving.
I'd say this - if the idea of direct equity/ownership doesn't get you excited about pushing simple & robust techniques, then you are completely in the wrong place. You should probably find a different problem space or industry to work in. Hollywood might be a better option if the notion of equity or straight ownership in the business still isn't enough to put your tech ego into a box.
I have a little bit of equity in the company that I work for now. It's super small and early stage, and still between me and the product decision there exists a Designer that reports to a CTO that reports to a CEO. For everything that I want to see done differently, I have to make a case that convinces all these stakeholders that it's the right way. Ultimately, equity or not, my job is to row the ship where the cap'n tells me.
>> Equity is the entirely the answer for cutting through all the bullshit.
I agree for small companies which are largely founder owned. I think outside of that, Equity doesnt do much because so much effort is put into obfuscating the value/share of the equity. If you cant see the cap table, and you cant see the preference overhang, the equity is as good as worth zero. There is no discernable value for a fraction with no denominator.
Equity is the answer, I work in investment banking and we all get a share of firm profits, I'll often sideline small projects in favour of projects that I think will be more valuable to the org and increase my/our pay cheque come bonus time
Doing the sort of simple solutions to your specific job's actual problems can also be something that constrains your ability to work anywhere else. Often the best simple solution that's tightly integrated into your job's environment is something that is inconceivable as a good idea anywhere else. You're optimizing around other old decisions, good or bad. You're often correctly overfitting a solution to your specific problems.
I've often found myself now having issues even updating my resume, because what I did for the last year at work barely is explainable to other people on my team, let alone to someone in HR at another company. Or the more simple explanation is something that sounds like I'm doing work barely more complex than an intern could have done. Which often isn't wrong, but the intern wouldn't know which simple work to do.
My years of experience in the company's stack and org is valuable to the company, and nontransferable elsewhere.
> Someone whose incentives are tightly aligned with the business is going to solve the actual problem and simply and effectively as possible.
On average, and depending on skill. Incentives are hugely important (probably the most important metric any manager could work on), but even they do not guarantee results. If you hire so many juniors that nobody is there to upskill them fast, you only get one lottery ticket per employee. Conversely, if you hire a bunch of geniuses and fail to give them incentives to work on realisable, useful problems together, you get two lottery tickets per employee at twice the price.
(This comment feels woefully incomplete. Does anyone know of good resources to learn more about incentive structures and how they relate to individual and company success? I feel like the problem is that incentive structures change massively when companies grow, so even for unicorns there's just a short sweet spot where we can actually learn how they are supposed to look.)
Author gets it half right. Too many developers trying to have fun by doing new things. Affordable initial software development tends to come from places that have boring templated solutions using 1-2 tools.
But the main argument is off - What some classify as imaginary problems is actually building in the wrong types of affordances. The only way to know the difference is by knowing your tools and the problem domain.
The root of bad software is leaders lacking practiced learning.
In the imaginary case of the merch selling android app i dont think it is justified to solely blame the developers for the bad product. The root cause for a bad outcome lies in the simple fact that nobody wanted to tell the client that there was already a solution for 20 bucks and absolutely no need for something custom built. NOBODY except the business owner gains anything in this and the client deserves the ripoff too. It comes as no surprise that developers look for interesting challenges without appearing too distracted by the stupid and boring bs work they are stuck with.
Reputation is what gets you more business in the future. You can expand your business or raise rates, or both.
Yes, this won't help if you're literally in danger of not meeting payroll next week, but I would hope most businesses don't operate so close to the edge.
The author is right, but doesn't seem to mention that this could be solved, at least in theory, with better project management. Devs focusing on the wrong thing? Project manager should be on it. Scope creep? Project manager should be on it. Client asking for irrelevant features? Project manager should be on it.
Replace those devs with ones who are focused on the right things? Great, but you still have the other problems to deal with (scope creep, uninformed clients), not to mention a host of other things that can derail a project, such as poor communication.
tl;dr Most projects are poorly managed. Poorly managed projects tend to fail.
I think the move from project management to "product management" really exacerbated this. The type of people that gravitate towards "product" vs "project" management is one thing.
Further, project managers tend to focus on delivering the product stakeholders have asked for on time and on budget.
Product managers however I find are frequently searching for a new expanded scope of stakeholders to brag they're solutioneering to. I find them more often in the "solutions looking for problems" business than I find developers, or at least developers are only doing it on a micro-scale, while product managers will take an entire team/org on 6 month fruitless missions to build software destined for the dustbin.
The moment project manager stops taking care of minutes, meeting agenda, getting all the stakeholders in the same meeting and ensuring all issues have an assignee and instead starts taking interest in the specifics of the project, I bail.
I agree with this to some extent. but there’s a flip side too.
This mentality is often taken way too far. I had an old boss who wouldn’t allow me to write unit tests citing this thought process.
Even at places with decent engineering practices, I’ve seen so many examples of software where you’re limited to a one to many relationship for something that could and easily should have been implemented as many to many, rendering the product useless to many because a product person couldn’t stand to think a couple months ahead.
Some people seem to take this idea too far and basically assume that if a problem is interesting or takes away a tedious task, it must be overengineering, premature optimization, or an imaginary problem.
Perhaps a better way to phrase the issue would be “artificial constraints”, which would encompass the flip side too.
I have seen it validated by reality several times… more times than the opposite. I had a boss refuse to let me do a refactor that changed these sketchy dynamic field tables into json columns because “it’s not customer facing.” They were unable to show off features in an important demo because the endpoints were timing out despite putting 2 other people on it for 2 weeks to find code-based optimizations.
3 days later I deployed my “nice to have” fix and the performance issues disappeared.
I’ve also seen a company stall out scaling for years and lose multiple million-dollar customers despite having a novel in-demand, market leading product because they refused to do anything to clean up their infrastructure.
>I had a boss refuse to let me do a refactor that changed these sketchy dynamic field tables into json columns because “it’s not customer facing
YAGNI isnt about not refactoring existing technical debt. It's about not trying to pre-empt future requirements.
If youre refactoring in anticipation of as yet unmaterialized requirements then YAGNI applies - e.g. generalizing code when when today there is 1 specific use case because tomorrow you think there will be 3+.
If youre cleaning up existing code while working on it and the boss stops you because "it's not customer facing" then he's just asking you to violate the boy scout rule.
All of these definitions are fuzzy... refactor versus upgrade versus feature. When the people wrote it the way they did, they were almost certainly thinking that they don't need to overthink or over-engineer, and that they should discount hypothetical future concerns.
I can give you an abundance of examples. We were creating a page that was going to use state in a certain way. I was trying to insist that we address the way state will be handled across pages ahead of time. These concerns were dismissed as premature optimization. A few months later we had 5 pages with the state being handled in 5 different ways, and being synced in different ways between each page, complete with if statements, sometimes passing state through URLs, sometimes through local storage, sometimes through session, sometimes through JWT data, generally through a combo of several of them. Then we'd end up with confusing redirect loops for certain edge cases, state getting overwritten, etc.. We spend weeks fixing these bugs, and, eventually, weeks refactoring to manage state in a simpler way. These bugs often got caught by customers, drawing us away from feature delivery that was critical for demos to large customers.
All of that could have been avoided by spending 1 day thinking a little harder and planning for the future.
It ultimately boils down to a couple assumption that people like to make.
(1) engineers know nothing about the domain, they can never predict what will be needed. That might be true in a large company with obscure domain-specific things for engineers who work far away from the day-to-day, but sometimes the engineers know exactly what's going to come up. (2) You can hill-climb your way into optimal program implementation. You can get to local maxima this way, but there are regular ways that programs grow based on how the business is growing and you can predict certain places where you will soon hit diminishing returns for current implementations. As long as you're up front about it and double-check your assumptions about the way the business is growing (and hence the application), I think there are ample places where you actually are going to need it.
>I can give you an abundance of examples. We were creating a page that was going to use state in a certain way. I was trying to insist that we address the way state will be handled across pages ahead of time. These concerns were dismissed as premature optimization. A few months later we had 5 pages with the state being handled in 5 different ways.
The right time to address this was probably a bit at a time after the 1st, 2nd and 3rd pages. Certainly not before the 1st and definitely not after the 5th.
>All of that could have been avoided by spending 1 day thinking a little harder and planning for the future.
The reason why you try as hard as possible to avoid planning for the future is because it's really hard to predict the future. Moreover humans have an inbuilt bias towards thinking we are better than we are at it (hence the gambling industry).
Refactoring as soon as possible after the fact will always produce better designs than up front planning for this reason.
>there are regular ways that programs grow based on how the business is growing and you can predict certain places
This is the kind of phrase that makes alarm bells go off in my head that somebody SHOULD be following YAGNI and isnt.
If it's defaults in a well worn framework that doesnt railroad you then fine but anything more than that - red flags all around.
Rule of 3 is often correct, first time just do it, second consider if it'll very likely happen a third time and when the third time happens it's darn well time to do it!
HOWEVER, this only works if you have the agency at an organization to allocate time for doing something. Contrary to this when you are in an organization of management that doesn't understand technical debt (or is fine with it because it just means more consulting hours) then it's absolutely the correct choice to stall and/or fix things "prematurely" (iff you can see what product they are trying to create without knowing it) if you'll be holding the shit-can of duplicated crap down the line getting knuckled because things aren't going fast enough due to technical debt.
All depends on what the I is in the YAGNI. I have seen development be parallised where the same work is being done again and again by different developers in different ways because YAGN { maybe 2-3 days of upfront architecture and design for a 6 month project }. This results in bugs and maintenance nightmares. This was before unit tests were common though, so maybe unit tests may have saved it. But surely it was slower to develop that way.
But tautologically you can't take YAGNI too far, if the "YAGN" part is actually true :-). But that is always under debate.
It certainly feels that way. 2 or 3 days of up front architecture and design with hindsight is always better than 2 or 3 days of up front design in reality, but of course you dont have that hindsight when you start.
I've had to do up front on multiple projects and it always results in overengineering - we focused on things that didnt matter, designed things that were inappropriate, etc.
I'd always rather take those 3 days and redistribute them as extra refactoring time.
I’ve had tons of times where YAGNI has bitten teams at a FAANG. It’s been responsible for re-orgs as products have to pivot to meet the goals that were dismissed but turned out to be needed.
I was creating a very important demo once, features i had said were important were classified as YAGNI. Leadership eventually saw that we couldn’t deliver without said features. YAGNI bit those teams in the butt.
these things happen all the time internally to companies but get ironed out internally as well.
Yes. While it’s less common, I’ve seen orgs struggle because they didn’t have enough imagination.
Every feature is done quick’n’dirty and eventually you have people whose full time job is to respond to customer complaints and fix data straight in the production database.
No, it’s bad business because it doesn’t scale. Software is lucrative because you make it once and sell it to thousands of customers. If you’re making every customer their own bespoke thing, you’ll spend all your time for little return.
“Billed to the customer” means you’re charging the customer by the hour / project. You can get plenty of return selling bespoke things this way. Accenture is a $200 billion company.
That’s called Professional Services. Professional Services assemble a solution for a customer from a variety of components and maybe build some glue or the equivalent of a dashboard. This is not the same as having a ton of “if” statements in code to handle customer X vs customer Y.
The secret, as a software vendor, is to generalise these bespoke customer requests so you can sell the solution to all your customers (and get more customers!). If you are really cheeky, you can even get that customer to help fund the development that will make your business more money (hey, it’s win-win). You need to ruthlessly follow this approach though, as the rot of bespoke code will quickly become an insurmountable quality nightmare that can sink your business.
Yeah I agree. I think the biggest mistake people make when applying YAGNI is not considering how difficult it will be to change later. If it's just some hard-coded value that you could easily add to a config later? Fine. YAGNI.
If it's something more fundamental like language choice or system architecture... Well fine YAGNI now but if you ever do need it you're screwed.
Earlier in my career I ate up the lean startup, move fast and break things, y combinator stuff. And while there are some very good lessons there, I’ve also come to realize that when you stop working on a part of the code, that may very well be the last time someone goes in there to make serious changes for a while. So sometimes it makes sense to do it right, even if it takes a few days longer (but not if it’s going to turn into some massive overengineering project).
> secure online banking is actually quite an easy problem to solve . . . The storage and transfer of numbers is not a particularly hard problem.
I don't know anything about banking software, but I have a hunch the author underestimates the complexity (as is typical for HN). The Efficient Markets Hypothesis would suggest that if it were so simple to make banking software, then someone would do it.
The shareprice of META fell by 75% from August 2021 to August 2022, wiping about three quarters of a trillion dollars off its market cap.
From October 2022 to now the shareprice tripled, adding half a trillion dollars to its market cap.
Explain that in terms of the Efficent Markets Hypothesis? "share prices reflect all information" - what half trillion worth of information change came out in 2022? and what in 2023?
What about insider trading laws? How can we say "share prices reflect all information" when we know there are people who have more information which would give them an unfair advantage, which means the current shareprice cannot be reflecting the information they have?
Not to disagree with you but if there are people with insider knowledge they will either dump a massive amount of stock or buy it up changing the stock price. So the insider knowledge should be priced in quite fast thanks to their greed.
> It should be noted that this issue isn’t unique to developers. Management, sales, HR, support, legal, and even accounting departments have their own unique ways of creating imaginary problems. They try to involve themselves too much in a decision, when their presence at a meeting is just a formality or wasn’t requested at all. They overemphasize a minute problem that is related to their role, or hire teams much larger than necessary to illustrate their importance.
I run into this more often than problems I imagine. A whole series of what ifs and folks imagining things. The worst part is the fixes to their imaginary problem are usually not well thought out and drive things towards worse choices.
I’m a big believer in getting version 1 out the door as problems people imagine often… are never relayed by the actual customer.
I often work with some routing software (let’s say routing packages). There is a simple mode that works great. Anyone can use it.
The issue is people want to establish say 20 rules about how / when / what is routed. Business folks insist that it be “easy” for “anyone to use” just like the existing easy mode.
This is doomed from the start. We can make it easier for sure, but if you have 20 rules with different weighted priorities:
1. It is complex for most people to think of. It will look complex because it is complex.
2. That’s ok because the guy with 20 rules probably has thought about them and understand they have 20 rules.
Then we give them UI to visualize it all and customer is happy.
But the business folks are upset because the visualization is complex… and there we are again.
For the record I usually get through this slog and everyone is happy in the end, but it is a slog due to imaginary problems.
Isn't this why companies have employees with different levels of experience?
Need a bog standard app to stream audio files? Give it to the person you hired right out of college, or maybe even the summer intern. She's never built something like this before, so to her it will be a challenging novel problem. A (somewhat) more experienced developer may need to provide initial guidance and review code, but that's a comparatively minor time investment, and besides, the act of mentoring someone else should keep it interesting.
> Isn't this why companies have employees with different levels of experience?
this runs into another common mistake businesses make: putting people onto a task because they're available, not because they're suited to it (in any sense).
the junior person (+ a little mentorship) would be great for the job, but they're mired in some big project. but hey, you've got this super senior fellow sitting around waiting for work, and we can bill the customer more for them, anyways.
It seems like the fundamental problem for the scenario presented in the article was an inability of the customer to communicate their strategy and a completely hands off approach during implementation when they should have had at the very least biweekly checkins with developers with pre-negotiated milestones.
The fundamental problem is that a non-technical person has specced out a technical product without input from technical people.
A restauranteur wouldn't develop a menu without input from a chef or prior experience as one. A layperson wouldn't design a real building without input from an architect or engineer.
Yet for some reason a myriad of non-technical people (read: laypeople) feel empowered to design, spec, and strategize about software. It still boggles my mind that "product management" is a real profession.
I absolutely agree with the premise. People just love building things even when they're not needed. After a while, your ego gets attached to whatever it is you've built and you can't let it go.
I remember working at one place where somebody built a new framework to solve a common problem we all had. He pitched it to all the other devs in a meeting and I remember being confused about it because there was a standard framework that solved the problem already and did it in much simpler and more elegant way. (To be fair to him, this standard functionality was only recently introduced.)
During his presentation, I asked why the standard solution wouldn't work for him. It turns out he wasn't familiar with it. Fair enough, so later I messaged him and showed him the standard way to do it and how much simpler it was. He couldn't be swayed.
He just couldn't accept that his complicated solution wasn't necessary. He constructed scenarios where his idea was needed, even though I saw solutions to those scenarios using the standard framework.
Interestingly enough, one of the scenarios where his custom thing was needed was in some tests he had written where he did some complicated things to set things up. I looked at the tests and even there saw those complicated things weren't necessary! There were ways to simplify what he was doing so that the tests were better written and didn't need his custom tool.
Anyway, he wouldn't be convinced. And because he couldn't be convinced, we got stuck with his solution and saw people continue to work on it, add more functionality to it, fix bugs, etc. All of that work was just a waste of time when we could've relied on a standard solution, which was way more mature and way simpler.
All of this drove me crazy, but I realized that sometimes people are just unable to see simple solutions to problems. Worse, having one complex solution begets more complex solutions elsewhere.
This is one reason I’m glad I came from a research background into software. If something doesn’t work it doesn’t upset me and I don’t personally feel aggrieved, you just chalk it up to experience and move on. The number of times I worked on a study that had to be abandoned either because it didn’t work or because another research group beat us to it!
> people are just unable to see simple solutions to problems
This person might just have been terrified of admitting that all the (very tangible) time they spent on payroll building this thing was for naught. It’s unclear how their manager might have responded to that. And they wouldn’t have been able to put „built system doing X used by Y developers and deployed to Z customers“ in their resume.
The reasonable choice often doesn’t have much skin in the game.
On one project I did it was essential to be able to record how much a tool was used so that we could charge for it. The tool ran locally on the customers' machine and reported to our service. The overall mechanism that sent and received this information had to be reliable or we'd lose money but even worse would be to in some way overcharge customers. Lots of aspects of the design were complicated by this concern.
Then we ended up deciding not to make money out of it that way. So we burned enormous effort and created a horrible design for no reason.
So IMO the problem is usually with the way requirements are not usually well understood even by the people asking for them. Later on it becomes clearer what is needed but you're stuck with false assumptions baked into your design in a way that you never have bandwidth to remove because you need so much bandwidth just to do normal work....because of those assumptions.
I think the most important function of a good Product Management team is to understand what parts of the go-to-market impact tech decisions and spend 80% of their energy into pinning those down as firmly as possible. There is a happy medium between JIT delivery of specs for random features and a hard two-year roadmap that can't react to business changes.
YNGNA is generally true, but Product's should have a very clear vision of what kinds of entities the system is going to handle over then next 36 months before they start asking for specific functionality. I've seen extra shit get built, but I've also seen e.g. a travel booking system that was built without a "flight" being a first class entity. Flights were deduced on the front end from attributes attached to seats... which worked well until a PM asked for the UI to show fully-booked flights, which HAVE no available seats that make it to the front end. Same product couldn't handle the booker and traveler being different people, when they knew from day 1 that it would be a necessary feature. It would have been little extra work to incorporate into the data model from the beginning, even if the two values were always the same for a while.
I think the majority of the technical debt I've seen that isn't ci/cd related is disconnect between the domain model the product team is working in and the data model the engineering team is working with. Formalizing that domain model is now one of the first things I do when joining a team, so everyone agrees on precisely what the major nouns and verbs are and how they interact. Not just for the current system, for where we think we will be in 2-3 years. With everyone doing agile, it's amazing how many incompatible, un-written assumptions you discover that hadn't been ironed out.
510 comments
[ 620 ms ] story [ 5062 ms ] threadOuch. The tone of the article is a little harsh, and I’m not sure if snippets like the above are intentionally hyperbolic, but there is a fair amount of truth in it.
Most of my career has involved convincing peers to do less, and solve simpler problems with simpler solutions.
As developers and smart folks in general, we like complicated problems that are big and far away. How many times have I heard in a meeting, "Yeah, but when we have 1M users..."
It's great fun to think your product will get to 1M users. It's also very unlikely. It's not nearly as fun to finish and ship and market and monetize the half-broken thing the team is working on now. Yet that's the only way out and the only way anyone gets to 1M users to begin with.
There is no technical problem. The problem is nobody worked on actually marketing the product. Build it and nobody shows up is the norm.
We never saw anywhere near those numbers in production, but I don’t really blame them - it was a big company and you do what you can to get ahead. A lot of money was spent on infrastructure that wasn’t needed but nobody seemed to care.
Whenever someone asks me this question, I reply with a question "How many simultaneous users do you/we have today and what is our projection say for 12-18 months from now ?". If the answer is not clear, I tell them not to worry yet. If the answer is very clear but numbers are much smaller today (say 5-10), then I challenge them on where they think they/we could be in 12-18 months. A lot of times, it helps the other side see that they are mostly asking "How long is a piece of string".
I told them I'd buy another stick of RAM and scale vertically until I had more customers, and save money on staff in the meantime. The interviewer went cold, I didnt get the job.
Edit: also, in 2004 when they got started, what else could they have used?
I believe Mark Zuckerberg simply used the technology he knew and took it from there. That's fine. I probably would have done the same thing.
But many people are making an ideology out of this, arguing that not giving a shit about performance is always the right choice initially because that's how you grow fast enough to be able to retool later.
I think this is based assumptions that are no longer true.
In the early 2000s, mainstream programming languages and runtimes were either fast and low productivity or slow and high productivity (Exceptions such as Pascal/Delphi did exist but they were not mainstream). And the cost of scaling up was prohibitive compared to scaling out.
Today, you can choose any fast high productivity language/runtime and go very far mostly scaling up.
Beyond that I think there is more to unlearn than to learn from the history of the Y2K batch of startups. The economics of essentially everything related to writing, running and distributing software have changed completely.
For me the biggest win was the changes they made to how arrays are stored in memory, I saw some systems drop by half in memory usage and had to change basically nothing - those kinds of wins are rare.
1) Pick technologies that you can optimize.
2) Don't over-optimize.
Also, the concept of "optimization" here has very little to do with the language itself. It's far more about the overall stack, and it definitely includes people and processes (like hiring). It's not like FB invested $0 toward performance before swapping out PHP interpreters! Its massive caching layer, for example, was already taking shape well before HPHP (the C++ transpiler which preceded HipHop), not to mention the effort and tooling behind the MySQL sharding and multi-region that still exists in some form today. Many backend FB services were already written in C++ by 2010. But they had already gone very, very far—farther than most businesses ever will—on "just" PHP. Heroics like HPHP only happened after enormous pipelines of money were already flowing into the company.
Absolutely. Otherwise they wouldn’t have been able to release the actual product and keep adding features to it the way they did with Facebook. They’d spend half the time learning the “right” language & environment. That would have slowed them down to the point they wouldn’t have been able to work on the actual product as much as they did.
And feature-wise, Facebook evolved really quickly.
Its ease of use allowed him to launch the site from his dorm room. Iirc, YouTube was also written in php (it had .php urls), before google bought it and rewrote it using python, so you could probably thank php for that site too.
In any case, in the 2000s a PHP programmer/designer was analogous to a JavaScript developer today. Lots of talent out there, and it only took a few weeks of orientation and familiarizing for new hires to be productive.
How long did it take until Facebook engineers realised that their technology stack was not the best tool for the job? It definitely wasn't the day when they decided to build a compiler and a storage engine.
That's why I think it can't hurt to remind people of the great lengths to which Facebook had to go in order to deal with the limitations of their chosen platform.
Of course if all you know is JavaScript, then that requires going a bit outside your comfort zone.
I think this started with the old Apache web server. When the www got started, that server did the job so everyone used it. The problem was it didn't scale, so all kinds of cool solutions (load balancers and such) were developed and everyone building something bigger than a personal blog used that stuff. For most the root problem was that Apache had terrible performance. Nginx has solved that now, and we also have faster hardware and networks, so anything less than HN can probably be hosted on an rpi on your home network. OK, I'm exaggerating, but only a little. Bottom line is that scaling is still treated like a big fundamental problem for everyone, but it doesn't need to be.
I'd say they underestimate how long "just throwing money" (servers) at the problem can work.
If you earn decent money now, scaling number of app servers 10x times to serve 10x times the traffic will still earn decent money. Doesn't matter that "PHP is slow", deal with it when your infrastructure cost warrant hiring more/better developers to fix it.
Especially now. Pair of fat servers on some NVMes and 1TB of RAM gonna cost you less than few dev-months and that can serve plenty of users in most use cases even before any extra caching will be needed.
It's only after a project is off the ground that caring about these decisions winds up wasting everyone's time, that's when you wind up slowing momentum tremendously due to dangling a potential new toy in front of your team.
And overestimate how expensive it is to just add another server, or underestimate how expensive a rebuild is. But then, part of that is also that IT departments want the budget; if they don't spend their annual budgets on innovation, their budget gets cut.
Or in my neck of the woods, the EU subsidies will stop if they don't have anything they can file as "innovation".
https://thehill.com/business/despite-widespread-protest-redd...
https://en.wikipedia.org/wiki/Reddit
https://backlinko.com/reddit-users
https://www.oberlo.com/blog/reddit-statistics
https://www.businessofapps.com/data/reddit-statistics/
Those constants represent the markets supported by that app. By the time the app supports even a few dozen markets, every engineer involved will have exercised their stock options and left.
Not necessarily. It's just that most developers have never worked in a setting where they got to work on problems properly.
Solving real problems for real people is very addictive. There is a reason some people like working for startups. It's because you live very close to your users and when you make them happy you know.
The second interesting fact is that if you just plow ahead and solve many real problems fast you will eventually run into problems that are both real and interesting.
After having tried that there has been no going back for me. I am allergic to imaginary problems. It feels as pointless as watching people who are famous for being famous argue on TV.
I think we are all victims of our feedback loops. (University) Education sublely teaches us that the only important problems are those that are very difficult and preferably have never been solved before. Those same problems also make for better blogposts. In the real world the incentives are mostly opposite. Problems with no known solutions (or only really difficult solutions) are generally bad. They can be worth it, but you should stay away from them until you know that are worth it. Software engineers seem to almost pride themselves on not knowing what their users want.
It takes a while to scrub all that bad learning out and replace it with something better. Unfortunately some people are stuck.
This was the feedback loop that worked best for me. imaginary problems and needless complexity go hand in hand, ruthless editing at the planning stage is necessary to combat it.
The endorphins from making someone's job less sucky are a way better high than solving some code puzzle.
This is a great insight. One can add value to other people's lives by applying known solutions in relatively novel contexts (e.g. building a CRUD form at XYZ employer), whereas it's very hard to add value to other people's lives by trying to develop entirely novel solutions (because the probability of success is so low). Most of our training however, focuses on the methodology used to develop these novel solutions, rather than on the application of the solutions themselves.
Unless you're working on something like the Linux kernel, that's an imaginary problem.
At the beginning of a project, before I write a line of code, I try to: a) if it's user-facing software, get some UI designer input to shape functionality/interactions, not styling. To end users, the UI is the software, not just the shell, so mediocre UI = mediocre software. It can also illuminate needs you didn't consider that affect architecture, etc.; then b) block out the broad stroke architecture, usually on paper, c) intentionally choose languages/environments/tooling/etc rather than reflexively going with whatever we've been using recently, and d) spend some design time on a reasonably sane and extensible, but not overly detailed data model.
There's no perfect solution, but at least in my cases, it seems like a good middle ground.
The reality is that MVP should be missing most planned functionality and should really just be a few core features and functions that the rest of the application builds off of, the trunk of the dependency tree so to speak. That idea is, unfortunately lost on the majority of PM's, and ultimately it costs more time/money to get to a finished v1 because of it.
> Simplify the problem you've got or rather don't complexify it. I've done it myself, it's fun to do. You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had.
[1] http://www.ultratechnology.com/1xforth.htm
But their day job actually isn't writing code, it's solving a problem for an end-user; the programming language is just a tool.
Rethink your job from a coder to a problem solver and you should be able to get over the compulsion to overcomplicate things for your own gratification.
I do think that “imaginary problem” is the answer most of the time though. Most things that look like unnecessary complexity or hobby horses really are.
Sometimes boring is just pure torture.
And the gap is widening...
Simple > Complex.
It's amazing how many otherwise brilliant people dive headlong into project without considering these basic principals or even intentionally brush them aside.
Except to make that work there's a lot of infrastructure, including runtime-generated-and-compiled C++ code that gets dynamically loaded by said Python code to perform the actual calculations. Quite complex!
The true skill comes in finding the right balance between simplicity and complexity for a given situation.
In the case of FEniCS, the complexity is worth it because it allows the system to be used by less skilled programmers (who might know more about the math and physics), and the complexity handled by experienced programmers.
For our codebase we've got junior programmers who might need to read and understand my code if I'm on vacation and shit hits the fan, so I err on the side of making it easy to read and reason about. Which might not be the "simplest" for some measures of simplicity (like fewer lines of code).
[1]: https://fenicsproject.org/
I started using it in about 2014. It was not exactly what I would call a simple project. It used to be an ordeal to build, could only easily be used via Docker, ported to Python 3 a long long time after most of it’s dependencies did, had an unstable C++ API that changes were not documented for but nonetheless you were required to use if you wanted for reasonable performance for some calculations, etc. The national supercomputing centre in my country managed to only get one version to build because it was so poorly specified at the time and basically only supported Ubuntu!
FEniCsX is considerably better usability wise, but that’s the result of hard lessons learnt.
There's always going to be a minimum level of complexity. Sometimes that minimal level calls for throwing a PHP script up on some shared hosting provider somewhere. Sometimes that minimal level calls for an Enterprise-level design with ten layers of abstraction because you need to be able to individually unit test every aspect.
Being aware of where and when to introduce complexity is half the battle.
Maybe optimal performance is a feature? Feature worth working on from start?
And that performance optimization done prematurely often achieve complete opposite result. Meanwhile performance optimization done at necessity rarely have the same problem.
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."
i now take great pride when my code looks boringly obvious.
You'll be better off without it and 99.9% of the time you won't have to retrieve it later anyway.
At work we have this terrible “Enterprise Architecture” team comprising of highly paid people who haven’t written a single line of code and who don’t know the intricacies of our business but keep proposing complicated “Event Driven Architectures” and “Micro this and Micro that” and just reciting the latest buzz words just to keep appearing cool.
It’s insane how much total cost they add to the organization, both directly and indirectly.
This still results in a lot of compromises and problems. Anyhow they should be there talking to the developers in a 2-way fashion so that then end result is not entirely "from on high".
IME architects tend to be people who tell your team that you should be using Azure Cosmos after a Microsoft salesman takes them out to lunch. They last coded 5 years ago.
They are acting rationally within their belief system of making money.
Enterprise software is exactly that, software used by enterprise -- it doesn't signify any good qualities (far from it) only that it provides a cost-effective software solution to a defined business need.
The issue is when corps get bailed out, overfunded, or have revenue mostly outside of software (e.g. gov't contracts) as it eliminates cost-effective from the equation... so you just end up with buggy messes (code as a cost center).
You'd have to work in the tiny niches where tech is the true product to find good development ...and even then...
This meant I needed to deal with more JavaScript than I originally intended with my career, and eyerolls from backend architect types, but projects go much smoother than in my past, that's for sure.
“You’ve just wasted $15,000 on two months work with a team of contractors”
The project may also have been doomed because $15k is not very much for something like they described.
But again, fully aware that they probably picked a random number. I’d have just added another zero to make it more realistic.
Exactly the type of team that is super productive for these kinds of job imo.
That sounds really high to me. I personally found 15k way too high already; guess it depends on the details/market/circumstances
But let’s say 15k for two months of a team of contractors. That’s 7.5k/mo. A team says at least two, but I think given they mention sales etc, that’s 3-5 people. Netting 2.5k/person before you take out built in profit margins, healthcare (because it’s dollars, I assume America) and more. That’s roughly 1.5k/person per month. (Of course they could have multiple projects but even then that’s a low amount imho for a contractor with sales)
If they’re expecting an off the shelf solution, then they’re getting fleeced on the costs and 15k is too high, but if they’re going to a company that does big solutions than they spent too little.
There’s too little info in general
Either you have the bootstrap founders that will trawl through fiverr to find the cheapest labor (the same types that will scoff and get insulted at anything over $10k), or you have the well-funded founders that will pay whatever it takes.
From experience: Cheapskates and low-ballers will never be happy. They always want something free, more haggle room, more discounts, and always have high demands and expectations. The best thing one can do is to price yourself away from them.
all the author needs is a wordpress site with a couple plugins and a few weeks of back and forth on the design work. if you can't make a good profit on that with a $15k invoice, you're doing something very wrong.
At no point is it clear that they’ve discussed off the shelf solutions, or infrastructure (storage, server hiding, CDN etc).
If they had, then they either messed up because 15k over two months is too high or too low. It’s squarely in the: “nobody has actually defined the project territory” for me.
Yes, a Wordpress solution would work, but then why even budget 2 months for it, unless bespoke design is involved. Again that becomes a: this is too low or too high to be realistic
why should they? the requirements are listed in the article. the job is to meet the requirements. "where should we store the files" is a job the client hires you to answer, not something requiring communication.
You see the most extreme versions of dangers of this in stories about governments and older companies having to pay insane salaries to bring FORTRAN or COBOL developers out of retirement to keep systems running. If you keep doing the simple solutions within the existing system, you risk creating a system so inbred that only the folks who built it can maintain it effectively.
For less extreme setups, it's still a balancing act to consider how much your unique and specific solution that is the simple option for you company starts closing you off of the larger hiring pools in more common technologies and patterns.
The demand for Fortran/Cobol experience has nothing to do with training. We need to make sure you are masochistic enough to trudge through the sludge.
“I think we should use a Kubernetes cluster!”
“You’re joking, surely? This is a tiny web site with mostly static content!”
Next project:
“For this web app, I propose we use Kubernetes…”
Equity is the entirely the answer for cutting through all the bullshit. At least in my head. I don't know how it plays in other people's minds but mine sounds like: "If we ship and go live, I get x% of all profit moving forward in my personal scrooge mcduck money bin". Pretty big carrot. It's kind of like time share in my own personal business, but I don't have much of the headache that goes along with running my own 100%.
This has some caveats, namely that equity in a 10k person org is often times not nearly as meaningful as equity in a 10 person org. Shipping your code 2 weeks early at Ford or Dell means what, exactly? If the code you are shipping is the business, then things are different. It also really helps if you care about the problem you are solving.
I'd say this - if the idea of direct equity/ownership doesn't get you excited about pushing simple & robust techniques, then you are completely in the wrong place. You should probably find a different problem space or industry to work in. Hollywood might be a better option if the notion of equity or straight ownership in the business still isn't enough to put your tech ego into a box.
I agree for small companies which are largely founder owned. I think outside of that, Equity doesnt do much because so much effort is put into obfuscating the value/share of the equity. If you cant see the cap table, and you cant see the preference overhang, the equity is as good as worth zero. There is no discernable value for a fraction with no denominator.
I've often found myself now having issues even updating my resume, because what I did for the last year at work barely is explainable to other people on my team, let alone to someone in HR at another company. Or the more simple explanation is something that sounds like I'm doing work barely more complex than an intern could have done. Which often isn't wrong, but the intern wouldn't know which simple work to do.
My years of experience in the company's stack and org is valuable to the company, and nontransferable elsewhere.
On average, and depending on skill. Incentives are hugely important (probably the most important metric any manager could work on), but even they do not guarantee results. If you hire so many juniors that nobody is there to upskill them fast, you only get one lottery ticket per employee. Conversely, if you hire a bunch of geniuses and fail to give them incentives to work on realisable, useful problems together, you get two lottery tickets per employee at twice the price.
(This comment feels woefully incomplete. Does anyone know of good resources to learn more about incentive structures and how they relate to individual and company success? I feel like the problem is that incentive structures change massively when companies grow, so even for unicorns there's just a short sweet spot where we can actually learn how they are supposed to look.)
But the main argument is off - What some classify as imaginary problems is actually building in the wrong types of affordances. The only way to know the difference is by knowing your tools and the problem domain.
The root of bad software is leaders lacking practiced learning.
Yes, this won't help if you're literally in danger of not meeting payroll next week, but I would hope most businesses don't operate so close to the edge.
Replace those devs with ones who are focused on the right things? Great, but you still have the other problems to deal with (scope creep, uninformed clients), not to mention a host of other things that can derail a project, such as poor communication.
tl;dr Most projects are poorly managed. Poorly managed projects tend to fail.
Further, project managers tend to focus on delivering the product stakeholders have asked for on time and on budget.
Product managers however I find are frequently searching for a new expanded scope of stakeholders to brag they're solutioneering to. I find them more often in the "solutions looking for problems" business than I find developers, or at least developers are only doing it on a micro-scale, while product managers will take an entire team/org on 6 month fruitless missions to build software destined for the dustbin.
This mentality is often taken way too far. I had an old boss who wouldn’t allow me to write unit tests citing this thought process.
Even at places with decent engineering practices, I’ve seen so many examples of software where you’re limited to a one to many relationship for something that could and easily should have been implemented as many to many, rendering the product useless to many because a product person couldn’t stand to think a couple months ahead.
Some people seem to take this idea too far and basically assume that if a problem is interesting or takes away a tedious task, it must be overengineering, premature optimization, or an imaginary problem.
Perhaps a better way to phrase the issue would be “artificial constraints”, which would encompass the flip side too.
3 days later I deployed my “nice to have” fix and the performance issues disappeared.
I’ve also seen a company stall out scaling for years and lose multiple million-dollar customers despite having a novel in-demand, market leading product because they refused to do anything to clean up their infrastructure.
YAGNI isnt about not refactoring existing technical debt. It's about not trying to pre-empt future requirements.
If youre refactoring in anticipation of as yet unmaterialized requirements then YAGNI applies - e.g. generalizing code when when today there is 1 specific use case because tomorrow you think there will be 3+.
If youre cleaning up existing code while working on it and the boss stops you because "it's not customer facing" then he's just asking you to violate the boy scout rule.
I can give you an abundance of examples. We were creating a page that was going to use state in a certain way. I was trying to insist that we address the way state will be handled across pages ahead of time. These concerns were dismissed as premature optimization. A few months later we had 5 pages with the state being handled in 5 different ways, and being synced in different ways between each page, complete with if statements, sometimes passing state through URLs, sometimes through local storage, sometimes through session, sometimes through JWT data, generally through a combo of several of them. Then we'd end up with confusing redirect loops for certain edge cases, state getting overwritten, etc.. We spend weeks fixing these bugs, and, eventually, weeks refactoring to manage state in a simpler way. These bugs often got caught by customers, drawing us away from feature delivery that was critical for demos to large customers.
All of that could have been avoided by spending 1 day thinking a little harder and planning for the future.
It ultimately boils down to a couple assumption that people like to make. (1) engineers know nothing about the domain, they can never predict what will be needed. That might be true in a large company with obscure domain-specific things for engineers who work far away from the day-to-day, but sometimes the engineers know exactly what's going to come up. (2) You can hill-climb your way into optimal program implementation. You can get to local maxima this way, but there are regular ways that programs grow based on how the business is growing and you can predict certain places where you will soon hit diminishing returns for current implementations. As long as you're up front about it and double-check your assumptions about the way the business is growing (and hence the application), I think there are ample places where you actually are going to need it.
The right time to address this was probably a bit at a time after the 1st, 2nd and 3rd pages. Certainly not before the 1st and definitely not after the 5th.
>All of that could have been avoided by spending 1 day thinking a little harder and planning for the future.
The reason why you try as hard as possible to avoid planning for the future is because it's really hard to predict the future. Moreover humans have an inbuilt bias towards thinking we are better than we are at it (hence the gambling industry).
Refactoring as soon as possible after the fact will always produce better designs than up front planning for this reason.
>there are regular ways that programs grow based on how the business is growing and you can predict certain places
This is the kind of phrase that makes alarm bells go off in my head that somebody SHOULD be following YAGNI and isnt.
If it's defaults in a well worn framework that doesnt railroad you then fine but anything more than that - red flags all around.
HOWEVER, this only works if you have the agency at an organization to allocate time for doing something. Contrary to this when you are in an organization of management that doesn't understand technical debt (or is fine with it because it just means more consulting hours) then it's absolutely the correct choice to stall and/or fix things "prematurely" (iff you can see what product they are trying to create without knowing it) if you'll be holding the shit-can of duplicated crap down the line getting knuckled because things aren't going fast enough due to technical debt.
In that case the temptation to close the ticket and skip the refactoring step can be too great.
If you're begging for time to refactor from the PM youre doing it wrong.
Preventing fires will never be a priority. Not even if you smell smoke.
But tautologically you can't take YAGNI too far, if the "YAGN" part is actually true :-). But that is always under debate.
I've had to do up front on multiple projects and it always results in overengineering - we focused on things that didnt matter, designed things that were inappropriate, etc.
I'd always rather take those 3 days and redistribute them as extra refactoring time.
I was creating a very important demo once, features i had said were important were classified as YAGNI. Leadership eventually saw that we couldn’t deliver without said features. YAGNI bit those teams in the butt.
these things happen all the time internally to companies but get ironed out internally as well.
Every feature is done quick’n’dirty and eventually you have people whose full time job is to respond to customer complaints and fix data straight in the production database.
The secret, as a software vendor, is to generalise these bespoke customer requests so you can sell the solution to all your customers (and get more customers!). If you are really cheeky, you can even get that customer to help fund the development that will make your business more money (hey, it’s win-win). You need to ruthlessly follow this approach though, as the rot of bespoke code will quickly become an insurmountable quality nightmare that can sink your business.
If it's something more fundamental like language choice or system architecture... Well fine YAGNI now but if you ever do need it you're screwed.
I don't know anything about banking software, but I have a hunch the author underestimates the complexity (as is typical for HN). The Efficient Markets Hypothesis would suggest that if it were so simple to make banking software, then someone would do it.
From October 2022 to now the shareprice tripled, adding half a trillion dollars to its market cap.
Explain that in terms of the Efficent Markets Hypothesis? "share prices reflect all information" - what half trillion worth of information change came out in 2022? and what in 2023?
What about insider trading laws? How can we say "share prices reflect all information" when we know there are people who have more information which would give them an unfair advantage, which means the current shareprice cannot be reflecting the information they have?
2. Interest rates affected the entire market
I run into this more often than problems I imagine. A whole series of what ifs and folks imagining things. The worst part is the fixes to their imaginary problem are usually not well thought out and drive things towards worse choices.
I’m a big believer in getting version 1 out the door as problems people imagine often… are never relayed by the actual customer.
I often work with some routing software (let’s say routing packages). There is a simple mode that works great. Anyone can use it.
The issue is people want to establish say 20 rules about how / when / what is routed. Business folks insist that it be “easy” for “anyone to use” just like the existing easy mode.
This is doomed from the start. We can make it easier for sure, but if you have 20 rules with different weighted priorities:
1. It is complex for most people to think of. It will look complex because it is complex.
2. That’s ok because the guy with 20 rules probably has thought about them and understand they have 20 rules.
Then we give them UI to visualize it all and customer is happy.
But the business folks are upset because the visualization is complex… and there we are again.
For the record I usually get through this slog and everyone is happy in the end, but it is a slog due to imaginary problems.
Crucial to fight against this kind of stuff.
Bad languages and tooling is the root of bad software.
Need a bog standard app to stream audio files? Give it to the person you hired right out of college, or maybe even the summer intern. She's never built something like this before, so to her it will be a challenging novel problem. A (somewhat) more experienced developer may need to provide initial guidance and review code, but that's a comparatively minor time investment, and besides, the act of mentoring someone else should keep it interesting.
this runs into another common mistake businesses make: putting people onto a task because they're available, not because they're suited to it (in any sense).
the junior person (+ a little mentorship) would be great for the job, but they're mired in some big project. but hey, you've got this super senior fellow sitting around waiting for work, and we can bill the customer more for them, anyways.
They shouldn’t be checking in after two months. They should have regular check ins.
They shouldn’t have had such a vague brief.
They should have discussed a wide range of off the shelf options from the get go to see if they needed something bespoke or not.
A restauranteur wouldn't develop a menu without input from a chef or prior experience as one. A layperson wouldn't design a real building without input from an architect or engineer.
Yet for some reason a myriad of non-technical people (read: laypeople) feel empowered to design, spec, and strategize about software. It still boggles my mind that "product management" is a real profession.
I remember working at one place where somebody built a new framework to solve a common problem we all had. He pitched it to all the other devs in a meeting and I remember being confused about it because there was a standard framework that solved the problem already and did it in much simpler and more elegant way. (To be fair to him, this standard functionality was only recently introduced.)
During his presentation, I asked why the standard solution wouldn't work for him. It turns out he wasn't familiar with it. Fair enough, so later I messaged him and showed him the standard way to do it and how much simpler it was. He couldn't be swayed.
He just couldn't accept that his complicated solution wasn't necessary. He constructed scenarios where his idea was needed, even though I saw solutions to those scenarios using the standard framework.
Interestingly enough, one of the scenarios where his custom thing was needed was in some tests he had written where he did some complicated things to set things up. I looked at the tests and even there saw those complicated things weren't necessary! There were ways to simplify what he was doing so that the tests were better written and didn't need his custom tool.
Anyway, he wouldn't be convinced. And because he couldn't be convinced, we got stuck with his solution and saw people continue to work on it, add more functionality to it, fix bugs, etc. All of that work was just a waste of time when we could've relied on a standard solution, which was way more mature and way simpler.
All of this drove me crazy, but I realized that sometimes people are just unable to see simple solutions to problems. Worse, having one complex solution begets more complex solutions elsewhere.
It's hard to admit that your baby is ugly.
This person might just have been terrified of admitting that all the (very tangible) time they spent on payroll building this thing was for naught. It’s unclear how their manager might have responded to that. And they wouldn’t have been able to put „built system doing X used by Y developers and deployed to Z customers“ in their resume.
The reasonable choice often doesn’t have much skin in the game.
Then we ended up deciding not to make money out of it that way. So we burned enormous effort and created a horrible design for no reason.
So IMO the problem is usually with the way requirements are not usually well understood even by the people asking for them. Later on it becomes clearer what is needed but you're stuck with false assumptions baked into your design in a way that you never have bandwidth to remove because you need so much bandwidth just to do normal work....because of those assumptions.
YNGNA is generally true, but Product's should have a very clear vision of what kinds of entities the system is going to handle over then next 36 months before they start asking for specific functionality. I've seen extra shit get built, but I've also seen e.g. a travel booking system that was built without a "flight" being a first class entity. Flights were deduced on the front end from attributes attached to seats... which worked well until a PM asked for the UI to show fully-booked flights, which HAVE no available seats that make it to the front end. Same product couldn't handle the booker and traveler being different people, when they knew from day 1 that it would be a necessary feature. It would have been little extra work to incorporate into the data model from the beginning, even if the two values were always the same for a while.
I think the majority of the technical debt I've seen that isn't ci/cd related is disconnect between the domain model the product team is working in and the data model the engineering team is working with. Formalizing that domain model is now one of the first things I do when joining a team, so everyone agrees on precisely what the major nouns and verbs are and how they interact. Not just for the current system, for where we think we will be in 2-3 years. With everyone doing agile, it's amazing how many incompatible, un-written assumptions you discover that hadn't been ironed out.