> The problem: to achieve the desired level of UX, we needed to convince stakeholders that a single-page-app approach would be needed, including some APIs to fuel them.
At one job I ran into this pretty badly. The team I joined had an internal app that was, basically, a bunch of forms connected to a database. It had a lot of problems -- the data models didn't actually map to what the business needed, the UI was unintuitive. My manager insisted that what we needed to do was rewrite the entire thing as a React SPA. I told him, well, we're just going to end up with the exact same problems, only now implemented in a different framework. This won't solve any of the actual problems. He insisted, though, saying that React was "modern."
Thankfully, the depth of mismanagement here was pretty exceptional in my experience. In most cases, even if I'm personally skeptical of the need for an SPA I can see a reasonable argument in favor. The cartoonish straight out admitting that the sole purpose was to be more "modern" with literally zero regard for actually serving any user needs is thankfully not something I've run across too often.
There are markets where being seen as old-fashioned is fatal, so this might not be irrational. Just because it's internal doesn't mean they're not worried about the wrong people seeing it in an old-fashioned state. People who worry about that (for good reason or not) don't always have a boundary on that concern, and sometimes internals leak.
In fairness, his audience was his manager, not the app's users. So it was perfectly rationale for him to push for the relatively easy re-write instead of worrying about the app's users, given that his manager was not a user of the app. As an engineer who finds value in making things people actually find useful, though, it was a pretty dispiriting experience.
It happens more often than you might think. Rewriting everything seems to be fun/easy to inexperienced developers. It takes lots of scars to realise that nope simply improving what is already somewhat working is usually a much better solution.
One thing that vendors have that internal teams don’t is million dollar marketing budgets. Selling the sizzle of successful internal projects is its own kind if work, but if neglected even the greatest of projects will succumb to the onslaught of vendors. Good technical leaders recognize this inherently and celebrate their teams creations, non-technical leaders don’t and take pride in buying from big brands and the fancy luncheons that come along with them.
If you break down what most vendors and consultants do, in terms of either dollars or hours, you'll usually find that they mostly do sales, PR, client relationship, etc. Projects are designed and managed to make the right people look good to the right people, manage expectations, etc.
They'll spend massive time giving non-tech executive buyers easy, fun choices.. let them feel strategic, visionary and empowered. They'll communicate the hell out of every piece of progress. Whoever hired them also has a massive incentive to buy into and sell onwards, that the project is a massive success.
IE... if your job is to make a clients' Salesforce instance do a thing, 80% you your job is stuff other than making a clients' Salesforce instance do the thing.
Internal dev teams will rarely have this pageantry. Meanwhile, no one outside that team knows what they do, how well they do it or such.
If you want the 'aha' moment have a look at most software vendors incomes statements.
They spend 2x more on selling than they do on R&D.
So when that Oracle Sales Guy is giving you the pretty talk, and taking you to lunch remember that all that money your sending to Oracle is literally mostly going to him! The product makers just get 'a cut'.
Now consider how much of high tech is just that, sales and marketing, conferences and luncheons.
That's a staggering waste for a developed economy.
We are very inefficient with these things, we need better approach to IT.
> Big solutions to the big problems. Exactly what consultancies sell. It was obvious to them that having any inhouse-developers is the biggest business risk at all.
I will never, ever, ever, do in-house software development ever again. Non-software companies treat their inhouse developers like crap. They don't get it. I don't know when non-software businesses thought it was a good idea to move software developers in house. Most of the company don't want them there. Or deal with them begrudgingly. It's hard to quantify their revenue so they are treated as an expense not cost saving or revenue generating. And they are the first on the chopping block when things turn down.
In my experience, if you are recruited by a company that doesn't sell software or software services you should tread very carefully, ask lots of questions. Sometimes you'll find out a 100 year old furniture company only has a dev team because the grandson thought it was a good idea. And then you run very far away.
My company sells robots, not software, and treats us like royalty. Perhaps “tech companies” fall under the umbrella and I’m just misreading you?
Edit: I’m confused then: if robots have software therefore it’s a software company, then tons of companies are software companies. But so many of those products, the software is just a hidden detail that no customer ever thinks about or cares about. Where do you draw the line?
Yeah, it's about proximity to the core business. People usually don't join FAANG to be HR or lawyers, but developers. I'm sure it's the same at other places - why join a bank as a programmer, or a law firm as a designer? Go where you're valued.
Obviously you're a software company, you just happen to have to sell the platform for the software to run on as well.
(There is an interesting discussion to be had about where the boundary between "software" and "hardware" companies is, but an environment with a high level of wilful misreading isn't one of them)
A robotics company is still a software company. Unless there is a human standing at a bank of switches directly controlling the robot. I'm talking about non-tech companies. The start a inhouse devteam when they really needed a consultant.
There's a large class of products for which the software is essentially an after thought. And it shows.
Appliances are an obvious example. But also things like cameras to a large degree. Or auto infotainment systems until Apple and Google software was integrated. (Embedded is sort of a different category that is functionally essentially part of the hardware.)
Not the same thing. Robots are literally programmed to perform a task. A travel booking agency? I don't give a shit if it's AI or 1000 gnomes making hotel reservations. In the former the software is part of the product, the later travel is the product.
The parent is likely referring to companies that needed a square space/Wordpress site and maybe a Shopify account but instead hires 3 scrum teams to build their e-commerce platform.
The software output at such a company is confused at best. Usually with some exec pitching digital transformation without definition. No one wants to deal with the tech team, pay and expectations suck, getting hardware can be a years long process, and eventually someone pulls the plug.
You mentioned pay, the pay at these places is terrible. I got an Engineering Manager offer for a company I worked with years and years ago, the top end of their budget was $90k. For the manager. Devs (there were five) were making $30-45k. At that level of pay you're simply not going to get good people. Even if you get one or two who are decent engineers, they're not sticking around when most of the team isn't that great. WLB is important but there are plenty of places paying $80-100k for mid-levels in LCOL/MCOL areas where a 40 hour week is the max.
Both Google and Netflix have unique software products that make their main business functional. I've worked on internal projects and the difference in management attitude toward them and the real product is rather palpable.
I think we really need better terms. What is meant by software company in that whole discussion is actually two things (at least):
- companies for which software is the sold product (or its support, hosting...)
- companies selling a service or a product that requires inhouse made software
It could be argued that this second one could be split in two as the software may not be delivered directly to the client but the border is a bit blurry when it is a website/app platform like netflix.
I was an in house dev and have friends that do it. Con is low pay. Pro is usually great WLB, as delays and confusion aren't up against a hard deadline.
The pay isn't really all that low when you factor in all the OT tech companies typically require. Also, the retirement packages are typically much better outside of the world of tech companies.
> Big solutions to the big problems. Exactly what consultancies sell.
The problem is most consultants are not good at what they do. It why the sales guys for the consulting company are not the devs. Most the time consultants come in for 3-6-12 months, cause some trouble and then just leave the mess to the company to address later. There are some good consultants but most are just people that couldn't work in house because they are not up to par for long term engagements of products that need to stand the test of time to add legitimate value.
> It's hard to quantify their revenue so they are treated as an expense not cost saving or revenue generating.
This depends on the business, most the time, unless you make software you are an expense. The company makes their money in different departments and making money is the name of the game. It's takes a different set of skills (albeit better/stronger, industry knowledge and communication skills) to be an in-house developer vs a contractor.
>The company makes their money in different departments and making money is the name of the game.
The challenge here is that under this model the different departments get credit for the revenue they generate. What their relationship is with the in-house dev team is very much something that one should be aware of if they choose to be in that situation. It's not too dissimilar a situation to every single 'change management' model that has executive commitment as the number one success factor.
I agree. It's a different animal that being just a developer at a consultancy. You are supporting other departments to become successful. If you are undeniably helpful people/departments/executives will know that. I personally do not care for credit at all for tasks accomplished. It's always the journey I find more rewarding. Give me my salary, benefits, opportunities to grow, etc... and Im good, but I know not everyone is like that, especially users on HN.
That's the thing unfortunately there exist places where the opportunities to grow are non-existent because not every executive realize what you are bringing. And their n+1 will hear about conflicting opinions and make their mind about that. You and people you helped may not have enough political power to save your ass.
That is true, there are always bad places to work at. But by and large Ive worked at more places that appreciate good ancillary work than oppose it. It's important to work efficiently and economically (which is the antithetical to consultancy), communicate well, (what are you doing, why its important and why it will help the company tomorrow) especially to non-technical people. It's good to be able to speak clearly to C-level individuals spending their annual budgets on "IT services" and to also speak well to the people implementing those services. Seeing the broad big picture, keeping it in focus and getting the job done to help the company is invaluable.
I have been in the same boat. It wasn't 100 years old, and it wasn't a furniture company, but it ended exactly the way yardie described it :-) No one wants you there because they see you as a waste of money and a threat. A lesson to be learnt I guess.
I think it comes down to if the company views your job as a cost center or a profit center.
Software developers at software companies/tech companies generate profits, so they are treated as such.
Software developers at more "traditional" businesses (retail, industry, manufacturing, etc) are often seen as an expense and something that should be treated like any other expense - try to minimize it.
Looking at the original article, it seems the author was working for a software company that was in denial about being a software company:
> The amount of customization by far exceeded the initial offering of the services, and had to be done within the constraints of each individual technology stack.
>
> This accumulated in hundreds of partially overlapping systems, wired together with brittle ad-hoc integrations.
They had a software solution and were officially spending zero dollars on software development. You can't beat that. Never mind the huge amounts of time they spent keeping it working. All of that money showed up in other places on the bottom line. Maybe a lot of it was IT, but a lot of the expense was probably people whose job title had nothing to do with technology spending hours doing repetitive and error-prone integration work with third-party systems, emailing around spreadsheets and CSV files, manually tracking down and fixing bad data, and answering the phone for confused customers. As soon as you start employing software developers to create a more efficient system, it looks like you're spending 100x more for software, on a trajectory to take a very, very long time to achieve parity with the existing system.
When I first got into manufacturing, I encountered this too, but I saw it as an opportunity. I was the sole (and first) developer when I came onboard, and within a few years, I had a team. A decade later, and the entire company depends on my team on a daily basis, because we've automated tasks and provided tools for every department. Not to mention that I've broken company records for salary increases on multiple occasions throughout my journey here. So, maybe it was a little more work for me to get them to "get it" with regards to software development, but I did it. That said, I do understand that not everyone wants to have to put in that extra work, but it does feel damn good if you do.
> That said, I do understand that not everyone wants to have to put in that extra work, but it does feel damn good if you do.
I do think this is valid, but feel I should add: that willingness needs to go both ways. You were willing to put in the extra work, and your employer was willing to "get it".
There exist organizations that will never "get it" despite the extra work, and these are the places others are talking about when they say "run away".
I experienced both variations of this earlier in my career. I was successful the first time, and tried to apply that success in a larger environment.
The lack of organizational willingness to "get it" in that second environment is what motivated me to move to a place where building software is the primary reason for existence. I do not intend to look back.
The lack of organisational willingness is usually a combination of managerial debt and veeps/senior managers whose annual bonus depends on keeping things as they are.
Managerial debt can happen anywhere, including tech firms. It's where you're dealing with people who don't understand what recent technology can do. Maybe they got it 10-50 years ago. Maybe they were very good at it 10-50 years ago. But today they're behind the curve and unwilling or unable to understand that a working modern solution is possible.
They may also be risk averse for perfectly rational reasons, in which case you need to sell damage management security promises and fall backs with any upgrades.
The bonus issue is a very real problem. If there's pushback from that level it's often because those people can see dollars moving from them to the newcomers. And they'll do almost anything they can to prevent that.
These are very difficult issues to deal with, because they're usually hidden behind spurious excuses and rationales and passive blocking actions.
I think such companies will tend to be replaced by companies built around IT - because they can gain the benefits of software, of flexibility etc (compared with physical objects, anyway).
1. I worked with a single internal guy to build a web-based ordering system, in the days where people faxed in orders. This replaced weeks of data entry, dozens of part time data entry people, and enabled a big change in materials ordering and production (orders were all in a database within a few hours, vs literally weeks of keyed-in faxes from before).
The one guy that was in charge of all this suddenly had a lot more power internally, and it pissed off a lot of other folks who suddenly had less influence.
2. More recently, worked with a team of external contractors, and internal folks were resistant to help any of us set up external dev systems. They all worked in the physical offices, and had no need for 'run a dev system on a laptop'. One of our guys spent a lot of time (and kept getting chastised by the client's mgt) for getting and documenting a build process for external contractors to use. The pandemic hits, office buildings are shut down, and internal folks have a much harder time figuring out... how to do anything from home.
In both cases, under-the-radar skunkworks projects ended up having large (and in one case transformative) impact and value, but were ignored or fought by those with entrenched power.
For decades now, per Alistair Cockburn, I assumed people just really, really hate change.
There's some research which shows our selfish little brains have an immunological type reaction to change. Anything that threatens our internal stable equilibrium is squashed like an invader. No matter how much we say we want to change. No matter how much we may need to change.
While wholly correct, turns out Cockburn was an optimist.
We now know that some fraction of people would rather watch the whole world burn than give up one iota of power.
There was an article earlier this week about why every engineer should do a stint in consulting.
This is why.
The reason companies like this use consulting companies is because the VPs want total control. Internal IT is purely a cost center, and unless your company (read: the business) is already on board with the product lifecycle op model, there is no point in trying to do any of this from within IT. Consultants work at the VP level and in a large enough company, we find a technical area (usually data as it has the biggest business returns) to modernize, then leverage success there build out the business case for the rest of the company (data is a good place to start building API gateways). And selfishly, the business is willing to pay higher hourly rates than internal IT (which we generally leave to the offshore MSPs).
When I sell transformational solutions to a company, I know it won’t be successful without top-down buy-in. Consultants are also better positioned to do these transformations because we do them all the time; and the exercises around stakeholder management and consensus building are extremely important. They’re the kinds of things internal teams are never resourced or trained for, and they don’t happen quickly.
The problem in consulting is that you almost always end up in an internal turf war. As mercs, you can pack up and leave once the paycheck runs out.....
Indeed, however for me personally being a merc is much more enjoyable, specially because each project is a different stack, so it easier to change than being labeled "X Developer".
Plus thanks to those political games there are lots of soft skills to learn as well.
> So while many team members were arguing about theoretical things, I onboarded important developers on to the new system and they were productive refactoring the old things we produced into the new technology stack.
I question the wisdom of somebody who started refactoring and replacing systems right after massive layoffs in an industry that was crippled by the pandemic.
Businesses in survival mode can't spend resources like this. The author points out the executives became enraged when the found this out and I agree with the executives.
The pandemic was also a chance to implement new stuff, as there was anyway no real revenue generated. Everything was chaotic, so if you made mistakes you could just brush them off with "because of covid".
> I question the wisdom of somebody who started refactoring and replacing systems right after massive layoffs in an industry that was crippled by the pandemic.
Well wisdom is gained from experience and it seems the author learned a few lessons.
And... when everything failed except the new system, which seemed to keep the company operational despite the executives' views... you still agree with them?
You agree with their 'enraged' state? You think it was justified? Even with hindsight, or just at that moment in time?
Just from the point of view of some manager: perhaps worked many years at a company with the ultimate goal of entering C-suite. Maybe gave up other good opportunities for this goal. In such situations it must suck a lot to see your power and influence wane. You might have made many sacrifices and then someone starts some projects that could ruin everything you worked so hard for.
Edit: It's a legitimate solution. I've survived 3 reorgs at my company because I could code. I read up on the new incoming system and became effective.
I think a lot of it is that managers aren't really the captain, or at least lack the same expectations. They are more a contracted operations manager who can hop in a lifeboat first or can get paid more steadily if the ship runs aground and therefore they are more needed to clean up the mess.
One of the few hyperlinks is to an article by PG about how using Lisp enabled his small team to build amazing stuff that ran circles around larger teams that were using other languages. Based on that link, with no other knowledge, I also assumed that meant the author might have used Lisp, but it also makes sense in the "higher order language", or even just "writing custom code to solve problems" sense.
I think it's more related to the language power continuum.... he was probably coding in a "more powerful language" while the rest of the org was coding in a less powerful one.
Interesting tone. The tone says, all these people are idiots, and they failed to understand my beautiful creations.
But he then outlines all his failings at the bottom, which I agree with, so he clearly knows better. It's it possible that this tone deaf attitude was a primary culprit in the downfall of the organization he and his handpicked successor built?
You can't build something in a vacuum when there's other people with power in the organization. You need to treat the system as a whole including the politics involved. Sometimes that mean a less optimal solution to gain some influence with key stakeholders.
This is one reason technologists can make bad managers. They don't think about the politics or don't want to play it.
His tone isn't "you are all idiots". His tone is "you all don't care about real progress, just selfish politics".
The exasperation comes from total breakdown of the internal story "we are all here to make the company better". Whilst at the same time people keep repeating that line.
Yes, you state it better, and my characterization was a bit facetious. The common thread is that, he implies, or explicitly states, that he has the organization's best interests at heart and others don't.
That might be true or not, but you can't operate with that mindset and expect people to come along with you. Whether or not it's true, you need to presume positive intent. Perhaps they see aspects of their jobs that won't automate well that you do not. Is that stubborn self interest, or standing up to a naive new leader who doesn't get it?
The hardest and most important mindset shift for me was when I realized almost every person who supports terrible policies/politicians does so thinking they do it because they care, and they do from their particular perspective. I can't usually bridge the gap with someone who commits or supports horrors out of "concern for souls[like mine!]," for example, but most situations aren't at such a timeless impasse.
Every family member who ever turned into a whiny baby while trying to do free IT for them saw themselves as the victim, or the person under threat. That's the kind of person with power in most organizations, and you have to find some way to meet them in the middle without compromising everything to get anything done. It's not enough to be right.
I believe this for a lot of the people who support policies or politicians.
I doubt whether this extends to the politicians / executives enacting those policies. They will certainly frame it like that, but for a lot of them, their underlying thinking is going to be much more self-serving and about manipulating the narrative.
Huh, I had a different reaction, I think his conclusion is: Technical correctness isn't enough, the enterprise is a social organism and you need to get people on board to come with you seeing how it will achieve their own goals and include them, and that's the lesson I learned from this experience.
Yes, and I agree with that conclusion. But your comment demonstrates an empathy and understanding that the blog post did not. I don't think it was quite a rant, but it had that "I was right" tone to it the whole time. An awkward fit with the lessons learned at the end, and coincidentally, another possible explanation for why things went so off the rails.
Not how I read it at all. To me it reads like a story of somebody thinking he is doing the right thing only later realising that it doesn’t matter if you don’t get buy in from the people who have the real power in the enterprise.
Sounds to me more like a failure of upper management. The CEO at least should have realized that such a change needed buy-in from everyone and creating that "digital business unit" was going to achieve the opposite.
Always plenty of blame to go around in situations like these, and CEO operates like the goodwill line item on a spreadsheet. He takes the blame so we can all move forward.
Not at all. He clearly points out that he underestimated the power of internal politics. And that creating working solutions is not enough to be successful. That’s a lesson we can all learn from.
I reread and I think you're right about most of it. But the scare quotes in Gaining Momentum really frame this as a battle between good guys and bad guys, which isn't a productive framework to think about politics. (Not productive, but can be effective depending on your goals, unfortunately).
Yep good point. Thinking in good/bad guy term makes everything a zero sum game. Finding ways to turn it into a win-win situation makes a huge difference. And often it all comes down to how a solution is presented to a decision maker and how much the decision maker feels that he/she is in control. The perception of the solution is often more important than the technical details of the solution.
> And then the obvious thing happened: a complete breakdown of all important systems, like a house of cards. And: most vendors were not working due to lockdowns or other important issues during the pandemic.
> The only system not affected by the complete meltdown was: our new one, which was designed to withstand outages from backend systems below.
I don't understand, how can their SPA frontend still be functional when the backend is totally down? You can display things to the user, but you cannot do anything meaningful from it?
Depends on what the backend systems were doing. They may have been data feeds that could be cached, with the “front-end” actually being a whole app with its own back end and database. Or perhaps taking orders and queueing them up for later delivery when the back-end systems came back on line.
I've been both in-house and consultant to a variety of big corps in a variety of industries, and I can say that invariably every single one is messed up and inefficient.
Yet still somehow customers buy the products and (more often than not) the company delivers a product that is half working enough to be tolerated and used.
Meh. When it seems like everyone is stupid but you... you are probably the one who is on the wrong track.
This is all very vague, so who knows.
But at the very least, I would guess the OP's attitude of I-know-better-than-everything-that-came-before arrogance plus ignorance of why things were the way they were made big failures inevitable.
It's a real limitation -- though a common one -- to need everyone to see things from your perspective, rather than being able to understand their perspective.
a vast amount of people think horse dewormer is a viable strategy. If anything there is a crisis of "my ignorance is just as valuable as your experience", and not the other way around.
I agree with the author that the big failure point was the "triumph" that resulted in a new fully-separate department, before any of the covid-triggered failures or new development. You've lost at that point. You can't make it me against you, in-house software vs everybody else, you have to share the pain and credit. Otherwise only you feel the pain and any of your "wins" will be improvements that require time and expense from other departments without them getting anything tangible comp- or recognition-wise in exchange. Let alone getting into the "this means the jobs of people I employ to do it manually in the old system will be gone" aspect, which the author touches on too. (And I imagine that even the big failures during Covid maybe weren't as terribly critical to them as to the dev team because of how low-volume the industry was.)
"When there just is no interest in actually improving things, I should leave it at that. "
Yep, if you can't convince them to care about and drive the improvement themselves, you're gonna have a hard time if driving it becomes the most important thing to you.
This is just how company politics works. Those executives wouldn't be in their positions if they didn't look out for themselves first. Also it is turtles most of the way down. This experience should be a good lesson.
Only the larger of non-tech companies can get away long term with internally built tech. This is because only they have the money to build and maintain teams and build and maintain the technical expertise.
Sometimes, it still takes a massive bite in the ass for these giants to wisen up. For example, see Home Depot and their credit card scandal and then their subsequent investment in self checkout, online sales, and lockers. They have done really well for themselves.
Smaller companies eventually either get swallowed up or destroyed by bigger companies, by disruptive new players, or get a new lease on life due to the creation of a service layer below them.
Nowadays those new disruptive players are often tech plays.
Also the new service layer is often tech. One example of a tech service layer is Shopify. AirBnB and Uber are examples of industries getting swallowed up by tech. Expedia is another example of disruption from the previous generation. Home Depot swallowing up more of the construction supply industry is an example of bigger players crushing smaller players. Bye OSH.
> I started talks with upper leadership to move all the people currently working on building/improving products internally into a dedicated business unit.
New things go with old things. The people who maintain a product ideally should own the improvement or replacement of that product. That's how you get everyone on board. And if you can't manage to do that, then you'll probably end up in some kind war.
> Other executives were afraid of the new VP. Not because he was inherently evil, quite the contrary, but the pure existence of his new role meant intermeddling with their own topics to them. Something executives tend to dislike.
How could you possibly know that they were "afraid"? More likely, a group of people just had an abrupt re-org with little-to-no input and now had a new poorly-defined internal "digital" group that failed to provide clear direction on how it was going to replace an existing 3rd-party process. If worked suddenly ground to a halt and the new department was the only thing that changed, maybe existing leadership wasn't the problem?
> Without any real support from any other executive, our poor VP was tasked the impossible: digitalize the enterprise... but without any decision making power outside of his own department.
What about the other poor VP's? How did this new digital VP engage with them, individually and as a leadership team, to ensure that this new department had a well-structured process by which it provided value to the company? Did this new department try to lead the agenda, or simply steer it? You should be aiming to provide a service, not expect service from others.
> Executives went full enrage when they heard about what we did there
Well, yeah? This reinforces my opinion that the new department completely failed to engage the company. I'm sure the executive leadership team had more important things it wanted accomplished. How did adding Phoenix LiveView to an existing mess of a stack add value to the company in a way that aligns with existing objectives? Was the digital VP even consulted on this decision?
It's also possible leadership at the top, a CEO/founder, didn't know how to manage an executive leadership team and the dysfunction went too deep for a VP to fix. If the leadership team isn't meeting regularly and on the same page and collaborating with the CEO to determine a roadmap, then something is very wrong. You can't fix bad communication between leadership stakeholders that can't decide amongst themselves what priorities are.
> they wouldn't even come up with such ideas on thir own
You're telling me that a group of people that had been successfully making demands of 3rd-party contractors and companies couldn't do the same of an internal team? I don't buy it. How did they get to such a tangled mess in the first place?
> there are sometimes no other options than doing it as we did
Maybe so, but how did you bring along other stakeholders in this decision-making process? I wouldn't tolerate this attitude from a contractor/freelancer, or a full-time employee. A big part of leadership is convincing others of the value of a certain path, and failure to do that isn't the fault of the other party.
> Without toplevel commitment, best efforts and even results don't matter.
Yes. In the vacuum of clear leadership direction, other people will find opportunity to fill the empty void with work. But separate leaders working in isolation, even creating one-hit-wonders, will never be successful as a team collaborating on even a mediocre agenda.
> When there just is no interest in actually improving things, I should leave it at that.
I don't believe this for a second. Either there was complete absence of top-level leadership, or the new department wasn't structured to service the business and failed to build the confidence of other stakeholders that it could deliver value. I see a lot of blaming the other guy, and very little other actual introspection other than "without including all people affected by this".
126 comments
[ 4.4 ms ] story [ 183 ms ] threadhttps://anonyfox.com/cv
I'd wager a SPA is a hammer/nail reaction here.
Thankfully, the depth of mismanagement here was pretty exceptional in my experience. In most cases, even if I'm personally skeptical of the need for an SPA I can see a reasonable argument in favor. The cartoonish straight out admitting that the sole purpose was to be more "modern" with literally zero regard for actually serving any user needs is thankfully not something I've run across too often.
If you break down what most vendors and consultants do, in terms of either dollars or hours, you'll usually find that they mostly do sales, PR, client relationship, etc. Projects are designed and managed to make the right people look good to the right people, manage expectations, etc.
They'll spend massive time giving non-tech executive buyers easy, fun choices.. let them feel strategic, visionary and empowered. They'll communicate the hell out of every piece of progress. Whoever hired them also has a massive incentive to buy into and sell onwards, that the project is a massive success.
IE... if your job is to make a clients' Salesforce instance do a thing, 80% you your job is stuff other than making a clients' Salesforce instance do the thing.
Internal dev teams will rarely have this pageantry. Meanwhile, no one outside that team knows what they do, how well they do it or such.
They spend 2x more on selling than they do on R&D.
So when that Oracle Sales Guy is giving you the pretty talk, and taking you to lunch remember that all that money your sending to Oracle is literally mostly going to him! The product makers just get 'a cut'.
Now consider how much of high tech is just that, sales and marketing, conferences and luncheons.
That's a staggering waste for a developed economy.
We are very inefficient with these things, we need better approach to IT.
I will never, ever, ever, do in-house software development ever again. Non-software companies treat their inhouse developers like crap. They don't get it. I don't know when non-software businesses thought it was a good idea to move software developers in house. Most of the company don't want them there. Or deal with them begrudgingly. It's hard to quantify their revenue so they are treated as an expense not cost saving or revenue generating. And they are the first on the chopping block when things turn down.
In my experience, if you are recruited by a company that doesn't sell software or software services you should tread very carefully, ask lots of questions. Sometimes you'll find out a 100 year old furniture company only has a dev team because the grandson thought it was a good idea. And then you run very far away.
Edit: I’m confused then: if robots have software therefore it’s a software company, then tons of companies are software companies. But so many of those products, the software is just a hidden detail that no customer ever thinks about or cares about. Where do you draw the line?
(There is an interesting discussion to be had about where the boundary between "software" and "hardware" companies is, but an environment with a high level of wilful misreading isn't one of them)
Appliances are an obvious example. But also things like cameras to a large degree. Or auto infotainment systems until Apple and Google software was integrated. (Embedded is sort of a different category that is functionally essentially part of the hardware.)
The software output at such a company is confused at best. Usually with some exec pitching digital transformation without definition. No one wants to deal with the tech team, pay and expectations suck, getting hardware can be a years long process, and eventually someone pulls the plug.
By that definition google is not a tech company, their product is selling advertising space. Tech backend just enables that.
Or Netflix. Certainly not selling tech or software, they rent movies.
For most companies building the physical product is a loss leader, what they’re selling is their custom software.
- companies for which software is the sold product (or its support, hosting...)
- companies selling a service or a product that requires inhouse made software
It could be argued that this second one could be split in two as the software may not be delivered directly to the client but the border is a bit blurry when it is a website/app platform like netflix.
So generally, IT vs product development.
I have been on IT projects (internal development) tied to revenue. It was pretty sweet while it lasted.
The problem is most consultants are not good at what they do. It why the sales guys for the consulting company are not the devs. Most the time consultants come in for 3-6-12 months, cause some trouble and then just leave the mess to the company to address later. There are some good consultants but most are just people that couldn't work in house because they are not up to par for long term engagements of products that need to stand the test of time to add legitimate value.
> It's hard to quantify their revenue so they are treated as an expense not cost saving or revenue generating.
This depends on the business, most the time, unless you make software you are an expense. The company makes their money in different departments and making money is the name of the game. It's takes a different set of skills (albeit better/stronger, industry knowledge and communication skills) to be an in-house developer vs a contractor.
The challenge here is that under this model the different departments get credit for the revenue they generate. What their relationship is with the in-house dev team is very much something that one should be aware of if they choose to be in that situation. It's not too dissimilar a situation to every single 'change management' model that has executive commitment as the number one success factor.
/r/oddlyspecific :)
Me: So what is it you do here. The job description says you are looking for a developer?
Them: Oh, we make and sell boxes.
Me: Like the cardboard boxes you get delivered?
Them: Yes, exactly!
Me: Okay, so where does software fit in that? Are you aware of development cycles? {Other interview questions I can't remember}
Them: Oh, we don't know the technical part too much. We want to automate some of it.
Me: ...
Software developers at software companies/tech companies generate profits, so they are treated as such.
Software developers at more "traditional" businesses (retail, industry, manufacturing, etc) are often seen as an expense and something that should be treated like any other expense - try to minimize it.
> The amount of customization by far exceeded the initial offering of the services, and had to be done within the constraints of each individual technology stack. > > This accumulated in hundreds of partially overlapping systems, wired together with brittle ad-hoc integrations.
They had a software solution and were officially spending zero dollars on software development. You can't beat that. Never mind the huge amounts of time they spent keeping it working. All of that money showed up in other places on the bottom line. Maybe a lot of it was IT, but a lot of the expense was probably people whose job title had nothing to do with technology spending hours doing repetitive and error-prone integration work with third-party systems, emailing around spreadsheets and CSV files, manually tracking down and fixing bad data, and answering the phone for confused customers. As soon as you start employing software developers to create a more efficient system, it looks like you're spending 100x more for software, on a trajectory to take a very, very long time to achieve parity with the existing system.
I do think this is valid, but feel I should add: that willingness needs to go both ways. You were willing to put in the extra work, and your employer was willing to "get it".
There exist organizations that will never "get it" despite the extra work, and these are the places others are talking about when they say "run away".
I experienced both variations of this earlier in my career. I was successful the first time, and tried to apply that success in a larger environment.
The lack of organizational willingness to "get it" in that second environment is what motivated me to move to a place where building software is the primary reason for existence. I do not intend to look back.
Managerial debt can happen anywhere, including tech firms. It's where you're dealing with people who don't understand what recent technology can do. Maybe they got it 10-50 years ago. Maybe they were very good at it 10-50 years ago. But today they're behind the curve and unwilling or unable to understand that a working modern solution is possible.
They may also be risk averse for perfectly rational reasons, in which case you need to sell damage management security promises and fall backs with any upgrades.
The bonus issue is a very real problem. If there's pushback from that level it's often because those people can see dollars moving from them to the newcomers. And they'll do almost anything they can to prevent that.
These are very difficult issues to deal with, because they're usually hidden behind spurious excuses and rationales and passive blocking actions.
1. I worked with a single internal guy to build a web-based ordering system, in the days where people faxed in orders. This replaced weeks of data entry, dozens of part time data entry people, and enabled a big change in materials ordering and production (orders were all in a database within a few hours, vs literally weeks of keyed-in faxes from before).
The one guy that was in charge of all this suddenly had a lot more power internally, and it pissed off a lot of other folks who suddenly had less influence.
2. More recently, worked with a team of external contractors, and internal folks were resistant to help any of us set up external dev systems. They all worked in the physical offices, and had no need for 'run a dev system on a laptop'. One of our guys spent a lot of time (and kept getting chastised by the client's mgt) for getting and documenting a build process for external contractors to use. The pandemic hits, office buildings are shut down, and internal folks have a much harder time figuring out... how to do anything from home.
In both cases, under-the-radar skunkworks projects ended up having large (and in one case transformative) impact and value, but were ignored or fought by those with entrenched power.
For decades now, per Alistair Cockburn, I assumed people just really, really hate change.
There's some research which shows our selfish little brains have an immunological type reaction to change. Anything that threatens our internal stable equilibrium is squashed like an invader. No matter how much we say we want to change. No matter how much we may need to change.
While wholly correct, turns out Cockburn was an optimist.
We now know that some fraction of people would rather watch the whole world burn than give up one iota of power.
This is why.
The reason companies like this use consulting companies is because the VPs want total control. Internal IT is purely a cost center, and unless your company (read: the business) is already on board with the product lifecycle op model, there is no point in trying to do any of this from within IT. Consultants work at the VP level and in a large enough company, we find a technical area (usually data as it has the biggest business returns) to modernize, then leverage success there build out the business case for the rest of the company (data is a good place to start building API gateways). And selfishly, the business is willing to pay higher hourly rates than internal IT (which we generally leave to the offshore MSPs).
When I sell transformational solutions to a company, I know it won’t be successful without top-down buy-in. Consultants are also better positioned to do these transformations because we do them all the time; and the exercises around stakeholder management and consensus building are extremely important. They’re the kinds of things internal teams are never resourced or trained for, and they don’t happen quickly.
Plus thanks to those political games there are lots of soft skills to learn as well.
I question the wisdom of somebody who started refactoring and replacing systems right after massive layoffs in an industry that was crippled by the pandemic.
Businesses in survival mode can't spend resources like this. The author points out the executives became enraged when the found this out and I agree with the executives.
Well wisdom is gained from experience and it seems the author learned a few lessons.
You agree with their 'enraged' state? You think it was justified? Even with hindsight, or just at that moment in time?
> refactoring the old things we produced into the new technology stack.
You don't see how that could fix things?
As a rule, when things are broken it's the best time to go out fixing them. But yeah, this has a non-zero cost that maybe can destroy the company.
Do not encroach upon my power, you cretin! It doesn’t matter if the business dies because of my stupidity, as long as I go down captain of this ship!
Edit: It's a legitimate solution. I've survived 3 reorgs at my company because I could code. I read up on the new incoming system and became effective.
As annoying as that is, I kind of feel you should be above that. You have all right to be annoyed and skeptical, but not to sabotage the effort.
If I had to guess Rust vs Java/C/Perl?
But he then outlines all his failings at the bottom, which I agree with, so he clearly knows better. It's it possible that this tone deaf attitude was a primary culprit in the downfall of the organization he and his handpicked successor built?
This is one reason technologists can make bad managers. They don't think about the politics or don't want to play it.
That might be true or not, but you can't operate with that mindset and expect people to come along with you. Whether or not it's true, you need to presume positive intent. Perhaps they see aspects of their jobs that won't automate well that you do not. Is that stubborn self interest, or standing up to a naive new leader who doesn't get it?
Every family member who ever turned into a whiny baby while trying to do free IT for them saw themselves as the victim, or the person under threat. That's the kind of person with power in most organizations, and you have to find some way to meet them in the middle without compromising everything to get anything done. It's not enough to be right.
I doubt whether this extends to the politicians / executives enacting those policies. They will certainly frame it like that, but for a lot of them, their underlying thinking is going to be much more self-serving and about manipulating the narrative.
> The only system not affected by the complete meltdown was: our new one, which was designed to withstand outages from backend systems below.
I don't understand, how can their SPA frontend still be functional when the backend is totally down? You can display things to the user, but you cannot do anything meaningful from it?
Yet still somehow customers buy the products and (more often than not) the company delivers a product that is half working enough to be tolerated and used.
This is all very vague, so who knows.
But at the very least, I would guess the OP's attitude of I-know-better-than-everything-that-came-before arrogance plus ignorance of why things were the way they were made big failures inevitable.
It's a real limitation -- though a common one -- to need everyone to see things from your perspective, rather than being able to understand their perspective.
Generally its great, but not a panacea, and while it would have been improvements it would have suffered fun new problems.
Source: been the young guy who thought if only they listened, and then they did.
33 districts in Uttar Pradesh are now Covid-free: State govt
Overall, the state has a total of 199 active cases, while the positivity rate came down to less than 0.01 per cent.
In an area with a statistically zero vaccination rate.
https://www.hindustantimes.com/cities/lucknow-news/33-distri...
I await your heartfelt apology due to new facts. /s
"When there just is no interest in actually improving things, I should leave it at that. "
Yep, if you can't convince them to care about and drive the improvement themselves, you're gonna have a hard time if driving it becomes the most important thing to you.
This is just how company politics works. Those executives wouldn't be in their positions if they didn't look out for themselves first. Also it is turtles most of the way down. This experience should be a good lesson.
Only the larger of non-tech companies can get away long term with internally built tech. This is because only they have the money to build and maintain teams and build and maintain the technical expertise.
Sometimes, it still takes a massive bite in the ass for these giants to wisen up. For example, see Home Depot and their credit card scandal and then their subsequent investment in self checkout, online sales, and lockers. They have done really well for themselves.
Smaller companies eventually either get swallowed up or destroyed by bigger companies, by disruptive new players, or get a new lease on life due to the creation of a service layer below them.
Nowadays those new disruptive players are often tech plays. Also the new service layer is often tech. One example of a tech service layer is Shopify. AirBnB and Uber are examples of industries getting swallowed up by tech. Expedia is another example of disruption from the previous generation. Home Depot swallowing up more of the construction supply industry is an example of bigger players crushing smaller players. Bye OSH.
New things go with old things. The people who maintain a product ideally should own the improvement or replacement of that product. That's how you get everyone on board. And if you can't manage to do that, then you'll probably end up in some kind war.
“Nobody gives a hoot about profit”.
So you’re a developer AND you’re nominating executives? Something about this story or the author stinks.
How could you possibly know that they were "afraid"? More likely, a group of people just had an abrupt re-org with little-to-no input and now had a new poorly-defined internal "digital" group that failed to provide clear direction on how it was going to replace an existing 3rd-party process. If worked suddenly ground to a halt and the new department was the only thing that changed, maybe existing leadership wasn't the problem?
> Without any real support from any other executive, our poor VP was tasked the impossible: digitalize the enterprise... but without any decision making power outside of his own department.
What about the other poor VP's? How did this new digital VP engage with them, individually and as a leadership team, to ensure that this new department had a well-structured process by which it provided value to the company? Did this new department try to lead the agenda, or simply steer it? You should be aiming to provide a service, not expect service from others.
> Executives went full enrage when they heard about what we did there
Well, yeah? This reinforces my opinion that the new department completely failed to engage the company. I'm sure the executive leadership team had more important things it wanted accomplished. How did adding Phoenix LiveView to an existing mess of a stack add value to the company in a way that aligns with existing objectives? Was the digital VP even consulted on this decision?
It's also possible leadership at the top, a CEO/founder, didn't know how to manage an executive leadership team and the dysfunction went too deep for a VP to fix. If the leadership team isn't meeting regularly and on the same page and collaborating with the CEO to determine a roadmap, then something is very wrong. You can't fix bad communication between leadership stakeholders that can't decide amongst themselves what priorities are.
> they wouldn't even come up with such ideas on thir own
You're telling me that a group of people that had been successfully making demands of 3rd-party contractors and companies couldn't do the same of an internal team? I don't buy it. How did they get to such a tangled mess in the first place?
> there are sometimes no other options than doing it as we did
Maybe so, but how did you bring along other stakeholders in this decision-making process? I wouldn't tolerate this attitude from a contractor/freelancer, or a full-time employee. A big part of leadership is convincing others of the value of a certain path, and failure to do that isn't the fault of the other party.
> Without toplevel commitment, best efforts and even results don't matter.
Yes. In the vacuum of clear leadership direction, other people will find opportunity to fill the empty void with work. But separate leaders working in isolation, even creating one-hit-wonders, will never be successful as a team collaborating on even a mediocre agenda.
> When there just is no interest in actually improving things, I should leave it at that.
I don't believe this for a second. Either there was complete absence of top-level leadership, or the new department wasn't structured to service the business and failed to build the confidence of other stakeholders that it could deliver value. I see a lot of blaming the other guy, and very little other actual introspection other than "without including all people affected by this".