Companies attempting to pay an engineer according to location, in my opinion, is another kind of discrimination. You are supposed to pay an employee based on his/her abilities, not her location.
We have rules in govts that companies should not discriminate against employees based on sex, religion, sexual orientation etc etc.. But it is fair to discriminate the salary of an employee based on location? For ex: I know a few friends who have moved from Europe to Asia with the same role and are getting paid less compared to what they were getting paid in Europe. Its the same role, its the same person, but getting paid less just because of location ?
Discrimination is around things that an individual can't choose (religion being the weird elephant in the room). Fair or not, this isn't discrimination.
Many people can’t choose where they live either. Getting a visa to the US is a ludicrous process and even if they wanted to they maybe tied by family obligations.
But people don't choose to be rich or poor exactly either. As a general rule, discrimination is around things for which there's no choice. Having a choice over where they live or whether they are rich doesn't mean it's easy or practical. But that can't make it grounds for discrimination, even if unfair.
In the UK, the Equality Act (2010) protects: age, disability, gender reassignment, marriage or civil partnership (in employment only), pregnancy and maternity, race, religion or belief, sex, sexual orientation.
Pregnancy, maternity, marriage, civil partnerships and gender re-assignment are usually chosen by individuals, not forced upon them.
I disagree. I believe a company should pay based on ability of employees to have comfort and wellness, not some universal measure of value (which I believe to be impossible). What you are advocating for inadvertently breaks community and exacerbates gentrification and destruction of social fabric via inequality. Location matters.
Yeah I absolutely hate that. I get why they do it from a business point of view but as an engineer, I'm instantly put off when I see something along the lines of "up to $x (depending on location)".
> You are supposed to pay an employee based on his/her abilities, not her location.
I don't believe that is a legal requirement, anywhere. Remuneration is based on many factors, which can include the cost of living. A company will not be able to hire someone in New York City, for the same price as someone in a less expensive jurisdiction.
This isn't discrimination, it's simple economic reality.
> Companies attempting to pay an engineer according to location, in my opinion, is another kind of discrimination. You are supposed to pay an employee based on his/her abilities, not her location.
So you’re saying, we should be paying engineers in Europe and in the US the same as an engineer in LATAM or India or Asia that has the same level of experience and skill.
The only way to be non discriminatory is to have a standardized formula of compensation that takes into account cost of living (rent, food, healthcare, taxes etc) where the final take home pay in locations around the world are equivalent - which I believe should be the case at most companies
A worker in country X or country Y are very different for companies' balance sheets. For instance, my company is Canadian, and we are eligible for significant tax credits through SR&ED[0] for software developers. If a software developer moved their permanent residence to outside Canada, even if we could magically pin exchange rates to pay them the CAD equivalent in local currency, it would be a significantly different financial impact on the company as they aren't eligible for that program. I'm not an expert, but I imagine there are many equivalent programs in every country, state, and even municipality.
It works both ways, anyway. If those friends had moved from Thailand to Switzerland, would it be discrimination to pay them more?
I agree with this in theory but I can't see how it will work in practice. There isn't a global "value of ability" to base the pay on. It's valued differently in every location.
Why are you supposed to pay based on abilities? Where is that stated?
As a company you need certain abilities, and you pay whatever the market decides these abilities are worth, and nothing more. Depending on location, the market will set a different price on these abilities, so you pay different.
> You are supposed to pay an employee based on his/her abilities, not her location.
You are supposed to pay them the minimum amount it takes to get them to show up to work. When someone moves to a less competitive market, where getting another job is harder, then they are more likely to show up for lesser pay.
And remember that a country may have a less competitive market, even if the workers are remote and not seemly bound by a local market, because governments often love to put up huge roadblocks when it comes to international hiring. If you are being paid less than you were in another country doing the same job for the same employer, this is almost certainly why you have agreed to take a pay cut.
That’s up to them. It’s the worker who chooses how much it takes to show up. I suppose if they want to truly play the part of being female they may want to accept less.
You could also argue that difference in pay is less discriminatory. You are paying employees to have the same quality of life, same type of housing, same opportunity to provide for family, send your kids to the same type of schooling. These things cost differently in different countries, so require different income.
Exactly this! Location-based pay is not so much about cost of living as it is about buying power. In the end, money is just a place holder for real value which comes in the form of goods and services. And the real value of the same amount of dollars wildly differs per location. So if you want to pay fairly and not discriminate, you have to try and make sure people can roughly buy the same things in their differing locations for the money you give them.
While I'm sure it's very kind of companies to care about my quality of life and the type of my housing, it's honestly none of their business. Even if they tell me I'm "family."
Fair pay to me, at least, means paying for results. Not paying for hours spent toiling. Not paying for where I am on the planet. Not paying for how I get the results, just for the results.
Instead, there are all of these gamey factors inserted into the mix. They're emotional. They're manipulative. Yuck!
That’s not how free markets/capitalism is supposed to work. Companies are expected to just shop around for the lowest price on the capabilities they need. Are you suggesting we should adopt something else then capitalism/free markets?
> You are supposed to pay an employee based on his/her abilities, not her location.
I don't think this make any sense on so many levels. First, "abilities" are not a good way to think about wages. If you hire a neurosurgeon to do your gardening, you won't pay them more than a run of the mill gardener.
Rather, you as the employer compete against other employers on different markets in a fairly classical supply and demand situation. The "abilities" of an compliance expert with tech skills did not change much when GDPR was introduced, but as all EU companies scrambled to figure out the regulation (and the DPO role was popularised by fiat), the compensation went way up.
If the employee can participate in e.g the SF labour market, you have to pay a competitive salary in that market, if not you don't have to. As long as there are barriers, e.g a on-location worker in SF has more opportunities for whatever reason, the location premium makes sense.
To take your example in the opposite direction. Let's say a east-european company want to expand into the US and open up a sales engineering office in SF, and want their best sales engineers to go work their, it would be completely insane to not raise their wage. "We pay people after ability here you have 40k USD, have fun finding housing".
I used to feel this way, but largely grown out of it. In the end, you're asking for SF salaries, and those in India are asking for NL salaries(or SF salaries). You'd largely find a race to the bottom, I think.
The simple fact is you made a good living for your area, and they made a good living for theirs. If you want more, move. If you won't move, there's likely a reason why.
Everyone wants a SF salary with an Indiana or even India cost of living. For obvious reasons, that can't work.
>It doesn't matter whether you're paying somebody in the Bay Area $100 000 per year, or somebody in the Philippines, because the cost for you as a business is the same.
The cost for you as business is NOT the same. Start from the taxes part, that 100k is what the employee gets but the cost for the business is higher depending on the country/area because they also pay tax on top, social benefits etc. Also need to have in many cases a local business, doesn't matter if that is virtual etc, since they need to adhere to local laws thus having resources supporting that etc.
So while I'm on the employee side here as I'm also working on a multinational coorp with global role yet paid with local standards, there is more than meets the eye
yep. People vastly underestimate how difficult the logistics of "pay an employee for services" can be, particularly when you don't have a legal corporate presence in the country where the employee resides. There are services that handle this for you and they charge a 30-40% premium on top of the employee's salary. And sometimes this is still worth it, because many countries charge an absolutely obscene incorporation fee for foreign-owned businesses.
It’s not. If I have to pay some service or agency a lot of money to be able to employ you, there’s less money left to pay you. Employers at distributed companies don’t look at your take-home salary, they look at the total cost to employ you. Fees and taxes differ wildly per country, there’s no other way to compare. So if a large % of that total cost goes to middle men, then that makes you a more costly employee at no benefit to either you or the company.
If I’m considering two people for a job but one is in a place where paying them well means I spend a huge amount on fees (or “employer-side taxes” for that matter, looking at you Austria), I might well choose the other.
Yep, and they choose not to have it unless there's some compelling reason to. We've got staff in many countries, but it's not a blanket "work from wherever you like".
Fwiw companies like Remote and Deel charge a fixed fee of substantially less than 30-40%. IIRC we pay about 600 euros per employee per month to Remote. That’s a lot of money that I’d rather give to the employees themselves, but it’s much much less than 30-40%.
so why not set a “total remuneration package” as it’s known where I live. It’s the total value, inclusive of compulsory payroll deductions and taxes, and set the salary to match that. The cost to the company is the same, but your take-home depends on where you live.
in many areas salaries must be specified as the take-home part (including the taxes you pay as employee, but not including the part that the employers pay, which is not part of your remuneration), because doing otherwise would be confusing and could be considered deceptive.
From the other angle: There exist companies that do not do location based pay. If you're hired, you get a US level salary, whether you're in San Francisco or Poland or Indonesia.
These positions are phenomenally competitive. You have swung open the doors to the world, and said "Give me what you got!", and you reap the benefits by having a much, much larger pool of applications and talent to sift through, with a lot of truly exceptional people in there - mostly very intelligent, very driven people from poorer countries.
That's how supply and demand works. If you are offer to pay people more, you usually end up hiring a higher quality candidate.
You also know that companies are looking to optimize margin, so those US level salaries are only temporarily. Given enough good people in India, or other low-income countries, that pay level will drop significantly.
I've worked with teams from Bangalore who were staff of the bank I was contracting for -- they were amazing, but also not appreciably (if at all) cheaper than employing someone in London or New York.
Several well-known banks had large offices, and competition for talent was high. No race for the bottom there.
It doesn't particularly matter where you employ people, if you're trying to save costs by paying people less then you're going to have a bad time.
Good engineers will accept lower payment if their costs are lower. Similar to how amazon is operating, by lowering costs, minimizing margin, they can win over customers and win the market.
Conversely, if there is a shortage of known-to-be-qualified engineers then anyone who is any good will command a high salary regardless of their cost of living.
If you want to pay them lower wages, they'll work for someone else instead.
I recently visited some teams in India, they indicated quite a few coworkers moved back to India to be closer to family and friends. Not everyone is dreaming about moving to the US. And yes, I agree that I couldn’t see myself living in India, with all the pollution and overpopulation…
Sure, that's of course what we would expect. However - and I don't know about you - if my options were between getting paid $200 a month as a farmer with my dad, or earning $8000 a month as a software engineer in my room, I would probably still take the $8000 option, even if it only seemed like it would be around for 3 months.
I may even be grateful for the chance, instead of angry that it wasn't a deal that was going to last in perpetuity. I admit I may be in the minority here.
What makes you think US level salaries are temporary? Isn’t this the case for last 40 yrs? Why it will change now? Is it due to advance in remote work or more supply?
To be clear: we don’t only hire in the US. We have employees in at least the US, Canada, and Europe at the moment.
We do want some overlap in working hours with the US, so it is true that we cannot realistically hire anywhere just yet, but not being in the US is not a dealbreaker.
A consequence of that is that local companies, that have local economy level income, can't compete on salary with those foreign companies. So they can't get the top-tier workforce they used to have access to. Ever increasing the economic disparity between countries.
They allow brain drain to happen, without the barriers of having to move countries.
But the money stay in the country and increase the chance that the employee eventually starts their own business, possibly using the cheaper workforce as an advantage.
You might be right, if we assume that everyone who takes these high paying remote jobs also makes sure to never ever spend the money they earn locally, either.
However, if I was earning an order of magnitude more money than I currently am, I might want to pay a little extra to go to the really good barber, or to eat at the really nice restaurant at the riverbank. Or, hell, I might just employ a cleaning service every week, to save myself a few hours' time vacuuming my apartment. These necessarily local services will also see their revenues rise. To me that seems to be a more important effect on the local economy at large.
But how would you feel about working as a barber, chef or cleaner, when you could earn two orders of magnitude more making Internet thingamajings for people on the other side of the world?
New York never has a critical shortage of barbers, chefs or cleaners even though for many decades it’s been possible to earn 100x as a Wall Street bond trader or quant.
A healthy growth economy can tolerate income differences. But the balance is certainly precarious, as the example of New York or London shows. It’s constantly on the edge of driving out the remaining barbers and chefs because they can’t afford rents.
I'm pretty sure someone making 2-4x their local salary for a remote company and paying taxes is healthier for the economy than working your ass off (or not) for a local startup that wants to end up getting acquired OR doing the same remote work with 2-3 layers of management extracting the difference in pay. At least in Poland I can't think of any single company I'd want to work for.
There is also the factor of the country receiving hard foreign currency, which I understand is generally quite desireable. This is less relevant for EU vs US compensation, but for more developing ("3rd world") nations could be significant.
If they don't have to move countries, it's not really brain drain at all. It's exactly the opposite in fact.
If they did have to move, then they would, and you'd have brain drain. But because they can remain in their communities (while earning the globally-competitive income that they would otherwise have to move for), they now pay taxes to their local government, buy from local businesses, mentor local youth, and so on. When they've earned enough money from their job, they may quit and start a startup in their own community, or become an angel investor supporting startups in their area, rather than yet another bay-area based fund. These are all good things!
Now see that's an interesting problem space to be in. Keen eye.
To me it seems like a really exciting place to apply recent innovations in LLMs. If LLMs can chew through thousand page legal binders like toilet paper, there's no reason they can't chew through a thousand 1-page resumes and spit out "These are the ten most promising ones based off of our statistical analysis." Firms can specialize in the production, hosting and fine tuning of these LLMs, and even play both sides of the market by allowing candidates to see how good their resume looks for a given job description.
I think this is much likely to become a lot more common in the latter half of the 2020s.
Whether or not it is pleasant for the applicants, it'll happen if it provides a benefit for the companies. They don't care about the applicant experience because they have no incentive to.
This is the dystopian future that is likely already happening. Slowly but surely we will lose control over our own labor. As the name implies, we're just human resources.
The flipside of this is that there will be an asymmetric advantage available to firms that are capable of finding excellent candidates that fall into the ML blind spots.
Yeah those companies don't exist because they would be wasting money. Salary isn't "location based", it's "competitive salary based". It just happens that competitive salaries strongly depend on location.
If you were to forget about location and just say "we'll negotiate all salaries" then you would end with exactly the same result because people in NL are willing to work for much lower salaries than people in SF.
I don't get why so many smart programmers don't understand this basic fact of economics. Eh maybe they do understand it and are just jealous of insane SF salaries (I certainly am!).
I would be wary of demanding equal pay by location anyway because you'll end up with all jobs moving to India.
Those companies do exist. I can confirm specifically that at least when I had an offer from Supabase they paid everyone, internationally, regardless of location, the same pay bands. Being USA based it was one of the reasons that I turned down the offer because I was able to get a much higher salary elsewhere but it would have been extremely competitive had I taken the role and moved to some place like Vietnam.
I wish I could have taken the Supabase role because it was definitely my top pick otherwise. One look at the output and caliber of people they hire also indicates that they have little issue finding talent.
FWIW this was a couple of years ago and I have no idea whether they are still doing this equal pay band thing or not. But they were doing it for awhile at least
The fact is that location is irrelevant for some roles. If you are looking for top talent, you'll pay top talent value. If you constrain your hiring to a single location, you're simply reducing your own pool of candidates.
Today, most labor arrangements are more and more like companies. If you were to select top companies to contract for some job that doesn't really care about location, you wouldn't be choosing companies based on that. You'd choose based on how good they are.
It's the difference between trying to buy the cheapest versus buying the best. Of course if you're always looking for the cheapest, you'll always move towards overseas jobs. But if you're looking for the best, you can get the best from all over the world by offering a single solid compensation package.
It's all a transaction, isn't it? At the end of the day my labor is worth however much I can get for it.
Sure, you could squeeze even more profit by paying overseas workers less, but then you create all sorts of imbalances that can and will hurt your business in the long run.
I always joke that if you want to hire me (I am not from the US) and pay 50-60% less just because I live here, why wouldn't I work 50-60% less?
You're getting the 1% of a lower income country, for an average local developer salary. If you want the 1% of SF you'll have to pay a lot more, even if they are equivalent in the value they provide. The company still wins, and as a result you get happier employees.
You can always cheap out, but it's never without consequences.
> Yeah those companies don't exist because they would be wasting money.
Basecamp has been around for 20+ years and they publicly mention that they hire based on SF rates, not even SF but the top 10% of SF[0] for positions around the world.
When you have only 30 employees, and your people at the top are buying multiple supercars and planes, and your revenue is somewhere in the ball park of $5M+ per employee... you sure as hell better be paying salaries commensurate with that, or people are going to be bitter.
Indeed. I don't understand why a remote company would want to pay top dollar for a mediocre developer living in the US, while refusing to pay the same for an exceptional developer living somewhere else.
Because of marginal returns on applicant motivation. A $200,000 position sounds great to a person expecting $150,000. $200,000 also sounds amazing to someone expecting $40,000. But, the same person expecting $40,000 will also be amazed by a $100,000 position, and certainly not half as amazed as the $200,000 one.
This may work if you are only hiring senior people, but imagine hiring some mid level or junior employee. if US junior salary is Poland senior salary, would you willing to hire the senior dev from Poland to junior position ?
The issue I see is that software engineering is a team sport. Having a bunch of intelligent driven people doesn't mean they will together act like an intelligent driven group. Work cultures differ greatly between countries including in some subtle unconscious ways. Even Western Europe versus the USA have a very different dynamic in terms of how ICs and managers interact with one another.
But when Indians, Germans, or Canadians move to the Bay Area for better pay, they don't immediately (or necessarily ever) become culturally Californian. They might put their complaints aside for the paycheque, but they'd probably have equally-happily done the same as a remote worker. Especially when their whole organization is remote anyway, so there's essentially no subtle cultural difference between a Canadian working from Canada, and a Canadian who just moved to Mountainview while zooming from their spare bedroom.
Furthermore, if the difference in compensation is really explained by differences in employee productivity caused by work culture barriers, then location-based pay must somehow maps to the productivity cost of that work-cultural barrier. There's no reason to think that this should map directly to cost of living, and it would change over time; if the workforce composition shifts towards Europeans for example, then now it's your workers in the bay area who are less productive due to missing subtle cultural cues from their German managers.
> you reap the benefits by having a much, much larger pool of applications and talent to sift through
Interviewing/hiring is incredibly noisy though. If SF engineers have a much higher average skill than the rest of the world then you might still end up with better people if you just hire from SF rather than the world in general, even if the latter has a much wider pool with more top people in absolute numbers.
A smaller company can do it. The post links to a post by Bryan at Oxide Computer. The salary scheme for the generally senior people they hire is quite egalitarian. It's also pretty modest by senior-level Bay Area (and even many other locations) standards.
Probably. But some percentage of people who buy into what the company is doing are willing to take lower pay to work there because the pay is "enough" or whatever.
in general i agree, but the problem with location based salaries is that they are working for locals, but not for expats. expats everywhere have higher living costs than locals, in part because they will get more expensive housing that is more up to the standard they are used to, and they also may have kids that they can't or don't want to send to a local school, and they will buy more imported food that they are used from home.
when i looked at their salariy calculator i figured that i could not live on the salary they would offer for my location. school costs alone for each child are as much as i pay for rent.
it's a problem because it means that many expats can't work for gitlab because they can't afford to live on that salary.
if i am working for gitlab in my home country, i can't move into the home country of my wife because my salary would be reduced below what we need to live there
But there’s no benefit to Gitlab or to society in general for Gitlab to subsidize people to move to countries where they can be rich compared to the locals. It’s a personal obstacle to working at Gitlab, not a problem with the setup.
strongly disagree. there is a big benefit to have cultures mix and help with integration into an international company, both to gitlab and to society. that's why i moved to china.
i am not demanding to maintain a lifestyle from elsewhere or even from my home. i just expect the acknowledgement that as an expat i simply can not avoid certain extra expenses that locals do not have. with regards to kids, there isn't even another option. children that did not grow up speaking chinese at home have no realistic chance to successfully pass through chinese schools.
why should i not be able to demand that?
this hinges in whether we want people to mix and integrate. if we do want that, then we do need to support people who make that choice. if we don't want to support that, well then we just won't get people who are willing to make that move.
so if gitlab is not paying a salary that allows me to do that, then gitlab is missing out on the benefits of the mix and integration that i am achieving. it's quite simple.
it's fine if gitlab (or any other company for that matter) does not want that. that's their choice. but they also have to accept that the consequence for the company is to not get those benefits. you get what you pay for.
If I work for you, and you're willing to double my salary to subsidize my moving to an area that has wildly more opportunities for me (which is precisely why the salary is higher there), don't mind if I do move there.
You just need to ask yourself if offering incentives to get your employees to move to hot job markets is in your benefit.
> you're asking for SF salaries, and those in India are asking for NL salaries(or SF salaries)
No. You got it all wrong.
You're asking for equal pay for equal work. If all your team members are paid in full but you, in spite of doing the exact same work, are paid a fraction of what they are paid, then something is terribly off.
Your contribution to your team does not depend on where you're currently located. How much time you waste on commute does not change the expectation placed on your output. If your office location is prohibitively expensive that means your company needs to sort their mess and work on a location that's more affordable. It makes no sense that you need to subsidize your employer's bad office location.
Why does it matter? Salaries should depend on the value you deliver. I don't think companies limit their profits depending on the location. Apple devices are often more expensive outside the US.
While I can empathize with this, this is a moral judgement. Price is not determined by fairness, but by offer and demand (and other factors). Otherwise teachers might earn more than football players. I think one could argue they should.
I agree. It's just supply and demand. I just disagree with people saying salaries are somehow pegged to living standards. No it isn't. Companies don't care about your living standards. They pay what the market commands thats all.
Whilst we all want SF wages, those wages are because housing and cost of living in SF are exceptionally high compared to say Bali.
So you want to earn 200,000 USD. Whilst living in a part of the world where that effectively places you in the top 1% of earners.
You're only looking at this from that angle. What about the person who's living in SF, and scrapes by paycheck to paycheck. Whilst you live in a much cheaper area to live in, and you have a very different financial situation. Your both paid the same wage.. because "fairness".
You might say, "JUST MOVE". But do you think that's a fair thing to tell someone?
Sure people move to places like SF, but people do that for more than money, they do it for a variety of reasons. And one of them is that there's a lot of demand in that area for your talents. So if you wanted to change jobs, grow your career, etc.. you can. But that area has a higher cost of living.
You sacrifice some of those things when you move to the small country town, there's no tech hub. No meetups, nothing.
So do you think it's still fair to say to that person "Hey I know you're living hard... But at least you can spend that little money you have left on public transport getting to a meetup!". Whilst the person living in the middle of nowhere can afford first class.
I get what you are trying to say. I'm one of those people who makes that kind of money in a cheap country. I'm just saying its fair from a value delivered perspective. Paying you less doesn't mean the company donates the money saved to a charity. It just goes to the companies balance sheet. I've had plenty of arguments with company exec's about this:
1. its not fair you get to live like a king! -> would you move here to live like "a king"? Oh you don't want to deal with the pollution and the bureaucracy and lack of safety. Ok. Ah so there is a cost I am paying by living in a bad country.
2. paying you SF salaries would be unfair to people living around you -> ok so you would be ok paying that extra amount directly to a charity right? Oh ok you aren't.
I'm saying it's only fair for a employee to think they should be paid proportional to the value delivered. Companies exist to make more money. This causes a clash. In my last company the CEO specifically wouldn't hire staff engineers from cheap countries because he didn't want to give the other engineers the idea that they too can command higher wages.
Just pay people the same amount and let them decide how to live their lives. They are adults.
> The problem with this line of thinking is that cost of living is the same everywhere. It is not.
I don't understand what point you think you're making. My disposable income is not my employer's business, and I definitely do not live below my means to subsidize my employer's business.
You wanted me to do my work in exchange for my salary. Pay me. Don't think for a minute you are entitled to go through my grocery bill to see if you impose pay cuts.
Equal pay for equal work doesn't mean the same pay. It means that an employee from Bay Area, one from Netherlands and one from India are able to buy the same amount of goods on their local markets from their wage.
Remember that all of those local markets sell iPhones at a higher cost in USD than the Bay Area. Exactly which goods should they be able to buy the same amount of?
Ok. So the salary difference should be the difference between the monthly median rent: for a 1 bed apartment that’s $1500/month cost difference, double it for taxes and fuzzy math, salary increase of $36k/year. Maybe it should cover “the median rent”, whatever size unit that is, so $1500 in Amsterdam vs $4k in SF: +$60k. Or should it cover buying a 3 bedroom house, though? That’s more like an extra $8k/month for SF, so salary difference is +$190k.
This kind of detail is where it falls down. Should people be able to purchase the same goods, or the same relative local position in society?
But money is portable, and Local market means a different thing than it did pre-pandemic.
The argument that local cost of living is the only factor falls apart when people are moving from high CoL markets to low CoL markets with their advantaged savings. Especially within a nation, or times of large movement (again, pandemic - where folks with higher CoL wages were better positioned to acquire prime real estate in lower CoL markets).
I mean, you aren't payed in buckets of rice or whatever. Or even a inflation adjusted basket of good and services.
You are paid in dollars (or equivalent). So they don't get to claim to be paying the basket of goods if they won't automatically match inflation either
> Equal pay for equal work doesn't mean the same pay.
Yes it does. You want me to deliver an output, not to slack off if I live X or Z. If you value my work, you pay me. You don't get to claim a benefit just because I chose to live somewhere cheap. Otherwise it would make sense to pay me millions if I chose to move to a penthouse in Manhattan.
Same here. The reality is that what company chooses to pay is the minimal amount they can get a candidate to accept for that location.
If the candidate had other better offers, then they can either reject the offer or propose a higher counter-offer. If the company received a counter-offer and chose to accept it, then this amount becomes the new minimal amount.
Over a period of time, if there are enough counter-offers (or rejections), this continues to increase. Since, the two (or however many) locations don't necessarily have the same demand or supply, it's inevitable that some location will end up with much higher compensation than others. It's the nature of free market and why some companies engage/d in hiring collusion (see https://en.wikipedia.org/wiki/High-Tech_Employee_Antitrust_L...)
This suggests that cost of living in SF is the cause of high salaries but in fact the only reason SF/Bay Area has high cost of living is because of the high concentration of highly capital efficient businesses with strong skilled labor demand coupled with the complete unwillingness to build high rise density most places in the Valley area. I know quite a few people who make SF level salaries working remote in random states across the country. Of course it can work. And SF should not get too cocky. Detroit used to be the Motor City, Music City, and a cultural and technological force in the world, but then its core competencies got disrupted by cheaper, more efficient labor elsewhere.
Why is that? It isn't obvious to me. It seems like an excuse companies use to pay people less, especially if the company is based in a high-paying city.
There are two ways to evaluate salaries / compensation packages: 1. What is your price in the market? 2. How much money do you need to be able to focus on your work and not worry about money. Most companies use a mix of both approaches, depending on the role and the individual. For most employees, the second perspective makes for a better experience. As long as you get enough each month to make the money problem a non-issue, you get to focus on the work itself, which can be a very positive experience. Some people (or the same people at different phases of their lives) want to optimise for getting the highest price they can get in the market. That's a valid choice, but one that in many cases results in a suboptimal work experience.
By that logic, the only one who wins is the company. I would like to believe that most of us, employees, want what’s best for us, employees.
If company X was paying N for a SF engineer, and suddenly it finds out that it can pay N/2 for an engineer that’s as good as the SF one (but lives, let’s say, in Mexico)… well, jackpot for company X.
> Everyone wants a SF salary with an Indiana or even India cost of living. For obvious reasons, that can't work.
Why should someone living in Indiana make less money than someone living in NYC for a remote tech job?
Both are located in the US separated by a 2 hour flight. Both are in the same timezone too. For a remote company where you have employees spread around the US, there's no difference to anything here.
> Why should someone living in Indiana make less money than someone living in NYC for a remote tech job?
There's no moral reason to satisfy "why _should_", but let's be honest. A company with equal salaries regardless of geo is more likely to cut the salaries of people in expensive places to live than raise the salaries of people in less-expensive places. This is especially true in places where both salary and fundamental necessities like health coverage are included in compensation, like the US.
Such companies then lose the employees who prefer to live in expensive places, to companies who _will_ pay them more to live in more expensive places.
Whether accepting that result produces a more efficient staff for a company is the more indicative question IMO. Or asking why some places are so much more expensive to live in.
> Why should someone living in Indiana make less money than someone living in NYC for a remote tech job?
Because on average they'll accept less money. Someone in NYC is more likely to have other options with higher pay. If a business wants to hire them, they have to offer more. Simple as that.
Salaries are based on the market economy of supply and demand. Salaries for a specific role are different in one location over another because that is what the market is in each location.
If you want developer salaries to be the same in all locations for the same level of work for reasons of fairness and anti-discrimination, you should ask why salaries of developers are higher or lower than other roles. Developers get paid highly because of supply and demand. Why not advocate for all workers, regardless of role, be paid the same - this would also be fair and anti-discriminatory. It seems we want to benefit from the market economy on one side and then want fairness on the other side. The logical conclusion of this kind of reasoning is communism where all workers at a company are paid the same, regardless of role. We know how that worked out.
If all developers, regardless of location, were paid the same, companies would much rather hire all their developers in a single location. Why bother hiring in locations far away? Jobs would have never flowed out of high paying locations to lower paying locations.
The Bay Area has the highest salaries because Silicon Valley has had many years to develop and the vast majority of tech companies are based in the Bay Area. Many companies are started in the Bay Area because people who work together in one company often break off and start another company. Many people at these successful companies have become wealthy and this has driven up home prices. This has made the Bay Area one of the most expensive places to live. Other workers cannot afford to live in the Bay Area and so there is a shortage of labor. This drives up the cost of every thing, including restaurants, groceries, and personal services. If the Bay Area does not add substantial housing, it will continue to see companies move to other metros.
In the US, salaries are different based on metro and state and are based on the market economy. Companies move to offices to particular metros if they is a healthy supply of workers and supply is greater than demand so that it is more cost effective. This location competition is healthy.
The role of any government, whether it is metro, state or country, is to create thriving economies so people want to move to that location. This means investing in critical mass in particular industries so many companies in that industry want to locate there. It also means ensuring other costs are low. In the US, health care insurance cost between $24k and $36k/year for anyone with a family. This is higher than the full salary in other countries. If the US does not figure out how to solve its health care costs, it will continue to see jobs leave for lower cost locations.
Supply and demand, yes, and when jobs are tied to locations the demand is concentrated. So supply needs to move to where the demand is, or face a lack of demand and lower prices. If the demand is willing to disregard geography, their supply will be that much greater.
With fully-remote working, the demand isn't as concentrated, so supply need not be as concentrated either.
As someone not currently living in the Bay Area, or London, or another tech hub, I'm quite happy not to be paying the cost of living there. And honestly I don't think it's fair that people who live there should be better compensated just because they decide to live somewhere expensive. But I understand why it happens, because when companies hire specifically in a tech hub there are lots of people willing to work with them but only if their pay is higher. It's a vicious circle for employers, a virtuous circle for employees, and only sustainable for as long as productivity remains above cost. It's not built on a stable economic foundation.
I don't want to accept lower pay for the same job, which may mean that I don't work for those companies. That's the market at work :). On the other hand, if a company wants to employ folk to be physically present in London, they're not going to want to employ me. While if a company is willing to pay the same to everyone, they'll get fewer people in London and more people outside London, and they might even be the same people just dropping their commute :).
There's enough global demand for Software Engineering to drag everyone up. It is universally the case that if you pay peanuts you'll get monkeys, but paying an equal wage for equal work benefits the company and wider society and I don't particularly care if that's lower than I might get if I was willing to work in London (or the Bay Area), so long as I'm not required to work in London (or the Bay Area).
the issue comes when the pay does not match the living standards.
despite having the least in absolute terms, it is possible to have a great life with the mid-to-high level income in IT there. can't say the same with what most people earn in europe.
Ruby's entire programming model was based on the premise that language speed does not matter since most of the time you are waiting on IO / Network. Well now, both on Node.js and JVM we have programming models which say that - while I wait for IO / Network let me do some other work or service more requests (Async / Webflux / Coroutines).
IMHO - using Ruby/Rails in 2024 can be wasteful, but of-couse for the right situation it can be a good choice. (Just) For example an enterprise app where you know the number of users will be limited, or when you know the development speed is paramount or where you want to build a quick proptype to test the market out. Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
Ruby now has Fibers and other constructs for async like Ractor.
> Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
It isn’t any less tricky than a Django or Express project. With any codebase discipline and regular tending to the garden of code is important to prevent weeds from growing.
If anything that Ruby (and Rails) has going against it is still the raw language performance and higher memory requirements/usage than its counterparts, which makes it less desirable for workloads that require low latency or small memory footprint
Meh, it's fine for 99% of use cases. Rails only becomes a pain in my experience if you have random, huge swings in traffic or a ton of active users with websocket stuff (mostly solved by anycable). But very few products are going to reach those limits and if they do those problems can be solved too.
Stripe, GitHub, and Shopify are the shining examples of Rails scaling in all kinds of ways without problem.
I believe Github still use Rails for their main application. Spinning services out into microservices, even ones that are other languages, isn't really a sign that Rails isn't working for them, it's just what happens when companies scale. The company I work for is a Rails backend (React frontend) and we have some services split out into Go based microservices (eg. Twilio webhook handling has been offloaded to Go in Lambda), but it's still very much a Rails app.
> Even their frontend I think is now just react
It remains to be seen how successful this is and what the reasons were for the change. Personally I don't like it, it's less reliable and slower than before.
Not sure how long you were there nor which team you were on, but as someone who's still working on Shopify infra after a decade, I disagree with you.
Scaling problems at Shopify are rarely if ever directly Rails related.
The biggest challenge was always scaling the database layer, and to some extend the deploy pipeline of the monolith given the amount of people working on it.
Perhaps we can debate whether this is really a Rails-specific problem per se, but we definitely had lots of problems scaling to prevent long-running IO from saturating web/job worker pools for things like taxes, shipping rates, syncing external inventory, ad syndication, etc -- anything where we had to make multiple network calls to the outside world from a Rails thread was a big pain.
Stripe doesn't use Rails. Stripe's application of Ruby is far removed from your typical Rails app. A lot of custom stuff was built into Ruby to make the multi-million line codebase work as well as it does.
I guess it depends on how one defines "productivity:" is it spitting out code as fast as your keyboard works, or is is not having features do weirdo things because who can reasonably say what the code does at any given time?
I have more than once tried to contribute fixes to GitLab's codebase, and every time I open the thing in RubyMine it hurpdurps having no earthly idea where symbols come from or what completions are legal in any given context. I trust JetBrains analysis deeply, so if they can't deduce what's going on, then it must take an impressive amount of glucose to memorize every single surface area one needs to implement a feature. That or, hear me out, maybe "it works on my machine" is a close to correct as the language is going to get without explicit type hints as a pseudo static typing
Bugs exist in all code written in all programming languages and you will find bugs in programs written in statically typed languages too as you know. Programming languages are rarely chosen for the absense of bugs alone though or we'd all be using SPARK Ada or something.
> spitting out code as fast as your keyboard works, or is is not having features do weirdo things
This is a straw man. No-one has suggested that "spitting out code as fast as your keyboard works" is what Rails allows you to do, or that Ruby code results in features that "do weirdo things".
In reality engineer productivity is important and Rails enables it in a web environment.
> I have more than once tried to contribute fixes to GitLab's codebase, and every time I open the thing in RubyMine it hurpdurps having no earthly idea where symbols come from or what completions are legal in any given context.
Yes, I prefer writing statically typed languages and I would /greatly/ prefer Ruby to be statically typed for a number of reasons, but it will likely never be so in a way I consider to be usable (see https://github.com/ruby/rbs). Not being able to definitively tell what a method returns or where one is defined is a total PITA, but it's one of the array of up and downsides to Ruby, with each language having a different set.
> I trust JetBrains analysis deeply, so if they can't deduce what's going on, then it must take an impressive amount of glucose to memorize every single surface area one needs to implement a feature.
You don't need to know how all of Rails works to write a Rails app, as I'm sure you know, so this seems like another mis-representation of the truth, just as you don't need to know how the compiler or CPUs work to do a lot of (most?) programming.
> That or, hear me out, maybe "it works on my machine" is a close to correct as the language is going to get without explicit type hints as a pseudo static typing
You seem to be suggesting that Ruby on Rails applications behave unpredictably on a machine to machine basis but that's not really how Ruby works in practice, or matching on my experience.
> Ruby's entire programming model was based on the premise that language speed does not matter since most of the time you are waiting on IO / Network.
Couple of points. 1) You probably mean Ruby on Rails and not Ruby and 2) that's not true of either. Ruby or Rails "entire programming model" was not based on thoughts around I/O at all, a huge about of the driving force behind Rails was DHHs ideas about developer happiness, which is intrinsically linked with speed of development.
Discussions about I/O /do/ happen in the Rails community because it's important when you're running a web server which is a lot of the use Ruby/Rails is involved in, but it's not the sole (or even a major) focus of the decisions made for the language or framework.
> Well now, both on Node.js and JVM we have programming models which say that - while I wait for IO / Network let me do some other work or service more requests (Async / Webflux / Coroutines).
Right, but if the goal is to maximise CPU usage of your web servers then you can completely do that using Rails, and have historically been able to do so by spinning up multiple processes in the same way you'd spin up multiple threads in, say, the JVM. Not as convenient perhaps, but a model that's been in use since at least the 90s when I started out. Luckily it turns out that web requests can generally be run in isolation so IPC isn't usually an issue (the same reason why multiple physical web servers is simple, just as multiple processes is).
> IMHO - using Ruby/Rails in 2024 can be wasteful, but of-couse for the right situation it can be a good choice. (Just) For example an enterprise app where you know the number of users will be limited, or when you know the development speed is paramount or where you want to build a quick proptype to test the market out. Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
Wasteful as a generalisation is blunt. Of course it can be wasteful, but so can writing your app in rust, it just depends where the pressures are. I'd suggest that not launching is a far bigger concern to most people at an early stage, and even mid to late stages developer speed is one of the most difficult things to scale.
Personally I'm the CTO of a company that uses Rails (backend, React frontend) and developer speed is /hugely/ important to us. We're always constrained by engineering resources and the speed of Rails development continues to be a huge win for us. We do have scaling issues, but 99% of the time they're not Rails issues, they're in the database.
The article is nice and well articulated, but I'd argue exactly the opposite on this point. The author seems to confuse equality and equity. He takes the Netherlands and Bay Area as an example and don't make a compelling argument as to how and why paying differently those employees is akin to discrimination based on race.
Were you to pay the same salary in Netherlands and Bay Area, you would effectively either (i) pay absurdly high salaries to your dutch engineers, similar to what the prime minister earns or (ii) pay absurdly low salaries to your Bay Area engineers (to the point that you'd basically have none of them).
Who cares if you don't have Bay Area engineers? Their ability isn't based on where they live, is it? If they want to lower their costs of living so they can compete they can "just move".
The world is moving towards more closed borders, so moving is getting harder. But in the end we’ll all need to move to low-income countries like Yemen, Togo, and Ethiopia, to lower our costs, and be most competitive for employers?
Define best engineers first. I seriously doubt that hiring in SF will give you the best engineers, it might have one of the best ratios of good:bad engineers compared to many other places but it's not a given whatsoever that hiring one living in SF will be a slum dunk. I worked with many terrible engineers from SF over the past 20 years.
Which makes the point: why would a company pay such a premium on hiring from a single location if, in the end, it can only get the best engineers from there if it pays an even higher premium than the ridiculous salaries from the place? It doesn't make much financial sense, seems to be mostly based on "feelings" during hiring, like yours.
> seems to be mostly based on "feelings" during hiring
Silicon Valley has a local bubble economy. If overpaid engineers spend their money for overpriced health insurance, rent, mortgage or juiceros the boat stays afloat.
But if the SV companies start sending money abroad the bubble would deflate...
Bottom line you want the best people, which are spread around the globe, but within each country is a different price point for what you pay the best people. Bidding too high is wasteful to your resources, too low makes it impossible to compete in higher pay areas, and middle ground is the worst of both worlds: loose out on high pay areas, be wasteful in low pay areas.
I hope future generations will better understand that accumulating wealth should not be the goal of life.
Maybe future generations will not have to worry as much about accumulating wealth when countries work harder to ensure a high quality of life in the middle class. I doubt that will happen for the west until they figure out how to relieve all the pressure the housing market is putting on their citizens.
Location-based pay means the company pays more for areas where the supply is priced higher. It's not about cost of living, because the company won't pay me more if I prefer a Ferrari over a Fiat.
Location-based pay pays people more if they're in a market where they have other options, so they don't go to those other options.
In essence, if you have location-based pay, you're incentivizing your employees to move to an area where they'll have vastly more employment options than to stay with you.
Also buying power is just not the same based on your location. A hundred dollars in the US and in the Netherlands will literally not buy the same things.
The Dutch Prime Minister gets paid €186k annually [0]. He has 13 years of tenure in this role and 22 years total in the politics industry [1].
I'd hope a high performing, vastly experienced engineer in NL could aspire to that, though they might not get the same benefits or future opportunities.
A prime minister usually has quite good offers after the political career. Rutte has been trying to get the leader role for NATO, for example, which is probably a step up in salary (?)
Taking the CoL in NL, particularly the Randstad area, that salary is not outrageous. Especially, if you're a new expat on a single income, without any social net, and with any hopes of owning property, within a reasonable commute distance from your work.
It's fairly universal that politicians are underpaid relative to the level of responsibility they hold.
I'd quite like my country (UK) to be run by folk who are competent, rather than by only those who can afford to run for office without being paid for it. Yes, MPs get a lot more than the median income. But many people who are well-qualified to be MPs are also well-qualified to hold roles that pay at least as well if not more, and without needing to run for election to get the job.
My conclusion after reading this is never give it your all unless it's your own company. Burnout after working for someone else in tech should not happen at all. The post overall has negative undertones so it feels like the burnout was really bad after all this time, I hope the author recovers and finds joy and peace working on their own projects.
Also, from my understanding, the burnout did not seem to stem from the actual work but more from interpersonal relations and disagreements. I am certain that is not uncommon
This is what burnout is. Burnout is not caused by working, but by stress. The root cause of this stress is personal and interpersonal factors. People can burn out on very small workloads if the ratio of interpersonal toxicity to hours of work is high enough.
Sometimes it looks like people are burning out because of extreme workloads. In these situations, I've always asked "why is the workload so extreme to begin with?" This gets you to the real source of stress.
It's amazing how many issues in life are caused by the act of thinking of these possible issues, than the issues themselves. My friend had horrible issues sleeping, he tried many things before going to psychiatrist. The conclusion was quick and accurate - your issues sleeping are caused by your fear of not falling asleep, basically anxiety. Anti-anxiety meds fixed it immediately.
Wu Tang has the best lines to describe corporate workers:
“You're just worms in the worst part of the apple that's rotten
You squirm and you turn from the right, still plottin
All slimy cause you stay grimy, petty crimey”
While some have no alternative others enjoy that kind of game and are part of the problem. Best stay away from such jobs, and undermine corporate at every turn.
Do just enough day to day. Show heroic effort here and there. Don't get too attached to your project, service, code etc. Always understand the undercurrents so that you can keep abreast of things.
Yes. To add, always try to escape, have a side project, dabble in opensource, wake up early and use your best brain capacity for that then go to work. I know it's hard but if you're just what you do at work then future employment prospects are lower than otherwise.
Exactly, every company and org has a meta that you need to understand. If you understand the meta and you can/want to play the game, then play else start searching.
> never give it your all unless it's your own company
It’s the classic “last founder or first employee” problem. If you are an employee at a startup, have no illusions: if “they”[1] don’t give a shit about you now, you can be sure “they” will not give a shit about you later.
[1]: the company is and always be an inanimate and abstract object, your boss and colleagues will change
It's important to know what 'your all' is for the company. I learned early on that working more than 40 hours a week writing code results in my introducing bugs into the code base. So doing more is counter productive.
Choosing to not care is much harder as we have all had years of education conditioning us that we need to achieve more and to be the best.
> Choosing to not care is much harder as we have all had years of education conditioning us that we need to achieve more and to be the best.
Very much so. I enjoyed reading Coddling of the American Mind as well. Not 1:1 but related to the conditioning, lack of maturity, and in some cases, learned helplessness.
I moved from using bitbucket, first own servers and later SaaS to gitlab.com SaaS in early 2018.
I cannot say anything else than gitlab performance and UI being much better. I was permanently complaining about bitbucket, but very happy with gitlab most of the time. They had rather frequent partial outages, but resolution was always quick.
Maybe starting 1 year ago I noticed gitlab.com reponse times going down. It's not that things hang or are super slow, but they seem to consistently take 2 seconds more than what they used to. (The number is a complete guess, I have no measurements either before or after.) It's still usable, but no longer the pleasant experience it used to be.
Being able to work from where I happen to live and I have thought about whether I would like to work for them.
That it's a good product would certainly be a motivation. However, frequent incidents being resolved fast must be incredibly stressful for the staff. I prefer to stay where we make heavy releases only a few times a year for regulatory reasons. (Even though stress levels can raise until the last MR for the release has been merged.)
I read this meme so often on HN, so I genuinely want to know: how do you organize work and get people to act on a common vision if there are no managers?
I think people mistake “manager” as being people without tech skills. I think a tech company should have managers to create alignment and those managers should spend part of their time “managing” and part of their time still honing their tech skills.
In my experience, people will use "managers" as a proxy for complaining about all the other things they don't like.
Of course clueless managers and extreme bureaucracy will destroy a company, but I very often see engineers that just want to do X and ignore everything around it as if they were working in a isolated environment. It's often the managers calling these engineers back to reality, which creates tension.
The issue is you literally cannot be an engineer if you don't know how to build things.
On the other hand, there are plenty of managers that do not know how to manage people, that do not even know what is expected of them. Should they optimize for cost? For productivity? For employee happiness?
Competent managers are a minority. They tend to rise quickly, leaving the inept ones to run the day-to-day.
> The time it takes to deploy code is vital to the success of an organization
I've found this to be one of the most important metrics (if not the most important) for maintaining developer velocity as a startup scales.
Context switching is one of the most expensive operations in a developer's day-to-day work, and also one of the most ignored.
Managers love to build team schedules around their personal schedules, but this is often disruptive to ICs' "maker schedules."
Many orgs identify this and focus on the scheduling cost w.r.t. context switching, but the build-to-deploy time is equally (if not more) important for developer ICs.
I recently spent a few minutes applying to a Gitlab position, only to get a link to some Gitlab wiki explaining the list of EU countries they hire at is limited, and mine isn't among them. I wish whoever posted the ad had spent a few seconds copying and pasting that list from their wiki.
> No matter how you try to spin this, it's by all accounts an act of discrimination to pay one person less than another purely based on where they live..
Companies aren't gods charged with dispensing fair wages to us mortals. They are participants in the capitalist market.
And in that market, all prices are a matter of negotiation.
In any transaction, the buyer wants to pay as little as possible, and the seller wants to earn as much as possible. This isn't evil or greedy. It's a basic incentive that comes from the fact that people/companies/literally everyone prefers to have more money than less money.
Thus, your company wants to pay you as little money as possible, and you want them to pay you as much as possible. The same is true when the roles are reversed and you're the one in the buyer position. For example, when you're shopping, you want to pay as low a price as possible, and the seller wants to charge as high a price as possible.
Again, all prices are a matter of negotiation..
Power in a negotiation comes down to your BATNA, i.e. your best alternative if negotiation fails. Put simply, whoever cares the least has the most power.
In a capitalist market, that BATNA for the buyer is the next cheapest option. The next furniture store that's selling this couch for cheaper. The next adequately qualified software engineer in the Bay Area who's selling their services for cheaper. And the BATNA for the seller is obviously the next most generous spender. The customer who's willing to pay a little more. The company that's willing to pay higher salaries.
If it sounds like I'm conflating two concepts by comparing customers with employers, and businesses with employees, it's because you've been brainwashed into thinking these things are different. They're not.
When money changes hands, whoever receives it is the business. Whoever pays it is the customer.
That means, in the relationship between you and your employer, you are the business selling a service, and they are a customer paying for your service. Your salary is just a price. Your resume was an advertisement. Posting it everywhere was marketing. Interviewing was your sales pitch. As an employee, you are a business. (Someone just came along and renamed all the terms to confuse you.)
Which raises the question: Why on earth would your customer voluntarily pay a higher price than they need to? If all the stores in your area are cheap, why would they pay you more for your services than your neighbor is asking for?
When you travel to a country cheaper than yours, do you pay every restaurant, store, and taxi driver the same way you would at home?
No, you don't. Because that would be foolish. You pay local rates, which are less.
That's the same thing companies are doing when they pay lower salaries in cheaper areas.
If businesses were perfectly rational economic actors then all bay area/sv devs would be out of jobs.
> When you travel to a country cheaper than yours, do you pay every restaurant, store, and taxi driver the same way you would at home?
> No, you don't. Because that would be foolish. You pay local rates, which are less.
But if you could magically teleport to store or restaurant anywhere in the world, would you pay high prices to go to SV restaurant or store? No, that would also be foolish.
So local prices make sense of physical businesses because people pay premium for convenient access, for not needing to fly half-way across the world to do your groceries or whatever. But that same does not apply to full-remote developers; the work(/value) is getting delivered to the business all the same regardless where the employee happens to be physically sitting.
If the work/value delivered would be the same, then all software development would have migrated to Vietnam already. For some reason, companies think it is important that some developers are physically in Silicon Valley and they pay a lot to get that.
Is it about language and timezone barriers? No, because there are plenty of alternatives in the US as well.
If physical location is significant then it's not full remote position.
Beyond that, this is not just about sv vs rest of world, but about doing per-location adjustment based on local economic conditions. And it is pretty difficult to argue that the value of a developer to business is in anyway linked to their cost of living
So, he joined as employee #28 reporting directly to the CEO, then gradually more managers were slotted in above him, then one of the managers put him on a PEP, then he got the hint and left. I wanna say this story is pretty standard, and probably a big part of the reason why people in big companies often don't do great things.
Heck, even if you start a project within a company and it gets successful, the company can just slot in people above you, so you become employee #n in a project that you started, and then these people can say you underperform and so on.
I hope I'm not violating any copyrights – page 143 of Setting the Table from Danny Meyer [1]
> To me, a 51 percenter has five core emotional skills. I’ve learned that we need to hire employees with these skills if we’re to be champions at the team sport of hospitality.They are:
1. Optimistic warmth (genuine kindness, thoughtfulness, and a sense that the glass is always at least half full)
2. Intelligence (not just “smarts” but rather an insatiable curiosity to learn for the sake of learning)
3. Work ethic (a natural tendency to do something as well as it can possibly be done)
4. Empathy (an awareness of, care for, and connection to how others feel and how your actions make others feel)
5. Self-awareness and integrity (an understanding of what makes you tick and a natural inclination to be accountable for doing the right thing with honesty and superb judgment)
Exactly! I’ve never been able to express a succinct list of why some teams and/or companies feel better than others, but “51%ers” explains it perfectly.
For future reference, you are certainly not violating US copyright law, because quoting a few sentences from a book falls under fair use. https://en.wikipedia.org/wiki/Fair_use
I was curious why the author decided to call them "51 percenters." A google search of the term suggests that the skills of this group of employees are divided by 51% hospitality and 49% technical excellence. Please feel free to correct me if there is anything wrong in my interpretation.
Someone "whose skills are divided 51-49 between emotional hospitality and technical excellence" [1]. Seems quite bizarre to me to define it so precisely. Even if skills were measurable in such a way, how many people will be exactly 51% emotional hospitality, and why is 52% or 50% not suitable?
i think the implication is if a 51%'er has to decide between technical excellence and emotional hospitality then, all other things equal, they will use emotional hospitality since that's the majority of their skills (51%). It sounds like preferring to hold a hand vs rejecting incompetence. I don't really agree, i get not being jerk is important but i would flip it to 51% technical excellence 49% emotional hospitality.
Do you have any practical experience with companies running as it's described in this motivational book?
Many of those books are selling well because they are well written and say exactly what reader thinks might work, but if you ask anyone else who worked with the author, the reality can be quite different.
> Do you have any practical experience with companies running as it's described in this motivational book?
I do not have experience running it at a company level, but at a team level (I have been an engineering lead in two companies for the last 6 years).
From all the books I've read (I read a lot), this is the one that was most "spot-on" about treating other humans and making them feel valued and therefore building a team with strong bonds.
> Many of those books are selling well because they are well written and say exactly what reader thinks might work, but if you ask anyone else who worked with the author, the reality can be quite different.
Absolutely agree.
In my experience I resonate most with any books when I have already, unbeknownst to me, been applying what they preach (which has been the case with Setting the table that I'm currently in the process of finishing).
I believe that it requires a lot of introspection to be able to apply new knowledge (ie, if you haven't thought about it or experienced it before reading about it)
That's what I like about "agile roles" like Product Owner or Scrum Master, they take a slice of traditional manager's responsibilities, but they don't have any reporting authority over other workers. My EM has like 30 direct reports and it works fine because he doesn't really have anything to do with our day to day work.
I'm not sure how you can evaluate 30 people you don't interact with closely.
NIMS, the National Incident Management System, talks of ICs having between 3-7 direct reports, when there is a need to be connected to what they are doing, because beyond that, you can't reconcile things easily.
Those semi-managerial roles are the biggest problem with that model, in my opinion. Sure, it works as long as everything is peachy. But as soon as there are any real conflicts of interest, it will show who is the real manager. And it's not the product owner or scrum master.
With authority comes responsibility for your actions. Without responsibility, no authority. The product manager is a manager in name only, and product owner even less so.
That doesn't mean you can't have several direct reports. The classic matrix organization for example. But it means semi-managers without real responsibility have no real mandate for doing a good job at the slightest hint of trouble.
> But as soon as there are any real conflicts of interest, it will show who is the real manager. And it's not the product owner or scrum master.
If there's a conflict of interest, it needs to be discussed based on merit, not based on who has the bigger authority.
If there's no agreement, it needs to be escalated to somebody who has the authority (manager). But IME this doesn't happen very often.
I like this model, because the default position is that none of the engineering, product, process is the "master", so you need to negotiate. If one of the roles also has reporting authority, that automatically skews the decision making towards yielding to them.
That list of “skills” is spot on. I also especially like his use of the term “skunking” to describe how somebody’s personal opinions/problems/issues impact the rest of the team. “51%ers” are exactly the kind of people I want to work with.
A better approach is to have separate career tracks for 'people' managers and 'project' managers.
People managers do the performance evaluations and various HR administrative tasks (signing time cards, hiring, firing, etc.) but they rely on feedback from their group which are both individual contributors and project managers.
Project managers lead the projects and have to select/attract the right combination of individual contributors to their project if they want it to succeed.
A project that 'gets more management' will usually have to justify the addition of PMs from a cost-benefit perspective. And a project that is overburdened with management types will usually see the ICs migrate to other projects in order to improve their impact.
All this happens organically, so individual contributors are empowered instead of being disenfranchised through organizational changes.
That sounds like basically matrix management which has many well documented issues. The biggest one in my experience is that the people managers need to be themselves judged on some rubric. If that rubric is success of projects then it tangentially aligns with business goals. If it's something else or they don't have power over projects then they are encouraged to play constant politics.
> Why would they need to play constant politics if they don't have power over projects? Not everyone is motivated by the same things.
They do have power over the projects. Being able to PIP someone is power over everything that person does including which projects they work on. Including which projects no one works on. Except it's not their direct power which means to leverage it they need to play politics. Adding layers doesn't remove that power but simply increases the amount of politics they play to make up for it.
The goal with the matrix management is to distribute the risk. If your people manager puts you on a PIP then at least your project managers will have some ability to push back on that.
But there is no good reason for the People manager to care anything about what projects have people working on them. If they start to care about which projects are successful instead of all projects are successful then they're not a good fit for the job. And yes I have experienced that, as well as it's opposite.
and this leads to 4 engineers and 6 managers sitting in a meeting, and nobody actually being responsible for anything.
no, of course, there's a lot of value in providing escalation/descalation/rehoming processes, and dedicated ways for org-wide feedback on people's and projects' impact, but people are not just three orthonormal roles on top of each other in a trenchcoat, if there's not clear hierarchy then - as others pointed out - the informal chaos takes over (because it's the human default)
I have that at the current company I work at. Some manager started a successful project with me and the management is too greedy and busy trying to replace me and the manager by bringing in politics, social dilemma and putting layers of management between everyone. Truly a sad state to be in. I learned my lesson as to why you should never truly care about what you do at work. Feels like the project will rather go belly up and disintegrate because of that.
You should care about the work you do. Just remember who you are doing it for. They own the work they being the company. If the company wants to reward incompetent kleptocrats then I salute them as long as I get PAID the second they stops happening they can with the greatest of respect get f’d. then you just take your trade and apply it somewhere else having learned expensive lessons they paid for about what worked.
> to replace me and the manager by bringing in politics, social dilemma and putting layers of management between everyone.
sorry, this seems ... too vague for me to understand what's going on, and how it connects to their greed.
are they trying to 'scale up' this project? why aren't they keeping you? replace you how? in what capacity? are you currently wearing too many hats according to them? can you please give some details?
I recognize the feeling of shouting into the void trying to get people to care about something.
You’ll be telling, showing, raw stats, massaged stats, appeals to authority, what have you, but if the only thing the directors care about is features, it’s all for nothing.
After 3 years it finally became a problem because they noticed that releases kept failing, but I don’t think it had anything to do with me so much as someone typing ‘how to stop my releases failing’ into Google.
I don’t know how to solve that, as I’m going through the same thing again now (with a different subject).
I thought it was a great write up of the lifecycle of a successful startup. If you stick around you get to see a lot of change. Some positive some negative. Of course at 1000+ people you can’t have everyone reporting to the CEO
> Heck, even if you start a project within a company and it gets successful, the company can just slot in people above you, so you become employee #n in a project that you started, and then these people can say you underperform and so on.m
The trouble with these stories is that they’re n=1 anecdotes and we only get one side of the story.
There’s an implicit claim in many comments here that we need to assume that the employee was actually a higher performer, didn’t need managers, didn’t deserve a PEP and so on. That’s understandable given that we tend to put ourselves in the shoes of the person writing a piece and being anti-corporate is always popular on HN.
However, those aren’t safe assumptions in cases like this. I’ve worked at a couple early stage startups that acquired early employees who couldn’t (or wouldn’t) grow with their role and the company. It’s common to keep early employees around out of respect for their past work, a belief that they hold difficult to replace historical knowledge, or simply because they’re well connected to founders and other early employees who have grown into leadership positions.
But in reality, simply being an early employee and being involved with early important projects doesn’t necessarily mean that person is the best person for the job or even a good fit for continuing to do it. Some of the early stage employees I worked with were great at cobbling something together from scraps and keeping it functional with a collection of cron jobs and manual interventions, but their operating style doesn’t work at a bigger company at all. If they can’t adapt and change or they become disgruntled about having to work on things the way you have to at a bigger company, they start to become more detriment than help. That’s just one example of many different potential failure modes of early employees as companies grow that we don’t like to talk about.
> I wanna say this story is pretty standard, and probably a big part of the reason why people in big companies often don't do great things.
It’s “standard” in the sense that every early company has a number of early employees who don’t grow with it, but it’s not the only or even a common fate. GitLab has plenty of employees who have been there for a long time, but you’re not hearing about them from disgruntled blog posts. Consider the selection bias when reading this.
As for the claim that they can’t do “great things”: I’ve used GitLab for a very long time and I disagree. The product continues to evolve. It’s not a stagnant product at all.
‘He worked remote’ thought popped into my head. He wasn’t there to “grow into leadership role” with the rest of the early employees. I do agree with your read of the situation.
btw, that restaurant in Amsterdam is apparently the go to place for startups. I am certain I sat in one of those chairs (was this on 2nd floor facing the canal?) with another fabled startup.
I largely agree, especially with the sentiment that there are two sides to any story.
I think one angle I'll nitpick though is this doesn't have to be related to whether or not the employee was a high performer or had particular leadership needs or whether they're adaptable to the current maturity level of the org. There can and likely are many other elements such as whether the manager/employee even get along, or agree on direction, strategy, problems, etc.
And the difficulty is, these types of issues can often show up as performance issues. In lots of cases, is the performance issue the employees fault or the leaders.
In my anecdotal experience where a job went from the best role I ever had to one of the worst happened over a very short week or two with the change in a manager. And I think to the point you're making, when I resigned I outright said... each of us is going to blame the other for my departure. The hard truth is the answer is likely if either of us had been different people I'd still be in that role doing work I really enjoyed, but in that particular leader/follower relationship we simply didn't get along well at all.
If I ever did a what it was like or why I left post, I'd probably have plenty of arguments for my side, because I think to your point, I'll be the hero in my own story.
> In my anecdotal experience where a job went from the best role I ever had to one of the worst happened over a very short week or two with the change in a manager.
Two managers in my case, both of whom were, let's say, not successful in their roles. And a glass ceiling so that I had to do most of the work _and_ be a tech lead, yet "could not manage the team because reasons".
Sudden reorgs + hire above + glass ceilings = golden ticket for horrors like this.
My read is that he was one of those early employees that is a huge pain to manage. They want to do what they think is fun and interesting and they have strong opinions about shit that is ultimately inconsequential. It's a nightmare to manage an IC who is friends with the CEO and doesn't think you're making the right choices. But ultimately they don't accept accountability for any decisions, they just move bop around and cause problems.
One tell-tale sign is endpoint security and using his own device. It's the kind of permissive cultural thing that does not scale because of compliance issues and developer productivity overhead. But it's very hard to wrench these long-time devs away from their preferred Linux distribution which requires conditional build stuff everywhere to support. Use a work computer for work, let them monitor the updates and stuff, as long as they're not using the webcam to record you who cares?
The database backup story - my guy you were on the database team. Backing up the databases is job 1. You can't just passive-voice away "oh there were no backups". But of course he's more interested in fighting about sharding architecture than actually keeping the site running day-to-day.
His big takeaway is that Gitlab didn't spend enough time on performance for their hosted offering which was a huge money loser. Just because he thinks performance stuff is fun to do, if the hosted offering is a money pit of course they're not throwing more resources at it. You have to make an actual business case for why your thing is more important and makes money more than another project. You don't just engineer in a vacuum for the fun of it.
The dev who makes shit work and and tracks measures the ops gains / reduced headcount needs and reports their impact up the chain. Who also gives talks on how everyone can make their stuff run as simple as they do.
Your read lines up with another point in the article that I found to be stated in strangely absolute terms:
> you need to be able to deploy your code fast, i.e. within at most an hour of pushing the changes to whatever branch/tag/thing you deploy from.
This sweeps under the rug all of the potential issues with fast deploys. I guess it depends on the product. I work on a managed database service, and one category of potential bug is that we accidentally delete or corrupt customer data. We have to be much more careful and can't just deploy what was submitted to mainline in the last hour without doing significant regression and performance testing.
But anyway essentially the main reason they give for fast deploys is:
> being able to see your changes live is nice because you actually get to see and make use of your work.
I think this is true. But, it lines up with this negative interpretation of this article. The author seems to prioritize themselves over the health of the product they're working on.
Sure, you could argue such people are needed only at the early stages of a company, and counterproductive when the company gets bigger. But I don't buy that. Why then are big companies asking all the time: "Oh, where's our internal startup spirit? How can we bring it back?" Right after firing the people who encapsulated that spirit, for being "hard to manage".
Because sometimes all they're after is for slideware on GenAI. The people they fired encapsulated a spirit of innovation that doesn't fit the operating model and therefore is of no value to them.
I think this trend to talk about startup practices in large orgs is more executive nostalgia and a complaint about all the processes put in place to catch mistakes that have been made before.
The same people as developers would be pursuing rewrites of rock-solid 20+ year mature software projects because there's a trendy framework.
Large organizations don't have 'startup spirit' because 'startup' companies fail. Employees of large mature orgs with 6000 employees didn't sign on to a company that's got a good chance of not existing next quarter. They're not taking massive risks and throwing halfbaked features into a brand new product with 1 client hoping to get bought by facebook or maybe an insurance company.
If those big companies are really complaining about not having startup spirit maybe they should provide an exit for the VCs and aquihire (briefly because the engineers will all leave asap) a startup!
Sometimes it is not even about the "startup spirit", but more like "why is it that we didn't do any projects of significance in the last year, and why is our incident rate 3x from what it was just 3 quarters ago, and why are all these migrations not finished yet?"
I can assure you that me being a pain to manage wasn't the problem, nor was it ever brought up. In fact, the only negative/improve-upon-this-thing kind of feedback I got once or twice was to adjust my communication style to be less blunt/harsh, something I agreed with and did end up improving upon.
I can also guarantee you that the work I did very much did good work instead of "cause problems". Feel free to ask anybody that worked at GitLab during the same time (or is still there) and see for yourself :)
> I didn't get along well with this manager,The resulting conflict lead to a "performance enablement plan"
Respectfully, in your post you literally say you got a new manager who PIPed you for being hard to work with. The whole tone of the article feels like rehashing old disagreements.
Respectfully, I didn't. I specifically wrote the following:
> I didn't get along well with this manager, The resulting conflict lead to a "performance enablement plan", a procedure meant to get things back on track before the need for a "performance improvement plan" (PIP). A PIP is meant to be used as a last attempt at improving the relationship between an employee, their work, and their employer.
Not getting along with someone doesn't imply or mean that a person is difficult to manage. Instead, it means there's simply friction between two people.
In this specific case, the main source of friction was that my messed up working hours resulted in me performing tasks later than expected (though still within any deadlines), though I recall those time frames not being well specified to begin with (i.e. it was more of an implicit assumption that X would be done by hour Y).
I believe I was also a little late for a meeting because I'd overslept. That's not good, but it certainly isn't a case of "Wow this person is so difficult", instead more of a case of "This person needs to get his schedule back together".
Either way, you seem to be interpreting the story in a way different from what's written down. I doubt I can change that, so I'm going to leave it at this comment.
how can you assure that? it seems like a classic self-fulfilling prophecy of arrogance.
changes bring some adjustment uncertainty, and sometimes it goes well, things get better, sometimes it gets worse. with enough time you'll draw a short stick. it sucks.
> how can you assure that? it seems like a classic self-fulfilling prophecy of arrogance.
Because colleagues have told me so, and the annual employee reviews were positive as well. In fact, outside the PEP the only actionable/to-improve feedback I got was essentially "Sometimes you can be a little harsh/blunt", which is vastly different from "this person is difficult to work with".
> It's a nightmare to manage an IC who is friends with the CEO and doesn't think you're making the right choices
Since I find this take harmful, let's rotate the scene: what if you have no business being in the manager's seat above that IC in the first place, and you are effectively being used as a tool by someone who is not the CEO, but, say, a newly-appointed CTO? What if you _are_ making the wrong choices? What if that IC is one of the few people in the org having the experience and the knowledge to tell you that you are? What if that IC is making the right calls and suggestions, but your implicit directive you got from the newly-appointed CTO is actually to manage that IC out "because they are such a pain for everyone"?
I have been on both ends of this situation and I do believe most scale-ups are _not_ doing a good job retaining and nurturing their early-stage staff+ ICs into their larger era. This has multiple reasons, but it does often feel like there is a lot of value to be had if just those staff+ ICs weren't so horribly mismanagemed. A hire-above-from-the-outside is prime example.
And if I sound bitter - let's just say I have experience being "that IC". It wasn't pretty, and no - I was not an asshole.
In every startup I've been in, there was undue deference to long-time ICs. They didn't normally stir up public fights with execs but they would openly undercut line managers and then move around in the org as people tried to stop being responsible for them.
It's not really clear what you do with a staff+ long-term IC. A lot of them don't want to manage or be involved in leadership, but they want to pull down a huge salary and just do work that is frankly replaceable by a senior eng.
To be clear I do believe there are levels of IC experience above staff, but being 21 and joining a startup doesn't make you a Principal Engineer just because you hung around until you're 30. Especially if you've only worked at that one startup.
> It's not really clear what you do with a staff+ long-term IC
Set clear expectations in the first place. "I want to run things my way here, I'm the new management and I don't like you" is not an expectation - and that's what I have observed happen all too often.
From my own experience as very early employee, I know how it feels, to have them slot in more leads and managers above you and step by step losing the mechanisms, that enabled you to be very effective early on, when you just got things done. Later every step needs to go through some slotted in manager position, making sure you don't work on something "wrong", no matter how much your early ideas and contributions enablef the company to rise in the first place.
I wonder if this is what is behind years old stale issues of gitlab, especially regarding Gitlab CI. From the outside you get the impression, that they completely lost ability to build great things, since they do not seem to care to fix years old bugs that still come up again and again.
I've seen some recent data on the salaries they're offering in my area. I like Gitlab but wouldn't be prepared to make those kinds of sacrifices to apply there.
> This then lead to the discovery that we didn't have any backups as a result of the system not working for a long time, as well as the system meant to notify us of any backup errors not working either.
This seems to be entirely normal for startups. I've asked about it during interviews.
Do these startups consciously decide not to have backups (for time/money reasons perhaps)? It makes no sense to me.
I've worked at some startups and most senior engineers know what's necessary to keep production systems running reliably. That includes monitoring and backups.
Could it be that this is entirely normal for startup with less experienced engineers doing things for the first time?
Terrible managers are the bane of our industry. OTOH, many startups owe their existence to their founders having had to deal with terrible managers in their previous jobs. The story of the traitorous eight and Fairchild semiconductors comes to mind. In my own life, if it weren't for getting a non-technical but political and vindictive manager in my last job, I probably wouldn't have had the motivation to start my own startup. In hindsight, I wish I'd been assigned a manager as bad as the last one I had, earlier in my career. :)
I wish you could flag companies that have toxic management without compromising your privacy or getting outright censored. It would also be great to find companies that truly need your skills, so you can hop more easily without the HR overhead.
Microsoft has toxic management but it also has M1 who are just trying to navigate the maze and do the right thing. It is never straightforward like this.
Besides toxicity, i believe the main issue of some bad management is uselessness. Perhaps it leads to toxic situations, but the root cause is that they don’t add value to anything in the org.
It’s worse than not adding any value to the org. These people actively undermine value in the org by depleting their direct reports’ morale. OTOH, any organisation that tolerates and turns a blind eye towards such subterfuge, perhaps deserves it.
“People don’t leave bad companies, they leave bad managers”, an aphorism that ringers truer with every departure I observe.
(But I also see how the incentives of the org can erode at the energy put into cultivating management skills and prioritisation of effective management work.)
I wish the ‘More people doesn't equal better results’ would be put into every manager door and desk. I would also add more results to the quote to make it even more accurate.
Read replicas are better suited for this situation and way less complicated. And you said it yourself: you need to be careful and you loose the ability to do joins.
Totally agree. Overall the author's arguments against sharding seem short-sighted, at best.
All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes.
For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB.
For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database".
Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.
Remember most of the business is on prem installs . Introducing the huge complexity of sharding to the codebase would have benefited these installs approximately zero.
The existence and uptime of gitlab.com is essential to the company's customer acquisition. If gitlab.com goes offline for an extended period of time (due to database size issues or anything else), it's a huge hit to the reputation of their brand. This will directly affect their bottom line even if it isn't the main direct source of their revenue.
And again, what's the alternative? If you offer a successful user-generated content platform, eventually you must shard. User-generated content only grows over time, it never shrinks, and at some point you exceed the maximum amount of storage that can be attached to a single server. And long before that point, you encounter horrendous operational headaches, such as full backups taking longer than 24 hours to make (let alone restore from).
It's also conceivably possible that their largest on-prem customers would be interested in sharding their self-hosted installations, so theoretically this could be a valuable high-cost enterprise feature.
I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines keep getting bigger, if your growth rate is low/moderate you might never reach storage limits. Or might theoretically reach them in 5+ years in which case many other things could change before then, in tech or the business. The alternative (to expensive front loaded investment in sharding) is just not doing it, until you have to, because its likely you might never actually have to.
Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab.
Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.
Sure, I'm not advocating premature optimization here. Don't shard unless it's clear that you eventually need to. But we're talking about a specific company here, not sharding in general.
I believe in GitLab's case that they absolutely will need to shard eventually. This is a decade-old company with a $12 billion market cap, not some early seed-stage venture with unproven success. And they're absolutely growing faster than hardware storage capacity limits are increasing.
Even aside from data size motivations, they've clearly already had blast-radius problems from using a single monolithic production database.
It's not like sharding is some major unsolvable problem in 2024. It was a lot harder 10-15+ years ago, when engineers with sharding experience were extremely difficult to find, frameworks didn't support it at all, premade solutions didn't exist, etc. But even then it was absolutely solvable, I can say from first-hand experience, even with much smaller teams than what GitLab can afford to throw at this.
Every time we'd look into this, we'd look into the challenges we'd have to face and see if they changed since last time. One of the problems GitLab faces is that it JOINs all over the place. In addition, while most data can be shared on a group/namespace basis, GitLab also does a bunch of cross-group queries in frequently accessed pages.
To put it differently, try to think of an application that does all the things you _don't_ want to be doing if you're going to (or want to) shard. Then imagine that that's basically GitLab, coupled with the load patterns simply not justifying it.
>In spite of all this, I'm not sure what alternative I would recommend instead of the combination of Ruby and Ruby on Rails. Languages such as Go, Rust or Node.js might be more efficient than Ruby, but none have a framework as capable as Ruby on Rails.
Go would be a sensible choice.
Both .NET and Java have very good frameworks and are good choices for large scale development.
Rails is nice for small to medium websites, but for large microservice based apps will present some issues.
How is that true? If a company pays $30k monthly in Bay Area and $10k in Netherlands, what should the unique, non discriminatory wage should be? $30k or $10k?
Some people believe that every employee's productivity and contribution to value can be precisely quantified, and fairly split between employer and employee.
I've always been sceptical of this myself - how do you value that code review where I stopped a junior engineer's XSS getting into production? Incredible value? Or 10 minutes of work?
Always wondered that. It means everyone's salary is determined by the highest-cost of living area you want to hire from. Doesn't really make any sense seen that way.
Everywhere I’ve worked there’s always been one burnout engineer like this who doesn’t really know what anyone else does or why but is absolutely convinced that everyone else except for him is an irredeemable moron.
> In practice this lead to GitLab building many features over the years that just weren't useful: a serverless platform nobody asked for and that was ultimately killed off, support for managing Kubernetes clusters that didn't work for three weeks without anybody noticing, a chatops solution we had to build on top of our CI offering (thus introducing significant latency) instead of using existing solutions, or a requirements management feature that only supported creating and viewing data (not even updating or deleting); these are just a few examples from recent years.
If my company ever grows to be as successful as Gitlab, I’m going to much more worried about innovation grinding to a halt than about people trying stuff that doesn’t work, or trying stuff the wrong way. It’s so much easier to shoot an idea down than to believe in it long enough for it to have a chance. I strongly agree with the author about making engineers be the decision makers on how to build out an engineering product, but I’m not convinced that this list of failed ideas is a bad sign at all.
It's a matter of focus: Having some time to tinker and try out stuff that might just fail is great. If it consumes so much development time that the must-haves deteoriate, you let down your customers.
I think it's less innovation and more failing to understand the requirements (i.e. only implementing half of what's needed, and then moving to the next thing). E.g. adding labels to gmail, but no way to have filters, so you have to manually apply them.
This part of the article resonated with me, as I used to work as a tech lead for a very similar startup to Gitlab. We were always pushed to "ship MVP's" and "move fast and break things" by product and design, which always resulted in a half-broken features getting pushed to production and left there once the sprint was over. This also led to a constant inexorable growth of tech debt, that made me feel like we'd never be able to get things remotely stable in our corner of the product. I've since come to the conclusion that it's much better to build small, flexible, feature-complete systems that can be used for multiple iterations of ideas.
Story of bad management aside, this was also a highlight for me on the risks of early stage startup even when the startup goes big
> In my case the amount of taxes would be so high I wouldn't be able to afford it, forcing me to wait until GitLab went public
I have seen a bunch of cases where people really want to leave the company but can't due to short exercise windows and taxes that come along with exercise.
One of the companies I know had to partner with a lender just so that their employees could exercise stocks. So now if the company IPOs you owe the lender a load compounding at 15%
Thankfully many newer companies seem to be going towards 10 year exercise windows, but they are still a minority
> Thankfully many newer companies seem to be going towards 10 year exercise windows, but they are still a minority
I'm actually a bit on the fence for the 10 year exercise window, as I'm not sure it's always a benefit. As I understand it, startup exits timelines are on an upward trend, so in 10 years will the average exit still be below 10 years, and even then are you working for a company that meets the average timeline. Then, from a taxation perspective, if you do want to exercise before the 10 year expiry, if it's cost prohibitive now, it should be bankrupting after a few more investment rounds and increases to the 409A valuation.
So I'm curious if there is a side effect here that the 10 year exercise window actually leads to many more shares becoming even more cost prohibitive to exercise right before the company reaches a liquidity event and as such not exercised (if you're in the early stage employee group).
Yeah, you are right on the point that if you are not able to exercise now you might not be able exercise later also. It's still lesser of the 2 evils imo because:
1/ many companies have liquidity events as they mature. So you may not have to wait for IPO/exit
2/ In 10 years you often know if the companies gonna go anywhere. So if you feel the company's ngmi, might as well not exercise.
I have also heard some companies fixing their exercise price (as in its not tied to 409A) to $1 or some small value like this. That seems like the best option
This is an interesting read and I am grateful for the author sharing their experience.
One thing that I could pick up is that the author is somewhat bitter about their experience, but isn't really taking responsibility for their own contribution to getting into the situation.
At most companies that are not evil and not exceptionally dysfunctional (and I presume GitLab is neither) getting to a state where one is put on a PEP/PIP is preceded by a longer period where they had indications that they are not performing in accordance with what is expected of them. Also, at most larger companies consistently "meeting expectations" is not hard, if you're in a role you're skilled for (which it sounds like the author was) and willing to receive the explicit and implicit signals from your environment, especially the hierarchy above you, and adjust.
Being able to make these adjustments and work effectively as part of a complex environment is an important part of working at a larger company. Not taking responsibility for the need to learn how this system works and adjust accordingly is counter-productive. It sounds like the author realised that rather late. Perhaps they could have had a better experience if they figured that out earlier.
Very interesting to read as a former employee of a company being Gitlab's customer, self-hosting EE production and solving everyday problems, with their support also, during the described years
557 comments
[ 2.8 ms ] story [ 354 ms ] threadWe have rules in govts that companies should not discriminate against employees based on sex, religion, sexual orientation etc etc.. But it is fair to discriminate the salary of an employee based on location? For ex: I know a few friends who have moved from Europe to Asia with the same role and are getting paid less compared to what they were getting paid in Europe. Its the same role, its the same person, but getting paid less just because of location ?
Pregnancy, maternity, marriage, civil partnerships and gender re-assignment are usually chosen by individuals, not forced upon them.
(Stress on might and some, probably still not enough in may rich countries to stop native population from shrinking.)
I don't believe that is a legal requirement, anywhere. Remuneration is based on many factors, which can include the cost of living. A company will not be able to hire someone in New York City, for the same price as someone in a less expensive jurisdiction.
This isn't discrimination, it's simple economic reality.
So you’re saying, we should be paying engineers in Europe and in the US the same as an engineer in LATAM or India or Asia that has the same level of experience and skill.
The only way to be non discriminatory is to have a standardized formula of compensation that takes into account cost of living (rent, food, healthcare, taxes etc) where the final take home pay in locations around the world are equivalent - which I believe should be the case at most companies
It works both ways, anyway. If those friends had moved from Thailand to Switzerland, would it be discrimination to pay them more?
[0] https://www.canada.ca/en/revenue-agency/services/scientific-...
Yes, but quite often, workers are in the same country (or even same state!) and still get paid differently based on CoL.
As a company you need certain abilities, and you pay whatever the market decides these abilities are worth, and nothing more. Depending on location, the market will set a different price on these abilities, so you pay different.
You are supposed to pay them the minimum amount it takes to get them to show up to work. When someone moves to a less competitive market, where getting another job is harder, then they are more likely to show up for lesser pay.
And remember that a country may have a less competitive market, even if the workers are remote and not seemly bound by a local market, because governments often love to put up huge roadblocks when it comes to international hiring. If you are being paid less than you were in another country doing the same job for the same employer, this is almost certainly why you have agreed to take a pay cut.
If I get hired in such a company, I'm moving to SF or Zurich the next day.
Fair pay to me, at least, means paying for results. Not paying for hours spent toiling. Not paying for where I am on the planet. Not paying for how I get the results, just for the results.
Instead, there are all of these gamey factors inserted into the mix. They're emotional. They're manipulative. Yuck!
I don't think this make any sense on so many levels. First, "abilities" are not a good way to think about wages. If you hire a neurosurgeon to do your gardening, you won't pay them more than a run of the mill gardener.
Rather, you as the employer compete against other employers on different markets in a fairly classical supply and demand situation. The "abilities" of an compliance expert with tech skills did not change much when GDPR was introduced, but as all EU companies scrambled to figure out the regulation (and the DPO role was popularised by fiat), the compensation went way up.
If the employee can participate in e.g the SF labour market, you have to pay a competitive salary in that market, if not you don't have to. As long as there are barriers, e.g a on-location worker in SF has more opportunities for whatever reason, the location premium makes sense.
To take your example in the opposite direction. Let's say a east-european company want to expand into the US and open up a sales engineering office in SF, and want their best sales engineers to go work their, it would be completely insane to not raise their wage. "We pay people after ability here you have 40k USD, have fun finding housing".
I used to feel this way, but largely grown out of it. In the end, you're asking for SF salaries, and those in India are asking for NL salaries(or SF salaries). You'd largely find a race to the bottom, I think.
The simple fact is you made a good living for your area, and they made a good living for theirs. If you want more, move. If you won't move, there's likely a reason why.
Everyone wants a SF salary with an Indiana or even India cost of living. For obvious reasons, that can't work.
>It doesn't matter whether you're paying somebody in the Bay Area $100 000 per year, or somebody in the Philippines, because the cost for you as a business is the same.
The cost for you as business is NOT the same. Start from the taxes part, that 100k is what the employee gets but the cost for the business is higher depending on the country/area because they also pay tax on top, social benefits etc. Also need to have in many cases a local business, doesn't matter if that is virtual etc, since they need to adhere to local laws thus having resources supporting that etc.
So while I'm on the employee side here as I'm also working on a multinational coorp with global role yet paid with local standards, there is more than meets the eye
That's the company's problem.
If I’m considering two people for a job but one is in a place where paying them well means I spend a huge amount on fees (or “employer-side taxes” for that matter, looking at you Austria), I might well choose the other.
The complication is the company prefers to work in one/a few jurisdictions only and have "proper" employees. It simplifies a lot for the company.
These positions are phenomenally competitive. You have swung open the doors to the world, and said "Give me what you got!", and you reap the benefits by having a much, much larger pool of applications and talent to sift through, with a lot of truly exceptional people in there - mostly very intelligent, very driven people from poorer countries.
That's how supply and demand works. If you are offer to pay people more, you usually end up hiring a higher quality candidate.
Results of offshoring to Bangalore is a stereotype, but it wasn't born from nothing.
I've worked with teams from Bangalore who were staff of the bank I was contracting for -- they were amazing, but also not appreciably (if at all) cheaper than employing someone in London or New York.
Several well-known banks had large offices, and competition for talent was high. No race for the bottom there.
It doesn't particularly matter where you employ people, if you're trying to save costs by paying people less then you're going to have a bad time.
A tangential proof of my initial statement can be observed in US immigrant IQ levels, as compared to the general population.
If you want to pay them lower wages, they'll work for someone else instead.
Nearly nobody is ever going back, even those who can't land a proper job just stay and drive ubers.
I may even be grateful for the chance, instead of angry that it wasn't a deal that was going to last in perpetuity. I admit I may be in the minority here.
Everyone after you is out of luck and can develop software for $200/m then.
Is it? The only place I know off the top of my head is maybe ten years old.
We do want some overlap in working hours with the US, so it is true that we cannot realistically hire anywhere just yet, but not being in the US is not a dealbreaker.
They allow brain drain to happen, without the barriers of having to move countries.
However, if I was earning an order of magnitude more money than I currently am, I might want to pay a little extra to go to the really good barber, or to eat at the really nice restaurant at the riverbank. Or, hell, I might just employ a cleaning service every week, to save myself a few hours' time vacuuming my apartment. These necessarily local services will also see their revenues rise. To me that seems to be a more important effect on the local economy at large.
A healthy growth economy can tolerate income differences. But the balance is certainly precarious, as the example of New York or London shows. It’s constantly on the edge of driving out the remaining barbers and chefs because they can’t afford rents.
The amount of money I poured into the local economy is probably an order of magnitude higher (maybe even 2) than if I had worked for a local company.
If they did have to move, then they would, and you'd have brain drain. But because they can remain in their communities (while earning the globally-competitive income that they would otherwise have to move for), they now pay taxes to their local government, buy from local businesses, mentor local youth, and so on. When they've earned enough money from their job, they may quit and start a startup in their own community, or become an angel investor supporting startups in their area, rather than yet another bay-area based fund. These are all good things!
But I'd expect a very low number of firms to succeed at that.
To me it seems like a really exciting place to apply recent innovations in LLMs. If LLMs can chew through thousand page legal binders like toilet paper, there's no reason they can't chew through a thousand 1-page resumes and spit out "These are the ten most promising ones based off of our statistical analysis." Firms can specialize in the production, hosting and fine tuning of these LLMs, and even play both sides of the market by allowing candidates to see how good their resume looks for a given job description.
I think this is much likely to become a lot more common in the latter half of the 2020s.
If you were to forget about location and just say "we'll negotiate all salaries" then you would end with exactly the same result because people in NL are willing to work for much lower salaries than people in SF.
I don't get why so many smart programmers don't understand this basic fact of economics. Eh maybe they do understand it and are just jealous of insane SF salaries (I certainly am!).
I would be wary of demanding equal pay by location anyway because you'll end up with all jobs moving to India.
I wish I could have taken the Supabase role because it was definitely my top pick otherwise. One look at the output and caliber of people they hire also indicates that they have little issue finding talent.
FWIW this was a couple of years ago and I have no idea whether they are still doing this equal pay band thing or not. But they were doing it for awhile at least
Today, most labor arrangements are more and more like companies. If you were to select top companies to contract for some job that doesn't really care about location, you wouldn't be choosing companies based on that. You'd choose based on how good they are.
It's the difference between trying to buy the cheapest versus buying the best. Of course if you're always looking for the cheapest, you'll always move towards overseas jobs. But if you're looking for the best, you can get the best from all over the world by offering a single solid compensation package.
It's all a transaction, isn't it? At the end of the day my labor is worth however much I can get for it.
Sure, you could squeeze even more profit by paying overseas workers less, but then you create all sorts of imbalances that can and will hurt your business in the long run.
I always joke that if you want to hire me (I am not from the US) and pay 50-60% less just because I live here, why wouldn't I work 50-60% less?
You're getting the 1% of a lower income country, for an average local developer salary. If you want the 1% of SF you'll have to pay a lot more, even if they are equivalent in the value they provide. The company still wins, and as a result you get happier employees.
You can always cheap out, but it's never without consequences.
Basecamp has been around for 20+ years and they publicly mention that they hire based on SF rates, not even SF but the top 10% of SF[0] for positions around the world.
[0]: https://signalvnoise.com/svn3/minimum-pay-at-basecamp-is-now...
Furthermore, if the difference in compensation is really explained by differences in employee productivity caused by work culture barriers, then location-based pay must somehow maps to the productivity cost of that work-cultural barrier. There's no reason to think that this should map directly to cost of living, and it would change over time; if the workforce composition shifts towards Europeans for example, then now it's your workers in the bay area who are less productive due to missing subtle cultural cues from their German managers.
Interviewing/hiring is incredibly noisy though. If SF engineers have a much higher average skill than the rest of the world then you might still end up with better people if you just hire from SF rather than the world in general, even if the latter has a much wider pool with more top people in absolute numbers.
when i looked at their salariy calculator i figured that i could not live on the salary they would offer for my location. school costs alone for each child are as much as i pay for rent.
if i am working for gitlab in my home country, i can't move into the home country of my wife because my salary would be reduced below what we need to live there
why should i not be able to demand that?
this hinges in whether we want people to mix and integrate. if we do want that, then we do need to support people who make that choice. if we don't want to support that, well then we just won't get people who are willing to make that move.
so if gitlab is not paying a salary that allows me to do that, then gitlab is missing out on the benefits of the mix and integration that i am achieving. it's quite simple.
it's fine if gitlab (or any other company for that matter) does not want that. that's their choice. but they also have to accept that the consequence for the company is to not get those benefits. you get what you pay for.
You just need to ask yourself if offering incentives to get your employees to move to hot job markets is in your benefit.
Kids would have to leave school and all their friends behind, wife hasn’t finished education here - there’s those reasons as well.
No. You got it all wrong.
You're asking for equal pay for equal work. If all your team members are paid in full but you, in spite of doing the exact same work, are paid a fraction of what they are paid, then something is terribly off.
Your contribution to your team does not depend on where you're currently located. How much time you waste on commute does not change the expectation placed on your output. If your office location is prohibitively expensive that means your company needs to sort their mess and work on a location that's more affordable. It makes no sense that you need to subsidize your employer's bad office location.
So you want to earn 200,000 USD. Whilst living in a part of the world where that effectively places you in the top 1% of earners.
You're only looking at this from that angle. What about the person who's living in SF, and scrapes by paycheck to paycheck. Whilst you live in a much cheaper area to live in, and you have a very different financial situation. Your both paid the same wage.. because "fairness".
You might say, "JUST MOVE". But do you think that's a fair thing to tell someone?
Sure people move to places like SF, but people do that for more than money, they do it for a variety of reasons. And one of them is that there's a lot of demand in that area for your talents. So if you wanted to change jobs, grow your career, etc.. you can. But that area has a higher cost of living.
You sacrifice some of those things when you move to the small country town, there's no tech hub. No meetups, nothing.
So do you think it's still fair to say to that person "Hey I know you're living hard... But at least you can spend that little money you have left on public transport getting to a meetup!". Whilst the person living in the middle of nowhere can afford first class.
1. its not fair you get to live like a king! -> would you move here to live like "a king"? Oh you don't want to deal with the pollution and the bureaucracy and lack of safety. Ok. Ah so there is a cost I am paying by living in a bad country.
2. paying you SF salaries would be unfair to people living around you -> ok so you would be ok paying that extra amount directly to a charity right? Oh ok you aren't.
I'm saying it's only fair for a employee to think they should be paid proportional to the value delivered. Companies exist to make more money. This causes a clash. In my last company the CEO specifically wouldn't hire staff engineers from cheap countries because he didn't want to give the other engineers the idea that they too can command higher wages.
Just pay people the same amount and let them decide how to live their lives. They are adults.
I don't understand what point you think you're making. My disposable income is not my employer's business, and I definitely do not live below my means to subsidize my employer's business.
You wanted me to do my work in exchange for my salary. Pay me. Don't think for a minute you are entitled to go through my grocery bill to see if you impose pay cuts.
Equal pay for equal work doesn't mean the same pay. It means that an employee from Bay Area, one from Netherlands and one from India are able to buy the same amount of goods on their local markets from their wage.
Of course a Bay Area employee needs to be paid more, their rent is insanely higher than the rent of employees in the NL
This kind of detail is where it falls down. Should people be able to purchase the same goods, or the same relative local position in society?
https://www.ktvu.com/news/in-bay-area-0-of-homes-were-cheape...
You're literally arguing that every area has different economics. But also failing to also understand that every area has different economics.
iPhones don't keep you warm at night.
The argument that local cost of living is the only factor falls apart when people are moving from high CoL markets to low CoL markets with their advantaged savings. Especially within a nation, or times of large movement (again, pandemic - where folks with higher CoL wages were better positioned to acquire prime real estate in lower CoL markets).
You are paid in dollars (or equivalent). So they don't get to claim to be paying the basket of goods if they won't automatically match inflation either
Yes it does. You want me to deliver an output, not to slack off if I live X or Z. If you value my work, you pay me. You don't get to claim a benefit just because I chose to live somewhere cheap. Otherwise it would make sense to pay me millions if I chose to move to a penthouse in Manhattan.
If the candidate had other better offers, then they can either reject the offer or propose a higher counter-offer. If the company received a counter-offer and chose to accept it, then this amount becomes the new minimal amount.
Over a period of time, if there are enough counter-offers (or rejections), this continues to increase. Since, the two (or however many) locations don't necessarily have the same demand or supply, it's inevitable that some location will end up with much higher compensation than others. It's the nature of free market and why some companies engage/d in hiring collusion (see https://en.wikipedia.org/wiki/High-Tech_Employee_Antitrust_L...)
Why is that? It isn't obvious to me. It seems like an excuse companies use to pay people less, especially if the company is based in a high-paying city.
For some lucky people that does work.
If company X was paying N for a SF engineer, and suddenly it finds out that it can pay N/2 for an engineer that’s as good as the SF one (but lives, let’s say, in Mexico)… well, jackpot for company X.
Why should someone living in Indiana make less money than someone living in NYC for a remote tech job?
Both are located in the US separated by a 2 hour flight. Both are in the same timezone too. For a remote company where you have employees spread around the US, there's no difference to anything here.
There's no moral reason to satisfy "why _should_", but let's be honest. A company with equal salaries regardless of geo is more likely to cut the salaries of people in expensive places to live than raise the salaries of people in less-expensive places. This is especially true in places where both salary and fundamental necessities like health coverage are included in compensation, like the US.
Such companies then lose the employees who prefer to live in expensive places, to companies who _will_ pay them more to live in more expensive places.
Whether accepting that result produces a more efficient staff for a company is the more indicative question IMO. Or asking why some places are so much more expensive to live in.
Because on average they'll accept less money. Someone in NYC is more likely to have other options with higher pay. If a business wants to hire them, they have to offer more. Simple as that.
If you want developer salaries to be the same in all locations for the same level of work for reasons of fairness and anti-discrimination, you should ask why salaries of developers are higher or lower than other roles. Developers get paid highly because of supply and demand. Why not advocate for all workers, regardless of role, be paid the same - this would also be fair and anti-discriminatory. It seems we want to benefit from the market economy on one side and then want fairness on the other side. The logical conclusion of this kind of reasoning is communism where all workers at a company are paid the same, regardless of role. We know how that worked out.
If all developers, regardless of location, were paid the same, companies would much rather hire all their developers in a single location. Why bother hiring in locations far away? Jobs would have never flowed out of high paying locations to lower paying locations.
The Bay Area has the highest salaries because Silicon Valley has had many years to develop and the vast majority of tech companies are based in the Bay Area. Many companies are started in the Bay Area because people who work together in one company often break off and start another company. Many people at these successful companies have become wealthy and this has driven up home prices. This has made the Bay Area one of the most expensive places to live. Other workers cannot afford to live in the Bay Area and so there is a shortage of labor. This drives up the cost of every thing, including restaurants, groceries, and personal services. If the Bay Area does not add substantial housing, it will continue to see companies move to other metros.
In the US, salaries are different based on metro and state and are based on the market economy. Companies move to offices to particular metros if they is a healthy supply of workers and supply is greater than demand so that it is more cost effective. This location competition is healthy.
The role of any government, whether it is metro, state or country, is to create thriving economies so people want to move to that location. This means investing in critical mass in particular industries so many companies in that industry want to locate there. It also means ensuring other costs are low. In the US, health care insurance cost between $24k and $36k/year for anyone with a family. This is higher than the full salary in other countries. If the US does not figure out how to solve its health care costs, it will continue to see jobs leave for lower cost locations.
With fully-remote working, the demand isn't as concentrated, so supply need not be as concentrated either.
As someone not currently living in the Bay Area, or London, or another tech hub, I'm quite happy not to be paying the cost of living there. And honestly I don't think it's fair that people who live there should be better compensated just because they decide to live somewhere expensive. But I understand why it happens, because when companies hire specifically in a tech hub there are lots of people willing to work with them but only if their pay is higher. It's a vicious circle for employers, a virtuous circle for employees, and only sustainable for as long as productivity remains above cost. It's not built on a stable economic foundation.
I don't want to accept lower pay for the same job, which may mean that I don't work for those companies. That's the market at work :). On the other hand, if a company wants to employ folk to be physically present in London, they're not going to want to employ me. While if a company is willing to pay the same to everyone, they'll get fewer people in London and more people outside London, and they might even be the same people just dropping their commute :).
There's enough global demand for Software Engineering to drag everyone up. It is universally the case that if you pay peanuts you'll get monkeys, but paying an equal wage for equal work benefits the company and wider society and I don't particularly care if that's lower than I might get if I was willing to work in London (or the Bay Area), so long as I'm not required to work in London (or the Bay Area).
despite having the least in absolute terms, it is possible to have a great life with the mid-to-high level income in IT there. can't say the same with what most people earn in europe.
IMHO - using Ruby/Rails in 2024 can be wasteful, but of-couse for the right situation it can be a good choice. (Just) For example an enterprise app where you know the number of users will be limited, or when you know the development speed is paramount or where you want to build a quick proptype to test the market out. Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
> Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
It isn’t any less tricky than a Django or Express project. With any codebase discipline and regular tending to the garden of code is important to prevent weeds from growing.
If anything that Ruby (and Rails) has going against it is still the raw language performance and higher memory requirements/usage than its counterparts, which makes it less desirable for workloads that require low latency or small memory footprint
Stripe, GitHub, and Shopify are the shining examples of Rails scaling in all kinds of ways without problem.
> Even their frontend I think is now just react
It remains to be seen how successful this is and what the reasons were for the change. Personally I don't like it, it's less reliable and slower than before.
As a former Shopify dev, I can tell you that "without problem" is highly inaccurate. We had lots of Rails scaling problems.
Scaling problems at Shopify are rarely if ever directly Rails related.
The biggest challenge was always scaling the database layer, and to some extend the deploy pipeline of the monolith given the amount of people working on it.
I would like to challenge that.
I guess it depends on how one defines "productivity:" is it spitting out code as fast as your keyboard works, or is is not having features do weirdo things because who can reasonably say what the code does at any given time?
I have more than once tried to contribute fixes to GitLab's codebase, and every time I open the thing in RubyMine it hurpdurps having no earthly idea where symbols come from or what completions are legal in any given context. I trust JetBrains analysis deeply, so if they can't deduce what's going on, then it must take an impressive amount of glucose to memorize every single surface area one needs to implement a feature. That or, hear me out, maybe "it works on my machine" is a close to correct as the language is going to get without explicit type hints as a pseudo static typing
> https://news.ycombinator.com/item?id=39159481
Bugs exist in all code written in all programming languages and you will find bugs in programs written in statically typed languages too as you know. Programming languages are rarely chosen for the absense of bugs alone though or we'd all be using SPARK Ada or something.
> spitting out code as fast as your keyboard works, or is is not having features do weirdo things
This is a straw man. No-one has suggested that "spitting out code as fast as your keyboard works" is what Rails allows you to do, or that Ruby code results in features that "do weirdo things".
In reality engineer productivity is important and Rails enables it in a web environment.
> I have more than once tried to contribute fixes to GitLab's codebase, and every time I open the thing in RubyMine it hurpdurps having no earthly idea where symbols come from or what completions are legal in any given context.
Yes, I prefer writing statically typed languages and I would /greatly/ prefer Ruby to be statically typed for a number of reasons, but it will likely never be so in a way I consider to be usable (see https://github.com/ruby/rbs). Not being able to definitively tell what a method returns or where one is defined is a total PITA, but it's one of the array of up and downsides to Ruby, with each language having a different set.
> I trust JetBrains analysis deeply, so if they can't deduce what's going on, then it must take an impressive amount of glucose to memorize every single surface area one needs to implement a feature.
You don't need to know how all of Rails works to write a Rails app, as I'm sure you know, so this seems like another mis-representation of the truth, just as you don't need to know how the compiler or CPUs work to do a lot of (most?) programming.
> That or, hear me out, maybe "it works on my machine" is a close to correct as the language is going to get without explicit type hints as a pseudo static typing
You seem to be suggesting that Ruby on Rails applications behave unpredictably on a machine to machine basis but that's not really how Ruby works in practice, or matching on my experience.
Couple of points. 1) You probably mean Ruby on Rails and not Ruby and 2) that's not true of either. Ruby or Rails "entire programming model" was not based on thoughts around I/O at all, a huge about of the driving force behind Rails was DHHs ideas about developer happiness, which is intrinsically linked with speed of development.
Discussions about I/O /do/ happen in the Rails community because it's important when you're running a web server which is a lot of the use Ruby/Rails is involved in, but it's not the sole (or even a major) focus of the decisions made for the language or framework.
> Well now, both on Node.js and JVM we have programming models which say that - while I wait for IO / Network let me do some other work or service more requests (Async / Webflux / Coroutines).
Right, but if the goal is to maximise CPU usage of your web servers then you can completely do that using Rails, and have historically been able to do so by spinning up multiple processes in the same way you'd spin up multiple threads in, say, the JVM. Not as convenient perhaps, but a model that's been in use since at least the 90s when I started out. Luckily it turns out that web requests can generally be run in isolation so IPC isn't usually an issue (the same reason why multiple physical web servers is simple, just as multiple processes is).
> IMHO - using Ruby/Rails in 2024 can be wasteful, but of-couse for the right situation it can be a good choice. (Just) For example an enterprise app where you know the number of users will be limited, or when you know the development speed is paramount or where you want to build a quick proptype to test the market out. Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
Wasteful as a generalisation is blunt. Of course it can be wasteful, but so can writing your app in rust, it just depends where the pressures are. I'd suggest that not launching is a far bigger concern to most people at an early stage, and even mid to late stages developer speed is one of the most difficult things to scale.
Personally I'm the CTO of a company that uses Rails (backend, React frontend) and developer speed is /hugely/ important to us. We're always constrained by engineering resources and the speed of Rails development continues to be a huge win for us. We do have scaling issues, but 99% of the time they're not Rails issues, they're in the database.
The article is nice and well articulated, but I'd argue exactly the opposite on this point. The author seems to confuse equality and equity. He takes the Netherlands and Bay Area as an example and don't make a compelling argument as to how and why paying differently those employees is akin to discrimination based on race.
Were you to pay the same salary in Netherlands and Bay Area, you would effectively either (i) pay absurdly high salaries to your dutch engineers, similar to what the prime minister earns or (ii) pay absurdly low salaries to your Bay Area engineers (to the point that you'd basically have none of them).
Which makes the point: why would a company pay such a premium on hiring from a single location if, in the end, it can only get the best engineers from there if it pays an even higher premium than the ridiculous salaries from the place? It doesn't make much financial sense, seems to be mostly based on "feelings" during hiring, like yours.
Silicon Valley has a local bubble economy. If overpaid engineers spend their money for overpriced health insurance, rent, mortgage or juiceros the boat stays afloat.
But if the SV companies start sending money abroad the bubble would deflate...
Most engineers seem to life in SF out of necessity, more than anything else. After all, that is where most jobs are.
I hope future generations will better understand that accumulating wealth should not be the goal of life.
Location-based pay pays people more if they're in a market where they have other options, so they don't go to those other options.
In essence, if you have location-based pay, you're incentivizing your employees to move to an area where they'll have vastly more employment options than to stay with you.
Also buying power is just not the same based on your location. A hundred dollars in the US and in the Netherlands will literally not buy the same things.
I'd hope a high performing, vastly experienced engineer in NL could aspire to that, though they might not get the same benefits or future opportunities.
[0] https://en.wikipedia.org/wiki/Prime_Minister_of_the_Netherla...
[1] https://en.m.wikipedia.org/wiki/Mark_Rutte
I'd quite like my country (UK) to be run by folk who are competent, rather than by only those who can afford to run for office without being paid for it. Yes, MPs get a lot more than the median income. But many people who are well-qualified to be MPs are also well-qualified to hold roles that pay at least as well if not more, and without needing to run for election to get the job.
Sometimes it looks like people are burning out because of extreme workloads. In these situations, I've always asked "why is the workload so extreme to begin with?" This gets you to the real source of stress.
“You're just worms in the worst part of the apple that's rotten You squirm and you turn from the right, still plottin All slimy cause you stay grimy, petty crimey”
While some have no alternative others enjoy that kind of game and are part of the problem. Best stay away from such jobs, and undermine corporate at every turn.
It’s the classic “last founder or first employee” problem. If you are an employee at a startup, have no illusions: if “they”[1] don’t give a shit about you now, you can be sure “they” will not give a shit about you later.
[1]: the company is and always be an inanimate and abstract object, your boss and colleagues will change
Choosing to not care is much harder as we have all had years of education conditioning us that we need to achieve more and to be the best.
Very much so. I enjoyed reading Coddling of the American Mind as well. Not 1:1 but related to the conditioning, lack of maturity, and in some cases, learned helplessness.
I moved from using bitbucket, first own servers and later SaaS to gitlab.com SaaS in early 2018.
I cannot say anything else than gitlab performance and UI being much better. I was permanently complaining about bitbucket, but very happy with gitlab most of the time. They had rather frequent partial outages, but resolution was always quick.
Maybe starting 1 year ago I noticed gitlab.com reponse times going down. It's not that things hang or are super slow, but they seem to consistently take 2 seconds more than what they used to. (The number is a complete guess, I have no measurements either before or after.) It's still usable, but no longer the pleasant experience it used to be.
Being able to work from where I happen to live and I have thought about whether I would like to work for them.
That it's a good product would certainly be a motivation. However, frequent incidents being resolved fast must be incredibly stressful for the staff. I prefer to stay where we make heavy releases only a few times a year for regulatory reasons. (Even though stress levels can raise until the last MR for the release has been merged.)
I think people mistake “manager” as being people without tech skills. I think a tech company should have managers to create alignment and those managers should spend part of their time “managing” and part of their time still honing their tech skills.
Of course clueless managers and extreme bureaucracy will destroy a company, but I very often see engineers that just want to do X and ignore everything around it as if they were working in a isolated environment. It's often the managers calling these engineers back to reality, which creates tension.
On the other hand, there are plenty of managers that do not know how to manage people, that do not even know what is expected of them. Should they optimize for cost? For productivity? For employee happiness?
Competent managers are a minority. They tend to rise quickly, leaving the inept ones to run the day-to-day.
I've found this to be one of the most important metrics (if not the most important) for maintaining developer velocity as a startup scales.
Context switching is one of the most expensive operations in a developer's day-to-day work, and also one of the most ignored.
Managers love to build team schedules around their personal schedules, but this is often disruptive to ICs' "maker schedules."
Many orgs identify this and focus on the scheduling cost w.r.t. context switching, but the build-to-deploy time is equally (if not more) important for developer ICs.
Companies aren't gods charged with dispensing fair wages to us mortals. They are participants in the capitalist market.
And in that market, all prices are a matter of negotiation.
In any transaction, the buyer wants to pay as little as possible, and the seller wants to earn as much as possible. This isn't evil or greedy. It's a basic incentive that comes from the fact that people/companies/literally everyone prefers to have more money than less money.
Thus, your company wants to pay you as little money as possible, and you want them to pay you as much as possible. The same is true when the roles are reversed and you're the one in the buyer position. For example, when you're shopping, you want to pay as low a price as possible, and the seller wants to charge as high a price as possible.
Again, all prices are a matter of negotiation..
Power in a negotiation comes down to your BATNA, i.e. your best alternative if negotiation fails. Put simply, whoever cares the least has the most power.
In a capitalist market, that BATNA for the buyer is the next cheapest option. The next furniture store that's selling this couch for cheaper. The next adequately qualified software engineer in the Bay Area who's selling their services for cheaper. And the BATNA for the seller is obviously the next most generous spender. The customer who's willing to pay a little more. The company that's willing to pay higher salaries.
If it sounds like I'm conflating two concepts by comparing customers with employers, and businesses with employees, it's because you've been brainwashed into thinking these things are different. They're not.
When money changes hands, whoever receives it is the business. Whoever pays it is the customer.
That means, in the relationship between you and your employer, you are the business selling a service, and they are a customer paying for your service. Your salary is just a price. Your resume was an advertisement. Posting it everywhere was marketing. Interviewing was your sales pitch. As an employee, you are a business. (Someone just came along and renamed all the terms to confuse you.)
Which raises the question: Why on earth would your customer voluntarily pay a higher price than they need to? If all the stores in your area are cheap, why would they pay you more for your services than your neighbor is asking for?
When you travel to a country cheaper than yours, do you pay every restaurant, store, and taxi driver the same way you would at home?
No, you don't. Because that would be foolish. You pay local rates, which are less.
That's the same thing companies are doing when they pay lower salaries in cheaper areas.
> When you travel to a country cheaper than yours, do you pay every restaurant, store, and taxi driver the same way you would at home?
> No, you don't. Because that would be foolish. You pay local rates, which are less.
But if you could magically teleport to store or restaurant anywhere in the world, would you pay high prices to go to SV restaurant or store? No, that would also be foolish.
So local prices make sense of physical businesses because people pay premium for convenient access, for not needing to fly half-way across the world to do your groceries or whatever. But that same does not apply to full-remote developers; the work(/value) is getting delivered to the business all the same regardless where the employee happens to be physically sitting.
Is it about language and timezone barriers? No, because there are plenty of alternatives in the US as well.
Beyond that, this is not just about sv vs rest of world, but about doing per-location adjustment based on local economic conditions. And it is pretty difficult to argue that the value of a developer to business is in anyway linked to their cost of living
Heck, even if you start a project within a company and it gets successful, the company can just slot in people above you, so you become employee #n in a project that you started, and then these people can say you underperform and so on.
- Organising the work and steering it in the right direction
- Ensuring that people work well together, help them grow, deal with "people problems"
If and when both of the above is achieved without a person holding the title "Manager", you don't need them.
This can be achieved by hiring 51%ers for example [1] and by actively monitoring the health of your organisation.
[1] https://news.ycombinator.com/item?id=39333921
[1-1] https://www.amazon.com/Setting-Table-Transforming-Hospitalit...
[*] YMMV: the hardest problem in any organisation is the "people" aspect, there's no silver bullet.
EDIT: Added the link to my other comment about 51%ers
> To me, a 51 percenter has five core emotional skills. I’ve learned that we need to hire employees with these skills if we’re to be champions at the team sport of hospitality.They are:
1. Optimistic warmth (genuine kindness, thoughtfulness, and a sense that the glass is always at least half full)
2. Intelligence (not just “smarts” but rather an insatiable curiosity to learn for the sake of learning)
3. Work ethic (a natural tendency to do something as well as it can possibly be done)
4. Empathy (an awareness of, care for, and connection to how others feel and how your actions make others feel)
5. Self-awareness and integrity (an understanding of what makes you tick and a natural inclination to be accountable for doing the right thing with honesty and superb judgment)
[1] https://www.amazon.com/Setting-Table-Transforming-Hospitalit...
This description helped me put words on the type of people I enjoy working with
That's why I love HN so much: a helpful answer with a source to boot!
Someone "whose skills are divided 51-49 between emotional hospitality and technical excellence" [1]. Seems quite bizarre to me to define it so precisely. Even if skills were measurable in such a way, how many people will be exactly 51% emotional hospitality, and why is 52% or 50% not suitable?
[1] https://www.nrn.com/corporate/meyer-51-percenters-have-five-...
Many of those books are selling well because they are well written and say exactly what reader thinks might work, but if you ask anyone else who worked with the author, the reality can be quite different.
I do not have experience running it at a company level, but at a team level (I have been an engineering lead in two companies for the last 6 years).
From all the books I've read (I read a lot), this is the one that was most "spot-on" about treating other humans and making them feel valued and therefore building a team with strong bonds.
> Many of those books are selling well because they are well written and say exactly what reader thinks might work, but if you ask anyone else who worked with the author, the reality can be quite different.
Absolutely agree.
In my experience I resonate most with any books when I have already, unbeknownst to me, been applying what they preach (which has been the case with Setting the table that I'm currently in the process of finishing).
I believe that it requires a lot of introspection to be able to apply new knowledge (ie, if you haven't thought about it or experienced it before reading about it)
EDIT: formatting
https://news.ycombinator.com/item?id=39333921
So what is his job then?
NIMS, the National Incident Management System, talks of ICs having between 3-7 direct reports, when there is a need to be connected to what they are doing, because beyond that, you can't reconcile things easily.
With authority comes responsibility for your actions. Without responsibility, no authority. The product manager is a manager in name only, and product owner even less so.
That doesn't mean you can't have several direct reports. The classic matrix organization for example. But it means semi-managers without real responsibility have no real mandate for doing a good job at the slightest hint of trouble.
If there's a conflict of interest, it needs to be discussed based on merit, not based on who has the bigger authority.
If there's no agreement, it needs to be escalated to somebody who has the authority (manager). But IME this doesn't happen very often.
I like this model, because the default position is that none of the engineering, product, process is the "master", so you need to negotiate. If one of the roles also has reporting authority, that automatically skews the decision making towards yielding to them.
That list of “skills” is spot on. I also especially like his use of the term “skunking” to describe how somebody’s personal opinions/problems/issues impact the rest of the team. “51%ers” are exactly the kind of people I want to work with.
People managers do the performance evaluations and various HR administrative tasks (signing time cards, hiring, firing, etc.) but they rely on feedback from their group which are both individual contributors and project managers.
Project managers lead the projects and have to select/attract the right combination of individual contributors to their project if they want it to succeed.
A project that 'gets more management' will usually have to justify the addition of PMs from a cost-benefit perspective. And a project that is overburdened with management types will usually see the ICs migrate to other projects in order to improve their impact.
All this happens organically, so individual contributors are empowered instead of being disenfranchised through organizational changes.
>> If it's something else or they don't have power over projects then they are encouraged to play constant politics.
Why would they need to play constant politics if they don't have power over projects? Not everyone is motivated by the same things.
They do have power over the projects. Being able to PIP someone is power over everything that person does including which projects they work on. Including which projects no one works on. Except it's not their direct power which means to leverage it they need to play politics. Adding layers doesn't remove that power but simply increases the amount of politics they play to make up for it.
But there is no good reason for the People manager to care anything about what projects have people working on them. If they start to care about which projects are successful instead of all projects are successful then they're not a good fit for the job. And yes I have experienced that, as well as it's opposite.
no, of course, there's a lot of value in providing escalation/descalation/rehoming processes, and dedicated ways for org-wide feedback on people's and projects' impact, but people are not just three orthonormal roles on top of each other in a trenchcoat, if there's not clear hierarchy then - as others pointed out - the informal chaos takes over (because it's the human default)
Words to live by.
sorry, this seems ... too vague for me to understand what's going on, and how it connects to their greed.
are they trying to 'scale up' this project? why aren't they keeping you? replace you how? in what capacity? are you currently wearing too many hats according to them? can you please give some details?
You’ll be telling, showing, raw stats, massaged stats, appeals to authority, what have you, but if the only thing the directors care about is features, it’s all for nothing.
After 3 years it finally became a problem because they noticed that releases kept failing, but I don’t think it had anything to do with me so much as someone typing ‘how to stop my releases failing’ into Google.
I don’t know how to solve that, as I’m going through the same thing again now (with a different subject).
The trouble with these stories is that they’re n=1 anecdotes and we only get one side of the story.
There’s an implicit claim in many comments here that we need to assume that the employee was actually a higher performer, didn’t need managers, didn’t deserve a PEP and so on. That’s understandable given that we tend to put ourselves in the shoes of the person writing a piece and being anti-corporate is always popular on HN.
However, those aren’t safe assumptions in cases like this. I’ve worked at a couple early stage startups that acquired early employees who couldn’t (or wouldn’t) grow with their role and the company. It’s common to keep early employees around out of respect for their past work, a belief that they hold difficult to replace historical knowledge, or simply because they’re well connected to founders and other early employees who have grown into leadership positions.
But in reality, simply being an early employee and being involved with early important projects doesn’t necessarily mean that person is the best person for the job or even a good fit for continuing to do it. Some of the early stage employees I worked with were great at cobbling something together from scraps and keeping it functional with a collection of cron jobs and manual interventions, but their operating style doesn’t work at a bigger company at all. If they can’t adapt and change or they become disgruntled about having to work on things the way you have to at a bigger company, they start to become more detriment than help. That’s just one example of many different potential failure modes of early employees as companies grow that we don’t like to talk about.
> I wanna say this story is pretty standard, and probably a big part of the reason why people in big companies often don't do great things.
It’s “standard” in the sense that every early company has a number of early employees who don’t grow with it, but it’s not the only or even a common fate. GitLab has plenty of employees who have been there for a long time, but you’re not hearing about them from disgruntled blog posts. Consider the selection bias when reading this.
As for the claim that they can’t do “great things”: I’ve used GitLab for a very long time and I disagree. The product continues to evolve. It’s not a stagnant product at all.
btw, that restaurant in Amsterdam is apparently the go to place for startups. I am certain I sat in one of those chairs (was this on 2nd floor facing the canal?) with another fabled startup.
GitLab is fully remote. Everyone works remote, including leadership.
I think one angle I'll nitpick though is this doesn't have to be related to whether or not the employee was a high performer or had particular leadership needs or whether they're adaptable to the current maturity level of the org. There can and likely are many other elements such as whether the manager/employee even get along, or agree on direction, strategy, problems, etc.
And the difficulty is, these types of issues can often show up as performance issues. In lots of cases, is the performance issue the employees fault or the leaders.
In my anecdotal experience where a job went from the best role I ever had to one of the worst happened over a very short week or two with the change in a manager. And I think to the point you're making, when I resigned I outright said... each of us is going to blame the other for my departure. The hard truth is the answer is likely if either of us had been different people I'd still be in that role doing work I really enjoyed, but in that particular leader/follower relationship we simply didn't get along well at all.
If I ever did a what it was like or why I left post, I'd probably have plenty of arguments for my side, because I think to your point, I'll be the hero in my own story.
Two managers in my case, both of whom were, let's say, not successful in their roles. And a glass ceiling so that I had to do most of the work _and_ be a tech lead, yet "could not manage the team because reasons".
Sudden reorgs + hire above + glass ceilings = golden ticket for horrors like this.
One tell-tale sign is endpoint security and using his own device. It's the kind of permissive cultural thing that does not scale because of compliance issues and developer productivity overhead. But it's very hard to wrench these long-time devs away from their preferred Linux distribution which requires conditional build stuff everywhere to support. Use a work computer for work, let them monitor the updates and stuff, as long as they're not using the webcam to record you who cares?
The database backup story - my guy you were on the database team. Backing up the databases is job 1. You can't just passive-voice away "oh there were no backups". But of course he's more interested in fighting about sharding architecture than actually keeping the site running day-to-day.
His big takeaway is that Gitlab didn't spend enough time on performance for their hosted offering which was a huge money loser. Just because he thinks performance stuff is fun to do, if the hosted offering is a money pit of course they're not throwing more resources at it. You have to make an actual business case for why your thing is more important and makes money more than another project. You don't just engineer in a vacuum for the fun of it.
> you need to be able to deploy your code fast, i.e. within at most an hour of pushing the changes to whatever branch/tag/thing you deploy from.
This sweeps under the rug all of the potential issues with fast deploys. I guess it depends on the product. I work on a managed database service, and one category of potential bug is that we accidentally delete or corrupt customer data. We have to be much more careful and can't just deploy what was submitted to mainline in the last hour without doing significant regression and performance testing.
But anyway essentially the main reason they give for fast deploys is:
> being able to see your changes live is nice because you actually get to see and make use of your work.
I think this is true. But, it lines up with this negative interpretation of this article. The author seems to prioritize themselves over the health of the product they're working on.
* Even in the USA the col based pay made them non competitive compared to what I could get elsewhere.
* Incidents like the database backups, and others, lead me to believe they weren't executing at a high enough level.
The same people as developers would be pursuing rewrites of rock-solid 20+ year mature software projects because there's a trendy framework.
Large organizations don't have 'startup spirit' because 'startup' companies fail. Employees of large mature orgs with 6000 employees didn't sign on to a company that's got a good chance of not existing next quarter. They're not taking massive risks and throwing halfbaked features into a brand new product with 1 client hoping to get bought by facebook or maybe an insurance company.
If those big companies are really complaining about not having startup spirit maybe they should provide an exit for the VCs and aquihire (briefly because the engineers will all leave asap) a startup!
Big organisations are stable, mostly due to the often frustrating inertia and lack of risk taking coupled with their existing, mature revenue streams.
I can also guarantee you that the work I did very much did good work instead of "cause problems". Feel free to ask anybody that worked at GitLab during the same time (or is still there) and see for yourself :)
Respectfully, in your post you literally say you got a new manager who PIPed you for being hard to work with. The whole tone of the article feels like rehashing old disagreements.
> I didn't get along well with this manager, The resulting conflict lead to a "performance enablement plan", a procedure meant to get things back on track before the need for a "performance improvement plan" (PIP). A PIP is meant to be used as a last attempt at improving the relationship between an employee, their work, and their employer.
Not getting along with someone doesn't imply or mean that a person is difficult to manage. Instead, it means there's simply friction between two people.
In this specific case, the main source of friction was that my messed up working hours resulted in me performing tasks later than expected (though still within any deadlines), though I recall those time frames not being well specified to begin with (i.e. it was more of an implicit assumption that X would be done by hour Y).
I believe I was also a little late for a meeting because I'd overslept. That's not good, but it certainly isn't a case of "Wow this person is so difficult", instead more of a case of "This person needs to get his schedule back together".
Either way, you seem to be interpreting the story in a way different from what's written down. I doubt I can change that, so I'm going to leave it at this comment.
changes bring some adjustment uncertainty, and sometimes it goes well, things get better, sometimes it gets worse. with enough time you'll draw a short stick. it sucks.
Because colleagues have told me so, and the annual employee reviews were positive as well. In fact, outside the PEP the only actionable/to-improve feedback I got was essentially "Sometimes you can be a little harsh/blunt", which is vastly different from "this person is difficult to work with".
Since I find this take harmful, let's rotate the scene: what if you have no business being in the manager's seat above that IC in the first place, and you are effectively being used as a tool by someone who is not the CEO, but, say, a newly-appointed CTO? What if you _are_ making the wrong choices? What if that IC is one of the few people in the org having the experience and the knowledge to tell you that you are? What if that IC is making the right calls and suggestions, but your implicit directive you got from the newly-appointed CTO is actually to manage that IC out "because they are such a pain for everyone"?
I have been on both ends of this situation and I do believe most scale-ups are _not_ doing a good job retaining and nurturing their early-stage staff+ ICs into their larger era. This has multiple reasons, but it does often feel like there is a lot of value to be had if just those staff+ ICs weren't so horribly mismanagemed. A hire-above-from-the-outside is prime example.
And if I sound bitter - let's just say I have experience being "that IC". It wasn't pretty, and no - I was not an asshole.
It's not really clear what you do with a staff+ long-term IC. A lot of them don't want to manage or be involved in leadership, but they want to pull down a huge salary and just do work that is frankly replaceable by a senior eng.
To be clear I do believe there are levels of IC experience above staff, but being 21 and joining a startup doesn't make you a Principal Engineer just because you hung around until you're 30. Especially if you've only worked at that one startup.
Set clear expectations in the first place. "I want to run things my way here, I'm the new management and I don't like you" is not an expectation - and that's what I have observed happen all too often.
I wonder if this is what is behind years old stale issues of gitlab, especially regarding Gitlab CI. From the outside you get the impression, that they completely lost ability to build great things, since they do not seem to care to fix years old bugs that still come up again and again.
This seems to be entirely normal for startups. I've asked about it during interviews.
I have trouble understanding this.
Do these startups consciously decide not to have backups (for time/money reasons perhaps)? It makes no sense to me.
I've worked at some startups and most senior engineers know what's necessary to keep production systems running reliably. That includes monitoring and backups.
Could it be that this is entirely normal for startup with less experienced engineers doing things for the first time?
(But I also see how the incentives of the org can erode at the energy put into cultivating management skills and prioritisation of effective management work.)
If you distribute your data carefully and don't need to do joins across shards, I fail to see how sharding doesn't help with reads.
All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes.
For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB.
For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database".
Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.
And again, what's the alternative? If you offer a successful user-generated content platform, eventually you must shard. User-generated content only grows over time, it never shrinks, and at some point you exceed the maximum amount of storage that can be attached to a single server. And long before that point, you encounter horrendous operational headaches, such as full backups taking longer than 24 hours to make (let alone restore from).
It's also conceivably possible that their largest on-prem customers would be interested in sharding their self-hosted installations, so theoretically this could be a valuable high-cost enterprise feature.
Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab.
Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.
I believe in GitLab's case that they absolutely will need to shard eventually. This is a decade-old company with a $12 billion market cap, not some early seed-stage venture with unproven success. And they're absolutely growing faster than hardware storage capacity limits are increasing.
Even aside from data size motivations, they've clearly already had blast-radius problems from using a single monolithic production database.
It's not like sharding is some major unsolvable problem in 2024. It was a lot harder 10-15+ years ago, when engineers with sharding experience were extremely difficult to find, frameworks didn't support it at all, premade solutions didn't exist, etc. But even then it was absolutely solvable, I can say from first-hand experience, even with much smaller teams than what GitLab can afford to throw at this.
To put it differently, try to think of an application that does all the things you _don't_ want to be doing if you're going to (or want to) shard. Then imagine that that's basically GitLab, coupled with the load patterns simply not justifying it.
There are countless counterexamples.
Go would be a sensible choice.
Both .NET and Java have very good frameworks and are good choices for large scale development.
Rails is nice for small to medium websites, but for large microservice based apps will present some issues.
How is that true? If a company pays $30k monthly in Bay Area and $10k in Netherlands, what should the unique, non discriminatory wage should be? $30k or $10k?
I've always been sceptical of this myself - how do you value that code review where I stopped a junior engineer's XSS getting into production? Incredible value? Or 10 minutes of work?
One part didn’t resonate with me though:
> In practice this lead to GitLab building many features over the years that just weren't useful: a serverless platform nobody asked for and that was ultimately killed off, support for managing Kubernetes clusters that didn't work for three weeks without anybody noticing, a chatops solution we had to build on top of our CI offering (thus introducing significant latency) instead of using existing solutions, or a requirements management feature that only supported creating and viewing data (not even updating or deleting); these are just a few examples from recent years.
If my company ever grows to be as successful as Gitlab, I’m going to much more worried about innovation grinding to a halt than about people trying stuff that doesn’t work, or trying stuff the wrong way. It’s so much easier to shoot an idea down than to believe in it long enough for it to have a chance. I strongly agree with the author about making engineers be the decision makers on how to build out an engineering product, but I’m not convinced that this list of failed ideas is a bad sign at all.
> In my case the amount of taxes would be so high I wouldn't be able to afford it, forcing me to wait until GitLab went public
I have seen a bunch of cases where people really want to leave the company but can't due to short exercise windows and taxes that come along with exercise.
One of the companies I know had to partner with a lender just so that their employees could exercise stocks. So now if the company IPOs you owe the lender a load compounding at 15%
Thankfully many newer companies seem to be going towards 10 year exercise windows, but they are still a minority
I'm actually a bit on the fence for the 10 year exercise window, as I'm not sure it's always a benefit. As I understand it, startup exits timelines are on an upward trend, so in 10 years will the average exit still be below 10 years, and even then are you working for a company that meets the average timeline. Then, from a taxation perspective, if you do want to exercise before the 10 year expiry, if it's cost prohibitive now, it should be bankrupting after a few more investment rounds and increases to the 409A valuation.
So I'm curious if there is a side effect here that the 10 year exercise window actually leads to many more shares becoming even more cost prohibitive to exercise right before the company reaches a liquidity event and as such not exercised (if you're in the early stage employee group).
1/ many companies have liquidity events as they mature. So you may not have to wait for IPO/exit
2/ In 10 years you often know if the companies gonna go anywhere. So if you feel the company's ngmi, might as well not exercise.
I have also heard some companies fixing their exercise price (as in its not tied to 409A) to $1 or some small value like this. That seems like the best option
One thing that I could pick up is that the author is somewhat bitter about their experience, but isn't really taking responsibility for their own contribution to getting into the situation.
At most companies that are not evil and not exceptionally dysfunctional (and I presume GitLab is neither) getting to a state where one is put on a PEP/PIP is preceded by a longer period where they had indications that they are not performing in accordance with what is expected of them. Also, at most larger companies consistently "meeting expectations" is not hard, if you're in a role you're skilled for (which it sounds like the author was) and willing to receive the explicit and implicit signals from your environment, especially the hierarchy above you, and adjust.
Being able to make these adjustments and work effectively as part of a complex environment is an important part of working at a larger company. Not taking responsibility for the need to learn how this system works and adjust accordingly is counter-productive. It sounds like the author realised that rather late. Perhaps they could have had a better experience if they figured that out earlier.