It's very hard to know what creates value without more senior leadership experience.
Sometimes it's only '1 thing out of 10' but that '1 thing' is absolutley essential.
But honestly, I suggest the entire 'big data' movement is a bit outlandish, there really isn't a need for most of it. The 'saving grace' frankly is AI, because maybe-maybe-maybe we can make use of that data in some future alg because god knows we're really not making that much of use of it.
I think the author wound have been very grateful if at least some of the seniors would have seen it as part of the leading task to craft some reward structure for their subordinates. Like that barber did. Life is so much nicer when money is for people, not the other way round.
The teams that brought value to the company were embedded alongside of the folks producing the data to be 'scienced' and had leadership that was able to understand and communicate the information (and its limits) to their leaders and adjacent teams and use their experience to point the data science folks in the right direction.
Anecdotally, it seems like companies advertising data + AI capacities are the worst offenders when it comes to buzzwords and grift. Also anecdotally, I know the founder of a small agency that does rapid prototyping of predictive models for logistics-heavy industries and adds a lot of value.
I think the problem is really one of scale - it's way harder to pin down your value add when you're part of a very large collective. Being responsible for 10% of a 10-man output is much more tangible than being responsible for 0.001% of a 10,000-man output, and the difference between 10% and 0% is similarly much easier to discern from a management and accountability perspective.
A lot of times senior leadership isn’t equipped to communicate the data requirement to achieve a task. So data people, technologists, dbas, are shooting from the hip trying to fit a square into a round hole.
Once the target is hit, most of the work to hit that target is worthless, but we’ve gotten smarter at taking the journey and creating repeatable steps, setting up data structures, etc.
I also worked at a company that had antiquated data storage practices and ugly political silos around it to the point where it was kafkaesque to access basic operating data which was actually valuable.
... and were the goddam strategy team!
I really depends so much on leadership as well.
I suggest most companies either have 'Zero IT' leadership knowledge, in which case you get the 'buy whatever Microsoft tells us to buy' - or they are tech companies where they understand it but are probably a bit overzealous.
But the same thing applies to everything.
'Legal' can be a useless waste of money, until all of a sudden they're the most important team in the company ...
The author seems to extrapolate from their experience of having a “six figure data job straight out of college clocking out at 5pm everyday producing no value” (paraphrasing) that all “data work” must be producing no societal value.
I don’t like throwing around out the phrase “privileged/out of touch” too often but this post doesn’t seem fit for the top position of HN.
Care to elaborate where the value created for millions of people is in an industry that pivoted years ago to focus on ad revenue and psychological tricks to extract money from 'whales'?
Sure, advertising is collectively a multi-trillion dollar industry propelled largely by computer/data science, but it is still only a fraction of overall economic activity driven by advancements in data analysis.
The past couple decades have seen huge advancements in safety, reduced workplace injuries, auto injuries, etc. all driven in some part by analytics.
This sentiment that “all data analysis = advertising = evil” seems very reductive. It reminds me of all the comments I see from my technical peers about how “useless” other departments are such as HR and Sales, when they’re fundamental to a healthy business which pays the salaries of programmers.
I found it resonated (although agree the title is click-baity, they just talk about their own experiences in multiple orgs).
It is hard to do good data work. Offhand it takes some combination of:
- business understanding and goals (that don't themselves come directly from data)
- using data to effectively orient those goals to the best opportunities
- using data to measure whether you are successful or not
Part of that can involve meta goals -- our data is not sufficient now to meet bullet 2, so we need to start doing something different to measure it.
I find many people in these roles act like they are just human machines producing reports. You really need to take agency in many situations IMO, and direct higher ups to look at data in the right way. If you just wait to be told what report to produce, it will not go well.
I disagree, the author has touched on something widespread and hard to articulate that I suspect a large part of the HN readers have had experience with.
Being a data scientist is what I imagine being a lawyer is for idealists who go into the profession. They think there is an underlying reality that holds bad actors to account—for attorneys, this is the institution of “The Law”—but, in fact, most of the job is helping those bad actors justify what they already wanted to do anyway.
And just as most legal disputes end in settlements, most data scientists are excess capacity, kept around because the programmers who will put up with typical dev nonsense aren’t smart enough to hit the high notes… when the fact is that said high notes only need to be hit very, very rarely in business. Being a corporate lawyer comes down to intimidation—no one wants to face off against Apple’s team of lawyers—and 99% of being a corporate data scientist is talking in maths to impress (or defraud) clients and investors.
The problem is that lots of organisations say they want to be "data driven" but then leave all the management decisions to the prejudices of individual managers as well as management as a whole within the organisation. It's a problem that's been with us since Taylorism and the dawn of "scientific management".
You end up with "we spent years and $ to get the data which says do X, but we don't feel like doing X, so we're just going to ignore it because data in and of itself has no power within the organization".
It's like buying a gym membership and not going to the gym. Having a data science department satisfies the organisation's need to believe it's self-improving.
I feel like the title is a bit clickbaity. This post is more of a rant against poor management than actual data work per se. That being said it's true that unlike other roles such as developers, artists, copywriters, etc. that still can produce tangible elements of the product under poor leadership, data will be the first victim of poor management and will have a higher probability of not producing anything at all.
Data buy-in within a company is absolutely essential for it to work.
This resonates. I've moved from data engineering into sales engineering. The variety of problems is fun, albeit the context switching with Account Execs that don't respect my calendar is... something.
A lot of my previous data work was someone saying "show me what's interesting"... "no, not that- something else"... "huh, this doesn't jive with my gut, start over".
Bringing _value_ out of data sciene / engineering is incredibly hard. You have to have the engineering skills as baseline, but I firmly believe everyone undervalues the amount of work to "sell" your analysis to less technical folks in a way that's inline with the needs of the org. It's incredibly difficult.
Long-time non-profit guy here. Totally get where the author is coming from and I'm in a very similar position in terms of where my role relates to my organisation, but I still love my job despite having exactly the same issues. Why?
Non-profits are a funny old space. In theory, they're not out there to make money, so people gravitate towards them seeking "purpose". Everything is fine once you realise that you're not going to get that, or at least in the way people come to it for.
Non-profits have ridiculous, completely unachievable goals, particularly when you put their budgets in context. Some provide truly excellent services and products (e.g., reports), most don't.
So what's so great about a non-profit tech job? I've got an interesting problem space, I love trying to work out how to measure things that are inherently difficult to measure. I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't.
I bust my arse because I'm interested in what I'm doing, but I could easily coast if I wanted to. The pay is enough for me, and my colleagues are for the most part super nice and interesting. If I want to learn something, I can make an excuse to play with it at work and run with it.
Seeking to have "impact" through a data job at an average non-profit is naive, but a lot of these jobs have stuff to recommend them beyond that.
----
Edit: slight addition to my list of joys of non-profit nerding
If you believe what the organization does furthers your purpose, then you are going to get that, although I agree that the day-to-day feeling may not be that "purposeful".
> Non-profits have ridiculous, completely unachievable goals
There a gazillion kinds of non-profits; some do, some don't.
If your non-profit is the town museum management foundation, you have a perfectly achievable goal (it's right there in the name).
If your non-profit is "the coalition to end world hunger", then maybe not so much. Then again, even such a non-profit might have the actual, stated, legally-filed goal of delivering food aid to areas in the world hit with natural disasters like floods or drought - and that's not an achievable goal in the sense of being ever done with it, but it is achievable in the sense of being able to do just that continuously.
> So what's so great about a non-profit tech job?
That there is almost never the situation of a huge pile of closed-source code developed internally which you have to live with; and no custom expensive hardware. So typically you need to both rely on, and contribute to, free software that basically anybody can use.
> Seeking to have "impact" through a data job at an average non-profit is naive
On the contrary. There is great potential of technological impact outside the organization due to what I said above; and there is potential for impact on what the organization does, because it's often more fluid, and using data, you can make convincing arguments about steering activity in a different way - appealing to your colleagues desire to serve noble purposes.
For what it's worth, I've mainly worked for "the coalition to end world hunger".
I've got a particularly fun gig at the moment, but generally tuned out of the mission a while ago.
I guess my point was that sure, as the original post suggested, this sort of work _can_ be meaningless when viewed as the author does, but there can still be lots to love in the sector.
> I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't
As someone who's now the only tech person at a small non-profit, this really really resonated with me (as did the rest of your post).
I feel it's often challenging to asses how robust/brittle something should be implemented because we don't sell (and maintain) software. A lot of "tech things" simply have a very short life span, often closely tied to project scope and duration.
I hope you don't mind me asking here, but it would be great to connect with someone in a similar position. If you're up for it, please feel free to shoot me an email, you can find the address in my profile :)
Thank you for the insight on this. It's true, I know that some non-profits that are good. Over Christmas, I had the privilege of spending time with an incredibly smart person that that set up a successful non-profit helping out healthcare workers during the pandemic. It really struck me how this person still turned up at the office day-after-day to help out, doing manual labour if necessary, even though it clearly wasn't necessary - they could bask in their early fame and the opportunities it had brought if they wanted to.
But that definitely did not seem to be the most common case, as you've said. I think maybe I might write some quick edits to make this clearer, both to myself and reader, but 'most data work' being clueless is the real issue, not 'all data work'. That is, I've moved around jobs quite aggressively, but if there's a 10% chance that any job will be okay for a while, that could take quite a few moves to find one, and you can't change over that often!
I've been in non-profit now for a few years and this rings true. We are a non-profit "business" that competes directly with for-profit companies. We try to impress upon the public that we are non-commercial, when the reality is we are just marginally less commercial because we still need revenue to operate and do normal revenue operations to get it. We just won't gate or restrict our offering no matter what and mostly avoid any commercial interference in the sanctity of our mission. And yeah, we're trying to rebuild our data systems for like the third time in 5 years and can barely even eke out a plan as to what to measure let alone what to action on. On the plus side, no one is deluded and everyone wants to do better.
I am surprised there are no private equity like entities that are driven by a single thesis which is to buy non-tech public corporations (or as far away from tech as possible: like commodities and real estate) then cancelling all 'digital transformation' or 'data engineering' contracts with consultancies like McKinsey, replicate the important 10% part of the contract in-house by recruiting a solid small tech team.
Then take the company public later at 10x the valuation, having slashed the cost by 10x without impacting operations or growth.
If you figure out how to identify those 10% and get the business on board with following that, I’d suggest skip the buying companies part and just sell your consulting hours at triple McKinsey rates.
McKinsey doesn't get brought in on the basis of their competence even though it's of course part of the image they project. If you want to go with quality work you'll have to buy your way through because the upper management doesn't have a clue about tech and you will not be able to justify your higher rates if you're a nobody.
Executives at those companies understand that risk and set up all kinds of landmines to scare private equity away.
A simple example would be, they could put the IT support department under the "VP of Crypto" instead of under the "Director of IT". So private equity comes in and fires the entire crypto department because they assume it is useless and then find out that no one can get their passwords reset or computer replaced.
More complex examples involve finance. You could transfer stock in your company to a banker as collateral for a loan with the condition that if the loan is paid off early there is a 1 billion dollar fee. Taking a company private requires that the buyer buy all the public stock.
Knowing which 10% is the important bit. The problem with these projects is that you need to extract the tacit knowledge from the lower level staff in the company, who have often been incentivised for years to keep it to themselves.
"Manage better" is not a scalable system. Although it does work for some individual takeover merchants; it's a big part of the Warren Buffet success.
This is not supposed to be a scalable model, it depends on deep domain expertise of enterprise IT and also would require some hands-on active management and most importantly recruitment. So it is not your usual private equity play "slash everything by 30%" gain short-term profit, flip it, then watch the company close down in a few years time due to morale and lack of growth strategy.
A semi-generalisable strategy to identify the 90% quick wins in consultancy contracts: look at the consultancy contract, anywhere it says Oracle, IBM, Microsoft, Data Science think about how you can replace that with an open-source stack run by a competent technology team that you will recruit and what impact scrapping all the data science stuff will have.
The crucial bit in this strategy is to offer a package to this new elite tech team that would be competitive with FAANGs, this will be completely surprising to most management of these companies as they are used to paying their tech stuff garbage when in effect they are paying 10x FAANG salaries via the daily rate of McKinsey consultants.
It would be interesting to see it tried. It's effectively "digital transformation" with the host/mutator roles reversed. Rather than have an organisation call in outsiders to do the transformation, the transforming group "eats" (buys) the transformed organisation.
The risk is definitely in the squishy unknown unknowns that are hard to quantify and therefore get obliterated early on in a digital transition.
It would take firing and replacing most of management...
And when you look at it, yes, there are plenty of funds driven by buying a company, replacing the management, and selling it. Most of them are not long-term driven (you really expect people that flip companies to be long-term driven?) and thus make all kinds of decisions that you'll probably disapprove.
I would qualify this as tech/IT management, but you will not need to touch the management team that is running the core business. In fact in most cases even a big chunk of the IT management is outsourced to third-parties.
I'm not about to praise the Musk takeover of Twitter as a good thing, but he did prove you can make massive cuts in a workforce and keep things running just fine. And he's emboldened the rest of tech to do the same.
It's not just data analysis. There's bloat all over tech.
For at least a couple of months, yes. The jury is still out as to how his actions affect the long-term sustainability of the company.
If you were a farmer you could stop buying seeds entirely and save a hell of a lot of money. And you'd be perfectly able to continue to grow and reap the current crop. If anything, productivity would increase because time spent buying and storing seeds is freed up to focus on the current crop.
You could say a lot of the work is speculative because you might be wrong about how much value it ends up producing. But if you say that 1 project in 10 pays off you just take the cost of the other 9 as the cost of the 10th.
Watching growing startups hire a junior data scientist to analyse their new mountain of data is very entertaining. One of their first findings is their revenue/growth seems to be dropping… “we need someone to focus on growth/revenue ASAP”. They hatch a plan, build a team and magically their growth starts to recover.
The issue was spotted in April/May, the building happens during the summer and is launched In September… I’ve watch this cycle happen so many times, I’ve given up trying to point out that the issue is just seasonality.
I worked with a guy who loudly pointed this out in executive meetings where marketing and analytics were presenting. Then he wouldn't let it go, loudly proclaiming the sham in the hallways and through various emails.
He was laughing at them in their faces, even, and was 100% correct. They excluded him from any further meetings.
Every time they brought up a new stat and set off alarm bells he would bring up their massive fuckup and rub it in their face, and then would dig through the data and question every point.
He was eventually fired for pointing out how incompetent our new owners were after an acquisition, and would have been a lot happier joining everyone in their sham.
Yeah, I haven't seen anyone do this firsthand, but all my social intuition tells me that you can't challenge the social reality being presented. You can yell at one manager, call his team incompetent, so on (all deeply inadvisable), but in my heart, I don't think anything would get you strategically blackballed and removed faster than saying "So... what do we all produce, exactly? Oh? You realize that's absolutely worthless, right?"
I'm going to take an educated guess that you've just turned 30 or are about to turn 30?
I had this exact same problem where I became entirely disillusioned with the output of my work after having become reasonably senior and realising that the majority of my time and effort ammounted to projects which either failed or treaded water. The unfortunate reality in business is that the projects that truly change the world are 1 in 100. They don't look or feel much different from any other at the begining but when traction really does start to take off you know you have something right. I think from your description, you need to work at a startup where you have genuine impact. This is the only way you truly "feel" meaning. You might still build something that's reasonably worthless to humanity, but you will do it your way and take your own risks.
I felt disillusioned 4-5 months in my first job after college. I could see the project I was working would take a long time to create something of value (3-5 years) but we were always put under pressure to show some stupid demos to management every quarter.
So at the end we were just doing these demos and not working on actual things that we needed to do. The project was important, if implemented successfully it would cut jobs of ~60 people in the organization. So management kept on adding more people until they realized that there was no real progress and they dumped the project and we got reassigned.
I realized that 1 year of my work as well as ~15 other engineers just went to waste.
Working in the ML/data space and have been fortunate to have largely steered clear of this problem so far. My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment.
Jobs like this exist, though you may have to take a pay cut compared to the bullshit. My first job after finishing my PhD was as a staff scientist in an academic lab using ML engineering + data science to support scientific research. There seems to be a fair number of grant supported jobs like this and pay isn’t terrible. Just under or just at 6 figures. You can make more in industry, but scientific work feels very meaningful.
Now I’m working in credit risk modeling, something I never expected to be doing, but so far it’s been a good fit. The models are applied directly in decision making for the business, and there’s a real incentive to get everything right because mistakes could harm real people’s lives. The team I’m on is strong and ethically sound and I feel good about what I’m doing.
For anyone in the data space who’s despairing about the state of the industry, non-bullshit jobs do exist, you just have to look for them, and use your judgment when scoping out new roles.
I’ve been doing “big data” in the retail space for almost a decade now and it’s pretty clear to me how it affects the bottom line. It really can be as simple as “we need more of this milk” and “the spicy queso is very popular”
And if stores had meals pre-shopped in bags you could just remove unwanted items from. Grocery stores seem ripe for simple optimizations even without big data
Grocery stores would be better served by not being giant chains with optimization problems, at least in North America.
Better zoning laws that allow mixed-use zoning would enable more, smaller grocers embedded in neighbourhoods. Personalized service would be much easier on a smaller scale. And you wouldn't have to travel far to pick it up.
I’d beg to differ. The current system optimizes for large grocery trips for bulk items that require long shelf stability.
Most people would agree that produce should make up the bulk of ones diet, yet it’s pretty plain to see that the footprint of the produce section doesn’t scale with the size of the store.
Produce is something best shopped for frequently, and is something I’d personally be willing to pay a higher upfront cost for the convenience of closer smaller stores where I can get in and out quickly. Because while we may pay a lower sticker price, the elephant in the room with the massive hidden cost that we pay but rarely account for: food waste.
You're also likely someone with a higher than average income, and general closeness to potential store locations. If you were poorer, or had to drive farther it's less likely you'd use those as optimization points.
Also fresh food is great for the diet and health, but when some even occurs that interrupts things like distribution of fuel, food, or power just having it around may lead you to getting hungry really quickly. This is something you have to look at on both a personal and national level. Food security can quickly lead to destabilization.
Conversely buying in bulk implies having enough room in your house to store food items.
It’s not like the typical Costco shopper is low income. Our grocery system is optimized for moderately well to do suburbanites with SUVs and large houses. Low income people tend to live in “food deserts”.
Consider the experience of shopping for food in America if you’re dependent on public transit, which a much larger proportion of the low income population are. Requiring a car to get groceries is extremely expensive.
Aside from specialty shops, I'm not sure there's any great virtue in smaller markets, often with higher prices and less selection, other than there are probably more of them you can walk to if you live in a denser urban area. And my observation at least in the UK is that most of those smaller grocers are giant chains like Tesco and Sainsbury.
Not smaller markets, just operating under a large one in a more decentralized fashion. Small grocers sourcing and selling local-first will of course be more responsive to local needs and more resilient against large supply chain shocks. Not to mention cut out a lot of traffic on trucking and shipping lanes.
"... this is just the rise of this new general store, and how all specialty stores look the same now, so none of them are actually really that special."
Being the giant chain is the optimization. Grocery stores have razor thin margins; operating at scale is the only way to keep prices reasonable.
This is also one of the few domains where people are very price sensitive, and they regularly see and think about costs. I couldn't tell you what my local automotive guy charges for an oil change, but I can absolutely tell you the price of flour and cheese.
No one is going to pay $10 for a single apple, and if you raise prices hard enough people are going to start gardening.
It's ironic that low food prices are not for ensuring equitable access to food.
Update: Go buy an apple in a food desert and I guarantee that if they have one you will pay more than a dollar for one. Grocery chains are optimized for maximizing profit not distribution.
My household would really like it if Amazon foods merely provide the same list we bought form last time to adjust for this delivery — we absolutely hate having to start from scratch every time (and that's not even mentioning their issues with substitution).
Heck, the local wine & beer outlet lets me just open up the last purchase list and buy that again (and adjust the quantities if desired).
WTF is wrong when a local store can get it so right, yet Amazon, with it's emphasis on "always be hungry like a startup", so totally forks it up?
Substitution was the real issue I had with online grocery delivery when I last used it 15 years ago. Things would get substituted that I didn't really think were equivalent and essential ingredients for some meal would be left off the order.
It was "OK" at the time; I was on crutches and could go to the store but not easily do a full grocery shopping. But I haven't done online grocery since.
I'm at least somewhat skeptical. The face of the last data warehousing fad in the 90s was around things like optimizing retail through things like putting diapers and beer [1] close together. But pretty much, even when there was some correlation, stores never did much about it.
I can imagine that one reason is that you also need customers to be able to find things, which is difficult when not sorted into some meaningful categories. That said, I think at the Target near me the beer and the diapers are relatively close.
Actually you want customers to not be able to find things, so they have to wander around, and 'find' other things (in large displays for example) that you want them to buy. If the beer and diapers are always together then they won't discover the new high-margin sausage rolls. So what's better for the customer from a correlated data perspective may not be better for the business from a maximizing profit perspective.
Taken to the extreme though, I'm going to be taking up employee time asking where something is and eventually stop shopping at a store if it's too frustrating.
a) most people don't take things to extremes;
b) employee time is not a problem, as they're there anyway;
c) if all grocery stores act like this (and they do), you have to keep shopping there anyway (and they know this).
So what will actually happen, with you and me and everyone else, is that we'll grumble about how they moved the cheese again, we'll wander around the store until we find it, and we'll be exposed to some more products and implicit advertising in our search, and we'll forget about it as soon as we exit the store. Over time the store will increase its profits 1.3%, and the competing store "where everything is always in the same place as last time" struggles because their prices aren't maximally optimized with the same grocery store software everyone else uses.
Eh. I have several grocery stores--and grocery departments--that I more or less favor for various reasons depending on what I'm buying. If one of them makes my life less pleasant whether because they're always moving stuff around, they overly rely on self-service, they tend to be crowded, or whatever--I'll probably go elsewhere. I do consider pricing but I don't really comparison shop so that only loosely factors in.
> you want customers to not be able to find things, so they have to wander around, and 'find' other thing
This seems to be the Loblaw/NoFrills model in Toronto. They relocate products with alarming frequency.
Want milk? That's in the dairy section. Want organic milk? That's in the health food section at the opposite end of the aircraft hangar sized store. Nuts? Either in the snack foods, health foods, or baking ingredients depending on unknown critera - one literally has to check all the places.
That's not quite what the article says. Although it doesn't explicitly say so, there was apparently a (possibly valid) correlation. But Osco didn't act on it, instead removing a bunch of slow-moving SKUs instead.
Though that also makes the point that making things easier for customers to find is probably more important than any minor co-purchase optimizations.
I think there have been more recent fads. (If fad is the word for it.) There was the story about Walmart stocking up on Pop-Tarts during hurricane season that was in the news fifteen or twenty years ago.
There was the original "Big Data" and Hadoop, Chris Anderson's "The End of Theory" and Clive Humby coining "data is the new oil" around the same time. Probably others.
I believe you, because the store is always out of the most popular products, and nobody ever seems to figure out that they could simply shift production towards the more popular products and make more money?
On a local basis I can go to two of the same store a few miles apart and one will out of products X, Y, and Z, but the other store may be out of X, T, and R. Localized buying trends can have significant differences.
Also, there may be many other effects here. For example, if a popular product is actually popular and the pipeline to make the product is months long, well there's going to be outages. Shifting production generally isn't easy and trends pass quickly.
In addition, perceived popularity can be used to manipulate consumer pricing. "X is always out. Oh look X is in stock now, I should buy it for 50% more"
Yes, I am sure. I go to grocery stores in multiple locations and they are all much more likely to be out of a particular version of a product than other versions. It has been this way for years. They do not price this version differently than the other versions. And this is true for multiple products even in different kinds of stores.
I think some suppliers just aren't that good at adapting supply to demand.
This is funny to me. In the old days, this was called: cost accounting to determine the profit of any particular item you sell and market analysis to see how many of which things your customers are likely to buy. Join the two results to optimize your profits. It's not rocket (/data) science.
Actually neither of these is particularly easy to do for unorganized companies.
I really like your heuristic, and I'll add to your list of examples where one could work
- Any tech company where the Stats/ML model is one of THE products and differentiators. You'll need to cut through a lot of buzz word and sales-speak to find the good ones.
- Banks and other financial institutions where making uneducated guess is a big no-no when it comes risk, pricing and anything related to financial products. Your example of Credit Risk Modeling is a classic example and very interesting problem.
- As much as consultancy gets a bad rep due to some shady practice from big players, there exists a solid demand for professionals that know Statistics in the Large Construction Projects space. Let's say modeling demand and return financial for a project, proving environmental impact, preparing/implementing/analyzing unbiased surveys in the area, and so on.
- Government agencies where data is one of the Key Outputs. Such as the Census, Bureau of Labor Statistics, CDC, and so many others.
As you said, sometimes it will mean a pay cut, especially if you want to remain in the technical work and not deal with the business and managerial side of things. But there's solid demand.
+1 for government. I work with state and local governments a fair bit, and while there’s a lot of red tape and weird politics, the overwhelming majority of clients I work with are smart, capable, highly mission driven people doing their best in a system designed to move slowly. Being able to look back on a project and see a positive impact in a community is way more rewarding than trying to move the needle on click through. And the pay cut for public sector consulting isn’t as much as one would think. Especially for federal.
Yes, that's a common approach. You can take a dataset of consumer features at the point in time when loans were opened along with information about the loan outcomes and try to predict the loan outcomes. You can't just take a kitchen sink approach with the features though because there are regulations that demand a level of explainability. To get a sense of the basics, I think the book Intelligent Credit Scoring by Naeem Siddiqi [0] is very good.
i would also like to add that modelling in credit risk is not just about yes / no answers around loan outcomes.
There are lots of other goals that are regularly modelled such as default rates, profit optimization, loss minimization, delinquency and payoff rates at specific parameters ... endless options
There are also lot of different ways in which these models are implemented ( decision trees, statistical analysis, ML... )
Some examples of (real life) projects include:
- if our institution offers this card to clients with 750 credit scores vs 790 credit scores, how does my profit move vs my losses and what the factors to limit losses while maximizing profits
- how do I minimize my costs for servicing this card while keeping profits at the max ?
- what rewards options lead to the highest number of preselected / qualifying clients taking up a product at the lowest cost
- what contact strategies are best for specific types of clients if they are late on payments - call or email or text or legal letter? which strategies are the cheapest? which strategies give what this institution considers to be the best response ? which lead to fastest full payment? fastest partial payment? which lead to getting back to a regular payment plan?
- how can we identify clients who have a lending product with us who might be on the market for another lending product in the next 12 months? in the 6 months?, those who might need a limit increase pro-actively? those who whose might need a limit decrease pro-actively?
And, one of the largest area pf credit risk evaluation is real time decisioning on transactions: 'is throwaway201606 really buying $6000 of apple products, in person, at this mall in Toronto, Canada right now when I (the system) know I he bought a daily Wendy's Spicy chicken sandwich 10 minutes ago in Dallas" and should we allow this payment
Some example of how models are used here include ( note that modelling helps establish which transactions to look at more carefully and which to ban outright among other things )
- predict what type of terminals are being targeted: scammers -> we have left bank ATM machines alone and started looked at gas pumps:
- predict where transactions of interest might come from: scammers -> we do scams on site A at Christmas, scams on site B in the summer or we do site A scams with brand Y card and do site B scams with brand Z card
- predict behaviour patterns of transactions of interest: we always test the cards we will use by purchasing a $5 'brand x' gift card online 10 minutes before
Yes, completely agree, I omitted the part about not just modeling binary outcomes for the sake of brevity. I was even considering linking the article Capital One: Exploiting an Information-Based Strategy [0], showing that one of Capital One's innovations was successfully modeling more complicated outcomes like profitability.
Yeah, I was basically filtering out jobs applications by whether or not I could see what effect ML was supposed to have on the functioning of the business, and importantly, whether or not they actually had access to the data to make it work.
Ended up at a place where ML doesn't just make things better, but is necessary to make things work at scale.
There were a lot of job ads I looked at that just seemed dreadful to me though. So many recruiting companies wanted ML Engineers/Data scientists. I could kinda see how it would work, but didn't think I would enjoy it.
Focusing on jobs where your output makes an immediate impact is an incredibly smart move to make sure you matter. Of course, most roles matter but there are too few managers who know how to vouch and articulate a team’s business value properly.
I actually found myself working on a credit risk modelling project on the capital markets side and it’s been great as well.
This is actually the most important career advice for tech jobs.
First junior dev job it's not as important because you just need to find something. After that you will want to be very careful to make sure the job has actual business impact, otherwise you are likely to end up in some kind of bullshit vanity project that only has to appear to work.
If you're a baseball player, you don't want to work for a football team. You'll be undervalued and hate your job.
Less metaphorically, I've always looked for jobs where my skills are aligned with the "company mission" - first at a bunch of startups, and now as an academic researcher where I get to define that mission.
I find it strange that AI/ML people would avoid "adtech for ethical reasons".
I really wish the websites I visited made better use of my actual history on those sites to tailor relevant ads to my interests. It seems like an ideal application of AI to me. They could do a far better job than serving me the lowest common denominator stuff they keep throwing at me.
My Twitter news feed these days:
* 10% - posts of interest from people I follow, i.e. the stuff I actually go to Twitter for
* 90% - Ads and recommendations of topics to follow that I have zero interest in
If it's going to insist on showing ads and recommending topics of interest, you'd think those could be better personalized, given that Twitter has years of my tweet, reply, and like history to train its AI on.
But no... what I get is crypto ads, Hollywood events, celebrity news, sports news, etc.
You understand that you're not the one paying for the service, so you don't get a say, right? Even once The Machine has perfect knowledge of you, the ads will not be tailored to your preferences. It will be what an advertiser has paid The Machine to show you.
> what an advertiser has paid The Machine to show you.
As Cory Doctorow has point out so eloquently in his "enshittification" series, the end point isn't even to the benefit of the advertiser. At the end state, The Machine also have perfect knowledge of the advertiser, and the adtech company can turn the full power of the The Machine to the benefit of itself, extracting value from both the audience and the advertisers.
The problem is that there are a loooot of ethical implication on using your own personal data in the first place, where that goes, who has access to it, how is it handled, and so on and so on. Then advertisements isn't anything but propaganda, which has its own set of implications. And then finally we have the ever present pressure to push more and more ads, thereby making the internet in general worse and worse, so the very field of ads is in itself unethical, as it is destroying the virtual environments we are building.
Also ads != recommendations. In a sense after a while these two are also at odds with each other. Cause there is again, the ever present need to sell you more stuff.
It's almost impossible to be competitive whilst remaining ethical since being unethical provides a massive advantage. In the drive to optimize profits, any hard advantage soon becomes table stakes.
Not to argue for OP, but I think it’s more of a slippery slope thing. A friend of mine does data science at an online bet site but doing more of the number crunching; and from what I gather, the advertising side get close to pandering to gambling addicts
Edit:ignore, op posted. I should stop jumping to answer so soon!
I find it strange that anyone would want companies to accurately model their behaviour and desires so that they might influence you to behave in a way that benefits that company.
The rare time an ad slips past my blocker, (and rarer still when I stop to think about them) I take solace whenever they are irrelevant.
> I find it strange that anyone would want companies to accurately model their behaviour and desires so that they might influence you to behave in a way that benefits that company.
If the companies in question are billing the ads by impression, as is common, they get paid whether the ad served is relevant or not. If I'm going to be served up native content or ads anyway, I far prefer they be relevant to my interests than they're not.
I'm failing to see how this addresses the concern of corporate entities trying to effectively shape my behaviour, with no regard for my well being, to favour their interests.
I'm not sure what particular interests you have in mind, but if I visit a site I don't have to pay for, subsidized by advertising, and that shows me content and ads relevant and tailored to my interests, I consider my own interests pretty well served.
> I far prefer they be relevant to my interests than they're not.
These companies are trying to extract my attention and money -- both limited resources -- and if I have to see ads better ones that I have no interest in, simply out of spite.
>I find it strange that AI/ML people would avoid "adtech for ethical reasons".
You founded and run an advertising company. Are you taking the piss? Surely you've aware of the ethical issues even if you don't give a fuck about them.
Or are you just saying you categorically expect AI/ML people's interest in the tech and/or money to override the ethics?
>Perhaps we can start from the points I'm making and assess them on their own merits instead?
As far as I can tell, the points you made are:
A) It's strange people make moral choices to stay out of advertising.
B) Advertisement could be more targeted.
B is boring and I don't care.
A isn't really a point with merits to be debated.
I enjoy eating meat. But I don't call it "strange" everytime someone turns out to be vegetarian. I understand why they refrain from eating meat. It's not like I disavow the existence of the question of "should we eat meat?". I just disagree.
Sounds pretty extreme. Telling anyone about a product you've built is really a form of advertising. I don't see how society advances if your viewpoint were to be implemented and taken to its rational conclusion.
When I was starting out my career, my father gave me a piece of advice that has turned out to be incredibly applicable and valuable - "Stay close to the money". What he meant by that is work in environments where the work you perform is very closely tied to the revenue of the company - or another way of saying it - "Do work that is directly part of the organization's mission". It sounds like you've applied this to your career with good results.
As a non-ad tech example, a significant component to what we're trying to do at https://www.auxon.io is provide tech to companies they can use for testing & analysis of robots and cyber-physical systems. From our perspective internally what we're getting is the ability to perform something akin to "materials science" for the increasingly critical software parts of these systems and construct models of their behavior to use for predictive & comparative analysis purposes.
We have a research partnership with the University of Ottawa & University of Luxembourg specifically on the statistical analysis end of things to go deep in some areas to later incorporate the findings into our products. In fact the first go around of that research cycle has already happened (https://arxiv.org/abs/2301.13807v1) and the insights are being integrated into our Deviant product (https://auxon.io/products/deviant).
It's definitely not ad-tech. It definitely has a specific applied use case. Most of our marketplace traction is in aerospace, energy, automotive, and defense. We're not immediately hiring for roles on the data analysis end of things (we're in much more immediate need of visualization & frontend help), but we will be this year.
> My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment.
Currently starting to look for a new role and couldn’t have articulated my goal more clearly. The problem is that this narrows the field considerably, to finance/insurance/fraud (“pure money work”), healthcare/EMR (which as far as I can tell is also actually just profit optimization work for hospitals), and then you have manufacturing (closer to where I currently am), but the actual applications of data science are more limited and data Eng / analytics are more valuable, bioinformatics (which is really interesting, but also a large learning curve), and then just traditional BI (kind of generic and boring).
Also, if you’re thinking about long term career development, many of these functions fall under the IT/CIO umbrella as orgs grow, which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc. which I’m not saying is a bad thing, just an observation. Just applying statistics and modeling can only rise so high in an org, unless like you said, their core business and value prop is being the best at modeling some phenomena.
anti-fraud is very unsexy work IMO, but as long as you believe that (insert some internet-enabled service) should exist, then it can't be safe and enjoyable to use without serious fraud mitigation and prevention work, you can be secure in knowing that you're making life a lot easier for people who otherwise would have suffered from that fraud
Curious why you think antifraud is “unsexy”? I worked in the field for a few years. My experience was that the bulk of fraud is easily detected with bog standard models (most people doing fraud are lazy). But, there’s a long tail of super sophisticated work for detecting more subtle operators. It’s also pretty fun to learn about the convoluted schemes people come up with for doing fraud and to imagine how someone would attack a new service.
Also theres quite a bit of money sloshing around and it’s relatively easy to quantify financial benefit from fraud reduction, so the departments tend to be well funded
I’m finishing my bioinformatics-centric PhD and I really resonated with the author’s sentiments. Except in the case of the bioinformatics field, I feel like there’s too much data for any person to make sense out of. And if you do anything beyond a volcano plot, everybody’s eyes roll back into their head and they get bored and ask for their volcano plot or their special gene/protein/transcript of interest.
Tbh I dread my future job of maintaining R and python scripts
> which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc.
I was going to suggest: we have a bunch of data scientists working on intrusion detection problems in cybersecurity, tied to an engineering organization instead of an IT organization. On the balance of considerations, I think it has a net positive impact. And I'm pretty sure plenty of other companies have something similar. Unfortunately, we also have an ongoing hiring freeze, so I can't recommend my own organization.
I agree they do exist, and this heuristic sounds sensible to me. It's the good old patio11 "go work in a profit center" situation, and I wish I had done so. I'd probably be grappling with a new existential concern, but that's life (I wish I had a job that didn't matter so I wouldn't have to worry about performance all day!).
- energy management (shifting loads to times when energy is cheap) for consumer/commercial/industrial use cases
- energy markets, especially power trading: often highly algorithmic, and driven by models that turn fundamentals data (weather, calendar, …) into supply/demand predictions, and from there into price predictions
Why not try to do
data work in science? Analysing astronomy data, particle physics data, or DNA sequences sounds high impact, and you have world experts on the data to make sure you’re getting quality curated data.
Absolutely an option! I've got my own strategy out, but it seemed self-absorbed to go on about it. People don't want to hear about me specifically beyond whatever minimum lived experience is required to make my point. Plus it hasn't panned out yet.
Maybe replace "Data" with "Information Technology". Most people don't work in information technology, and occupations like teacher, construction, health care worker, lawyer etc. are certainly not worthless.
I have learned that sometimes it is better to wait work out and see how it plays out.
To some I'm sure it looks like laziness, but I consider it more weaponized procrastination. With experience I've become better at it.
Sometimes I get that gut feeling about a project, and just can't bring myself to get started. Some basic research happens, but I spend close to zero actual time on it, sticking to other things that matter.
In two weeks, priorities have changed and all of what I was asked for has now changed or been cancelled anyway. Glad I didn't get too invested.
> The absolutely fucked up thing is that everyone I've met in this space seems to have totally given up on doing anything meaningful at work. The goal is to get paid, not stress out, have a happy office where everyone can collect their strange handout, and not think too deeply about how unfulfilling is it to produce nothing for forty hours a week.
this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
> people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices
Private money being wasted due to inefficiencies at the office (as you say, a misallocation) is surely entirely different than a government-granted right to not produce anything yet still be supported.
At different rates though. You can absolutely choose not to pay the lazy-tax and buy from a different vendor that doesn't employ an army of people who do nothing, you can even build whatever it is you need, but you can't easily choose to not pay the taxes that the government collects for the same purposes.
Pretty sure Atlassian has a lower per-employee efficiency than what a smaller startup providing a product in the same market would have, but there might be specific sticky features in the Atlassian product that make it difficult for one to migrate. The effective market hypothesis is great at finding local maxima.
Likely, yes, but it's "do I want to work around this", not "there's no way". There's enough people getting fed up with Atlassian that "Like Atlassian X, but sane" could be a category.
There's basically no way you can start a new state if you'd want to opt out of the existing ones, and even moving from one to another is quite the undertaking. Imagine Atlassian saying "sure, you're free to stop working with and paying us, but we believe that we've been instrumental in building your company, and we believe that entitles us to 20% of your company if you want to work with someone else".
You might find it ethical to not use a superior product to try and force the world to be more like the one described by the efficient market hypothesis, but I definitely don't believe this is on the to-do list of people who get stuff done and get promoted to make impactful changes in their organization.
Even if that were true, and I don't think it is, there are millions of jobs that are subsidized with public money and the public has no meaningful choice in them. Think defense contractors, public sector consultants, etc.
"Jobs paid for by public money" and "completely unproductive jobs paid for by public money" are not the same though.
But even if it actually was millions of completely unproductive jobs, I fail to see how that would be an argument for adding a few dozen millions more and not a reason to get rid of the ones that exist and cost us money that we could otherwise invest into important things that are currently underfunded.
Because they are not jobs. They would free up people to other pursuits, in the hope at least a few of them would be useful. It would also be a massive boon to those who cannot make ends meet.
I wasn't making the argument that every government job is useless, I was simply reminding that it's not true that unproductive private jobs can be weeded out simply with competition or customer care.
> Private money being wasted due to inefficiencies at the office is surely entirely different than a government-granted right to not produce anything yet still be supported.
Why?
If you are in the camp that think companies sole purpose is shareholder profit then this is perverse.
If you are not in that camp you can probably agree that we should aspire to the best life for most people.
(As a European this comment really dissonates. I do realise that it does not for Americans)
Private money wasted usually leads to that entity dwindling away from competition, with the end result (of efficiency) that benefits society at large. Public money wasted takes a much longer time to dwindle away and the end result is usually catastrophic for society at large, i.e. a revolution.
As a European, odds are that you're used to a monoethnic country where "most people", "society", and "government" are kind of interchangeable, and people find it easier to sacrifice themselves for the greater good -- as you say, aspire to the best life for most people, even if it might mean that the decisions being made are not optimal for one's own circumstances.
I myself hail from such a country (though outside of Europe), and you are right that the American viewpoint, or that of any other large, multiethnic country, is quite different. Implementing policies that require taxpayers to support others that are culturally different and therefore harder to empathize with is an order of magnitude more difficult; and it is becoming yet more difficult in America as time goes by, due to the reduced emphasis in assimilating to the mainstream ("white") culture.
From a society point of view, it doesn't really matter "who" wastes it. Ultimately, the outcome is the same: Someone spends time on something that society does not benefit from.
... sidenote; I personally think that something completely inefficient "is not that bad". Much worse is work that is directly detrimental for society as a whole, even if it does make a few people rich. Now that is terrible.
I think this discussion rather gets to the bottom of the objection to UBI: nobody objects quite so strenuously to wasted work as they do to "idleness". In the view of these people, it is morally superior to pay people to dig holes and fill them up, or the white collar equivalent of digging holes in spreadsheets and filling them up again, than it is to just give them money with no strings attached.
The important thing is that people are prevented from enjoying themselves for at least eight hours a day.
No, not at all. The important thing is that the person making poor choices bears enough of the cost of those choices.
Otherwise, they will not grow.
True UBI would create the largest, most indolent, and most socioeconomically isolated consumption class the world has ever seen — until it collapses under its own weight.
They're not usually bearing the cost, though? In a lot of places it's better to have hundreds of unproductive staff than a few productive ones, it inflates your standing and pay. It's ultimately the investors who bear the cost, but they also deem it not worth their time to try to manage the managers.
Sorry to just jump in here, but growth isn't for everyone. If some people don't want to work, I'd rather pay them to get out of the way than risk the chance of them adding negative net value.
That’s the other major issue of UBI — it’s an ideal tool for ghettoization, if those in power decide it’s more profitable to give “undesirables” just enough that they’ll get out of the way — and stay there, quietly.
The wealth and culture gap would become nearly insurmountable for anyone otherwise capable of upward mobility out of the UBI class.
UBI would just socialize the cost which at least now, the private sector partly pays.
UBI is a yearly recurring expense in the trillions.
Lots of people would work less, (remember COVID?) lowering tax revenue.
This means higher demand for workers, who will require even higher pay because of supply constraints and high taxes.
Unlocked demand, decreased labor supply and higher taxes means inflation. Lots of inflation, specially in services.
Businesses will offshore or automate to try and reduce cost, but that will push even more workers onto UBI.
What will the new equilibrium be? Nobody knows. It could stabilize with extreme income inequality creating a permanent underclass, It could drive the United States into hyperinflation, creating civil unrest and destabilizing world security. Nobody knows. It would be a society wide experiment that would be politically impossible to undo democratically.
If you think BS jobs are bad, how about no jobs in a bankrupt United States, sliding into Venezuelan hyper inflation?
Imagine China, Russia, and North Korea undeterred and the death and human suffering that would follow.
The US may not be perfect, but the world is a harsh harsh place and people have no idea how good they have it right here, right now.
So good that professionals not only demand high pay from their employers, but think it's totally reasonable to require "meaning" as well.
Were he still alive, I would pay good money to see the author try and read his essay out loud to Victor Frankel.
I just experienced something like a flashback, I was in debate club, listening to yet another kid claim that social change X would lead inevitably to global nuclear annihilation. Strongest case possible, right?
We can't possibly re-allocate the trillions of dollars the federal government spends, or the more trillions the collective states spend, because it would lead inevitably to mass die-offs comparable to soviet-era famine conditions.
Existing UBI programs don't directionally move that way? Pandemic productivity actually rose, despite a couple of quarters of setbacks? The actual evidence doesn't support gloom and doom? Just make the claims of potential harm bigger! Nobody can counter hyperinflation, civil unrest, or world war!
>Pandemic productivity actually rose, despite a couple of quarters of setbacks?
Covid did not create Universal Basic Income. Everyone knew that Covid money will run out, so most people didn't quit their jobs (although still many did, which was GP's point). Also, the amount of debt raised during covid was no joke, and we don't know the long term effects. But even disregarding that last point Covid isn't a useful datapoint for disproving GP's claim. The fundamental issue is with UBI is people will not want to work and that is GP's central claim.
So GP using involuntary unemployment numbers in support of an anti-UBI case didn't trigger your alarm, but me using voluntary employment numbers to demonstrate the falsity of the original claim did?
If Covid isn't a useful datapoint for disproving GP's hysterical claims, then it definitely isn't a useful datapoint for supporting them.
The number of people who quit their jobs to depend on "Covid money" rounds down to zero. There just wasn't enough "Covid money" for that. Many people were unemployed during the early days of the pandemic, but not voluntarily.
Your "fundamental issue" lacks evidence and the counter-evidence from limited trials of UBI points the other way. But carry on. Civil unrest and a world war await.
>So GP using involuntary unemployment numbers in support of an anti-UBI case didn't trigger your alarm
Obviously, becasue the cause is unemployment, not whether it is voluntary or not.
>but me using voluntary employment numbers to demonstrate the falsity of the original claim did?
Yes, because it is a logical error, as I explained above.
>Your "fundamental issue" lacks evidence
The evidence is that most people don't like working. I've never met anyone who went to work on a day off; I've seen countless people to get out of it.
>counter-evidence from limited trials of UBI points the other way.
They are as you say: limited. We would only see the effects after UBI becomes "a thing" that allows people to slack off, not a trial. And then children won't take eduaction seriously as there is no more carrot.
Hahahahahaha! I had no intention of replying at all, but this, this is so outrageous I laughed out loud:
> And then children won't take eduaction seriously as there is no more carrot.
From the misspelling of education to the apparent breathlessness with which this prognostication is offered up as a fait accompli, it's perfection itself!
We live in a world in which for some strange reason people try to educate children and work at jobs making more than subsistence wage, and even try to get raises and promotions, and even when they get a job making six times the average, they keep working and advancing, but no, if UBI is enacted, civil unrest and world war will be the least of our problems. CHILDREN WILL CEASE TO LEARN!
Your're right that "And then children won't take edu[ca]tion seriously as there is no more carrot" is wrong. It's more accurate to say there is less incentive to learn, and education will be taken less seriously. But the main point still stands that when work is optional it will be taken less seriously. 13% of people are on SNAP already,[0] and a large percent of those people are not actually looking for a job or only work part time. The idea that this number will increase as a result of UBI is obvious. Claiming that it won't is not, which is why you need to resort to high school debate team tactics of taking your "oppenent" in bad faith and ignoring their points. Grow up.
A slightly different version of the arguments you write here could have been made against pensions or a 40h work week when they were introduced. I'm not saying UBI would not massively change society and potentially include risks, I am just questioning your certainty that implementing it would lead to disaster.
Costs don't disappear just because it's the private sector that has them.
On the national economic level, waste is waste.
But I agree that with UBI people who have actually important jobs might also quit, or demand higher salaries.
Some would say that sounds fair and sensible compared to the current status quo.
About software development specifically: Note how many of the most valuable software projects have grown not from profit driven corporations, but from open source maintainers working in their spare time.
this is actually not supported by any data. its pure speculation. plus the handful of UBI trials don't show the catastrophic breakdown of labor supply you claim to inevitably happen... the reality is until a proper ubi trial is done with actually livable wage we don't know what would happen...
A lot of your reasoning requires a big [cite needed] - specifically the "lots of people would work less". UBI pilot programs have proved the opposite. Also several states cut unemployment benefits to goose their active worker numbers and the opposite happened - less people working.
I'm not saying that nationwide UBI would be a good thing but your logic isn't clear.
The late David Grabber had a great book about this - Bullshit Jobs.
And when you draw the line, we do keep a lot of work and a lot of jobs around purely because if we don't a lot of people are going to have a lot of free time to ask a lot of questions about the way that the world works. And the very foundation of our system is that you have to go work, sell your labour to make someone else rich, then buy things to make someone else rich and eventually die.
People don't object to UBI because they don't like the idea of people getting money for no work.
They dislike UBI because either
a) It's financially impossible. Assuming the basic income is "enough to live a basic life", say £1500/month in the UK, that is totally impossible.
b) They don't like the idea of people getting free money and not having to work when they do work. I mean you see that already with benefits/welfare, but UBI makes it more extreme.
Although obviously with UBI the people paying lots of taxes have the option of stopping work and moving somewhere cheap. Which is what a huge number of people would do, and the whole system would collapse.
But yeah, nobody is against a magical utopia where nobody has to work. They just know it can't exist.
> this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
Absolutely. I've worked in multiple different companies, of different sizes, since about 2011 with breaks in between. One common theme was the amount of pointless work going on, stuff that could've been streamlined or automated, or manual reports that no-one reads or acts upon.
Everyone I know who works in these kind of jobs are financially secure but extremely depressed.
One of the mistaken beliefs underlying the UBI argument is that the fulfillment part will come automatically for most people. Who wouldn’t want to spend every day painting, writing, or playing music? The problem is that most people who are not ideologically self-motivated are unable to find sources of fulfillment without some sort of economic influence.
Are the workers depressed because they are "not ideologically self-motivated", or because they must spend time doing something they don't see the point in doing?
To rephrase the title a bit: most data work seems fundamentally like research.
You look at a bunch of data seeking a nice, clean X/Y relationship that yields positive ROI for the org. These types of situations are rare - that’s the sort of e-commerce landing page / conversion optimization problem where a bunch of other factors (particularly whether anyone needs or wants the widget) are answered already.
That doesn’t sound like the nature of these data engineering type roles. Sounds like there’s an explicit “Get us this spreadsheet!” BI-grunt functionality, with the overhead of doing researchy type work that might pay off for the company later.
I’d guess that the “get the business unit their spreadsheets” is the actual valuable part of this role, and any other useful insights about the data are just considered gravy by the org.
"There are usually at least a few tasks required for the organisation to function, such as producing some sort of report for the government, that technically does need to happen, so we can't simply lay everyone off." - At highly regulated companies and companies of sufficient size, reporting to upper management, government and external stakeholders is more than just "a few tasks". This, alongside a continuously developing product ("terrible data"), product pivots ("vision is hard") and growing data volumes IS what "most" data work is, and it is not fundamentally worthless.
So the premise of the post seems a bit flawed. The author seems a bit jaded by bureaucracy and lack of vision at a few organizations they have worked at, which is unfortunate, but by no means an accurate representation of data work.
That's ignoring that much of that government mandated regulation is worthless in itself, not producing the originally intended results and only leading to useless jobs like the author's.
The data is adjusted so that it fits the regulatory demand, not the other way around. In many cases it's still pointless work that produces no value for society.
Yes, I considered raising this point. My work is sometimes necessary to comply with regulation, but frequently that regulation is inherently pointless. That said, this seems to be a different category of worthless. My regular work can literally not be done. This category of work must be done or you risk terrible consequences, but it probably shouldn't be done.
324 comments
[ 3.6 ms ] story [ 334 ms ] threadSometimes it's only '1 thing out of 10' but that '1 thing' is absolutley essential.
But honestly, I suggest the entire 'big data' movement is a bit outlandish, there really isn't a need for most of it. The 'saving grace' frankly is AI, because maybe-maybe-maybe we can make use of that data in some future alg because god knows we're really not making that much of use of it.
But it's hard.
So many people underestimate the value of experience. Seeing patterns comes with seeing many different things over an extended period of time.
I think the problem is really one of scale - it's way harder to pin down your value add when you're part of a very large collective. Being responsible for 10% of a 10-man output is much more tangible than being responsible for 0.001% of a 10,000-man output, and the difference between 10% and 0% is similarly much easier to discern from a management and accountability perspective.
Once the target is hit, most of the work to hit that target is worthless, but we’ve gotten smarter at taking the journey and creating repeatable steps, setting up data structures, etc.
... and were the goddam strategy team!
I really depends so much on leadership as well.
I suggest most companies either have 'Zero IT' leadership knowledge, in which case you get the 'buy whatever Microsoft tells us to buy' - or they are tech companies where they understand it but are probably a bit overzealous.
But the same thing applies to everything.
'Legal' can be a useless waste of money, until all of a sudden they're the most important team in the company ...
I don’t like throwing around out the phrase “privileged/out of touch” too often but this post doesn’t seem fit for the top position of HN.
The past couple decades have seen huge advancements in safety, reduced workplace injuries, auto injuries, etc. all driven in some part by analytics.
This sentiment that “all data analysis = advertising = evil” seems very reductive. It reminds me of all the comments I see from my technical peers about how “useless” other departments are such as HR and Sales, when they’re fundamental to a healthy business which pays the salaries of programmers.
It is hard to do good data work. Offhand it takes some combination of:
- business understanding and goals (that don't themselves come directly from data)
- using data to effectively orient those goals to the best opportunities
- using data to measure whether you are successful or not
Part of that can involve meta goals -- our data is not sufficient now to meet bullet 2, so we need to start doing something different to measure it.
I find many people in these roles act like they are just human machines producing reports. You really need to take agency in many situations IMO, and direct higher ups to look at data in the right way. If you just wait to be told what report to produce, it will not go well.
And just as most legal disputes end in settlements, most data scientists are excess capacity, kept around because the programmers who will put up with typical dev nonsense aren’t smart enough to hit the high notes… when the fact is that said high notes only need to be hit very, very rarely in business. Being a corporate lawyer comes down to intimidation—no one wants to face off against Apple’s team of lawyers—and 99% of being a corporate data scientist is talking in maths to impress (or defraud) clients and investors.
You end up with "we spent years and $ to get the data which says do X, but we don't feel like doing X, so we're just going to ignore it because data in and of itself has no power within the organization".
It's like buying a gym membership and not going to the gym. Having a data science department satisfies the organisation's need to believe it's self-improving.
Data buy-in within a company is absolutely essential for it to work.
A lot of my previous data work was someone saying "show me what's interesting"... "no, not that- something else"... "huh, this doesn't jive with my gut, start over".
Bringing _value_ out of data sciene / engineering is incredibly hard. You have to have the engineering skills as baseline, but I firmly believe everyone undervalues the amount of work to "sell" your analysis to less technical folks in a way that's inline with the needs of the org. It's incredibly difficult.
Non-profits are a funny old space. In theory, they're not out there to make money, so people gravitate towards them seeking "purpose". Everything is fine once you realise that you're not going to get that, or at least in the way people come to it for.
Non-profits have ridiculous, completely unachievable goals, particularly when you put their budgets in context. Some provide truly excellent services and products (e.g., reports), most don't.
So what's so great about a non-profit tech job? I've got an interesting problem space, I love trying to work out how to measure things that are inherently difficult to measure. I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't.
I bust my arse because I'm interested in what I'm doing, but I could easily coast if I wanted to. The pay is enough for me, and my colleagues are for the most part super nice and interesting. If I want to learn something, I can make an excuse to play with it at work and run with it.
Seeking to have "impact" through a data job at an average non-profit is naive, but a lot of these jobs have stuff to recommend them beyond that.
----
Edit: slight addition to my list of joys of non-profit nerding
If you believe what the organization does furthers your purpose, then you are going to get that, although I agree that the day-to-day feeling may not be that "purposeful".
> Non-profits have ridiculous, completely unachievable goals
There a gazillion kinds of non-profits; some do, some don't.
If your non-profit is the town museum management foundation, you have a perfectly achievable goal (it's right there in the name).
If your non-profit is "the coalition to end world hunger", then maybe not so much. Then again, even such a non-profit might have the actual, stated, legally-filed goal of delivering food aid to areas in the world hit with natural disasters like floods or drought - and that's not an achievable goal in the sense of being ever done with it, but it is achievable in the sense of being able to do just that continuously.
> So what's so great about a non-profit tech job?
That there is almost never the situation of a huge pile of closed-source code developed internally which you have to live with; and no custom expensive hardware. So typically you need to both rely on, and contribute to, free software that basically anybody can use.
> Seeking to have "impact" through a data job at an average non-profit is naive
On the contrary. There is great potential of technological impact outside the organization due to what I said above; and there is potential for impact on what the organization does, because it's often more fluid, and using data, you can make convincing arguments about steering activity in a different way - appealing to your colleagues desire to serve noble purposes.
I've got a particularly fun gig at the moment, but generally tuned out of the mission a while ago.
I guess my point was that sure, as the original post suggested, this sort of work _can_ be meaningless when viewed as the author does, but there can still be lots to love in the sector.
As someone who's now the only tech person at a small non-profit, this really really resonated with me (as did the rest of your post).
I feel it's often challenging to asses how robust/brittle something should be implemented because we don't sell (and maintain) software. A lot of "tech things" simply have a very short life span, often closely tied to project scope and duration.
I hope you don't mind me asking here, but it would be great to connect with someone in a similar position. If you're up for it, please feel free to shoot me an email, you can find the address in my profile :)
But that definitely did not seem to be the most common case, as you've said. I think maybe I might write some quick edits to make this clearer, both to myself and reader, but 'most data work' being clueless is the real issue, not 'all data work'. That is, I've moved around jobs quite aggressively, but if there's a 10% chance that any job will be okay for a while, that could take quite a few moves to find one, and you can't change over that often!
Then take the company public later at 10x the valuation, having slashed the cost by 10x without impacting operations or growth.
A simple example would be, they could put the IT support department under the "VP of Crypto" instead of under the "Director of IT". So private equity comes in and fires the entire crypto department because they assume it is useless and then find out that no one can get their passwords reset or computer replaced.
More complex examples involve finance. You could transfer stock in your company to a banker as collateral for a loan with the condition that if the loan is paid off early there is a 1 billion dollar fee. Taking a company private requires that the buyer buy all the public stock.
https://en.wikipedia.org/wiki/Shareholder_rights_plan
"Manage better" is not a scalable system. Although it does work for some individual takeover merchants; it's a big part of the Warren Buffet success.
A semi-generalisable strategy to identify the 90% quick wins in consultancy contracts: look at the consultancy contract, anywhere it says Oracle, IBM, Microsoft, Data Science think about how you can replace that with an open-source stack run by a competent technology team that you will recruit and what impact scrapping all the data science stuff will have.
The crucial bit in this strategy is to offer a package to this new elite tech team that would be competitive with FAANGs, this will be completely surprising to most management of these companies as they are used to paying their tech stuff garbage when in effect they are paying 10x FAANG salaries via the daily rate of McKinsey consultants.
The risk is definitely in the squishy unknown unknowns that are hard to quantify and therefore get obliterated early on in a digital transition.
And when you look at it, yes, there are plenty of funds driven by buying a company, replacing the management, and selling it. Most of them are not long-term driven (you really expect people that flip companies to be long-term driven?) and thus make all kinds of decisions that you'll probably disapprove.
It's not just data analysis. There's bloat all over tech.
For at least a couple of months, yes. The jury is still out as to how his actions affect the long-term sustainability of the company.
If you were a farmer you could stop buying seeds entirely and save a hell of a lot of money. And you'd be perfectly able to continue to grow and reap the current crop. If anything, productivity would increase because time spent buying and storing seeds is freed up to focus on the current crop.
But next year...
The issue was spotted in April/May, the building happens during the summer and is launched In September… I’ve watch this cycle happen so many times, I’ve given up trying to point out that the issue is just seasonality.
He was laughing at them in their faces, even, and was 100% correct. They excluded him from any further meetings.
Every time they brought up a new stat and set off alarm bells he would bring up their massive fuckup and rub it in their face, and then would dig through the data and question every point.
He was eventually fired for pointing out how incompetent our new owners were after an acquisition, and would have been a lot happier joining everyone in their sham.
I had this exact same problem where I became entirely disillusioned with the output of my work after having become reasonably senior and realising that the majority of my time and effort ammounted to projects which either failed or treaded water. The unfortunate reality in business is that the projects that truly change the world are 1 in 100. They don't look or feel much different from any other at the begining but when traction really does start to take off you know you have something right. I think from your description, you need to work at a startup where you have genuine impact. This is the only way you truly "feel" meaning. You might still build something that's reasonably worthless to humanity, but you will do it your way and take your own risks.
So at the end we were just doing these demos and not working on actual things that we needed to do. The project was important, if implemented successfully it would cut jobs of ~60 people in the organization. So management kept on adding more people until they realized that there was no real progress and they dumped the project and we got reassigned.
I realized that 1 year of my work as well as ~15 other engineers just went to waste.
Jobs like this exist, though you may have to take a pay cut compared to the bullshit. My first job after finishing my PhD was as a staff scientist in an academic lab using ML engineering + data science to support scientific research. There seems to be a fair number of grant supported jobs like this and pay isn’t terrible. Just under or just at 6 figures. You can make more in industry, but scientific work feels very meaningful.
Now I’m working in credit risk modeling, something I never expected to be doing, but so far it’s been a good fit. The models are applied directly in decision making for the business, and there’s a real incentive to get everything right because mistakes could harm real people’s lives. The team I’m on is strong and ethically sound and I feel good about what I’m doing.
For anyone in the data space who’s despairing about the state of the industry, non-bullshit jobs do exist, you just have to look for them, and use your judgment when scoping out new roles.
Better zoning laws that allow mixed-use zoning would enable more, smaller grocers embedded in neighbourhoods. Personalized service would be much easier on a smaller scale. And you wouldn't have to travel far to pick it up.
Most people would agree that produce should make up the bulk of ones diet, yet it’s pretty plain to see that the footprint of the produce section doesn’t scale with the size of the store.
Produce is something best shopped for frequently, and is something I’d personally be willing to pay a higher upfront cost for the convenience of closer smaller stores where I can get in and out quickly. Because while we may pay a lower sticker price, the elephant in the room with the massive hidden cost that we pay but rarely account for: food waste.
Also fresh food is great for the diet and health, but when some even occurs that interrupts things like distribution of fuel, food, or power just having it around may lead you to getting hungry really quickly. This is something you have to look at on both a personal and national level. Food security can quickly lead to destabilization.
It’s not like the typical Costco shopper is low income. Our grocery system is optimized for moderately well to do suburbanites with SUVs and large houses. Low income people tend to live in “food deserts”.
Consider the experience of shopping for food in America if you’re dependent on public transit, which a much larger proportion of the low income population are. Requiring a car to get groceries is extremely expensive.
Fresh food without preservatives lasts at least a week.
https://www.marketplace.org/2023/02/02/why-are-boutiques-and...
This is also one of the few domains where people are very price sensitive, and they regularly see and think about costs. I couldn't tell you what my local automotive guy charges for an oil change, but I can absolutely tell you the price of flour and cheese.
No one is going to pay $10 for a single apple, and if you raise prices hard enough people are going to start gardening.
Update: Go buy an apple in a food desert and I guarantee that if they have one you will pay more than a dollar for one. Grocery chains are optimized for maximizing profit not distribution.
If the giant chain (or a network of smaller grocers) had good personalization, it would make the stocking the small local pickup point more practical.
I live in a diverse multicultural community. A small local grocer can't stock everything my neighbors and I want.
Heck, the local wine & beer outlet lets me just open up the last purchase list and buy that again (and adjust the quantities if desired).
WTF is wrong when a local store can get it so right, yet Amazon, with it's emphasis on "always be hungry like a startup", so totally forks it up?
It was "OK" at the time; I was on crutches and could go to the store but not easily do a full grocery shopping. But I haven't done online grocery since.
I also mostly would just as soon go to the store.
[1] https://canworksmart.com/diapers-beer-retail-predictive-anal...
So what will actually happen, with you and me and everyone else, is that we'll grumble about how they moved the cheese again, we'll wander around the store until we find it, and we'll be exposed to some more products and implicit advertising in our search, and we'll forget about it as soon as we exit the store. Over time the store will increase its profits 1.3%, and the competing store "where everything is always in the same place as last time" struggles because their prices aren't maximally optimized with the same grocery store software everyone else uses.
This seems to be the Loblaw/NoFrills model in Toronto. They relocate products with alarming frequency.
Want milk? That's in the dairy section. Want organic milk? That's in the health food section at the opposite end of the aircraft hangar sized store. Nuts? Either in the snack foods, health foods, or baking ingredients depending on unknown critera - one literally has to check all the places.
Though that also makes the point that making things easier for customers to find is probably more important than any minor co-purchase optimizations.
On a local basis I can go to two of the same store a few miles apart and one will out of products X, Y, and Z, but the other store may be out of X, T, and R. Localized buying trends can have significant differences.
Also, there may be many other effects here. For example, if a popular product is actually popular and the pipeline to make the product is months long, well there's going to be outages. Shifting production generally isn't easy and trends pass quickly.
In addition, perceived popularity can be used to manipulate consumer pricing. "X is always out. Oh look X is in stock now, I should buy it for 50% more"
I think some suppliers just aren't that good at adapting supply to demand.
prior to covid? last few years have not been representative about how well the supply chain works over the past 50 years.
Actually neither of these is particularly easy to do for unorganized companies.
- Any tech company where the Stats/ML model is one of THE products and differentiators. You'll need to cut through a lot of buzz word and sales-speak to find the good ones.
- Banks and other financial institutions where making uneducated guess is a big no-no when it comes risk, pricing and anything related to financial products. Your example of Credit Risk Modeling is a classic example and very interesting problem.
- As much as consultancy gets a bad rep due to some shady practice from big players, there exists a solid demand for professionals that know Statistics in the Large Construction Projects space. Let's say modeling demand and return financial for a project, proving environmental impact, preparing/implementing/analyzing unbiased surveys in the area, and so on.
- Government agencies where data is one of the Key Outputs. Such as the Census, Bureau of Labor Statistics, CDC, and so many others.
As you said, sometimes it will mean a pay cut, especially if you want to remain in the technical work and not deal with the business and managerial side of things. But there's solid demand.
Is it trying to take a huge dataset of consumer features and join it to a dataset of loan outcomes and then predict loan outcomes?
[0] https://towardsdatascience.com/book-review-intelligent-credi...
i would also like to add that modelling in credit risk is not just about yes / no answers around loan outcomes.
There are lots of other goals that are regularly modelled such as default rates, profit optimization, loss minimization, delinquency and payoff rates at specific parameters ... endless options
There are also lot of different ways in which these models are implemented ( decision trees, statistical analysis, ML... )
Some examples of (real life) projects include:
- if our institution offers this card to clients with 750 credit scores vs 790 credit scores, how does my profit move vs my losses and what the factors to limit losses while maximizing profits
- how do I minimize my costs for servicing this card while keeping profits at the max ?
- what rewards options lead to the highest number of preselected / qualifying clients taking up a product at the lowest cost
- what contact strategies are best for specific types of clients if they are late on payments - call or email or text or legal letter? which strategies are the cheapest? which strategies give what this institution considers to be the best response ? which lead to fastest full payment? fastest partial payment? which lead to getting back to a regular payment plan?
- how can we identify clients who have a lending product with us who might be on the market for another lending product in the next 12 months? in the 6 months?, those who might need a limit increase pro-actively? those who whose might need a limit decrease pro-actively?
And, one of the largest area pf credit risk evaluation is real time decisioning on transactions: 'is throwaway201606 really buying $6000 of apple products, in person, at this mall in Toronto, Canada right now when I (the system) know I he bought a daily Wendy's Spicy chicken sandwich 10 minutes ago in Dallas" and should we allow this payment
Some example of how models are used here include ( note that modelling helps establish which transactions to look at more carefully and which to ban outright among other things )
- predict what type of terminals are being targeted: scammers -> we have left bank ATM machines alone and started looked at gas pumps:
- predict where transactions of interest might come from: scammers -> we do scams on site A at Christmas, scams on site B in the summer or we do site A scams with brand Y card and do site B scams with brand Z card
- predict behaviour patterns of transactions of interest: we always test the cards we will use by purchasing a $5 'brand x' gift card online 10 minutes before
[0] https://www.computer.org/csdl/proceedings-article/hicss/1998...
Ended up at a place where ML doesn't just make things better, but is necessary to make things work at scale.
There were a lot of job ads I looked at that just seemed dreadful to me though. So many recruiting companies wanted ML Engineers/Data scientists. I could kinda see how it would work, but didn't think I would enjoy it.
I actually found myself working on a credit risk modelling project on the capital markets side and it’s been great as well.
First junior dev job it's not as important because you just need to find something. After that you will want to be very careful to make sure the job has actual business impact, otherwise you are likely to end up in some kind of bullshit vanity project that only has to appear to work.
Less metaphorically, I've always looked for jobs where my skills are aligned with the "company mission" - first at a bunch of startups, and now as an academic researcher where I get to define that mission.
I really wish the websites I visited made better use of my actual history on those sites to tailor relevant ads to my interests. It seems like an ideal application of AI to me. They could do a far better job than serving me the lowest common denominator stuff they keep throwing at me.
My Twitter news feed these days:
* 10% - posts of interest from people I follow, i.e. the stuff I actually go to Twitter for
* 90% - Ads and recommendations of topics to follow that I have zero interest in
If it's going to insist on showing ads and recommending topics of interest, you'd think those could be better personalized, given that Twitter has years of my tweet, reply, and like history to train its AI on.
But no... what I get is crypto ads, Hollywood events, celebrity news, sports news, etc.
I may not be voting with my dollars as a user here, but I am voting with my attention span. Dollars are not the only asset of value in play.
As Cory Doctorow has point out so eloquently in his "enshittification" series, the end point isn't even to the benefit of the advertiser. At the end state, The Machine also have perfect knowledge of the advertiser, and the adtech company can turn the full power of the The Machine to the benefit of itself, extracting value from both the audience and the advertisers.
Also ads != recommendations. In a sense after a while these two are also at odds with each other. Cause there is again, the ever present need to sell you more stuff.
Edit:ignore, op posted. I should stop jumping to answer so soon!
The rare time an ad slips past my blocker, (and rarer still when I stop to think about them) I take solace whenever they are irrelevant.
If the companies in question are billing the ads by impression, as is common, they get paid whether the ad served is relevant or not. If I'm going to be served up native content or ads anyway, I far prefer they be relevant to my interests than they're not.
― Upton Sinclair
These companies are trying to extract my attention and money -- both limited resources -- and if I have to see ads better ones that I have no interest in, simply out of spite.
You founded and run an advertising company. Are you taking the piss? Surely you've aware of the ethical issues even if you don't give a fuck about them.
Or are you just saying you categorically expect AI/ML people's interest in the tech and/or money to override the ethics?
Perhaps we can start from the points I'm making and assess them on their own merits instead?
As far as I can tell, the points you made are:
A) It's strange people make moral choices to stay out of advertising.
B) Advertisement could be more targeted.
B is boring and I don't care.
A isn't really a point with merits to be debated.
I enjoy eating meat. But I don't call it "strange" everytime someone turns out to be vegetarian. I understand why they refrain from eating meat. It's not like I disavow the existence of the question of "should we eat meat?". I just disagree.
Be careful before you start tossing around the Latin, someone might hit you with a 'cui bono?'.
I've been doing ML for over 20 years and if it were up to me, advertising would be illegal.
We have a research partnership with the University of Ottawa & University of Luxembourg specifically on the statistical analysis end of things to go deep in some areas to later incorporate the findings into our products. In fact the first go around of that research cycle has already happened (https://arxiv.org/abs/2301.13807v1) and the insights are being integrated into our Deviant product (https://auxon.io/products/deviant).
It's definitely not ad-tech. It definitely has a specific applied use case. Most of our marketplace traction is in aerospace, energy, automotive, and defense. We're not immediately hiring for roles on the data analysis end of things (we're in much more immediate need of visualization & frontend help), but we will be this year.
This reminded me of those Microsoft Viva emails I keep getting. Anyone finding those useful for anything at all?
To me it seems like a solution in search of a problem.
Currently starting to look for a new role and couldn’t have articulated my goal more clearly. The problem is that this narrows the field considerably, to finance/insurance/fraud (“pure money work”), healthcare/EMR (which as far as I can tell is also actually just profit optimization work for hospitals), and then you have manufacturing (closer to where I currently am), but the actual applications of data science are more limited and data Eng / analytics are more valuable, bioinformatics (which is really interesting, but also a large learning curve), and then just traditional BI (kind of generic and boring).
Also, if you’re thinking about long term career development, many of these functions fall under the IT/CIO umbrella as orgs grow, which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc. which I’m not saying is a bad thing, just an observation. Just applying statistics and modeling can only rise so high in an org, unless like you said, their core business and value prop is being the best at modeling some phenomena.
Also theres quite a bit of money sloshing around and it’s relatively easy to quantify financial benefit from fraud reduction, so the departments tend to be well funded
Like security, it's a field where success is when (externally) no one knows anything ever happened.
Tbh I dread my future job of maintaining R and python scripts
I was going to suggest: we have a bunch of data scientists working on intrusion detection problems in cybersecurity, tied to an engineering organization instead of an IT organization. On the balance of considerations, I think it has a net positive impact. And I'm pretty sure plenty of other companies have something similar. Unfortunately, we also have an ongoing hiring freeze, so I can't recommend my own organization.
- energy management (shifting loads to times when energy is cheap) for consumer/commercial/industrial use cases
- energy markets, especially power trading: often highly algorithmic, and driven by models that turn fundamentals data (weather, calendar, …) into supply/demand predictions, and from there into price predictions
- retail pricing, both offline and e-commerce
I feel like this is true for IT in general. Projects often fail or don't deliver on their promises.
Years of my work either never reached production or were scrapped and re-written because some key requirement changed.
For this reason I appreciate time spent with my family - it's always rewarding one way or another.
To some I'm sure it looks like laziness, but I consider it more weaponized procrastination. With experience I've become better at it.
Sometimes I get that gut feeling about a project, and just can't bring myself to get started. Some basic research happens, but I spend close to zero actual time on it, sticking to other things that matter.
In two weeks, priorities have changed and all of what I was asked for has now changed or been cancelled anyway. Glad I didn't get too invested.
this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
Private money being wasted due to inefficiencies at the office (as you say, a misallocation) is surely entirely different than a government-granted right to not produce anything yet still be supported.
There's basically no way you can start a new state if you'd want to opt out of the existing ones, and even moving from one to another is quite the undertaking. Imagine Atlassian saying "sure, you're free to stop working with and paying us, but we believe that we've been instrumental in building your company, and we believe that entitles us to 20% of your company if you want to work with someone else".
But even if it actually was millions of completely unproductive jobs, I fail to see how that would be an argument for adding a few dozen millions more and not a reason to get rid of the ones that exist and cost us money that we could otherwise invest into important things that are currently underfunded.
I wasn't making the argument that every government job is useless, I was simply reminding that it's not true that unproductive private jobs can be weeded out simply with competition or customer care.
Why?
If you are in the camp that think companies sole purpose is shareholder profit then this is perverse.
If you are not in that camp you can probably agree that we should aspire to the best life for most people.
(As a European this comment really dissonates. I do realise that it does not for Americans)
As a European, odds are that you're used to a monoethnic country where "most people", "society", and "government" are kind of interchangeable, and people find it easier to sacrifice themselves for the greater good -- as you say, aspire to the best life for most people, even if it might mean that the decisions being made are not optimal for one's own circumstances.
I myself hail from such a country (though outside of Europe), and you are right that the American viewpoint, or that of any other large, multiethnic country, is quite different. Implementing policies that require taxpayers to support others that are culturally different and therefore harder to empathize with is an order of magnitude more difficult; and it is becoming yet more difficult in America as time goes by, due to the reduced emphasis in assimilating to the mainstream ("white") culture.
go buy treasury bonds, they pay almost 5 percent. ibonds pay more. why? they just do. and companies and rich people buy them all the time.
... sidenote; I personally think that something completely inefficient "is not that bad". Much worse is work that is directly detrimental for society as a whole, even if it does make a few people rich. Now that is terrible.
They then buy food, housing, medical, transportation, fuel, entertainment, (optionally) raise kids, by things kids want, take vacations, have hobbies
They're still most likely a net positive to the economy.
The important thing is that people are prevented from enjoying themselves for at least eight hours a day.
Otherwise, they will not grow.
True UBI would create the largest, most indolent, and most socioeconomically isolated consumption class the world has ever seen — until it collapses under its own weight.
The wealth and culture gap would become nearly insurmountable for anyone otherwise capable of upward mobility out of the UBI class.
UBI is a yearly recurring expense in the trillions.
Lots of people would work less, (remember COVID?) lowering tax revenue.
This means higher demand for workers, who will require even higher pay because of supply constraints and high taxes.
Unlocked demand, decreased labor supply and higher taxes means inflation. Lots of inflation, specially in services.
Businesses will offshore or automate to try and reduce cost, but that will push even more workers onto UBI.
What will the new equilibrium be? Nobody knows. It could stabilize with extreme income inequality creating a permanent underclass, It could drive the United States into hyperinflation, creating civil unrest and destabilizing world security. Nobody knows. It would be a society wide experiment that would be politically impossible to undo democratically.
If you think BS jobs are bad, how about no jobs in a bankrupt United States, sliding into Venezuelan hyper inflation?
Imagine China, Russia, and North Korea undeterred and the death and human suffering that would follow.
The US may not be perfect, but the world is a harsh harsh place and people have no idea how good they have it right here, right now.
So good that professionals not only demand high pay from their employers, but think it's totally reasonable to require "meaning" as well.
Were he still alive, I would pay good money to see the author try and read his essay out loud to Victor Frankel.
We can't possibly re-allocate the trillions of dollars the federal government spends, or the more trillions the collective states spend, because it would lead inevitably to mass die-offs comparable to soviet-era famine conditions.
Existing UBI programs don't directionally move that way? Pandemic productivity actually rose, despite a couple of quarters of setbacks? The actual evidence doesn't support gloom and doom? Just make the claims of potential harm bigger! Nobody can counter hyperinflation, civil unrest, or world war!
Covid did not create Universal Basic Income. Everyone knew that Covid money will run out, so most people didn't quit their jobs (although still many did, which was GP's point). Also, the amount of debt raised during covid was no joke, and we don't know the long term effects. But even disregarding that last point Covid isn't a useful datapoint for disproving GP's claim. The fundamental issue is with UBI is people will not want to work and that is GP's central claim.
If Covid isn't a useful datapoint for disproving GP's hysterical claims, then it definitely isn't a useful datapoint for supporting them.
The number of people who quit their jobs to depend on "Covid money" rounds down to zero. There just wasn't enough "Covid money" for that. Many people were unemployed during the early days of the pandemic, but not voluntarily.
Your "fundamental issue" lacks evidence and the counter-evidence from limited trials of UBI points the other way. But carry on. Civil unrest and a world war await.
Obviously, becasue the cause is unemployment, not whether it is voluntary or not.
>but me using voluntary employment numbers to demonstrate the falsity of the original claim did?
Yes, because it is a logical error, as I explained above.
>Your "fundamental issue" lacks evidence
The evidence is that most people don't like working. I've never met anyone who went to work on a day off; I've seen countless people to get out of it.
>counter-evidence from limited trials of UBI points the other way.
They are as you say: limited. We would only see the effects after UBI becomes "a thing" that allows people to slack off, not a trial. And then children won't take eduaction seriously as there is no more carrot.
> And then children won't take eduaction seriously as there is no more carrot.
From the misspelling of education to the apparent breathlessness with which this prognostication is offered up as a fait accompli, it's perfection itself!
We live in a world in which for some strange reason people try to educate children and work at jobs making more than subsistence wage, and even try to get raises and promotions, and even when they get a job making six times the average, they keep working and advancing, but no, if UBI is enacted, civil unrest and world war will be the least of our problems. CHILDREN WILL CEASE TO LEARN!
Hahahahahahahaha! Okay, NOW carry on.
[0]https://www.census.gov/library/stories/2022/05/who-is-receiv...
Amen.
A slightly different version of the arguments you write here could have been made against pensions or a 40h work week when they were introduced. I'm not saying UBI would not massively change society and potentially include risks, I am just questioning your certainty that implementing it would lead to disaster.
Oh, and it's Viktor Frankl by the way.
But I agree that with UBI people who have actually important jobs might also quit, or demand higher salaries.
Some would say that sounds fair and sensible compared to the current status quo.
About software development specifically: Note how many of the most valuable software projects have grown not from profit driven corporations, but from open source maintainers working in their spare time.
I'm not saying that nationwide UBI would be a good thing but your logic isn't clear.
That should would be a frightening and radical departure from today.
They dislike UBI because either
a) It's financially impossible. Assuming the basic income is "enough to live a basic life", say £1500/month in the UK, that is totally impossible.
b) They don't like the idea of people getting free money and not having to work when they do work. I mean you see that already with benefits/welfare, but UBI makes it more extreme.
Although obviously with UBI the people paying lots of taxes have the option of stopping work and moving somewhere cheap. Which is what a huge number of people would do, and the whole system would collapse.
But yeah, nobody is against a magical utopia where nobody has to work. They just know it can't exist.
Absolutely. I've worked in multiple different companies, of different sizes, since about 2011 with breaks in between. One common theme was the amount of pointless work going on, stuff that could've been streamlined or automated, or manual reports that no-one reads or acts upon.
One of the mistaken beliefs underlying the UBI argument is that the fulfillment part will come automatically for most people. Who wouldn’t want to spend every day painting, writing, or playing music? The problem is that most people who are not ideologically self-motivated are unable to find sources of fulfillment without some sort of economic influence.
You look at a bunch of data seeking a nice, clean X/Y relationship that yields positive ROI for the org. These types of situations are rare - that’s the sort of e-commerce landing page / conversion optimization problem where a bunch of other factors (particularly whether anyone needs or wants the widget) are answered already.
That doesn’t sound like the nature of these data engineering type roles. Sounds like there’s an explicit “Get us this spreadsheet!” BI-grunt functionality, with the overhead of doing researchy type work that might pay off for the company later.
I’d guess that the “get the business unit their spreadsheets” is the actual valuable part of this role, and any other useful insights about the data are just considered gravy by the org.
So the premise of the post seems a bit flawed. The author seems a bit jaded by bureaucracy and lack of vision at a few organizations they have worked at, which is unfortunate, but by no means an accurate representation of data work.
The data is adjusted so that it fits the regulatory demand, not the other way around. In many cases it's still pointless work that produces no value for society.