There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result.
Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start.
Maybe what the author is trying to say is 'small-scale big data is dead, but big data chugs on.'
I kind of doubt they trained chatgpt on petabytes of application logs and web server logs. Is keeping all of this crap even useful for more than a small amount of time at this scale?
Actual good information will always be useful, most of this "big data" seems to be the equivalent of recording background static.
>ChatGPT is literally changing how school will test their students, for a start.
Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.
All math homework through the high school level is now as simple as figuring out how to describe it to ChatGPT (or maybe ChatGPT 2.0 for particularly tricky examples). Paper-writing is now a matter of figuring out how to rephrase LLM output in your own words to get around any watermarking or pattern detection.
Wolfram alpha has been around for math cheats (and people like me who just needed a more visual representation to learn) for a while now. Including proof of work.
Oh yeah, it's why most of my classes restricted us to TI-83s. The TI-89 was restricted in schools to basically calc and above, and the TI-92 was just banned. Lol
In my school TI-89s were unrestricted, but I think that was mostly because teachers were only trained with 83s and assumed the 89 had equivalent capabilities. The SAT permitting the 89 probably had something to do with it too, since the 92 was banned (because qwerty as I understand it.)
I mean, that's certainly part of it. But you're also seeing very fast restructuring of individual courses (because programs overall will definitely take years to restructure because higher education moves so damned slow) to account for these tools.
In the small institution I am currently working with, the English courses, in one week, integrated chatgpt as a tool for students to work with. It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes, and that happened in one week. I cannot stress enough how unbelievably fast that is for higher ed. That's faster than light speed.
And we're not even that well resourced. I have to imagine there are other examples where it's more than just running through a bot to scan for known outputs.
Sounds like they simply panicked and threw something together with little thought or preparation, what, right in the middle of an actual course? And they want to charge kids for this kind of 'expert instruction'? I'd be pissed as a student.
What a weird assumption you made there. In what way does what I wrote sound panicked? Because it was a week? Yes it was fast, but it was a massive effort of the entire english faculty.
It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; that was another strange assumption for you to make.
Just a tip, and don't read tone in this statement, but don't assume things. 9/10 times you're going to be incorrect. It's much better to ask questions, instead of making statements with question marks at the end of them.
I simply don’t buy that they had anywhere close to enough time to integrate this into an existing curriculum in a way that would meet the high standards paying students should have for a university education. I don’t have to ask how an English department became experts in using a just released AI in the classroom in a week because I don’t think that’s what they actually achieved.
Again, stop putting words in my mouth, please. I never said they became experts. I said they integrated it into existing processes. I said that they were doing something other than just scanning for chatgpt hits like plagiarism checkers.
>It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes
>It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class.
I don't honestly know how else to say that. I legitimately do not know how to help you understand what I'm saying.
Not who you are replying to. I found your example fascinating - would you be able to share one or two concrete examples of this integration you mentioned? I would like a low-level peek or two into how the teaching landscape is changing in light of the rise of LLMs like chatGPT.
Sure - I'll use one of the creative writing classes.
In the past, the class would be centered around ideas and themes the class came up with together during the first week of the semester. They would then read and discuss short stories from various authors centered around that theme, preferably from different eras and/or cultures. From there, they would work in pairs/small groups to flush out original ideas they came up with for their own stories based on the themes and styles from the first month of reading. Then they would work individually to write the stories. Finally, they would come back together to edit and work through that process as a group, with a reading and discussion in the last week or two.
Now, the class works alone the first day or two to discuss themes with chatgpt, to identify relevant and appropriate literature (if possible), and to flush out initial ideas for the readings and discussion topics. Then we come back together for group work to figure out themes for the semester and possible readings, like it used to be, but primed with whatever information they had already discussed with Chatgpt. From there they work through the readings, using chatgpt to bounce ideas for discussion off of before coming to class. Then they work in teams, like before, to flush out their own stories, but again, using chatgpt as if it were another member of the group. They also have to track their question/comment and response, to evaluate their own thought processes and look for weak links in their reasoning and logic. Students are then free to use the software to edit their writing before coming together with their creative assignments.
We haven't made it past the step of reading, as it's still the first month of the semester. But,the discussion for the first section has been much more in-depth and (to use a word that is impossible to quantify) vibrant. The students had already aired the ideas they thought may be dumb, and would therefore be less willing to voice in a public setting; this allows them to really dive further into whatever is in each story, and connect dots between stories that generally took a week or two. Because they have another 'person' to talk to whenever they want, and however much they want, they tend to really get into the work. Further, and unexpectedly (and anecdotally sadly) it has allowed a couple of students I know personally who would not have been willing to participate in public discussion (anxiety disorder and TBI) a stronger voice, because they know their ideas are flushed out already, and it provides them with a 'script' of ideas they know are novel and valuable for the class. In other words, if they were able to get the idea from Chatgpt, they knew they had more work to do to build that thought, because that is really just a baseline.
I'm interested to see how the actual writing process goes. Chatgpt is okay at creative writing, but not at the level that we expect to see. Some faculty expressed concerns that students would just have the software do the work for them, but I'm not really bothered by that. First off, if someone can figure out how to make a career out of publishing chatgpt prompts, well, good for them. Second, if the software is able to write better than the students are, they don't really belong in this class in particular (graduate level writing class).
Anyway, like I said earlier in the thread - we're really just treating this as another 'person' in the class, but who is available to everyone, all the time. It's beneficial for the brainstorming sections, but not for the hard skills, from what I can tell. I am most excited to watch them evaluate their own thought processes when working with chatgpt to flush out their ideas. In the past, this has been a barrier, because every single person I’ve ever met never gives a partner 100% of their thoughts all the time. They always hold back. My theory is that, because it’s a dead-end conversation, they will be more willing to push i...
Are we trying to produce adults who are able to think critically and creatively, and who reach their full intellectual potential, or are we trying to produce adults who can push a few buttons and blindly believe what the machine tells them?
> In fact, [writing] will introduce forgetfulness into the soul of those who learn it: they will not practice using their memory because they will put their trust in writing, which is external and depends on signs that belong to others, instead of trying to remember from the inside, completely on their own.
- attributed to Socrates by Plato. c.399-347 BCE. “Phaedrus.”
The point is that all technology is a tool. Whether it be writing, calculators, or various narrow AI software. We can either bemoan the loss of a now-less-useful skill (memorization, long division, longform writing), or learn how to use these tools to better achieve our goals.
Technologies develop for different purposes and have different effects. Modern digital technology has a very inhumane origin story, as noted in “New Dark Age” by James Bridle, and other works.
https://jamesbridle.com/books/new-dark-age
While likely not how it would be work out in practice, you would hope that with better tools would also come higher standards. If you expect more complex, more thorough, and/or less error-prone output from students using AI then you don’t necessarily have to lower how much critical and creative insight they need to have. Like the difference in a test that does and doesn’t allow calculators, you always have to fit the assignments to the tools that are used for them.
As long as it pushes the students into mental effort I have no problem with changes in education. The world changes, it’s normal to have changes. But there’s nothing wrong to compare against prior models, sometimes we revisit abandoned ideas, artworks, models etc
Isn't the problem that kids (like all of us) are lazy and would rather use tools than their brain, rather than people trying to keep school "just like it was"?
That's how we teach today, but that doesn't mean it's the only way. As a counter example, in chess, AI makes a great learning partner, and from beginners to world champions the level of proficiency has increased immensely in the last 30 years due to better chess engines.
It appears that it's hard to detect AI generated content. E.g. true detection rates are only around 25% and there are also techniques to further mask output [1].
You only need enough to flag suspect content, and then the teacher calls the student in for a quick oral exam - the fakers will flounder, the reals will pass.
Even if you understand the material well, doing assigned tasks takes a lot of time. Especially if it's free form text. And "I want to do something more interesting right now" is at least as powerful a motivator to cheat assignments as "I don't know how to do this".
Aye, if I mostly know a subject why bother to take the time to write 10 pages when I can outsource that to a bot, and if they question me I can do a 20 min oral interview and save myself the trouble?
> At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?
Careful with the scales. 2x improvement could be interpreted from 80% human performance to 160% human performance. Or going from 10% error rate to 5% (which again crosses into superhuman territory on some tasks).
Those last few bits are the critical ones.
We are! Don't even have to reach for fusion potentially being commercial technology to show it. Solar is already approaching $0.03/kilowatt hour and likely to be half of that by the end of the decade. Energy getting very cheap coupled with computing capacity continuing to go way up is going to enable lots of interesting new technologies beyond LLMs
Teachers assign better scores to papers with better penmanship. I forget how strong the effect was, but using a keyboard does help equalize some biases.
In Italy we test with pen and paper and the quality of italian schools is abysmal. The point is not the test, is the quality of the teachers. They are the only hope to form good humans, not some standardized test.
I don't even think we're just talking about children either. Test taking in academia (university and above) could stand a much needed fresh look.
I am hopeful that a change happens in academia to prepare students for jobs, which is why they are going to school in the first place. Yes, students need to learn how to "think", but really they are wanting to get the technical skills to perform their duties more than anything.
We have bestowed too much credence in traditional academia not useful to the average person or average job. College is a "game" for most students, and they put up going through the motions of testing, etc. for the sake of the diploma at the end.
I hope we're going to enter a new era of what college means for those looking to get something different out of it.
That's a completely different topic. "Data" is obviously a pretty generic term and "large sets of data" are going to be more and more relevant to the world in general. What he's talking about is the Big Data trend in industry specifically around Business Intelligence (BI). That is, collecting as much data as possible on your users to optimize your product experience and profits. Tracking clicks, purchases, form dropoffs, email opens, ad impressions. It's mostly going to be first-party data (ie what did they do with our own products and content).
ChatGPT and the like are not going to get much use from that kind of data and instead are looking at a giant corpus of text and images scraped from a variety of public sources to infer what humans might think sounds smart. It's possible the two worlds will meet, but that's probably not what's going to be announced this week.
Yes, this occurred to me as well. The counter narrative here is in fact that the story of the last 2-3 years has been break throughs in AI have come about mostly by scaling up their network sizes and training data sets 5 orders of magnitude or so.
I guess the take away however is still that regular businesses really just can't play in this game and should not be assuming they have big data until that fact asserts itself out of necessity rather than the other way around.
I am writing an essay series on this topic: last-mile analytics and how an abundance of data must be ultimately converted into (measurably correct) action.
If anyone wants to follow along, the series is here!
That looks like a huge undertaking, but kudos for taking the time. I'll be following along. Totally agree that all data should be tied to the business value that it's driving.
Unfortunately, I've found that many data teams focus more on making the data clean and available. They never drive the conversation about what actions are being taken with the data. That leads to them being treated as cost centers. Wrote a similar post about my perspective on it - https://bytesdataaction.substack.com/p/transform-your-data-t...
I'd love to chat about the space more with you if you're interested! Email in bio.
My cheap no-name old laptop SSD writes with 170MB/s.
A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach.
My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.
Presumably the "order" you mention is a primary key to another table, likely one that references the individual items that make up that order, so the data will be much larger than you estimate.
It will grow larger still if you include web logs from your e-commerce site and event data from your mobile app so that you can correlate these orders with items that customers considered but ultimately didn't buy. How will your laptop and SSD perform when you then build a user-item matrix to generate product recommendations for each of those 1.2 billion customers?
While plenty of organizations unnecessarily use Big Data tools to store and analyze relatively small amounts of data, there are plenty of customers with enough data to require them. I've seen plenty of them firsthand.
There are functionally less than 1000 organizations that currently require distributed compute for data analysis. You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. The cost of compute has decreased faster than the amount of data we have to store and process. What we used to do with spark jobs we can do with python on a single box.
I/O performance is just one of many characteristics that impact performance and from experience the one you least need to worry about. RAID 0 across multiple high-end NVME drives with OS file caching is going to be more than fast enough for most use cases.
The issue is running out of CPU performance and being able to seamlessly scale up/down compute with live running workloads.
But ... we are ... basically by definition. Vanishing little projects actually need cloud scale infrastructure.
And, to address your previous statement: one beefy server is actually pretty scalable. Soft threads spin up in microseconds to serve incoming requests, communication between threads is blazing fast, caching is simpler on one machine, etc. You don't even have to worry to much about scaling, the CPU just throttles itself when there is no load.
And every once in a while you just upgrade to the next gen beefy machine.
Even if this is off by two orders of magnitude and it's only 100,000 companies that need distributed compute, that means that almost all companies just need a single large computer.
Looking at the distribution of companies by employee count and assuming that data scales with employee count (dangerous assumption, but probably true enough on average), that means that companies don't need distributed compute until they get several hundred employees. [0]
This is not true. Any column store database (bigquery, Redshift, snowflake) implements distributed compute behind the scenes. When an analyst/business intelligence people have a query return in 3 seconds instead of 15 seconds, it's actually huge. Not just in aggregate amount of time saved, but in creating a quick feedback loop in testing hypothesizes. This is especially true considering that most analyst type people look at data as aggregates across some dimension (e.g. sales per month , unique visitors per region, etc...)
These types of questions are orders of magnitude faster with a distributed backend.
I was just playing with some data from our manufacturing system, about 30 GB. I pulled the data to my laptop (very expensive Apple one) and while it fits on my disk just fine, it took about 15 minutes to download.
I imported it to ClickHouse which took a while due to figuring out whatever compression and LowCardinality() and so on. I ran a query and it took ClickHouse about 15 seconds. DuckDB pointed to the parquet files on my SSD took 19 seconds to do the same. Our big data tool took 2 seconds, while working with data directly in cloud storage.
Now of course this is entirely unfair - the big data thingie has over twenty times more CPUs than my laptop, and cloud storage is also quite fast when accessed from many machines at once. If I ran ClickHouse or DuckDB on 100 CPU machine with terabyte of RAM it might have still turned out faster.
But this experiment (I was thinking of using some of the new fancy tech to serve interactive applications with less latency) made me realize that big data is still a thing. This was a sample - one building from one site, which we have quite a few of.
I'd love to understand the shape of this data and some of the types of queries you're performing. It would be very helpful as we build our product here at motherduck.
I have no doubt that there are situations where the cloud will be faster, especially when provisioned for max usage [which many companies do not]. However, there are a lot of these situations even where the local machine can supplement the cloud resources [think re decisions a query planner can make].
Feel free to reach out at ryan at motherduck if you want to chat more.
Let's assume your completely made-up 1000 organisations claim is true.
Right now I work for one of them: a global investment bank.
Within that organisation we have at least 100+ Spark clusters across the organisation doing distributed compute. And at least in our teams we have tight SLAs where a simple Python script simply can't deliver the results quick enough. Those jobs underpins 10s of billions of dollars in revenue and so for us money is not important, performance is.
So 1000 x 100 = 100,000 teams, all of whom I speak for, disagree with you.
Disagree with what? I never said _you_ are a dummy for using distributed compute. There are many good applications for distributed compute. I used spark and flink at a big tech job. The stack worked well for some things, and for others it was a hammer looking for a nail. What you do not see is that for every team that you work with and consider a peer group to you, there are 100 teams that really do not need distributed compute, because they have an org wide infra budget of <3M dollars and a total addressable data lake of less than 1TB, but they are implementing very expensive distributed compute solutions recommended from either a Deloitte consultant or a very junior engineer. Should an IB with an infra budget in the 100M+ infra budget zone use distributed compute solutions, absolutely. There just aren't that many of these orgs.
> You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc.
Hold your horses... the beefiest servers that are in production today, unless you count custom-made stuff go to somewhere between 128 and 256 cores per board. These are hugely expensive. Also, I don't know if you can rent those from Amazon.
Typical, affordable servers range between 4..16 cores. Doesn't matter if you buy them yourself, or you rent them from Amazon. It's much cheaper to command a fleet of affordable servers than to deal with a high-end few. This is both because the one-time price of buying is quite different and because with smaller individual servers you have a fighting chance to scale your application with demand. Especially this is true in case of Amazon as you could theoretically buy spot instances and by so doing you'd share the (financial) load with other Amazon's customers.
Now... storage. Well, you see, in Amazon you can get very expensive storage that's guaranteed to be "directly" attached to the CPU you rent, the so-called ephemeral storage. This is the storage that's included with the VM image you use. It's very hard to get a lot of it. I couldn't find the numbers for Amazon, instead, I know that Azure tops out at 2 TB. In principle, this kind of storage cannot exceed a single disk, so, think Amazon probably offers the same 2 TB, maybe 4. But, again, it's cheaper to have a bunch of EBS's attached... but then you'll have to have more of them as the latency will suffer, and in order to compensate for that you would try to increase throughput, perhaps.
Also, think that, in practice, you'd want to have a RAID, probably RAID5, and this means you need upwards from 3 disks. Also, if you are using something like a relational database, you'd most likely want to put the OS on a single device, the database data on a RAID and database journal on a yet another device, and, probably, you'd want that device to be something like persistent memory / optane / something from higher-tier disks with dedicated power supply. And all this is not due to size, but due to different contingencies you need to have in order to prevent huge data loss... Now, add to this backups and snapshots, perhaps replication in 2-3 different geographical areas if you are running an international business... and that's quite a bill to foot.
There are similar problems with memory, since there can only be so many legs on memory bus and only so many pieces of memory you can attach to a single CPU, and if you also want a lot of storage, then, similarly, there can be only so many individual storage devices attached and so on.
Bottom line... even to reproduce the performance of your laptop in the cloud you would probably end up with some distributed solution, and you would still struggle with latency.
Azure has the LS series of VMS [1] which can have up to ten 1.92TB disks attached directly to the CPU using NVMe. We use these for spilling big data to disk during high-performance computation, rather than single-machine persistence, so we also don't bother with RAID replication in the first place.
Though it is a bit disappointing that while Microsoft advertises this as "good for distributed persistent stores", there are no obvious SLAs that I could rely on for actually trusting such a cluster with my data persistence.
Well, attaching 10 PCIe devices is going to give a very hard time to your CPU if all of them should be used. The speed of copying from memory or between devices will become a bottleneck. Another problem is that on such a machine you will also need huge amount of memory to allow for copying to work. And, if you want this to work well, you'd need some high-end hardware to be actually able to pull that off. In such a system, your CPU will prevent you from exploiting the possible benefits of parallelization. It seems beefy, but it's entirely possible that a distributed solution you could build with a fraction of the cost would perform just as well.
This situation may not be reflected in Azure pricing (the calculator gives 7.68 $/h for L80as_v3) since if MS has such hardware, it would be a waste for it to stand idle. They'd be incentivized to rent it out event at a discount (their main profit is from traffic anyways). So, you may not be getting an adequate reading of the situation, if you are trying to judge it by the price (rather than cost). But, this is only the price of the VM, I'm scared to think about how much you'd pay if you actually utilize it to its full potential.
Also, since it claims to have 80 vCPUs, well... it's either a very expensive server, or it's, again, a distributed system, where you simply don't see the distributed part. I haven't dealt with such hardware firsthand, but we have in our DC a Gigaio PCIe TOR switch which would allow you to have that much memory (in principle, we don't use it like that) in a single VM. That thing with the rest of the hardware setup costs some six-digit number of dollars. I imagine something similar must exist for CPU sharing / aggregation.
> The high throughput and IOPS of the local disk makes the Lasv3-series VMs ideal for NoSQL stores such as Apache Cassandra and MongoDB.
This is cringe-worthy. Cassandra is an abysmal quality product when it comes to performing I/O. It cannot saturate the system at all... I mean, for example, if you take old-reliable PostgreSQL or MySQL, then with a lot of effort you may get them to dedicate up to 30% CPU time to I/O. Where the reason for relatively low utilization (compared to direct writes to disk) is the need to synchronize that's not well-aligned with how the disk may want to deal with destaging.
Cassandra is in a class of its own when it comes to I/O. You'd be happy to hit 2-3% CPU utilization in the same context where PostgreSQL would hit 30%. I have no idea what it's doing to cause such poor performance, but if I had to guess, some application logic... making some expensive calculations sequentially with I/O, or just waiting in mutexes...
So, yeah... someone who wanted Cassandra to perform well would probably need that kind of a beefy machine :D But whether that's a sound advise -- I don't know.
Don't forget the cool JS library you included to track mouse movements so you can optimize your UI to make sure Important Money Making Things are easily clickable.
That's 8.4 hojillion megabytes per second right there.
It never sat well with me that none of the production services could leverage my local computation and storage power. I don't need to store my contacts on a remote server that could index my contacts when mixed with every other contact in a single table. That’s a blatantly oversimplified example but you get the gist.
Developing apps as local-mostly with remote being "just storage" might've been interesting approach but oh so many stuff moved to webshit from native apps and browsers still don't even have decent data management.
I don't see incentive to host a bunch of stranger's stuff on your machine. The moment you make it easy and "bulletproof", the bad kind of content nobody wants coming from their IP will come with it.
One of my "computers are really fucking fast" experiments, almost a decade ago, was when I was trying to do a histogram plot of a function that I was 98% sure was terribly broken. It was expected to give a uniform distribution so I figured I'll just plot a bunch of values into a 2d space and then convert it to a greyscale image.
At first I tried to puzzle out a good sampling strategy to make sure I didn't bias the output, then on a whim I tried 2^32 samples and went to lunch. It took something like a half an hour to do 4 billion samples. Took me a couple times to figure out how to squeeze 4k megapixels into a graph so I ran it a few more times, but the results showed a very distinct banding pattern that confirmed that the problem was every bit as bad as I suspected, which was a blocking issue for our release. A couple of hours well spent, running through an 'intractable problem' that really wasn't.
Big Data is far from dead. On the contrary, people (on most daily projects) are more mindful now wrt all Big Data liabilities and benefits (infrastructure cost vs. what you get from it) thanks to the experience of the failed ones. But many analytics companies are thriving.
Also, using BigQuery as a metric of how Big Data is used is, IMHO, wrong. Real analytics companies usually have custom solutions because BigQuery is too expensive for any serious usage unless you are Google.
MongoDB grew revenue 52.8% in the previous financial year [1].
And if there is any levelling off it's going to be because of the move towards cloud managed options e.g. Snowflake, DocumentDB rather than because PostgreSQL decided to add JSONB support.
"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size."
The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
"customer obsession" was always at the mercy of the real obsession: "making money hand over fist". The former will ALWAYS lose out to the latter given enough cycles.
Yeah, "customer obsession" really just means "market share / growth obsession" which is a means to (eventually) making monopoly profits. Which Amazon seems to have achieved.
You can also “coast upwards” for quite awhile - as an example we now pay Microsoft less than we were for periodic upgrades to Office for the entire Microsoft 365 suite (including email hosting, etc) but all the machines are now Macs. They make more from us in one way, but less in total dollars.
All the amazon corporate values are derived from making profit. Two-pizza teams? More like "three slices isn't frugal", one or two should be enough for you.
I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.
Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).
Data Scientist's job is to launder management's intuition using quantitative methods :)
This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.
We totally are, it's just that rationality is a tool, not a guide. If you want to work out the truth, rationality will help you do that, but if instead you want to justify a decision you already made, well, it'll help you do that too.
Hypothesis don't come from rationality either, they result from well informed intuition. All of the formality of science is about tricking ourselves into discovering our intuition is wrong using a rational series of steps even when everything in our nature is to use that ability to reason to do the opposite.
Thats because psychologists dont understand or choose to ignore how chemistry influences our personalities and emotions. An extremely simple example from the same medical/health profession is the use of SSRI's to make people feel happy. The legal system recognises how chemicals influence our feelings because of the laws that exists on illegal drugs or drink driving.
The definition of rational is being informed enough to know what said chemicals will do in the short term and long term in order to make an informed decision, but then I'm reminded we dont get taught any of the above unless we specialise at a Uni, so most people cant make any sort of informed decision.
I think the answer is simpler: people care about their careers and their family first. Think, "If the data says something that gets in the way of my career well I don't care about the data."
Had the same problem when I was an economics researcher -- publication bias for what stakeholders want to hear (often the government) is rampant because that's where funding for the economics department mostly comes from.
It's only rational. The company certainly doesn't care about that individual first, as evidenced by e.g. its decision to lay them off when it doesn't think the individual is serving them, so why should the individual put the company first?
This is also known as The Iron Law of Institutions.
Huh. I've not thought of it as laundering, but I think you've basically summarised consulting in healthcare. Pay to legitimize and push through a pre-existing idea (eg let's close down a few ERs) or a delusion (e.g. lean, we don't need a waiting room) and say it was recommended by consultants to stakeholders and the public.
all consulting is like that, Partners/MDs at consulting companies meet with Board/CEOs to get rough idea of what they want/need, and quickly negotiates a consulting engagement contract to create PowerPoint with all the evidence and analysis gathered that supports CEO's initial idea.
This is the only reason why a 60+ PowerPoint slide deck can cost several millions dollars
Economics is the go-to for conservative, status-quo maintaining arguments because there's a wealth of statistics and information available for "how things have always been", and precious little-to-none for "how things could be if..."
It's easier to poke holes in predictions of the future than in interpretations of the past, especially when the people making those decisions have likely reached their decision-making status through "how things have always been".
That's like a portrait artist that finds success by painting people more beautiful than they really are vs a starving one that paints them true to life due to sense of artistic integrity.
Reminds me how Garth Brooks started doing metal after becoming a country music star.
> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
Parents can be listening to their kids 99% of the time it will be transparent and uneventful. When there’s confrontation/divergence in opinion, by definition it didn’t work out through the usual channel, and of course a third party weighting in the balance will have visible effects.
My Mother-in-Law was called by a tech support scammer. Her bank was unwilling to accept their charges, and the scammer wanted her to call the bank to tell them to accept them anyway. My Brother-in-Law was telling her "no, this is a scam, do not do this", but she was unwilling to listen. Eventually, he told her to call me, thinking if she wouldn't listen to her son, maybe she'd listen to her son-in-law. Which she did.
As a consultant with roots in backend dev, I fully understand the scrutiny that we receive because unfortunately, it is often very warranted... It feels a bit refreshing to read your comment and see someone articulate what I am trying to convey to my clients. I am a tool, and yes, this pun is intended.
Sure, it is actually not very complicated in my case. I did backend development for a short while during and after university and then moved into IT consulting fairly quickly.
It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most junior level and joined my first project with 3 very senior colleagues after a few weeks.
The learning curve was very steep both on the technical level and also regarding the consulting aspect - at first there was nothing I could 'consult' on due to lack of experience. This changed with growing experience, with the guidance of senior colleagues and my private efforts to gain skills and expertise.
This almost reads like my trajectory so far, but I'm at the point where I can't really consult due to the lack of experience, but I did make a good impression so far. Can I ask you, into what efforts should I put my private time? More technical knowledge? Into very fine details, or brief insights into different areas? Any good resources?
Here's my take on this not listening to the "expert":
A few years ago there was a problem with storm-water infiltration into my (elderly at the time) mother's property from her neighbor. I, being a dutiful son and a civil engineer, investigated it and came up with the probable cause, the likely effects of non-action and the most cost-effective solution. I presented it to my mother in the most layman-like terms that I could. She said she'd think about it – meaning she'd refer i.e. defer – to her daughters. In the meantime I had a very layman-like chat with my mother's carer and told her the situation in layman's terms. The carer listened and said that what I said I made total sense to her. Later on, one of my sisters accosted me and stated that it was completely obvious what the problem and the solution was – "even the carer could see it". Human foreheads don't have the real estate for where my eyebrows wanted to ascend.
My advice is to consider whether the message should be separated from the messenger somehow.
This could be a reason why Data Scientist as a job title exploded in last years, every middle manager could afford one/two/few headcounts of data scientists to produce analysis that advances that middle manager's corporate agenda (more growth, empire building, expansion to certain de-novo areas, etc).
Recent tech layoffs is the other side of that growth, when cheap money is gone and company is forced to stick to core competencies and shutdown growth plans
In the news business, if your story or opinion backs up the preconceived notions of the investigative reporter then you are a 'source' otherwise you are a 'conspiracy theorist'.
Also a big reason McKinsey and BCG exists - provide cover for business plans intended by management to protect them from shareholder lawsuits. My friend did a sojourn at McKinsey and 6 months of his life was producing PowerPoints and memos backing up an expansion to AIPAC region. Was already happening but he was providing all manner of business justification for board meetings and whatnot.
I wonder what a data scientist could really find out about executive (over?) compensation. employee compensation. working from home. office cubicle size and layout. tool expenditure for employees vs productivity.
As a data scientist at a large corporate I find this is often the push… but I resist every time and tell people what they don’t want to hear. Maybe I’m playing this whole corporate ladder thing wrong :/
I’ve been there, we wanted to release a feature, it kept coming back with issues that made it perform much worse than control, after 5 or so iterations with bug fixes it came back positive.
It took a lot of analysis and time to clarify to higher ups that we weren’t just P-hacking , but at least they were concerned about that.
> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).
> Data Scientist's job is to launder management's intuition using quantitative methods :)
It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if he proceeds with whatever he already wants to do.
The beards might be a bit shorter or nonexistent, but the story hasn’t changed.
Absolutely. If you don't like what K-Means is telling you, change a variable and re-run. (that's one great thing about business data: there's no shortage of variables! True, there's usually a shortage of independent variables, but fixing that is difficult and underfunded).
And you'd better hope the bones actually say something useful.
I was the infra lead on a data lake project and got take part all the way to breaking down the data and turning into PowerBI reports. The result was "sell more" and to clients who marketing already identified, years ago, as whales.
There were some interesting other insights, esp. w/r/t to niche products that sold around weird dates (Easter, Memorial Day, 4th July -- but not obvious gift days like Valentines or X-Mas), but it led to a lot of "you're doing it wrong!" recriminations and follow up projects.
I think this depends a lot on the org. In a place I used to work we collected and analysed a lot of data which convinced management to significantly change the spec of the product and spend a lot more time and effort on testing, because the product was being used in unexpected ways.
I would say it was a very engineering driven org however, so if you could present compelling data it could go a long way.
This phenomenon is true to varying degrees in academic medicine (maybe all of academia) as well - personally have seen excellent data and methods disregarded when they don't confirm existing agendas. The choice for the researcher can become one of burning out trying to do good work and getting nowhere, or acquiesce and only present data that is uncontroversial. Huge existential threat to knowledge advancement.
> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
I mean, that makes sense does it not? If you're confirming something people already had a hunch about, why would they challenge it? And if it does go against their belief, they are going to want to make sure the data is correct before they change the course of the ship.
Yes.
I worked in the data org of a moderately sized financial firms tech org.
The tech org claimed to be hugely data driven. Was in the org mottos and all of that.
Nonetheless, the CTO went on a multi-year, 10s of millions of dollars, huge data tech stack & staffing reorg shake up... with really zero data points explaining the driver, or what we would measure to determine it was successful.
So it became a self referential decision that we are successful by doing what he decided, and we are doing it because he decided it.
how is this really different from any other aspect of life? Very few people really like to be told counter information, and it is always easier when providing data that aligns with the current group think. Doesn't matter if it is business, politics, or really anything. Being the outlier trying to change the direction of things is a struggle.
> Data Scientist's job is to launder management's intuition using quantitative methods
Ouch. This is savage, but sadly correct in many cases.
HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to happen.
The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.
Positive claims are about what is true. Normative claims are about what should be true, or rather what decisions we should make. Put another way, positive claims deal only with facts while normative claims deal also with values.
GP is saying that it's the data-scientist's job to give the executive the facts and it's the executive's job to decide what to do about the facts.
> The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.
Where do business analysts fit into this dichotomy? Their whole job is to poke around in Tableau in order to surface high-ROI strategies for the business to pursue. (Where, in choosing which proposals to surface to management, they're effectively making 90% of the strategic decisions.)
Or how about corporate buyers in trading and retail companies?
People who poke around in Tableau might not get a lot of respect in the hierarchy of DataFolk, but descriptive statistics and thoughtfully chosen visualizations can be immensely useful. Exploratory data analysis sometimes reveals patterns that are so obvious that to apply statistical inference is just vanity.
If understanding the data generating processes is the goal, I'd rather see some useful plots than wade through a technical description of some model whose assumptions were flagrantly violated.
Good point. Data is one aspect of making a decision. The other aspect is understanding the industry and environment. Often data scientists give just one variable needed to make a decision. In health care for example you need to factor in a whole host of legislation. You also need to factor aspects of the industry not reflected in the data. As an example doctors not wanting to use iPads is something you can't measure and can't force as company. Even though data analysis might suggest this is the way to go.
This isn't just "Data Scientist" but scientist as well. The more a finding is in contradiction, either with existing scientific consensus or even with just popular culture, the more the science is criticized. I've seen unequal criticism based on how much people wanted the results to be true/false and even after responding to the criticism I've seen people just ignore science they don't like.
The skepticism isn't a problem, the unequal application of it, the potential to harm careers, and the chilling effect as people wisen to how best meet their own personal goals is.
> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.
>The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.
Sadly true. As humorously depicted here[0]:
In the beginning was the DEMO Project. And the Project was
without form. And darkness was upon the staff members
thereof. So they spake unto their Division Head, saying, "It
is a crock of shit, and it stinks."
And the Division Head spake unto his Department Head,
saying, "It is a crock of excrement and none may abide the
odor thereof." Now, the Department Head spake unto his
Directorate Head, saying, "It is a container of excrement,
and is very strong, such that none may abide before it." And
it came to pass that the Directorate Head spake unto
the Assistant Technical Director, saying, "It is a vessel of
fertilizer and none may abide by its strength."
And the assistant Technical Director spake thus unto the
Technical Director, saying, "It containeth that which aids
growth and it is very strong." And, Lo, the Technical
Director spake then unto the Captain, saying, "The powerful
new Project will help promote the growth of the
laboratories."
And the Captain looked down upon the Project, and He saw
that it was Good!
Authoritarian types consider any information derived by science which is contrary to their position as invalid or irrelevant because facts challenge their authority and ability to exercise control.
yes. I used to think the Church had a honest disagreement with Galileo about heliocentricity. When I grew up I realized the Church never cared about orbits at all, what they care about is maintenance of status quo.
And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so in their mind, they are doing the right thing.
The church was not upset about heliocentrism. They were upset that Galileo was attempting to reinterpret the words of Bible in order to bolster his astronomical authority.
"[Therefore,] when God willed that at Joshua’s command the whole system of the world should rest and should remain for many hours in the same state, it sufficed to make the sun stand still. In this manner, by the stopping of the sun, the day could be lengthened on earth—which agrees exquisitely with the literal sense of the sacred text.”
"The Church" wasn't then and isn't now a monolith of opinion.
A modern characterisation of "The Galileo Affair" would be that he was SWAT'ed by someone he was really really mean online to.
Thus the whole "Galileo affair" starts as a conflict initiated by a secular Aristotelian philosopher, who, unable to silence Galileo by philosophical arguments, uses religion to achieve his aim. [1]
and
While delle Colombe was almost alone in arguing publicly against Galileo, there was a group of scholars and churchmen who supported his Aristotelian views. After Galileo referred disparagingly to delle Colombe as 'pippione' ('pigeon'), his close friend the painter Lodovico Cigoli coined the nickname 'Lega del Pippione' ('The Pigeon League') for delle Colombe's group.[2]
Galileo literally refered to delle Colombe (and friends) as Simplicio (simple minded) and worse in his highly popular Dialogue Concerning the Two Chief World Systems [3] and within a year or so the Pigeon League got their revenge, using their influence to have religuous charges bought against Galileo.
The affair was complex since very early on Pope Urban VIII had been a patron to Galileo and had given him permission to publish on the Copernican theory .. this was very much a case of personal vendettas and internal politics rather than a straight up case of "The Church Versus Galileo".
>>I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.
>> Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
It's a good sign at the company that I run, anytime our analysts/data scientists come up with metrics that say we're killing it, or that our ideas should bear a ton of fruit, the kneejerk reaction is to be extremely skeptical of the results. Usually they're still right.
When the data scientists say we're fucking something up, we tend to pay a lot more attention.
I found it incredibly stressful to discover and provide analyses (even experimental results) that wasn't expected, or contradicted prior beliefs. The findings were always very harshly scrutinized, and typically lead to tons of pointless extra work to 'understand what is going on'.
My friend runs a successful market research agency and she says she gets called in when management have decided they need to make a change but need evidence to sell it to the shareholders and staff.
"data" is usually just twisted to make leadership look good or justify what decision they wanted to make anyway. Analytics data is sliced at arbitrary time periods to make growth in whatever metric look good, certain subsets are just removed, etc.
doesn't help that most of this data goes through multiple layers of BS where each person is putting it through filters to make themselves look better. And a good chunk of people don't have enough understanding of stats to understand when they are being tricked
Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc.
Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.
Exactly. It isn't just time to set up all the data in a way that makes the right query possible. It is also having queries fast enough to be able to run a vast number of them in order to find what you are looking for (or even things you were not looking for).
It’s moving the data around that is slow… and expensive. Getting the data into the data warehouse, then getting it to the processors then moving it around to filter and transform.
Getting your data to the cloud is expensive, but then you can’t do anything with it because distributing it to process in multiple stages is too expensive and you’re already paying so much to keep all that useless data.
Agree with some of what you've said, but disagree with a lot:
> Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
100% Individual humans are emotional creatures with their own wants and needs, and it's important to understand how organizational incentives drive decision making.
> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven".
This has not been my experience, though. The more common thing I've seen is that, sometimes data is boring and doesn't really show much actionable insight, but as everyone wants to justify their job, I've seen data scientists come up with really questionable conclusions that fell apart on further inspection (call it "p-hacking the enterprise").
Plus, a lot of this data in these data wearhouses is messy. Often times data scientists are siloed at the end of the process, but then you get "garbage in/garbage out" results, where there is some bug in data tracking that isn't understood until it's too late. Much better in my opinion to have data engineers and data scientists work much more closely with product engineering teams up front so they can help ensure the data they collect is accurate.
I find a lot of organizations don't have the discipline to harness whatever power their data may have. Sure collect everything, but god forbid you have any sort of data governance, or spend a single resource minute of time manually tagging or organizing or validating it. Then they try to make shitty ML models or products out of it, but don't care if the models actually work or not, just that they have AI now. Then a year later when the model has provided no value they are like, oh well big data is worthless I guess.
Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile.
Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been paid handsomely, while some higher ups can now put "implemented agile and big data at X" on their LinkedIn profiles.
I'm one of those PhDs with zero domain knowledge analyzing data and I share the sentiment.
Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (like asking for average of something that follows a very fat tailed distribution with a few observations dominating the phenomenon. Of course it's usually impossible to explain this to people).
The more I think about this, the more I think that in truly data powered companies, both the decision making and data analysis have to be carried out by more or less the same people. The organizational hierarchies have to be much flatter. Essentially the employees will have to be some kind of "secret agents" who have both the skills and the mandate to steer the company in the direction they see fit. I sort of see this already happening in the FAANG companies where, or so I hear, it's very difficult to get hired, the staff count is quite small compared to traditional companies and the senior engineers have a lot of power in the company.
Using math PhDs or Palantir or whatever as a sort of modularized black box for "insights" while giving them no real skin-in-the-game does not work.
I can confirm this way of worling with data from my time at Amazon operations. We used data all the time, everywhere and for everything. But we did not have data scientist in our time, we did it ourselves. Quite peculiar, but so damn effecient and effective. I kind of miss that. It also showed that most of the data analytics stuff I know, Six Sigma, is just plain overkill for a lot of practical applications.
My favorite example is the WW2 bomber diagram shown to illustrate survivorship bias. Sure, working from data and first principle one could identify the vulnerable spots of the bomber. Or one could asl the designers or have an engineer, heck even a contemporary field mechanic, take a look at the actual drawings of the plane. And reach the same conclusion, faster, with a ton of additional insight and improvement ideas that can actually be implemented...
Agree on the size not being the issue. I transitioned from a data engineering manager to a data product manager at a new company. You know how much data a typical customer generates? Less than 1TB a year.
I told my VP that the engineering foul ups in the current product are easily fixable. Standard tooling and patterns exist to re-architect and solve the bottlenecks. What is much harder is a data architect to make sense of the complex data and make sure there is good value for our customers.
Guess what position I don't have on the team, and won't have due to budget issues.
The other thing to consider is that data simply has nothing of value. Part of the marketing of big data is the almost fairy tale belief in "insights" existing in any data set if you just look hard enough.
Correct, much data has no value. The cost for storing the data, maintaining the data [in the day-and-age of privacy requirements especially], and combing through the data is often much greater than the value obtained from the data itself.
The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)
Replace “data scientists” with “software engineers” and you have another accurate insight. They don’t want to listen to us about how to write software or derive value from data.
I think the recent industry layoffs reflect that in part.
Though I think there’s a better way— that is executive data science, what I do at Zapier. The key is that I’ve built up a huge amount of econometric and economic / business skills that I apply to affect good change in collaboration with company leaders. It allows me to work with Execs using sensible analysis. I improve growth and output by helping us catch errors of assumption before they go into production and cost growth / bad surprises. I also help the executives gain alignment around good information. This multiplies their departments’ output by allowing them to work better together, more in concert. That helps avoid issues with data being bent to decisions.
They typically carry a lot of hard-won valuable domain knowledge (that I combine with my economic-statistic knowledge and skills for rigor).
It’s my job to ensure Execs start with good sensible information regarding objectives. They usually ask fantastic questions and share a lot of great analysis of their own.
There are times I learn about what I might call controversial implications. This is typical of innovation using technology. It’s in these moments that I feel I create the most value by highlighting the trade offs I believe we face / potential regret.
I get to apply my causal inference and measurement skills to a lot of compelling situations in collaboration with some really intelligent and skilled people. I truly feel like I get to perform technically excellent work, and I feel very privileged in that way.
My day to day is mostly economic / econometric analysis, conceptualization and formulation of technology, and writing, but also occasional meetings to gather information, feedback, answer questions, teach and collaborate with others.
I try to effectively “sample” the org by interacting with people up and down the org, so I can assist Execs in incorporating directors, managers and ICs knowledge / pool good information / systematize planning, etc.
What “data driven” normally looks like: business people ask questions that are not answerable, but the DS team often goes off and tries to answer them anyway. I’ve been in so many meetings, in multiple organizations, where a senior leader asks a question like, “why did this number go up? We need an owner to deep dive this.” Then, someone disappears for a few hours or days and comes back with some nice narrative that more or less might make some sense, and is directionally consistent. Then, best case, they go “hmm, ok” and worst case, they say it’s an interesting view and we need to add it to the tracking. Then, move on to a new question + action item, rinse and repeat. And then, 6 months or a year later, some brave soul goes, “geeze, there are way too many things to look at here, can we streamline this reporting to focus on the handful of most important metrics, and then assign owners to review the others and surface anything interesting?” And then all those things people asked for over time get stuffed into an appendix or deleted, and the whole thing starts all over again.
> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven".
I worked in a "data-driven" company of 5,000+ employees, with 4 data scientists who were spilt into two separate non-collaborating teams. In effect, they were so under resourced they got nothing done.
I would argue that insights from statistical models “[are] just one parameter in a complex equation.” No doubt PR from political issues can help the bottom line of a company. Can also hurt it. Just to be clear I’m not arguing political choices are inherently good or bad.
I've found this to be true in the (albeit small) # of samples I've run up against.
There's an agenda by some level of management, and they use "data" to forward their agenda, or disregard it due to "unexplainability" (a legitimate concern for BD/ML) if it disagrees with the agenda.
I agree with a lot of the sentiments of the MotherDuck people, but boy are they loud and proud for someone who never delivered anything more than blogposts and vague promise to somehow exploit the MIT licensed DuckDB.
Meanwhile for example boilingdata.com seems to have already done that - by using AWS Lambda + DuckDB as distributed compute engine which I can't decide if its awesome, deranged or both.
I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm).
Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing about these is that once they've been observed and understood, the value of ongoing observation drops rapidly.
A great example of this is soil moisture sensors. Every farmer that puts these in goes geek-crazy for the first year or so. It's so cool to see charts that illustrate the effect of their irrigation efforts. They may even learn a little and make some adjustments. But once those adjustments and knowledge have been applied, it's not like they really need this ongoing telementry as much anymore. They'll check periodically (maybe) to continue to validate their new assumptions, but 3 years later, the probes are often forgotten and left to rot, or reduced in count.
Most of the time this story is true, but think this way, the person that was using the system was an expert on the subject. If you can replace the expert with a person just looking at a graph from time to time to know if you have to irrigate the soils it's a different thing. Most of the data or ML tools show us something that the client as an expert already knows, but the true power of this tools is to give them to a non expert user and have roughly the same level of proficiency
I think there's a problem at the heart of the matter, specifically the idea that the act of measurement is in itself powerful when in point of fact that this isn't universally the case. As the old adage goes: "garbage in, garbage out." Even more troubling, there is a physical limit to our ability to model what we measure. Take the retina, it has around a million light receptors and even if you assumed they only have two valid states then you're left with around 10^300,000 bits of information to process, so good luck with that. Same thing applies to whatever firms are measuring and what they think is conveying relevant information as they'll have similarly exponential increases if they don't filter out the vast majority of irrelevant data points and states.
> it has around a million light receptors and even if you assumed they only have two valid states then you're left with around 10^300,000 bits of information to process...
That would only be a million bits (1 Mb). You're counting potential states, not bits.
I've observed the same in manufacturing … and fitness trackers a la FitBit.
There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter.
Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few new operational tools in their quiver.
Same is true of fitness trackers: you excitedly get one, learn how much you really are sitting(!), adjust your patterns, time passes … then one day you realize you haven't put it on for a week. It stays in the drawer.
Not unless they're threatened with ruin will people make changes to the standard way of doing things. This is actually … not bad! Continuity is important, and this is kind of a subconscious gating function to prevent deviation from a proven way of working. So, the change has to be so compelling or so pressing that they're forced to. Not a bad thing.
While we think things change overnight in this world, they generally take awhile … stay patient … it's worth it.
I used to track sooo much health and fitness data... Then I realized it mostly wasn't actionable, or at least, I wasn't altering my decisions based on it. The answer was always, "more training." So I stopped.
I think they are only useful to runners, where weekly distance, speed and heart rate are massively useful. These are all very accurate things that can be captured, unlike weight resistance (swinging a kettlebell, dead lifting).
I went on a diet a few years ago. I obsessively recorded every food I ate in MyFitnessPal. To this day, I know roughly how many calories pretty much everything I eat is. So, I've learned from the process and don't need the process as much any more. (I'm kidding about that - it's easy to underestimate how much you eat, and an extra 200 cal a day adds up over the years.)
I did something similar in my early 20’s. 15 years later the weight is gone and hasn’t come back. It is pretty nice being able to eyeball my plate of food at a BBQ knowing it will not add any flesh to my mid section.
I definitely did not need a multi-PB storage array to reach that goal. Or did I? I am sure someone out there knows roughly the human brain storage capacity in MB.
This is interesting, and I think you're on to something. I got a fitbit when I got serious about running because I had no idea what it felt like to run in zone 2. I can read about it, but that only gets you so far. While running I could actively check my heartrate and adjust. A year into the hobby and now I don't bring my fitbit on probably 50% of my runs. I know how fast I need to go.
Where as a Garmin is a useful ongoing tool for me a runner. I have got through about six watches. I don't use gimmicks like step counters though, turned it off. "Congratulations you completed 10,000 steps!" - well duh, I ran 10 miles this morning.
if youre running 10 miles per morning, even half that, on the regular you're an extreme outlier. the vast, vast majority of folks don't get anywhere near that, and a step counter that gamifies making them move is a good thing.
Oh, those soil moisture sensors, they are so fascinating.
I spent a number of exciting year developing a high frequency soil impedance scanner and finally understood why I was doing it. To confirm the obvious :)
But isn't this the essence of industrialization and automation? Messure, adjust process; repeat until feedback loop is stable - document and keep doing the thing that works, over and over?
If you want Toyota style continuous improvement you would need to improve in new areas of the process / new metrics, most of the time?
Very interesting. Left AgTech last year but had similar experiences, even worse where often the single most prominent use-case was to follow some painful necessary documentation of ag inputs (chemical, seeds, fertiliser) to get subsidies.
Real inputs from data? Nah!
I tend to think the problem is the "random digging for correlations" part.
Having tons of data is a Good Thing, so long as you can afford the marginal cost of gathering and managing all that data so that it's ready at hand when you need it later.
It's how you use the data that makes all the difference. If you're facing an issue you don't understand at all, don't go digging for random correlations in your mountain of data to find an explanation.
Think like a scientist: you need a valid hypothesis first! Once you have a hypothesis about what your issue might plausibly be, then you make a prediction: "If I'm right, I suspect our Foobar data will show very low values of Xyzzy around 3AM every weekday night". Only then do you go look at that specific data to confirm or refute the hypothesis. If you don't get a confirmation, you need to go back to hypothesizing and predicting before you look again. You can't prove causation by merely correlating data.
> It's how you use the data that makes all the difference. If you're facing an issue you don't understand at all, don't go digging for random correlations in your mountain of data to find an explanation.
Absolutely. But in my experience, there's this massive trend across the tech world that flat out rejects the value of domain/subject matter expertise. Instead, all you need is an engineer who can throw some ML at the uncurated mountain of data your organization has collected. Little to no value is placed on the resources that can frame an actionable hypothesis, even though the entire value proposition arises from this exercise!
Maybe I'm just jaded. I end up wasting a lot of time trying to re-direct data scientists and engineers down more appropriate pathways than if the problem they're solving was just brought to my attention earlier. Sorry, I understand you spent two weeks shoe-horning dataset X into our analysis system for your work, but it's invalid for the question you're asking - use dataset Y instead, and you'll have an answer in an hour or two.
> But in my experience, there's this massive trend across the tech world that flat out rejects the value of domain/subject matter expertise. Instead, all you need is an engineer who can throw some ML at the uncurated mountain of data your organization has collected. Little to no value is placed on the resources that can frame an actionable hypothesis, even though the entire value proposition arises from this exercise!
Sounds like the data scientists need to get together with the MBAs and they can do companies where nobody needs to actually know what they're doing.
This!!!!
The amount of data scientists i come across who don't get this. If you're not knowledgeable enough to form reasonable hypothesis about the data, you have no business touching data
Fine-grained measurement is useful when you have options for fine-grained action.
You don't need a chip to tell you that the soil is dry, but if you can use that chip to regulate drip irrigation that can apply substantially different flow to different plants, then you can get a not-too-much, not-too-little watering even if you have a big variation in conditions.
You don't need a big analysis to acknowledge that everybody knows that a particular competitor has lower or higher prices and adjust your pricing; but doing that continuously on a per-product basis does require data and analysis.
Agreed. But how many executives will agree to take these fine-grained actions to achieve value from the data? How many data teams are able to build up a strong-enough argument to convince them?
I've worked on many product-led-growth initiatives in the software industry. The software industry is probably the biggest 'believer in data' there is -- many scientific-forward minds who understand the value. However, even in the software industry, it's really hard to convince folks that if you make 5 improvements that net 1% conversion gain each, you can dramatically improve revenue.
On the flip side, there's some great action coming from data insights. Look at Strella Biotech - they're putting sensors in sealed warehouses to detect spoilage for certain vegetables and fruits. That's something that can have great returns with just a few IoT devices and a novel sensor.
This analysis reminds me of the big interest in the use of hyperspectral imaging for agriculture. The idea was the greater spectral resolution (greater than Landsat) would result in more interesting information. Agriculture was one of the applications. But, once you did find the interesting stuff, you no longer needed a hyperspectral sensor. You could just look at one spot with a much lower cost sensor.
So hyperspectral, like big data, is useful up front. But in the end, much simpler tools and algorithms will solve the problem on a continuing basis.
Yeah, there is perhaps a data usefulness duck curve. First data useful for specific immediate problems, not a lot of use for a while after that, then 15 or 20 years later, the big picture trends start to provide value for big decision making.
Not many orgs keep their data that long, though. Or even think about the future that far.
Interesting. Sounds as if what's really needed isn't so much collecting and analysing lots of data, but an alarm that's triggered when observations deviate from a set of assumptions. Observations that confirm some definition of "normalcy" -- as most observations would -- can be discarded.
Or they could put on their work boots and go walk around the field and kick a few dirt clods, or science forbid! put a hand in the soil to check the moisture content.
Right on point. In the past I have been obsessed with big data, looking for insights. Then I realized that a medium-sized specific data set is always better than a gargantuan general big data monster. There is so many applications in my field where only outliers matter anyways, and everything is very "centralized" to a few relevant observations. So the only thing about big data is that you maybe throw away 99.9% of the data right away and then you have some observations that you actually care about. There is soooo much data out there that is just noise, and so little that I actually care about. And that's why I still end up hand collecting stuff every now and then.
My personal definition of Big Data has always been when you gather/store data without having a planned use for it. Do we need this data? Don't know, let's just store it for now.
The article does allude to this definition when it states that "Most data is rarely queried". We have become data hoarders. Technology has made it easy (and relatively cheap) to store data, but the ideas of what to do with this data have not scaled in comparison.
I love DuckDB and am cheering for MotherDuck, but I think bragging about how fast you can query small data is really no different than bragging about big data. In reality, big data's success is not about data volume. It's about enabling people to effectively collaborate on data and share a single source of truth.
I don't know much about MotherDuck's plans, but I hope they're focused on making it as easy to collaborate on "small data" as Snowflake/etc. have made it to collaborate on "big data".
I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database.
Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can query whatever you want, …
And if the database may get up to 1TB, still no problem with SQL. If exceed that, you may need a professional OPs team for your database and a few giant servers, but they should easily be able to go up to 10 TB, offload some queries to secondary servers, …
I think a lot of data tech has come full circle is now mostly just relational databases. Our org is invested in redshift which lets us mostly pay as we go. The DB itself is just a Postgres facade on scalable storage with some native connectors to file stores and third-parties. After rolling over our stack like three times, we're now just dumping tons of raw data into staging tables, then creating views on top of them. It's 97% raw SQL with a smattering of python for clunky extractions. And we're now true believers in ELT vs ETL.
I think a key driver of this is not having to use SQL. I like DynamoDB and EdgeDB because I can use a more modern and reasonable language to interact with the database.
its really difficult to do any kind of analysis without relational queries. The standard way you do this is to have an app datastore in DDB, and an ETL job that pipes your data into some data warehouse env.
EdgeDB works really well for relational queries, it's a graph native query language that renders into Postgresql. Check it out: https://www.edgedb.com/
That’s a good point, I also think that there should be some modern alternative to SQL. I really like how you can query databases with LinqPad (c#) and how it renders it into a nested table tree. All relations are clickable/expandable, so if you find something interesting in your result set, you can just expand additional rows from other tables. In the background it just creates sql via an ORM, not only once I more or less copy and pasted that generated sql into a view.
But linqpad is not useful if you don’t get the pro version, only then you get code completion. So it’s not really the answer to the problem.
This reminds me of a great blog post by Frank McSherry (Materialize, timely dataflow, etc) talking about how using the right tools on a laptop could beat out a bunch of these JVM distributed querying tools because... data locality basically.
The title might be hyperbole (intentionally), but the observations are more or less in line with what I experienced through a few the Big Data initiatives over the years under different enterprise environments (although I have reservation about the one 1%er comment). To me, Big Data was never about how "big" the data was, but more about the tools/system/practice needed to overcome the limitation of the previous generation. From that perspective, yes, the "monolith" may be having a "coming back" for now due to the improvement of underlying single node performance. But I do think Data size will keep growing, everything needed to make Big Data work would still be there when the pendulum swings back where a single node can't handle it anymore.
Not that big data is dead, more like real time data is coming to life, but you need the old stuff around to make a buck or two… Well, that my view. LLMs are transformer model technique are making data more relevant than ever. If you are a business, well you are in for a “now real” digital transformation.
Making data the centerpiece of your business business could mean that your effectiveness of business process could increase several order of magnitudes. Funny thing is, you will not use some else’s model, unless you are building a ChatBox to infer, but you will need to build your own model and be trained in your own business process to be successful.
Consider a bank, here is my prediction of expected outcomes:
Enhanced Customer Experience: The system can act as a virtual banking assistant, providing customers with instant access to their account information, real-time transactions, and balance updates. The system can also answer customer inquiries and provide relevant information, improving the overall customer experience.
Improved Fraud Detection: The system can monitor the bank's financial transactions in real-time and identify any potential fraud, helping the bank reduce its exposure to financial losses.
Automated Loan Processing: The system can analyze loan applications, credit scores, and other relevant data to approve or reject loan applications in real-time, reducing the time and effort required for manual loan processing.
Personalized Marketing: The system can analyze customer behavior, transaction history, and demographic information to provide personalized marketing and cross-selling opportunities, increasing the bank's revenue and customer loyalty.
Real-Time Insights: The system can provide real-time insights into the bank's financial performance, customer behavior, and market trends, enabling the bank to make informed decisions and respond to market changes quickly.
What is interesting to me is, this is just the beginning of what could be…
Yeah, I've noticed more applications just need to focus on making sense of raw information really quickly, but usually don't need an archive to make decisions.
There are lots of interesting things that can happen with "big streaming" than necessarily "big data". Like, cybersecurity is evolving to monitoring and reacting what everyone's machine is doing in the last 15 minutes, instead of having a huge database of hashes you trust. But not a ton of things really utilize what happened, say, 10 years ago on people's machines.
There's definitely some things that can use massive archives of old data, but I have found far, far fewer things that would benefit from it, and often that comes with some very big maintenance hassles. Most of the time, you can just set data retention to 30 days and be done.
They've been working to implement your ideas for decades and none of it requires LLMs or any machine learning techniques. Basic old ETL is more than sufficient.
The issue is that (a) the calculations they need to perform are complex and take time to run (b) there are financial regulations that weave its way through those system and (c) there is a lot of legacy code especially in the core ledger system which "just works" and people are reluctant to touch.
That said depending on your bank you can get real-time account activity, loan approvals in < 5 minutes etc.
Well, that is an understatement. I do agree with you that banks have been trying to fix decades old application.
But in this process, you don’t need ETL, nor all the process and development to accomplish these ideas. Conceptually the idea builds its self (it learns) how to threat the data, quite revealing and near real time. Considering you account for security and privacy, then you basically shift your input into the data stream and using a natural language get the data output you need, not clunky apps.
Imagine I just login, and say:
me>how much do I have?
bank>You have 100$
me>Please send 50$ to 1003
bank> Are you sure? Please add your security code to confirm
bla bla…
All this with little intervention.
Banks spend hundreds of man hours developing a lacking application while delivering a very poor customer experience. They spend millions on running decades old applications because it so expensive to change them… and thus the circle continues…
I’m really exited to see DataBases disappear conceptualy, data entry, mostly all that just disappear… I will ask my ChapBot for statement, give me a personal investment advice, and classify all my purchases and see where my wife has been spending all my money, all from the confort of my phone.
it’s a brave new wold we are wakening up to, that to me is exciting. And coming from having helped several major banks build their infrastructure, it’s just a boost to talk about something fresh, no more Hypervisor, core count, db licenses, ect. Ok, I’ll concede it’s pretty much the same old, just the nemonics will be different… How many GPUs, how quickly can you spin a container, how fast if your S3 datastore… oh wait, there is that circle again… >:D
Pretty funny to see this when every other headline on this site is about how large language models are about to revolutionize dentistry, beekeeping, etc.
I feel like big data has rarely lived in most organizations. My own experience working in large orgs largely supports the point that collected data is rarely queried. But this is rarely due to a lack of interest, it is mostly because a) nobody really has a great overview over what even is collected b) even if you know/assume something is collected, you usually have no idea where c) if you find the data, there is a decent chance that it is in some sort of weird format that requires a ton of processing to be usable.
This has been - to varying extends - my own experience working in large organizations that don't have tech as their core business.
Although there are some successful data analysis project, the potential of the collected data remains largely underutilized.
443 comments
[ 0.23 ms ] story [ 321 ms ] threadBig data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start.
Maybe what the author is trying to say is 'small-scale big data is dead, but big data chugs on.'
Actual good information will always be useful, most of this "big data" seems to be the equivalent of recording background static.
Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.
In the small institution I am currently working with, the English courses, in one week, integrated chatgpt as a tool for students to work with. It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes, and that happened in one week. I cannot stress enough how unbelievably fast that is for higher ed. That's faster than light speed.
And we're not even that well resourced. I have to imagine there are other examples where it's more than just running through a bot to scan for known outputs.
Why pay for the college?
It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; that was another strange assumption for you to make.
Just a tip, and don't read tone in this statement, but don't assume things. 9/10 times you're going to be incorrect. It's much better to ask questions, instead of making statements with question marks at the end of them.
Isn't that... an assumption?
>It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes
>It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class.
I don't honestly know how else to say that. I legitimately do not know how to help you understand what I'm saying.
In the past, the class would be centered around ideas and themes the class came up with together during the first week of the semester. They would then read and discuss short stories from various authors centered around that theme, preferably from different eras and/or cultures. From there, they would work in pairs/small groups to flush out original ideas they came up with for their own stories based on the themes and styles from the first month of reading. Then they would work individually to write the stories. Finally, they would come back together to edit and work through that process as a group, with a reading and discussion in the last week or two.
Now, the class works alone the first day or two to discuss themes with chatgpt, to identify relevant and appropriate literature (if possible), and to flush out initial ideas for the readings and discussion topics. Then we come back together for group work to figure out themes for the semester and possible readings, like it used to be, but primed with whatever information they had already discussed with Chatgpt. From there they work through the readings, using chatgpt to bounce ideas for discussion off of before coming to class. Then they work in teams, like before, to flush out their own stories, but again, using chatgpt as if it were another member of the group. They also have to track their question/comment and response, to evaluate their own thought processes and look for weak links in their reasoning and logic. Students are then free to use the software to edit their writing before coming together with their creative assignments.
We haven't made it past the step of reading, as it's still the first month of the semester. But,the discussion for the first section has been much more in-depth and (to use a word that is impossible to quantify) vibrant. The students had already aired the ideas they thought may be dumb, and would therefore be less willing to voice in a public setting; this allows them to really dive further into whatever is in each story, and connect dots between stories that generally took a week or two. Because they have another 'person' to talk to whenever they want, and however much they want, they tend to really get into the work. Further, and unexpectedly (and anecdotally sadly) it has allowed a couple of students I know personally who would not have been willing to participate in public discussion (anxiety disorder and TBI) a stronger voice, because they know their ideas are flushed out already, and it provides them with a 'script' of ideas they know are novel and valuable for the class. In other words, if they were able to get the idea from Chatgpt, they knew they had more work to do to build that thought, because that is really just a baseline.
I'm interested to see how the actual writing process goes. Chatgpt is okay at creative writing, but not at the level that we expect to see. Some faculty expressed concerns that students would just have the software do the work for them, but I'm not really bothered by that. First off, if someone can figure out how to make a career out of publishing chatgpt prompts, well, good for them. Second, if the software is able to write better than the students are, they don't really belong in this class in particular (graduate level writing class).
Anyway, like I said earlier in the thread - we're really just treating this as another 'person' in the class, but who is available to everyone, all the time. It's beneficial for the brainstorming sections, but not for the hard skills, from what I can tell. I am most excited to watch them evaluate their own thought processes when working with chatgpt to flush out their ideas. In the past, this has been a barrier, because every single person I’ve ever met never gives a partner 100% of their thoughts all the time. They always hold back. My theory is that, because it’s a dead-end conversation, they will be more willing to push i...
More Big Data!
- attributed to Socrates by Plato. c.399-347 BCE. “Phaedrus.”
https://marshallbrain.com/manna1
People are very attached to school being just like it was when they went.
If you don’t do it that way, you’ll always be fooled by whatever the screen tells you.
[1] https://www.nbcnews.com/tech/innovation/chatgpt-can-help-foo...
seems like a no-brainer in the long term.
At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?
Careful with the scales. 2x improvement could be interpreted from 80% human performance to 160% human performance. Or going from 10% error rate to 5% (which again crosses into superhuman territory on some tasks). Those last few bits are the critical ones.
Here's a novel idea: test students using pen and paper?
I am hopeful that a change happens in academia to prepare students for jobs, which is why they are going to school in the first place. Yes, students need to learn how to "think", but really they are wanting to get the technical skills to perform their duties more than anything.
We have bestowed too much credence in traditional academia not useful to the average person or average job. College is a "game" for most students, and they put up going through the motions of testing, etc. for the sake of the diploma at the end.
I hope we're going to enter a new era of what college means for those looking to get something different out of it.
That's pretty much exactly what the author says in the article.
ChatGPT and the like are not going to get much use from that kind of data and instead are looking at a giant corpus of text and images scraped from a variety of public sources to infer what humans might think sounds smart. It's possible the two worlds will meet, but that's probably not what's going to be announced this week.
I guess the take away however is still that regular businesses really just can't play in this game and should not be assuming they have big data until that fact asserts itself out of necessity rather than the other way around.
If anyone wants to follow along, the series is here!
https://alexpetralia.com/2023/01/19/working-with-data-from-s...
Unfortunately, I've found that many data teams focus more on making the data clean and available. They never drive the conversation about what actions are being taken with the data. That leads to them being treated as cost centers. Wrote a similar post about my perspective on it - https://bytesdataaction.substack.com/p/transform-your-data-t...
I'd love to chat about the space more with you if you're interested! Email in bio.
My cheap no-name old laptop SSD writes with 170MB/s.
A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach.
My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.
It will grow larger still if you include web logs from your e-commerce site and event data from your mobile app so that you can correlate these orders with items that customers considered but ultimately didn't buy. How will your laptop and SSD perform when you then build a user-item matrix to generate product recommendations for each of those 1.2 billion customers?
While plenty of organizations unnecessarily use Big Data tools to store and analyze relatively small amounts of data, there are plenty of customers with enough data to require them. I've seen plenty of them firsthand.
I/O performance is just one of many characteristics that impact performance and from experience the one you least need to worry about. RAID 0 across multiple high-end NVME drives with OS file caching is going to be more than fast enough for most use cases.
The issue is running out of CPU performance and being able to seamlessly scale up/down compute with live running workloads.
And, to address your previous statement: one beefy server is actually pretty scalable. Soft threads spin up in microseconds to serve incoming requests, communication between threads is blazing fast, caching is simpler on one machine, etc. You don't even have to worry to much about scaling, the CPU just throttles itself when there is no load.
And every once in a while you just upgrade to the next gen beefy machine.
Looking at the distribution of companies by employee count and assuming that data scales with employee count (dangerous assumption, but probably true enough on average), that means that companies don't need distributed compute until they get several hundred employees. [0]
[0] https://www.statista.com/statistics/487741/number-of-firms-i...
These types of questions are orders of magnitude faster with a distributed backend.
I was just playing with some data from our manufacturing system, about 30 GB. I pulled the data to my laptop (very expensive Apple one) and while it fits on my disk just fine, it took about 15 minutes to download.
I imported it to ClickHouse which took a while due to figuring out whatever compression and LowCardinality() and so on. I ran a query and it took ClickHouse about 15 seconds. DuckDB pointed to the parquet files on my SSD took 19 seconds to do the same. Our big data tool took 2 seconds, while working with data directly in cloud storage.
Now of course this is entirely unfair - the big data thingie has over twenty times more CPUs than my laptop, and cloud storage is also quite fast when accessed from many machines at once. If I ran ClickHouse or DuckDB on 100 CPU machine with terabyte of RAM it might have still turned out faster.
But this experiment (I was thinking of using some of the new fancy tech to serve interactive applications with less latency) made me realize that big data is still a thing. This was a sample - one building from one site, which we have quite a few of.
I have no doubt that there are situations where the cloud will be faster, especially when provisioned for max usage [which many companies do not]. However, there are a lot of these situations even where the local machine can supplement the cloud resources [think re decisions a query planner can make].
Feel free to reach out at ryan at motherduck if you want to chat more.
Right now I work for one of them: a global investment bank.
Within that organisation we have at least 100+ Spark clusters across the organisation doing distributed compute. And at least in our teams we have tight SLAs where a simple Python script simply can't deliver the results quick enough. Those jobs underpins 10s of billions of dollars in revenue and so for us money is not important, performance is.
So 1000 x 100 = 100,000 teams, all of whom I speak for, disagree with you.
Hold your horses... the beefiest servers that are in production today, unless you count custom-made stuff go to somewhere between 128 and 256 cores per board. These are hugely expensive. Also, I don't know if you can rent those from Amazon.
Typical, affordable servers range between 4..16 cores. Doesn't matter if you buy them yourself, or you rent them from Amazon. It's much cheaper to command a fleet of affordable servers than to deal with a high-end few. This is both because the one-time price of buying is quite different and because with smaller individual servers you have a fighting chance to scale your application with demand. Especially this is true in case of Amazon as you could theoretically buy spot instances and by so doing you'd share the (financial) load with other Amazon's customers.
Now... storage. Well, you see, in Amazon you can get very expensive storage that's guaranteed to be "directly" attached to the CPU you rent, the so-called ephemeral storage. This is the storage that's included with the VM image you use. It's very hard to get a lot of it. I couldn't find the numbers for Amazon, instead, I know that Azure tops out at 2 TB. In principle, this kind of storage cannot exceed a single disk, so, think Amazon probably offers the same 2 TB, maybe 4. But, again, it's cheaper to have a bunch of EBS's attached... but then you'll have to have more of them as the latency will suffer, and in order to compensate for that you would try to increase throughput, perhaps.
Also, think that, in practice, you'd want to have a RAID, probably RAID5, and this means you need upwards from 3 disks. Also, if you are using something like a relational database, you'd most likely want to put the OS on a single device, the database data on a RAID and database journal on a yet another device, and, probably, you'd want that device to be something like persistent memory / optane / something from higher-tier disks with dedicated power supply. And all this is not due to size, but due to different contingencies you need to have in order to prevent huge data loss... Now, add to this backups and snapshots, perhaps replication in 2-3 different geographical areas if you are running an international business... and that's quite a bill to foot.
There are similar problems with memory, since there can only be so many legs on memory bus and only so many pieces of memory you can attach to a single CPU, and if you also want a lot of storage, then, similarly, there can be only so many individual storage devices attached and so on.
Bottom line... even to reproduce the performance of your laptop in the cloud you would probably end up with some distributed solution, and you would still struggle with latency.
Though it is a bit disappointing that while Microsoft advertises this as "good for distributed persistent stores", there are no obvious SLAs that I could rely on for actually trusting such a cluster with my data persistence.
[1]: https://learn.microsoft.com/en-us/azure/virtual-machines/las...
This situation may not be reflected in Azure pricing (the calculator gives 7.68 $/h for L80as_v3) since if MS has such hardware, it would be a waste for it to stand idle. They'd be incentivized to rent it out event at a discount (their main profit is from traffic anyways). So, you may not be getting an adequate reading of the situation, if you are trying to judge it by the price (rather than cost). But, this is only the price of the VM, I'm scared to think about how much you'd pay if you actually utilize it to its full potential.
Also, since it claims to have 80 vCPUs, well... it's either a very expensive server, or it's, again, a distributed system, where you simply don't see the distributed part. I haven't dealt with such hardware firsthand, but we have in our DC a Gigaio PCIe TOR switch which would allow you to have that much memory (in principle, we don't use it like that) in a single VM. That thing with the rest of the hardware setup costs some six-digit number of dollars. I imagine something similar must exist for CPU sharing / aggregation.
> The high throughput and IOPS of the local disk makes the Lasv3-series VMs ideal for NoSQL stores such as Apache Cassandra and MongoDB.
This is cringe-worthy. Cassandra is an abysmal quality product when it comes to performing I/O. It cannot saturate the system at all... I mean, for example, if you take old-reliable PostgreSQL or MySQL, then with a lot of effort you may get them to dedicate up to 30% CPU time to I/O. Where the reason for relatively low utilization (compared to direct writes to disk) is the need to synchronize that's not well-aligned with how the disk may want to deal with destaging.
Cassandra is in a class of its own when it comes to I/O. You'd be happy to hit 2-3% CPU utilization in the same context where PostgreSQL would hit 30%. I have no idea what it's doing to cause such poor performance, but if I had to guess, some application logic... making some expensive calculations sequentially with I/O, or just waiting in mutexes...
So, yeah... someone who wanted Cassandra to perform well would probably need that kind of a beefy machine :D But whether that's a sound advise -- I don't know.
That's 8.4 hojillion megabytes per second right there.
That is not random access speed. For random access my relatively high-performance SSD only does 42MB/s reading and 80MB/s writing.
At first I tried to puzzle out a good sampling strategy to make sure I didn't bias the output, then on a whim I tried 2^32 samples and went to lunch. It took something like a half an hour to do 4 billion samples. Took me a couple times to figure out how to squeeze 4k megapixels into a graph so I ran it a few more times, but the results showed a very distinct banding pattern that confirmed that the problem was every bit as bad as I suspected, which was a blocking issue for our release. A couple of hours well spent, running through an 'intractable problem' that really wasn't.
Also, using BigQuery as a metric of how Big Data is used is, IMHO, wrong. Real analytics companies usually have custom solutions because BigQuery is too expensive for any serious usage unless you are Google.
Sighs...
mongo's got sharding out of the box - which is nice - but you have to get your key right or it will suck.
Also no one should want to host a mongo db - unless that's your business.
And if there is any levelling off it's going to be because of the move towards cloud managed options e.g. Snowflake, DocumentDB rather than because PostgreSQL decided to add JSONB support.
[1] https://www.macrotrends.net/stocks/charts/MDB/mongodb/revenu...
The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
It's a pretty strong argument to say that Microsoft under Gates was technically obsessed, but that really faltered under Balmer.
Microsoft continued to win profits, but they made major strategic missteps that cost the revenue.
Amazon feels like it's going down the same path: empowering the tree-gazers without remembering that the forest also matters.
{Revenue attributable to previous R&D} (aka coast) vs {Revenue attributable to current R&D} (aka acceleration)
I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.
Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).
Data Scientist's job is to launder management's intuition using quantitative methods :)
We are accepting a whole lot of assumptions every day.
The definition of rational is being informed enough to know what said chemicals will do in the short term and long term in order to make an informed decision, but then I'm reminded we dont get taught any of the above unless we specialise at a Uni, so most people cant make any sort of informed decision.
Had the same problem when I was an economics researcher -- publication bias for what stakeholders want to hear (often the government) is rampant because that's where funding for the economics department mostly comes from.
This is also known as The Iron Law of Institutions.
it isn't clear to me how the grounds for realizaing the theory are reconciliable with its conclusion
-- Robert A. Heinlein, Tunnel in the Sky, 1955
This is the only reason why a 60+ PowerPoint slide deck can cost several millions dollars
It's easier to poke holes in predictions of the future than in interpretations of the past, especially when the people making those decisions have likely reached their decision-making status through "how things have always been".
So, a synonym for 'consultant?' :)
I've always used them as "independent 3rd parties" who were listened to.
This has been going on for mellennia, unfortunately.
I want to make the move from development into consultancy and would appreciate hearing how you did it.
I cannot DM you, but if you have the time to type a small paragraph, I'm all ears.
If you don't want to make that public, you can directly email me (<HN-username>@gmail.com)
It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most junior level and joined my first project with 3 very senior colleagues after a few weeks.
The learning curve was very steep both on the technical level and also regarding the consulting aspect - at first there was nothing I could 'consult' on due to lack of experience. This changed with growing experience, with the guidance of senior colleagues and my private efforts to gain skills and expertise.
Let me know if this was helpful!
Thanks a lot!
A few years ago there was a problem with storm-water infiltration into my (elderly at the time) mother's property from her neighbor. I, being a dutiful son and a civil engineer, investigated it and came up with the probable cause, the likely effects of non-action and the most cost-effective solution. I presented it to my mother in the most layman-like terms that I could. She said she'd think about it – meaning she'd refer i.e. defer – to her daughters. In the meantime I had a very layman-like chat with my mother's carer and told her the situation in layman's terms. The carer listened and said that what I said I made total sense to her. Later on, one of my sisters accosted me and stated that it was completely obvious what the problem and the solution was – "even the carer could see it". Human foreheads don't have the real estate for where my eyebrows wanted to ascend.
My advice is to consider whether the message should be separated from the messenger somehow.
Recent tech layoffs is the other side of that growth, when cheap money is gone and company is forced to stick to core competencies and shutdown growth plans
It took a lot of analysis and time to clarify to higher ups that we weren’t just P-hacking , but at least they were concerned about that.
> Data Scientist's job is to launder management's intuition using quantitative methods :)
It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if he proceeds with whatever he already wants to do.
The beards might be a bit shorter or nonexistent, but the story hasn’t changed.
I was the infra lead on a data lake project and got take part all the way to breaking down the data and turning into PowerBI reports. The result was "sell more" and to clients who marketing already identified, years ago, as whales.
There were some interesting other insights, esp. w/r/t to niche products that sold around weird dates (Easter, Memorial Day, 4th July -- but not obvious gift days like Valentines or X-Mas), but it led to a lot of "you're doing it wrong!" recriminations and follow up projects.
I would say it was a very engineering driven org however, so if you could present compelling data it could go a long way.
https://www.youtube.com/watch?v=kAichhoZrKs
Also someone to blame if it doesn't work out
Nonetheless, the CTO went on a multi-year, 10s of millions of dollars, huge data tech stack & staffing reorg shake up... with really zero data points explaining the driver, or what we would measure to determine it was successful.
So it became a self referential decision that we are successful by doing what he decided, and we are doing it because he decided it.
Ouch. This is savage, but sadly correct in many cases.
HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to happen.
The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.
Positive claims are about what is true. Normative claims are about what should be true, or rather what decisions we should make. Put another way, positive claims deal only with facts while normative claims deal also with values.
GP is saying that it's the data-scientist's job to give the executive the facts and it's the executive's job to decide what to do about the facts.
So often, those models proved to be over-fitted and not generalizable.
But too many decision makers simply can't properly judge such results.
Where do business analysts fit into this dichotomy? Their whole job is to poke around in Tableau in order to surface high-ROI strategies for the business to pursue. (Where, in choosing which proposals to surface to management, they're effectively making 90% of the strategic decisions.)
Or how about corporate buyers in trading and retail companies?
Or quantitative investment managers?
If understanding the data generating processes is the goal, I'd rather see some useful plots than wade through a technical description of some model whose assumptions were flagrantly violated.
The skepticism isn't a problem, the unequal application of it, the potential to harm careers, and the chilling effect as people wisen to how best meet their own personal goals is.
https://ryxcommar.com/2022/11/27/goodbye-data-science/
The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.
Sadly true. As humorously depicted here[0]:
[0] https://www.anvari.org/fortune/Best_Fortunes/820_in-the-begi...And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so in their mind, they are doing the right thing.
- Galileo Galilei
A modern characterisation of "The Galileo Affair" would be that he was SWAT'ed by someone he was really really mean online to.
and Galileo literally refered to delle Colombe (and friends) as Simplicio (simple minded) and worse in his highly popular Dialogue Concerning the Two Chief World Systems [3] and within a year or so the Pigeon League got their revenge, using their influence to have religuous charges bought against Galileo.The affair was complex since very early on Pope Urban VIII had been a patron to Galileo and had given him permission to publish on the Copernican theory .. this was very much a case of personal vendettas and internal politics rather than a straight up case of "The Church Versus Galileo".
[1] https://en.wikipedia.org/wiki/Galileo_affair#cite_note-Spell...
[2] https://en.wikipedia.org/wiki/Lodovico_delle_Colombe
[3] https://en.wikipedia.org/wiki/Dialogue_Concerning_the_Two_Ch...
>> Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
It's a good sign at the company that I run, anytime our analysts/data scientists come up with metrics that say we're killing it, or that our ideas should bear a ton of fruit, the kneejerk reaction is to be extremely skeptical of the results. Usually they're still right.
When the data scientists say we're fucking something up, we tend to pay a lot more attention.
Only the paranoid survive, after all.
I found it incredibly stressful to discover and provide analyses (even experimental results) that wasn't expected, or contradicted prior beliefs. The findings were always very harshly scrutinized, and typically lead to tons of pointless extra work to 'understand what is going on'.
My friend runs a successful market research agency and she says she gets called in when management have decided they need to make a change but need evidence to sell it to the shareholders and staff.
That was a plot point in Dirk Gentley's Holistic Detective Agency (1989), though the observation much pre-dates this.
doesn't help that most of this data goes through multiple layers of BS where each person is putting it through filters to make themselves look better. And a good chunk of people don't have enough understanding of stats to understand when they are being tricked
Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc.
Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.
https://didgets.substack.com/p/data-science-and-serendipity
Getting your data to the cloud is expensive, but then you can’t do anything with it because distributing it to process in multiple stages is too expensive and you’re already paying so much to keep all that useless data.
kthnxbye
> Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
100% Individual humans are emotional creatures with their own wants and needs, and it's important to understand how organizational incentives drive decision making.
> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven".
This has not been my experience, though. The more common thing I've seen is that, sometimes data is boring and doesn't really show much actionable insight, but as everyone wants to justify their job, I've seen data scientists come up with really questionable conclusions that fell apart on further inspection (call it "p-hacking the enterprise").
Plus, a lot of this data in these data wearhouses is messy. Often times data scientists are siloed at the end of the process, but then you get "garbage in/garbage out" results, where there is some bug in data tracking that isn't understood until it's too late. Much better in my opinion to have data engineers and data scientists work much more closely with product engineering teams up front so they can help ensure the data they collect is accurate.
Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile.
Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been paid handsomely, while some higher ups can now put "implemented agile and big data at X" on their LinkedIn profiles.
Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (like asking for average of something that follows a very fat tailed distribution with a few observations dominating the phenomenon. Of course it's usually impossible to explain this to people).
The more I think about this, the more I think that in truly data powered companies, both the decision making and data analysis have to be carried out by more or less the same people. The organizational hierarchies have to be much flatter. Essentially the employees will have to be some kind of "secret agents" who have both the skills and the mandate to steer the company in the direction they see fit. I sort of see this already happening in the FAANG companies where, or so I hear, it's very difficult to get hired, the staff count is quite small compared to traditional companies and the senior engineers have a lot of power in the company.
Using math PhDs or Palantir or whatever as a sort of modularized black box for "insights" while giving them no real skin-in-the-game does not work.
My favorite example is the WW2 bomber diagram shown to illustrate survivorship bias. Sure, working from data and first principle one could identify the vulnerable spots of the bomber. Or one could asl the designers or have an engineer, heck even a contemporary field mechanic, take a look at the actual drawings of the plane. And reach the same conclusion, faster, with a ton of additional insight and improvement ideas that can actually be implemented...
Which itself both takes time and is wildly unpredictable - neither of play well with today's Taylorist managements schemes.
I told my VP that the engineering foul ups in the current product are easily fixable. Standard tooling and patterns exist to re-architect and solve the bottlenecks. What is much harder is a data architect to make sense of the complex data and make sure there is good value for our customers.
Guess what position I don't have on the team, and won't have due to budget issues.
The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)
But I have thousands of data points, thousands, I tell ya!
Though I think there’s a better way— that is executive data science, what I do at Zapier. The key is that I’ve built up a huge amount of econometric and economic / business skills that I apply to affect good change in collaboration with company leaders. It allows me to work with Execs using sensible analysis. I improve growth and output by helping us catch errors of assumption before they go into production and cost growth / bad surprises. I also help the executives gain alignment around good information. This multiplies their departments’ output by allowing them to work better together, more in concert. That helps avoid issues with data being bent to decisions.
They typically carry a lot of hard-won valuable domain knowledge (that I combine with my economic-statistic knowledge and skills for rigor).
It’s my job to ensure Execs start with good sensible information regarding objectives. They usually ask fantastic questions and share a lot of great analysis of their own.
There are times I learn about what I might call controversial implications. This is typical of innovation using technology. It’s in these moments that I feel I create the most value by highlighting the trade offs I believe we face / potential regret.
My day to day is mostly economic / econometric analysis, conceptualization and formulation of technology, and writing, but also occasional meetings to gather information, feedback, answer questions, teach and collaborate with others.
I try to effectively “sample” the org by interacting with people up and down the org, so I can assist Execs in incorporating directors, managers and ICs knowledge / pool good information / systematize planning, etc.
I worked in a "data-driven" company of 5,000+ employees, with 4 data scientists who were spilt into two separate non-collaborating teams. In effect, they were so under resourced they got nothing done.
There's an agenda by some level of management, and they use "data" to forward their agenda, or disregard it due to "unexplainability" (a legitimate concern for BD/ML) if it disagrees with the agenda.
Meanwhile for example boilingdata.com seems to have already done that - by using AWS Lambda + DuckDB as distributed compute engine which I can't decide if its awesome, deranged or both.
[1] https://duckdblabs.com/news/2022/11/15/motherduck-partnershi...
Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing about these is that once they've been observed and understood, the value of ongoing observation drops rapidly.
A great example of this is soil moisture sensors. Every farmer that puts these in goes geek-crazy for the first year or so. It's so cool to see charts that illustrate the effect of their irrigation efforts. They may even learn a little and make some adjustments. But once those adjustments and knowledge have been applied, it's not like they really need this ongoing telementry as much anymore. They'll check periodically (maybe) to continue to validate their new assumptions, but 3 years later, the probes are often forgotten and left to rot, or reduced in count.
"Mate, we don't need a chip to tell us the soil's dry"
https://doi.org/10.1145/1858171.1858211
That would only be a million bits (1 Mb). You're counting potential states, not bits.
There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter.
Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few new operational tools in their quiver.
Same is true of fitness trackers: you excitedly get one, learn how much you really are sitting(!), adjust your patterns, time passes … then one day you realize you haven't put it on for a week. It stays in the drawer.
Not unless they're threatened with ruin will people make changes to the standard way of doing things. This is actually … not bad! Continuity is important, and this is kind of a subconscious gating function to prevent deviation from a proven way of working. So, the change has to be so compelling or so pressing that they're forced to. Not a bad thing.
While we think things change overnight in this world, they generally take awhile … stay patient … it's worth it.
I definitely did not need a multi-PB storage array to reach that goal. Or did I? I am sure someone out there knows roughly the human brain storage capacity in MB.
I spent a number of exciting year developing a high frequency soil impedance scanner and finally understood why I was doing it. To confirm the obvious :)
If you want Toyota style continuous improvement you would need to improve in new areas of the process / new metrics, most of the time?
Having tons of data is a Good Thing, so long as you can afford the marginal cost of gathering and managing all that data so that it's ready at hand when you need it later.
It's how you use the data that makes all the difference. If you're facing an issue you don't understand at all, don't go digging for random correlations in your mountain of data to find an explanation.
Think like a scientist: you need a valid hypothesis first! Once you have a hypothesis about what your issue might plausibly be, then you make a prediction: "If I'm right, I suspect our Foobar data will show very low values of Xyzzy around 3AM every weekday night". Only then do you go look at that specific data to confirm or refute the hypothesis. If you don't get a confirmation, you need to go back to hypothesizing and predicting before you look again. You can't prove causation by merely correlating data.
Absolutely. But in my experience, there's this massive trend across the tech world that flat out rejects the value of domain/subject matter expertise. Instead, all you need is an engineer who can throw some ML at the uncurated mountain of data your organization has collected. Little to no value is placed on the resources that can frame an actionable hypothesis, even though the entire value proposition arises from this exercise!
Maybe I'm just jaded. I end up wasting a lot of time trying to re-direct data scientists and engineers down more appropriate pathways than if the problem they're solving was just brought to my attention earlier. Sorry, I understand you spent two weeks shoe-horning dataset X into our analysis system for your work, but it's invalid for the question you're asking - use dataset Y instead, and you'll have an answer in an hour or two.
Sounds like the data scientists need to get together with the MBAs and they can do companies where nobody needs to actually know what they're doing.
The problem is that they don't stay geek-crazy?
You don't need a chip to tell you that the soil is dry, but if you can use that chip to regulate drip irrigation that can apply substantially different flow to different plants, then you can get a not-too-much, not-too-little watering even if you have a big variation in conditions.
You don't need a big analysis to acknowledge that everybody knows that a particular competitor has lower or higher prices and adjust your pricing; but doing that continuously on a per-product basis does require data and analysis.
I've worked on many product-led-growth initiatives in the software industry. The software industry is probably the biggest 'believer in data' there is -- many scientific-forward minds who understand the value. However, even in the software industry, it's really hard to convince folks that if you make 5 improvements that net 1% conversion gain each, you can dramatically improve revenue.
So hyperspectral, like big data, is useful up front. But in the end, much simpler tools and algorithms will solve the problem on a continuing basis.
1. Keep the raw full data for short period of time, at most 1 month.
2. Downsample what you need for longer period of time (5-10% of the full data).
3. Aggregate your metrics on a yearly basis to save money and compute costs.
Not many orgs keep their data that long, though. Or even think about the future that far.
Right on point. In the past I have been obsessed with big data, looking for insights. Then I realized that a medium-sized specific data set is always better than a gargantuan general big data monster. There is so many applications in my field where only outliers matter anyways, and everything is very "centralized" to a few relevant observations. So the only thing about big data is that you maybe throw away 99.9% of the data right away and then you have some observations that you actually care about. There is soooo much data out there that is just noise, and so little that I actually care about. And that's why I still end up hand collecting stuff every now and then.
The article does allude to this definition when it states that "Most data is rarely queried". We have become data hoarders. Technology has made it easy (and relatively cheap) to store data, but the ideas of what to do with this data have not scaled in comparison.
I don't know much about MotherDuck's plans, but I hope they're focused on making it as easy to collaborate on "small data" as Snowflake/etc. have made it to collaborate on "big data".
Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can query whatever you want, …
And if the database may get up to 1TB, still no problem with SQL. If exceed that, you may need a professional OPs team for your database and a few giant servers, but they should easily be able to go up to 10 TB, offload some queries to secondary servers, …
Both are distributed compute. Except that Spark allows you to mix/match code with SQL.
But linqpad is not useful if you don’t get the pro version, only then you get code completion. So it’s not really the answer to the problem.
https://github.com/frankmcsherry/blog/blob/master/posts/2015...
Volume != Quality
Making data the centerpiece of your business business could mean that your effectiveness of business process could increase several order of magnitudes. Funny thing is, you will not use some else’s model, unless you are building a ChatBox to infer, but you will need to build your own model and be trained in your own business process to be successful.
Consider a bank, here is my prediction of expected outcomes:
Enhanced Customer Experience: The system can act as a virtual banking assistant, providing customers with instant access to their account information, real-time transactions, and balance updates. The system can also answer customer inquiries and provide relevant information, improving the overall customer experience. Improved Fraud Detection: The system can monitor the bank's financial transactions in real-time and identify any potential fraud, helping the bank reduce its exposure to financial losses.
Automated Loan Processing: The system can analyze loan applications, credit scores, and other relevant data to approve or reject loan applications in real-time, reducing the time and effort required for manual loan processing. Personalized Marketing: The system can analyze customer behavior, transaction history, and demographic information to provide personalized marketing and cross-selling opportunities, increasing the bank's revenue and customer loyalty.
Real-Time Insights: The system can provide real-time insights into the bank's financial performance, customer behavior, and market trends, enabling the bank to make informed decisions and respond to market changes quickly.
What is interesting to me is, this is just the beginning of what could be…
There are lots of interesting things that can happen with "big streaming" than necessarily "big data". Like, cybersecurity is evolving to monitoring and reacting what everyone's machine is doing in the last 15 minutes, instead of having a huge database of hashes you trust. But not a ton of things really utilize what happened, say, 10 years ago on people's machines.
There's definitely some things that can use massive archives of old data, but I have found far, far fewer things that would benefit from it, and often that comes with some very big maintenance hassles. Most of the time, you can just set data retention to 30 days and be done.
They've been working to implement your ideas for decades and none of it requires LLMs or any machine learning techniques. Basic old ETL is more than sufficient.
The issue is that (a) the calculations they need to perform are complex and take time to run (b) there are financial regulations that weave its way through those system and (c) there is a lot of legacy code especially in the core ledger system which "just works" and people are reluctant to touch.
That said depending on your bank you can get real-time account activity, loan approvals in < 5 minutes etc.
But in this process, you don’t need ETL, nor all the process and development to accomplish these ideas. Conceptually the idea builds its self (it learns) how to threat the data, quite revealing and near real time. Considering you account for security and privacy, then you basically shift your input into the data stream and using a natural language get the data output you need, not clunky apps.
Imagine I just login, and say: me>how much do I have? bank>You have 100$ me>Please send 50$ to 1003 bank> Are you sure? Please add your security code to confirm
bla bla…
All this with little intervention.
Banks spend hundreds of man hours developing a lacking application while delivering a very poor customer experience. They spend millions on running decades old applications because it so expensive to change them… and thus the circle continues…
I’m really exited to see DataBases disappear conceptualy, data entry, mostly all that just disappear… I will ask my ChapBot for statement, give me a personal investment advice, and classify all my purchases and see where my wife has been spending all my money, all from the confort of my phone.
it’s a brave new wold we are wakening up to, that to me is exciting. And coming from having helped several major banks build their infrastructure, it’s just a boost to talk about something fresh, no more Hypervisor, core count, db licenses, ect. Ok, I’ll concede it’s pretty much the same old, just the nemonics will be different… How many GPUs, how quickly can you spin a container, how fast if your S3 datastore… oh wait, there is that circle again… >:D
In that case, chat-bots have existed for years and consumers largely don't like them.
In your scenario you can transfer money in a few clicks rather than having to write out an entire conversation.
This has been - to varying extends - my own experience working in large organizations that don't have tech as their core business.
Although there are some successful data analysis project, the potential of the collected data remains largely underutilized.