University data is getting outdated very quickly. Large companies have taken over the role of data curators.
I personally think this is wrong. Big tech companies should build hardware, while government agencies should use that hardware to curate our data. Like how it was near the beginning of the internet. Companies should be as far away from our data as possible.
I think the exact opposite: government should know as little as is required to provide its services, as should any company. If you imagine a company can do damage with your data, wait til a bad government has it.
I can't access the original study however taking data, explicitly naming security cameras for example, to "use it or merge it with data from external parties such as publishers or public or private sector organizations" will surely not seriously degrade student and staff privacy, right?
The rush to "exploit" data reminds me of the dot com hype. It's one thing to use available data to make more informed decisions about things from course content to building occupancy. It's quite another to rush towards total surveillance because of a Fear of Missing Out of "exploitable data".
Make of it what you will, I just saw on the project censored website that protecting student data is seen by some as a major priority. The article seems more PII and cybersecurity focussed, rather than recognizing the threat of intentional monetization.
This is about digital security, i.e. protecting data from unauthorized access and use. It says nothing about protecting students themselves from surveillance and abuse by their own institutions.
This is the type of article that makes me wonder: "therefore, what?"
Yes, absolutely, universities are awash in data they do little to anything with outside of hyperfocused laboratories. I wonder if having central data management groups would be an enabler or a bottleneck, however.
They have 0 visibility into any of the operations and therefore can’t derive any insights or install controls.
They have no idea if:
Some professors more effective teachers than others.
Which days students buy more food.
Whether taking a given prerequisite leads to better mastery of advanced material.
If additional spend is correlated with better student outcomes.
For people in tech businesses, it’s such a shocking thing to read it’s hard to fathom how any of the leadership at universities find this situation acceptable.
I didn't go to a prestigious US university but even mine had all this data and used it. In tech we like to think we are the only ones to "get" data. We are not.
Teaching outcomes are evident from exam results, grants won, papers published etc and are routinely used to determine promotions, staffing and more. Food/footfall etc are monitored and predicted by the private catering companies on campus. Budget reviews are comprehensive and require yearly and greater analysis of university performance on a per department basis. All that doesn't even include competitive processes like ranking tables for universities, applications for grants, participation in international research groups etc etc.
> They have no idea if:
Some professors more effective teachers than others.
What are you even talking about? Literally every course at most universities end with an evaluation. Obviously, there are other problematic ways to measure effectiveness (publications). How much that data gets used is another question and varies widely by institution. “People in tech businesses” love to exaggerate the quality of the data they have and its usefulness, hence the Subprime Attention Crisis that is rapidly eroding their cash on hand as the Fed’s cheap money and QE dries up.
My school did course evaluations every year, and I’m under the impression that the Professor’ performance is tracked in the sense that if somebody was failing too many students, it would be noticed. Beyond that, I’m not sure how one could evaluate teaching without also being a subject matter expert. So this seems more like the sort of thing that ought to be handled by the professor’s expert peers, than data people or administrators.
> For people in tech businesses, it’s such a shocking thing to read it’s hard to fathom how any of the leadership at universities find this situation acceptable.
I’m in a tech business and it doesn’t sound shocking at all. The only things people really have data for is how many people are giving them how much money.
I run a centralized data and analytics team at a large, online university that primarily serves adult learners. I’ve been responsible for our data and analysis for 10 years.
I think this article rings true for most universities, but for some, like mine, we’ve had unified systems for the last 20 years. While that’s been helpful in a lot of ways, the primary thing we’re looking for - helping students graduate and get better jobs than they have today - is surprisingly challenging. Adult learners often fail to finish school for a variety of reasons, most of which aren’t academic and often can’t easily be forecast using all available data.
Unifying data sources or mining them further doesn’t solve student health issues, childcare, jugging work responsibilities with school, etc.
What’s more, universities are slow-moving, heavily bureaucratic, and highly regulated. So even when we have good ideas to relieve burden those often come head to head with accreditation rules, financial aid regulations.
In my opinion, for my students at least, data is the least of our problems.
I had to laugh at one of the authors being Christine L. Borgman. I'm sure all our data will be treated respectfully when fully integrated into their unaccountable profit-driven total surveillance system.
Security cameras in universities seem to be accomplishing… nothing. I am getting very frequent emails (2-3 per week) from one of the top US universities with blurry unusable screenshots of perpetrators stealing items, breaking in, assaulting students, etc. Makes me think - should they not invest into better security instead?
Starting with the premise "Data is useful", it proceeds to pick at all
the ways we've failed to make it useful... and we're damned well going
to make it useful if it kills us!
Maybe, just maybe (for those that dabble in the sceptical, explanatory
game we call science) it might be that "bare data" has little use
within certain contexts... say those that by definition are on the
cutting edge of knowledge and reality, best steered by imaginative
vision of leading experts.
Data driven market cybernetics may be very useful in some industries,
such as ones with physical logistics, complex supply chains, rapidly
shifting supply sources and demand sinks. But the primacy of "data
uber alles" should not be one-size-fits-all.
Having data can undeniably be useful. The question is, how much ROI does the insights in the data provide, relative to the cost to build the organizational infrastructure capable of extracting these insights?
And there's a bootstrapping problem, because often it's hard to calculate the potential ROI without actually going through the work of building the infrastructure to process the data.
Indeed, absolutely true. And furthermore, sometimes the value of data,
and of possible other sources, doesn't emerge until you start analysis
and then deductively see new collection opportunities. It's a
to-and-fro process.
> how much ROI does the insights in the data provide, relative to the
cost to build the organizational infrastructure
Having seen the workings of academia for a long time now I'd argue
very little.
But I think the problem is more subtle. A Heisenbergian problem. When
data collection involves people you have a triple problem of
intrusion, distraction, and ossification. the act of trying to create
a "data driven culture" in some contexts simply kills what's
there. The cost of this, in addition to the bootstrapping costs of
sensors and analytics are paralysing.
Academia, by its very nature, if properly functioning, demands to be
dynamic. If it's good, it will not stand still long enough for you to
look at it. And if you measure it "too hard" it will evaporate as all
the good people who resent ossification leave.
I think it's simpler: imagine you're the NSA and decide to transcribe every phone call ever made to text: you'll spend gazillion in storage, connectivity and processing for the capture, and spend a gazillion squared on post-capture text analysis to find "something", like "who is communist in Atlanta", and not even be able to make it find the communists pre-emptively, before it's too late and they already donated $3 to a local chapter.
There can be too much data. There are questions you cannot answer even by collecting all data, unless you have no time or food cost constraints but the value of your answer will decrease with the time distance from the moment the question was asked. In a million year you might know for sure who was communist in Atlanta during the 2000s, year by year, block by block, but you may care less by then.
One man's troubling tone is another man's dystopic reality.
I read a comment on here months ago about someone who enabled a hotel chain to process the data coming from the motion activated lights in every hotel. The chain was able to track workers with this data and cracked down on people taking too long of breaks.
And now a personal anecdote. The business I used to work at had an ID badge door you had to use to enter the smoking area. I was told by someone I trusted that when layoffs came, a member of management asked for a list of the people that used that door. The people on that list were prioritized for termination.
"Universities are literally awash in data. From administrative data offering information about students, faculty and staff, to research data on professors’ scholarly activities and even telemetric signals"
Why are professors' telemetric signals even being discussed? Are publicly listed office hours not enough? My dogs got chipped without their consent. Do professors deserve more dignity than my dogs?
This is the problem. The data could be used for good: to improve the education being provided to the students, and to improve conditions for those working for the university, but history tells that most of the time it will end up being used against/at the expense of those same people and that there will be a much stronger profit incentive to do the wrong thing rather than the right one.
I have no doubt the problem is worse in academia, but to be quite honest nobody is really doing a good job at curating, governing and unifying data across large organizations (at least without extreme expense.)
Just look at the treadmill of architectures designed to "solve" this problem: integrated databases to "data warehouses" to "linked data" to "data lakes" to "data fabric" to "data mesh."
Organizations that succeed do so because they have the resources to prioritize data management, and do so at significant team size and expense. But any organization that doesn't (or can't) view data curation as first class, top-line budget isn't in a position to capture even a small part of the theoretical value of their data.
Consider the "data mesh" architecture, the latest fad in this space. It basically assumes that not only do you have a dedicated data curation team, each data source also has it's own dedicated team of curators capable of productizing data sources.
That kind of thing is simply out of reach for most organizations.
I lived through an acquisition in the teens that I feel was an illustrative story. An old financial services provider embarked on a massive, whole organization modernization effort. The new CEO went all in and actually made the necessary investment in staff and headcount, which was remarkable in itself.
A big part of the effort involved providing customers with big data solutions and all the implied benefits that would entail. They jumped aboard the hype train with gusto. Each quarter progress was reported and margins improved with a clear upward trend. From the trenches it was clear that even with the investment made, it was still not enough to reach any of the goals.
After a year of “modernization” the company was acquired, the buyer literally salivating over the coming profit margin increase and “synergies” in selling into related markets. After another year post acquisition it became clear that the benefits were simply out of reach, and the profits did not arrive. The acquiring company was left with the low margin business it had always been. Layoffs ensued as the buyer sought to salvage something from the transaction.
Turned out the entire modernization and big data effort had been an elaborate bait and switch committed by the board and CEO of the acquired company, masterfully executed.
>Turned out the entire modernization and big data effort had been an elaborate bait and switch committed by the board and CEO of the acquired company
Are you saying that rhetorically, or do you actually believe (perhaps with evidence) that the modernization and big data effort was not undertaken in earnest?
Observation/mini rant: unfortunately the data industry is both very susceptible to fads and hype, and does not have widespread standardization or a generally-accepted set of best practices. The subset of data "influencers" whose discourse occurs mostly through Twitter and Substack have especially heavy sway in what is the current Data Big Thing. These people introduce new buzzwords and ride them to the mainstream, or redefine what were previously more-or-less anchored down concepts into new interpretations. It feels tepid and arbitrary, almost postmodern. On top of this, much of the "thought leadership" is being driven by individuals who lean heavily towards the soft-skill side. So we have a heavy overindexing on strong opinions around organizational methodologies, team structure and roles/responsibilities, and other Big Ideas without much engineering representation or consideration. What spawns from this are things like the "data mesh" and the bastardization of both data concepts and clear communication/thinking. The data mesh white paper perfectly encapsulates this[1]. I challenge anybody to try to read it and understand what the hell the author is even vaguely trying to say after a few passes.
I do see what you are saying, though I think a lot of what they say makes somewhat more sense if you are familiar with their consulting "bubbles" and the language used within them.
But the problem exists even on a concrete technical level. Look at how many different "big data" products are out there. The Apache project alone has probably 15 different tools that largely overlap.
> We need more ad-tech startups for the university/higher education industry NOW
I had the same thought - intrusive analytics and a focus on exploiting and monetizing client data seem to have damaged the web, made gaming less fun, and made everything user-hostile and creepy. Do we really want universities to go down that path?
While I agree with your general skepticism, I also believe there are ways these data could benefit the students. Extracting value from operational data could mean modernizing curriculum, offering more office hours, etc.
I think most of the value of "data" is captured in empowering individual contributors to observe their working conditions and the impact of their actions and adapt their strategy in response. The power of the central office spreadsheet wielders is very secondary. (Which is of course not to say that spreadsheets are not valuable, as any teacher knows.)
In an ideal world - yes. But seeing how overworked professors and other staff members already are in top universities, I doubt this is top of mind for them. There are courses with hundreds of enrolled students, weekly homework, practical projects, etc. IMO somewhat centralized DS tools are more suited to handle this load than an individual contributor (a professor in this context).
You have a point about the amount of money spent on admin at Universities, but I think you (and most commenters here) don't really realize how bad the data situation is at a lot of major universities. If we want to streamline admin overhead at public universities, there's an incredible amount of time and money wasted due to extremely inefficient use/access to data.
I work as a BI dev at a large Big 10 school (50k students/35k faculty and staff). For all those students and staff, there are a grand total of 3 data engineers, who are responsible for all our central data warehouses, including DBAing our oracle and redshift DBs. Just to get a new (untransformed) table added to the data warehouse from a source takes 6+ months, and that's only if it's from a source with an existing integration.
On top of that, there are at least 50 people I know of whose job is basically to produce one or two reports manually in excel every week, and this isn't even considering people in finance or accounting. I'm talking about simple things like how many active research grants do we have, or how many students have enrolled in certain courses. These are reports (and entire fte positions) that could easily be automated with a single SQL query. Speaking of SQL, outside of the data engineering team, there are only 4 or 5 people who know any SQL out of the 100 I know of in reporting/BI positions. The "advanced" data teams are using MS access as an ETL tool to pull together data for tableau reports.
However, there are a lot of institutional issues that make fixing those problems difficult. For one, while we have a central IT dept, we also have about 10 individual college-level IT teams, which means that data isn't just in different databases, but on a whole separate network. For example, if I want to create a report on student faculty ratio, I need to connect to VPN 1 to export faculty data from redshift, then switch to VPN 2 to export student data from Oracle, then switch to VPN 3 so that I can upload both datasets to our depts SQL server. After all that I can finally write a SQL query to get a student faculty ratio. Oh and when we need to update that ratio in a month, I'll have to go thru the whole manual extract/load process again. Forget automating that, since the network teams have no incentive to allow any tunnelling or bridging from one network to another.
I could rant about this all day, but I think it's fair to say that there are still a ton of low-hanging fruit inefficiency-wise at universities. If we could get universities to value their data more highly, maybe that wod have the additional effect of solving some of these problems and even be a net money saver.
If such proposals would result in reduced expenditure on administration, it would obviously be beneficial.
However, I note that your comments largely remark on an insufficient attention given to administration.
The trend in Universities over the past few decades has largely been to increase administration efforts, without that having a notable benefit on student outcomes.
I was in a social network research laboratory. Some of the best people at the lab went into industry while the others did do post docs. The professor that worked at the lab also went to Microsoft Research. Honestly if you were to go the route of being a professor or from MANGA what would you choose? Also, I think there are laws and also rules within Universities that would preclude the study to capture value and even then you would have to hire data analysts that would be external to being a professor so I think it would be hard to compete with MANGA in that regard also.
Oh good - an opportunity to use "begging the question" for its original meaning.
I don't see a single example of "success" in the article. (They said "We unexpectedly found a pervasive void of infrastructure thinking and a relatively limited set of data-informed planning successes" but I must have missed the successful example.)
So the entire article describes what they think should be done, without any successful examples to show that anything useful can be done in the first place.
Obviously I’m 20 years past university and nearer being the paying parent, but why oh why should university “capture the value of data”. They should offer affordable education (if you believe in the value of tertiary education) and affordable signaling of abilities (if you don’t). A recent post by John Cochrane [1] somewhat pointed out the absurdity in US education, Stanford specifically. Stanford itself mentions 15750 non teaching staff members on
2288 professors. My kids K12 (EU) runs on about 6 administrators (excluding cleaning, but including all other staff) for 22 teaching staff, including upstream shared services in the city. In a high school the ratio is probably even better.
15750 administrators wanting to capture the value of data is just plain silly. (I’m going overboard here, but it’s about an order of magnitude of difference! I know of 50% differences in FTE/value in the market I work in, but x10…)
Public schools depend on a lot of administrative support outside the physical school its self. Buss drivers for example aren’t managed at the individual school level because they transport students to multiple different schools.
This extends through a huge range of administrative functions for everything from calling snow days to collecting taxes to pay for the school etc.
A big chunk of that 15,000 are professionals engaged in research activities (mostly funded by grants and other external funds) e.g. the 1600 folks involved in running the Stanford Linear Accelerator, and activities like fundraising and finances. They mostly are NOT folks 'administering' the faculty in any way. So the comparison with your kid's school is not meaningful. Not to say there are perhaps more administrators than might be ideal. But these numbers aren't helpful in understanding whether that's true.
Also, it's worth noting that Stanford is in fact free for folks under the median household income in the U.S. (roughly speaking) which seems pretty 'affordable'.
Of course the economics of one of these large-endowment, research intensive institutions is pretty much unrelated to their teaching function. But that just highlights the weakness of using gross, whole-institution numbers of people/dollars for any sort of comparison. Big universities serve lots of ends (not just teaching undergrads) and so the teasing apart the economic picture (and whether it's efficient at meeting it's many goals) is complicated.
As a data scientist, I think most data is useless, but there is an addictive, video game-like quality to throwing lifeless spreadsheets into a machine and having colorful visualizations come out. It's kind of like a very boring video game for adults that makes them feel like they're working, when they're actually just enjoying colorful abstract shapes and colors. To be honest, this is probably a sizable piece of why I'm in this line of work.
Agreed! It all probably started with big tech promoting the “data-driven decision” paradigm. Of course, in many cases this approach is effective, but it’s not a panacea and has its limits. It’s tempting to interpret availability of any data as an amazing untapped resource, but in many cases analyzing it is just a massive waste of resources and could be more effectively replaced with traditional tools (surveys and such).
> It's kind of like a very boring video game for adults that makes
them feel like they're working,
Absolutely love it.
Let's distinguish a few things though. "Data science" seems like a
pretty weird name. I mean, it's just "Science" right. Of course
there's statistics, mathematics, signal processing, systems analysis,
machine learning... all the good things that you and I are into.
But how does this get huddled uncomfortably beneath the umbrella "Data
science"?
I think the answer is found by asking about the ends of data
science, the old Cui Bono?
There's the raw entertainment value you mention. It's cool to have
knowledge and visualise it. Sensors, transducers, processing is fun.
Then there's legibility. That is political and is about control.
What most scientists are doing with data is either hypothesis
testing or combing for causal relations to then abductively feed back
into hypothesis generation.
What most business people are trying to do is optimise, and adjust
constraints and parameters. It's modelling for the most-part. It's
ancient and goes back to linear analysis and regression from before
the last century.
Security people are looking for stress signifiers, suspicious patterns
with various triggers, selectors and tripwires.
Financial people want fortune telling. They want the models to
extrapolate into beautiful hockey sticks.
Within any organisation we may need to do one, a few, many or none at
all of the above. The problem then is that "Valuable data" is such a
broad, open prospect it seduces gushing, credulous administrators into
valuing the process, and the tools, but not the ends.
The job description for a data scientist isn't necessarily consistent across companies, but I can imagine a good description would be to act like a real scientist in that you collaborate with product teams to figure out what experiments to run, what data to collect, and how to visualize that for both the product and business team to decide on their approach to make the company more money, ie. specializing in a/b testing and experimentation. A problem I see is creating a data science team just to follow the trends and specifically requesting they try to "study user behavior" or something in hopes the data shows some underlying trend that triggers an "eureka!" moment.
Agree that the term data science is strange. Data engineer seems like a better name. Science implies rigorous hypothesis testing guided by a theories of how particular systems work. Not sure that applies to most “data science” work.
Only slightly related to your very funny comment here, you might enjoy “The Visual Display of Quantitative Information” by Edward Tufte. Not only does he help you to best make the abstract shapes and colors, but there’s an added aspect of the visuals that tickles the mind. That added aspect is how easily some information is communicated along with the image. It’s along the lines of how sometimes someone can tell a joke just by saying something as concisely as possible (even if the content isn’t humorous).
Like you said, whether or not the visuals provide any value is a completely different line of discussion.
This definitely applies to programming as well. It's not uncommon to see people happily churning out mountains of highly redundant and repetitive code, or zealously pursuing far-reaching nit-picky refactors of dubious benefit and I think it's simply because the act of writing code and running tests and watching them go green is fun.
You perfectly described why I became disillusioned with any kind of data science job. The feeling that most of the time data is used to support preexisting bias and can be arranged to say anything you want. Interesting that you know this and enjoy it despite that.
It seems like a somewhat weird mismatch — the described benefits are things like curriculum improvements and hiring better instructors, which I think most people (including most professors, I think, despite the fact that they’d be on the receiving end of this) would be in favor of.
But including security cameras in the list of tools seems… odd? What would they tell you? Classroom attendance I guess. But all that tells you is that the instructor is either so bad that nobody bothers showing up, or so good that their slides, recordings, etc are enough for the students to skip class sometimes. (And if you measure the number of students in the classroom, attendance will just become mandatory, this is dumb).
In general universities should be pretty inefficient I think. They are places where young people go to learn and try things out. Including student employees. If a university has no waste, that indicates that it hasn’t given enough possibly-unqualified undergrads enough resources to accidentally misuse.
> In general universities should be pretty inefficient I think. They are places where young people go to learn and try things out. Including student employees. If a university has no waste, that indicates that it hasn’t given enough possibly-unqualified undergrads enough resources to accidentally misuse.
The problem is all the extra money (yes, pretty much all of it) has gone to building up the administrations, not into resources that actually help students. And the students certainly haven't turned out smarter as a result.
Universities have turned into a jobs program for people looking to work in education administration.
Working in a top-tier computer science department, I find our ability to answer basic questions about the health of our degree program fairly troubling. I think non-academics may be surprised by how much we don't know, and how little useful and continuous data analysis is taking place.
For example: What is our retention rate? Meaning, what percentage of students who start our degree programs complete it. A fairly standard and important indicator of program health. Next, break this down by various cohorts: What is our retention rate among women? And so on. Heck, frequently we can't even answer questions about the current gender ratio within our program—and this is something that has been a focus of our diversity efforts recently.
I've had people say with a straight face that we _cannot_ calculate retention because we don't know when students leave our program. But of course someone knows this! And I've been able to produce rough estimates even given the limited data that I have access to. But a lot of educational data is fairly siloed, and frequently the people assigned to perform these tasks don't have much training and tend to give up quickly.
I suspect that many departments just don't have anyone assigned to do even basic educational data analysis on a regular basis, and with access to enough data to run interesting reports. My department is in the process of creating a faculty leadership role around academic data analytics, but my sense is that this will be a very unusual position. (And don't worry—it'll be filled by a faculty member, and not a new administrator.)
And don't even get me started about student evaluations of teaching. Yes, we give a survey at the end of every semester and ask students whether they liked a particular course and professor. No, those answers have very little to do with how much they actually learned. Yes, we could measure learning in other better ways—success in downstream courses, for example. No, people don't tend to do that.
There's a lot of room for improvement here, just working with the data we already have. No need for additional "telemetric signals".
Then you attempt to apply those ideas. It is at this point that the person responsible for applying those ideas says "been there done that".
Point is, most data dashboards are non actionable. The challenge is to create a good actionable dashboard (i.e. if values cross a certain threshold, then the user should take some action on it).
Once you create an excellent actionable dashboard, you realize it doesn't need a dashboard. It can be a notification.
So, while the data is important, the questions around it might just lead to the same work that was being done anyway.
> The challenge is to create a good actionable dashboard
Huh? You don’t need a dashboard to make use of data. The best use of data in my mind is asking and answering questions.
Eg, maybe your program has a low number of graduating female engineers. Why? Maybe they’re dropping out along the way. Maybe female intake is low. With the data you can answer these questions.
The graduating rate of female CS students is low, but is it abnormally low compared to other schools? You investigate and - everyone has an equally low rate except one place where it’s 50/50. The data has led to a question - Why? What are they doing differently? And so on.
> Point is, most data dashboards are non actionable.
Maybe you find out that one specific class or set of classes is responsible for a lot of people leaving the program. So you zero in on that part of the curriculum and improve it. Sounds pretty actionable to me. We've actually done similar things on a smaller scale (week by week) to improve student success in my course.
But of course you don't know anything until you do the analysis.
In my experience it is generally worse than that. If a program has a retention rate of, say, 30% then it isn't like that is going to come as a surprise. Data generally reveals things that are very obvious. It is easier to read a situation by talking to a few people and asking simple questions. Which raises further questions about why a data-driven approach is needed.
The value in data driven approaches is high, but it takes a rare person to figure out why. Traditionally data has actually been a communication tool for things that are already known. That isn't at all how people expect it to be used, everyone seems to anticipate it is used to make decisions.
An organisation resisting data is bad news because it will struggle to talk about things that everyone knows to be true.
Hazarding an uninformed idea: If organizationally the department isn't geared toward this kind of analysis, what if you opened up the data, and let whoever wanted to work with it do their thing. Probably a perfect environment for crowd-sourcing, given the expertise and resources.
Perhaps one problem is that every college has its own bespoke curriculum and processes, so every data problem is a "little data" problem. Of course there are lots of rationalizations for why every program needs to be unique and special, but does it really benefit the students?
A similar problem in medicine: Every clinic system has a unique set of business processes, and a custom build of Epic. Granted the clinics are competing on which one can develop the most efficient processes, but does the patient benefit?
> Of course there are lots of rationalizations for why every program needs to be unique and special, but does it really benefit the students?
Of course not. What would benefit the students would be having a lot more standardization so that we compare ideas and approaches and determine what works. But the problem with standardized evaluation is that half of the programs suddenly discover that they aren't in the top half—as most of them had previously thought. This seems like more or less what happened to standardized testing in K–12 education.
But, in the context of a specific curriculum, having no idea what is happening and therefore no way to improve your bespoke curriculum is even worse than just deciding to do things your own way.
And it's worse than medicine, because at least they have some common metrics for what it means to be healthy. Whereas, faculty get to assign grades however they want! Imagine you ran a diet study where you both controlled the meals and got to reposition the numbers on the scale at will.
An anecdote from someone in IT at a major university: The registrar has two employees whose sole job is to write SQL queries. These are to answer basic questions such as, "How many undergraduates are currently enrolled." And it turns out that this is a nearly impossible question to which to give a definitive answer.
What does it mean to be enrolled? What if someone takes a semester off? What if they are part time? What if they are on a foreign exchange program this year? What if ...
The above are the type of questions that can result in different definitions of enrolled.
Oh, so we need a Vice-provost of Data. That's what is keeping universities, otherwise homogenous and well structured, from aligning on data infrastructure and strategy.
Because it does not "belong" to them. It belongs to all science and all society, whom will through creative destruction find ways to extract the value for everyone.
The same applies to corporations, landholders, monarchs and parliaments and governments.
The abstract suggests higher ed administrators collect data from “research, administrative operations and other sources” but there is one source not mentioned: donors.
I am absolutely sure that donor data is organized, systematized, useful and generates positive ROI.
It’s surprising that this study would not mention donor data (unless they had a predetermined “struggle with data” point of view in mind)
> The study’s authors contend that universities have been slower than organizations in other economic sectors to create senior-level positions focused on data quality, strategy, governance and privacy matters.
Universities are not an economic sector; they aren't profit-making enterprises. They are knowledge- and education-making enterprises.
Do they need more employees who aren't creating knowledge and educating? My understanding is that universities have been greatly expanding such non-core functions.
That’s exactly what universities are these days. They sell a signal that says “this worker can do the bare minimum you need” called a Degree. We make the worker get certified themselves, but this is the purpose. To act like modern Universities do anything else today is disingenuous in my opinion. It’s like saying people who work in finance are there to allocate capital efficiently. No, they’re there to make money. That’s it.
It's a popular meme but IMHO does not reflect university-level analysis. That exists, but reductionist arguments merely take the worst of someone or something and blow that up to 100%.
For example, everyone feels self-interest, but the reductionist claim it that it's all we are. A simple look at the evidence shows that it's manifestly untrue; people are much more than self-interested.
Similarly universities do and are much more. I know university students and they are learning a lot - I'm very impressed with the thought and creative effort put into the cirriculum.
The reductionists really hurt themselves and people who listen to them. They wall themselves off from all the good in the world, including education.
True, the reductionism is no good. I guess to me I’m tired of discourse that leans towards “universities are places where we make you a complete human”. The real state of things is more what I describe, with a dash of making you deeper. Not vice versa.
I agree it can go too far the other way too. People come with many different motives and experience many different things.
> I’m tired of discourse that leans towards “universities are places where we make you a complete human”.
I'm surprised that's said much. I see the reductionism as the mainstream; when I say otherwise, I'm 'politically incorrect'. On HN, certainly the reductionism is much more common. Isn't what you quote now the outsider view?
1. Systems in higher Ed often do not talk and departments tend to be siloed and have drastically different goals and objectives.
2. There are a lot of privacy concerns as most of the data is students data.
In addition to this, I question what the end goal of the data analysis is? What is the value for a not-for-profit? The goal is going to be based on the department as a college doesn’t have the same profit motive. The individual departments typically don't have the resources to do deep data analysis.
There are so so so so many reasons for this. Research data: hard to centralize because subject to impossible conflicting demands from funders and human subjects IRBs, and in the hands of ferociously independent faculty. Students: privacy laws and worse than that over-cautious university counsel and administrators who are afraid of running afoul of those laws. Administrative data: stuck in a bunch of individual bureaucratic buckets that don't talk to one another because universities are badly managed in general, also in the clutches of horrific enterprise software platforms made by the worst companies in the world (like oracle) and administered by IT people who aren't paid market rates.
University data projects tend to succeed if and only if they're turned over to librarians, who are typically the only people on campus who have any clue how to do such a thing.
Data is not information. Lots of data is worthless if you don't know what you're looking for.
Perhaps we could actually do something about this if we focused strongly on falsification, but right now there's just going to be a whole lot of Wittgenstein's ruler going on; what people want to look at or see will come first with such an abundance, and there's going to need to be some kind of real filter to make it useful.
If it's bad in universities, how much of this can we expect to be true for the entire "big data" economy we've built a gigantic part of the internet off of?
96 comments
[ 0.20 ms ] story [ 190 ms ] threadI personally think this is wrong. Big tech companies should build hardware, while government agencies should use that hardware to curate our data. Like how it was near the beginning of the internet. Companies should be as far away from our data as possible.
The rush to "exploit" data reminds me of the dot com hype. It's one thing to use available data to make more informed decisions about things from course content to building occupancy. It's quite another to rush towards total surveillance because of a Fear of Missing Out of "exploitable data".
That's the primary value to be captured right there! Their privacy is highly valued, monetarily:
> authors contend that universities have been slower than organizations in other _economic_ sectors
Education in the US is just pure business.
https://www.projectcensored.org/ferpa-and-higher-ed-should-p...
Yes, absolutely, universities are awash in data they do little to anything with outside of hyperfocused laboratories. I wonder if having central data management groups would be an enabler or a bottleneck, however.
They have 0 visibility into any of the operations and therefore can’t derive any insights or install controls.
They have no idea if:
Some professors more effective teachers than others.
Which days students buy more food.
Whether taking a given prerequisite leads to better mastery of advanced material.
If additional spend is correlated with better student outcomes.
For people in tech businesses, it’s such a shocking thing to read it’s hard to fathom how any of the leadership at universities find this situation acceptable.
Teaching outcomes are evident from exam results, grants won, papers published etc and are routinely used to determine promotions, staffing and more. Food/footfall etc are monitored and predicted by the private catering companies on campus. Budget reviews are comprehensive and require yearly and greater analysis of university performance on a per department basis. All that doesn't even include competitive processes like ranking tables for universities, applications for grants, participation in international research groups etc etc.
What are you even talking about? Literally every course at most universities end with an evaluation. Obviously, there are other problematic ways to measure effectiveness (publications). How much that data gets used is another question and varies widely by institution. “People in tech businesses” love to exaggerate the quality of the data they have and its usefulness, hence the Subprime Attention Crisis that is rapidly eroding their cash on hand as the Fed’s cheap money and QE dries up.
I’m in a tech business and it doesn’t sound shocking at all. The only things people really have data for is how many people are giving them how much money.
I think this article rings true for most universities, but for some, like mine, we’ve had unified systems for the last 20 years. While that’s been helpful in a lot of ways, the primary thing we’re looking for - helping students graduate and get better jobs than they have today - is surprisingly challenging. Adult learners often fail to finish school for a variety of reasons, most of which aren’t academic and often can’t easily be forecast using all available data.
Unifying data sources or mining them further doesn’t solve student health issues, childcare, jugging work responsibilities with school, etc.
What’s more, universities are slow-moving, heavily bureaucratic, and highly regulated. So even when we have good ideas to relieve burden those often come head to head with accreditation rules, financial aid regulations.
In my opinion, for my students at least, data is the least of our problems.
The funniest part was the example of security cameras. What big breakthroughs are universities hoping to achieve with this data?
This 4 parter I wrote for Techrights [1] and the Times article that seeded it [2] go into detail.
[1] http://techrights.org/2022/12/28/andy-farnell-on-british-uni...
[2] https://www.timeshighereducation.com/campus/eliminating-harm...
It's a prophecy that demands to be fulfilled.
Starting with the premise "Data is useful", it proceeds to pick at all the ways we've failed to make it useful... and we're damned well going to make it useful if it kills us!
Maybe, just maybe (for those that dabble in the sceptical, explanatory game we call science) it might be that "bare data" has little use within certain contexts... say those that by definition are on the cutting edge of knowledge and reality, best steered by imaginative vision of leading experts.
Data driven market cybernetics may be very useful in some industries, such as ones with physical logistics, complex supply chains, rapidly shifting supply sources and demand sinks. But the primacy of "data uber alles" should not be one-size-fits-all.
Having data can undeniably be useful. The question is, how much ROI does the insights in the data provide, relative to the cost to build the organizational infrastructure capable of extracting these insights?
And there's a bootstrapping problem, because often it's hard to calculate the potential ROI without actually going through the work of building the infrastructure to process the data.
> how much ROI does the insights in the data provide, relative to the cost to build the organizational infrastructure
Having seen the workings of academia for a long time now I'd argue very little.
But I think the problem is more subtle. A Heisenbergian problem. When data collection involves people you have a triple problem of intrusion, distraction, and ossification. the act of trying to create a "data driven culture" in some contexts simply kills what's there. The cost of this, in addition to the bootstrapping costs of sensors and analytics are paralysing.
Academia, by its very nature, if properly functioning, demands to be dynamic. If it's good, it will not stand still long enough for you to look at it. And if you measure it "too hard" it will evaporate as all the good people who resent ossification leave.
There can be too much data. There are questions you cannot answer even by collecting all data, unless you have no time or food cost constraints but the value of your answer will decrease with the time distance from the moment the question was asked. In a million year you might know for sure who was communist in Atlanta during the 2000s, year by year, block by block, but you may care less by then.
I read a comment on here months ago about someone who enabled a hotel chain to process the data coming from the motion activated lights in every hotel. The chain was able to track workers with this data and cracked down on people taking too long of breaks.
And now a personal anecdote. The business I used to work at had an ID badge door you had to use to enter the smoking area. I was told by someone I trusted that when layoffs came, a member of management asked for a list of the people that used that door. The people on that list were prioritized for termination.
"Universities are literally awash in data. From administrative data offering information about students, faculty and staff, to research data on professors’ scholarly activities and even telemetric signals"
Why are professors' telemetric signals even being discussed? Are publicly listed office hours not enough? My dogs got chipped without their consent. Do professors deserve more dignity than my dogs?
Just look at the treadmill of architectures designed to "solve" this problem: integrated databases to "data warehouses" to "linked data" to "data lakes" to "data fabric" to "data mesh."
Organizations that succeed do so because they have the resources to prioritize data management, and do so at significant team size and expense. But any organization that doesn't (or can't) view data curation as first class, top-line budget isn't in a position to capture even a small part of the theoretical value of their data.
Consider the "data mesh" architecture, the latest fad in this space. It basically assumes that not only do you have a dedicated data curation team, each data source also has it's own dedicated team of curators capable of productizing data sources.
That kind of thing is simply out of reach for most organizations.
A big part of the effort involved providing customers with big data solutions and all the implied benefits that would entail. They jumped aboard the hype train with gusto. Each quarter progress was reported and margins improved with a clear upward trend. From the trenches it was clear that even with the investment made, it was still not enough to reach any of the goals.
After a year of “modernization” the company was acquired, the buyer literally salivating over the coming profit margin increase and “synergies” in selling into related markets. After another year post acquisition it became clear that the benefits were simply out of reach, and the profits did not arrive. The acquiring company was left with the low margin business it had always been. Layoffs ensued as the buyer sought to salvage something from the transaction.
Turned out the entire modernization and big data effort had been an elaborate bait and switch committed by the board and CEO of the acquired company, masterfully executed.
Are you saying that rhetorically, or do you actually believe (perhaps with evidence) that the modernization and big data effort was not undertaken in earnest?
I feel like this part you omitted clarifies my opinion on the matter.
[1]: https://martinfowler.com/articles/data-mesh-principles.html
But the problem exists even on a concrete technical level. Look at how many different "big data" products are out there. The Apache project alone has probably 15 different tools that largely overlap.
Or: This lecture brought to you by Pepsi.
I had the same thought - intrusive analytics and a focus on exploiting and monetizing client data seem to have damaged the web, made gaming less fun, and made everything user-hostile and creepy. Do we really want universities to go down that path?
There's no need to encourage them to start treating their mission as one of data mining in order to "capture" more value from the students.
I work as a BI dev at a large Big 10 school (50k students/35k faculty and staff). For all those students and staff, there are a grand total of 3 data engineers, who are responsible for all our central data warehouses, including DBAing our oracle and redshift DBs. Just to get a new (untransformed) table added to the data warehouse from a source takes 6+ months, and that's only if it's from a source with an existing integration.
On top of that, there are at least 50 people I know of whose job is basically to produce one or two reports manually in excel every week, and this isn't even considering people in finance or accounting. I'm talking about simple things like how many active research grants do we have, or how many students have enrolled in certain courses. These are reports (and entire fte positions) that could easily be automated with a single SQL query. Speaking of SQL, outside of the data engineering team, there are only 4 or 5 people who know any SQL out of the 100 I know of in reporting/BI positions. The "advanced" data teams are using MS access as an ETL tool to pull together data for tableau reports.
However, there are a lot of institutional issues that make fixing those problems difficult. For one, while we have a central IT dept, we also have about 10 individual college-level IT teams, which means that data isn't just in different databases, but on a whole separate network. For example, if I want to create a report on student faculty ratio, I need to connect to VPN 1 to export faculty data from redshift, then switch to VPN 2 to export student data from Oracle, then switch to VPN 3 so that I can upload both datasets to our depts SQL server. After all that I can finally write a SQL query to get a student faculty ratio. Oh and when we need to update that ratio in a month, I'll have to go thru the whole manual extract/load process again. Forget automating that, since the network teams have no incentive to allow any tunnelling or bridging from one network to another.
I could rant about this all day, but I think it's fair to say that there are still a ton of low-hanging fruit inefficiency-wise at universities. If we could get universities to value their data more highly, maybe that wod have the additional effect of solving some of these problems and even be a net money saver.
However, I note that your comments largely remark on an insufficient attention given to administration.
The trend in Universities over the past few decades has largely been to increase administration efforts, without that having a notable benefit on student outcomes.
I don't see a single example of "success" in the article. (They said "We unexpectedly found a pervasive void of infrastructure thinking and a relatively limited set of data-informed planning successes" but I must have missed the successful example.)
So the entire article describes what they think should be done, without any successful examples to show that anything useful can be done in the first place.
15750 administrators wanting to capture the value of data is just plain silly. (I’m going overboard here, but it’s about an order of magnitude of difference! I know of 50% differences in FTE/value in the market I work in, but x10…)
[1] https://johnhcochrane.blogspot.com/?m=1
This extends through a huge range of administrative functions for everything from calling snow days to collecting taxes to pay for the school etc.
Also, it's worth noting that Stanford is in fact free for folks under the median household income in the U.S. (roughly speaking) which seems pretty 'affordable'. Of course the economics of one of these large-endowment, research intensive institutions is pretty much unrelated to their teaching function. But that just highlights the weakness of using gross, whole-institution numbers of people/dollars for any sort of comparison. Big universities serve lots of ends (not just teaching undergrads) and so the teasing apart the economic picture (and whether it's efficient at meeting it's many goals) is complicated.
Absolutely love it.
Let's distinguish a few things though. "Data science" seems like a pretty weird name. I mean, it's just "Science" right. Of course there's statistics, mathematics, signal processing, systems analysis, machine learning... all the good things that you and I are into.
But how does this get huddled uncomfortably beneath the umbrella "Data science"?
I think the answer is found by asking about the ends of data science, the old Cui Bono?
There's the raw entertainment value you mention. It's cool to have knowledge and visualise it. Sensors, transducers, processing is fun.
Then there's legibility. That is political and is about control.
What most scientists are doing with data is either hypothesis testing or combing for causal relations to then abductively feed back into hypothesis generation.
What most business people are trying to do is optimise, and adjust constraints and parameters. It's modelling for the most-part. It's ancient and goes back to linear analysis and regression from before the last century.
Security people are looking for stress signifiers, suspicious patterns with various triggers, selectors and tripwires.
Financial people want fortune telling. They want the models to extrapolate into beautiful hockey sticks.
Within any organisation we may need to do one, a few, many or none at all of the above. The problem then is that "Valuable data" is such a broad, open prospect it seduces gushing, credulous administrators into valuing the process, and the tools, but not the ends.
Instead we have social science, computer science and now data science.
Have you looked at the Financial Modeling World Cup? https://www.youtube.com/channel/UCOlnCUAKLENyFC8wftR-oNw
Its tagline is: "Excel Esports. Yes, It's a thing"
Like you said, whether or not the visuals provide any value is a completely different line of discussion.
But including security cameras in the list of tools seems… odd? What would they tell you? Classroom attendance I guess. But all that tells you is that the instructor is either so bad that nobody bothers showing up, or so good that their slides, recordings, etc are enough for the students to skip class sometimes. (And if you measure the number of students in the classroom, attendance will just become mandatory, this is dumb).
In general universities should be pretty inefficient I think. They are places where young people go to learn and try things out. Including student employees. If a university has no waste, that indicates that it hasn’t given enough possibly-unqualified undergrads enough resources to accidentally misuse.
Universities have turned into a jobs program for people looking to work in education administration.
For example: What is our retention rate? Meaning, what percentage of students who start our degree programs complete it. A fairly standard and important indicator of program health. Next, break this down by various cohorts: What is our retention rate among women? And so on. Heck, frequently we can't even answer questions about the current gender ratio within our program—and this is something that has been a focus of our diversity efforts recently.
I've had people say with a straight face that we _cannot_ calculate retention because we don't know when students leave our program. But of course someone knows this! And I've been able to produce rough estimates even given the limited data that I have access to. But a lot of educational data is fairly siloed, and frequently the people assigned to perform these tasks don't have much training and tend to give up quickly.
I suspect that many departments just don't have anyone assigned to do even basic educational data analysis on a regular basis, and with access to enough data to run interesting reports. My department is in the process of creating a faculty leadership role around academic data analytics, but my sense is that this will be a very unusual position. (And don't worry—it'll be filled by a faculty member, and not a new administrator.)
And don't even get me started about student evaluations of teaching. Yes, we give a survey at the end of every semester and ask students whether they liked a particular course and professor. No, those answers have very little to do with how much they actually learned. Yes, we could measure learning in other better ways—success in downstream courses, for example. No, people don't tend to do that.
There's a lot of room for improvement here, just working with the data we already have. No need for additional "telemetric signals".
Then you find out why retention is low.
Then you brainstorm ideas to increase retention.
Then you attempt to apply those ideas. It is at this point that the person responsible for applying those ideas says "been there done that".
Point is, most data dashboards are non actionable. The challenge is to create a good actionable dashboard (i.e. if values cross a certain threshold, then the user should take some action on it).
Once you create an excellent actionable dashboard, you realize it doesn't need a dashboard. It can be a notification.
So, while the data is important, the questions around it might just lead to the same work that was being done anyway.
Huh? You don’t need a dashboard to make use of data. The best use of data in my mind is asking and answering questions.
Eg, maybe your program has a low number of graduating female engineers. Why? Maybe they’re dropping out along the way. Maybe female intake is low. With the data you can answer these questions.
The graduating rate of female CS students is low, but is it abnormally low compared to other schools? You investigate and - everyone has an equally low rate except one place where it’s 50/50. The data has led to a question - Why? What are they doing differently? And so on.
Maybe you find out that one specific class or set of classes is responsible for a lot of people leaving the program. So you zero in on that part of the curriculum and improve it. Sounds pretty actionable to me. We've actually done similar things on a smaller scale (week by week) to improve student success in my course.
But of course you don't know anything until you do the analysis.
The value in data driven approaches is high, but it takes a rare person to figure out why. Traditionally data has actually been a communication tool for things that are already known. That isn't at all how people expect it to be used, everyone seems to anticipate it is used to make decisions.
An organisation resisting data is bad news because it will struggle to talk about things that everyone knows to be true.
And anonymising data sources is a notoriously fraught task.
A similar problem in medicine: Every clinic system has a unique set of business processes, and a custom build of Epic. Granted the clinics are competing on which one can develop the most efficient processes, but does the patient benefit?
Of course not. What would benefit the students would be having a lot more standardization so that we compare ideas and approaches and determine what works. But the problem with standardized evaluation is that half of the programs suddenly discover that they aren't in the top half—as most of them had previously thought. This seems like more or less what happened to standardized testing in K–12 education.
But, in the context of a specific curriculum, having no idea what is happening and therefore no way to improve your bespoke curriculum is even worse than just deciding to do things your own way.
And it's worse than medicine, because at least they have some common metrics for what it means to be healthy. Whereas, faculty get to assign grades however they want! Imagine you ran a diet study where you both controlled the meals and got to reposition the numbers on the scale at will.
The above are the type of questions that can result in different definitions of enrolled.
Stop running dragnet data collection on unwilling participants and stop using it to try to “optimize” your interactions with them.
This data shouldn’t exist, it’s value shouldn’t be exploited.
The same applies to corporations, landholders, monarchs and parliaments and governments.
New year is bringing out my fundamentals :-)
I am absolutely sure that donor data is organized, systematized, useful and generates positive ROI.
It’s surprising that this study would not mention donor data (unless they had a predetermined “struggle with data” point of view in mind)
Universities are not an economic sector; they aren't profit-making enterprises. They are knowledge- and education-making enterprises.
Do they need more employees who aren't creating knowledge and educating? My understanding is that universities have been greatly expanding such non-core functions.
For example, everyone feels self-interest, but the reductionist claim it that it's all we are. A simple look at the evidence shows that it's manifestly untrue; people are much more than self-interested.
Similarly universities do and are much more. I know university students and they are learning a lot - I'm very impressed with the thought and creative effort put into the cirriculum.
The reductionists really hurt themselves and people who listen to them. They wall themselves off from all the good in the world, including education.
> I’m tired of discourse that leans towards “universities are places where we make you a complete human”.
I'm surprised that's said much. I see the reductionism as the mainstream; when I say otherwise, I'm 'politically incorrect'. On HN, certainly the reductionism is much more common. Isn't what you quote now the outsider view?
1. Systems in higher Ed often do not talk and departments tend to be siloed and have drastically different goals and objectives.
2. There are a lot of privacy concerns as most of the data is students data.
In addition to this, I question what the end goal of the data analysis is? What is the value for a not-for-profit? The goal is going to be based on the department as a college doesn’t have the same profit motive. The individual departments typically don't have the resources to do deep data analysis.
University data projects tend to succeed if and only if they're turned over to librarians, who are typically the only people on campus who have any clue how to do such a thing.
Perhaps we could actually do something about this if we focused strongly on falsification, but right now there's just going to be a whole lot of Wittgenstein's ruler going on; what people want to look at or see will come first with such an abundance, and there's going to need to be some kind of real filter to make it useful.