Even if storage is cheap, they can charge for their service. Or possibly abusers are too aggressive, see r/DataHoarder. Also Chia faming on cloud storage is possible.
I'm surprised they weren't relying on internal HPC resources. Here Stanford has centralized almost everything to use Sherlock[0], and apparently it's quite a nice environment.
Starfish is getting great traction in universities and labs. It can do amazing things for storage analysis and migration across filesystems and cloud buckets.
What is it with Google pulling off the bait and switch on so many products at once? Unlimited high quality photos for Pixel owners, free workspace for personal/family users and unlimited drive storage for education are all going paid over the span of ~a year. It's almost like Google's exploding from within.
This isn't the first time, nor the last. They've been doing this for years as part of their kill old products and replace them with similar new one products culture. Google Music comes to mind.
If that's the case, and not a deliberate aggressive strategy to acquire users ("bait and switch"), Colab users playing with GPT on free TPUs and GPUs should be by far the most worried.
The cost of storage at Google's scale is low. GPUs/TPUs are another thing entirely.
The community of https://fakeyou.com users has been using Colab to train TTS models. Changes late last year to available free instance types forced most of our users to have to pay for the pro plan.
Storage is cheap, but egress/ingress in and out of gcp is not. I suspect the real reason Google are doing this is a mixture of the two, to reduce network costs and because they've got enough users now.
My guess is where it's been going for a while, in line with the signals sent by hiring their new CFO - juicing shareholder value by cost-cutting now that the easy gains have been had and "natural growth" is slowing.
So they just added an entire 2018 or 2019 size profit to their outcome in the past year.
That didn't come from squeezing pennies on little storage products and the like. That's from one of the world's largest snowballs continuing to roll down hill, piling up tens of billions in trivially easy profit addition as they go. When it comes to expanding margins, for an organization the size of Google that is still expanding relatively quickly in sales, it's far more important to hold down cost expansion as the profit soars (don't add those entirely unnecessary 20,000 heads) than it is to cut rounding error costs. Microsoft has similarly been masterful about doing that lately (unlike during the Ballmer era).
Most likely. Search continues to be strong and nothing has come close to bypass the moat. Facebook on the other hand, their moat has been broken through and they're knocking at the door of the castle :P
Their deal with Apple ensures their staying power. They own the other half of the Mobile ecosystem so even when lockdowns end and travel opens up there may be even more usage.
Same cash cow as always. Money from companies who think we browse the web without AdBlockers. A little bit YouTube adds. They tried to spin it as growth on Google Cloud, but its minimal, compared to the cash cow.
It's hard to tell if it's constantly shifting priorities and budgets causing projects to be retired, or a strategy to bring users into their walled garden and then upcharge once they are onboard.
It is incredibly doubtful they ever really intended to give away boundless amounts of storage forever. It is far more probable that they were trying to entice customers.
This is most probably the effects MBA's / late stage growth cycle.
Google had all this potential profit lying around, and then someone came around and wanted to get a bonus out of it.
Google should watch out, if people no longer get any perceived utility from their services, they will find themselves destitute. A true ad company with a search engine and no innovation as Peter Thiel describes them.
Opposite effect for me. I was getting workspaces for free to basically send and receive five emails a month or so. I pay Google for Google One storage, Stadia, and YT Music. Their reneging of workspaces is going to lead to me giving Google zero dollars per month. I'm sure I'm a rounding error to them, but perhaps others are in the same boat.
But you missed the key point. The goal was not to get more money for Google. The goal was to get a fat bonus. Doubling an orgs profit by destroying value requires some good, high-level holistic analysis to uncover. Meanwhile the VP in question will be doing everything in their power to obfuscate that, at least until after the next comp planning cycle.
I mean this is what companies build up revenue for, to be able to set prices when inflation happens. Googles services are undeniable. Now they are in position to start justifying their high multiples.
I think Google is losing money on their cloud division while trying to gain market share, so they are cutting costs in every other place to make the bottom line earnings look better.
... I think I've been screwed over enough times by Google now that I would never, ever, ever, ever, ever rely on them for anything where I expect business continuity. I've done too much business with Google to ever do business with Google.
I can't imagine their cloud division ever winning with Amazon and Azure exactly because there are happy GSuite/Youtube/etc. customers like me around, making sure no one ever trusts them for anything again.
100% agree... the people making cloud purchasing decisions are also usually the people who are early adopters/enthusiasts for all of these google products.
I think that for a long time the average non SV consumer had the mentality of “I’m not paying a monthly fee for that crap” when it came to online stuff but all the streaming services have shifted that and it’s more normal now to pay a small monthly fee for little subscriptions of everything imaginable. So, my wild guess is that Google is seeing that culture change and realizes it’s now a good time to join everyone else and finally start milking it directly. Probably not accurate but I could see that being part of the conversation.
I'm wondering if privacy regulation (GDPR, CCPA, VCDPA) slowly closes the targeted ads market. Hence, Google starts charging for services that previously were paid for from someone else's marketing budget.
A lot of people seem to have missed the point I was trying to make. I'm not saying that it's unusual for Google to engage in anti-competitive dumping or kill their products; I'm wondering why so many of these actions were timed so close to each other. As it stands, the narrative around Google dumping is more blatantly clear than if they waited longer to phase out some other products (like storage for Universities) or simply didn't, which makes me think that internal or external pressure on Google to monetise its properties more aggressively is ramping up.
The obvious rationale, not wanting to really give unlimited storage, is exactly what their storage policy section said. This is simply a sales tactic that people keep falling for over and over, as if there is really something for nothing forever.
Makes one wonder why it's even legal for companies to just... lie in their marketing. Why aren't they getting sued? For that matter, why aren't cell companies getting sued for "unlimited" data with data caps? Why aren't there laws requiring advertising to be truthful?
If there is a cap of 1 byte per hour, there is absolutely a limit of 720 bytes per 30 day month. Whenever there is a cap, it is not "technically unlimited".
With that logic, everything is capped by the network speed.
And that's why there's typically a 2nd asterisk about speeds dependent on network conditions. Still legally advertised as unlimited, thanks to the fine print.
The BBB's National Advertising Review Board in the U.S. told Mint Mobile to stop using the word "unlimited" to describe its plan that included 35 GB of data and unlimited throttled data (at 2G speeds). (The plan is still not a bad deal.)
According to the blog post, the admings of those organizations were also notified a year ago minus a few weeks:
> We will contact impacted institutions directly in the coming weeks to discuss a range of options for getting the storage they need
Given the quota model, it doesn't seem like notifying individual users would have made sense. It doesn't sound like the original poster would have been a domain admin.
I'm not saying they didn't, but that sentence you quote is future tense. We don't know that they actually did in the time frame that they said they would. Just that they said they would.
I've helped maintain a g suite org, they're pretty rigorous about getting in touch when they're making changes. I know we had plenty of notice about the storage policy changes for example.
Probably when the organization's admin does so. There are many institutions that allow alumnus to keep the Education account. With this change, they have to hastily notify the alumnus that this will no longer be honored, and they should move whatever important out of it before the account gets deleted.
And they can always pay for storage they use above the new free limit:
> Large institutions will be provided supplemental storage later this year and all schools can gain additional storage through Education Plus and the Teaching and Learning Upgrade.
I was told by an SRE on Drive that the unlimited offer was legit because the at the rate you could upload data to drive the costs were still far bellow the bill.
Of course that seemed wrong because if you are uploading for years $5 a month isn't going to keep covering all of that storage. I imagine that part of this assumption that storage costs would keep dropping and presumably this isn't happening as quickly as originally predicted.
Indeed, for a while with Google there have been nominal ingress quota limits of (iirc) 750 GB per day.
It seems that in the context of research universities there may be a mismatch in communication and use of "unlimited". Google said something it didn't mean, then eventually realises and claws it back.
This has happened twice in a few weeks - once for "forever" on free G Suite, and now with "unlimited" for universities.
I'm surprised that they don't get more marketing collateral for selling the credibility of G Suite to enterprise clients from being able to talk about case studies like storing and processing multi dimensional time-series medical imaging data on their storage platform as a way to say "we're ready for your data, and you can feel good while you pay us, as we support this great research"...
Clearly the revenue push has arrived for these services. 100 TB per institution isn't really a lot when you consider research data, but might be more tenable if you consider purely students with the odd slideshow and video. Unfortunately this makes life easier for the less scientifically rigorous fields of study that make a few slide decks, rather than the ones where a student might collect a few TB of data in an hour or two.
A service could support unlimited data by throttling the upload speed, and the more you upload, the slower it gets. Then, they could offer paid accounts that increased the speed.
1) There is no such thing as "unlimited" anything with technology. Either you haven't read the fine print of your provider's terms or your provider will at some point have to change them.
2) What's interesting is that storage costs are dropping precipitously while they're restricting their customer's use of storage space. I just bought an 18 TB SATA drive for my main desktop that I paid $320 for. 100 TB just isn't that much space today.
>I just bought an 18 TB SATA drive for my main desktop
I think one of the big issues is the performance of the disk. At 200 MB/s (which is generous for a spinning disk) it would take more than 24 hours to read all the data from the disk.
With those kind of performance characteristics, I suspect that it makes redundancy and checking for integrity much more complicated.
On point 2, you can build a 100 TB "NAS" with an 8 drive chassis and 8 of the new 20 TB drives. Pricing should be under 5 or 6k USD unless you go crazy on the server. That should give enough extra to allow for a good bit of RAID parity and filesystem overhead etc.
I think it's less about storage quantity now, and more about the available bandwidth for access to that storage, if we assume 100 TB is reasonable for a whole university. If we assume parallel reads from all 8 drives at, say, 200 MB/s (which is likely optimistic and assumes one single linear workload), this will only get you 12 Gbps.
Throw multiple users and a need to seek, and rapidly the cost seems to be to become random access, latency and bandwidth (SSD caching in a tiered architecture etc).
This might help Google justify higher pricing for storage than the "raw" storage cost, if the market will bear it. Alternatively, maybe the solution is for Google to badge up a non Google Drive storage service in the education workspace suite for research data, based on more appropriate cloud storage offerings with glacier type tiers etc? Offer universities unlimited storage and people will find incredible things to store there you never dreamed of....
I wonder what's going to happen if or when YouTube starts to implement restrictions on how many videos you can upload. People are being conditioned year after year that they can go back to YouTube and expect to be able to upload videos at any time. There will probably be some level of strife if that were to suddenly change.
That would also bring into question if "significant" content creators would have more leverage to upload more videos than the average user. Then all the people that expect to be able to upload unlimited videos would either move off to a competitor, condition themselves to think otherwise, or something else.
Google would need more than one raw GB per GB, because there is some risk the drive would fail. I don't know what the answer would be, my NAS is a two bay solution, so I waste 50% of the capacity, a 4 bay one might only waste 25%. And that is without accounting for disk replacement cost, operating and electrictal costs and so on.
Back a few years ago when I was in grad school (University of Texas) they went all in telling everyone to use Google Drive for everything because we were getting unlimited storage for life. Last week they just sent out an email saying that as of later this year our quotas are going to be 1GB.
Yeah. I guess when you promise a perpetual account to all students for life, that adds up quick. New policy is 5GB for active students faculty and staff, and 1GB for alumni, retirees, and former students.
> As of November 2021, UTmail is utilizing over 6.4 PB of pooled storage. However, due to a change in the G Suite storage policy, Google will implement a 545 TB pooled storage quota.
> Of the customers impacted: 70,000 are students, faculty, and staff, and 200,000 are alumni and retirees.
Edit: just read the help files closer and turns out that alumni and retirees lose Google Drive entirely -- 1GB is just for email.
O.o I guess I never really think about the scale these systems operate at.
Actually, considering that, the fact they give you 15GB on a free consumer account must add up to at least many hundreds of PBs?! I guess storage is cheap and/or targeted ads make a lot of money!
This is happening to all universities that used Google to provide email and drive space for students and faculty. For the University of Texas alone, there are approximately 270,000 affected accounts. I personally have 895 GB in my Drive. Will you seed it for me?
As someone who used to work for a bioinformatics lab, I'm confused by this tweet.
The storage now being limited is for Gsuite- Classroom, Meet, Gmail, Calendar, Drive, Docs, Sheets, Slides, etc.
These aren't the products you'd store a lab's datasets in. They simply aren't designed for that and won't provide acceptable APIs, performance, etc. Thus although this change may stink for university IT, I don't see an alternate world where this lab had embraced Gsuite storage and was crippled by it.
Given that people have tried stuffing hundreds of TB of data in every cloud service ever that offered "unlimited" storage, I would not be surprised if someone actually did that.
Under funded university labs often have need for archival storage to "keep results for future".
You name the technology, I've seen it be used to store results. If a system is centrally funded by the main university (and not the lab itself), researchers will find a way to get it to hold research data if it has an interface to ingest a file...
They don't need acceptable APIs or performance - they need something as opposed to nothing, and they'll find a way to make it work! More to the point, they need basic IT infrastructure funded by something central in the university. If there was a centrally funded way to use s3 with glacier for archival, (so they don't need to spend lab money on it forever more), many would use it.
In my experience, often the issue is a lack of local budget, meaning infinite ingenuity to find a way to use what is available to get the best outcome for their research. Any ability to upload and store a file for later retrieval can, with enough low-cost RA time, be made into a apparently infinite free storage system!
I’ve backed up my master thesis and all relevant data to google drive.
I’m I’ll move a copy somewhere else if I ever need to conduct new research on it, but I’m sure if I had gone on to take a phd each paper, published or not would end up in a folder on gdrive if google had promised me the space.
Like everything else in the software world, I am sure they always had a "we can change or cancel anything anytime we want for whatever reason or no reason at all" clause in their written contracts.
Previously worked at Google in infrastructure but opinions are my own.
I am always frustrated/embarrassed at Google’s messaging of these things, but it seems to me like this tweet is trying to stir the pot for attention. The Pixel phone with unlimited uploads marketed at consumers is one thing, but this unlimited plan for universities assuredly contained terms that didn’t guarantee it forever and a large institution should be aware of that. I also don’t think unlimited is a synonym for eternally unlimited.
It also seems like it would be a bad idea to store frequently accessed experimental data in Google Drive and based on follow up tweets the professor agrees and doesn’t use Google Drive for this… so it seems like they are mad about a hypothetical.
I don’t think Google is always the most thoughtful with things like this but this seems like something to take up with their IT department and not a second Google Reader incident.
Indeed - given much of Google drive marketing seems to be trying to encourage it's use for routine frequently accessed data, this is an interesting question.
Much of the marketing for drive seems to still be (in sentiment at least) about dumping it all into Google drive and never having to worry about storage again. I assume they mean storing huge files in Google drive, but actually that's probably a significant use case for many users of cloud storage.
I think the biggest issue is provenance and access control/logging—Drive doesn’t have a lot of great tools for that. Backups as well.
That said, I’m struggling to imagine how you could even efficiently process on the order of 1 PiB of data from Drive. Having that in a data warehouse or some object storage close to your compute seems required to make any good use of it.
Logging in Drive is pretty robust (backups are not). There's a few tools that can move that level of data, but google's now-first-party migrate product could likely handle it.
Just coming out of a higher education position I held for about 10 years...we were really only using GSuite for Education on the periphery of everything else we were doing (which was all Microsoft-based). So we were really only setting up the needed GSuite integrations to give some access for students/faculty/staff to use the GSuite apps if they wished, so the "unlimited" storage side of things was an extra bonus.
While our actual students/staff have not abused the unlimited-ness at all, there's been a big trend in recent years for fake spam/bot type accounts to submit online applications (particularly for the inexpensive community college level here in California) which would allow for those fake people to get issued accounts within the college systems (typically giving them a student email account).
In essence though, one bad account could in turn create multiple shared drives it seems, and then from there grant access to other external Gmail accounts and swap the ownership to them, while still leaving the unlimited storage capabilities in place (at least that's been my perception of what I had seen in our instance). Before I left recently, I had seen pages upon pages of shared drives in our GSuite Admin area of Shared Drives that were terabytes large (one as high as 100 TB or some ridiculously high number like that)...with very little mitigation options being provided by Google to help avoid that from occurring in the first place (aside from taking the extreme measure of locking things down after the fact...but cleaning up that existing mess seemed to not have a solution...supposedly there were going to be some new tools released early this year to help out, but not sure if those are available yet).
Suffice to say...some bad apples are making it hard for all of the good people to take advantage of what was otherwise a wonderful offering for educational institutions (having administered many systems over the years...it's always nicer to not have to worry about license limits and other baloney like that since it simplifies things for what is usually an already overloaded IT staff).
I could clearly see all of the bad accounts abusing our instance...if only there was an easy way to filter all accounts > 2 TB and then delete them I could have reduced our storage usage down considerably for Google.
That multiplied by hundreds of institutions where this may have occurred during the pandemic and I can see how storage usage could have went up dramatically on Google's end...but I think that Admins in those institutions would be more than happy to clean things up if they had the tools/ability to do so.
Just a note for those of you who may be in a similar situation. YMMV.
If you have a couple of very large folders (even 1 TB or more) you care about holding onto from your unlimited drive, share it with a few (even throwaway) personal Gmail accounts. Once my HS disabled access to my account after I graduated, the school admin account somehow inherited these folders (the organization name is the new owner).
I still have access to these many TB large folders many years on in the “Shared with me” section of gdrive.
Check out my comment above here in this same thread which basically confirms exactly what you just shared here...this is the same tactic I was observing being used with fake student accounts sharing with outside external Gmail addresses which then appeared to take over the Shared Drives being created in GSuite, allowing for TBs and TBs to be added in to the drives by outside users. It seems like a big problem to me...but at the moment I don't think Google is providing good enough tools for Admins to see/eliminate these situations very quickly/easily.
A smart IT administrator would put that down on paper, but not "unlimited space", just a large amount of space, per user, committed on paper.
It's like decision makers don't understand business and human psychology.
Firstly, when you tell someone they have "unlimited" space, it will be filled up with "unlimited" data. Its basically a law.
Secondly, since the dawn of the concept of transactions between people and businesses, everyone knows there is "no free lunch". I just don't get why enterprise environments allow "unlimited" and "perpetual license" and other weasel terms into their contracts.
Understand that no business will give anything away for free. Be extra careful when a business says so.
I think Google is realizing that software companies like Blackboard and Peoplesoft are milking universities for millions of dollars while Google is losing potential revenue in the name of product exposure.
Since universities need to provide email and storage services to their students anyway, Google is realizing that they will happily pay the bill rather than switching providers.
What is a good way to sort and deduplicate about 700 GB of photos stored in Google Drive (not in Google Photos but as image files in Drive)? Can it be done incrementally instead of downloading the entire thing?
What are the best options for deduplication and organization if I have to do it at once?
Unfortunately, Google doesn't support good API read access of your files: pulling them down via their API results in several bits of metadata being stripped (including GPS coordinates). (If someone knows otherwise, I'd love to be wrong!)
I tried downloading about 200GB of data off of my university drive last night. Took a good hour or two of "zipping" on the drive site, then... hundreds of separate zips ended up dumped on my browser at some point later. And those downloads mostly failed. So now I'm taking a look at rclone to move data off of my drive, maybe slowly, through my raspberry pi.
113 comments
[ 3.0 ms ] story [ 176 ms ] threadNot just G Suite. Google Photos + Pixel unlimited storage[0], along with Google Photos free storage[1].
[0] https://9to5google.com/2022/01/31/pixel-3-google-photos-back...
[1] https://support.google.com/photos/answer/10100180?hl=en
Why is Google cutting back so much?
I thought storage was cheap. How much have they overextended themselves?
[0] https://www.sherlock.stanford.edu/docs/
https://www.starfishstorage.com
The cost of storage at Google's scale is low. GPUs/TPUs are another thing entirely.
2020: $41b operating income. 2021: $78b operating income (+$37b in one year).
2018: $31b operating income. 2019: $36b operating income.
So they just added an entire 2018 or 2019 size profit to their outcome in the past year.
That didn't come from squeezing pennies on little storage products and the like. That's from one of the world's largest snowballs continuing to roll down hill, piling up tens of billions in trivially easy profit addition as they go. When it comes to expanding margins, for an organization the size of Google that is still expanding relatively quickly in sales, it's far more important to hold down cost expansion as the profit soars (don't add those entirely unnecessary 20,000 heads) than it is to cut rounding error costs. Microsoft has similarly been masterful about doing that lately (unlike during the Ballmer era).
Their deal with Apple ensures their staying power. They own the other half of the Mobile ecosystem so even when lockdowns end and travel opens up there may be even more usage.
https://abc.xyz/investor/static/pdf/2021Q4_alphabet_earnings...
But when other companies do it, they tell you that it’s not forever. I suspect other companies want you to still be a customer in 5 years.
End storage for new users, don’t just go telling scientists you’re going to delete their stuff.
Google should watch out, if people no longer get any perceived utility from their services, they will find themselves destitute. A true ad company with a search engine and no innovation as Peter Thiel describes them.
I would rather pay Apple than Google.
https://www.macrumors.com/2021/06/29/icloud-data-stored-on-g...
I can't imagine their cloud division ever winning with Amazon and Azure exactly because there are happy GSuite/Youtube/etc. customers like me around, making sure no one ever trusts them for anything again.
[0] https://blog.google/outreach-initiatives/education/google-wo..., the "new storage policy" section.
Buy a new phone and get unlimited data*
* Speeds may be throttled after a certain about of data usage
Technically still unlimited.
If there is a cap of 1 byte per hour, there is absolutely a limit of 720 bytes per 30 day month. Whenever there is a cap, it is not "technically unlimited".
And that's why there's typically a 2nd asterisk about speeds dependent on network conditions. Still legally advertised as unlimited, thanks to the fine print.
https://www.theverge.com/2021/12/21/22848536/mint-mobile-pre...
Deprioritized data is still allowed to be advertised as "unlimited" if it is not throttled to 2G speeds.
> We will contact impacted institutions directly in the coming weeks to discuss a range of options for getting the storage they need
Given the quota model, it doesn't seem like notifying individual users would have made sense. It doesn't sound like the original poster would have been a domain admin.
> Large institutions will be provided supplemental storage later this year and all schools can gain additional storage through Education Plus and the Teaching and Learning Upgrade.
Absolutely with all the small players.
But even the big players as we see here.
Of course that seemed wrong because if you are uploading for years $5 a month isn't going to keep covering all of that storage. I imagine that part of this assumption that storage costs would keep dropping and presumably this isn't happening as quickly as originally predicted.
It seems that in the context of research universities there may be a mismatch in communication and use of "unlimited". Google said something it didn't mean, then eventually realises and claws it back.
This has happened twice in a few weeks - once for "forever" on free G Suite, and now with "unlimited" for universities.
I'm surprised that they don't get more marketing collateral for selling the credibility of G Suite to enterprise clients from being able to talk about case studies like storing and processing multi dimensional time-series medical imaging data on their storage platform as a way to say "we're ready for your data, and you can feel good while you pay us, as we support this great research"...
Clearly the revenue push has arrived for these services. 100 TB per institution isn't really a lot when you consider research data, but might be more tenable if you consider purely students with the odd slideshow and video. Unfortunately this makes life easier for the less scientifically rigorous fields of study that make a few slide decks, rather than the ones where a student might collect a few TB of data in an hour or two.
Zero.
Not even the owners watching them to make sure they uploaded correctly.
I think you are limited to 12 hours / 250 GB per clip?
The bandwidth / storage demands of video are not insignificant.
Someday someone will do something - ie, insure you are paying for some kind of service.
1) There is no such thing as "unlimited" anything with technology. Either you haven't read the fine print of your provider's terms or your provider will at some point have to change them.
2) What's interesting is that storage costs are dropping precipitously while they're restricting their customer's use of storage space. I just bought an 18 TB SATA drive for my main desktop that I paid $320 for. 100 TB just isn't that much space today.
I think one of the big issues is the performance of the disk. At 200 MB/s (which is generous for a spinning disk) it would take more than 24 hours to read all the data from the disk.
With those kind of performance characteristics, I suspect that it makes redundancy and checking for integrity much more complicated.
I think it's less about storage quantity now, and more about the available bandwidth for access to that storage, if we assume 100 TB is reasonable for a whole university. If we assume parallel reads from all 8 drives at, say, 200 MB/s (which is likely optimistic and assumes one single linear workload), this will only get you 12 Gbps.
Throw multiple users and a need to seek, and rapidly the cost seems to be to become random access, latency and bandwidth (SSD caching in a tiered architecture etc).
This might help Google justify higher pricing for storage than the "raw" storage cost, if the market will bear it. Alternatively, maybe the solution is for Google to badge up a non Google Drive storage service in the education workspace suite for research data, based on more appropriate cloud storage offerings with glacier type tiers etc? Offer universities unlimited storage and people will find incredible things to store there you never dreamed of....
That would also bring into question if "significant" content creators would have more leverage to upload more videos than the average user. Then all the people that expect to be able to upload unlimited videos would either move off to a competitor, condition themselves to think otherwise, or something else.
Back a few years ago when I was in grad school (University of Texas) they went all in telling everyone to use Google Drive for everything because we were getting unlimited storage for life. Last week they just sent out an email saying that as of later this year our quotas are going to be 1GB.
RIP huge swaths of research data I guess.
> As of November 2021, UTmail is utilizing over 6.4 PB of pooled storage. However, due to a change in the G Suite storage policy, Google will implement a 545 TB pooled storage quota.
> Of the customers impacted: 70,000 are students, faculty, and staff, and 200,000 are alumni and retirees.
Edit: just read the help files closer and turns out that alumni and retirees lose Google Drive entirely -- 1GB is just for email.
O.o I guess I never really think about the scale these systems operate at.
Actually, considering that, the fact they give you 15GB on a free consumer account must add up to at least many hundreds of PBs?! I guess storage is cheap and/or targeted ads make a lot of money!
This is happening to all universities that used Google to provide email and drive space for students and faculty. For the University of Texas alone, there are approximately 270,000 affected accounts. I personally have 895 GB in my Drive. Will you seed it for me?
The storage now being limited is for Gsuite- Classroom, Meet, Gmail, Calendar, Drive, Docs, Sheets, Slides, etc.
These aren't the products you'd store a lab's datasets in. They simply aren't designed for that and won't provide acceptable APIs, performance, etc. Thus although this change may stink for university IT, I don't see an alternate world where this lab had embraced Gsuite storage and was crippled by it.
You name the technology, I've seen it be used to store results. If a system is centrally funded by the main university (and not the lab itself), researchers will find a way to get it to hold research data if it has an interface to ingest a file...
They don't need acceptable APIs or performance - they need something as opposed to nothing, and they'll find a way to make it work! More to the point, they need basic IT infrastructure funded by something central in the university. If there was a centrally funded way to use s3 with glacier for archival, (so they don't need to spend lab money on it forever more), many would use it.
In my experience, often the issue is a lack of local budget, meaning infinite ingenuity to find a way to use what is available to get the best outcome for their research. Any ability to upload and store a file for later retrieval can, with enough low-cost RA time, be made into a apparently infinite free storage system!
I’m I’ll move a copy somewhere else if I ever need to conduct new research on it, but I’m sure if I had gone on to take a phd each paper, published or not would end up in a folder on gdrive if google had promised me the space.
Like a written contract or this University just assumed there was unlimited Google storage?
I am always frustrated/embarrassed at Google’s messaging of these things, but it seems to me like this tweet is trying to stir the pot for attention. The Pixel phone with unlimited uploads marketed at consumers is one thing, but this unlimited plan for universities assuredly contained terms that didn’t guarantee it forever and a large institution should be aware of that. I also don’t think unlimited is a synonym for eternally unlimited.
It also seems like it would be a bad idea to store frequently accessed experimental data in Google Drive and based on follow up tweets the professor agrees and doesn’t use Google Drive for this… so it seems like they are mad about a hypothetical.
I don’t think Google is always the most thoughtful with things like this but this seems like something to take up with their IT department and not a second Google Reader incident.
Much of the marketing for drive seems to still be (in sentiment at least) about dumping it all into Google drive and never having to worry about storage again. I assume they mean storing huge files in Google drive, but actually that's probably a significant use case for many users of cloud storage.
That said, I’m struggling to imagine how you could even efficiently process on the order of 1 PiB of data from Drive. Having that in a data warehouse or some object storage close to your compute seems required to make any good use of it.
With regards to data locality, I was more thinking about running a map reduce or similar ETL pipeline.
While our actual students/staff have not abused the unlimited-ness at all, there's been a big trend in recent years for fake spam/bot type accounts to submit online applications (particularly for the inexpensive community college level here in California) which would allow for those fake people to get issued accounts within the college systems (typically giving them a student email account).
In essence though, one bad account could in turn create multiple shared drives it seems, and then from there grant access to other external Gmail accounts and swap the ownership to them, while still leaving the unlimited storage capabilities in place (at least that's been my perception of what I had seen in our instance). Before I left recently, I had seen pages upon pages of shared drives in our GSuite Admin area of Shared Drives that were terabytes large (one as high as 100 TB or some ridiculously high number like that)...with very little mitigation options being provided by Google to help avoid that from occurring in the first place (aside from taking the extreme measure of locking things down after the fact...but cleaning up that existing mess seemed to not have a solution...supposedly there were going to be some new tools released early this year to help out, but not sure if those are available yet).
Suffice to say...some bad apples are making it hard for all of the good people to take advantage of what was otherwise a wonderful offering for educational institutions (having administered many systems over the years...it's always nicer to not have to worry about license limits and other baloney like that since it simplifies things for what is usually an already overloaded IT staff).
I could clearly see all of the bad accounts abusing our instance...if only there was an easy way to filter all accounts > 2 TB and then delete them I could have reduced our storage usage down considerably for Google.
That multiplied by hundreds of institutions where this may have occurred during the pandemic and I can see how storage usage could have went up dramatically on Google's end...but I think that Admins in those institutions would be more than happy to clean things up if they had the tools/ability to do so.
Interested to hear on which products Google makes any promises regarding lifetime (ie. guaranteeing at least 10 years availability for example).
If you have a couple of very large folders (even 1 TB or more) you care about holding onto from your unlimited drive, share it with a few (even throwaway) personal Gmail accounts. Once my HS disabled access to my account after I graduated, the school admin account somehow inherited these folders (the organization name is the new owner).
I still have access to these many TB large folders many years on in the “Shared with me” section of gdrive.
It's like decision makers don't understand business and human psychology.
Firstly, when you tell someone they have "unlimited" space, it will be filled up with "unlimited" data. Its basically a law.
Secondly, since the dawn of the concept of transactions between people and businesses, everyone knows there is "no free lunch". I just don't get why enterprise environments allow "unlimited" and "perpetual license" and other weasel terms into their contracts.
Understand that no business will give anything away for free. Be extra careful when a business says so.
Since universities need to provide email and storage services to their students anyway, Google is realizing that they will happily pay the bill rather than switching providers.
Is Google in trouble?
What are the best options for deduplication and organization if I have to do it at once?
I'd download a Google Takeout.
If you need something to deduplicate your takeout, feel free to try PhotoStructure (I'm the author). https://photostructure.com/faq/takeout/
(You'll be left with a single datestamped-based folder hierarchy: just enable "automatic organization". The folder format is customizable, too: https://forum.photostructure.com/t/how-to-change-the-naming-... )