And it would have been pretty simple to run a query for users with over 4 million objects, and fire off an email to them all saying "We are about to implement a limit, and you will be impacted. Here are potential workarounds/mitigations".
I suspect though that the simple process of dropping a few hundred customers who will be directly impacted an email requires approval from 27 middle managers, and it's easiest to just ignore them.
More often, this kind of error out of Google comes from the left hand not knowing what the right hand is doing.
Joe or Jane Noogler has just been brought into the team, and their starting project was to improve indexing. They succeeded in doing so by attaching the Drive data to an indexing service created after Drive existed. It's working great and has a 10-20X speedup. But oops... That new indexing system has a 5-million-element hard limit built in, and nobody caught it until they went to production. J. Noogler is too new to have realized this could have been an issue so they never started the escalation / customer messaging processs.
Precisely what I felt skimming through the comments on the issue.
A lot of high scale businesses have taken a huge risk. A company like Google had over 10y long tenure engineers who were swept by the layoffs. The argument that the year prior was a hiring spree only makes it worse: now you have hands to throw at the problem but unlikely the right pairs, which, given the climate of job loss fear, aren't likely to admit they are incompetent.
Off topic, but this isn't an isolated case of mega size top tier companies seemingly dropping the ball more far often and/or with greater blast radius impacts. See github leaking secrets publicly on their own git repo.
Call me paranoid, but to me one of these things is going on:
1/ tipping point of too many under qualified for the job engineers running things, due to C-suites looking at keyboard monkeys the way they see factory workers, naively applying "cost cutting" measures, plucking through spreadsheets who "are the biggest seemingly disposable weights" and fire those as part of waves of x% layoffs
2/ Overworked remaining know-how crews - added the weight of dealing with inspiring politicians turned enginners for the juicy 200k + rsu package who, demand better articulated instructions, knowledge transfers and more comprehensive formal documentation materials, so that they too can shine too - on top. Competent folks having less and less time but more more tears and blue bags below their eyes finally realising that ain't worth it as it keeps getting worse anyway.
3/ sabotage
Or all those, since there could be some smowball effects there.
Call me paranoid, that it isn't what's going on, we clearly aren't seeing the slow fall of major infrastructures the few who were there to build them now having mostly packed their bag and long gone, replaced by swaths of bootcamp trophy coders hoarding cloud certificates like North korean generals like to collect shiny pins.
Maybe it isn't, the future is still in the making anyway, but I felt glimpses of that several years ago, then about each passing year, now every few months. like an seeing accelerating meteroite seemingly going away but which keeps looking bigger and bigger.
More so related to the topic: just drop drive as a centralized storage, there are e2e solutions out there and open source scripts to entirely migrate out of these wall gardens. You won't miss google sheets: you can still use it!
5M items sounds suspiciously like one machine has to keep in RAM the complete list of a users files for certain operations. User accounts with large numbers of objects were probably causing those machines to OOM. 5M items, at about 1000 bytes per item (name, metadata, a few uuids, etc) is 5Gbytes. 5Gbytes is about the amount of RAM most tasks will be given.
I could imagine that some services within Google haven't been carefully designed - for example, perhaps the quota service reads a complete file listing into RAM, adds up the sizes, and then writes the available quota.
This is probably the easy fix, rather than redesigning every service to be able to stream objects correctly.
It could also be a sharding/checkpointing issue. Imagine you want to scan every file in every user account, for example for blank documents (maybe because you want to downrank blank 'untitled documents' in search results, because they are easily accidentally made).
You write your scanner to divide up the list of user accounts into chunks, process all chunks in different worker machines, and combine the results. Simple mapreduce.
However, if there is one huge user account, then you either have to wait for the entire process to take far longer, or you need to have multiple workers working on the single account (adding a lot of complexity to every operation you wish to run across all accounts).
Sounds like they want to protect against resource exhaustion. Touch a file, it's empty with a size of zero bytes. But depending on the block size of the storage device, it's likely to be between 4k and up to even 1M.
They should incorporate a minimum size for every file when calculating how much quota you have.
Because they have no retrieval fees for gsuite, I bet it is expensive dealing with drives with tons of small files.
Agree they are rolling this out terribly. They should have granted everyone 2x the number of files they currently have in their drive quota (or 4M, whichever was smaller).
I see folks describe Google Drive errors / limits like this and people describe "a business critical operational system" and I wonder. Is Google Drive even supposed to do this thing?
Not pointing the finger back at them, if Google doesn't make it clear they probably should, but at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service.
> at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service.
That may be how you view the product, but Google Drive absolutely is used as an enterprise cloud service by many large companies. And as a result, there are lots of applications that integrate with its API, etc., to serve those enterprise use cases.
At Google’s scale for consumer users, on their backend it has to be an enterprise scale product anyway or the whole thing would have imploded shortly after launch.
Sometimes there are random UX/UI/API limitations, but those usually get fixed relatively quickly when someone hits them, because they’re typically just some hardcoded limit someone put in a front end layer because it was ‘more than enough’ and they needed something. A bit of elbow grease to clean up that path if it seems useful, and it’s usually good again.
It's funny to read your comment, because to me, Google Drive feels like an enterprise product that incidentally happens to provide a consumer-facing version as well, not the other way around.
The assumption stems from the facts that a/ it started as personal use, like gmail. Only way later workspace and that the gsuite name even came out. Certainly no enterprise offering for years after personal use was broadly well adopted. b/ contrary to most SaaS businesses, Google is rather subtle and gentle when it comes to promoting or upselling, many people don't even know a paid/pro version of this thing exists.
But you are right, google offers enterprise grade solutions so is expected to support over a few million files per account or even per user. Maybe something changed due to hasty cost cutting measures given the sudden interest by many on the issue tracker.
> The assumption stems from the facts that a/ it started as personal use, like gmail. Only way later workspace and that the gsuite name even came out. Certainly no enterprise offering for years after personal use was broadly well adopted.
Yes - I mean, sort of. It depends on what you define as the starting point for Google Drive. Like many Google products, the brand has been reorganized and reused a number of times.
If you look at Google Drive as an extension of Google Docs, which existed prior to 2012, then yes, it started off as personal use. But if you look at it as a thing that launched in 2012 (the introduction of the Google Drive brand), then it arguably belongs as part of the reorganization that involved ending G Suite and turning it into Google Apps for Work[0], which is now known as Google Workspace.
In fact, now that the consumer version of Google Drive has been reorganized into Google One, you could argue that the enterprise version of Google Drive is actually older than the consumer version!
Of course, all of these distinctions are somewhat arbitrary, because it's the same product that you've used all along, and the only thing that changes is the pricing and the marketing. But if we're talking about consumer vs. enterprise versions of a commodity product that essentially operates at enterprise-scale for consumers, that's also the only real difference.
It's actually quite funny that the same product (which has been improved slightly, but is largely identical to what it was over a decade ago) has had so many different brands attached to it, especially when you consider that some of those brands are essentially container formats that repackage the same underlying brand (e.g., Google Workspace contains Google Drive).
[0] There was actually a rebrand in between, from 2012-2015, but because Google's product names are Alphabet soup, I can't be sure of what they called it in the interim, though I'm positive it was a distinct product from before 2012.
You don't have anything in your HN profile, so I can't be certain, but I'm guessing you haven't been in the Research/Education space? It's scary how much Google Drive is used for storage of stuff.
+1
And, it's not like Google didn't make huge marketing pushes to enter that space and businesses at large. Gsuite, Classroom, ever increasing storage limits. There is even an education package offering "unlimited" storage.
I work at a university that is fully invested in Google Apps.
We did have unlimited storage, however last week they announced that's soon to change:
> A quota will soon be introduced on the amount of data that staff and students at the University can store in their Google account. This is part of a larger piece of work being carried out by the Google Workspace team in IT Services, to review the overall amount of storage being used by the University.
google drive has unlimited storage, they opened that pandora's box themselves. yes it's janky as hell, but I'm looking at tens of thousands per year to backup to cloud services for my lab's data
You can't download anything without 3PC. It's the only Google service set up that way, which isn't a coincidence. Drive is essentially a loss leader to protect their cash cow.
What's to do with cookies? I understand the comment to refer to ads being by far their main revenue making business, and shaping users browswer preference to keep cookies helps Google make more money via ads.
I didn't even know 3PC was used by google drive. Not sure what could be non cpincidental otuer than a technical reason for it given the very collaborative nature of this tool.
So that's the reason they're required: to preserve the credentials story between the static resource provider and the non-static layer. Nothing nefarious; just an implementation detail.
That's how all dark patterns are constructed. Present a plausible cover story. Keep the underlings in the dark about the true reason for the system architecture. Don't write anything down.
That's also how all patterns are constructed: present a plausible cover story (because it is true). Keep the underlings in the dark about the true reason for the system architecture (because the delta between the true reason and apparent reason is null). Don't write anything down (because there's nothing else to write down).
Suspicion of dark pattern is not proof of dark pattern.
(ETA: at Google in particular, the dark-pattern-conspiracy you're describing between two unrelated teams, in this case Drive and static storage, is hard to pull off because coordinating those teams requires documenting design, and it turns out a lot of folks working at Google are Hacker News-type personalities who will ask inconvenient questions if they come up. But what they have that we lack is observation of the larger security picture and the ways failure to, for example, 3PC-guard your static files can result in failure modes).
It's "free" cloud hosting. You'd be surprised how often people mix their personal google account with work stuff. It annoys me greatly, but that's how it is.
There is general ignorance of what google does/is/isn't and how online storage works. I agree with the sentiment, but, I've seen many a thing from recipes, shopping lists to FPA/Proposals all hosted on google drive. Even people sell content that is hosted on google drive.
Disclaimer: I'm a Googler but I don't work anywhere near this stuff, and this is just my own personal opinion.
In my mind, Drive isn't a file sharing / storage system, but rather a document sharing / storage system. If you want to store large numbers of files, you should use Cloud Storage, which can handle that. Drive is the cloud equivalent of LAN shared folders - you know those janky NFS and SMB shares that were always a giant headache because of permissions and other nonsense? That's what Drive is trying to be.
Seems like they could do it without the jankiness, though.
But the average user is not writing automated systems to populate google drive with millions of files. Once you are doing something that unusual you should understand that nuance. Sure it is fun to hack a tool designed for one purpose to serve another, but if you are writing code for use in production you have to understand these types of design differences.
Drive is very closely linked with Docs, Sheets, Slides. These aren't "files" in the traditional sense. These, plus things like PDFs, videos, etc are what Drive is designed to help manage.
If you have 100,000 source code files, art assets or something like that, you probably don't want to use Drive to manage them. The UI just isn't built for it and it doesn't scale.
I’ve never had a problem with NFS, and modern filers would laugh at 3 million files.
I really dislike Google Drive. I can’t imagine people would use it at all if not for the fact that it’s bundled, monopoly style, with the office suite and gmail.
Since Google won’t ever fix it, I’m hoping they rapidly make it so crappy some other competitor crops up.
Of course, with our luck, it’ll be some Microsoft offering.
I work it in a fly neuroscience lab and we use it to store all our electrophysiology and video data. Each person in the lab is storing on average 5TB of data, and the lab as a whole stores 100TB.
The graphical user interface combined with unlimited storage for Google Workspaces is essentially an unbeatable deal. Researchers can upload their data easily through the interface. Any custom solution based on S3 or equivalent would take some time to teach and more time to maintain. Also, we're paying about $200 / month total to store 100TB of data in the cloud, which is hard to beat with other services.
I tried setting up a single account for the whole lab once, but we ran into the above 5M file limit, so we just have individual accounts per researcher and it's mostly fine for now.
> Also, we're paying about $200 / month total to store 100TB of data in the cloud, which is hard to beat with other services.
You should expect this to go away soon. I support science researchers and our unlimited storage option is going away in the coming months. Options for purchasing space are limited, and not cheap.
How expensive are we talking? Looking at the workspace pricing page the enterprise plan gets you a pooled 5TB per user with the option to add more by talking to them, does that mean you have to pay more for the extra storage?
This is with a regular Google Workspace plan, independent of the university-wide research plan. The research plan unlimited storage is indeed going away for us as well ( https://itconnect.uw.edu/tools-services-support/software-com... ), which is why we migrated to our own private Google Workspace.
I haven't heard about the google workspace enterprise unlimited storage going away anytime soon, although perhaps you know something I don't?
For personal use, Google Drive creates a way for me to have automatic backups of files I wouldn't want to lose in case of drive failure. It's also a way to easily share files between my desktop and my phone.
For professional use, I've usually seen it used as a document repository that can be shared by teams.
In both use cases, 5 million files seems like it'd be a hard limit to hit.
Google Drive has the "MS Excel" problem. It's made cloud storage really accessible to people, so they turn to it for everything, even stuff that's more appropriately put somewhere else. In Excel's case, it's database data. In Google Drive, it's files. My company uses it for video assets, because the media folks are really familiar with it, but we're moving it to S3 now because it's not got the chops for our seriously "larger" needs.
If you're using Google Earth Engine or a similar tool, it's generally highly recommended to use Google Drive as the storage backend for it - especially since Google sells Enterprise plans with "as much storage as you need."
Geospatial artifacts can be very large, and also can be spread across thousands and thousands of files (20+ zoom levels on Earth, with enough 256x256 tiles to cover the planet at each zoom level, for example).
Many use some rsync-like tools to use gdrive as a flat backup system. Perfectly fine use case. You would be over a million as a single user. Some account have thousands of users.
See some comments on the tracker, some orgs have a multi sites set up across an entire nation. I wouldn't be surprised if 10k+ volunteers at some NGOs are on that sort of plan and use it as their personal storage solution on top of all the things they sync for document counts heavy work.
Where do you get the idea that Google Drive is a "primarily-personal file storage/transfer system"?
That may be what you use it for, but that doesn't mean other people use it in that way. And it doesn't mean that's how Google represents it in their documents.
I tried to look for documentation about what Google Drive should and shouldn't be used for.
In the FAQ Google Drive said it could be used for storing any file type. There are instructions for how to backup/sync any folder on your computer to Google Drive. If you're an organization that has several people backing up folders with large numbers of files, there's nothing Google says that makes you think you shouldnt use it that way.
The only thing that indicated what it shouldn't be used for was content policy page. It basically says it shouldn't be used for illegal activity and that it shouldn't be used for streaming video.
All this comment proves is that it's not not available to companies, not that it's built with corporations in mind. There's a MASSIVE difference in software for regular home customers and for business customers.
You can easily reach this number if you are doing something like storing json/xml user data from some service. But I just can't imagine you could ever hit this limit as a normal user. And if you do, just zip some old stuff in to one file and carry on.
From reading the thread it sounds like this is 5M per user.
Google Drive appears to be targeted at "human" usage, i.e. people uploading or creating files. I would guess that this is also worked into the cost – that the assumptions of the amount of work a human can do are a part of the price formulation. The reporter of this bug seems to be using this as a storage backend for software though, which I don't believe is the intended use-case.
Looking at S3 pricing, just the storage is ~5x the cost of Google Drive, and then you need to add transfer and API calls on top of that.
I don't personally think that there are reasonable use-cases for human users with 5 million files. There may be some specialist software that produces data sets that a human might want to back up to Google Drive, but that software is unlikely to run happily on drive streamed files so even those would be unlikely to be stored directly on Drive.
(Disclaimer, I work at Google, not on Drive, this is my personal reading and interpretation of the public info, I don't have any inside info here)
I did some quick math on this. The top plan Google sells is 30TB, to reach 5M files in 30TB, you'd have to fill the entire thing with 0.75MB files. So yeah, if you are using Google drive as some kind of JSON blob storage, it isn't designed for your use case.
For 30TB I think it would be 6MB, but regardless, that appears to be per-user. If you're an individual needing 5m files, maybe you need a specialist storage solution, and if you're a business then that's one hell of a bus factor.
Makes me wonder if the limit was actually as much for finding run away internal code. I’m sure there’s cases deploying something like Google Drive where code went rampant making duplicates or something.
My guess is someone wrote some stupid "Google drive file name FUSE mount" script that attempted to exploit unlimited file storage in metadata. So some team working on Drive ran a query and saw 5M files is enough for basically anyone.
This reminded me of early days running AWS SQS. We didn't charge for storage as message size was limited and the expectation was any message was going to be read immediately... But sure enough some users would just leave messages there forever and we could never figure out what they were trying to do except maybe exploit it for free storage somehow.
I recently moved on to using Google Drive as a backup drive for my computer, the tool I'm using for it, Restic[1], does file deduplication at file block level. Restic allow multiple hosts to upload to the same backend/repository. So far I already have 25k files, even though I started backing up a only few weeks ago.
I imagine that if you setup multiple servers/personal computers backups that could scale significantly.
This is the way to go. Files will count towards the Service Account, not your account. If you use Google Workspace I strongly recommend using a Shared Drive (instead of sharing a folder in “My Drive”) so the files will be owned by your org, not by the Service Account (otherwise deleting the SA would result in the files being deleted as well, because in “My Drive” the creator of a file keeps ownership even when creating in a shared folder). If you have problems with the 400,000 files limit in a Shared Drive: create a new one, rclone has a “union” backend. It can be set to create new files in the new Shared Drive while still showing files from the other drive(s). Also Shared Drives have their own recent activity views so it doesn’t clutter your “my drive” view.
Create a SA in GCP and generate a JSON credentials file. Create a folder or Shared Drive via web and share it with the SA.
I’m using this setup for years with zero problems.
Your analysis of why this limit exists is probably correct.
However, when you're paying for a product, the limits should be disclaimed. If I'm investing in Google Drive, I should be able to easily see the limits which apply to the product I've bought; that means total storage caps, file count caps, bandwidth caps, and whatever else.
If they listed every configuration to this level of granularity, the doc would be huge and no one affected would even see it. So the easiest way would still be to just pay for a month, try to use it as intended, and if you hit a limit, try another product.
While I agree with you that restrictions should be clear, this feels like a limit well beyond what anyone would encounter. Even abnormal usage with something like many big git repos with tons of little nested files inside the data directory isn’t going to hit 5M files unless you’re really committed to pushing until you find the invisible wall.
It would be pointless if your car came with a giant list of the melting point of every material used in the structure of your car so you know what the “real” temperature limits are. Oh boy, better not let the car get to 1450C or else the steel frame might melt!
I store way more than 5M files on consumer storage devices all the time. It truly isn't a ridiculous number, with 2TB it's not even hard to hit with a mix of text files and smaller media files.
If the storage vendor doesn't want people to use their product for the many use-cases where you end up with a bunch of small files, advertise the file count cap. Simple as that.
But the number of files you're allowed to store is a pretty basic feature. You know cars are made of materials which will break down at 1450C. You have absolutely no way to know whether Google's file limit is 100k, 1M, 5M, 100M, 1T, or what. How are you supposed to know if Google Drive fits your use-case when they don't document their limits?
I thought about this, but I accidentally put the Webkit repo (~1m files) into Dropbox once and it didn't go well. Git needs a consistent view of the filesystem across many files to work, and that's not really possible with that many files stored in a sync system that works per-file like Drive does. I put Git in the category of specialist software that shouldn't be operated in Drive. By all means tar and backup, that should be effective, but there's no need to store all the individual files.
Storing a git working tree in a cloud drive does seem like a bad idea, but the actual .git directory itself might be okay? I think most of the files in there are immutable or infrequently changed.
Minecraft saves are (were? Idk anymore) just big folders with tons of files representing each chunk of the world. These would be a common thing a kid might want to back up to drive and could potentially be zillions of files.
> "I don't personally think that there are reasonable use-cases for human users with 5 million files."
A 5 million file limit might be quite reasonable if you're paying for the basic, 100 GB storage tier. But Google Drive offers multiple tiers with up to 30 TB of storage. 30 million MB!
That means if your average file size is less than 6MB - very likely if you're storing JPEGs, audio files, text records, or whatever; you'll never be able to fill your Google Drive storage.
What's the average size of a file on a regular macOS or Windows HDD? It wouldn't surprise me if it's much less than 6MB. Shouldn't the file count limit scale with the storage limit?
Part of this seems like some of the wonky design that seems to be under the hood of Drive. Just looking around at how rclone has to do things, the difference between shared and individual drives, the fact I couldn't even originally find a way in the console to see my shared drive usage (even though I was the owner of it and it was on my account) it just seems like there is a very very wonky odd design to how the system works. Not like an object store under the hood with a layer of "document" logic applied, but rather what started as initially some API-esque thoughts on document storage that grew into a monstrosity. I wouldn't be shocked if there's just a ton of tech debt there.
While a 5 million file limit sounds reasonable at first glance, it isn't just the free tier that is affected. Google Drive also sells storage plans for up to 30 TB for an individual user, and there is no mention of such a limitation anywhere on any plan. If I bought and actually wanted to use that 30 TB (or heck even 5 TB), it isn't unreasonable to want to store a few million files. Regardless of the technical reasons behind it, not mentioning this restriction up front is before the sale is inexcusable.
I did the math and to reach 5M files in 30 TB, they all have to be averaging 6MB. If you actually have this many files, you should be using S3 or mongodb.
if you were using google drive to store images (e.g., you automated a camera system to take high res pictures), you can easily reach this limit.
Of course, it definitely is much nicer to have S3 over google drive for such a solution, but if you didn't want to write software (which you'd need for S3), and simply used the google drive's sync'ing features...
In this case you can just delete the oldest files, or tar/zip them in to one file. I doubt anyone needs instant granular access to 5 million images.
The chance of hitting this limit is so insanely slim and the workarounds are so trivial that I don't really see any issues here. I bet there would have been some abuse where someone tried to store data in file names and metadata which is not counted to your quota.
It seems like there is a much lower limit for "download all". There is a 100 or so item directory that I want to download, and "download all" tries to wrap them in a zip file and fails. I think it wants to pre-scan all the files for viruses before zipping, but is unwilling to do that many. So I have to download the files one at a time.
I've definitely downloaded folders with 100 items before. I think it might have to do with how big the individual files are because g drive won't create a zip that's larger than 2GB.
Hmm, thanks, that looks promising, but it appears to be an all singing all dancing complicated client that does two way transfers to gdrive and expects you to have a google account. I don't have a google account (don't want to enroll one), and am just trying to download a public folder that someone gave me a link to. I'll keep looking to see if I can find a way to use it.
We've been using GDrive in one form or another since 2014 when they released Google Drive for Work. Primary use case nowadays for my team is an old legacy tool(been in place since 2014) that connects to our ERP and other internal tools to generate reports and export data with combined metadata from multiple tools/sources. Then if needed, makes a copy to different share drives and teams, assigns access rights for internal and external users. We have custom web front ends that allow them to interact with these exports along with fat clients if they need to manipulate a larger dataset of files.
My last check had the service account that I'm responsible for somewhere around 8.7 Million files with an average size around 9.2MB.
We have multiple of these "accounts", with similar use-cases across other teams. This is excluding the normal use of Gdrive with normal "rank-and-file" office works managing standard office docs.
We've felt the squeeze from google over the past few years, so we've already started migrating off of google services about 2.5 years into a 4 year project.
Yeah this doesn't seem to be in line with the expected use or target market of Drive. It sounds like your tool is probably creating all the files acting as one service account, and therefore hitting the limit if it is indeed per user.
I think the rule of thumb I would use is that if you're creating the files in an automated way, and then accessing them outside of Drive interfaces, in all cases, then it's probably a better fit for a web app with cloud storage backend. Google Cloud Storage or S3 would work well for this.
138 comments
[ 3.2 ms ] story [ 197 ms ] threadI suspect though that the simple process of dropping a few hundred customers who will be directly impacted an email requires approval from 27 middle managers, and it's easiest to just ignore them.
Joe or Jane Noogler has just been brought into the team, and their starting project was to improve indexing. They succeeded in doing so by attaching the Drive data to an indexing service created after Drive existed. It's working great and has a 10-20X speedup. But oops... That new indexing system has a 5-million-element hard limit built in, and nobody caught it until they went to production. J. Noogler is too new to have realized this could have been an issue so they never started the escalation / customer messaging processs.
A lot of high scale businesses have taken a huge risk. A company like Google had over 10y long tenure engineers who were swept by the layoffs. The argument that the year prior was a hiring spree only makes it worse: now you have hands to throw at the problem but unlikely the right pairs, which, given the climate of job loss fear, aren't likely to admit they are incompetent.
Off topic, but this isn't an isolated case of mega size top tier companies seemingly dropping the ball more far often and/or with greater blast radius impacts. See github leaking secrets publicly on their own git repo.
Call me paranoid, but to me one of these things is going on:
1/ tipping point of too many under qualified for the job engineers running things, due to C-suites looking at keyboard monkeys the way they see factory workers, naively applying "cost cutting" measures, plucking through spreadsheets who "are the biggest seemingly disposable weights" and fire those as part of waves of x% layoffs
2/ Overworked remaining know-how crews - added the weight of dealing with inspiring politicians turned enginners for the juicy 200k + rsu package who, demand better articulated instructions, knowledge transfers and more comprehensive formal documentation materials, so that they too can shine too - on top. Competent folks having less and less time but more more tears and blue bags below their eyes finally realising that ain't worth it as it keeps getting worse anyway.
3/ sabotage
Or all those, since there could be some smowball effects there.
Call me paranoid, that it isn't what's going on, we clearly aren't seeing the slow fall of major infrastructures the few who were there to build them now having mostly packed their bag and long gone, replaced by swaths of bootcamp trophy coders hoarding cloud certificates like North korean generals like to collect shiny pins.
Maybe it isn't, the future is still in the making anyway, but I felt glimpses of that several years ago, then about each passing year, now every few months. like an seeing accelerating meteroite seemingly going away but which keeps looking bigger and bigger.
More so related to the topic: just drop drive as a centralized storage, there are e2e solutions out there and open source scripts to entirely migrate out of these wall gardens. You won't miss google sheets: you can still use it!
I could imagine that some services within Google haven't been carefully designed - for example, perhaps the quota service reads a complete file listing into RAM, adds up the sizes, and then writes the available quota.
This is probably the easy fix, rather than redesigning every service to be able to stream objects correctly.
You write your scanner to divide up the list of user accounts into chunks, process all chunks in different worker machines, and combine the results. Simple mapreduce.
However, if there is one huge user account, then you either have to wait for the entire process to take far longer, or you need to have multiple workers working on the single account (adding a lot of complexity to every operation you wish to run across all accounts).
That doesn't make sense because this is a new restriction.
https://www.computerworld.com/article/2534312/the--640k--quo...
Because they have no retrieval fees for gsuite, I bet it is expensive dealing with drives with tons of small files.
Agree they are rolling this out terribly. They should have granted everyone 2x the number of files they currently have in their drive quota (or 4M, whichever was smaller).
I see folks describe Google Drive errors / limits like this and people describe "a business critical operational system" and I wonder. Is Google Drive even supposed to do this thing?
Not pointing the finger back at them, if Google doesn't make it clear they probably should, but at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service.
That may be how you view the product, but Google Drive absolutely is used as an enterprise cloud service by many large companies. And as a result, there are lots of applications that integrate with its API, etc., to serve those enterprise use cases.
I just wonder .... is that what Google built Drive for? / how much of that they were thinking of.
Sometimes there are random UX/UI/API limitations, but those usually get fixed relatively quickly when someone hits them, because they’re typically just some hardcoded limit someone put in a front end layer because it was ‘more than enough’ and they needed something. A bit of elbow grease to clean up that path if it seems useful, and it’s usually good again.
They sell enterprise plans, and have for years. It's definitely an intended use of their product.
https://workspace.google.com/products/drive/
It's funny to read your comment, because to me, Google Drive feels like an enterprise product that incidentally happens to provide a consumer-facing version as well, not the other way around.
But you are right, google offers enterprise grade solutions so is expected to support over a few million files per account or even per user. Maybe something changed due to hasty cost cutting measures given the sudden interest by many on the issue tracker.
Yes - I mean, sort of. It depends on what you define as the starting point for Google Drive. Like many Google products, the brand has been reorganized and reused a number of times.
If you look at Google Drive as an extension of Google Docs, which existed prior to 2012, then yes, it started off as personal use. But if you look at it as a thing that launched in 2012 (the introduction of the Google Drive brand), then it arguably belongs as part of the reorganization that involved ending G Suite and turning it into Google Apps for Work[0], which is now known as Google Workspace.
In fact, now that the consumer version of Google Drive has been reorganized into Google One, you could argue that the enterprise version of Google Drive is actually older than the consumer version!
Of course, all of these distinctions are somewhat arbitrary, because it's the same product that you've used all along, and the only thing that changes is the pricing and the marketing. But if we're talking about consumer vs. enterprise versions of a commodity product that essentially operates at enterprise-scale for consumers, that's also the only real difference.
It's actually quite funny that the same product (which has been improved slightly, but is largely identical to what it was over a decade ago) has had so many different brands attached to it, especially when you consider that some of those brands are essentially container formats that repackage the same underlying brand (e.g., Google Workspace contains Google Drive).
[0] There was actually a rebrand in between, from 2012-2015, but because Google's product names are Alphabet soup, I can't be sure of what they called it in the interim, though I'm positive it was a distinct product from before 2012.
We did have unlimited storage, however last week they announced that's soon to change:
> A quota will soon be introduced on the amount of data that staff and students at the University can store in their Google account. This is part of a larger piece of work being carried out by the Google Workspace team in IT Services, to review the overall amount of storage being used by the University.
Makes me wonder if the timing is somehow related.
https://blog.google/outreach-initiatives/education/google-wo...
I didn't even know 3PC was used by google drive. Not sure what could be non cpincidental otuer than a technical reason for it given the very collaborative nature of this tool.
Suspicion of dark pattern is not proof of dark pattern.
(ETA: at Google in particular, the dark-pattern-conspiracy you're describing between two unrelated teams, in this case Drive and static storage, is hard to pull off because coordinating those teams requires documenting design, and it turns out a lot of folks working at Google are Hacker News-type personalities who will ask inconvenient questions if they come up. But what they have that we lack is observation of the larger security picture and the ways failure to, for example, 3PC-guard your static files can result in failure modes).
There is general ignorance of what google does/is/isn't and how online storage works. I agree with the sentiment, but, I've seen many a thing from recipes, shopping lists to FPA/Proposals all hosted on google drive. Even people sell content that is hosted on google drive.
Drive is paid (consumer via One, business via Workspace) cloud hosting with a free consumer tier.
In my mind, Drive isn't a file sharing / storage system, but rather a document sharing / storage system. If you want to store large numbers of files, you should use Cloud Storage, which can handle that. Drive is the cloud equivalent of LAN shared folders - you know those janky NFS and SMB shares that were always a giant headache because of permissions and other nonsense? That's what Drive is trying to be.
Seems like they could do it without the jankiness, though.
If you have 100,000 source code files, art assets or something like that, you probably don't want to use Drive to manage them. The UI just isn't built for it and it doesn't scale.
I certainly joined GDrive believing it is designed as a cloud drive where I put whatever I'd like.
It definitely lives up to that. I'm always getting sent links to files I can't open.
That does seem how it operates. A lot of the UI is directed at document sharing.
That's what makes me double take when I hear about folks using it for some very creative, but not what I think of uses for Google Drive.
I really dislike Google Drive. I can’t imagine people would use it at all if not for the fact that it’s bundled, monopoly style, with the office suite and gmail.
Since Google won’t ever fix it, I’m hoping they rapidly make it so crappy some other competitor crops up.
Of course, with our luck, it’ll be some Microsoft offering.
I wish software would stop getting worse.
I work it in a fly neuroscience lab and we use it to store all our electrophysiology and video data. Each person in the lab is storing on average 5TB of data, and the lab as a whole stores 100TB.
The graphical user interface combined with unlimited storage for Google Workspaces is essentially an unbeatable deal. Researchers can upload their data easily through the interface. Any custom solution based on S3 or equivalent would take some time to teach and more time to maintain. Also, we're paying about $200 / month total to store 100TB of data in the cloud, which is hard to beat with other services.
I tried setting up a single account for the whole lab once, but we ran into the above 5M file limit, so we just have individual accounts per researcher and it's mostly fine for now.
You should expect this to go away soon. I support science researchers and our unlimited storage option is going away in the coming months. Options for purchasing space are limited, and not cheap.
https://www.nyu.edu/life/information-technology/about-nyu-it...
https://case.edu/utech/about/utech-news/unlimited-google-sto...
https://scripps.ucsd.edu/it/google-storage-changes
https://it.wisc.edu/about/it-project-portfolio/reshaping-the...
etc...
I haven't heard about the google workspace enterprise unlimited storage going away anytime soon, although perhaps you know something I don't?
For professional use, I've usually seen it used as a document repository that can be shared by teams.
In both use cases, 5 million files seems like it'd be a hard limit to hit.
Like anything, people can push limits where it’s no longer productive to extend.
The number of stored files: 400,000
Nevertheless, there must be something related to their storage model that ranks count of files > storage size.
Geospatial artifacts can be very large, and also can be spread across thousands and thousands of files (20+ zoom levels on Earth, with enough 256x256 tiles to cover the planet at each zoom level, for example).
See some comments on the tracker, some orgs have a multi sites set up across an entire nation. I wouldn't be surprised if 10k+ volunteers at some NGOs are on that sort of plan and use it as their personal storage solution on top of all the things they sync for document counts heavy work.
Just because your workflow doesn't involve lots of tiny files doesn't mean other people's don't.
Besides, most of those commenting in the issue are talking about their organization having hit the limit, not individuals.
That may be what you use it for, but that doesn't mean other people use it in that way. And it doesn't mean that's how Google represents it in their documents.
I tried to look for documentation about what Google Drive should and shouldn't be used for.
In the FAQ Google Drive said it could be used for storing any file type. There are instructions for how to backup/sync any folder on your computer to Google Drive. If you're an organization that has several people backing up folders with large numbers of files, there's nothing Google says that makes you think you shouldnt use it that way.
https://support.google.com/drive/answer/10838124?hl=en
The only thing that indicated what it shouldn't be used for was content policy page. It basically says it shouldn't be used for illegal activity and that it shouldn't be used for streaming video.
https://support.google.com/docs/answer/148505
Its SLA.
Or more precisely, its lack of one.
It seems odd that someone would say that you shouldn't use Google Workspaces for work.
Google Drive appears to be targeted at "human" usage, i.e. people uploading or creating files. I would guess that this is also worked into the cost – that the assumptions of the amount of work a human can do are a part of the price formulation. The reporter of this bug seems to be using this as a storage backend for software though, which I don't believe is the intended use-case.
Looking at S3 pricing, just the storage is ~5x the cost of Google Drive, and then you need to add transfer and API calls on top of that.
I don't personally think that there are reasonable use-cases for human users with 5 million files. There may be some specialist software that produces data sets that a human might want to back up to Google Drive, but that software is unlikely to run happily on drive streamed files so even those would be unlikely to be stored directly on Drive.
(Disclaimer, I work at Google, not on Drive, this is my personal reading and interpretation of the public info, I don't have any inside info here)
I imagine that if you setup multiple servers/personal computers backups that could scale significantly.
[1]: https://restic.net/
Given this, it is impossible to exceed the 5M file limit with the top plan google offers of 30TB. You'd have to lower the pack size to 6MB or less.
This is the way to go. Files will count towards the Service Account, not your account. If you use Google Workspace I strongly recommend using a Shared Drive (instead of sharing a folder in “My Drive”) so the files will be owned by your org, not by the Service Account (otherwise deleting the SA would result in the files being deleted as well, because in “My Drive” the creator of a file keeps ownership even when creating in a shared folder). If you have problems with the 400,000 files limit in a Shared Drive: create a new one, rclone has a “union” backend. It can be set to create new files in the new Shared Drive while still showing files from the other drive(s). Also Shared Drives have their own recent activity views so it doesn’t clutter your “my drive” view.
Create a SA in GCP and generate a JSON credentials file. Create a folder or Shared Drive via web and share it with the SA.
I’m using this setup for years with zero problems.
Example rclone config: https://pastebin.com/KdsFQz5K
However, when you're paying for a product, the limits should be disclaimed. If I'm investing in Google Drive, I should be able to easily see the limits which apply to the product I've bought; that means total storage caps, file count caps, bandwidth caps, and whatever else.
It would be pointless if your car came with a giant list of the melting point of every material used in the structure of your car so you know what the “real” temperature limits are. Oh boy, better not let the car get to 1450C or else the steel frame might melt!
If the storage vendor doesn't want people to use their product for the many use-cases where you end up with a bunch of small files, advertise the file count cap. Simple as that.
A 5 million file limit might be quite reasonable if you're paying for the basic, 100 GB storage tier. But Google Drive offers multiple tiers with up to 30 TB of storage. 30 million MB!
That means if your average file size is less than 6MB - very likely if you're storing JPEGs, audio files, text records, or whatever; you'll never be able to fill your Google Drive storage.
What's the average size of a file on a regular macOS or Windows HDD? It wouldn't surprise me if it's much less than 6MB. Shouldn't the file count limit scale with the storage limit?
% find $HOME -type f | wc -l
4969169
Almost 5M there.
Of course, it definitely is much nicer to have S3 over google drive for such a solution, but if you didn't want to write software (which you'd need for S3), and simply used the google drive's sync'ing features...
The chance of hitting this limit is so insanely slim and the workarounds are so trivial that I don't really see any issues here. I bet there would have been some abuse where someone tried to store data in file names and metadata which is not counted to your quota.
https://rclone.org/
My last check had the service account that I'm responsible for somewhere around 8.7 Million files with an average size around 9.2MB.
We have multiple of these "accounts", with similar use-cases across other teams. This is excluding the normal use of Gdrive with normal "rank-and-file" office works managing standard office docs.
We've felt the squeeze from google over the past few years, so we've already started migrating off of google services about 2.5 years into a 4 year project.
I think the rule of thumb I would use is that if you're creating the files in an automated way, and then accessing them outside of Drive interfaces, in all cases, then it's probably a better fit for a web app with cloud storage backend. Google Cloud Storage or S3 would work well for this.
The Google Drive landing page (https://www.google.com/intl/en-US/drive/) as of April 1, 2023 still doesn't mention the 5M per user maximum files count limit.
Is it an April fool's joke? /s.
Do better $GOOG.