At least a totally separate account. Probably better to use a totally separate set of IP addresses and browsers and maybe even computers. Google will definitely link accounts created from the same browser and potentially ban your main account if you violate their TOS on another account also owned by you.
This is a complete hack job and probably useless if Google changes free storage for docs.
That being said, they currently allow the guys at /r/datahoarder to use gsuite accounts costing £1 for life with unlimited storage quotas. These are regularly filled to like 50TB and Google doesn't bat an eye.
Couldn't you also embed data into images and upload them to Google photos, or is that discarded when they convert and compress the image in the backend?
I mean include binary data in an image file. So you would have a 300x300px jpg picture of a flower that's 20mb which you could unpack to a binary file.
I think that depends on their tools and how they evaluate data usage. If the reporting states that the accounts are using very little storage because it's using the same measuring stick that the client does them it's invisible. The question comes up during an audit of the system when the disk usage doesn't match the report. Then again, if this is used by few people it may just look like a margin of error.
It'll more be that the Google docs "live editing" backends are expensive to use disk and memory wise. They store complete version history with each keystroke of a document.
There's a good chance a megabyte of "document" costs Google a gigabyte of internal storage...
They don't store a complete version history. It just uses checkpoints in their timeline of real-time edits and computes the differences when you need them. Those deltas can also be compressed.
I had an (evil; don't do this) idea a while back to create a Dropbox-like program that stores all your data as binary chunks attached to draft emails spread across an arbitrary number of free email accounts.
Definitely would make an interesting learning exercise--I learned way more about SMTP/POP protocols* than I did before when I implemented demonstration SMTP/POP servers for my libgmail library before Gmail offered alternate means of access.
These days there's even the luxury of IMAP. :D
[*] About the only thing I remember now is the `HELO` and `EHLO` protocol start messages. :)
Me and a friend came up with a similar idea of a sort of distributed file system implemented across a huge array of blog comment sections. Of course you’d need a bunch of replication and fault tolerance and the ability to automatically scrape for new blogs to post spammy-looking comments on, but I thought it was a pretty funny and neat idea when we came up with it.
Even scarier than that would be a Turing complete language where the code is stored and memory is written to comments sections. The actual execution could be done by reading, execution function, and writing comments to store working memory and results. I guess with cryptographic encryption you could even hide what your doing.
I heard about a subreddit a while ago, where every post/comment was a random string. It was speculated at the time that something similar was going on.
It's even more interesting to think about this in the context of preserving banned information for future generations. For example, if all the countries in the world united to ban the New Testament. But you eventually realize the ephemeral nature of the net will probably prevent it from fulfilling such long-term data-archiving roles and you're better off burying manuscripts deep underground.
Has anyone actually tried storing a large amount of data like this? I feel like creating a new google account and using it as a backup for a 300gb folder I have.
Yes. It's called: Post to alt.binaries.* on Usenet.
It's effectively the same thing under the hood. Binaries are split and converted to text using yEnc (or base64, et al.) and uploaded as "articles". An XML file containing all of the message-IDs (an "NZB") is uploaded as well so that the file can be found, downloaded, and reassembled in the right order.
This form of binary distribution has been around since the '80s if you change some of the technical details; e.g. using UUencode rather than yEnc.
Spend $5 for a 3-day unlimited Usenet account with e.g. UsenetServer.com and upload it.
If you want it to stay up, then make another account in 3925 days (the retention period), download it, and then reupload it for another 10+ years of storage.
That references "for education", but it's also true for GSuite Business (and enterprise, but not basic). You'll need to be paying for at least 5 users, or $60/month.
In the same spirit, I made a few "just for fun" plugins for my (now abandoned) encrypted-arbitrary-storage Dropbox-like application Syncany:
The Flickr plugin [1] stores data (deduped and encrypted before upload) as PNG images. This was great because Flickr gave you 1 TB of free image storage. This was actually super cool, because the overhead was really small. No base64.
The SMTP/POP plugin [2] was even nastier. It used SMTP and POP3 to store data in a mailbox. Same for [3], but that used IMAP.
The Picasa plugin [4] encoded data as BMP images. Similar to Flickr, but different image format. No overhead here either.
All of this was strictly for fun of course, but hey it worked.
Anything that persists can be used to store arbitrary data... I remember (around a decade ago now, I'm not sure if these still exist) coming across some blogs that ostensibly had images of books, details about them, and links to buy them on Amazon and such... I only understood when I came across a forum posting from someone complaining that his ebook searches were clogged with such "spam blogs", and another poster simply told him to look more carefully at those sites, but not to say anything more about his discoveries. You can probably guess what you got if you saved the surprisingly large "full-size" cover image from those blogs and opened it in 7zip!
I feel less hesitant about revealing this now, given how long ago it was and that more accessible "libraries" are now available.
IIRC the “mods are asleep, post […]” 4chan meme originally came from “mods are asleep, post high res” threads where to an outside observer they were just posting high-resolution images of inane things, but there was actually steganography of some sort going on to hide child porn (I think) inside the files.
I can't remember if I tried, but it's important that you get the exact data back that you put in, which is why JPEG obviously won't work.
BMP is the easiest to encode/decode because it's literally a bitmap of RGB, no fancy compression and such, which, if you're storing arbitrary data is obviously not necessary.
PNG was trickier, because of its "chunks" and generally more structure. And compression.
Base85 would probably be a better choice for storing binary as text, since it has a ratio of 5:4 instead of 4:3.
On the topic of "unusual and free large file hosting", YouTube would probably be the largest, although you'd need to find a resilient way of encoding the data since their re-encoding processes are lossy.
I like the "Linux ISO" and "1337 Docs" references ;-)
The nice thing about yEnc is that it only has to escape NUL, LF, CR, and the escape character itself '=', so it essentially uses all but 3 characters out of the 255 possible values.
While this works over NNTP, SMTP and IMAP (and possibly POP), I'm not sure if it will work over HTTP if any of the servers use the Transfer Encoding header.
I don't think you want your backups in Google Docs either, given that Google may decide to ban you for TOS violations at any time.
I really do think videos would would work, reliably, given sufficient redundancy. Again, we have QR codes already, so this is a proven idea. You can't make QR codes unreadable without removing lots of perceptual visual details. The risk, as with using Google docs, isn't that Google will change their encoding, but that Google will just take down the videos for service misuse.
I think it would be comparatively more difficult for Google to detect this stuff in a video compared to a text document, because you expect some videos to be long and large. The entirety of the Encyclopedia Britannica comes out to less than 500 MB in a .txt document, so using any reasonable amount of space in a Google Doc should quickly raise red flags.
Back in the day of email gateways between different networks, there used to be a terrible problems with all the tin-pot dictator IBM SYSADMINs at BITNET sites who maintained their own personal styles of ASCII<=>EBCDIC translation tables, so all the email that passed through their servers got corrupted.
EBCDIC based IBM mainframe SYSADMINs on BITNET were particularly notorious for being pig-headed and inconsiderate about communicating with the rest of the world, and thought they knew better about the characters their users wanted to use, and that the rest of the world should go fuck themselves, and scoffed at all the unruly kids using ASCII and lower case and new fangled punctuation, who were always trying to share line printer pornography and source code listings through their mainframes.
"HARRUMPH!!! IF I AND O ARE GOOD ENOUGH FOR DIGITS ON MY ELECTRIC TYPEWRITER, THEN THEY'RE GOOD ENOUGH FOR EMAIL! NOW GET OFF MY LAWN!!!" (shaking fist in air while yelling at cloud)
It was especially a problems for source code. That was one of the reasons for "trigraphs".
>Trigraphs were proposed for deprecation in C++0x, which was released as C++11. This was opposed by IBM, speaking on behalf of itself and other users of C++, and as a result trigraphs were retained in C++0x. Trigraphs were then proposed again for removal (not only deprecation) in C++17. This passed a committee vote, and trigraphs (but not the additional tokens) are removed from C++17 despite the opposition from IBM. Existing code that uses trigraphs can be supported by translating from the source files (parsing trigraphs) to the basic source character set that does not include trigraphs.
youtube probably doesn't save the originals (though they could in some cold-storage tape drives, perhaps). But even still, it's not difficult to imagine that there may at some point exist a compression algorithm that can be applied to existing compressed video that could change a couple bits around in whatever encoding scheme you've chosen. Depending on the file type, that could be enough to corrupt the whole thing.
Sure you can get around this by adding ECC, but that isn't implemented here.
A few years ago I also found a backup tool that converted backups to DV videos, so that you could write them on cheap DV cassettes. It was something like more than 10 GB per cassette. Definitely not bad for a few years ago.
Base64 has the advantage of relative ubiquity (though Base85 is hardly rare, being used in PDF and Git binary patches). It also doesn't contain characters (quotes, angled brackets, ...) that might cause problems if naively sent via some text protocols and/or embedded in XML/HTML mark-up.
> YouTube ... you'd need to find a resilient way of encoding the data [due to lossy re-encoding]
That should be easy enough: encode as blocks or lines of pixels (blocks of 4x4 should be more than sufficient) in a low enough number of colour values (I expect you'd get away with at least 4bits/channel/block with large enough blocks so 4096 values per block) and you should easily be able to survive anything the re-encoding does by averaging each block and taking the closest value to that result.
Add some form of error detection+correction code just for paranoia's sake. You are going to want to include some redundancy in the uploads anyway so you can combine these needs in a manner similar to RAID5/6 or the Parchive format that was (is?) popular on binary carrying Usenet groups.
Reminds me of the old programs that would turn your Gmail storage into a network drive by splitting everything into 25MB chunks. Utterly miserable experience with terrible latency and reliability.
But I don't understand why Google would do that. For most users, aren't Google Docs files a substantial part of their usage? Or do people mainly store backups?
FTA "A single google doc can store about a million characters. This is around 710KB of base64 encoded data."
This means that in order to reach the limit of the drive space given away for free, they'd need something like 15,000 Google Doc files (15GB) if they counted toward your space limit. I doubt a lot of paying customers even reach that.
The real limit (file size) is reached by binaries. Videos and PDFs, usually.
I bet implementing such a limit would be 3 or more months of engineering effort.
Think about the difficulties. It has to take into account shared directories. It has to know about systems which auto-create documents (like results sheets for Google forms). It has to work with gsuite sysadmins who need to take ownership of files from deleted accounts. The UI to show when you have hit the limit has to be designed. And the support team has to be trained on how to resolve that error. And you're going to have to get that error message translated into 30 languages. Users already over the limit are going to be unhappy - are you going to write extra code to give them a grace period? How will you notify them of that? Will you have a whitelist of users allowed to go over the limit? How will you keep the whitelist updated and deployed to the servers? Who will have access to add/remove people from the whitelist?
The actual system itself has race conditions:. What if that 15000th file was simultaneously created in the USA and Europe? There is no way to prevent that without a cross-ocean latency penalty. Do you want to pay that penalty for every document creation? How do you deal with a net-split where cross ocean network traffic is delayed?
Finally, how will you monitor it? Will you have logging and alerting for users hitting the limit? Will there be an emergency override for engineers to remove the limit if necessary?
At big-web-service scale, simple engineering problems become complex problems fast...
Back when the commercial internet was just getting its act together there were companies that would give you free online access on Windows 3.1 machines in exchange for displaying ads in the e-mail client. (I think one was called Juno.)
The hitch was that you could only use e-mail. No web surfing. No downloading files. No fun stuff.
But that's OK, since there were usenet- and FTP-to-email gateways that you could ping and would happily return lists of files and messages. And if you sent another mail would happily send you base64-encoded versions of those binaries that you could decode on your machine.
The free e-mail service became slow-motion file sharing. But that was OK because you'd set it up before you went to bed and it would run overnight.
The original Juno ad server proxied the ads from the internet to the email client, and the proxy was wide open for several months. The first time I ever accessed the open internet at home was by dialing into the email service and bouncing through the proxy. I believe it was closed due to it being shared in the letters section of a hacker zine.
Yep and the Shar command that created a bash wrapper round sections of uuencoded data, so you could email a file in segments and conveniently recompose and run it to get the file back, without needing Shar at the other end. Good times.
That brings back memories. Of using an email gateway to get an Amiga fred fish disk - delivered as shar pieces to my uni email account (only staff had telnet, ftp, etc. access). Then assembling the pieces in /tmp on the departmental unix server. Then switching to a PC to use Kermit to get the contents onto a PC floppy. Then using an Amiga utility to be able to read PC format disks to copy them to an Amiga floppy.
I've no memeory of what motivated me to spend so much time just to be able to view some low-res, low-fps 3 second video clip, listen to 8-bit tracker "tunes" and try out some free application that invariably crashed the machine after a minute or two of use.
That reminds me of the first time I accessed World Wide Web. Back in '96 I was browsing a computer magazine and happened upon a listing of useful mailing lists, one of which returned the contents of web pages for a requested HTTP address. Same magazine had an install CD for the free Juno email service.
Being a teenager, the first web page I ever requested was www.doom.com, which returned a gibberish of text to Juno's email client. It was an HTML file full of IMG tags (one of those "Click here to enter" gateway pages), but I had no idea what I was looking at at the time. Somehow figured out to open the file in IE2 and saw... a bunch of broken images :)
I still vividly remember the sense of wonder that the early Internet evoked.
EDIT: Just checked the Wayback Machine. Looks like www.doom.com was not affiliated with the game at the time, so I must have browsed to www.idsoftware.com instead.
It's really sad thinking how kids these days totally miss the wonder of the early internet.
In my case, it was at the public library. The lone internet computer was constantly booked. But by watching over a library clerk's shoulder, I was able to see the password needed to unlock the text-based library catalog terminals (which terminals were plentiful and always available). (My parents worked at the library, or else I never could have pulled that off.) Once unlocked, I was able to use Lynx to telnet into my favorite MUD game. Unfortunately it didn't last long until a librarian caught me, which I think resulted in me being grounded from the library for a month or something like that.
First time I was able to access the WWW via a graphical browser I had a dial-in shell account at an ISP (or BBS or whatever they called themselves back then), then there was a program called "slirp" (which, amazingly enough, seems to have a wiki page at https://en.wikipedia.org/wiki/Slirp ) which allowed one to run "SLIP" (IP-over-serial) over the terminal connection to get IP access from my computer. Amazingly I got it to work, considering I barely knew what I was doing back then.
One big reason why I became a Linux user was that the TCP/IP stack for Win 3.1, Trumpet Winsock, was amazingly unstable and would regularly crash the entire OS. Linux had, even back then, a stable TCP/IP stack. And fantastic advancements like preemptive multitasking running in protected mode so errant user-space applications didn't crash the OS.
Very cool! About a year ago I had a similar idea, but to store arbitrary data in PNG chunks[1] and upload them to "unlimited" image hosts like IMGUR and Reddit.
Although if Picasa (predecessor to Google Photos) worked with BMP, it may be better to do that because it's much easier and more space efficient to encode arbitrary data in than PNG.
Anyone remember Gdrive? I can’t find it now, but I think it was probably early or mid 2000s. It let you store files as a local disk (FUSE) via Gmail attachments.
I remember using it back in 2005 iirc, and it was amazing. The files had a label called gmailfs.gDisk which is how it could keep the "file system" separate from the rest.
Now Google generously offers Drive with 15Gigs of space.
There were a couple of different projects at the time (listed in "Other Resources" on the project page) that sought to provide a programmatic Gmail interface.
I still have a "ftp" label in Gmail (checks notes 15 years later...) from the experimental FTP server I implemented as a libgmail example. :D
The libgmail project was probably the first project of mine which attracted significant attention including others basing their projects on it along with mentions in magazines and books which was pretty cool.
I think my favourite memory from the project was when Jon Udell wrote in a InfoWorld column ( http://jonudell.net/udell/2006-02-07-gathering-and-exchangin... ) that he considered libgmail "a third-party Gmail API that's so nicely done I consider it a work of art." It's a quality I continue to strive for in APIs/libraries I design these days. :)
(Heh, I'd forgotten he also said "I think Gmail should hire the libgmail team, make libgmail an officially supported API"--as the entirety of the "team" I appreciated the endorsement. :) )
The library saw sufficient use that it was also my first experience of trying to plot a path for maintainership transition in a Free/Libre/Open Source licensed project. I tried to strike a balance between a sense of responsibility to existing people using the project and trusting potential new maintainers enough to pass the project on to them. Looking back I felt I could've done a better job of the latter but, you know, learning experiences. :)
My experiences related to AJAX reverse engineering of Gmail (which was probably the first high profile AJAX-powered site) later led to reverse engineering of Google Maps when it was released and creating an unofficial Google Maps API before the official API was released: http://libgmail.sourceforge.net/googlemaps.html
I've seen a few stories of businesses and their employees all losing their Google accounts, just because the company hired a freelancer who had previously been banned, and Google detected the association. (Pretty sure they got the accounts back after some public outrage.) I wouldn't risk intentionally violating their terms if you're not quite ready to wake up one day 100% Google-free, or very good at hiding your tracks.
The fact that this is even a remote possibility should worry everyone of the ugly monopoly that Google became.
I found myself in a similar situation a couple months ago. An android App falsely charged me on the Play store. After trying to contact Google for multiple weeks I gave up and disputed the charge on my credit card. This resulted in Google coming after me for 8.99$ and threatening me to close all my Google accounts including gmail, calendar, photos, drive and everything I rely daily in Google.
That was a wake-up call for me. I decided to move everything OUT of Google. That company got too much power, it should worry way more people.
The sooner you start, the better. I've moved most of my email/contacts/calendar away [0], and the longer you give yourself to catch the things you've signed up for but forgotten, the better. Youtube was also a pain, but I transitioned my subscriptions manually to a different account. Maps seems like it'd be the trickiest if you're invested. I wasn't a heavy user, and maps still works pretty good when you're logged out.
[0] I use fastmail + custom domain, which works great, but you have to guard the domain very closely.
I think OP means that you have to make sure you don't forget to/neglect to renew it and make sure you don't accidentally lose the domain for any reason.
spot on, basically you now have to worry about the domain being lost or hijacked also. for me, the flexibility to change email providers behind a domain is worth it though
Thank you for the clarification. I use a dedicated card for domain hosting (with autorenewal enabled) to prevent this specific issue but I recognize most people likely don't do the same.
It means if you slip up and lose your domain, nobody can send you email (including 2FA, reset password, add a new email to your account, etc). You can imagine how inconvenient that would be. I use fastmail with a custom domain and that scenario gives me nightmares.
Mostly off-topic, but related: this is one of the major reasons email needs to finally go away. It was never intended to be the backbone of peoples lives in the way it has become.
Access to my email account probably gives you more access to my life and identity than my SSN [0].
I long for the day that we [1] all get assigned a public/private keypair instead of SSNs. That won't fix everything, but it's a huge step above a shared secret that is limited to 9 digits [2].
[0]: Even without signing up for a bunch of services, it's basically impossible at this point (at least in the US) to not have an email address associated with your bank account, car loan, mortgage, credit card, or even just watching TV.
[1]: "We" meaning "US citizens" or anyone else with a similar system.
[2]: I realize you also need info about the person and not just their number, but also apply that to keypairs.
Have the organization responsible of managing the PKI to generate a new subkey from your primary key (kept in cold storage) and publish a certificate revocation for the previous subkey lost/leaked.
Most of our ID cards (health, driving license) already have an expiration date and the subkeys should have one anyway.
No reason you can't have more than one, either. You could even issue keys for people to act on your behalf (e.g. they get access to it on your death as part of your will).
I have been doing it for a long time, the hardest for me is all the registered users I have around the web linked to the email. After a few years of changing each one that mattered I finally get close to zero mail on gmail.
Search I moved to ddg, that was the easy one.
Android can work fine with just f-droid since I noticed I rarely even use the store any more and I need just a few essential apps. For storage, I tend to store only documents and I like to use mega.nz.
The only thing I haven't managed to find a even close to decent alternative it's photos. Google Photos is just simply too good. I would be even willing to pay but really, all the other apps struggle to get sync right or have some other crappy stuff that makes them barely usable.
As I wrote that comment I went on another small search as I do every so often and I found Canon Irista and I have to say, I am impressed. The sync seems to work fine, it's pretty fast and the UI both of the website and the app is pretty solid. I suggest giving it a try if you are on the lookout for a new photo hosting service.
Nextcloud, preferably on a machine you own (but there are companies selling Nextcloud hosting as well). It replaces Google Drive, Contacts, Calendar, Photos (face recognition can be done with a third-party app), has an RSS reader, bookmarking service etc. Just look at its app store, you can install any of this with two clicks: https://apps.nextcloud.com/
It really is a suite that can combat Google's suite — and you can truly own it. Other than that, DDG for search, and your own domain for email (so that you could transfer it between different hostings if necessary).
I do have a Google account, but I use it for precisely two purposes: Google Play (my phone wouldn't work without one) and YouTube subscriptions (I can use an RSS reader for this, but it's a bit inconvenient). You can create a Google account without creating a Gmail account.
May I suggest using NewPipe[1] for a Google-account-free experience to follow channels? You can import them from your current subscription list, and easily export them when you switch phone or for backup purposes.
Are they? Email, calendar, online office, cloud storage etc. are all available from various other companies(even beside the big few corporations). The only two areas where you'd really have to sacrifice features would be Android apps, and YouTube if you're running a channel.
tell me which provider has an integrated single signon service for all of those above? Which provider has apps for their service for all major OS'es (including mobile), and is mostly free (or low cost)?
Microsoft does. OneDrive for storage, Outlook webapps does email and calendar. Office online has Word, Excel, etc. All accessed with one Microsoft account. All free.
You might not want to be tied to Microsoft but Google is not the only option.
Edit: Overlooked the comment about Apps. Microsoft offers apps for mobile, but not Linux. Although even on Windows I use the browser to access the services which will work on Linux.
Microsoft's office offerings comes very close (cept for the free part - which i guess is just a bonus and not a requirement). Although i have to say, despite microsoft's attitude for keeping compatibility and old stuff working, they too could chuck a google reader one day, and deprecate/remove a needed service (along with all your data).
What's needed is a syndication of data, and inter-operable apps. Like how xmpp worked. But of course, all vendors don't like this, because it turns themselves into a commodity.
Yeesh. I had the same happen - Except I never followed through on reversing the charge on my credit card. I spent multiple hours trying to dispute $2.99 or something. Clearly not for the monetary value - just from pure frustration!
However, I was scared for my Google account so just ended up dropping it. Ridiculous.
"An android App falsely charged me on the Play store."
I'm curious to know how this happened. Would you mind sharing more info?
As I understand it, the only way for an app to 'charge you on the play store' is to:
1) Be a paid app (in which case you pay before the app starts installing), or
2) via in-app purchases, which are handled by the app initiating the IAP, and then Play services taking over to ask for confirmation.
In either case, the transaction is only confirmed by a user action (tapping a button) with the app having no control.
Sure, it's possible for an Android app to trick you, by covering everything apart from the button with something fake, but I'd be surprised if such an app found its way into the Play Store.
Sure, I might have hit a corner case but I made an in-app purchase for a one year subscription for a service.
After using the app for a couple of days and restarting the phone, the app seemed to hit a bug and behave like if I didn't buy the subscription, prompting me to buy another subscription which I did thinking that this would unblock the backend and somehow merge with the fact that I already had a subscription.
Unfortunately, Google Play charged me again for a subscription I already had. Both the app creator AND Google Play were difficult to join. The App creators never replied to any of my emails. Google Play got an automated support website that decided that "I was not eligible for a refund" and there was nothing I could do about it. It also seems to be impossible to contact a real human being to explain the situation.
That gives a new meaning to "google bombing" -- a bad actor could cultivate a terrible google rating, then hire onto a low level freelance gig at a big company they wanted to bomb by association! Let's just say Oracle as a hypothetical example -- Russia, if you're listening...
> Pretty sure they got the accounts back after some public outrage.
This may happen only if you manage to get it to the front page of HN or have many Twitter followers. In most cases you don't stand much chances though.
Indeed. I love the "sorry @ the guys from google internal forums who are looking at this" line at the github. All tongue in cheek and aware of the situation.
TBH this is not unlike reporting a security bug to a company as a white hat, but more like a grey hat here.
> If the few blokes using this scam their way into few hundred terabytes of free storage, so be it, it's not worth the hassle for Google, imo.
This. They probably thought of this exact scenario before adding unlimited docs. They probably even expected somebody to make a script for it. Hell, a few of them might even have a script.
As long as a lot of people don't start abusing it or make a file-sharing service based on it, then they probably won't care. Basically, not until it's a significant enough threat to their bottom line.
Ultimately, it's no different than the inevitable person that just has a script to generate garbage and upload it to Google Docs as fast as possible. That's what the 250 docs a day limit is there for.
Hehe, I was just thinking how simple it will be for Google to identify accounts using this technique from simple usage analytics. I suspect this will not work for long... but still super cool!
Very neat, but it seems to me the issue with all wink-wink schemes like this is that you're ultimately getting something that wasn't explicitly promised, and so might be taken away at any time. So while interesting you couldn't really ever feel secure storing anything that mattered this way.
On the edonkey network, the file size would be reported raw but the clients could compress and transfer chunks to each other. Some guy had created an empty IL-2 sturmovik iso and seeded it. We lived at a government facility with ill-policed high speed (for the time) internet but even then I knew that I didn’t have a 400 Mbps connection. Maybe 2002/2003.
The whole thing only transferred a few kB. It looked like an entire disc though.
Maybe I'm missing a reference or joke here, but the size of a file means little with respect to how much it can be compressed. You can get a 1 petabyte file down to a few bytes if it's just `\0` repeated over and over.
250 comments
[ 2.7 ms ] story [ 253 ms ] threadI advise a totally seperate account when using this tool.
But anyways, something inside me likes it. Nicely done. Good job :)
I didn't know this was plausible.
That being said, they currently allow the guys at /r/datahoarder to use gsuite accounts costing £1 for life with unlimited storage quotas. These are regularly filled to like 50TB and Google doesn't bat an eye.
only high quality is unlimited (reduced size)
so, if your data survive photo compression technology, you can do it.
There's a good chance a megabyte of "document" costs Google a gigabyte of internal storage...
I would expect them to only store a diff between each version instead of storing the whole thing. Couldn't find much about this after a quick search.
> sorry @ the guys from google internal forums who are looking at this
These days there's even the luxury of IMAP. :D
[*] About the only thing I remember now is the `HELO` and `EHLO` protocol start messages. :)
Nowadays stuff like Dropbox is much more convenient and reliable.
If it’s only 300GB check out Backblaze B2. It would cost you $1.5 per month for that amount of data.
It's effectively the same thing under the hood. Binaries are split and converted to text using yEnc (or base64, et al.) and uploaded as "articles". An XML file containing all of the message-IDs (an "NZB") is uploaded as well so that the file can be found, downloaded, and reassembled in the right order.
This form of binary distribution has been around since the '80s if you change some of the technical details; e.g. using UUencode rather than yEnc.
Spend $5 for a 3-day unlimited Usenet account with e.g. UsenetServer.com and upload it.
If you want it to stay up, then make another account in 3925 days (the retention period), download it, and then reupload it for another 10+ years of storage.
https://support.google.com/a/answer/139019?hl=en#6_storage
https://youtu.be/y2F0wjoKEhg
tl;dw Upload is limited to 750GB per day per account
You need 5 users for GSuite Business, if you use those alone the limit is now 3750GB/day
The Flickr plugin [1] stores data (deduped and encrypted before upload) as PNG images. This was great because Flickr gave you 1 TB of free image storage. This was actually super cool, because the overhead was really small. No base64.
The SMTP/POP plugin [2] was even nastier. It used SMTP and POP3 to store data in a mailbox. Same for [3], but that used IMAP.
The Picasa plugin [4] encoded data as BMP images. Similar to Flickr, but different image format. No overhead here either.
All of this was strictly for fun of course, but hey it worked.
[1] https://github.com/syncany/syncany-plugin-flickr
[2] http://bazaar.launchpad.net/~binwiederhier/syncany/trunk/fil...
[3] http://bazaar.launchpad.net/~binwiederhier/syncany/trunk/fil...
[4] http://bazaar.launchpad.net/~binwiederhier/syncany/trunk/fil...
I feel less hesitant about revealing this now, given how long ago it was and that more accessible "libraries" are now available.
One of many Internet jokes with sinister origins.
BMP is the easiest to encode/decode because it's literally a bitmap of RGB, no fancy compression and such, which, if you're storing arbitrary data is obviously not necessary.
PNG was trickier, because of its "chunks" and generally more structure. And compression.
On the topic of "unusual and free large file hosting", YouTube would probably be the largest, although you'd need to find a resilient way of encoding the data since their re-encoding processes are lossy.
I like the "Linux ISO" and "1337 Docs" references ;-)
While this works over NNTP, SMTP and IMAP (and possibly POP), I'm not sure if it will work over HTTP if any of the servers use the Transfer Encoding header.
Build in a bit of redundancy, and I think it would work.
I really do think videos would would work, reliably, given sufficient redundancy. Again, we have QR codes already, so this is a proven idea. You can't make QR codes unreadable without removing lots of perceptual visual details. The risk, as with using Google docs, isn't that Google will change their encoding, but that Google will just take down the videos for service misuse.
I think it would be comparatively more difficult for Google to detect this stuff in a video compared to a text document, because you expect some videos to be long and large. The entirety of the Encyclopedia Britannica comes out to less than 500 MB in a .txt document, so using any reasonable amount of space in a Google Doc should quickly raise red flags.
EBCDIC based IBM mainframe SYSADMINs on BITNET were particularly notorious for being pig-headed and inconsiderate about communicating with the rest of the world, and thought they knew better about the characters their users wanted to use, and that the rest of the world should go fuck themselves, and scoffed at all the unruly kids using ASCII and lower case and new fangled punctuation, who were always trying to share line printer pornography and source code listings through their mainframes.
"HARRUMPH!!! IF I AND O ARE GOOD ENOUGH FOR DIGITS ON MY ELECTRIC TYPEWRITER, THEN THEY'RE GOOD ENOUGH FOR EMAIL! NOW GET OFF MY LAWN!!!" (shaking fist in air while yelling at cloud)
It was especially a problems for source code. That was one of the reasons for "trigraphs".
https://stackoverflow.com/questions/1234582/purpose-of-trigr...
https://en.wikipedia.org/wiki/Digraphs_and_trigraphs
>Trigraphs were proposed for deprecation in C++0x, which was released as C++11. This was opposed by IBM, speaking on behalf of itself and other users of C++, and as a result trigraphs were retained in C++0x. Trigraphs were then proposed again for removal (not only deprecation) in C++17. This passed a committee vote, and trigraphs (but not the additional tokens) are removed from C++17 despite the opposition from IBM. Existing code that uses trigraphs can be supported by translating from the source files (parsing trigraphs) to the basic source character set that does not include trigraphs.
Sure you can get around this by adding ECC, but that isn't implemented here.
Back in the day there were systems to back up to VHS tapes and those were way more lossy than YouTube https://youtu.be/TUS0Zv2APjU
Base64 has the advantage of relative ubiquity (though Base85 is hardly rare, being used in PDF and Git binary patches). It also doesn't contain characters (quotes, angled brackets, ...) that might cause problems if naively sent via some text protocols and/or embedded in XML/HTML mark-up.
> YouTube ... you'd need to find a resilient way of encoding the data [due to lossy re-encoding]
That should be easy enough: encode as blocks or lines of pixels (blocks of 4x4 should be more than sufficient) in a low enough number of colour values (I expect you'd get away with at least 4bits/channel/block with large enough blocks so 4096 values per block) and you should easily be able to survive anything the re-encoding does by averaging each block and taking the closest value to that result.
Add some form of error detection+correction code just for paranoia's sake. You are going to want to include some redundancy in the uploads anyway so you can combine these needs in a manner similar to RAID5/6 or the Parchive format that was (is?) popular on binary carrying Usenet groups.
Also, "Utterly miserable experience with terrible latency and reliability." is such a great customer endorsement quote. :D
http://www.viksoe.dk/code/gmail.htm
But I don't understand why Google would do that. For most users, aren't Google Docs files a substantial part of their usage? Or do people mainly store backups?
This means that in order to reach the limit of the drive space given away for free, they'd need something like 15,000 Google Doc files (15GB) if they counted toward your space limit. I doubt a lot of paying customers even reach that.
The real limit (file size) is reached by binaries. Videos and PDFs, usually.
But then, I suspect (as others note) that Google will notice when you have 50GB of Google Docs full of base64.
Seems an iffy way to store stuff.
They'd have to write a base64 detectors and automate the detection and banning of the accounts without the engineer ever seeing your files.
Any bugs in that code, and they'll ban innocent people.
Think about the difficulties. It has to take into account shared directories. It has to know about systems which auto-create documents (like results sheets for Google forms). It has to work with gsuite sysadmins who need to take ownership of files from deleted accounts. The UI to show when you have hit the limit has to be designed. And the support team has to be trained on how to resolve that error. And you're going to have to get that error message translated into 30 languages. Users already over the limit are going to be unhappy - are you going to write extra code to give them a grace period? How will you notify them of that? Will you have a whitelist of users allowed to go over the limit? How will you keep the whitelist updated and deployed to the servers? Who will have access to add/remove people from the whitelist?
The actual system itself has race conditions:. What if that 15000th file was simultaneously created in the USA and Europe? There is no way to prevent that without a cross-ocean latency penalty. Do you want to pay that penalty for every document creation? How do you deal with a net-split where cross ocean network traffic is delayed?
Finally, how will you monitor it? Will you have logging and alerting for users hitting the limit? Will there be an emergency override for engineers to remove the limit if necessary?
At big-web-service scale, simple engineering problems become complex problems fast...
So what? They might end up nuking some accounts, when some symptom pops up. And there'd be no recourse, whether or not it was a false positive.
Back when the commercial internet was just getting its act together there were companies that would give you free online access on Windows 3.1 machines in exchange for displaying ads in the e-mail client. (I think one was called Juno.)
The hitch was that you could only use e-mail. No web surfing. No downloading files. No fun stuff.
But that's OK, since there were usenet- and FTP-to-email gateways that you could ping and would happily return lists of files and messages. And if you sent another mail would happily send you base64-encoded versions of those binaries that you could decode on your machine.
The free e-mail service became slow-motion file sharing. But that was OK because you'd set it up before you went to bed and it would run overnight.
Thank you, whomever came up with base64.
The original Juno ad server proxied the ads from the internet to the email client, and the proxy was wide open for several months. The first time I ever accessed the open internet at home was by dialing into the email service and bouncing through the proxy. I believe it was closed due to it being shared in the letters section of a hacker zine.
I've no memeory of what motivated me to spend so much time just to be able to view some low-res, low-fps 3 second video clip, listen to 8-bit tracker "tunes" and try out some free application that invariably crashed the machine after a minute or two of use.
I do :)
It was the wonder of doing something for the first time. Of feeling like you were pioneering and learning and having fun all at the same time.
Truly, fun times :)
Being a teenager, the first web page I ever requested was www.doom.com, which returned a gibberish of text to Juno's email client. It was an HTML file full of IMG tags (one of those "Click here to enter" gateway pages), but I had no idea what I was looking at at the time. Somehow figured out to open the file in IE2 and saw... a bunch of broken images :)
I still vividly remember the sense of wonder that the early Internet evoked.
EDIT: Just checked the Wayback Machine. Looks like www.doom.com was not affiliated with the game at the time, so I must have browsed to www.idsoftware.com instead.
In my case, it was at the public library. The lone internet computer was constantly booked. But by watching over a library clerk's shoulder, I was able to see the password needed to unlock the text-based library catalog terminals (which terminals were plentiful and always available). (My parents worked at the library, or else I never could have pulled that off.) Once unlocked, I was able to use Lynx to telnet into my favorite MUD game. Unfortunately it didn't last long until a librarian caught me, which I think resulted in me being grounded from the library for a month or something like that.
And before that, the wonder of bulletin board systems. I got my first modem in 1985!
[1] https://en.wikipedia.org/wiki/Agora_(web_browser)
One big reason why I became a Linux user was that the TCP/IP stack for Win 3.1, Trumpet Winsock, was amazingly unstable and would regularly crash the entire OS. Linux had, even back then, a stable TCP/IP stack. And fantastic advancements like preemptive multitasking running in protected mode so errant user-space applications didn't crash the OS.
Good times.
[1] https://en.wikipedia.org/wiki/Mary_Ann_Horton
[1] http://blog.brian.jp/python/png/2016/07/07/file-fun-with-pyh...
Although if Picasa (predecessor to Google Photos) worked with BMP, it may be better to do that because it's much easier and more space efficient to encode arbitrary data in than PNG.
Now Google generously offers Drive with 15Gigs of space.
"Gdrive" (here: http://pramode.net/articles/lfy/fuse/pramode.html ) and the "Gmail Filesystem"/"GmailFS" (here: https://web.archive.org/web/20060424165737/http://richard.jo... as mentioned elsewhere in this thread) were both built on top of `libgmail` (here: http://libgmail.sourceforge.net/ ) a Python library I developed.
There were a couple of different projects at the time (listed in "Other Resources" on the project page) that sought to provide a programmatic Gmail interface.
I still have a "ftp" label in Gmail (checks notes 15 years later...) from the experimental FTP server I implemented as a libgmail example. :D
The libgmail project was probably the first project of mine which attracted significant attention including others basing their projects on it along with mentions in magazines and books which was pretty cool.
I think my favourite memory from the project was when Jon Udell wrote in a InfoWorld column ( http://jonudell.net/udell/2006-02-07-gathering-and-exchangin... ) that he considered libgmail "a third-party Gmail API that's so nicely done I consider it a work of art." It's a quality I continue to strive for in APIs/libraries I design these days. :)
(Heh, I'd forgotten he also said "I think Gmail should hire the libgmail team, make libgmail an officially supported API"--as the entirety of the "team" I appreciated the endorsement. :) )
The library saw sufficient use that it was also my first experience of trying to plot a path for maintainership transition in a Free/Libre/Open Source licensed project. I tried to strike a balance between a sense of responsibility to existing people using the project and trusting potential new maintainers enough to pass the project on to them. Looking back I felt I could've done a better job of the latter but, you know, learning experiences. :)
My experiences related to AJAX reverse engineering of Gmail (which was probably the first high profile AJAX-powered site) later led to reverse engineering of Google Maps when it was released and creating an unofficial Google Maps API before the official API was released: http://libgmail.sourceforge.net/googlemaps.html
But that's a whole other story... :)
AFAIK it mostly still works. The older "grive" might not.
On the one hand, I think this is great, on the other, I hope it doesn't force google to add limits that bother me in the future :P
I found myself in a similar situation a couple months ago. An android App falsely charged me on the Play store. After trying to contact Google for multiple weeks I gave up and disputed the charge on my credit card. This resulted in Google coming after me for 8.99$ and threatening me to close all my Google accounts including gmail, calendar, photos, drive and everything I rely daily in Google.
That was a wake-up call for me. I decided to move everything OUT of Google. That company got too much power, it should worry way more people.
Have you been successful in it? Any guidelines / tips? How hard is it?
[0] I use fastmail + custom domain, which works great, but you have to guard the domain very closely.
What do you mean by guarding the domain? To prevent large volumes of spam?
I'm intrigued by this, would you kindly share more on this!?!
Access to my email account probably gives you more access to my life and identity than my SSN [0].
I long for the day that we [1] all get assigned a public/private keypair instead of SSNs. That won't fix everything, but it's a huge step above a shared secret that is limited to 9 digits [2].
[0]: Even without signing up for a bunch of services, it's basically impossible at this point (at least in the US) to not have an email address associated with your bank account, car loan, mortgage, credit card, or even just watching TV.
[1]: "We" meaning "US citizens" or anyone else with a similar system.
[2]: I realize you also need info about the person and not just their number, but also apply that to keypairs.
What is the remedy for when someone loses or leaks their keypair?
Most of our ID cards (health, driving license) already have an expiration date and the subkeys should have one anyway.
- Use Yandex Maps instead of Google Maps. It's very accurate and navigation is smooth. Way better than Gmaps imo.
- Use Office 365 instead of Google Docs.
- You can use One Drive for cloud storage or buy a cheap VPS.
The only thing I haven't managed to find a even close to decent alternative it's photos. Google Photos is just simply too good. I would be even willing to pay but really, all the other apps struggle to get sync right or have some other crappy stuff that makes them barely usable.
It really is a suite that can combat Google's suite — and you can truly own it. Other than that, DDG for search, and your own domain for email (so that you could transfer it between different hostings if necessary).
I do have a Google account, but I use it for precisely two purposes: Google Play (my phone wouldn't work without one) and YouTube subscriptions (I can use an RSS reader for this, but it's a bit inconvenient). You can create a Google account without creating a Gmail account.
[1]: https://newpipe.schabi.org/
- Bought a new domain name and moved my mails in fastmail. I have been super happy with it so far.
- My Gmail address is now only for spam or very low importance emails.
- All my Pixel pictures are still uploaded to Google Photos, but I backup everything once a month or so.
- I don't use Google Drive for anything anymore. I have an Evernote account and a Dropbox account.
- Completely switchecd to DDG and Firefox.
- I'm still using my Pixel2 as of now but my next upgrade will be an IPhone, or a rooted Google-Independent Android phone.
You might not want to be tied to Microsoft but Google is not the only option.
Edit: Overlooked the comment about Apps. Microsoft offers apps for mobile, but not Linux. Although even on Windows I use the browser to access the services which will work on Linux.
What's needed is a syndication of data, and inter-operable apps. Like how xmpp worked. But of course, all vendors don't like this, because it turns themselves into a commodity.
However, I was scared for my Google account so just ended up dropping it. Ridiculous.
I'm curious to know how this happened. Would you mind sharing more info?
As I understand it, the only way for an app to 'charge you on the play store' is to:
1) Be a paid app (in which case you pay before the app starts installing), or
2) via in-app purchases, which are handled by the app initiating the IAP, and then Play services taking over to ask for confirmation.
In either case, the transaction is only confirmed by a user action (tapping a button) with the app having no control.
Sure, it's possible for an Android app to trick you, by covering everything apart from the button with something fake, but I'd be surprised if such an app found its way into the Play Store.
After using the app for a couple of days and restarting the phone, the app seemed to hit a bug and behave like if I didn't buy the subscription, prompting me to buy another subscription which I did thinking that this would unblock the backend and somehow merge with the fact that I already had a subscription.
Unfortunately, Google Play charged me again for a subscription I already had. Both the app creator AND Google Play were difficult to join. The App creators never replied to any of my emails. Google Play got an automated support website that decided that "I was not eligible for a refund" and there was nothing I could do about it. It also seems to be impossible to contact a real human being to explain the situation.
This may happen only if you manage to get it to the front page of HN or have many Twitter followers. In most cases you don't stand much chances though.
TBH this is not unlike reporting a security bug to a company as a white hat, but more like a grey hat here.
If the few blokes using this scam their way into few hundred terabytes of free storage, so be it, it's not worth the hassle for Google, imo.
edit: Apparently an account can create up to 250 docs a day https://developers.google.com/apps-script/guides/services/qu...
This. They probably thought of this exact scenario before adding unlimited docs. They probably even expected somebody to make a script for it. Hell, a few of them might even have a script.
As long as a lot of people don't start abusing it or make a file-sharing service based on it, then they probably won't care. Basically, not until it's a significant enough threat to their bottom line.
Ultimately, it's no different than the inevitable person that just has a script to generate garbage and upload it to Google Docs as fast as possible. That's what the 250 docs a day limit is there for.
The README.md says that UDS can store ~710kb per doc.
:shrug:
https://rehmann.co/blog/10-gb-27-kb-gzip-file-present-http-s...
The whole thing only transferred a few kB. It looked like an entire disc though.