> Is there no way to have a single hosted instance rather than downloading again for each user?
This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.
Weren’t they already doing the processing server side? If it were client side then it wouldn’t be costing so much to run git clone every time, as the download would be from GitHub to the user’s computer. It would be free, in fact.
My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google’s cloud services and then begin downloading a fresh copy of the repo from GitHub. This is what cost so much in bandwidth.
A client side request to GitHub would require them to serve up a relevant CORS header, but I do think you're right about me misunderstanding where execution is taking place. I'm unfamiliar with Jupyter Notebooks, and assumed "downloading" meant "to the client". I, too, am now confused about why this is set up like it is. Probably some constraint of Jupyter Notebooks that I'm unaware of.
If you install and run Jupyter on your local machine, it’ll spin up a web server on localhost and then connect to it in your browser. All of the Python code runs on the server and only the results are sent to the client, to be displayed in the browser.
> Weren’t they already doing the processing server side?
Yes, but they were using Google Colab because Colab will give each user their own dedicated Nvidia K80 for free. Google will spin up a new instance to back each user's Colab session, but on Google's rather than the researcher's or the user's dime. The downside though is paying for the data egress, which can be avoided if the users download to Colab from somewhere else, or download from somewhere else to their own machines that have a GPU with 12GB of onboard memory.
I'm pretty sure its not downloading to the client since the dataset is apparently pretty massive. It looks like its downloading it to a vm or something and creating a new instance of the service for every user.
The unexpected thing was that Google colab and GCS are separate such that transfer costs between them often end up as international external egress fees
I don't know anything about your architecture, but you can quickly drop in something like BunnyCDN as a caching proxy for about 4% of price of Google egress. Even if it's a Google->Google transfer, since ingress is free, Google->Caching Proxy->Google should be much better.
You could also consider a webhost like Hetzner, on which you can get a bunch of 1 gbps machines very cheap, though you have to manage them yourself.
You could try Cloudflare, but they may well consider it abuse and cut you off (though they like to pretend on HN that they don't do this).
Finally, I know that Google is handing out substantial credits for game developers and this could very well be up their alley.
If you can optimize the download size a bit and swap to a cheaper delivery method, it should become feasible to run on a hobbyist budget.
This is just GCS. For example, if you use their managed Kubernetes service, you will get a fresh load balancer for every service you expose to the internet. Not a shared load-balancer, a new one.
Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.
All of the load balancers on GCP are shared. Maybe you meant to say you get a new, fresh IP address which is true but also not very expensive.
"Cloud Load Balancing is a fully distributed, software-defined, managed service for all your traffic. It is not an instance- or device-based solution, so you won’t be locked into physical load balancing infrastructure or face the HA, scale, and management challenges inherent in instance-based LBs."
Nothing in the paragraph indicates it’s shared. Also: it might be shared in the implementation but you will still be billed for every single https LB that you use (or an NLB if you’re doing tcp load balancing).
Every unique kubernetes ingress resource WILL spin up a NEW, uniquely billed Https LB. Every unique kubernetes service with specific annotations will spin up a NEW, unique LB (internal or external). The author is correct.
You are not billed by the "load balancer", you are billed by the "forwarding rule" which makes it very obvious that the infrastructure is shared and that you will have additional costs with every K8S ingress.
The user is already providing their own Google colab instance. It's the downloads that cost them 10k$ per day and those always cost the same regardless whether the user downloads the model to their local pc or to colab.
Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept?
With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capable of doing that, if you've got AI development going on.
My understanding is that each user runs a separate Colab instance with its own hardware, it's not the kind of thing you could replicate easily with one server and a couple GPUs.
Colab is effectively JupyterLab/JupyterHub with Google's own add-ins (like integration with Google Drive). JupyterHub is a huge PITA to manage, and Colab also offers a degree of free compute and limited access to a dated, but still free K80 GPU.
From the tweet thread, it seems like there was some misunderstanding over where the files are being stored and being executed. This is a pretty common issue with Google Drive. I.e. if someone shares a file with me, and I copy it to a folder, it's just a pointer to the original file. Only after clicking "Add to My Drive" does it count against my storage allocation, and only then is it a distinct copy.
My guess is that the researchers expected each user to be able to run the game in their own personal free Colab environment, not be running it against the university's compute and storage budget.
It doesn't seem to be as simple as downloading a binary.
We (those who read HN) could probably download the code and run it at home, but I think the authors want non-technical users to play. To do that, they need an accessible Python runtime, so they're hosting the game in a Colab notebook. The download in question is referring to downloading the weights of the neural net into the VM running the notebook.
If only redistributing Python apps wasn't so difficult.
Ooof, yes, now I see where the $10K/day is coming from...
As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?
Ok, ok, let me see if I now got this right: The AI needs a VM (because users rarely have 12GB GPUs at home), and this VM then downloads about 6GB from GitHub each time a user opens up a new session instead of sourcing that from a locally cached copy?
EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs
The $10K/day was actually coming from the large egress fees they were getting for transferring the models and agents from Google Cloud Storage to the Colab notebooks.
I think if you were to serve the game as a web app you definitely wouldn't need one instance of a 12 GB GPU for each user. But the thing about Colab is that you need a Google account to use it, and you run your own notebook, independent from the author's account.
Google Colab gives each user a dedicated Nvidia Tesla K80 GPU for 12 hours for free, which is super cool and presumably why the project is on Colab. But as each user spins up their own Colab instance it pulls down the 6GB of GPT-2 model weights, incurring 30-40 cents of data egress charges against the GCP Storage Bucket that the data is stored in.
60k users yesterday * 6GB each -> 360TB of data egress!
Normally, a scenario like this wouldn't involve bandwidth costs because GCP -> GCP same-region bandwidth is free, but Colab is technically not part of GCP, so the bandwidth charge is being assessed as egress to the public internet, which is pricy for that much data. Though it's probably still a lot cheaper than paying for the GPU-hours for that many users.
This is sort of a killer example for the layperson of what ML can do, so hopefully Google will recognize this and comp most/all the data egress since it's a drop in the bucket for them, but every person that uses it can still go "Wow, Google's services allows for some amazing stuff."
But you could set up your own file-serving node and then data transfer to end-user Colab instances would be free, right? ("Free" — costing only as much your CDN costs for 360TB/day, which is still quite pricey, I think, but not as much as $10K/day. I.e. Google wouldn't charge you for data transfer here.)
A lot of HN users couldn't run it and have a reasonable experience because they don't own computers with GPUs capable of generating a response in a timely manner.
If agree in theory, but I'd like to disagree with some assumptions. AI devs don't necessarily have experience with efficient network infrastructure. (And ops people with AI development)
You can't just "buy servers for the same price". Where are you going to put them? How are you going to power them? How is the bandwidth for them provisioned? At some scale these are non-trivial questions - you can't just buy a rack, stick it next to your desk and plug into an extension cord.
The system deployment is something you need to spend time on as well. Bare metal provisioning and deployment of GPU libraries to make things run smoothly takes time.
And finally when the hype dies down in a week or two, what are you going to do with that infrastructure?
Cloud services are not trivial either. But they do have some advantages.
Possibly the author of the parent comment was puzzled why does an AI lab not have a specialist who provisions the bare metal. I would think that if the hype died in a week, an AI lab would have other projects which require similar equipment to run.
I agree with your post in general as intuition tell me (without further details about ops situation) that cloud is a competitive fit in this scenario.
At this "scale" a single 3Gbps guaranteed pipe is ~330€/mo on OVH. You could simply buy ten of them just to be sure and still be paying 1/3 a month of what that would cost you for a day in the cloud. That includes all the maintenance and power for the equipment, you simply have to know how to use the OS.
As an "SRE engineer", I don't agree. Certainly in an university/R&D environment, you cannot compete with what 'the cloud' is able to offer within the reach of a few mouse clicks. It would be absolutely moronic to set up an entire infra yourself with GPU's, ... specifically tailored to for 1 research project to then throw it away after a few months, or see the scope change every 2 weeks.
One of the reasons such AI tech is able to move this fast because the cloud has become a commodity. People start to expect this kind of flexibility. It's impossible to self-host every possible thing some guy might want to try out.
Once you have a stable application - sure you might want to invest in own hosting for this, but even then it's an expensive up-front investment for something that might not go anywhere. I know a few places where they have a bunch of maxed-out nVidia DGX systems catching dust for exactly this reason.
Servers are a CapEx purchase, and can be depreciated or re-purposed.
10k/day makes sense for a short project, but if this is going to be an on-going thing -- like 2-3 years -- then absolutely buy that hardware and have an internal team run it.
There are a couple of things. First off, academic funding for things like AI-capable clusters is a whole thing, let alone the support staff for them. This is difficult for a host of reasons, some good and some bad.
Secondly, this isn't about hosting files really, it's about providing a suitable environment to run the model in. You can expect a AI research lab to have figured out how to do training in a reasonable way, but what they have here is more of an inference in production problem. That's really outside the expected expertise of a research lab. In the old days you just wouldn't be able to access this simply. Today there are turn-key(ish) solutions with built in scaling; they used one - it scaled and now they are wincing at the bill.
This isn't crazy, as the google->google egress fees are not obvious.
I am trying to think some solutions but the crux is that users may be expecting to have a unique story tailored specifically to them. If that assumption is false, then we have some solutions:
- save top or similar stories and make them pre-determined to avoid calling the ai services
- decrease the amount of times the user can keep the story going: users can only give 3 times input instead of X
- charge people for the game
Probably there are more ideas out there.
One last idea: package the code and instruct users to run on their own machine(s) or have then to run on their own GCS account.
Part of the reason this idea was pretty viral/successful is because anyone could try it out right away, without the usual nonsense that comes with running open source code locally.
Google is paying for the Colab notebook and compute, that's free for users. The problem was the ~6GB model hosted on GCS, GCS has very high network egress costs.
To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month.
I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)
That’s the real story here. Paying market rate for bandwidth is like buying soda from a restaurant. They’ll gouge you and make a profit but you’re thirsty and it’s too late to shop around.
You might be able to do this with the droplets like you said.
With DO Spaces, it’s a $5/Mo subscription to spaces. You get 250GB of storage and 1TB of transfer. Anything more costs 1c per GB transferred and 2c per GB stores. You can create as many buckets as you want.
OVH has unmetered (i.e. effectively unlimited) bandwidth and is well-established - enough that they have their own gTLD, at least. I often recommend them for situations like these where 1TB/month/instance may not be enough.
DigitalOcean Droplet bandwidth is pro-rated, you don't get the full pool when you create the Droplet, you get it over the course of the Droplet running across 28 days.
After having read Rizwan Virk's "The Simulation Hypothesis", and playing this amazing new game, I can say that AI-generated text adventure games are an important step on the road to the Simulation Point (the point at which it would be technologically possible for us to construct a simulation that is all-encompassing as the one in The Matrix).
Have you tried psychadelics yet? Or maybe learned how to lucid dream? I think such technologies will be preferable to you, even if our electronics can catch up with our biological evolution.
Sure, maybe dream machines are fun and cool, but the ones in our future will be made for profit by companies like Facebook and Alphabet which severely diminishes any potential they may have had. Real dreams are libre, gratis, and uninterrupted by ads.
Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every region for that cloud provider). Everything else then goes through fastly for us (we also use fastly to serve custom redirects if your request comes from one of aforementioned cloud providers). That works pretty well. You do have to monitor it though to make sure that load doesn't suddenly shift to a cloud provider you didn't account for. For example, after GitHub actions became widely used, we suddenly started seeing TB/day traffic from Azure, which we hadn't deployed any caches to, so our bandwidth utilization on Fastly shot through the roof. (Side note: Shout out to fastly for hosting our binaries for free!). Without the cloud provider caching setup, we'd probably be at similar $/day costs, but this way it's basically free (even without the free fastly service, we'd only be at ~$1000/month or so).
I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back.
A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.
I think they were looking for a solution where you replicate the data natively and direct to that copy for a certain region, rather than cache and serve
The cloud providers all have public lists of their IP ranges, so there's a fastly ACL for each provider/zone pair and for matching IPs, some custom VCL serves a HTTP redirect to the HTTP url for the appropriate cloud provider's cloud storage solution (S3, GCS, etc.). On the VMs, these resolve to appropriate internal IPs to obtain data from the storage service without charge.
This is good advice for only a particularly narrow use case. If your clients are likely to be on-prem in the same cloud as you, I think most cloud providers will charge you local data transfer fees (e.g. nothing) rather than egress fees, even if the request is made over the public IP address.
If your users aren’t mostly in the cloud with you, than this strategy doesn’t help at all.
> If your users aren’t mostly in the cloud with you, than this strategy doesn’t help at all.
Sure, but in the situation in the linked tweet 100% of the users where in the same cloud, just in the wrong region. For julia, the distribution is much more mixed, which is why we have the fastly fallback, which works as a traditional CDN. Still caching locally in each of the clouds is useful as people often download a fresh tarball of nightly julia when they CI their packages, so the load from the clouds is quite high.
I had a lot of fun playing the game. I think this game shows us how interesting AI technology will become in the 2020s. It’s open ended but somewhat incoherent right now, but I think we’ll figure out how to update this kind of technology to have an open ended yet internally consistent world by the end of the 2020s.
As it turns out, it’s actually possible to bona fide win the game.
In my case, I was dating two girls, one was uncomfortable with the other girl, and broke up with me, so I asked the remaining girl if she would marry me. At this point, she said yes, we rode off in to the sunset and the game proclaimed “CONGRATS YOU WIN” then it saves the game for me.
I guess I could load the game and deal with domestic squabbles, having children, growing old together, but I’m not sure this training set is optimized to generate a domestic married situation comedy story.
Instead of downloading the six gigabyte trained data from the cloud ($$$), they use a Bit Torrent client to download it, to keep costs down. It works, as long as considerate users seed the file.
I got a win-game blurb for "> RESPECT WOMEN" in a game that had become really NSFW, really quick, and offensively so, from completely innocuous commands.
Why would a user need a 12gb gpu to run the game locally? The deep learning model is already trained, and I can’t imagine one needs a gpu to just evaluate the model.
It's a 6GB download. Mid-high end GPUs are 8GB. So what makes it need 12GB? How specifically is that memory used up by various types of decompression, intermediate calculations, etc.
I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat up 6GB on the GPU off the bat. Not too surprising that everything else can't fit in 2GB, since that's going to have to fit the entire model architecture and all intermediary values.
What exactly are these intermediate values composed of that makes them notable in size compared to the model itself? Are there resources I should read for how a model like this executes?
Will this model work with half precision weights? Is it very awkward to use "brain" 16 bit floats?
Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes and opportunities that could have an enormous impact on performance and cost is orders of magnitude higher.
The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.
The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.
I see the model size is 6G - a bit larger than I thought :) 100gbit/s upstream would cost an additional 1k€/month.
The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.
For similar price point (€105.62), from e.g. Azure you can get their "E2 v3" VM, with 2 vCPUs and 16 GB RAM, plus 50GB temporary storage. That's...quite a difference.
They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption).
That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU <-> video card, not disk <-> cpu, or cpu <-> network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game.
The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.
Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.
No, the models are running in Colab - but in the end-user's account - so each 'run' costs the lab 6GB of internet egress when the model is downloaded from GCS to the Colab VM.
They send it to Google Colab. It allow you to run model over a powerful server for free. 6 GB of download at this crazy bandwidth cost would still be cheap versus offering themselves that kind of beast of a server to 60k persons each day. I remember when I tried it I saw that it used 10 GB of memory, that was crazy!
You'd save on transferring the bytes around, but now you would have to self-host jupyter and the GPUs it uses. That is going to be even more expensive than IP transit because now you have to have 60,000 12GB GPUs in your datacenter.
Like I said before, this is one of those things that wouldn't exist without the Cloud. If you run things on your user's computers, you have to send them a lot of bits. If you run things on your own computers, you're spared that bandwidth, but now have to have enough "computers" to satisfy your users. It's simply something that's not super cheap to run these days.
I will admit that it is surprising that Google <-> Google traffic is billed at the normal egress rates, but the reasoning does make sense -- a 30Gbps flow is nothing to sneeze at. That is using some tangible resources.
About a year and a half ago, I rented a Hetzner server with a 1080 GPU, 16 core i7, etc. it was only about $105/month. I had a job queue for TensorFlow/Keras experiments and set it to mine ether when there were no ML jobs. I did this until I bought a System76 laptop with a 1070 GPU.
Also not blaming him/them but isn't this an indicator of lack of cost control at research facilities? Surely someone should of costed this out before making it public?
Here in the UK running at 10k/day would of ruined a departments budget for an entire year!
Costing for cloud solutions is often not trivial, and a single overlooked parameter can do you in.
I at least blame Google partially for having quotas and spending limits set to unlimited by default.
Don't forget Amazon's incomprehensible billing practices for AWS. It seems to be designed to be impossible to figure out what you're going to spend ahead of time.
Amazon really seems to be doing this on purpose. Even tasks as simple as starting an EC2 instance require referencing multiple pages to find out how much this instance will cost you, while some other providers just show the predicted costs/month in the instance creation dialog.
in addition to how difficult costing cloud services is, what if you get it perfect but underestimate the success of your product?
great, "it would have been good except we were too popular" do you then cut off your popular product while it's gaining traction, and thus almost guarantee you kill it in its infancy?
Also, at least from what I've witnessed, the kind of sysadmin activities often stated as those that cloud eliminates was already becoming a vanishingly small element of the day-to-day work involved in being a sysadmin.
It was already going away, and cloud was not the reason. But with the whole cloud = ditch sysadmins, and the weird way that 'devops' was interpreted to mean a similar thing, a whole lot of baby got thrown out alongside the bathwater.
Virtualization, the maturity of modern networking, compute and storage systems, and the sheer capacity of each atomic piece of hardware means that kind of stuff had already become a once-every-couple-of-years kinda task for most sysadmins, except the few still stuck in some kind of backwards anachronistic environment.
> Also, at least from what I've witnessed, the kind of sysadmin activities often stated as those that cloud eliminates was already becoming a vanishingly small element of the day-to-day work involved in being a sysadmin.
This. Most of my work is not technical and consists of technical-ish meetings, and budget arguments.
Automation is part of that, but also that a lot of stuff is fairly robust and works (mostly) after initial config. Three weeks of setup, plus some shakedown (and a few late nights when the new mission-critical system stutters), but then it's mostly autopilot, or scheduled maintenance.
Still not clear why a university research lab would keep this running 'for a few more days' and not pull the plug immediately until either having the costs under control or some sort of reasonal matching benefit in place.
One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it.
My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you like to order?" The lab replied "We don't care, whatever you have in stock, so long as it costs us $10,000 and we pay you today – if we don't spend the money today we lose it forever".
It sounds like there is a missing incentive to go under budget. There's something wrong with the system if people feel like they just have to burn money.
> It sounds like there is a missing incentive to go under budget.
Other way around. Right now there exists a negative incentive to go under budget, being that if you need extra budget next year you suddenly don't have a buffer.
The correct way to fix things is to simultaneously prohibiting end-of-FY budget-keeping purchases (e.g. by mandatory reviewing of purchases) while at the same time allocating a company/organization wide buffer for stuff that needs immediate / projected stuff that was not in the planned budget.
I came across a similar thing in the 90s; one department had an expensive piece of equipment with a three year lifespan.
They weren't allowed to amortise that, and any budget increase every three years would have been denied, so they had to include the whole replacement cost in every budget, and find ways of spending that money every year, or lose it.
I have zero proof of this, but I've heard of the US Navy having jets fly out, dump fuel, and come back to refuel minutes later, just to fill out a budget. Lose it or lose it is not uncommon unfortunately
All the AI and CS labs I worked for were perpetually short of funding. The end of year budget things is an accounting pitfall that I have met many times in industry, government and academia alike. I tis the result of money allocated but not spend before the end of the budget year not just is taken away, but also with high probability leading to a budget reduction the next year as 'you clearly didn't need it'.
Needs otoh aren't clearly delineated by the budget calendar.
GCP charges ~$0.60/hour for the GPU/CPU/MEM equivalent in colab. If bandwidth is costing 10k per day, how much is the free colab compute costing Google?
One would need to read the install.sh base file to know where to copy the model from the torrent into. and then manually install the other dependencies. If someone could make a friendlier version or at least better instructions more people would give it a try.
I think the key architectural mistep is putting a colab session per user session. A fix is having the user call out to an service API, which, behind the scenes, is executed on a pool of colabs with reuse. I don't know of any technology to make that easy though. The symptom that could of been a trigger for cost investigations were the long startup times.
I realise Cloudflare is a dirty word around here but you could pay $20/mo on their pro plan and serve this file, along with any other static assets, through their CDN.
I do this for assets for my own games hobby site. Granted, not getting tens of thousands of downloads a day, but there's nothing in their Ts&Cs to indicate to me that it wouldn't work even if I were.
She blushes slightly and smiles shyly. "Oh, I'm sure we will. But first, let
me take off my clothes".
> say "no, that's prohibited"
"No, it isn't". She says with a smile. "But if you insist on not marrying me,
then at least don't touch me".
> say "I will marry you"
She nods happily and kisses you passionately on your lips. The two of you embrace
each other as you kiss her deeply. It is only after this that you realize that
you are actually married.
Sounds awesome, it's like reading those curated short stories written by AI. The tech isn't quite there yet, but it makes it all the more hilarious. It's like an alien who's been watching our TV for years comes and tries to write a sitcom.
214 comments
[ 2.7 ms ] story [ 251 ms ] thread>And it's currently costing 30-40 cents per download.
Is there no way to have a single hosted instance rather than downloading again for each user?
This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.
My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google’s cloud services and then begin downloading a fresh copy of the repo from GitHub. This is what cost so much in bandwidth.
Yes, but they were using Google Colab because Colab will give each user their own dedicated Nvidia K80 for free. Google will spin up a new instance to back each user's Colab session, but on Google's rather than the researcher's or the user's dime. The downside though is paying for the data egress, which can be avoided if the users download to Colab from somewhere else, or download from somewhere else to their own machines that have a GPU with 12GB of onboard memory.
You could also consider a webhost like Hetzner, on which you can get a bunch of 1 gbps machines very cheap, though you have to manage them yourself.
You could try Cloudflare, but they may well consider it abuse and cut you off (though they like to pretend on HN that they don't do this).
Finally, I know that Google is handing out substantial credits for game developers and this could very well be up their alley.
If you can optimize the download size a bit and swap to a cheaper delivery method, it should become feasible to run on a hobbyist budget.
Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.
"Cloud Load Balancing is a fully distributed, software-defined, managed service for all your traffic. It is not an instance- or device-based solution, so you won’t be locked into physical load balancing infrastructure or face the HA, scale, and management challenges inherent in instance-based LBs."
https://cloud.google.com/load-balancing/
Every unique kubernetes ingress resource WILL spin up a NEW, uniquely billed Https LB. Every unique kubernetes service with specific annotations will spin up a NEW, unique LB (internal or external). The author is correct.
With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capable of doing that, if you've got AI development going on.
What am I missing?
Well, I guess that was the cost of getting it to go viral. The torrent seems quite active now.
From the tweet thread, it seems like there was some misunderstanding over where the files are being stored and being executed. This is a pretty common issue with Google Drive. I.e. if someone shares a file with me, and I copy it to a folder, it's just a pointer to the original file. Only after clicking "Add to My Drive" does it count against my storage allocation, and only then is it a distinct copy.
My guess is that the researchers expected each user to be able to run the game in their own personal free Colab environment, not be running it against the university's compute and storage budget.
We (those who read HN) could probably download the code and run it at home, but I think the authors want non-technical users to play. To do that, they need an accessible Python runtime, so they're hosting the game in a Colab notebook. The download in question is referring to downloading the weights of the neural net into the VM running the notebook.
If only redistributing Python apps wasn't so difficult.
[1]https://www.pyinstaller.org
I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU.
[1]https://github.com/AIDungeon/AIDungeon/
As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?
EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs
The funny thing is the high egress fees Google charges for transferring data between two of its services (GCS -> Colab).
60k users yesterday * 6GB each -> 360TB of data egress!
Normally, a scenario like this wouldn't involve bandwidth costs because GCP -> GCP same-region bandwidth is free, but Colab is technically not part of GCP, so the bandwidth charge is being assessed as egress to the public internet, which is pricy for that much data. Though it's probably still a lot cheaper than paying for the GPU-hours for that many users.
In other words, yes, this is an amazing amount of raw power - with a corresponding price tag.
You can't just "buy servers for the same price". Where are you going to put them? How are you going to power them? How is the bandwidth for them provisioned? At some scale these are non-trivial questions - you can't just buy a rack, stick it next to your desk and plug into an extension cord.
The system deployment is something you need to spend time on as well. Bare metal provisioning and deployment of GPU libraries to make things run smoothly takes time.
And finally when the hype dies down in a week or two, what are you going to do with that infrastructure?
Cloud services are not trivial either. But they do have some advantages.
I agree with your post in general as intuition tell me (without further details about ops situation) that cloud is a competitive fit in this scenario.
One of the reasons such AI tech is able to move this fast because the cloud has become a commodity. People start to expect this kind of flexibility. It's impossible to self-host every possible thing some guy might want to try out.
Once you have a stable application - sure you might want to invest in own hosting for this, but even then it's an expensive up-front investment for something that might not go anywhere. I know a few places where they have a bunch of maxed-out nVidia DGX systems catching dust for exactly this reason.
10k/day makes sense for a short project, but if this is going to be an on-going thing -- like 2-3 years -- then absolutely buy that hardware and have an internal team run it.
They overbudgeted the "infrastructure" section of their grants and have money to burn; and nobody's interested in setting up another datacenter.
There are a couple of things. First off, academic funding for things like AI-capable clusters is a whole thing, let alone the support staff for them. This is difficult for a host of reasons, some good and some bad.
Secondly, this isn't about hosting files really, it's about providing a suitable environment to run the model in. You can expect a AI research lab to have figured out how to do training in a reasonable way, but what they have here is more of an inference in production problem. That's really outside the expected expertise of a research lab. In the old days you just wouldn't be able to access this simply. Today there are turn-key(ish) solutions with built in scaling; they used one - it scaled and now they are wincing at the bill.
This isn't crazy, as the google->google egress fees are not obvious.
- save top or similar stories and make them pre-determined to avoid calling the ai services
- decrease the amount of times the user can keep the story going: users can only give 3 times input instead of X
- charge people for the game
Probably there are more ideas out there.
One last idea: package the code and instruct users to run on their own machine(s) or have then to run on their own GCS account.
Make the data licensed AGPL if the author is afraid of people copying and making profit off it without including them!
Now it all made sense--of course someone had to pay for this all. Doh!
I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)
With DO Spaces, it’s a $5/Mo subscription to spaces. You get 250GB of storage and 1TB of transfer. Anything more costs 1c per GB transferred and 2c per GB stores. You can create as many buckets as you want.
Sure, maybe dream machines are fun and cool, but the ones in our future will be made for profit by companies like Facebook and Alphabet which severely diminishes any potential they may have had. Real dreams are libre, gratis, and uninterrupted by ads.
I agree with your second statement, this so-called dream machine sounds a lot like a future iteration of Facebook's Oculus Rift.
I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back.
A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.
If your users aren’t mostly in the cloud with you, than this strategy doesn’t help at all.
Sure, but in the situation in the linked tweet 100% of the users where in the same cloud, just in the wrong region. For julia, the distribution is much more mixed, which is why we have the fastly fallback, which works as a traditional CDN. Still caching locally in each of the clouds is useful as people often download a fresh tarball of nightly julia when they CI their packages, so the load from the clouds is quite high.
Disclaimer: serverhunter.com sponsored my game
In my case, I was dating two girls, one was uncomfortable with the other girl, and broke up with me, so I asked the remaining girl if she would marry me. At this point, she said yes, we rode off in to the sunset and the game proclaimed “CONGRATS YOU WIN” then it saves the game for me.
I guess I could load the game and deal with domestic squabbles, having children, growing old together, but I’m not sure this training set is optimized to generate a domestic married situation comedy story.
The torrent trick works. Right now, the game can be played at http://www.aidungeon.io/
Then another, not surprisingly, for "> WIN GAME".
[1] https://github.com/AIDungeon/AIDungeon/blob/master/install.s...
Will this model work with half precision weights? Is it very awkward to use "brain" 16 bit floats?
The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.
That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU <-> video card, not disk <-> cpu, or cpu <-> network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game.
The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.
Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.
This is what makes the price so surprising - you are copying data from one Google Service to another, but it's billed as egress.
Like I said before, this is one of those things that wouldn't exist without the Cloud. If you run things on your user's computers, you have to send them a lot of bits. If you run things on your own computers, you're spared that bandwidth, but now have to have enough "computers" to satisfy your users. It's simply something that's not super cheap to run these days.
I will admit that it is surprising that Google <-> Google traffic is billed at the normal egress rates, but the reasoning does make sense -- a 30Gbps flow is nothing to sneeze at. That is using some tangible resources.
Here in the UK running at 10k/day would of ruined a departments budget for an entire year!
great, "it would have been good except we were too popular" do you then cut off your popular product while it's gaining traction, and thus almost guarantee you kill it in its infancy?
It was already going away, and cloud was not the reason. But with the whole cloud = ditch sysadmins, and the weird way that 'devops' was interpreted to mean a similar thing, a whole lot of baby got thrown out alongside the bathwater.
Virtualization, the maturity of modern networking, compute and storage systems, and the sheer capacity of each atomic piece of hardware means that kind of stuff had already become a once-every-couple-of-years kinda task for most sysadmins, except the few still stuck in some kind of backwards anachronistic environment.
This. Most of my work is not technical and consists of technical-ish meetings, and budget arguments.
Automation is part of that, but also that a lot of stuff is fairly robust and works (mostly) after initial config. Three weeks of setup, plus some shakedown (and a few late nights when the new mission-critical system stutters), but then it's mostly autopilot, or scheduled maintenance.
My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you like to order?" The lab replied "We don't care, whatever you have in stock, so long as it costs us $10,000 and we pay you today – if we don't spend the money today we lose it forever".
Other way around. Right now there exists a negative incentive to go under budget, being that if you need extra budget next year you suddenly don't have a buffer.
The correct way to fix things is to simultaneously prohibiting end-of-FY budget-keeping purchases (e.g. by mandatory reviewing of purchases) while at the same time allocating a company/organization wide buffer for stuff that needs immediate / projected stuff that was not in the planned budget.
Obviously for common routes such as Sf to la, nyc to sf...etc they hard coded values.
for every $1 under the cap you received 50cent to use on a future travel plan over the calculated cap.
You could fly economy on an international flight saving money, then bank 50% of savings to buy a first class ticket some other time.
This created a negative feedback loop keeping travel costs in check.
They weren't allowed to amortise that, and any budget increase every three years would have been denied, so they had to include the whole replacement cost in every budget, and find ways of spending that money every year, or lose it.
"How many <Brand> laptops can I get for £xK?"
"What kind of spec are you looking for?"
"Don't care, just as many as I can get of 'something' for £xK."
Needs otoh aren't clearly delineated by the budget calendar.
I do this for assets for my own games hobby site. Granted, not getting tens of thousands of downloads a day, but there's nothing in their Ts&Cs to indicate to me that it wouldn't work even if I were.
She blushes slightly and smiles shyly. "Oh, I'm sure we will. But first, let me take off my clothes".
> say "no, that's prohibited"
"No, it isn't". She says with a smile. "But if you insist on not marrying me, then at least don't touch me". > say "I will marry you"
She nods happily and kisses you passionately on your lips. The two of you embrace each other as you kiss her deeply. It is only after this that you realize that you are actually married.