HN Explorer

Ask HN: Anybody using Amazon Machine Image for AWS Deep Learning?

68 points by DrNuke ↗ HN
Hello, I do not own GPU hardware and am trying to understand if public amis on AWS are worth the fuss at professional level or at least good enough for the likes of course.fast.ai moocs. Newest here https://aws.amazon.com/marketplace/pp/B06VSPXKDX for Ubuntu. Any experience? Thanks!

42 comments

[ 3.0 ms ] story [ 96.7 ms ] thread
lekker
GPU hardware on Amazon is quite pricey. I would be careful since it can get out of hand rather quickly.
benmanns
I don't have experience with the Amazon AMIs for deep learning, but I'll give the tip that you can use spot instances with limited lifespans (x instance at $y price for at most z hours) to reduce cost and the risk you forget to shut something down.
Fripplebubby
Doing spot requests is a great way to keep the costs down if you just want to rent a GPU for a while. But bear in mind that GPU instance types tend to spike in price all-at-once when somebody brings a huge compute job online, which will exceed your spot request's max bid with no warning and immediately terminate your instance. So make sure whatever you're doing can be interrupted!
bhouston
GPUs in the cloud right now are so pricy, in part I think because NVIDIA is making people put in K40s and K80 - the Quadro and Telsa class cards that are super expensive.

We started to put on premise machines that have GPUs in them to reduce our costs. We can get a desktop class Core i7 machine with a GeForce 1060 6GB for around $800 all in, where as on Amazon you have to have a G2 2.xlarge on demand it is $400/month, or $3478 one year term (that is for 1/4 of a G2 8.xlarge achine, thus you get one of its 4 Quadro K5000 4GB cards to yourself.) I could put 2 1060s in that machine for another $200 or so. This just beats AWS pricing in <4 months of use.

My expectation is that these 1060s will be good for two years, maybe even three.

Because our GPU tasks are low bandwidth and non-critical we can run them on our existing office internet. Our rent also includes power. So there are not too many additional costs over the bare machine costs.

mabbo
I think you're definitely winning on price- provided you plan to run things continuously for that time period. The AWS GPU instances are probably not aimed at you though.

It's far more likely to be aimed at the Master's student in ML who has this new awesome model but it takes goddamn forever to train on his laptop and he just needs 3 days of time with a big-ass GPU instance.

He spins up the instances, runs his model, spins it down and is out what, $100 maybe? That fits into his limited budget.

Additional benefit: if I buy a kickass GPU, I'm going to reduce my overall productivity in research because now I can finally play some of these hot new games on full graphics!

tmsldd
Hopefully everything works fine for the student at the first run ;)
bhouston
> It's far more likely to be aimed at the Master's student in ML

I am pretty sure AWS isn't targeting its services at Master's students.

chrisfosterelli
> It's far more likely to be aimed at the Master's student in ML

As a counter example, I'm a master's student in ML and my lab wants to push more for an onsite machine since it's easier to expense one machine than a bunch of monthly AWS bills for each student in the lab :)

Additionally, most Master's students (at least in Canada) already have access to one or more very large compute clusters for jobs like this. I have used AWS a few times when I was short on time though!

Baeocystin
No. CUDA-capable hardware is dirt cheap, and once you have your own, you can run experiments 24/7.
chaosmail
I used the ami with a p2.large for a deep learning project at university. In my opinion it was quite pricey and reading from disk storage was disappointingly slow (we had a 60gb dataset). What i found really great was using the same machine first as a t2.micro instance for setup and testing s3 connection, a m4.large for some preprocessing and finally the p2 for deep learning, all on the same AMI and SSD.
uberneo
If you are just getting started then https://www.floydhub.com/ is a good place as they provide initial 100 mins on GPU for free and then later on approx 50% cheaper than AWS
gm-conspiracy
So, what is a recommended, buildable GPU rig (bang for buck)?

i7 w/ dual 1060 GPUs?

lloyd-christmas
my year and a half old rig is 2x 980ti and a 5930k, which is also my generic home computer. I only use AWS when it's a long running job and the weather is hot. It can really heat up your home on long running jobs, so keep that in mind.
StavrosK
I've been wondering: Is information processing "free" in terms of energy? I.e. if I put 100 watts in my GPU to make it do computations, will it heat my room for 100 watts' worth of heat?

I'm guessing yes, because where else is the energy going to go, but then we should probably be using heaters that are made of old/cheap GPUs, rather than just a resistor.

ivan_ah
As a Canadian, I always feel good running compute-intensive jobs in the winter. It's heating as usual, with a side-benefit of computation.
Neliquat
I have a system to duct in, out or recurculate inside air as the seasons change. I can heat my home with 'waste' energy at idle, or switch to cool outside air for heavy jobs or late night gaming. In summer I exhaust outside, pulling cooler air from underground. It is slowly spawning its own system, only 1.5 years in.
graphene
Yes, according to the first law of Thermodynamics, energy is conserved; regardless of whether there is computation being done, 100W of electric power will result in 100W of heat being dissipated.

About heating with computers, There is a Dutch company doing this, https://cloud.nerdalize.com/. It's interesting to think about how their economics work, because the heaters will be switched off for large parts of the year, and for much of the day. They must be banking that computation won't get much cheaper over time (in terms of FLOPS/$) because otherwise it'd be hard to recoup their initial investment. In a sense, this almost looks like a bet against Moore's law!

bckygldstn
Yup, all that energy ends up as heat in the room, just like a resistive heater.

However a heat pump can put even 400W of heat in a room for every 100W of electricity. That would be the benchmark for a lot of people in cold climates.

StavrosK
> a heat pump can put even 400W of heat in a room for every 100W of electricity

You mean it will transfer heat from the outside of the house to the inside? Over here we have the more traditional halogen space heaters (or central heating, with oil).

bckygldstn
Yeah exactly, 100W is the thermodynamic "penalty" you pay to move that heat from outside to inside.

Space heaters have the same thermal efficiency as a GPU or a toaster (100%), though could still be an improvement if they heat more evenly or heat skin directly via radiation.

projproj
I just built one partly for this purpose. I went with AMD's new Ryzen 1700. It has 8 cores and 16 threads. More threads are great if your application can use them. My understanding is that TensorFlow, for example, does[1]. i7s do better in applications that only use a single thread, but you get a really good cost/performance ratio for multi-threaded work from Ryzen [2]. The 1700 also comes with a serviceable air cooler, so you save some money there too.

The other components are a geforce 1060, Asus Prime X370-pro (lowest-priced x370 motherboard), and 16gb ram.

[1] https://stackoverflow.com/questions/39395198/configuring-ten... [2] https://www.phoronix.com/scan.php?page=article&item=amd-ryze...

edit: added other specs

Fripplebubby
I've been using this AMI for CNTK since the AMI was made available recently. It's been working well for me - they seem to update the AMI for big tagged releases of the libraries, and all the CUDA drivers are good to go out of the box. Personally I found it easier to use than Docker images, and both are easier than manual install.

With a spot request you can get a p2.xlarge for around $0.22/hr, which for me is sufficient for moocs and side projects, but for beefier instances the prices get high.

farhanhubble
I'm using p2.xlarge for Udacity Self Driving card nanodegree and Fast.AI assignments. It's super pricey as everyone has mentioned but I prefer using AWS to owning my own rig because I can download large datasets quickly.
minimaxir
A helpful on-topic note: Amazon has 1x K80 cloud GPU instances for $0.90/hr (not prorated), but very recently, Google announced GPU instances of the same caliber (https://cloud.google.com/compute/docs/gpus/add-gpus) for ~$0.75/hr (prorated to the minute), although Google Compute Engine requires configuring the GPU drivers first.
rocketcity
I have been using it a good bit for some of the deep learning assignments on Siraj Raval's channel (https://www.youtube.com/channel/UCWN3xxRkmTPmbKwht9FuE5A). I have had a really good experience so far. The trick is to find the AMI id for whichever region you want to use the image in and then just put in a spot request for your instance. I just checked the current going rate for a P2.XL in us-east-1 and it was at 0.2355 per hour.
ShirsenduK
Siraj is _the_ bomb! Fellow wizard here :)
ap46
Just get the GTX 1060 6GB variant & that should be enough RAM to run most models without suffering pagination.
KaiserPro
I'm using them at $work. But they are expensive, but so is EC2.

Unless you are able to get a decent deal (far better than reserved instances) then for raw CPU/GPU power hosting your own is better. Its the whole package that makes AWS attractive. (RDS, S3, instant spinup)

Spot prices is a good place to start if you are doing batches. just make sure you checkpoint little and often.

However if you are on your own, with limited funds, Get a second hand workstation (dell or hp z620/600) they tend to have decent PSUs, loads of ram and decent support, and a couple of Geforces with more Vram. (Quadros you are paying for the double float performance and ram. transfer Bandwidth and speed are the same or slightly better on geforce class cards.)

hackpert
Yeah. I mostly end up using p2.xlarge spot instances, and they have worked fine for my jobs so far. Be sure to checkpoint often and you'll be okay with spot instances.
chrischen
A pascal titan x is the best card for deep learning.

Even my gtx 1080 is faster, albeit with slightly less ram than the K80 or M60s available in the cloud.

A titan X would fix the ram issue though.

The most economical would probably be to colocate some titan X boxes. Not only cheaper, but more percormant than anything you can get from IaaS right now.

jph00
To answer your question about the http://course.fast.ai MOOC: the course provides an AMI that has everything you need pre-installed. Just follow the directions in the setup video, which includes a script that sets up the instance for you.

To save around 80% (but slightly less convenient) there are two ways to use spot instances provided on http://forums.fast.ai that are largely automated and work well.

In part 2 of the course you'll learn how to set up your own server. As others have mentioned that's much better value if you're using deep learning somewhat regularly.

(I'm the founder/teacher of this course.)

shoshin23
I can totally vouch for the ease with which you can setup your deep learning AMI. I followed the instructions on the site and the scripts provided by Jeremy abstract a lot of the stuff away. Super cool work!
kshnell
www.paperspace.com/ml has much more affordable GPUs than AWS and a new Pascal card. There's a pre-configured Deep Learning template and a couple interesting features like a browser based terminal and desktop (you can still SSH if you prefer).
dkobran
Paperspace co-founder here -- happy to discuss our AWS alternative which is focused on affordability, simplicity and performance. Recently, we've had a huge influx of people in moocs offloading their work to our GPUs/cloud. Would love any feedback you have.
[deleted]
(comment deleted)
coffeepants
https://paperspace.com is great. Takes about 2 minutes to go from no account to running a VM with all of the ML frameworks and drivers installed. The K80 on AWS/GCP is three years old and is just not built for Deep Learning. Paperspace has newer, more modern GPUs and I heard they are adding more cards soon (forget where I saw that).
eshvk
> is just not built for Deep Learning

What does a "built for Deep Learning" GPU do that the K80 doesn't do?

coffeepants
Sorry, should have elaborated on that a bit. The K80 is Kepler architecture (from 2012) and underperforms Pascal on most Deep Learning benchmarks. Here's a comparison of several NVIDIA cards and you'll see the K80 at the bottom: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learnin... This article emphasizes consumer cards which means you're building a box ie you lose all the benefits of offloading your work to the cloud. That said, there are cards that are better suited for DL that can run in a datacenter. It's not mentioned here but Pascal includes new GPU instructions designed to accelerate DL operations -- CuDNN is already taking advantage of these new capabilities. Perhaps, more importantly than the K80 card, is the P2 instance type itself, which includes significantly more RAM than you would need for DL (hence the high cost of the instance). In other words, the instance was evidently tuned for general GPU compute tasks like graphics-intensive applications/video encoding and not DL.
dhruvp
Hey OP! I've been using it extensively and it's definitely more than good enough for most tasks you'll be doing. If you're going this route, I'd highly recommend following the Keras' authors guide on how to set up the instance and connect to it with a jupyter notebook: https://blog.keras.io/running-jupyter-notebooks-on-gpu-on-aw.... This will boost your productivity a ton in the long run.

Best of luck!

[dead] nanistheonlyist
I have been using vagrant to spin up boxes on various cloud providers. The important thing is to remember to turn them off when you are done! I guess it would be possible to create a script to spin up boxes for each single job but I have not figured that out yet.