I really wanted to run the deep dream and deep dream animation scripts people have created. I am now re-sharing the instructions I found (with links to original guides). It isn't an automated install. You pretty much have to paste the lines one by one. But I did replace any "edit" steps with append or patch.
To do this on a fresh Ubuntu EC2 g instance there are a lot of steps- but I have tested them and put them all in one place (with links to the original sources and guides). I have CUDA up but not CUDNN as I haven't found how to legitimately download CUDNN without registering on the NVIDIA website.
Again: credit to the actual creators and all the original guide authors.
For better or for worse, with shell, the line between "a list of instructions to be typed in" and "shell script" is very, very thin.
Toss a
#!/usr/bin/bash
set -e
on top of that, call it "setup.sh", and give it a runthrough, and you're halfway to it being as "automated" as it needs to be to start with. You're fall more likely to get people start PR'ing a running shell script than dead text files.
(set -e will make the script stop when something returns with an error exit code. Despite temptation, given the size of this script I do advise doing your best to work with that... this is just the sort of script that if you let it blunder ahead blindly, when something doesn't work you'll need an hour just to figure out what didn't work, let alone fix it....)
Some of the steps require user input. Is the "set -e" for dealing with that? Also at least one reboot is needed, the script depends on a lot of external sources being up, and I have seen different behavior on the kernel upgrade steps.
No, see edit. But there are ways to do user input in shell, see Google. (By which I mean I literally do not personally know the best answers and that's what I'd be doing anyhow if I tried to explain here.)
Thanks for updating and organizing all of this. I got a working EC2 instance with CUDA and CUDNN working yesterday but it took some time. Having all of this is one place is excellent.
Yeah, I don't normally "do my own infrastructure" (I know, "it shows"). Usually the client or vendor has supplied the build. And I didn't want to commit to a package manager early in the exercise (in case it turned out to be incompatible with something one of the dependent projects did). So I just tried to start from the Caffe instructions and then fix things that broke.
The main components turned out to be: NVIDIA/CUDA, python, Caffe, ffmpeg.
Would you be willing to publish an AMI of the completed image? Seems like that'd make it really easy to run without all of the package management work.
Yes, I have a running instance up now in USWest. If it is free and somebody could shoot me instructions I'll make it an AMI (after lunch, on the way out the door right now).
Actually I started looking into it. I don't think I am going to make the AMI.
First I am not sure if the NVIDIA driver licenses do or do not allow this. Second I would have to tear down my current run and pay S3 charges to distribute the AMI. Third the instructions are long and ugly ( http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-... ) even after seeing the setup steps. Finally as people have mentioned in other comments there is a good to go Docker image for people to also play with.
For what it's worth, I've had luck just using the AWS EC2 web console, and choosing "Create Image" and "No reboot" from the instance's context menu. That spits out an EBS snapshot and an EBS-backed AMI based on the snapshot, which is probably what you were after. I went through the instance store image creation steps a long time ago for a project that deployed on custom AMIs and it was pretty convoluted; we ended up going EBS-backed and skipping the instance store altogether.
These steps are cool and very detailed, also check out https://github.com/VISIONAI/clouddream which is containerized so you can try it on local machine as well as on EC2.
That is definitely the more convenient option for running the defaults, but I think this is for a different usecase. If someone wanted faster results and full access to edit the python code and caffe models, this would be the better option.
At $2.60 an hour you'd think the Python code would be smart enough to expand "~" -- and call "avconv" instead of "avconf" -- or use a recent version of Ubuntu that avoids the ffmpeg vs avconv drama.
You'd also think people rendering movies would input and output PNG instead of JPEGs at 75% quality.
But you'd be wrong.
I appreciate the excitement around this but the people on the fringe hacking this shit together really should be ashamed of themselves. The core is so great! The script kiddie stuff wrapping it is SO BAD! This is perhaps the 15th attempt I've seen at it. Maybe the dumbest one yet.
We have Docker. We have AMI. We have scripts. And... we have people spinning up servers at $2.60/hr where you have to wait an indeterminate amount of time for some marketing intern to enable your NVIDIA developer application.
Google anything related to this technology and you'll immediately see a ton of people having the same three problems over and over. All due to sloppy packaging by people who basically Googlebombed keywords with their half-baked github experiments.
Slapping some half-baked shit onto Github isn't open source. It's littering.
And everyone knows "here's how to install!" walkthroughs never work for more than a week. STOP DOING IT.
$2.60/hr is way overpaying. For this task people should definitely be using spot instances, which are $0.32/hr. Big difference! It seems to me like they should also switch from g2.8xlarge to g2.2xlarge because caffe does not benefit much from multiple GPUs. That brings the cost to $0.07/hr; more than an order of magnitude difference.
Actually multiple instances of CAFFE rendering these models is working well for me. I'm paralleling 16 processes across 8 GPUs and everything fits. (Training might not work as well.)
Spot instances are great until the random hedgefund peak sneaks in and deletes all your work.
Your guidance for people just noodling around is right on the mark, though! The 2x machines are fine for playing and, once you get the AMI set up (hint hint) it's a two minute upgrade.
> Slapping some half-baked shit onto Github isn't open source. It's littering.
What a negative attitude...
Every programmer has written plenty of shitty code.
I know that I wrote lots of garbage code when I was a teenager just learning how to code. Unfortunately that was long before Github (or even Git) was a thing, and most of that code is either lost forever, or inaccessible on old floppy disks, etc. But today I would love it if my shitty Basic A text based games, and C code full of memory leaks still existed somewhere like Github, where I could look back at it and see how far I've come.
Projects like this one aren't intended to be open source masterpieces. They are fun learning projects by people who enjoy building stuff. If you don't like it then don't pay it any attention and build your own that is better. At least this guy put something together and documented the experience so someone else could follow up and improve on it if they wish.
Dunno. When I was a teenager I was shipping large bug-free apps. In things I was good at. This dude doesn't know how to fork, create a script OR use environment variables yet here he is on the front page of HN. That's litter.
Congratulations, you were some sort of child prodigy who could code perfectly from day one. Or maybe you just started earlier.
What exactly are you trying to achieve by hating on someone who's done nothing but share what they've learnt with other people. He isn't making you run this. He didn't even make you read it.
And nobody's making you read my comments either. So why is your meta-criticism valid but mine somehow not?
Look, I'm not trying to be a killjoy here. But there were no fewer than twenty versions of this on Github already. I know it, and I did have to read it, because I set this shit up and a good portion of my time was wasted after reading "guide" after "guide" from people who really shouldn't be allowed anywhere near a computer, let alone a publishing platform.
Once you took more than a half step away from the real caffe docs (which are totally fine, incidentally) every install guide was idiotic and broken. Every Python script was total bullshit. Publishing things that are hardcoded to your username in the home directory? Really? 29 public commits of an 80 line script and you still haven't sorted that out?
I submitted pull requests for the path bugs, the jpeg save at 75% idiocy, people spelling "avconf" instead of "avconv." But no0bs were copying the stuff around (not forking) faster than I could keep up. And nobody was accepting patches anyway. They just dumped the shit on Github and abandoned it.
Dump enough crap on a site and it disappears -- whether its Geocites or Google Code. Don't treat these amazing resources like your personal scrapbook. Show a little self respect and respect the time of others, too. That's the essence of being a professional.
I agree that two comments upthread is inappropriately personal.
Several people downvoted all my comments in bulk, including the one below that's a purely technical explanation of using multi CPU/GPU caffee. I personally do not care about points (moreover the same comment in another thread more than offset the 'karmic punishment' in this one) but since you're here, please have a look at the way some people are treating the downvote button.
I will keep things up a level and more broadly constructive in the future.
I didn't write any of the Python code you are complaining about. The install steps do use environment variables (and in fact set some in the bashrc). The install steps are not a script for reasons I gave in other comments in this thread (reboots, remote dependencies). You mention elsewhere that the effect includes JPEG artifacts if you use JPEG as your intermediate state. That is unfortunate, so I have added a links to other Python scripts for making movies so more variation is available.
I started this because none of the instructions I could find actually worked or had prominent enough links into the original BVLC source and Wiki. And why I did not submit to somebody else's Github or Wiki- mostly so I wouldn't have to beg to have pull accepted or a Wiki edit not reverted.
I apologize if you felt I was picking on you personally however I was in the same position as you. Nothing out there works. Now including your recipe.
Meanwhile the top GitHub issue for your referenced Python project (also on the HN home page is):
"Unable to run script with default arguments, get an invalid syntax exception."
I poked in to see if he'd managed to rename "avconf" to "avconv" yet.
Right below:
print("SHITTTTTTTTTTTTTT You're running CPU man =D")
I found this:
# this line is not tested cuz i don't have avconv :(
It's just all so gloriously half-assed. Sorry if you got caught in the cross-fire but it's hard to soar with the eagles when you're hanging out with turkeys.
32 comments
[ 4.0 ms ] story [ 66.0 ms ] threadTo do this on a fresh Ubuntu EC2 g instance there are a lot of steps- but I have tested them and put them all in one place (with links to the original sources and guides). I have CUDA up but not CUDNN as I haven't found how to legitimately download CUDNN without registering on the NVIDIA website.
Again: credit to the actual creators and all the original guide authors.
Toss a
on top of that, call it "setup.sh", and give it a runthrough, and you're halfway to it being as "automated" as it needs to be to start with. You're fall more likely to get people start PR'ing a running shell script than dead text files.(set -e will make the script stop when something returns with an error exit code. Despite temptation, given the size of this script I do advise doing your best to work with that... this is just the sort of script that if you let it blunder ahead blindly, when something doesn't work you'll need an hour just to figure out what didn't work, let alone fix it....)
The main components turned out to be: NVIDIA/CUDA, python, Caffe, ffmpeg.
I was very surprised how many steps ffmpeg took.
First I am not sure if the NVIDIA driver licenses do or do not allow this. Second I would have to tear down my current run and pay S3 charges to distribute the AMI. Third the instructions are long and ugly ( http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-... ) even after seeing the setup steps. Finally as people have mentioned in other comments there is a good to go Docker image for people to also play with.
Sorry about that.
For what it's worth, I've had luck just using the AWS EC2 web console, and choosing "Create Image" and "No reboot" from the instance's context menu. That spits out an EBS snapshot and an EBS-backed AMI based on the snapshot, which is probably what you were after. I went through the instance store image creation steps a long time ago for a project that deployed on custom AMIs and it was pretty convoluted; we ended up going EBS-backed and skipping the instance store altogether.
You'd also think people rendering movies would input and output PNG instead of JPEGs at 75% quality.
But you'd be wrong.
I appreciate the excitement around this but the people on the fringe hacking this shit together really should be ashamed of themselves. The core is so great! The script kiddie stuff wrapping it is SO BAD! This is perhaps the 15th attempt I've seen at it. Maybe the dumbest one yet.
We have Docker. We have AMI. We have scripts. And... we have people spinning up servers at $2.60/hr where you have to wait an indeterminate amount of time for some marketing intern to enable your NVIDIA developer application.
Google anything related to this technology and you'll immediately see a ton of people having the same three problems over and over. All due to sloppy packaging by people who basically Googlebombed keywords with their half-baked github experiments.
Slapping some half-baked shit onto Github isn't open source. It's littering.
And everyone knows "here's how to install!" walkthroughs never work for more than a week. STOP DOING IT.
Spot instances are great until the random hedgefund peak sneaks in and deletes all your work.
Your guidance for people just noodling around is right on the mark, though! The 2x machines are fine for playing and, once you get the AMI set up (hint hint) it's a two minute upgrade.
What a negative attitude...
Every programmer has written plenty of shitty code.
I know that I wrote lots of garbage code when I was a teenager just learning how to code. Unfortunately that was long before Github (or even Git) was a thing, and most of that code is either lost forever, or inaccessible on old floppy disks, etc. But today I would love it if my shitty Basic A text based games, and C code full of memory leaks still existed somewhere like Github, where I could look back at it and see how far I've come.
Projects like this one aren't intended to be open source masterpieces. They are fun learning projects by people who enjoy building stuff. If you don't like it then don't pay it any attention and build your own that is better. At least this guy put something together and documented the experience so someone else could follow up and improve on it if they wish.
What exactly are you trying to achieve by hating on someone who's done nothing but share what they've learnt with other people. He isn't making you run this. He didn't even make you read it.
Look, I'm not trying to be a killjoy here. But there were no fewer than twenty versions of this on Github already. I know it, and I did have to read it, because I set this shit up and a good portion of my time was wasted after reading "guide" after "guide" from people who really shouldn't be allowed anywhere near a computer, let alone a publishing platform.
Once you took more than a half step away from the real caffe docs (which are totally fine, incidentally) every install guide was idiotic and broken. Every Python script was total bullshit. Publishing things that are hardcoded to your username in the home directory? Really? 29 public commits of an 80 line script and you still haven't sorted that out?
I submitted pull requests for the path bugs, the jpeg save at 75% idiocy, people spelling "avconf" instead of "avconv." But no0bs were copying the stuff around (not forking) faster than I could keep up. And nobody was accepting patches anyway. They just dumped the shit on Github and abandoned it.
Dump enough crap on a site and it disappears -- whether its Geocites or Google Code. Don't treat these amazing resources like your personal scrapbook. Show a little self respect and respect the time of others, too. That's the essence of being a professional.
Several people downvoted all my comments in bulk, including the one below that's a purely technical explanation of using multi CPU/GPU caffee. I personally do not care about points (moreover the same comment in another thread more than offset the 'karmic punishment' in this one) but since you're here, please have a look at the way some people are treating the downvote button.
I will keep things up a level and more broadly constructive in the future.
I started this because none of the instructions I could find actually worked or had prominent enough links into the original BVLC source and Wiki. And why I did not submit to somebody else's Github or Wiki- mostly so I wouldn't have to beg to have pull accepted or a Wiki edit not reverted.
Meanwhile the top GitHub issue for your referenced Python project (also on the HN home page is):
"Unable to run script with default arguments, get an invalid syntax exception."
I poked in to see if he'd managed to rename "avconf" to "avconv" yet.
Right below:
I found this: It's just all so gloriously half-assed. Sorry if you got caught in the cross-fire but it's hard to soar with the eagles when you're hanging out with turkeys.Alternatively, is there a way to specify I don't want to enhance on certain outputs, like "dogs" and "faces"?