HN Explorer

Ask HN: I just got $100k in AWS credits, how should I use it?

109 points by chrischen ↗ HN
I got $100k in AWS Credits, with a 1-year time limit. I built a scrapy-powered image crawler that crawls over 300 art sites and finds the most popular posts with clustering algorithms and perceptual hashing (www.arthunted.com), but in the end it takes at most a few hours of a high CPU-instance per day to scrape and process (at most several dollars per day). At a year it'd barely make a dent.

I'm looking to build something that would make a splash, that would otherwise be constrained by budget, and that would have long-term self-sustaining value after the $100k runs out.

So no arbitrage, reselling, bitcoin mining, etc.

What type of project would require high-storage or high-amounts of processing? What can I build that would only be possible with that much money in infrastructure or compute power? Preferably the monthly budget would be about $10-20k.

Also I have a 40-instance limit on EC2 (which I may be able to raise).

168 comments

[ 3.4 ms ] story [ 219 ms ] thread
gesman
>> ... no arbitrage, reselling, bitcoin mining

You just scrapped my first 3 ideas :)

tlrobinson
Would running a PaaS on top of EC2 be considered "reselling"?
chrischen
Only if it is purely a subset of EC2.
gesman
Actually it's interesting task to come up with creative way to convert amazon credit to hard cash (that does not expire).

I don't think bitcoin mining via CPU (virtualized especially) is feasible for that anymore.

waterlesscloud
Altcoin mining on the amazon gpu machines was still viable, last I checked.
n4ru
Well, it WAS, until Scrypt ASICs came out about a year ago. Now it's not viable. You'd be lucky to get a percent back.
fasteo
>>> I got $100k in AWS Credits

May I ask how ?

chrischen
Its a credit given to companies backed by an accelerator.
anigbrowl
Even ones that can't think of anything to do with it? I wish I had these problems :-/
dm2
It's given out like candy, if not the 100k one you can surely get several thousands worth of credits for server hosting, either on Azure or through Amazon.
pc86
Can confirm - with my MSDN license I get a $150/mo credit for any development/test instances. As long as I have a valid license (and they offer the deal) I'll get the credit.
[dead] Chronic31
We're talking about $100,000 here. Not $150/month on some shitty cloud. I used to work at Microsoft.
bwb
which one?
zerni
I know Startupbootcamp has these deals for their batches. A similar deal is available for Google business users.
pliny
You can try turning 100k into 150k:

https://www.eff.org/awards/coop

fragsworth
Unless OP finds some way to turn our understanding of prime numbers on its head, the best case in one year is turning the 100k into 50k by finding the first prime with 1,000,000 digits.
dm2
Is that prize still being offered?

I think it was claimed in 2000, "The $50,000 prize will go to Nayan Hajratwala of Plymouth, Michigan, a participant of the Great Internet Mersenne Prime Search (GIMPS), for the discovery of a two million digit prime number found using the collective power of tens of thousands of computers on the Entropia.com network."

https://www.eff.org/press/releases/big-prime-nets-big-prize

bentpins
The article linked refers to the smallest of four prizes offered.

$50,000 to the first individual or group who discovers a prime number with at least 1,000,000 decimal digits (awarded Apr. 6, 2000)

$100,000 to the first individual or group who discovers a prime number with at least 10,000,000 decimal digits (awarded Oct. 22, 2009)

$150,000 to the first individual or group who discovers a prime number with at least 100,000,000 decimal digits

$250,000 to the first individual or group who discovers a prime number with at least 1,000,000,000 decimal digits

chrischen
Unfortunately that prize has already been awarded :(.
dm2
Current largest prime: 2^57885161-1

So take random prime numbers larger than 57885161 (such as 57885167), and find a script that can calculate with numbers that high using EC2s server constraints, then see if 2^(large_prime_numb)-1 is prime. Is that the correct method of doing this?

https://www.eff.org/awards/coop/primeclaim-43112609

What are the stats on testing large prime numbers on EC2 instances?

beckler
3d render farm?
nomel
He's a bit limited on instances though.

Still, could find some animation studios, offer them your farm for some % of profits.

Pretty big gamble though.

chrischen
One idea I had was to turn http://www.arthunted.com into a service. So you can specify parameters and sites to crawl and gather data, on demand. $5 a day times xxxx users.
boomzilla
scrapinghub seems to be the top dog in this space. You might want to check it out.

What's your background? Are you looking for business idea that can be built upon your existing site, or are you looking for something completely new?

chrischen
It doesn't have to be a business idea! Just an interesting idea that would otherwise require a lot of compute or storage, or something else provided by AWS.
sinsanati456
chrischen. I am also facing similar situation. So, which out of all the suggestions did you found the most interesting one ? And what have you actually planned doing with it ?
automathematics
I assume you had two options, 100k for a year or 10k for 2 years (I think)? (I've seen these offers)

Is that the case? And if so, why not take the other option?

chrischen
$100k is so much bigger number.
automathematics
Sure but it's like saying "here's a meal for 20, you have to eat it all today... or I'll feed you and only you for a year"
fragmede
Take the meal for 20 and host an improtu banquet. Charge a per-person fee and make some money.
heeton
No, it's like saying "here's $20, you have to use it all today. Or I'll give you $4, half today, half tomorrow".
galactus
mine dogecoin? (j/k)
bbcbasic
The altcoin ideas are getting downvoted, but with the right alt coin he could make $10k+ in the bank to keep.

Anything related to creating an ongoing service whether for love or profit, is madness, because in 12 months that service has to stop unless the OP has $100k to spare to keep it going for the next year.

sinsanati456
What are the right alt coins that your hinting at ?
[deleted]
(comment deleted)
VikingCoder
Well, if you're literally not going to use it - and you'll lose it at the end of a year... Can you donate it to Folding@Home or something?

I'm guessing AWS wouldn't want you to...

api
Look up some of the prime factorization prizes and do the math to see if it is achievable.
rememberlenny
Why can't you mine bitcoin?
ecesena
If you do the math, you won't get much more than 4-5k$ in bitcoin...
Retric
Because that only makes ~1,000$ or less which seems like a huge waste.
lockes5hadow
As above, but also mining crypto's is against AWS terms of service.
nomel
Why is that? Are they oversubscribed?
MichaelGG
Citation please. I just searched the AWS ToS, AUP, and customer agreement and found no such restriction.

Or do you mean that the startup $100K credit is restricted from obvious reselling/mining/anything that's not doing a value add? Cause of course AWS doesn't want to give away money, but promote startups to actually use their services.

[deleted]
(comment deleted)
[deleted]
(comment deleted)
th0br0
why don't you use the credits to try to create a publicly queryable index of the web in a standardised format? read: open source search engine. as you've got the money, just ignore efficiency.

else... commit part of it to one of the computing @ home proejcts?

Smerity
If you're interested in a publicly queryable index of the web, you could try running a search server such as ElasticSearch on the Common Crawl[1] corpus. ElasticSearch runs the search backend of WordPress, 600 million+ documents in total[2], so extending it to a Common Crawl archive seems possible.

n.b. I'm a data scientist at Common Crawl, so have a vested interest!

Also, whatever experiment you end up pursuing, remember to use spot instances if your setup allows for transient nodes - it'll substantially decrease your burn rate (usually 1/10th the price) allowing for even larger and more insane experiments :)

[1]: http://commoncrawl.org/

[2]: http://gibrown.com/2014/01/09/scaling-elasticsearch-part-1-o...

th0br0
I'd never heard of Common Crawl before but it looks like an awesome project! Keep up the good work!
Eridrus
I had a crawling project where I wanted to get a sense for a few ad-related things on the internet and came upon common crawl and was initially excited since I thought it would have incidentally captured the data I wanted, but I was disappointed to find that they did not do any kind of JS execution, which limited the effectiveness for me pretty drastically.
chrischen
How up-to-date is commoncrawl data?
sp332
Artificial intelligence? It needs huge datasets and lots of CPU to train. Then you could put it to work, maybe captioning video or something.
braum
do a better version of iThenticate which helps to prevent and find plagiarism in published content. We use it to help verify that content someone has sent to us is unique and is not just copy pasted in part or whole. It also helps us find uses of our published content and course material that has been re-purposed or copied verbatim. Entry level price point for iThenticate is about $5k per year. And unfortunately copyscape.com is not the same as iThenticate.
kefka
This is what I would do.

3 Steps:

1. Scrape every possible image along with location data (if available). Save all these images in Amazon storage. It's best if you can scrape photo galleries that include building names, sites, or other location descriptive data. Questionable gray area, but this is a mashup of thousands of images.

2. This is now your photogrammetry grid. Take all those photos and generate 3d scenes from the data you scraped.

3. Open up shop with these 3d assets. Charge for quality of object. Extra money if you make it easy to import into UE4, Unity, or Torque and make it "Ready for the Oculus Rift".

radicalbyte
4. Get sued for copyright infringement.
kefka
Ok. Prove I used Exhibit A in making of this 3d scene.
chralieboy
"Here is a subpoena showing the IP address of the AWS instance you controlled along with server logs showing you accessed that image.

Based on public statements stating how you compiled the images and comparison between the client's photo and your image, it is not beyond reasonable doubt that infringement likely occurred."

pbhjpbhj
Well we've got the server logs showing your AWS instance accessing that image on our server. As copyright is tort the usual burden of proof is balance of probabilities [UK, "preponderance of evidence" in USA I think; please correct if this is wrong in your jurisdiction], I'd say that's enough to swing it that you're going to need to prove you didn't use that image ... oh and we have a HN post replying to you suggesting you do this, which swings the balance a touch further.

We can probably have an expert witness testify the scene could use that image (ie they don't visually disagree so much that the scene couldn't have derived from inter alia that image).

Not enough perhaps to prove a criminal case ...

[deleted]
(comment deleted)
3327
thats part of business
pc86
You should try getting into a more honest business, then.
matt_morgan
It's a new work, with substantial creativity involved. I don't see it getting anywhere.
teraflop
Having "substantial creativity involved" does not prevent something from being legally considered a derivative work.
davidmr
And furthermore, you cannot pay your lawyer in Amazon credits.
btown
Unless your lawyer's name is Ed Felten, your name is Barack Obama, and the Amazon credits can be applied to GovCloud!
jhayward
"Substantial creativity" may get you past being a derivative work in Europe, but not in the US. Copyright law differs between jurisdictions. You'd have to be very careful.

I like the original idea. What I'd do is make sure the resulting images don't have any significant reliance on any small set of originals. So if challenged, you could re-create the scene w/o the challenged images and show a court that the scene is not closely derived from any single source.

hauget
Unrelated, but are you talking about creating 3d models out of point cloud data or something else?
kefka
Nope. Pointcloud is just depth data and potentially color data.

Photogrammetry is the technique of 3d scanning that correlates feature points within multiple pictures in order to back-project a 3d scene.

The more pictures you have of an area, the higher quality the overall scan. So if we have 3000 images of a building's exterior in NYC, we can recreate the building in 3d.

My idea was that, for X thousand images, a single image is a trivial datapoint, and could be easily removed with little loss in quality of scan. It may technically be in violation of copyright, but is used for a substantially different work.

I believe it could possibly qualify as fair use.

nightski
If you have correlated feature points you essentially have a point cloud no? It's just that photogrammetry adds an additional step of back-projecting the 3d scene?
deepnet
Using a SIFT pipeline for photogrammettry, I have successfully recreated small objects, Comet 67P and some buildings from quadcopter pics.

First I search for features with DoG and match them with SIFT , do a bit extra crawling along matched edges and the result is a dense coloured point cloud.

The point clouds are converted to a mesh with poisson surface reconstruction and retextured with fragments of the original images.

The Poisson surfaces are never quite as nice as the point clouds - I am using Meshlab for this part.

Processing a few hundred big images takes ages so I send the jobs up to EC2 for a few hours so each job is usually a couple dollars.

It is pretty useful as a 3D scanner.

hauget
That-sounds-amazing. Any chance you could share a link of the end result? Also, any suggestions on how to get started with photogrammetry & computer vision?
dharma1
would love to chat about this - do you have a contact email?
sciencerobot
Genome sequencing requires a lot of CPU power and disk space. You could build an application that performs these tasks and use the first $100k in compute power to help it grow. There is a lot of domain-specific knowledge in this field and I hear bioinformatics is difficult to monetize.
[dead] Chronic31
Genome sequence is a dead end and full of bioinformatics folks who have an unachievable dream.
[dead] sciencerobot
Genome sequencing requires a lot of CPU power and disk space. You could build an application that performs these tasks and use the first $100k in compute power to help it grow. There is a lot of domain-specific knowledge in this field and I hear bioinformatics is difficult to monetize.
tarequeh
You could build a video conversion website where one uploads original high resolution video and it'd spit out 1080p/720p/320p and other formats of video that are suitable for delivering on different devices/bandwidth. This could be an alternative for people hosting video on youtube and getting slapped with ads. An effort like this would use a lot of CPU but once it's converted it's just the storage cost. Common challenges are copyright issue but I can see different ways to promote it as a professional service than collection of random videos. $100K would cover cost of offering it for free to people for a limited period.
kgrin
That sounds pretty much identical to Zencoder, encoding.com, etc.
developer1
As strange as it sounds, that much bandwidth would probably suck up $100k faster than you'd imagine. There's a reason a lot of large companies that deal in video have their own data centres and crazy bandwidth deals that make it cheap. I don't think $100k at Amazon's prices would last more than a few months if the service became popular.
jordhy
I suggest you take your algorithm and apply it to other niches. There might be good horizontal scalability there. For example why not try to crawl photography sites to discover popular photos and sort them by style or "taste"

You could then create your own photo discovery service and call it Find my Style or something like that.

There are two ways to do this, you can either directly scan the photos looking for graphical patterns or you could analyze the text into which the photo is posted.

chrischen
I like this idea. Crawling art/photos online, classifying them, and otherwise mining the data would tie in well with my existing business, and also effectively convert $100k of computing into reusable stored assets.
skizm
crypto-currency mining? You could burn through that very quickly with enough machines.
pearjuice
If someone would have an idea how to get value out of $100K in AWS credits, would you think they would tell you? Why wouldn't they take the idea, pitch it, get $100K in AWS credits and then double it themselves?
dm2
Because humans are many times irrational and lazy.
jliptzin
Because he's not asking for ways of generating more than $100k profit out of his $100k AWS credits which would be quite difficult. He's in the unique situation of having $100k in free AWS credits and would probably be happy getting back half that as cash.
chrischen
Well I'm less interested in converting it to cash, as I could easily just start reselling the credits.

I'm trying to do some hacky project that would make a big splash... as there's $100k of value to be consumed.

jliptzin
Out of curiosity how'd you get the credits?
sinsanati456
Any idea, where you could resell the credits ?
[deleted]
(comment deleted)
larrys
I will let others answer "how should I use it" what I want to know is how did you get 100k in AWS credits.
jdorward
some considerations

- budget for bandwidth... esp if you are doing something more than text crud and want to serve it - e.g. image/video

- video is "heavy" and so necessarily takes a lot of compute, ram and storage and can also leverage gpu and pricier instances, depending on what you are trying to do... e.g. understanding video content with opencv/opencl or specific types of drawing like raytracing

- instance limit increases with aws are perfunctory... so i wouldn't consider 40 a limit, certainly don't design anything interesting with that limit in mind

- spot instances can save you a lot and stretch that $100K 1.5x - 4x depending on region, availability zone, and instance type

- unless you've done it before, time is your enemy to get into position to spend that money on something useful... so your monthly budget target range makes sense