Many many years I was this annoying jr team member that came and asked other people tons of questions. In my case, the other members of the team has something like 10 - 15 years more experience than I. Ultimately what…
I maintain web crawlers for a large internet portal and this stuff is a pain enough to deal with when people unintentionally make their websites recursive, much less INTENTIONALLY. facepalm At least google crawler had…
The needs of our customer are too transient for reserved yet, so it's primarily been as needed (on demand?) We run into problems during provisioning sometimes and Amazon has to then raise our quota. Some customer…
Most of our instances we keep up for months at a time, although they can turn flaky. We tend to use EBS, so we reprovision instances when needed. The biggest overall issue we have with EC2 is clock skew issues that can…
It fluctuates, but most recently somewhere between 34 - 56 instances for dev, staging and production search, crawler and content build clusters. (we index terabytes of data and hundred of millions of documents)
In the immortal words of Cave Johnson, "Why not?!" :-) Dealing with structured data is easy compared to unstructured data, but there is a vast wealth of unstructured content just waiting to be tapped. I suspect there is…
Many many years I was this annoying jr team member that came and asked other people tons of questions. In my case, the other members of the team has something like 10 - 15 years more experience than I. Ultimately what…
I maintain web crawlers for a large internet portal and this stuff is a pain enough to deal with when people unintentionally make their websites recursive, much less INTENTIONALLY. facepalm At least google crawler had…
The needs of our customer are too transient for reserved yet, so it's primarily been as needed (on demand?) We run into problems during provisioning sometimes and Amazon has to then raise our quota. Some customer…
Most of our instances we keep up for months at a time, although they can turn flaky. We tend to use EBS, so we reprovision instances when needed. The biggest overall issue we have with EC2 is clock skew issues that can…
It fluctuates, but most recently somewhere between 34 - 56 instances for dev, staging and production search, crawler and content build clusters. (we index terabytes of data and hundred of millions of documents)
In the immortal words of Cave Johnson, "Why not?!" :-) Dealing with structured data is easy compared to unstructured data, but there is a vast wealth of unstructured content just waiting to be tapped. I suspect there is…