How about using Crawl-delay at robots.txt? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
Was there a reason for not doing MVP? You could have made it easy to publish and scrap everything else.
How about using Crawl-delay at robots.txt? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
Was there a reason for not doing MVP? You could have made it easy to publish and scrap everything else.