And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.
I mean given that you've had this problem repeatedly, I'd call it "past-proofing", but I suppose you know your codebase better than I do.
Did you ever consider refactoring the code so that you don't have to do shotgun surgery every time you make this kind of change?
I work on the kind of big online scientific database that this article is about. 100% of our data is available from a clearly marked "Download" page. We still have scraper bots running through the whole site constantly.…
And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.
I mean given that you've had this problem repeatedly, I'd call it "past-proofing", but I suppose you know your codebase better than I do.
Did you ever consider refactoring the code so that you don't have to do shotgun surgery every time you make this kind of change?
I work on the kind of big online scientific database that this article is about. 100% of our data is available from a clearly marked "Download" page. We still have scraper bots running through the whole site constantly.…