Ask HN: Best practices for technical SEO for domains with more than 100k pages?
What's special about domains with 100,000+ pages (e.g. niche search engines and large e-commerce sites) is that most basic SEO advice is necessary but far from sufficient.
I'm on a project with 1,000,000 pages and only about 50,000 are indexed. My "crawl budget" is absolutely small, but relatively large. How do I make sure the "right" content gets indexed? How does Google even define "right"?
I seem to be writing a lot of code to answer basic question like "which of my 1,000,000 pages are indexed?" and "which pages should I be actively seeking to index and de-index?"
If you work on "technical SEO" for something like Stack Overflow or Amazon I'd love to learn from you.
And if these are the wrong questions to ask, I'd especially like to learn from you.
21 comments
[ 5.5 ms ] story [ 63.2 ms ] threadDo you have a sitemap.xml file? Does it include accurate last updated dates or suggest realistic crawl schedules (eg: only index static pages every month, but frequently updated pages every day)?
Have you run your site through an SEO checker (or at least some of the ignored pages) to make sure that they are accessible and not being blocked/ignored by a crawler? Any server responses or canonical links that may signal the wrong thing to a bot?
Do you have any deep links to your more overlooked pages on the homepage (or at least on your more popular pages)?
Do you have backlinks in the wild that point to the pages that you want to have crawled? Are they marked as nofollow? Do you post links to these pages on any social media accounts?
We use SEMRush and that's partly what's driving my questions: Their guidance is largely irrelevant for us: We're a niche search engine.
We have several thousand backlinks and I wish I had a better strategy here, but our content is broadly unshareable (versus something like Stack Overflow, which is very shareable).
https://support.google.com/webmasters/answer/7440203
I'll talk with the team about richer data as it furthers our SEO goals. I think the idea here would be that better engagement will improve our ranking? Is that correct?
The tricky part is getting impressions and clicks but this could help.
Absolutely. Rather than performing optimizations for SEO, it would be better at this point to add more to the site. Once that's done, post about it in different places like on Reddit. You should start to see more traffic from both web crawlers and people alike
I noticed that many of the vehicle images didn't load for me. It looks like they're hosted on whatever site they're pulled from. It would be a good idea to save thumbnails of the images that you find so you can serve them locally and include them for every listing
Since you're scraping car data too, it might be a good idea to get some basic descriptions for each type of vehicle. You might be able to opt into the Kelley Blue Book Developers program or similar:
http://developer.kbb.com/
It would also be good to provide at least some snippets of the descriptions of the vehicles that you're listing. More textual content would give Google something extra to dig into and would keep users from leaving the site and just browsing ebay or Craigslist themselves as often
It would be a bigger effort, but having basically a page for each listing locally would be interesting. You could show the original text and posts from the original source. Make the source clear, along with the last time you retrieved the information from it. Then if the original listing gets removed, you could keep it around. Show that it was unlisted or sold, but have a banner at the top that says something like "This listing is no longer available. Click here to look for available [make] [model]s". It could help people to find why one vehicle was cheaper than another (condition, location, etc)
How unique is the content on each page? What do the crawled and not indexed vs discovered not crawled ratios look like relative to indexed pages? What does linking to the detail pages look like (category pages, similar items)? How much authority do category pages have in terms of backlinks? Page speed issues-- in GSC, can you manage average download time <100ms (makes a huge difference)?
Noindexing may help-- if you just initially handle that based on page content uniqueness and/or presumed MSV potential for the page. Otherwise, based on your language, can you request a quota increase from the 200/day notifications for the Indexing API?
If you like reading about the topic, maybe head to BHW forums (very poor SNR, though) and read about what some of the more unscrupulous people are doing to index MM+ pages/mo and extract some of the less-risky strategies (e.g. don't abuse the news/job posting APIs to get spam content indexed).
As additional commentary, the last 2-3 months have been chaos in terms of algo updates. There was essentially a 30 day period where Googlebot all but disappeared for most sites, then the two HCUs and other random updates tossed in for good measure, stacked with desktop infinite scroll and the switch to mobile first indexing. I can't remember as many changes happening in a single quarter in over 15 years of being SEO or SEO adjacent.
I use SEMRush, GSC, GA and Mouseflow religiously. That is, regularly and with faith. I don't think they necessarily point me in the right direction.
I'll check out the forums.
There are a few factors that you need to adjust for: authority, quality, structure.
Authority is mostly out of your hands as a technical SEO scope but as more content gets indexed, more links and signals will come in. The real culprits are quality and structure. As others have said, low-quality pages need to be addressed. They can always be worked back in later but you’re better to be ruthless about it now and add them back later.
Profile pages, thin content pages, duplicates all need to go. No index them first, then block in robots eventually and doesn’t hurt to canonical them where applicable. If you jump straight to robots, they won’t pickup the noindex directive.
The structure, a lot of people think of folder structure as IA but really it’s linking structure. It’s important to know where you’re placing emphasis on pages by how you’re internal links are setup. This is also the best way to surface deep pages as you can link to them to bring Google deeper into your site.
Also you can try refreshing dates and seeing if that helps. Short term solution but works plenty well.
I know this is pretty general but it’s hard without seeing your issues specifically. If I can help you get on the right track, lmk. Gladly take a look. I run a small(free) community with some strong technical SEOs in it as well who like helping. Not sure how to connect but lmk if interested and we will figure it out.
Brent [at] CarGraph [dot] com