HN Explorer

Show HN: I scraped 25M Shopify products to build a search engine (searchagora.com)

317 points by pencildiver ↗ HN
Hi HN! I built Agora as a side-project leading up to the holiday season. I wanted to find an easier way to find Christmas gifts, without needing to go store-by-store.

My wife asked me for a a pair of red shoes for Christmas. I quickly typed it into Google and found a combination of ads from large retailers and links to a 1948 movie called 'Red Shoes'. I decided to build Agora to solve my own problem (and stay happily married). The product is a search engine that automatically crawls thousands of Shopify stores and makes them easily accessible with a search interface. There's a few additional features to enhance the buying experience including saving products, filters, reviews, and popular products.

I've started with exclusively Shopify stores and plan to expand the crawler to other e-commerce platforms like BigCommerce, WooCommerce, Wix, etc. The technical challenge I've found is keeping the search speed and performance strong as the data set becomes larger. There's about 25 million products on Agora right now. I'll ramp this up carefully to make sure we don't compromise the search speed and user experience.

I'd love any feedback!

285 comments

[ 4.0 ms ] story [ 293 ms ] thread
misterbwong
What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?
cldellow
(not the OP, but I have some experience with Shopify)

Shopify stores publish their product catalog at /products.json. From personal experience, you can hammer it pretty hard without being rate limited.

A challenge is that the pricing info in that endpoint is based on the stock Shopify catalog fields, and can be misleading depending on the specific theme customizations that the merchant uses.

pencildiver
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end.

Here's an example: https://www.wildfox.com/products.json

fermisea
Oh nice, you deserve great things in life for this comment!
satvikpendem
How did you detect that it was a Shopify store?
capableweb
Not OP but:

"Has the site a /products.json file?" is a good first check :) And if it does, "Does that format match with the format a Shopify store?" is another good followup question.

8372049
In another comment, OP wrote:

> Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).

satvikpendem
Ah, that makes more sense, I used BuiltWith before.
xp84
There are lots of telltale endpoints that you could just HEAD for a 200 vs 404. Or even just the products.json itself is a pretty good giveaway.

Or an even better way I’ve done in the past (to check which competitor’s platform a list of prospects is using in bulk) is just to use the DNS — a Shopify shop will be CNAMEd to a certain Shopify hostname.

stef25
Looked in the source of a random Shopify store, there are 200+ occurrences of "shopify", that's a clue :)
bomewish
What’s the trade off using js for this? Would it have been much faster to use go or something?
qdequelen
Did you only get the schema.json?
glohbalrob
wow! Nice work. I've been trying to build an index of shopify stores. Did you search for all domains pointing to shopify's name servers?
selcuka
I don't think that would work as many people also use Cloudflare etc.

You may try using BuiltWith which is a paid service:

https://trends.builtwith.com/websitelist/Shopify

ttt3ts
It works fine. Just issue a HEAD request when you are unsure and rotate proxies a lot l. Takes a bit of infra but definitely possible.
selcuka
I mean simply querying nameservers won't work.
hipadev23
https://www.searchagora.com/search?query=red%20shoes

It doesn’t work very well.

owlninja
I see some Red Wing shoe products first, then some bearings, some shoes, then someone in lingerie with red heels, and then more red shoes. I know Google is the horse to beat right now but they showed me nothing but red shoes to buy plus a few ancillary results for such a generic search - and if I was a good husband I would add a few words to narrow my search.
cmcconomy
That's funny, I made a domain-specific version of this for canadian coffee deals.

https://beangrid.mcconomy.org/

pencildiver
Super cool project (especially as a coffee lover myself)!
cmcconomy
The fun part was figuring out how I was going to put the site up without hosting ;)
ska
github?
cmcconomy
Yes - I have a daily cron-based scrape & commit job which updates the table data source CSV, along with github hosting for the static components.
boringg
Which coffee seems to hit the best in Canada (your take). I find the espresso in Canada hasn't been as good as the coffee brands in the US but I'm open to possibilities.

Also like the project!

wcarss
personal somewhat-pedestrian list: Pilot, Detour, Reunion, Propeller, Phil and Sebastian
cmcconomy
I enjoy Café St-Henri's Godshot when I can get it at a discount. Anything from Stereo is great, and I've enjoyed Monogram and De Mello. If I was buying at regular price, I would often get Social Coffee.

Of course I put this together because Black Friday is when I load up on (relatively) cheap coffee and chuck it in the freezer, so this time of year I always branch out and try new places and new offerings from familiar places. I built this list mostly from a reddit compilation I found, and I've been slowly updating the source url list as I learn of new canadian roasters that happen to be Shopify customers.

konschubert
Hey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me?

https://shop.invisible-computers.com

pencildiver
Super cool product! I'm currently using a list of Shopify stores, so it's still limited (i.e. wanted to start with a relatively small list to focus on the search experience). I'll submit your URL to the crawler now. If you want to reach out to support @ searchagora.com , I'd love to get your feedback as a Shopify store owner.
shubham_sinha
Hi, you could drop an email to onboard@peppyhop.com and we will be happy to onboard you. Please add target geography like you would like to target Indian market or US market
muratsu
Agora also doesn't return red shoes for the search query "red shoes". Seems like you haven't fully solved the problem yet :)

From a technical perspective, crawling 25M products is impressive but the search itself doesn't provide much value to me. I already use large e-commerce sites (amazon, wallmart, ...) and targeted ones (Nordstrom, SSENSE, ...). Sure I may not be searching through all the shopify, wix stores but I need to know why that's valuable to me to begin with. Perhaps understanding the value prop of SMBs and educating me about it would be a better positioning for Agora than simply being a search engine.

pencildiver
Definitely have not solved the problem yet! The search algorithm prioritizes the brand called "Red Wing Shoe" so still figuring out ways to show real 'red shoes'. Have been thinking about passing the images through a detection tool and tag them to enhance the search experience.

Re: Value Proposition. Absolutely, I think focusing on the SMB-angle and 'local shopping' will help direct users better. I'll definitely take this into account.

charliejuggler
You could try the method we used for our vector search demo for e-commerce (all open source, natch) - use CLIP to get vector embeddings for product pictures and then use these for boosting or matching. https://opensourceconnections.com/blog/2023/03/22/building-v... Our demo works pretty well for searches like 'blue network cable' when the colour isn't always explicitly mentioned in the product data.
[deleted]
(comment deleted)
taimurayaz
Cool project but Shopify already has this. https://shop.app

https://shop.app/search/results?query=red%20shoes

dinkleberg
Was about to share the same link, it seems like competing against Shopify would prove quite the challenge.

The real way to differentiate IMO is with a targeted UX for different niches rather than the one search engine to satisfy all queries.

pencildiver
Yeah absolutely. I hadn't heard of Shop until today but the value proposition is definitely similar. In the next week, I'll add other e-commerce platforms like BigCommerce, WooCommerce, support for custom built sites, etc. to really differentiate the user experience.
gardenhedge
Oh cool. That works a lot better than OPs
treesciencebot
This is amazing for finding cute collectibles from my favorite TV show that I would otherwise not noticed among random t-shirt and other "slap the picture and call it co-branded" products! I'm not super sure how long it is going to be around, but I think I'm gonna keep playing with it for a while.
pencildiver
Really happy to hear this. I'll do my best to keep this around :)
Asparagirl
Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
xnx
https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.
prayze
Did this suddenly get changed? Nothing but "# ,: # ,' | # / : # --' / # \/ />/ # /" is shown now.
calebegg
For some reason, "view source" gets the right list. Maybe a referer issue like someone else said.
wizzwizz4
It's just your browser's HTML parser. Line 6:

  #                         / <//_\
This is being interpreted as a malformed HTML closing tag, which (according to the HTML5 parsing algorithm published by WHATWG) gets treated as a comment. The file doesn't contain any > past this point. This leaves the uncommented contents from lines 1–6:

  #                               ,:
  #                             ,' |
  #                            /   :
  #                         --'   /
  #                         \/ />/
  #                         /
Or, with whitespace collapsed:

  # ,: # ,' | # / : # --' / # \/ />/ # /
Which should be exactly what you observe.

Ref: https://html.spec.whatwg.org/multipage/parsing.html https://developer.mozilla.org/en-US/docs/Web/CSS/white-space...

calebegg
It seems sort of questionable to use the list of things to not scrape as a starting point for scraping.... I mean, I get it's not actually enforced.
das_keyboard
Not really sure why all the answers here are flagged, but you may be mistaken.

The robots.txt does not exclusively list what not to scrape.

It provides information on which parts are allowed and wich are not (disallowed).

It also provides sitemaps for crawlers as a starting point with more information (eg. which sites are available and how often are they updated, etc.)

KomoD
Looks like it's just Shopify's own pages and not anything related to actual stores.
pencildiver
Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
russum
> and between $100k - $1m in revenue

Does "Built With" provide that data? How accurate do you think it might be?

patatero
Shopify shops always have /collections, /products, and /pages in their URL. If you have a regular Shopify site, you're not allowed to change them. I don't know if Shopify Plus clients can change them.

Shopify sites also have shop-name.com/products.json which has URLs that point to cdn.shopify.com

[dead] Uptrenda
[flagged]
rsweeney21
Google is a data scraper. OpenAI is a data scraper. Pretty much every major company uses data obtained from a data scraper.
jokethrowaway
Why? They're cataloguing public data. Do you think the same of search engines - which are equally scraping the web?

If it's not useful to you, nobody is forcing you to use this product.

Uptrenda
You really didn't think about this. Search engine crawlers are specifically designed to ignore sites that have the right setups. In other words: they try to be good citizens of the web. Data scrapers on the other hand are known to steal others IP and use it to jump start their own services.

When you do that you're using all the bandwidth of the site while devaluing everything they've built. They will have invested significant time and resources building that IP. But data scrapers think that just because its on a public website it means they can leech it and do what they like. No, there is such a thing as copy right and respecting authors. Fuck anyone who says otherwise.

If you want to be a little kiddiot and steal content you're free to do so. But you're just undermining the work of everyone you steal from. Making it less and less viable for them to make more of it in the future. I stand by what I said. Data miners are fucking parasites and the web would be better off without them.

twelve40
hm in this case, he is driving business to these websites. Quite the opposite from OpenAI and scrapers you are talking about, no?
system2
He is putting a high amount of stress on Shopify servers. OpenAI is already in trouble with data scraping practices. Hypothetically if this site becomes successful one day, people will do whatever it takes to become #1 on the search results page and will drive business to only a handful of companies.
twelve40
You're right in that it probably would be nice to have a civilized opt-out button like Google. But that's not hard to add (especially hypothetically if this site becomes successful). In my mind, total opposite of OpenAI thieving bulk data and giving 0 back to anybody.
mandeepj
Bad experience with someone?
peddling-brink
Arrested? That’s a hot take.

What is op doing that would warrant such a response? If anything, they’re providing free advertising to their “victims”.

xnx
Great project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.
pencildiver
Great call! I am doing back-ups on Mongo and this is a good use-case for tracking changes. Also trying to figure out how to detect is a product is sold out or not being sold anymore.
Minor49er
I worked on a competitor to CamelCamelCamel years ago. We had this exact issue since people would often click through to a page where the price or availability were different from what we were showing

Ultimately, we ended up adding an interstitial page between the product listing on our site and the page on the seller's site

This interstitial checked to see if we checked the price in the last couple of minutes, and if not, it would run a quick scrape of the page to ensure that we had the most up to date information

I can't remember exactly what the messaging or behavior was when there was a difference. I think there was a message that was displayed if the prices were different. Or if the product was actually out of stock, it would pull the user back into our site with a toast explaining that the product was no longer available

Anything less aggressive than this resulted in more customers experiencing price/availability errors or simply leaving the site, and anything more aggressive resulted in angry site owners who were losing bandwidth to our bots

> Also trying to figure out how to detect is a product is sold out or not being sold anymore

In these cases, either the page will say as much (eg: "Product Unavailable"), have some kind of stock or status code hidden beneath the UI to show that it's not available, or the target page will simply vanish from the web. However, none of these are guarantees. A site could say that a product has been discontinued, but the item could come back later, or under a different SKU, or whatever else

secabeen
For all of Amazon's faults, the fact that they tolerate CCC does drive a lot of my online purchases there. CCC used to track other sites, and was eventually blocked on all of them. If more sites want my business, showing their pricing history (either from internal data, or by letting someone build the DB) would go a long way.
moneywoes
is camel camel whitelisted by amazon? or can any scraper work
tehlike
Amazon associates program doesn't normally allow price trackers except with written approval
tehlike
Amazon doesn't allow price alerting/tracking on their affiliate program anymore, you need explicit written consent.

I am the owner of https://pricetracker.wtf and got the boot today.

nocoiner
Is it somehow known that CCC hasn’t been co-opted by Amazon? Frankly I figured Amazon would have bought them out a decade ago, but maybe the CCC founders have a stronger ethical compass than I do.
joshuamcginnis
I love your approach; you found a problem and developed a solution for it. And then you got the courage to share with the larger technical community. Good on you.

There's obviously some rough edges (multiple duplicate products, issues with product links linking to empty pages, and no results for broad terms), but don't let that stop you. I'm certain they can all be fixed.

Keep going! At the least, you'll come out of this with an excellent project in your portfolio.

pencildiver
Thank you, that means a lot. It has definitely been a whirlwind of emotions since posting on HN but glad I did. It's definitely an MVP so going to work fast to improve it.
thih9
When I search for “op-1”, partial match like “Frontier Co-op Turkey Rub, Organic 1 lb. -- Frontier Co-op” gets ranked higher than “teenage engineering op-1”. I would expect the opposite.
mandeepj
Do you plan to add filters: price etc?

I was about to 'reviews' as well in the above list but decided not to as they are not always trustworthy. Now AI is so advanced, that it can be used to detect fake reviews and ignore them from sampling.

pencildiver
Yes. There's a very basic price range filter right now. Working on adding a ships to, location, and a few others. Open to any ideas that would help in the shopping experience.

There are 'reviews' now and made the decision to only let authenticated users leave users so they are more trustworthy (i.e. thinking is that adding more friction will lead to higher quality reviews).

quickthrower2
I like it.

I need to be able to filter search to if it will deliver to my country.

It desperately needs some indication that your action is being processed, like a spinner, when you search.

pencildiver
Absolutely. Working on a "ships to" filter and enhancing the 'price' filter.

Also fixing the loading experience as we speak. Wasn't expecting this level of traffic so didn't account for slow server speed with the front-end experience.

quickthrower2
Thanks! It is reasonably fast but slow enough that a cue is needed so I know the input triggered.
stef25
Or even "ships from" / "located in".

I'm in Europe and don't want to deal with custom hassles or delays from shipping. Etsy and Reverb both have this option which I never fail to use.

[dead] mvdtnz
[flagged]
8372049
OP said in another comment:

> ... narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).

jonnycoder
Awesome! It would be good to listen to the enter key when typing in a search query. Your privacy and terms links point to what appears to be the saas code framework you used (just a guess). I was looking for your contact/email so I can ask you some questions.
pencildiver
Enter key should work, but the loading speed is very slow right now. Fixing that right now :)

You can reach out at support @ searchagora.com. Would love to talk!

bsbechtel
I searched for 'pão francês' and my store was the #1 result. I think you're doing it right! :)
dns_snek
I'm sorry, but I have to question where this heartfelt story about looking out for your wife is in any way real?

The website certainly doesn't look like a side project, it has a fully fledged system for merchants to advertise on Agora for a fee, an affiliate system offering $50 commissions to onboard merchants and the ToS and Privacy policy link to a website with the following mission statement:

> We buy, build, and invest in software companies with recurring revenue and product-led growth.

pencildiver
OP here. Yup, haven't launched the holding company yet but idea is to have an LLC for all my software projects. Still a work-in-progress in both thinking and execution. Agora specifically came from a personal need and is obviously still an MVP project.

Spun up a Merchant Page and Affiliate Program page in a few hours on Webflow using a template. There is a merchant dashboard built but the 'affiliate program' is a test.

ruune
How is "Jaggi Enterprises LCC" involved, where for example the terms lead to?
[deleted]
(comment deleted)
[deleted]
(comment deleted)
screye
What was the process for scraping 25M products ?

I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale.

Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ?

______________

A recommendation for how to improve search.

Your base captions will be pretty bad. You can use spot instances on a smaller GPU machine to run a dense captioning model (https://portal.vision.cognitive.azure.com/demo/dense-caption...) and generate captions for all your images.

Then for search, a simple vector store index would be a great retrieval solution here. It is better to do search using those as well.

Both are pretty cheap and can be done reliably within 20-30 lines of code each in python. 3rd party tools for these are pretty stable.

Ninjinka
As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.
pencildiver
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools.

For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json.

Built a crawler in simple Javascript to run through a list that I bought on a site called "Built With", access their JSON file with the product listing data, and scrape the exact data we want for Agora. Then storing it in Mongo and, currently, using Mongo Atlas Search (i.e. saw they released Vector Search but haven't looked at it). It has been a process of trial and error to pick the right data fields that are required for the front-end experience but not wanting to increase the size of the data set drastically. And after initially using React, switched to NextJS to make it easier to structure URLs of each product listing page.

Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well.

A few improvement that has helped so far:

- Having 2 separate Search Indexes, one for the 'brand' and on for the 'product'. There's a second public JSON file that is available on all Shopify stores with relevant store data at [Base URL]/meta.json For example: https://wildfox.com/meta.json

- Removing the "tags" that are provided by store owners on Shopify. I believe these are placed for SEO reasons. These were 1 - 50 words / product so removing these reduced the data size we're dealing with. The tradeoff is that they can't be used to improve the search experience now.

Hope this helps. Still wrapping my head around all of this.

ljm
2.2k/mo right off the bat is pretty steep, especially if you're paying that while the search response reliably takes over 10 seconds.

Why would you shovel 1.5k into MongoDB's pockets right off the bat? Especially when ElasticSearch is much better suited to what you're trying to do?

dinobones
You could run this entire stack (yes, even for 25 million products) using Kubernetes in a $40/month Linode + Elasticsearch + Cloudflare free plan.
altdataseller
Sounds like someone drank the Mongo kool-aid. You absolutely do not need Mongo, let alone Mongo Atlas. 25 million documents with ecommeece products is measly and should fit in a single 600 GB server
ljm
Probably not even that - 25mil is nothing really. A normalised schema in an RDBMS would handle that without sweating.
jabo
I'm biased, but I'd recommend exploring Typesense for search.

It's an open source alternative to Algolia + Pinecone, optimized for speed (since it's in-memory) and an out-of-the-box dev experience. E-commerce is also a very common use-case I see among our users.

Here's a live demo with 32M songs: https://songs-search.typesense.org/

Disclaimer: I work on Typesense.

keybits
I can also highly recommend TypeSense and have no affiliation. You'll save a lot of money and get much faster results.
k12sosse
> I'm currently not storing the image files, so that reduces the cost as well.

I wonder if someone catches on and replaces all your image URLs to the fuzzy testicle egg cup[0], will that negatively impact reputation?

0: http://i.imgur.com/32R3qLv.png

hipadev23
You’re spending $2k/mo run this?? Holy hell.
mfrye0
If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661.

Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo.

- https://www.postgresql.org/docs/current/textsearch.html - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-rd...

neeleshs
how big is the disk for the biggest instance?
mfrye0
Pretty small still at 500gb. It only stores hot data right now and a subset of what's important. Most of our data is in S3.
LunaSea
I have troubles seeing how this is possible.

$220 dollars per instance gets you 8Gb of RAM which is way, way, below the index size if you are indexing billions of vectors.

[deleted]
(comment deleted)
philippemnoel
You can even do Elastic-level full text search in Postgres with pg_bm25 (disclaimer: I am one of the makers of pg_bm25). Postgres truly rules, agree on the rec :)
slt2021
managed elastic search could slash your cost by an order at least
tehlike
Disclaimer: I am building https://pricetracker.wtf

You may want to look at Hetzner, and cut your costs by about 90%.

Feel free to reach me, email in profile.

adentranter
hey! this is cool, I take it you are based in the US?

How long have you been working on this?

tehlike
On and off for a year, with more time allocated since June. Yes I am in California.
katella
In your footer you have a lot of links like "kitchenaid price tracker" and "best buy price tracker". Have these helped links helped?
wolfgang42
I’ll second the comments that $2k/month is alarmingly high, especially for the performance that you seem to be getting. When I shoved ~40M webpages into a stock ElasticSearch instance running on a 2013-era server I bought for $200 (on eBay), it handled the load when I hit the HN front page just fine. Either you’re being drastically overcharged or there’s something horribly inefficient in your setup that could probably be tweaked fairly easily to bring your prices down.
DeathArrow
>Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well.

It will probably cost you just $100 to rent a server from Hetzner and do the same thing. I would also use Redis or another kind of cache to hit the DB less.

berkes
> site called "Built With",

Do you have Alink. And are they any good?

stef25
Google ?

https://builtwith.com/

Karl-Heinz
Berkes wanted to do good by sharing a provision with the OP, in case he/she buys something at builtwith.

We all know how to Google. :)

berkes
I specifically asked the author if he could add some extra info on Builtwith.

I can Google. But then I don't know if its truly the site the author was talking about. And I certainly don't know his or her insights on that site.

dangoodmanUT
Yo fuck mongo just use RDS or some digitalocean DB. Or really just use opensearch/elasticsearch, or even typesense (don't bother with raft it's so broken) or meilisearch
jabo
We’ve interacted before on Twitter and GitHub, and I want to address your point about Raft in Typesense since you mention it explicitly:

I can confidently say that Raft in Typesense is NOT broken.

We run thousands of clusters on Typesense Cloud serving close to 2 Billion searches per month, reliably.

We have airlines using us, a few national retailers with 100s of physical stores in their POS systems, logistic companies for scheduling, food delivery apps, large entertainment sites, etc - collectively these are use cases where a downtime of even an hour could cause millions of dollars in loss. And we power these reliably on Typesense Cloud, using Raft.

For an n-node cluster, the Raft protocol only guarantees auto-recovery for a failure of up to (n-1)/2 nodes. Beyond that, manual intervention is needed. This is by design to prevent a split brain situation. This not a Typesense thing, but a Raft protocol thing.

Oras
Take a look at TypeSense. Faster, better filtering, and much much cheaper if you’re going the cloud version
leobg
I index 40M paragraphs of legal text, bm25 and vector similarity search, at < 200ms query time, on a single $80/month Hetzner server. Email in profile if you’d like to talk.
KomoD
Oh... no... $1500/mo?
4runner
Sounds like you used an incorrect instance type/size on Atlas
helsinki
25 million products is really not much at all to scrape.
DeathArrow
>I have always used standard python tools like selenium, bs4 and the like

There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal.

Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.

Redster
Have Swedish family. Searched dala because family wants traditional Christmas ornaments. Sure enough, there were several results that were 10x cheaper than what I could find on the first page of Big Search Company. Great job!
pencildiver
Amazing, glad you were able to find it. I also just learned about what a "Dala" is :)
quaxar
Great site. Having built a search engine that needed to handle product data on a similar scale, it's not an easy thing to manage.

Some observations:

- Don't use infinite scrolling, it's an outdated UI practice that leads to bad user experience. It also makes the footer entirely unviewable.

- Clicking on a product card image does not reliably open up the product. I have to randomly click on it a few times (Chrome, Brave)

- Clicking on product card image and title leads to different actions, this is a bit unexpected, should show some hint of the difference.

- The product page pop up will reset the search list when closed, this messes up my search navigation, breaks the flow of browsing.

moneywoes
how did you avoid ip based blocking? rotating proxies?
moneywoes
where did you find a list of shopify stores to scrape
8372049
> Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).