I've worked on elaborate projects with the goal of world domination over the past few years, and all have failed. Now, I'm working on building a portfolio of smaller apps to slowly build passive income (with the potential for a big win).
TidyFinder is the first step towards that goal. I'd appreciate any feedback/criticism/suggestions that you can offer. I value the HN community's thoughts. Thanks.
Some basic details below, maybe I'll write a blog post with more thorough descriptions...
App and scrapers are written in Python with MongoDB storage. I was pleasantly surprised with MongoDB's query speed given I only index keywords. There are a small number of products in the database, but its still very fast.
To get the data I've created scrapers for each site. Currently, only Target and Amazon are implemented (with Container Store soon to come). Target and Container Store provide a product feed when you sign up as an affiliate. They FTP their product info to your site and I use that to parse the links. Using each link, I then scrape the page to store the relevant information for various products in the database. Amazon provides a nice API and there is a Python wrapper which makes it easy to query (python-amazon-product-api).
I guess you scrape websites of retailers to populate your database? Have there been any retailers which said you are not allowed to scrape their site for gathering product info, pictures and prices? (I can imagine that certain retailers only want their products to be showcased in the context of their own site)
Is storage furniture the best niche for this? I wouldn't know, since I'm not in the market for that, but I don't think the pain of comparison shopping in that subset is particularly high (I could be wrong of course). Another slight foible - how often does someone buy a dresser? You only have a brief window to acquire a user between the time they decide to buy but before they make the purchase. The polar opposite would be something like diapers where parents purchase on a regular basis. Are there any niches more like that?
These are all good questions, of which I don't have answers for (yet). I created this in response to my cousin (a professional organizer) complaining about finding products within certain dimensions. If she had this problem, I figured others might to. Only time will tell if this will actually provide value, but its certainly not all dressers.
Is this just furniture? I searched 3-6 x 7-9 x .5-1.5, which is about the size of art supply box I was looking for a while back, and it couldn't find anything.
No, this is not just furniture. It is everything in Storage & Organization categories from Target and Amazon. I'm not sure whether the art supply box fits that category, but I'd hope so. I do not have the dimensions for all products as they are not always formatted identically on product pages, and my parser is fairly dumb at the moment.
As people may notice, there are some issues with the dimensions search. For instance: how do you differentiate between width/height/depth when there is no real "front" of the product. The dimensions depend on the orientation the user expects, not necessarily what is stored in the DB.
These are both issues I hope to fix, but at the moment I still think it performs fairly well. (and will improve)
Many times I go shopping for something relatively expensive where I don't know exactly what I want. In almost every case I am disappointed with the typical online research/purchasing decision and wish there were way more intelligent vertical shopping engines.
Recently I can think of this happening to me for video cameras, regular camera, honeymoon spots, projectors, health insurance, apartments, outdoor camping gear, etc. Flight & hotel search has made real progress in the last couple years here but there's still tons of opportunity.
I can't say this category of items is super important to me, but I certainly welcome the idea of making purchasing anything online smarter/easier. With so many affiliate programs out there they should also lead to decent/good businesses.
But seriously, Ikea's EXPEDIT series should be in here, at the very least, and considering they have a whole "Storage Furniture" section, is a better target than Target.
Storage furniture:
Pantry
Wardrobes
Bed storage
Headboards
Nightstands
Dressing tables
Bookcases
Cabinets & sideboards
Chests of drawers
Clothes storage systems
Drawer units
DVD & CD furniture
Filing cabinets
Heavy duty storage systems
Shelving units
TV stands & media solutions
Wall shelves
I was planning to drive to Ikea tomorrow and was thinking about a search that would show me the options that fit the wall space in question.
Assuming this is to make some actual money, and not just a proof of concept -- sites like this live or die on the strength of their SEO. People need to find it when they're searching Google for storage products.
You need an individual product page with its own URL for every item, and you need a strategy for getting inbound links, including to those deep product pages. You need content on those pages - reviews and ratings (scrape Amazon, maybe, or product catalogues).
In other words, to make some cash out of this (and the five gajillion other sites you could set up in other niche verticals), you have to content spam the holy hell out of the Internet. Not something friendly to the HN crowd, but that's what works.
Once you've done that, and you have some traffic, you can sell CPC traffic to online stores in exchange for favorable placement. And once you've done that, you're probably making enough money to start measuring the expected value of a user for each inbound search term and buying more, paid traffic - especially when your SEO doesn't have you in position #1.
And then you'll have a comparison shopping site, which can print money - until Google updates its search engine and pulls the rug out from under you. Print the money, but put it in your pocket.
Immediate steps for expanding your product selection / hopefully making more cash - most of the major comparison shopping sites today have affiliate product feeds.
Well, back in the day, it was stuff like setting up blogs that linked back to your site, making somewhat-relevant blog comments elsewhere, submissions to directories, issuing 'press releases', making forum posts (especially with signature links), making somewhat-relevant contributions to wikis, 'contributing' to bookmarking sites like del.icio.us, providing somewhat-relevant answers on Yahoo! Answers, you get the idea. If you could stick a link in it, you stuck a link in it.
For all I know any or all of these tactics could currently be the search engine kiss of death - so don't do any of them without doing your research first. Search engines have probably gotten a wee bit smarter.
The other, more ethical way to do it is to produce great content (reviews, tutorials, photo shoots, etc.) that people might actually want to link to of their own free will, but this is time-consuming and hard to scale up.
21 comments
[ 3.1 ms ] story [ 66.1 ms ] threadTidyFinder is the first step towards that goal. I'd appreciate any feedback/criticism/suggestions that you can offer. I value the HN community's thoughts. Thanks.
App and scrapers are written in Python with MongoDB storage. I was pleasantly surprised with MongoDB's query speed given I only index keywords. There are a small number of products in the database, but its still very fast.
To get the data I've created scrapers for each site. Currently, only Target and Amazon are implemented (with Container Store soon to come). Target and Container Store provide a product feed when you sign up as an affiliate. They FTP their product info to your site and I use that to parse the links. Using each link, I then scrape the page to store the relevant information for various products in the database. Amazon provides a nice API and there is a Python wrapper which makes it easy to query (python-amazon-product-api).
As people may notice, there are some issues with the dimensions search. For instance: how do you differentiate between width/height/depth when there is no real "front" of the product. The dimensions depend on the orientation the user expects, not necessarily what is stored in the DB.
These are both issues I hope to fix, but at the moment I still think it performs fairly well. (and will improve)
Recently I can think of this happening to me for video cameras, regular camera, honeymoon spots, projectors, health insurance, apartments, outdoor camping gear, etc. Flight & hotel search has made real progress in the last couple years here but there's still tons of opportunity.
I can't say this category of items is super important to me, but I certainly welcome the idea of making purchasing anything online smarter/easier. With so many affiliate programs out there they should also lead to decent/good businesses.
But seriously, Ikea's EXPEDIT series should be in here, at the very least, and considering they have a whole "Storage Furniture" section, is a better target than Target.
I was planning to drive to Ikea tomorrow and was thinking about a search that would show me the options that fit the wall space in question.Your app is great -- just needs the goods.
I can see them picking it up if they haven't though.
It's a great idea IMO. Kudos.
Make the price adjuster control move in increments of at least $10 (maybe $100).
You need an individual product page with its own URL for every item, and you need a strategy for getting inbound links, including to those deep product pages. You need content on those pages - reviews and ratings (scrape Amazon, maybe, or product catalogues).
In other words, to make some cash out of this (and the five gajillion other sites you could set up in other niche verticals), you have to content spam the holy hell out of the Internet. Not something friendly to the HN crowd, but that's what works.
Once you've done that, and you have some traffic, you can sell CPC traffic to online stores in exchange for favorable placement. And once you've done that, you're probably making enough money to start measuring the expected value of a user for each inbound search term and buying more, paid traffic - especially when your SEO doesn't have you in position #1.
And then you'll have a comparison shopping site, which can print money - until Google updates its search engine and pulls the rug out from under you. Print the money, but put it in your pocket.
Immediate steps for expanding your product selection / hopefully making more cash - most of the major comparison shopping sites today have affiliate product feeds.
"content spam the holy hell out of the Internet"
How? (In what way would you go about doing that?)
For all I know any or all of these tactics could currently be the search engine kiss of death - so don't do any of them without doing your research first. Search engines have probably gotten a wee bit smarter.
The other, more ethical way to do it is to produce great content (reviews, tutorials, photo shoots, etc.) that people might actually want to link to of their own free will, but this is time-consuming and hard to scale up.