HN Explorer

Launch HN: Webhound (YC S23) – Research agent that builds datasets from the web

112 points by mfkhalil ↗ HN
We're the team behind Webhound (https://webhound.ai), an AI agent that builds datasets from the web based on natural language prompts. You describe what you're trying to find. The agent figures out how to structure the data and where to look, then searches, extracts the results, and outputs everything in a CSV you can export.

We've set up a special no-signup version for the HN community at https://hn.webhound.ai - just click "Continue as Guest" to try it without signing up.

Here's a demo: https://youtu.be/fGaRfPdK1Sk

We started building it after getting tired of doing this kind of research manually. Open 50 tabs, copy everything into a spreadsheet, realize it's inconsistent, start over. It felt like something an LLM should be able to handle.

Some examples of how people have used it in the past month:

Competitor analysis: "Create a comparison table of internal tooling platforms (Retool, Appsmith, Superblocks, UI Bakery, BudiBase, etc) with their free plan limits, pricing tiers, onboarding experience, integrations, and how they position themselves on their landing pages." (https://www.webhound.ai/dataset/c67c96a6-9d17-4c91-b9a0-ff69...)

Lead generation: "Find Shopify stores launched recently that sell skincare products. I want the store URLs, founder names, emails, Instagram handles, and product categories." (https://www.webhound.ai/dataset/b63d148a-8895-4aab-ac34-455e...)

Pricing tracking: "Track how the free and paid plans of note-taking apps have changed over the past 6 months using official sites and changelogs. List each app with a timeline of changes and the source for each." (https://www.webhound.ai/dataset/c17e6033-5d00-4e54-baf6-8dea...)

Investor mapping: "Find VCs who led or participated in pre-seed or seed rounds for browser-based devtools startups in the past year. Include the VC name, relevant partners, contact info, and portfolio links for context." (https://www.webhound.ai/dataset/1480c053-d86b-40ce-a620-37fd...)

Research collection: "Get a list of recent arXiv papers on weak supervision in NLP. For each, include the abstract, citation count, publication date, and a GitHub repo if available." (https://www.webhound.ai/dataset/e274ca26-0513-4296-85a5-2b7b...)

Hypothesis testing: "Check if user complaints about Figma's performance on large files have increased in the last 3 months. Search forums like Hacker News, Reddit, and Figma's community site and show the most relevant posts with timestamps and engagement metrics." (https://www.webhound.ai/dataset/42b2de49-acbf-4851-bbb7-080b...)

The first version of Webhound was a single agent running on Claude 4 Sonnet. It worked, but sessions routinely cost over $1100 and it would often get lost in infinite loops. We knew that wasn't sustainable, so we started building around smaller models.

That meant adding more structure. We introduced a multi-agent system to keep it reliable and accurate. There's a main agent, a set of search agents that run subtasks in parallel, a critic agent that keeps things on track, an...

46 comments

[ 3.0 ms ] story [ 71.9 ms ] thread
Oras
When I read your description, I thought, "This is just like the Exa dataset, no?" but then I gave it a try, and I am genuinely impressed.

Great decision to make it without a login so people can test.

Here is what I liked:

- The agent told me exactly what's happening, which sources it is checking, and the schema.

- The agent correctly identified where to look at, and how to obtain the data.

- Managing expectations: Webhound is extracting data Extraction can take multiple hours. We'll send you an email when it's complete.

Minor point:

- There is no pricing on the main domain, just the HN one https://hn.webhound.ai/pricing

Good luck!

giancarlostoro
This is how I use Perplexity, I will have to give this a try, I am always on the lookout for newer tools.
nc
Pretty cool, how does it compare to Parallel?
jackienotchan
AI crawlers have lead to a big surge in scraping activity, and most of these bots don't respect any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits, user agents, etc.).

This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).

Does Webhound respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?

Poomba
When it finds a page that has the results I'm looking for in a particular website, does it paginate through all of them? When I searched for "stores that are on the Faire marketplace" it seems to return just the first page of results without paginating through all of them.
whinvik
Nice, this looks interesting.

> It uses a text-based browser we built

Can you tell us more about this. How does it work?

lubujackson
Great job on the launch. This is fills an exact need my company has, and your UI is fantastic. One nit, it would be nice to be able to manually edit the schema instead of only interacting with the LLM.

I am concerned about your pricing, as "unlimited" anything seems to be fading away from most LLM providers. Also, I don't think it makes sense for B2B clients who have no problem paying per usage. You are going to find customers that want to use this to poll for updates daily, for example.

Are you using proxies for your text-based browser? I am curious how you are circumventing web crawling blocking.

admn2
Can I give it a list of products and have it provide data enrichment for each of them?
nextworddev
Is there an open source alternative to building this type of UI / job scheduling backend that works for grids?
codingdave
It looks promising overall, and I can see where this will save a ton of time for some types of data gathering. But I'm also seeing quite a lot of queries and processing going on, and it has only found about 3% of the data that I know is publicly available, after 20 minutes.

It does say that extraction can take hours, but I was expecting it would be more of an 80/20 kind of thing, with a lot of data found quickly, then a long tail of searching to fill in gaps. Is my expectation wrong?

I worry for two related reasons. One, inefficient gathering of data is going to churn and burn more resources than necessary, both on your systems and on the sites being hit. Secondly, although this free opportunity is an amazing way to show off your tool, I fear the pricing of an actual run is going to be high.

artembugara
Congrats on the HN Launch!

It's probably the best research agent that uses live search. Are you using Firecrawl, I assume?

We're soon launching a similar tool (CatchALL by NewsCatcher) that does the same thing but on a much larger scale because we already index and pre-process millions of pages daily (news, corporate, government files). We're seeing so much better results compared to parallel.ai for queries like "find all new funding announcements for any kind of public transit in California State, US that took place in the past two weeks"

However, our tool will not perform live searches, so I think we're complementary.

i'd love to chat.

mips_avatar
This is really cool, I could imagine a number of ways I'd want to incorporate this into my data pipeline. Your agent is actually doing a great job of finding useful sources, though I wish there was a way it could really dig into some of these sources and pull hard from them. For my query it actually found some cool sources I didn't know existed, but it only pulled a single query from them. I'm now thinking I should go and write a custom scraper for those sources to actually get the full data I want.
arjunchint
Since it looks like you are built on FireCrawl, and FireCrawl itself has similar products like FireEnrich, how do you see yourself maintaining a differentiation and compete with FireCrawl directly, if they just decide to copy you?

As an aside, we are about to launch something like similar at rtrvr.ai but having AI Web Agents navigate pages, fill forms and retrieve data. We are able to get our costs down to negligible by doing headless, serverless browsers and our own grounds up DOM construction/actuation (so no FireCrawl costs). https://www.youtube.com/watch?v=gIU3K4E8pyw

jquaint
Used this in a "vibe coding hackathon" the other day. Works really well!
_false
I found the ability to stop and clarify a task in "one-shot" mode impressive. In my original prompt it misunderstood MCP to stand for Medical Care Plan. I was worried I wasted a generation but being able to stop and clarify fixed it.
4ndrewl
Congratulations on the launch. I'm currently using it to run a very specific task that I was thinking about just earlier today. Will let you know how it gets on.
zachmoore
VC will want this tool for creating a competitive landscape. Startups should do it to but it will replace hundreds of hours of junior analyst time. For better or worse. THis is going to be much better research than what a junior analyst could do. Or maybe jr analyst will pay for it.
rick1290
Are you able to share some of the tech stack choices behind the scenes? React, fastapi, django, go, pydantic ai, ag-ui, firecrawl, vertex ai, etc. Would love to see the tools and frameworks others are deploying with.
cco
As someone that works in the auth space (Stytch), this is the first time I've _ever_ seen someone's landing page be a login screen.

I have to ask, how's that going? Genuinely curious to know!

mfkhalil
Fair point, most of our users have come from referrals/word of mouth so it hasn't really been an issue for us, but you're probably right that we should have more information on the landing page
cco
Oh definitely not trying to make a point, I'm really just curious about how it is working out in case it is something I should update in my recommendations to customers.

Seems like y'all are doing well with it!

piecerough
Have you tried 2.5 Flash Lite to cut costs further?
mfkhalil
We have, 2.5 Flash is about as small as we've been able to go while still delivering consistent results.
ashivkum
this is very polished and intuitive
bananamansion
“I see you’re dynamically generating schemas per dataset. Have you explored using a domain ontology or building a reusable ontology layer so the agent’s outputs can interoperate more consistently across projects
ukuina
Awesome platform.

I wanted to upgrade!

But your "upgrade to Pro" button on the Account page gets stuck on "Processing..."