Launch HN: Meticulate (YC W24) – LLM pipelines for business research
Some background on “business research”: investment and consulting teams sink many hours a week into researching companies, markets, and products. This work is time-sensitive and exhausting, but crucial to big decisions like company acquisitions or pricing model changes.
At large financial services firms, much of this work is offshored to external providers, who charge thousands of dollars per project and are often slow and low-quality. Small teams lack the budget and consistent flow of work to employ these resources. We’re building an automation solution that brings a fast, easily accessible, and defendable research resource.
Meticulate uses LLMs to emulate analyst research processes. For example, to manually build a competitive landscape like this one: https://meticulate.ai/workflow/65dbfeec44da6238abaaa059, an analyst needs to spend ~2 hours digging through company websites, forums, and market reports. Meticulate replicates this same process of discovering, researching, and mapping companies using ~1500 LLM calls and ~500 webpages and database pulls, delivering results 50x faster at 50x less cost.
At each step, we use an LLM as an agent to run searches, select and summarize articles, devise frameworks of analysis, and make small decisions like ranking and sorting companies. Compared to approaches where an LLM is being used directly to answer questions, this lets us deliver results that (a) come from real time searches and (b) are traceable back to the original sources.
We’ve released two workflows: building competitive landscapes and market maps. We designed it with an investor running diligence on a company as the target use case but we’re seeing lots of other use cases that we didn’t originally have in mind—things like founders looking for alternative vendors for a product they’re purchasing; sales reps searching for more prospects like one they’ve already sold to; consultants trying to understand a new market they are unfamiliar with, and more.
The main challenges we’ve been overcoming are preventing quality degradation along multi-step LLM pipelines where an error on one step can propagate widely, and dealing with a wide range of data quality. We’re working hard on our next set of workflows and would love for you to give it a try at https://meticulate.ai and would appreciate feedback at any level!
41 comments
[ 3.7 ms ] story [ 108 ms ] threadI've heard a few investor types say something like "You know what's surprisingly fun? Popping an edible and making market maps"
Here is an example output: https://meticulate.ai/workflow/3b3fe891f16fc437acca87c0
It was really nice to go away for a few minutes and come back to this. Output is not perfect, but I wouldn't expect it to be at this stage.
I assume slide deck output is on the way?
I have a friend who looked into doing something similar but they couldn't figure out a way to get the cost low enough. This was like a year ago so I'd guess it's much cheaper now and you could do something like fine-tuning a smaller domain specific model on GPT-4 outputs.
Any ballparks on pricing / cost? What models are ya'll using?
Using GPT-4 and GPT-3.5 currently, and costs can be $1.50+ per request right now (have been benefiting from YC cloud credits!). Definitely steep at the moment but we expect costs to come down at least 10x over time.
Not super clear on pricing yet (only a few weeks post-launch)
> $1.50
Thanks for sharing! I think they were ~10x that but hadn't done a ton of optimization yet. To me, having a swag at cost makes these tool demos a lot more interesting because you can start figuring out what types of businesses you can/can't build with them.
Definitely... we're lucky to be in an industry where there's a lot of money at stake
I'm actually working with a number of companies who are exploring this space. Many of them are in the current YC batch. We're helping to provide the core business data, then we're exploring how we can leverage our scraping infrastructure in other ways to bring costs down.
I'm open to chat if you're interested: michael (a t) bigpicture.io
What about quality of results? Are you measuring that too? Did you do so for the traditional reference practice? Using what sort of methodology? How did it your technique compare in quality? What kind of errors was it most likely to make? What techniques have you devised for spotting those errors? Are they the same kind of errors that users would experience when outsourcing? Are the errors easier or harder to spot for one than the other? Are they faster to remediate with one?
I see a clever concept but given the state of LLM's and the nature of how they work, I don't know that nominal cost and speed differences are really enough to sell on. Not for something "crucial to big business decisions." I'd want to know that my failure/miss rate is no worse than when outsourcing and that my net cost and time (including error identification and recovery) still end up ahead. I don't see either of those vital issues touched upon here.
Our method of evaluating quality is not super systematic right now. For this competitive landscape task, we have a "test suite" of ~10 companies and for each we have a sort of "must-include", "should-include", "could-include" set of competitors that should be surfaced. We run these through our tool and others and look at precision and recall on the competitor sets.
In terms of errors, right now our results are a little noisy, since we're biased towards being exhaustive vs selective. There are obviously irrelevant companies in the results that no human would have ever included. Our users can fairly easily filter these out by reading the one sentence overviews of the companies but it's still not a great UX. Actively working on this.
Not that an AI can’t do that too!
Though I may hang up…
Their clients want to know that the research report was written by a real person and not a bot.
But that doesn’t mean it actually has to be written by a real person and not a bot.
I've been in a meeting with a senior executive, PhD, who argued that Total Addressable Market for a supply chain tracking software (basically just a database) is same as the market for the goods it tracks.
It is as if a startup which prints restaurant menus would claim the value of all food sold in restaurants as their TAM.
And this person was otherwise reasonable...
So, yeah, never underestimate the power of Natural Stupidity.
Perhaps
If I have want to help you in your roadmap- specifically around Find Companies, how can I contribute?
I told the boss that actually that is spelled meticulous. They said that he was pretty sure that it was spelled meticulate. I got the job offer, but I refuse the job. The thought of working as and inferior for somebody who would sucker down on the spelling "meticulate" was just way too much for me. What other harebrained bull** would they have me commit to?
Not sure why I bring that up here. It's a personal experience that makes me have no interest in this product whatsoever. Which is not rational. I guess I'm just a hater and I wanted to share this funny story while also maybe causing a loss of revenue to a random company I have nothing to do with.
Cheers!
We have millions of customers to serve and billions of dollars at stake, but you're going to nitpick over this guy's colloquial word misuse?
Come on.
This person had so many red flags you could have called him Stalin. A bumpkin with missing teeth and a meth habit. I was applying as a mechanic in 2008. I didn't take the job because I didn't want to get f**ed around. So how about you give me a break? :wink:
De ce este fericit omul chel? Why is a bald man happy?
Pai, părul îi crește spre interior și îi gâdilă creierul! Well, his hair grows inwards and tickles his brain!
- for each company, find their main product lines and their respective market share (i.e. my firm does multiple things such as architecture, data governance, data engineering, data analysis, etc) but we're not necessarily the best on every type of service. I'm assuming that's a common pattern (e.g. some automotive companies are great at trucks but not the best at smaller cars, etc). - get basic financial stats (revenue, market share) - (bonus) get time-dependent stats (what was their revenue like 3 years ago ? What's the YoY growth ?)