Launch HN: Promoted (YC W21) - Search and feed ranking for marketplaces
For buyers, this is finding what you want quickly. For sellers, this is finding an audience despite competition. For marketplaces, this is increasing total conversion rates and new seller success rates. Matching buyers with sellers is the engine that drives marketplaces, and doing it better is how marketplaces grow.
Deciding who sees what in a list on an app is the core business of the biggest, most profitable companies in the world: Facebook, Google, Amazon. We have a decentralized, identity-free solution that’s more efficient for sellers and a better experience for users. Today, we optimize within marketplaces, but we believe that our approach can eventually match buyers and sellers across many apps and turn into a network between top marketplaces. We aren't an ad company. We use technology from ad tech to make marketplaces work better.
We met at Pinterest ads engineering. Previously, we helped build ad systems at Facebook and Google respectively. We learned that marketplace companies were all trying to build ads and hire ML engineers, but we knew from experience that most of these efforts would only have easy wins at first, then stall with huge costs and user loss. To build things right, we decided to form our own company. We started with just ads for marketplace, but quickly learned that our tech could produce much better marketplace search, so we expanded to that. It makes sense: every listing in a marketplace is something advertised for sale. They’re just not called ads.
Ironically, bad ad tech is easy. Anybody can sell a dumb banner, and this makes early money fast—but it’s bad money, because it’s a bad long-term strategy. Even with hiring awesome engineers from Facebook, Google, or Amazon, you still need to build a system that kills the easy money, doesn’t drive away users, keeps sellers happy, and maximizes sales in the long run. To do that, you have to go the hard route. You have to generate all listings in real-time: no caching. You have to try to deliver everything, not just top content, and explain why to sellers. You need to solve for how much you want to show something, in other people’s dollars, and it has to be correct all the time. Your inventory is always changing, anything can be shown to anybody, and people game your system. Models must always be evolving and depend on external data and market dynamics, so quant SRE and DevOps are crucial. Measurement has to be correct or you could be sued, or at least produce poor results. You need a manager tool so that busy people can run their campaigns and test how they perform in real-time.
Our tech has three parts: (1) Metrics: We log impressions, clicks, and conversions in our web and mobile client SDK. In our backend, we attribute conversions to impressions and join and aggregate data in real-time to power delivery. (2) Delivery: We use machine learning to predict user behavior to decide what to show. This is the “The Algorithm” famously from social media applied to e-commerce. (3) Manager: Sellers can run their own listings like ads, even if they are not ads, with self-service real-time reporting and A/B testing. This makes listings better by helping sellers improve themselves versus only sorting listings as they are today.
We like to say that we’ve built Paul Graham’s revenue loop, advanced twenty-five years (http://www.paulgraham.com/6631327.html).
We run both “organic” commercial search and feed and ads. Our insight was that these are actually the same systems for commercial search. Existing recommendation and search systems don’t run a...
36 comments
[ 4.9 ms ] story [ 80.2 ms ] threadThe idea of being able to project the unofficial market of search engine results (SEO/SEM on Google), and explicitly allowing marketplaces to actually commodify the search result space is fascinating.
Tell us about GSP!
The mysticism around GSP (generalized second-price auction) for ads is absurd. In twenty years, for “p(click)*bid”, the p(click) factor has advanced from a simple ratio to huge neural networks. But bids? “Sort and take the second price.” "Somebody once won a Nobel Prize." Long story short, GSP doesn't have many useful properties in practice except that it's easy to implement and compute and it's a "standard."
Major problems with GSP are: 1) Useful economic properties depend on non-repeated, "stable" auctions of 1D ordered lists, which doesn't describe most modern media. 2) GSP gets mixed with a "complementary bid" to control for user quality, which also complicates any theoretical properties 3) It's still complicated to understand and requires layers of control systems.
We originally started as “Algorithmic Auctions” to solve ad auctions fairly, but we didn't find a market for this.
1) GSP doesn't promise any specific price. FPA promises "the price you bid." If that's not what people are paying by simple math, it will be confusing. That hurts trust. This could happen if you have a user quality control system that penalizes poor quality. GSP gives you a second control (price) in combination with delivery volume and placement to manage user experience.
2) People expect GSP. Claiming FPA is an admission that you need to build an autobidder system versus letting people discover this for themselves.
Like, the trouble with FPA is that it extracts "too much" money from the bidder, as they pay their marginal cost. GSP (second price more generally) does have the nice property that you'll always pay less than your maximum bid, which is the margin (assuming you're bidding your maximum profitable LTV).
Also, congratulations Andrew, I'm sure you guys are gonna do incredibly well (and if you have problems in this space, I strongly recommend giving promoted.ai a look).
In a GSP, I bid $1. I'll pay the second price (not really but for purposes of clarity assume it) which was $0.90.
In this case, I always (almost) pay less than my maximum bid, which allows for margin given to me (assuming true value and a bunch of other unrealistic things) rather than in the FPA, where I pay $1, assuming my bid is highest.
I'm not talking in incentive-compatibility terms, rather in terms of where the "margin" goes.
https://medium.com/promoted/when-goog-fb-is-bad-for-your-per...
Yeah, as I think we've discussed before, GSP is a lovely economic model that doesn't have much applicability to the real world.
It's a great thing to explain to clients though, and it makes it look like the platform is not overcharging them, which is probably why people still keep using it.
Today, how much of your attention is spent between how many apps? I bet that the sum useful attention across many different apps exceeds attention on Facebook. But why does Facebook dominate performance marketing? How can these apps and their users find each other in a better way without aggregating into a centralized Big Tech company?
We're passionate about finding the answer to that. It has to start with making individual marketplaces run better by deeply understanding them.
how did you settle on the positioning as being for marketplaces?
sounds like this would be useful for any company that has a feed, search or recommendations like any retailer or publisher.
Promoted would be interesting for this new marketplace feature I'm working on right now, but this minimum makes it impossible to try it out. Any thoughts on pay-per-use pricing? Or is this only interesting for large established marketplaces with lots of data to train on?
$30k/mo minimum is roughly the cost of 1 FTE. If you're big enough to start hiring a team of specialist roles just for search and ML, then we're a better fit today.
Email me at ayates@promoted.ai, and let me see how we can help!
[0]: note that google and facebook try to solve this problem broadly by seeing more and more of your behavior and trying to better infer intent with essentially unlimited resources, and basically fail at it.
Part of the way we solve this is NOT with machine learning, but with tools to empower internal merchandizing teams and product teams in a way that fits nicely with the automated system. If you're a on a search team and had to goof around with elastic-search scores or hack in inserts for a new market merchandizing team, you've felt this pain. The path forward is ML + human expertise, which is better than either alone.
> basically fail at it
Our goal is to figure out "why" and "how to make it better." These are $T companies and dominate all performance ad spend. It's hard to think about such big numbers. One problem is that they start with crappy inventory (people who want to advertise) and it's really hard to actually _do_ something on these platforms with promotions that you do see. On marketplaces, you don't have these problems as much, because everything is already vetted and you can convert in the marketplace. That's why you're there, so it's a great experience.
So, we start from there, media matching that people love, and work backwards.
Pretty curious about how well you can build a generalized solution and still get uptake from SMBs. I'd think that marketplaces would tend to want to keep that kind of expertise in house, but I guess my experience shows that there are some less eng-focused companies that would pay for that kind of thing.
> When we started, we were shocked at how little marketplace companies measure anything.
For the travel company mentioned above, our model was built on hotel bookings only. That is, they gave us a list of every booking made on their platform, and then at search time they gave us the parameters of the search (city, dates, incoming flight) and hotel availability, and we were supposed to return the ranked list of hotels. Not in that training set: anything about unconverted searches, what hotels were shown to searchers at any point, or anything about the customers. Again, our model built on super sparse data outperformed their ranking by ~15% over a period of multiple years. (We had even better results over shorter time frames with another customer that never signed a contract.) I kept on telling people that these (Europe-based) companies could have signed a reasonably competent data scientist for like $50k/year, outperformed our models within 6 months or so, and saved themselves 6 figures/year.
I would not ;)
> less eng-focused companies that would pay
Actually, our experience has been the opposite. The more sophisticated the engineering team, the more they recognize how big of a pain unified search ranking is to build and maintain, and the more they appreciate what we offer.
On the forever "model-bakeoff": our approach is to include all existing models as features into an omni-model. If you are experienced in ML ops, you should be cringing, but we pull it off because from a customer development standpoint, we never want to be competing with some other new technique. Instead, we want to have a big ball of systems and progress is always "add more stuff." Then, the business and product teams can focus on how they want their product to work versus technical details of specific recommendation systems.
> I would not ;)
Ha, yeah, that was unclear. My comment was targeted more towards the general HN audience, because _I_ was pretty amazed when I got into that business. "Wait, so you've built the integration, you've built the metrics, you've run A/B tests at various levels for months and months that show beyond a shadow of a doubt that you'll make more money, and you don't want to sign...why?"
>> less eng-focused companies that would pay
> Actually, our experience has been the opposite. The more sophisticated the engineering team, the more they recognize how big of a pain unified search ranking is to build and maintain, and the more they appreciate what we offer.
I see now from the other comments that you're targeting a higher point in the market than I originally thought. Still somewhat surprised that companies at that level would buy rather than build, but I suppose my view of the space is biased since I was only working at places that would want to build that in-house.
> our approach is to include all existing models as features into an omni-model
Makes sense to me if you're trying to be a generalized solution. I assume there's no crossover between any of your customers and you have to build a unique model for each of them?
Candidly, this is our biggest challenge. If we can surmount this, we'll be huge. This is also why care so much about "upmarket brand". Decisions aren't made by numbers alone. We need to show that we're the smarter way to grow faster because other top companies are doing it.
This is a big bet. Is there a threshold for engineer TAC in search, discovery, and ads? $1M? 2? More? Hire a dozen? One hundred? I've seen these numbers. They happen because the potential value is there and the VC funding is there, but I haven't always seen the delivered engineering results. Us? We've been there, done that. We'd like to focus on delivering the results, and we know from experience that's going to happen better from the outside.
> no crossover between any of your customers
The model architecture and infra are the same, but the literal weights in memory are totally independent. Most of the heavy customization is in allocation rules and blender, and we have a DSL for that. https://github.com/promotedai/schema/blob/main/proto/deliver...
I've seen this dynamic in sales as well, with people complaining about how much they're paying in commission, seemingly forgetting that the commission comes out of sales that you wouldn't otherwise have
How do you overcome the customer's mindset of build vs buy, and having an internal competition/enemies from your customers?
It might be a more straight-forward decision when the customer is starting from scratch. However, when the customer is invested in their in-house solution, what does it take to convince them to try your solution?
1) Win ICs: Do the "crappy" work of running marketplace search really well. This is ops, data logging and correctness, A/B testing, and managing the complexity of requirements from all competing teams who want to manipulate search results and boost things. These are things that backend search teams usually don't love, but we solve their problems so that they can focus on their expertise and ship features.
2) Don't Compete, combine: Our approach allows us to combine all competing recommendation systems together into a unified model. There is never a this-or-that decision, or a feeling of losing out. This also applies to other vendors. This is a pain for ML ops, but it's worth it. From an ML approach, mixing different systems typically outperforms any component system so long as you have the infra and parameter complexity management to handle them.
3) Build a brand of being the best: Not everything in big companies is engineering experience and metrics. Decisions get made when you're the hot solution that the cool people that you want to be like use. We deliberately focus on working with hot marketplaces and hiring awesome engineers with top experience to built this brand.
Anybody of any size can use our open source mobile SDK: https://www.promoted.ai/client-metrics-libraries