Ask HN: My GPT project has become a local hit but I can't afford the bill
I'm an HN user for 5-6 years but I wanted to use an alt for this as it's a bit sensitive.
Last week I created a ChatGPT clone for my city. It's basically a laravel app hooked up to GPT3.5-turbo with a custom prompt that makes local references and talks in the way that people in my city talk.
It's been a surprise hit and now I've had 20,000 chats coming through in one day - people where I live really seem to love it
The problem is that I likely can't afford to keep hosting this. It's cost me $50/day for one day, and Adsense doesn't allow 'chat apps', so I'm at a loss at how to cover the bill for this app. I've already optimised the prompt and reduced the number of tokens I'm sending
The app a joke really, but it's a local joke that seems to be quite popular and connecting with my city. Should I try to raise donations? Is there an advertising provider I could use that would potentially cover the costs? Is there an alternative to OpenAI that is comparable to GPT-3.5 that I could self-host cheaper?
Any and all advice appreciated. I don't care about profit - I just want to keep the app online so people can enjoy it.
139 comments
[ 4.3 ms ] story [ 200 ms ] threadI bet you could find local sponsors who can pay you on a monthly basis
It might be hard to implement it in a week but a hastily put AWS & stripe solution might work.
If you do ever implement it, do mention it on HN, I would definitely buy couple of slots.
> The problem is that I likely can't afford to keep hosting this. It's cost me $50/day for one day
If your traffic continues to increase at a rapid clip, you may want to look into investors for your project. You could still keep it mostly nonprofit if you think it is adding a lot of value for people.
Hugging face and others should help verify if anything can, or how close it is.
Alternatively you could try swapping out an open source model like llama on the back end (possible licensing issues here).
Another option would be to seek out ads for local companies directly since it is an explicitly local product. Cold calling would probably be the best place to start. You could also pay attention to who is paying for billboards in your area and try contacting them.
If done right, it won't even be adblocked (because you can do it with no tracking at all).
Depending on the size of the city, and the queries, you should have a good idea where to look. If not, check the local church bulletins, the local softball/school fields, and anywhere else you find local business advertisements.
Sell it as a 'Google ads back in 2002', frame it in terms of high risk high reward. Sure its unproven, but back in those days, a lucrative click may have only costed $1,$2. When search ads got fully proven, they started costing $100/$200.
And you could try some alternatives to AdSense like:
Ezoic, MediaVine, or Adthrive
Another online possibility: throw up some Amazon (or similar) affiliate links to products and services, maybe even based on what people type (if that's accessible).
And next to them "opt out of the ads" link that goes to a donations website, to get fancy. Helps clear the conscience about having the ads in the first place.
But the donations route could also be a fun way to lean into the local element of it if the cost remains high. Could have a simple line of text when the page loads that thanks one of the donors for that day.
You could also compute embeddings for the questions (don’t have to be OpenAI embeddings), and reuse the answer if the question is sufficiently similar to a prevously asked question.
Wouldn't it be almost impossible to hit a duplicate when the users each form their own question?
Another issue I see is that these chat AIs usually have "history", so the question might be the same, but the context is different: the app might have received "when was he born", but in one context, the user talks about Obama and in another, she talks about Tom Brady.
If there are ways around these issues, I'd love to hear it, but it sounds like this will just increase costs via cache hardware costs and any dedup logic instead of saving money.
The embeddings approach would increase the likelyhood of finding the same question, even if phrased slightly differently.
Regarding context, that should be a part of the input for the embeddings.
I assume only a small percentage of users would put in the same prompt twice, and even then, why would they be upset at getting the same response?
Call them up one by one and say "hey I run [project] that everyone has been using lately, would you be interested in having your business name listed as a sponsor? It'll be shown on 1/4 page loads which is currently 5000 times a day. $500/month".
You'll know in like an hour if there's any advertising potential.
One day two salespeople contacted me and suggested taking over ad sales, and managed to sell thousands (I think it was somewhere between $5k-$10k/month, but it was a long time ago) a month in ads for the site. This was for a city with a population of 200k.
Don’t think just because you are very smart you can do sales. This is like business people trying to “learn coding” so they can put together some huge product. It doesn’t work well and isn’t scalable.
Try a little, see if you can grow.
All it took was seeing how professionals always do the job far better to change my mind. It doesn’t matter if a dev picks up some skills, if you plan to be a dev then a dev is all you’ll ever be, focus on the development and let sales worry about sales.
I see a lot of founders struggle with delegation. It seems they would rather juggle things they don’t do well instead of learning to trust and onboard people who can do things they don’t specialize in.
OP has a product he enjoyed making, is providing real world utility, and that he isn't trying to turn in to a billion dollar company. The obvious solution is for him to pick up a phone, send out a few emails, and generally reach out to the local business community instead of sitting back and waiting to see if it will magically make money and he can hire a sales person to do those tasks instead for his hobby project that has grown too expensive for a single person but would barely qualify for justification at a business.
I get that once a business grows to a certain point it needs "professionals" to do some of the tasks. Ideally start by hiring people with skills you don't have. Sales is a skill.
That said many people have more the one skill. Context matters. I write libraries for programmers. I attend trade events where I talk to, and sell to programmers. I do a lot of marketing in this space. I have an authenticity that a non-programmer does not have, and I believe that is reflected in our sales.
In this case the developer is perfectly capable of picking up the phone. They may or may not maximise potential income (but that's not their stated goal). If there's enough demand then they can pivot to a sales person later on.
Incidentally, I would likely figure out who does the marketing for the local businesses you see. They buy the ads, not the business directly. They're easier to talk to (they -want- more exposure opportunities) and they may bring multiple customers to the table.
OP can totally give some basic marketing a go. The product isn't turning even creating revenue - hiring marketing now, without at-least testing the water himself would be silly.
Lots of devs start in sales because those jobs/careers are incredibly prevalent. Think basic sales jobs like retail, call centers, etc.
Lots of business people unironically "learn coding" to great success. Some even build "huge products" that solve big problems.
---
Anecdotally, I know people who fall into all of these categories. I don't think we should condemn people to one function/talent... and, I don't think we need to talk down to cross-functional devs or business people.
> I've already optimised the prompt and reduced the number of tokens I'm sending
Lets assume you've found, or will find, all possible technical optimizations for your AI spend on ChatGPT.
You can:
* Ask for donations
* Accept a $50/day cost for X days as a market research budget to find a monetization plan
* Have some "first 5 questions are free, next are paid (or whatever)" to rate limit cost while maybe raising revenue
* Look for a common pattern of questions to see if there's some value / insight into your users that can be used as a source of revenue
* Find a cheaper LLM that supports your needs while being serviceable
* Pull the plug because its a joke, claim credit for it, and take the learnings onto the next thing
edit: grammar & formatting
Either way, there's definitely an opportunity for funding. In the case of Glasgow, there's a huge populous out there that are descendants of folk from the city/country taking pride in it (arguably more than a bunch of us that live here!) and like to get in on the banter.
That's certainly an avenue worth exploring if there's a similar relationship with folks or families of folks that emigrated away, but feel a strong connection.
Edit: You'll have to excuse my unfamiliarity with LLMs, but if you're storing messages and responses, and if there's a way to check that a message has already been asked, would it be possible to replay that (at least on the first message) instead of running a new query against the API? If they then engage further, then it would send the job out?
https://thetab.com/uk/glasgow/2023/02/15/glasgow-uni-working...
https://www.investglasgow.com/
Also, maybe you can use sometime kind of caching combined with some mbeddongd search to serve the previous response, if the input is similar above a certain threshold.
Switch out the backend, there are open source alternatives, and the results could be "good enough".
Get a list of local sponsors, offer them 1k ad placements for X dollars/euros/etc... , and inject them into your prompts.
As it is a joke site, I don't believe people will mind if the chat has ads, especially if you mention somewhere that it's financially hard to keep the site up.
One option to get the costs down would be to use a local LLM. It isn't likely to be as fast/good as GPT-3.5 but at least would give you a fixed cost per month (hosting).
If you had local hardware, given that traffic is going to be pretty low, if you have decent bandwidth at home and some hardware, you could even host there.
It's not that cheaper if you want similar level quality and inference speed. For commercial usage, OP will require training his own model since no good commercial alternative exists.
As opposed to API based where you pay more per use.
Remember this isn't a commercial service that OP is creating, it's an amusing Chatbot, they have no revenue, so fixed costs are likely to be a better model.
Could be a feature rather than a bug? :)
Not impossible, but tricky.