Launch HN: Sidekick (YC W23) – AI support bot for developers
Or if you prefer to not sign up to our Slack, here's a demo video showing the same thing: https://youtu.be/fPhP1325RkI
We’re in the process of making everything open source (there are some contractual issues we’re working through), but our client side code and basic infra is here: https://github.com/ai-sidekick/sidekick.
Providing technical support to developers has been expensive for companies because they need to hire skilled engineers to do it. We’ve seen community support channels with a 2000:1 ratio of developers to support engineers - there’s no way every question will get answered. We built Sidekick to make this much easier. It’s particularly helpful for open-source companies/projects because many OSS communities have a lot of people asking questions, but hardly anyone helping troubleshoot.
We integrate with Slack and Discord, since that’s where developer support is already happening. On the backend, we use Weaviate to index the data and OpenAI’s text-davinci-3 model to generate responses.
In addition to answering questions, Sidekick can also update .md files automatically with new information. When someone reacts to a message in Slack with the emoji, Sidekick will use Weaviate to find the part in your documentation that’s most related to the message, then use GPT3 to merge the new info into the documentation. Finally, it will submit a pull request on Github with the changes. We saw that devrel teams are already making product announcements and helping users troubleshoot common issues in the community, so we built this feature to save them even more time.
We use GPT for generating the responses and new documentation, but are relying less and less on it after learning that you hit a ceiling on answer quality very quickly by using only GPT and prompt engineering techniques. Here’s some of what we learned trying to prevent hallucinations in our answers: https://medium.com/@jfan001/how-we-cut-the-rate-of-gpt-hallu...
What we found makes a much bigger difference is the breadth and quality of the content you can search through, which is why we now rely a lot more on cleaning and annotating data, which yields far better results when combined with prompt chaining. For example, instead of naively chunking data into 1000 token blocks, we parse the markdown into semantically meaningful sections (e.g. paragraphs, lists, code blocks) and tag the content with the header name and document name so it’s more likely to surface for searches that are match for the section it’s from, even if it doesn’t exactly match the content in that chunk.
One fun thing we also learned is that when Sidekick gets added to a #help channel, people who otherwise wouldn’t ask questions start using it. It turns out, there are a lot of “lurkers” who come to these channels to find answers, but don’t want to bother anyone with their issue. Adding a tool that they can get answers from instantly brings these people out into the community, giv...
71 comments
[ 3.3 ms ] story [ 146 ms ] thread1. How exactly does this automatically keep docs up to date?
2. Does sidekick work well across multiple repos? (my org builds client facing apps so we have about 30 repos)
3. Is the doc auto updating expected to be in a particular format? My docs are very disorganized throughout all of these projects since I inherited a lot of codebases.
Overall looks really interesting. If you can actually deliver on the use cases you describe I think this will have value to a lot of people
1. If someone reacts to a message in Slack with the :memo: emoji, Sidekick will take the contents of the message and search for a chunk in the docs that closely matches the content. Each chunk is tagged with the source url on Github. Then it uses Github's Octokit client to pull that file, parses out appropriate chunk, and use GPT3 to merge the old content with the information from the message. Finally, it'll submit a pull request to the repo where the docs are stored with the changes.
2. Yep! It can ingest docs from multiple repos and link out to docs across all of them depending on which is most relevant. You can also add custom logic e.g. restricting it to search through certain repos for specific channels
3. It works better if there's a consistent structure throughout the docs, but it will try its best to merge info into docs that aren't structured. The risk here is that if you let Sidekick go wild it'll lead to very haphazard looking docs, so some curation is still needed.
[0]: https://news.ycombinator.com/item?id=35181139
The only devs I know on discord are crypto and AI generators.
I see it used in Rust, Gamedev, AI/ML (as you say), DevOps, JavaScript, ...
[1] https://onejs.com
I rarely see alive Glitter community, but I see plenty of active Discord communities, so active that I have to leave some of them. Every time I see "ask us on glitter", I'm already made peace with never getting an answer.
I'm not sure why it needs to be focused on developer docs vs all docs - any reason you decided on that niche?
I'd suggest creating a "demo" slack AND discord to try out. Is the demo slack working as intended? I see "Hey there! Looks like I'm not set up yet for this Slack workspace. Please send over a link to your documentation to founders@getsidekick.ai and we'll get you set up!"
And even better - make it so individuals can use it for legal documents.
https://media.makeameme.org/created/SAM-SHUT-UR.jpg
I tried to do this decades ago in hollywood (top media company) using DAM/AssetMgt tech and ran up against the entire attorney corp being protective of their deal notes/documents both for confidentiality with the talent and to monopolize their connection to that talent (only they can negotiate) aka gatekeepers. They purposely wanted "islands" of data. Likely similar for any legal outfit in a global 2000.
And yep we're running into some issues the new Slack we set up for public testing. Sorry for the inconvenience, we should have tested it more thoroughly. It should be up in the next hour or so.
I think just like (imo) much of the value of the data science craze of the last 10 years came not from the final analysis of the data but from inducing businesses to start measuring and storing operational metrics from every corner of the company in a structured way in the first place, much of the business value of ChatGPT will come from things like forcing businesses to start writing and maintaining good quality internal documentation.
The journey of getting your business to a point where AI can be useful for you is itself just as valuable as what AI will do for you.
With language models being where they are today, companies can get some really valuable insights out of these conversations, but like you said they do need to measure and store the right things.
Have you thougth in integrating it to update docs in Confluence?
I’d love the latter use case. A bot that scanned our internal engineering chat and answered our own team’s questions. I can provide many examples.
The difference is we don’t have public API docs. Instead we have code comments + confluence (reluctantly) + readme files in GitHub repos + the code itself. I imagine you could train off that and build the knowledge corpus over time?
In theory this should work just as well on Confluence as on Github - we'd love to try it out soon.
The main painpoint is probably around the UI for their cloud version. The web interface is not great, but if you do most of your testing with the client libraries it works very well.
As for wishlist - maybe I'm biased but ironically the docs could be better haha it's a bit difficult to find what you're looking for sometimes.
Do you have some stats on how often this even happens? In my experience, I have rarely seen these logprob thresholds actually get activated, because for binary token choices the probability of one of them is almost always >95%, but maybe this is because my prompts were not set correctly.
It happens pretty rarely, off the top of my head probably 5% of the time. However we do see a pretty wide range in binary token probabilities anywhere from low 50% to something that asymptotically approaches 100%.
- "Add to Slack" opens a Stripe page which says $49 is due today while the landing page claims there's a 30 day trial
- "Get Sidekick now" opens a contact us form
Have you had any instances of it writing a load of bullshit to a user, and if so how did you address this issue from a technical perspective?
We've also made some improvements in how we chunk and parse the content to make sure the information it finds is useful, since we've noticed hallucinations tend to happen when the context you give it is irrelevant.
And some other similar ones, but it good to see projects that will take a OSS approach, they are a few.
We at IngestAI[1] are in similar space albeit with different focus - we answer from all internal documentation in Slack/Discord/Whatsapp. We did a show hn just a few weeks ago[2] & has been wild since then - we have close to 10k+ users and counting.
If you would like build slack/discord/whatsapp/telegram bots answer your customer questions from internal docs - please try us
[1] : https://ingestai.io/
[2] : https://news.ycombinator.com/item?id=34909921
What seems to happen when the chatbot says "Hmmm, I don't know..." is that the DevRel team uses that to guide them to missing documentation sections. That's why we're building a whole analytics suite and documentation heatmap around the product to give insights about doc usage. Here's a quick demo [2].
[1] https://www.kapa.ai/ [2] https://www.loom.com/share/cb2b008141ae4ccda50b526a7597873e
Personally I think there's less value in an interface over the docs vs a community. Engineers might prefer faster answers from docs, but companies are going to prefer having their users in a central place where they can talk to them.